Can't figure out how to handle extra folders in Gmail
I have been using Thunderbird for many years with multiple mail accounts accessed by POP. However, I recently switched to IMAP so I can sync email with a smartphone. Most of my accounts are hosted on Google Apps, or GMail, and I am having trouble understanding how to handle the extra folders (really labels) that GMail creates.
I have five GMail accounts under different domain names and one outside account. In the past, I downed loaded new email using POP into Inbox. When I deleted, emails went into a local Trash folder. At least once a day, I emptied the trash and compacted folders on all my accounts. Every two months or so, I archived my active folders (Inbox, Templates, Drafts, Sent) using MailStore Home, and then I deleted the older emails in all my Inboxes. This kept the last about two months available in the Inboxes, and earlier email in the MailStore Home archive, and kept my Thunderbird folders from getting too big to function well. (I originally had indeterminate behavior when my local email folders got very big.
As suggested in a post I found, I disabled my POP accounts to not check for new mail at any time. Then I created IMAP accounts for each email using the automatic settings. This creation worked fine. However, the first time each account synced, GMail sent over the extra folders AllMail, Important, and Starred in addition to its Drafts, Sent Mail, Spam, and Trash. I really did not want multiple copies of my emails stored locally. I know Google sorts this all using labels, but Thunderbird seems to treat each Gmail folder as a real folder that should be updated with current emails. After searching, I set each account to sync only the last 30 days of email, and I also used Advanced under Server Settings to check for only Inbox, Drafts, and Sent Mail to be downloaded. I also noticed that AllMail contained several years of email (probably headers only at that point), around 20,000 messages, which I did not want on my local machine. I deleted all those emails in AllMail before 11/1/2015. I also added missing Sent mails from my local folders to the respective GMail Sent Mail files by copying them over from the disabled accounts. Later, Thunderbird seemed to download the actual emails whose headers remained in AllMail, so I had a 160 MB AllMail folder on one email account. I also ran the plugin Xpunge to empty the trash and compact folders on all accounts. This started around 1 pm and was still going at 10 pm. I let it run all night, and it apparently completed. I am not sure why it was taking all that time, but it seemed to be checking with the server regularly during the process.
Yesterday, I learned of the subscribe feature from a post in this forum. I used that to Subscribe only to Inbox, Drafts, Sent Mail, Spam, and Trash. This seemed to get me closer where I want to be. On all the GMail accounts but one, the extra folders went away (including the 160 MB AllMail) when I restarted Thunderbird. I have deleted emails going to the GMail Trash folder, and a copy of sent messages to the GMail Sent Mail folder. This looks much better, but I am not sure of what is happening. My goal is to have Inbox on each account contain all new mail and recent old mail. When I delete emails from Inbox, I want Thunderbird to sync with the server so I don't see those emails on the smartphone later. Again, I plan to empty the trash and compact folders regularly. In addition, I will archive Inbox, Drafts, and Sent Mail periodically into MailStore Home, and delete older emails as I did for the POP accounts. Is this going to work properly? Will I have all the full emails I want in Inbox, Drafts, and Sent Mail for archival? Can I empty the trash and compact files on these accounts without taking all night? MailStore Home will archive IMAP accounts in Thunderbird, but obviously not headers. When I delete the old emails in Inbox, will Thunderbird then cause them to be deleted on the server (I have set to sync only last 30 days)? Am I doing something here that will not work or does not make sense? I think it would be helpful to have a tutorial that goes into more of these issues when using GMail with its extra accounts that contain many of the same emails. Perhaps this thread can help others, also.
The other issue is that, on one email, the extra folders did not disappear from the left panel listing in Thunderbird when I unsubscribed them. I think I am only subscribed to Inbox, Drafts, Sent Mail, Spam, and Trash correctly. When I look at the choice of accounts under the server sync settings, for example, I am only offered those accounts. However, the unsubscribed folders including AllMail still appear in the left panel folder listing for that email account only. If I click on AllMail in the list, it quickly fills with emails. I am assuming it has downloaded headers only from the server. I don't know if Thunderbird will download any full emails later. This seems to be a bug of some kind. I am not sure how to get rid of the display of those unsubscribed folders. I gave up last night. When I rebooted the computer this morning, the Important and Starred GMail folders are missing from that account, but AllMail is still there. However, the profile folder on disk has no AllMail folder, just AllMail.msf, which I assume must hold the headers. Any suggestions on getting rid of that last unsubscribed folder? I think I read somewhere I could try just deleting the AllMail.msf file in that folder on disk, but I have not tried that.
Thanks for any help or advice.
Kaikki vastaukset (7)
In your profile folders for the imap mail account, the 'AllMail.msf' is only a file full of headers.
All Mail Gmail IMAP accounts have a All Mail folder which tracks every message. This is an artifact of how Gmail implemented labels, not a Thunderbird quirk. That folder is also used as the archive folder.
Tools -> Account Settings -> Synchronization & Storage for mail account Advanced is configured to keep a local copy of all IMAP folders on your hard disk. The safe way to unsubscribe and free the related disk space is: (Note this is the process if you want to remove anything subscribed/synchronised)
- Uncheck it in Synchronization & Storage -> Advanced button
As you do not have the mbox file called 'AllMail' no extension, it is possible that 'AllMail' was not selected to synchronise.
- Unsubscribe it - right click on mail account name in folder Pane and select 'Subscribe' - uncheck folders and clickon OK
- Help -> Troubleshooting Information
- click on 'show folder' button to view the profile directory using Windows Explorer.
- Exit Thunderbird now - this is important
- click on 'ImapMail' folder
- click on gmail mail account name
- Delete both "All Mail." and "All Mail.msf". ('AllMail' with no extension holds downloaded emails if folder was synchronised - the .msf file only has headers)
- Restart Thunderbird.
One thing you have to what out for is gmails habit of moving emails to other folders such as 'Important' without your knowledge. I thought all incoming mail would be in either Inbox or Spam. I had to get rid of these options via the webmail account.
note when you synchronise the last eg: 30 days. It will download the last thirty days the first time, but when it synchronises tomorrow, day 31, it downloads the additional new emails, but it does not delete the first day. So, after a period of time, you will discover the amount of emails you have downloaded will increase.
Deleting is also a bit different with gmail. You can delete an email from a folder eg: Inbox and when the account synchs with server to update server, that email will disappear from the server/webmail account Inbox (If webmail is already open rememeber to refresh the page), However, that email will probably still be in the 'AllMail' folder. I would recommend that every so often you check the size of the 'webmail 'AllMail' folder as it is the gmail Archive folder and also stores all emails both sent and received. Some people have queried about how they can only have a few emails in their inbox yet have eg: 96% quota.
If you want to get copies into Thunderbird, synchronising subscribed folders will do this, but those folders keep synchronising.
I have found best method is to synchronise the folders for offline use, then go into offline mode and use :
right click on highlighted emails, select 'Copy to' and select suitable folders in the 'Local Folders' account.
Emails stored in Local Folders do not synch with server, they are emails on your computer, just like the pop mail account. Easy to back up.
When finished, go back to online mode.
then should you delete emails (deliberately or accidentally) and lose them off the server, you still have a copy in Local Folders or in a backup.
re : I also ran the plugin Xpunge to empty the trash and compact folders on all accounts.
Useful tool which I've not used.
Just to let to know there is an alternative.
Tools > Account Settings > Server Settings for gmail imap account Bottom right - Message Storage
- Select: 'Clean up (Expunge) inbox on Exit.'
- Select: 'Empty Deleted folder on exit'
- click on OK
I also set the Account Junk Settings to delete after xx days. So all of this is occuring anyway when I exit Thunderbird.
To compact all folders: File > Compact folders or often I just hold down 'Alt' key and press 'F twice. when it is done - message in status bar -, I would exit letting the above do it's stuff.
Thanks for your extensive comments. I am beginning to understand. I do understand that these issues are not a Thunderbird issue but come from the way Google handles accounts. I still have questions about archiving locally and compacting folders.
FYI. I only had those extra Google folders such as AllMail set for download by default when I created the accounts. I created them using the automatic settings. I did see the synchronize all history or 30 days only because it is in the account settings directly. I changed this to 30 days right after creating each account. I only found the Message Sync Settings>Advanced that allowed me to uncheck to download those extra folders later. At that point I unchecked all but Inbox, Drafts, and Sent Mail for every account. I only discovered the Subscribe feature two days later when I was searching the Thunderbird forum in more detail than I did the first time. That seemed to get me closer to where I want to be except for the one rogue account. By the way, the AllMail folder listing in that one account did go away sometime the next day after the original post. I have no idea why it waited that long after I unsubscribed. I see you suggested I could just have deleted the corresponding folders in the profile if they did not stop being displayed. That is good to know. So I am now seeing only the folders I have subscribed on each account.
By the way, I use Xpunge because I have six accounts. That way I don't have to select the Empty Trash and Compact Files settings seven times (each account plus local folders). I did see that I could set Thunderbird to do this for each account on exit. However, I prefer not to do that since it could take a while and it gives me errors from time to time. I presume Thunderbird would not fully exit until that was all completed, which might not be what I want at that time. Xpunge lets me decide when to do that operation.
On archiving, I understand that I could just copy all the emails I want to archive to a local folder when I am ready to archive, and archive from there. That is much more trouble with six email accounts and multiple folders per account. I would prefer to archive from the IMAP folders if at all possible, but I need to make sure that the full emails are in the folder for the archive to work. I thought if I have all the folders I want to archive set to subscribe and download this would be the case, but I am not sure now. I also want to make sure there are no deleted emails or blank spaces in those folders that MailStore Home might try to archive, which is why I do the Empty Trash and Compact Folders before any archive. It is confusing because the IMAP accounts always show the same listing of emails whether they are actually in the account or just headers that can be downloaded. Are there ways of making sure the folders are ready for archiving, or can I be sure after compacting they are ready?
I noticed that Thunderbird will not empty the trash or compact folders for IMAP if offline. This surprised me because I thought compacting was just changing the local folder on disk, which does not change the existence of any emails. I guess Thunderbird also syncs on compacting and perhaps also on emptying trash. When I started using Thunderbird with POP, there were sometimes problems when folders were compacted and new mail received during the process, so since then I have been going offline to compact. Perhaps that is not a problem any more. Or perhaps Thunderbird is syncing during the operation, so I guess it would safely be in control of any download activity that would cause an issue. I want to be sure that if I compact while online there won't be those earlier problems, and that after the compact the folders are ready to be archived.
Another thing that made me confused. I added a local Templates folder to two of my email accounts, one of which is the same account that displayed the AllMail folder in the listing for a while after I unsubscribed it. When I tried to compact folders in that account, I got a message that the Template folder could not be compacted because there is not enough disk storage available. I noticed that I had messages displaying in that folder, but I had only a Templates-1.msf, no folder with no extension. I ended up transferring all the emails in the account to a local folder. After that, I could compact that folder. I then transferred those same emails from the local folder back into Templates. This took a while since I guess it was syncing with the server as it went. Now I can compact that folder. I now have a Templates-1 folder in the profile on disk, but is show 0 bytes. I seems like it sent them to the server and did not keep copies locally, even though that folder is set to download. This is an additional situation that makes me wonder if the IMAP folders will always contain the real emails.
The final question is on the 30 days. As you note, once I start downloading and syncing, those folders will continue to build up. What I plan to do (as in the past) is to go back every couple of months and archive all emails in the folders, then delete those older usually than two months. That way I have about 2 months of emails active in Thunderbird. If I do this, will Thunderbird tell the server to remove them from the Inbox there? The emails would be more than 30 days old, but the delete operation would be current. I am not sure how the 30 days works. I could either delete the old ones, or transfer them out to a local folder and delete from there. I guess it would be OK since the Google Inbox would not have them but the Google AllMail should still have them, unless I delete them into Trash, which Google says causes them to be deleted on the server.
I had some minor confusion with the Subscribe settings because of the buttons presented for Subscribe, Unsubscribe, and Refresh. It seemed like I had to play with them somehow, not just check the folders I wanted to subscribe and uncheck those to unsubscribe and select OK, but I may have just been confused.
Thanks again for your extensive advice. There is a lot of detail here that comes into play that is confusing. Any thoughts or advice you have is welcome.
I would add, for emphasis, that having all of gmail's folders/labels subscribed does not cost you much disk space. Although being subscribed to All Mail can increase sync overhead.
You've written an awful lot, which to me is a hint that perhaps you are trying to achieve something that isn't worth attaining. It sounds like mostly you just want to understand what gmail is doing, and hopefully Toad-Hall's excellent description has helped.
That said it totally escapes me what you are trying to fix And it's not clear to me why limit your desktop to only the last 30 days of email? (unless you are short on disk space)
Thanks for your response, Wayne. Basically, I am trying to get my 6 email accounts where they are synced to my smartphone and so they are available to me locally offline when I want them.
Years ago, when I first started using Thunderbird, I left emails in their folders indefinitely. However, I began to experience strange behavior with emails disappearing and the like. I learned from this forum that Thunderbird could become unstable if the mail folders grow very large. So I found a good archiving program MailStore Home, and I started archiving my local folders (after deleting spam and less important emails) every 3-4 months and keeping only the last 2-4 months in Thunderbird. All the instability went away, and this has worked fine for me. Essentially, I have all my emails locally with the last 2-3 months in the respective Thunderbird folders and older ones in the MailStore Home archive, which is easy to search.
The other reason for archiving locally is that I do incremental backups every day, one of the whole disk and one of all user files. Since the mail folders change every day, they will always be picked up in the incremental backups. I don't want large folders of emails I hardly ever look at increasing those back ups dramatically. MailStoreHome divides archived emails into smaller files that are indexed, so they hardly ever change once they are archived.
I hope this clarifies my requests. And, yes, I am also trying to better understand IMAP and Thunderbird and Google, for that matter.
You mention that subscribing to the extra Gmail folders won't take up much room. I assume you mean if they are not set to download. When I first set up my emails, using default settings, Thunderbird created several hundred megabyte folder files for Allmail, for example. That went away when I unsubscribed.
Now that I understand the subscribe and download options, I think I am set except for a few questions. I am set to subscribe to Inbox, Drafts, Sent Mail, Spam, and Trash. However, only Inbox, Drafts, and Sent Mail are set to download. (Specific emails in the other accounts will obviously download if I click on a given one.) I still have checked sync only the last 30 days. I want to locally archive Inbox, Drafts, and Sent Mail. Am I correct that I should be able to archive those three folders for each mail account directly from Thunderbird (with Thunderbird not running), with the understanding that Thunderbird will sync and make sure all the emails in those accounts are downloaded locally when it is running? Perhaps I need to compact folders before doing the archive, as I understand this causes folders to sync and clean up. Second, after I do a local archive operation, and delete the older emails from the local Thunderbird folders, will they be deleted in the server Inbox, also, even though they would be more than 30 days old?
Thanks again for the valuable advice and understanding.
marklang said
Years ago, when I first started using Thunderbird, I left emails in their folders indefinitely. However, I began to experience strange behavior with emails disappearing and the like. I learned from this forum that Thunderbird could become unstable if the mail folders grow very large. So I found a good archiving program MailStore Home, and I started archiving my local folders (after deleting spam and less important emails) every 3-4 months and keeping only the last 2-4 months in Thunderbird. All the instability went away, and this has worked fine for me. Essentially, I have all my emails locally with the last 2-3 months in the respective Thunderbird folders and older ones in the MailStore Home archive, which is easy to search. The other reason for archiving locally is that I do incremental backups every day, one of the whole disk and one of all user files. Since the mail folders change every day, they will always be picked up in the incremental backups. I don't want large folders of emails I hardly ever look at increasing those back ups dramatically. MailStoreHome divides archived emails into smaller files that are indexed, so they hardly ever change once they are archived. I hope this clarifies my requests.
I can see the appeal of http://www.mailstore.com/
marklang said
You mention that subscribing to the extra Gmail folders won't take up much room. I assume you mean if they are not set to download. When I first set up my emails, using default settings, Thunderbird created several hundred megabyte folder files for Allmail, for example. That went away when I unsubscribed.
I mean even when they ARE set to download. As mentioned, Thunderbird's implementation of gmail no longer keeps multiple copies of message accros folders.
marklang said
Now that I understand the subscribe and download options, I think I am set except for a few questions. I am set to subscribe to Inbox, Drafts, Sent Mail, Spam, and Trash. However, only Inbox, Drafts, and Sent Mail are set to download. (Specific emails in the other accounts will obviously download if I click on a given one.) I still have checked sync only the last 30 days. I want to locally archive Inbox, Drafts, and Sent Mail.
Makes sense.
marklang said
Am I correct that I should be able to archive those three folders for each mail account directly from Thunderbird (with Thunderbird not running), with the understanding that Thunderbird will sync and make sure all the emails in those accounts are downloaded locally when it is running? Perhaps I need to compact folders before doing the archive, as I understand this causes folders to sync and clean up.
Is it a feature of Mailstore that it accesses your server using imap directly, not going through Thunderbird? If it does, then it should work with Thunderbird running AND not running.
Compact cleans the Thunderbird folders of deleting imap mails. It doesn't specifically cause folders to sync. OTOH, I can see that it probably has that effect.
marklang said
Second, after I do a local archive operation, and delete the older emails from the local Thunderbird folders, will they be deleted in the server Inbox, also, even though they would be more than 30 days old?
absolutely, if there is no connection between Mailstore and Thunderbird
Wayne, that is helpful. MailStore Home does allow one to archive directly from Gmail and other popular online accounts. However, I am archiving from the local Thunderbird folders only. I rarely go online to look at Gmail directly, and I don't really want MailStore Home to find and archive a bunch of old emails. (Remember, I used POP up to now, so Gmail has lots of old junk.) Further, I don't believe my non-Gmail accounts can be accessed directly by MailStore Home. I refer almost exclusively to what I have in my local folders. That way I know what I am archiving because it is what I see in my Thunderbird UI. In fact, I go through the folders to clean them up right before I archive. That worked well with POP.
Archiving from local folders, MailStore Home will not do the archive unless Thunderbird is not running. I have always compacted folders before an archive so MailStore Home does not find any deleted emails. I am not sure what would happen if some local folders had only headers, without the downloaded emails themselves. Since MailStore Home, I believe, goes through the folders, not the index files, it would likely miss those, and I would not know they were missed. That is what I want to avoid.
On the compacting, I noticed that Thunderbird will not do the compacting unless I am online. In the past, it was recommended to go offline when compacting to make sure there was no interference with new emails coming in. That is what makes me think Thunderbird must be checking with the server through IMAP at the time of compacting. So I am now compacting when online, and hoping this leaves my local Thunderbird folders ready to be archived.