Mail Folder Gets Corrupted with Thunderbird Daily
I am running Thunderbird Daily. I had a problem recently where a number of my mail folders got corrupted; mail messages were missing. So, what I did was to restore from a good backup before the corruption the folder and folder.msf files to a new directory. One corrupt folder is my Sent folder. I then copied the restored Sent and Sent.msf to my TB Local Folders directory as Sent-restored and sent-restored.msf (while TB was not running). I started TB and looked at the sent-restored folder. The messages are there, and they are intact. I then select ~2400 mail messages from Sent-restored and copy into Sent. When I look at the Sent directory, I see that all of these messages have been copied, and the directory seems OK - all the correct dates, recipients, and subjects. But ALL of the messages I copied now have the same text - the text of one (I believe) of the messages.
I have tried this twice, with the same results. What can I do to debug/correct this? Thanks.
Modified by Wayne Mery
All Replies (7)
I am having something similar happen also, but I am using the beta's. I was on Beta 5 and at least one of my folders with a few thousand messages somehow seems to have corrupted. It was fine one day, now, a few days ago I went in and I have a BUNCH of messages that are duplicates with weird dates, subjects and from fields and, about 90% of the messages are now missing.
Deleting the index doesn't help any.
If I look into the regular folder file (The one I am looking at is for Amazon), the file has a normal size and reads fine and looks fine in Notepad++. I can see the individual messages and all the correct info.
I have upgraded to 95.0b5 and the issue is still there.
I am by far not a programmer nor am I a DBA, but, if corruption happens, isn't there some sort of a script that can be run on the folder files to check and repair? (In my past, I did some work on an application that used a SQLite DB and our dev team had a script that just went through the DB and cleaned and repaired it magically). To me, this would be gold to have.
The mail file is in mbox format. In this format, each mail message begins with a line that begins with "From " starting in column 1. Any line that begins with those characters is treated as the start of a new message. That is why, when the body of a mail message has a line that begins with those characters, a ">" character is pre-pended to that line to ensure that that line is not treated as the beginning of a new mail message. If your mail file is not too large (I once had to extract a few mail messages from an mbox file that was over 2 Gb in size), you can check to see if it is in proper format. Try
grep "^From " mboxfile
to see all the lines that signify the beginning of a new mail message In one of my files I see
From - Sat Jan 11 21:01:54 2020
This line gives a timestamp for the message. In mbox format, following this initial line is the RFC 5322 mail header piece. Then the first null line separates the header from the body of the mail message. I would check to see if your file is in proper mbox file. I did verify that if the .msf file is missing, Thunderbird will rebuild it when that folder is first re-accessed. I am now back on the non-daily build (91.3.2), and I have not had a chance to test my message-copy problem. I will try to find a few minutes this afternoon to test.
The mbox file is a straight character-based file. It may contain characters in a non-ASCII character set, if a mail message is encoded therein, but the file is NOT a binary file. Any text editor should be able to open the file, assuming that it is not too large for the editor.
Note that in Windows 10 I have GOW (GNU On Windows) installed; it gives me the GNU Unix utilities - grep, sed, awk, et alia.
I just ran another test. I copied six messages from a sent-211118 folder to my Sent folder. The first copied correctly, but the other five did NOT. The .msf file showed the date, time, sender, subject of the mail message copied, but the content of the message was from another message. I now have to determine from where that message came. I looked at one message via cntl-u, and it is not a complete message; it does NOT begin with a "From " line. I cannot believe that my Sent folder or the sent-211118 folder are corrupted.
I did some testing, and the results are in a .txt file. I cannot attach it, so I will include it in multiple replies.
Thunderbird Sent Folder Corruption d:\computer\tb-sent-corruption.txt 05Dec21 0422PM Barry Finkel [email protected]
01) At some point many of my Thunderbird folders got corrupted;
mail messages from 3/2020 through 08/2021 were missing. I have no idea what happened.
02) I "restored" some of the folders from a weekly Monday
backup before the corruption to a new directory.
03) One corrupted file was "Sent", so I copied the restored
Sent file to the Thunderbird pop.att.yahoo.com directory as Sent-restored.
03) I then copied one mail message from Sent-restored to Sent.
Thunderbird automatically built a Sent-restored.msf file. The one mail message copied fine; it was intact, had the correct text, and the index information displayed (sender, recipient(s), date, time, subject) were all OK.
04) I then copied a bunch of mail messages from Sent-restored
to Sent. The firs one copied was fine, but all the rest were corrupted. The displayed index was fine, but the text of the mail messages was all the same. I have tried this a number of times, all with the same result.
05) I finally took my current Sent file and extracted (via grep,
head, and tail) all of the mail messages I had sent since the restored backup (sent-restored) file.
06) I combined the Sent-restored file with the new Sent messages
mbox file, and I created a new Sent file. This was done with Thunderbird shut down. Thunderbird then re-created a new Sent.msf file. This new Sent folder looks OK; I can search it to find messages. This was all with Thunderbird Daily.
07) I regressed from Daily to 91.3.2 on 11/30/2021, as someone on.
the Thunderbird Forum said that the Daily build had some known corruption problems.
08) I tried coping one mail message from Sent-restored to Sent,
and it copied fine. I then deleted the now-duplicate message in the Sent folder.
09) When I tried to copy two messages at the same time, The first
copied OK, but the second had the same corruption I saw with the Daily build.
10) I now am trying to determine from where the text of the corrupted
mail message came. The beginning text of the message in the mbox file (seen via CNTL-U) does not begin with a "From: " line. It does contain five complete messages, with the "From: " line "escaped" with the standard ">" character. The last piece of the message is an incomplete mail message; the end of a base64-encoded JPEG attachment is missing. There are 1,355 lines in the message.
11) I the corrupted text coming from the source file (Sent-restored)
or from the destination file (Sent)?
12) Right now I cannot believe that the two mbox files nor the
corresponding .msf files are corrupt.
13) The first line of the corrupted message has this text:
you like your email forwarded? And this text line does NOT appear in the source file. It appears three times in the destination file. This implies that the messages being corrupted during the copy operation are getting the text from the destination file.
14) The last line of the corrupted message has this text:
naM/LxzTbi9+0ZRV2r3FQCQLjjOKAHOrO4d2JY+lNKFVOM1bgD3cyrjAFbN1p8AiVUQh8DOa It appears ONCE in the source file and FOUR times in the dest file.
15) The 1,355-line message appears in the destination file in three
places, starting with these lines:
16627687
16629577
16631296 This is the last message in the destination file.
16) tail +16630933 Sent > \temp\sent-16630933
The resulting file has
a) a message I sent Tue, 16 Nov 2021 14:26:46 GMT
two null lines
the 1,355 lines that comprise the corrupted message
b) three messages I sent today (12/05/2021)
17) In my Sent file, line 16627300 is the beginning of a message
I sent Fri, 03 Dec 2021 20:39:56 GMT
tail +16627300 Sent > \temp\sent-16627300
The resulting file has 5,487 lines and contains:
a) the message I sent: From - Fri, 03 Dec 2021 20:39:56 GMT
b) a null line
c) the message I sent: From - Tue, 16 Nov 2021 14:26:46 GMT
d) two null lines
e) "you like your e-mail forwarded?" [line 388]
"naM/..." [line 1742]
"ou like your mail forwarded" [line 1744]
"u like your e-mail forwarded?" [line 1779]
" like your e-mail forwarded?" [line 1804]
"like your e-mail forwarded?" [line 1865]
"you like your e-mail forwarded?" [line 2278]
"naM/..." [line 3632]
"you like your e-mail forwarded?" [line 3997]
"naM/..." [line 5351]
f) the three messages I sent today: From - Sun, 05 Dec 2021 15:08:46 GMT [line 5353]
There is only one line with a non-ASCII character - in a message I sent today.
18) The index displayed for the Sent folder in Thunderbird has
the message I sent Dec 03 followed by the three messages I sent today (Dec 05). The displayed index looks fine, and each of the four messages looks fine. So, what in the Sent mbox file is causing the corruption?
D:\computer>cat tb-sent-corruption.txt
Thunderbird Sent Folder Corruption
d:\computer\tb-sent-corruption.txt
05Dec21 0236PM Barry Finkel [email protected]
01) At some point many of my Thunderbird folders got corrupted;
mail messages from 3/2020 through 08/2021 were missing. I have no idea what happened.
02) I "restored" some of the folders from a weekly Monday
backup before the corruption to a new directory.
03) One corrupted file was "Sent", so I copied the restored
Sent file to the Thunderbird pop.att.yahoo.com directory as Sent-restored.
03) I then copied one mail message from Sent-restored to Sent.
Thunderbird automatically built a Sent-restored.msf file. The one mail message copied fine; it was intact, had the correct text, and the index information displayed (sender, recipient(s), date, time, subject) were all OK.
04) I then copied a bunch of mail messages from Sent-restored
to Sent. The firs one copied was fine, but all the rest were corrupted. The displayed index was fine, but the text of the mail messages was all the same. I have tried this a number of times, all with the same result.
05) I finally took my current Sent file and extracted (via grep,
head, and tail) all of the mail messages I had sent since the restored backup (sent-restored) file.
06) I combined the Sent-restored file with the new Sent messages
mbox file, and I created a new Sent file. This was done with Thunderbird shut down. Thunderbird then re-created a new Sent.msf file. This new Sent folder looks OK; I can search it to find messages. This was all with Thunderbird Daily.
07) I regressed from Daily to 91.3.2 on 11/30/2021, as someone on.
the Thunderbird Forum said that the Daily build had some known corruption problems.
08) I tried coping one mail message from Sent-restored to Sent,
and it copied fine. I then deleted the now-duplicate message in the Sent folder.
09) When I tried to copy two messages at the same time, The first
copied OK, but the second had the same corruption I saw with the Daily build.
10) I now am trying to determine from where the text of the corrupted
mail message came. The beginning text of the message in the mbox file (seen via CNTL-U) does not begin with a "From: " line. It does contain five complete messages, with the "From: " line "escaped" with the standard ">" character. The last piece of the message is an incomplete mail message; the end of a base64-encoded JPEG attachment is missing. There are 1,355 lines in the message.
11) I the corrupted text coming from the source file (Sent-restored)
or from the destination file (Sent)?
12) Right now I cannot believe that the two mbox files nor the
corresponding .msf files are corrupt.
13) The first line of the corrupted message has this text:
you like your email forwarded? And this text line does NOT appear in the source file. It appears three times in the destination file. This implies that the messages being corrupted during the copy operation are getting the text from the destination file.
14) The last line of the corrupted message has this text:
naM/LxzTbi9+0ZRV2r3FQCQLjjOKAHOrO4d2JY+lNKFVOM1bgD3cyrjAFbN1p8AiVUQh8DOa It appears ONCE in the source file and FOUR times in the dest file.
15) The 1,355-line message appears in the destination file in three
places, starting with these lines:
16627687
16629577
16631296 This is the last message in the destination file.
Sorry for the mis-formatting/posting. Here are items 19-end.
19) I again did
grep -n "^From " Sent > \temp\sent-from-211205-1500 and compared it with the sent-from-211205-0822 file I created this morning. The only differences are the three (now four) messages I sent today after 08:22. So, the "^From " lines in the mbox file seem OK.
20) But ...
c:\temp>tail sent-from-211205-0822
16625167:From - Fri, 03 Dec 2021 01:55:27 GMT
16625201:From - Fri, 03 Dec 2021 03:02:55 GMT
16625427:From - Fri, 03 Dec 2021 14:47:04 GMT
16627237:From - Fri, 03 Dec 2021 14:53:48 GMT
16627257:From - Fri, 03 Dec 2021 17:20:58 GMT
16627280:From - Fri, 03 Dec 2021 17:23:51 GMT
16627300:From - Fri, 03 Dec 2021 20:39:56 GMT
16627324:From - Tue, 16 Nov 2021 14:26:46 GMT
16629214:From - Tue, 16 Nov 2021 14:26:46 GMT
16630933:From - Tue, 16 Nov 2021 14:26:46 GMT
This shows that there are three messages at the end of the Sent file from 16 Nov. These contain the corrupted mail message.
21) Where are the "^From " lines at the end of the Sent file?
tail +16627300 sent > \temp\sent-16627300
grep -n "^From " sent-16627300
1:From - Fri, 03 Dec 2021 20:39:56 GMT
25:From - Tue, 16 Nov 2021 14:26:46 GMT
1915:From - Tue, 16 Nov 2021 14:26:46 GMT
3634:From - Tue, 16 Nov 2021 14:26:46 GMT
5353:From - Sun, 05 Dec 2021 15:08:46 GMT
5400:From - Sun, 05 Dec 2021 15:23:16 GMT
5434:From - Sun, 05 Dec 2021 16:01:40 GMT
5488:From - Sun, 05 Dec 2021 20:42:06 GMT
So, at the end of the 03 Dec message, Thunderbird is not recognizing the three "16 Nov" "From" lines and treating them as the starts of three mail messages.
22) Where are the null lines in this "end-of-Sent" file?
c:\temp>grep -n "^$" sent-16627300
16:
20:
22:
24:
45:
50:
53:
55:
61:
386:
387:
393:
409:
424:
426:
428:
435:
450:
452:
454:
464:
485:
491:
495:
501:
516:
518:
520:
523:
538:
544:
546:
547:
564:
582:
598:
608:
613:
630:
635:
640:
645:
652:
1743:
1749:
1765:
1778:
1784:
1800:
1803:
1809:
1825:
1840:
1842:
1844:
1851:
1864:
1870:
1886:
1901:
1903:
1905:
1912:
1914:
1935:
1940:
1943:
1945:
1951:
2276:
2277:
2283:
2299:
2314:
2316:
2318:
2325:
2340:
2342:
2344:
2354:
2375:
2381:
2385:
2391:
2406:
2408:
2410:
2413:
2428:
2434:
2436:
2437:
2454:
2472:
2488:
2498:
2503:
2520:
2525:
2530:
2535:
2542:
3633:
3654:
3659:
3662:
3664:
3670:
3995:
3996:
4002:
4018:
4033:
4035:
4037:
4044:
4059:
4061:
4063:
4073:
4094:
4100:
4104:
4110:
4125:
4127:
4129:
4132:
4147:
4153:
4155:
4156:
4173:
4191:
4207:
4217:
4222:
4239:
4244:
4249:
4254:
4261:
5352:
5368:
5370:
5381:
5393:
5397:
5399:
5418:
5420:
5431:
5433:
5451:
5460:
5467:
5471:
5479:
5482:
5485:
5487:
5505:
5507:
5509:
5514:
5516:
5517:
5518:
5559:
5560:
There are many null lines in this file, and there is a null line preceding each of the "^From" lines. So, I contend that the end of this mbox file is in proper mbox format. There is a null line before each "^From " line.
23) The only thing I see that could be causing the problem is
that the three 16 Nov messages are not in date sequence. I just looked at RFC 4155 (mbox format), and I do not see anything that says that the mail messages have to be in date order. Is Thunderbird getting confused? It does not show these messages. But I do see four messages from 16 Nov in the Sent folder as displayed by Thunderbird. The Subjects displayed are:
Where our Members Live
Where our Members Live
Re: Review of Recent Activity
Re: Review of Recent Activity
There are different "^Subject" linies in this file, as there are multiple messages in the last three 16 Nov mail messages. If there are multiple "Subject:" lines in a mail message, which one does Thunderbird use? I asume the first one. But I see in the file only one "Where Our Members Live" subject line and no "Review" subject lines. Thus, I conclude that maybe the Sent.msf file is corrupt.
25) Shut down Thunderbird, rename the Sent.msf file, and
restart Thunderbird. The Sent folder now displays messages from 16 Nov properly.
26) Copy two 15 Nov messages from Sent-211118 to Sent.
Both messages appear fine in the Sent-211118 Folder when I view them in Thunderbird. The first copied properly, but the second has the same corruption as before. Now I have no idea what may be causing the corruption.
Your issue is likely related to https://bugzilla.mozilla.org/show_bug.cgi?id=1742975
It seems likely. Is there anything I can do to give more info for the developers? My case involves copying/moving messages from one folder to another; the cases in the bug report seem to be different. For now, if I have to copy a few messages, i will copy one-at-a-time. For larger blocks of messages I will wait for more info on the bug. Thanks.