Adaptive spam filter suddenly crazy aggressive [solved] by resetting training data
I've been using TB for many years, currently 60.5.3 32-bit on Windows 7. I get a lot of automated emails and a lot of spam. I think I understand how TB classifies spam/ham using naive Bayes or something similar. I have been conscientious to train the classifier in the 2+ years that this installation of TB has been running.
The account > settings > junk settings are: adaptive junk mail filtering enabled (and I've trained the classifier); do not automatically mark mail ... (in all my address books); trust junk mail headers set by SpamAssassin; move junk emails to Junk; and (don't delete junk)
and my general junk settings are: when I set the spam flag, move to account's junk folder; mark junk/spam as read; and log adaptive filtering
In the past couple weeks (I noticed the problem 4 days ago), TB has suddenly become crazy aggressive in classifying spam (identifying it as spam, moving it to a spam folder and setting the spam flag). In some cases, I unflag it (and TB sends the email to my inbox) and it immediately gets reclassified as spam and moved again into my spam folder. This is not behavior I've ever seen. The misclassification rate just a few weeks ago was close to 0% and biased in favor of slightly more false negatives and very, very few false positives (i.e., occasionally I'd see a spam email in my inbox, rarely did a legitimate email go into spam). In the space of maybe 10 days, about 25-50% of my legitimate/ham email is being classified as spam. It's like night and day and it's a maddening problem.
I checked with my ISP and they deny having changed SpamAssassin settings. I can see emails that were junked that have "good" headers. For example, I had to recover my username and password to post this and one email (from mozilla.org, that was marked as spam) had these headers, which is fine, right?
X-Spam-Check-By: mail1.g14.pair.com
X-Spam-Status: No, hits=-5.5 required=5.0 tests=DCC_CHECK,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,HEADER_FROM_DIFFERENT_DOMAINS,HTML_IMAGE_ONLY_28,HTML_MESSAGE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=disabled version=3.004002
X-Spam-Flag: NO
X-Spam-Level:
X-Spam-Filtered: 440df9e0c6f90fd3f1744f287d3847e0
I've checked that I haven't changed anything (I haven't modified the configuration of TB or any controls my ISP makes visible to me), my ISP claims not to have changed anything (and many, perhaps all, of these emails are not marked as spam by SA). I'm left with a few hypotheses: (1) TB has a defect introduced (e.g., recently in the code that classifies spam, or owing my the amount of data in my installation); (2) recent spam has poisoned the spam filter; or (3) the Bayesian model is shared across devices using the same IMAP account and a Linux laptop I've been using for the past three weeks has poisoned the model.
Regarding #1, I note that the adaptive filter log ends on 3/5 (around the time I noticed the problem). There's no log entries for mail that I watched get misclassified today, for example. I wonder if I have enough data to overflow whatever data structures could overflow?
Regarding #2, I find that hard to believe on two fronts. First, IIUC, there should be years of data (my adaptive filter log goes back to 2014), so I find it hard to believe spammers could poison the Bayesian model so quickly (I noticed the change rapidly, not gradually). To be honest, if this is true, I'd still say it's a design flaw in TB if spammers can poison the filter that quickly.
And second, while there are some emails similar to spam (all emails from myself and being marked spam; that could be because I get a torrent of "your account has been hacked" emails that are spam), but many other emails that don't (to my eye) bear any resemblance of spam emails are being mis-classified. If there's a common thread, automated emails (cron output, log reports, host monitoring, etc.) are very affected. But maybe that's a matter of base rates (a very large proportion of my emails are automated).
Regarding #3, I've regularly used TB on more than one device, including on Linux desktops, without this issue having previously arisen.
Solutions: I'm not interested in using any kind of filter or whitelist approach, because I will leave someone/something off and miss emails.
I suspect someone will point me towards instructions for blowing away the accumulated Bayesian model and tell me to retrain it. That's not the end of the world, but I get a torrent of email and I'd like to avoid retraining it if possible. Also, if the use of the Linux laptop (running whatever version of TB comes stock on CentOS 7) is causing the trouble, then retraining the model on my primary Windows computer won't actually solve the issue. And finally because that's just a band-aid and the problem with reoccur at some point.
I know first-hand how "black-box-y" Bayesian models are, so examining the model is out, I think? I'd be curious, though, if there are any tools available to examine the model or troubleshoot this issue. The log just shows the subjects of the emails that are classified as spam.
Modified
All Replies (1)
I went ahead and reset the training data and spam flagging is back to normal and surprisingly accurate. I haven't had to reclassify too many emails.