i have just imported about fifteen to twenty years of MBOX files (using import export tools) that are now named in a sort of willy nilly fashion (like no naming convention whatsoever except to be able to keep the imported data separate and to not overwrite everything). this was done in order to import the complete mess of data i seem to have created. 

short question: i have about fifteen years of various email profiles and am wondering what the status is of DUPLICATE DELETION these days? specifically - is there something that would compare an actual message and an actual date before deleting? or what data are the various (?) software functions using to determine and delete duplicates? longer explanation: i have just imported about fifteen to twenty years of MBOX files (using import export tools) that are now named in a sort of willy nilly fashion (like no naming convention whatsoever except to be able to keep the imported data separate and to not overwrite everything). this was done in order to import the complete mess of data i seem to have created. meaning it is probably outlook data migrated to TB and probably back again, TB data migrated to mail and probably back again and then some other third party mail software that went toes up after buying it etcetera etcetara. SO i have about ten TB profiles worth of data i "archived" basically willy nilly because i didn't know what i was doing and didn't hire anyone to straighten this out for me like a dope. so now i am dealing with it. - anyway, are there any third party or TB methods for dealing with what must be a MASSIVE amount of duplication? what are the technical issues or tags or ID's involved with such a process? pic of some of the "naming conventions" of the mbox files i have been importing...