You can look at the network.protocol-handler.external.mailto prefs on the about:config page.

If network.protocol-handler.external.mailto prefs are user set (bold) then right-click and Reset them to the default value.

You can also try to change the network.protocol-handler.warn-external.mailto pref to true.

• http://kb.mozillazine.org/about%3Aconfig

Hallelujah!

Maybe I should have figured that out on my own, so extra Thank You!

You're welcome

I'm trying to get it to open in gmail. The about:config setting is correct, I believe, at false ... gmail is not external. And in the tools>options, gmail is set. But I get that error message

I had this problem too and it was driving me crazy. I then opened aboutconfig and searched for anything to do with mail. I found entries having to do with the Google Toolbar which I no longer have. I guess the entries are left even after the Toolbar is uninstalled?

One of these entries was: google.toolbar.mailto.gmail.configured;true

I toggled that to false and all was fixed. I suppose I could delete all the entries having to do with the Google toolbar, but this toggle was all that was necessary to get the mailto working with Outlook again.

I found entries having to do with the Google Toolbar which I no longer have. I guess the entries are left even after the Toolbar is uninstalled?

Will_T, You are correct. See http://kb.mozillazine.org/Changing_the_mail_program_used_in_Firefox#Extension_issues

The Google Toolbar for Firefox includes a "Send with Gmail" feature that changes the mail program used in Firefox to Gmail, regardless of what is set in Firefox options or your system default mail program. It is recommended that you do not select the "Send with Gmail" option in the Google Toolbar. Firefox already has a "Use Gmail" option available by default in the Applications listing for mailto actions. More importantly, uninstalling the Google Toolbar extension with the "Send with Gmail" feature enabled may leave you with Firefox opening nothing at all for mailto links. You can resolve this issue by resetting the Firefox user preference network.protocol-handler.external.mailto to the default "true" value in [[about:config]]. See Reset Firefox preferences to troubleshoot and fix problems for details.
Note: the Google Toolbar is no longer supported by Google for Firefox 5 and up.

where is the network.protocol-handler.external.mailto?

I also want it to open gmail, and it does not open mailto links. Firefox is becoming too much trouble. I plan to switch to Google Chrome. My mailto links work there.