Firefox shows "Use custom settings for history" as an indication that at least one of the history and cookie settings is not the default to make you aware that changes were made. If all History settings are default then the custom settings are hidden and you see "Firefox will: (Never) Remember History". "Never Remember History" means that Private Browsing is active and "Always use private browsing mode" gets a checkmark. "Use custom settings for history" stays selected if at least one of the History or Cookie settings is not the default to make you aware that changes from the default setting have been made.

I have this problem too...I select "never remember history" click OK, restart firefox and...bingo, it's back to "use custom settings" again

It is not a problem. You only need to understand how this works in Firefox. "Use custom settings for history" doesn't make changes to settings, but if you see this setting then this means that not all settings are the default and changes have been made. If you see this setting and you would change the cookie and history settings to the default then the next time you will no longer see "Use custom settings for history", but you get "Remember History" or "Never Remember History" (PB mode).