If you right-click the download button you should see an option Hide Button When Empty.

I didn't describe it well, but that's not what I was looking for. I want the downloads button to show up when I download something, to hide when the list is empty, AND for an item to automatically be cleared from a list once I've clicked on it once. Right now there's only the option to clear all downloads at once manually, which is not that convenient.

You can find "Remove From History" in the right-click context menu of an item. This removes the file from the browsing history unlike Delete that removes the downloaded file.

Downloads are considered history items: clearing history removes corresponding download items and vice versa.