a clue in your post is that you say you got hassled by "the feds". So is this to assume that you were on federal land? There's a lot of people on md'ing forums who have simply taken the stance to avoid all federal land (stick to state or city or county or private, etc...). Why? Because ARPA, which is a federal law (and thus only applicable to federal sites, unless specifically drawn in by inclusion on some lower/lesser level entity by-choice).
However, with that said, believe it or not, not all federal land is off-limits to detecting. For example: NFS land and BLM land are not altogether off-limits (even some specific verbage ALLOWING detecting, and not simply a matter of "silence on the subject). However, the "gotcha" is going to be that there *might* be some cultural heritage verbage (if you asked enough desk-bound bureaucrats), that disallow the taking of older items (I guess over 50 yrs. old blah blah). So for example, you can hunt for new items, or gold nuggets, or meteorites, etc..
So I'm not sure what type of federal land you were on. Perhaps you want to expand a little?
I was stopped one time in NFS land, while hunting a CCC era campground. The ranger barked "you can't do that". However, after some casual conversation, he changed his tune and said "well.... you can .... but you must turn in any coin over 50 yrs. old to the ranger station". Doh! (my math never was that good).
And also: it sort of depends on what you mean by ghost town too. Because if you mean that in the glamorous and colorful sense of the word (standing buildings like Bodie type of "sphagetti western" move scene), then perhaps you were ... uh ... not using a little discretion on the type places you hunt. Because go figure, no matter WHAT entity governs/manages a site (whether fed, state, city or county), you've got to figure it's always not a good idea to be hunting around "obvious historic monuments". That's like swatting a hornets nest, no matter whether there really was a rule or wasn't.
But if you meant just lone foundations (or a naked spot where nothing is left to mark the site, etc...), then .... people are going to care far less about that (especially if it's out in the middle of nowhere). JMHO.