One thing I noticed was I used to wear my worn out jeans to metal detect and they had holes in the knees and looked like I was a bum. Since I started wearing new jeans or khakis to public places and not looking like some homeless dude, I get more smiles and how you doing from people. I'm not advocating dressing up to metal detect, but you shouldn't look like a hobo either. The first impression you make is the most important one and if you follow that first impression by using smart recovery methods, you add to your reputation as someone to be trusted. It's all about trust, I believe.
Lou Holtz, famed football coach and one of my personal hero's, asserts that this is the fundamental human question - Can you be trusted?
And as you note, this trust has to be earned face to face. It is the root of respect, and all other human interactions.
I've had people accuse me of being a scavenger for metal detecting, but I've learned to try and be diplomatic and educate these people. If you confront their fear with anger, you reinforce their fear in the first place and justify their lack of trust.