What sort of audio do you get from the internal speaker if you disconnect the headphones?
Does it sound distorted just like the headphones, or does it sound ok?
A few thoughts:
Corrosion or a dirty jack is entirely possible, as is mechanical stress on the jack contacts.
In any of these are suspected, and the internal speaker sounds like crap too, that still doesn't rule out the jack. On almost any piece of gear with an audio output where plugging in the headphones turns off the internal speaker (whether it be a detector or something else like a cellphone or boombox), the headphone jack has a normally-closed switch in it.
If the headphones are not inserted, the contact for the audio output presses against the contact for the internal speaker, and the sounds comes out the speaker.
When the headphones are inserted, those 2 contacts are forced apart and the output contact touches the contact on the headphone plug, sending the audio out the phones instead.
So dirt, corrosion, or mechanical fatigue in the jack could explain audio problems on both internal speakers or headphones. And sometimes it can produce strange things like no audio at all on either the internal speaker or phones, where neither are making contact at all. I had a palm treo cellphone once that did this after abused the headset jack enough (by doing things like slamming the cord for my wired headset in a car door and trying to walk away without realizing it, tearing the wires off completely) I had to use the bluetooth headset to use the phone at all after that until I got rid of it.
I'm not sure I buy the idea of leaving the phones plugged in being bad for the jack.
Can dissimilar metals react in way that cause them to corrode long-term? Sure, but I'll bet dirt and dust are a far bigger concern.
Also, if either the jack in the explorer or the plug on your phones is gold-plated, any sort of bimetallic reaction is unlikely.
And the constant plugging and unplugging of the phones will cause its own wear and tear on the jack, which might be worse. If you leave them plugged in, you'll get rotational wear on the jack from the plug spinning in the jack as your move around, part of why I loop my cord once around the back end of the machine and tie it off to reduce stress on the jack. If you unplug them, you'll still get that rotational wear while hunting, plus the wear from constantly plugging and unplugging, which may be made worse by any grit or dust present on the contacts.
The occasional use of canned air or a compressor to blow dust out of the jack isn't a bad idea though. A q-tip lightly (not soaking) wet with alcohol might not be a bad idea either, just be careful not to leave cotton strands in the jack. I suppose I should take my own advice and give my jack a cleaning..
Wear issues are a good part of the reason why the SE has a 1/4" jack instead of 1/8" like older models. 1/8" jacks should NEVER be used on equipment that's going to take a beating in the field, in my opinion.
Too bad it's so hard to get at the jack and inspect it, the way they put the machine together.