"John67"
What your trying to do is normally very simple and I have the same setup for earbuds in my take along detector bag. The adapter to size the 1/4" male to a 1/8" female works really well.
In your case you may have the right adapter but it could be defective, if you can dig up the package it came in there should be enough information on it to determine if its the correct adapter or not.
One point in my above post that I was trying to make is to confirm your headphones and earbuds have audio in both ears, if you don't have a small audio device to test your earbuds you can just plug them into the headphone jack of any computer! (or laptop)
Once your equipment is tested working then the fault falls to your adapter and that being either defective or the wrong thing even though the ports on it are the right size. The adapter you need is very common.
Now, the whole idea you have for using earbuds is also a good option to have! certain locations with very little surrounding noises and where your not hunting extra deep the earbuds are GREAT! they are very lightweight and they are for sure not as hot on the ears.
Mark