Other than go somewhere that isn't trash infested, you can do the following:
1.. Use the 8" coil to get some of the older and deeper coins that you might be able to between the trash that you can't with the 11" coil.
2.. Go with a 5" coil and do a better job of picking between the trash than the 8", knowing that while you won't get quite the depth, you'll still be able to hit on higher-conductors between some of the trash that mask them with the 11" and 8" coils.
3.. Grid and clean a site, hunting in ALL METAL (0 iron mask) so as to remove ALL targets to include the desired coins, and the masking trash and good targets they hampered you getting before.
For maximum success, remove the maximum number of targets. #3
To be able to pull some keepers in very dense trash, you can't beat a small coil (#2) that is worked slow and methodically.
If you just want to do "better than average," use the medium-sized coil (#1) and cross your fingers. That's what you're doing now, so you have two choices left. Get a 5" coil and get what you can still,knowing some good finds will remain due to masking, or do you best to clean a site out.
There are times I like to use an 11" coil when I know that I can get the best depth and coverage when hunting a site that has very few to targets, especially those that can cause masking. Often I will use a mid-sized coil if there are an 'average' number of undesired targets AND I am working a site that I don't plan to revisit often. I'll get what I can. When trash is rather heavy, I slap on my lower rod w/5" coil and cherry pick all what I can. larger coils are almost a waste at such suites because a dense concentration of trash means I'll have a lot of crappy signals and little chance of making a good recovery. The 5" coil ups my odds.
if there is a vacant lot i can visit often, or if there is an old park where I can grid off an area and spend ample time on several trips to scour it of any metal, then I still start with a 5" or mid-sized coil. I'll be pulling all targets, but I will hear more decent hits in amongst the trash and that can alert me to using more care in a recovery. I'll still yank the litter form the site. When "all is clear" I make sure I work the 11" over it for any remaining deeper targets.
Now, here's the interesting part. I've been detecting for over 45 years, and I preferred (still do) to use coils in the 5"-7" size for most of my hunting. I have worked old lots and parks and homesteads and military encampments and such, either alone or with others, and have concluded the following. If a site has little to no trash at all, and if there are coins and tokens there in honest the 5"-8' range, a larger coil like an 11" ought to do just fine. However ... I have also found that most of the time, even at older sites, unless there has been a lot of ground disruption (plowing for example) or a lot of deposition, good targets just are not all that deep.
I have worked with others to clean a site out, most of us using coils in the 4