Frank,
There is no formula that I know of as to the size of the receive coil. However, the general standard is to make the receive coil diameter 50% of the transmit but there has been various other sizes with most between 35% and 70% of the TX for the RX windng.
There is a discussion going on right now on the Geotech forum as to what might be the best size ratio of the tx and rx coil. The discussion is more about the ideal SNR ratio, but all aspects more than likely follow similar results. As for where the center coil is placed, the normal placement is centered inside the outer transmit winding. Based upon what is coming out of that discussion, the 50% ratio is a good safe size to use. So as an example, if you build a 12" coil, then the receive winding should be 6" diameter. In your case, the 18" coil with the 6" diameter receive probably should have worked while centered. Since it didn't I suspect the inductance is way off on one or both of the windings.
Now, with that said, I don't recommend you try to match the resistance of your orignal coil. Furthermore, I am not sure how you do this for a concentric tx/rx coil? Matching resistance is only important for ML PI's and doesn't mean that much for most other designs. Even then, are you matching the wire size? Resistance isn't relative to anything without the size of the wire used.
What detector are you trying to build this coil to use on? Most PI's are very forgiving when it comes to coils but there are some basics that should be thought about. First, work with inductance and not resistance. Second, is the detector a low power PI and if so, the resistance isn't much of a concern.
As Eric Foster mentioned very recently on the Geotech forum, an inductance meter is a must. Considering the fact you can buy one for about $40 off ebay, it makes sense to have one.
What inductance do you have on your 3" coil? If you don't know, then how many turns are in the small coil? Did you use any program to determine the inductance?
As to why your coil didn't work, well it sounds like you don't have the right inductance combination as well as a good tx to rx ratio.
You need to read the various coil building articles on the Geotech forum and a lot of the discussions. There is a lot of good information for a beginner on that site.
One last note, when trying something new, I recommend you stay with safe values such as the same size as your factory coil. In other words, if your factory coil is 11" diameter, then try to build a concentric coil the same size first and get it to work right. From there you can try different sizes. By following this procedure you will most likely have better luck.
One last note, concentric means everything shares a center focal point or axis. Thus all windings are centered within any larger one.
Reg