The Cadet corps numbered 274 men at New Market in 1864. The first VMI class in 1839 had 16 Cadets. So if you figured the average class over those 26 years had 200 cadets (a high estimate) then by 1865 there may have only been 5200 alumni. A VMI button from the Civil War would be extremely rare, assuming Alumni where wearing their VMI uniforms in the Confederate Army. Also, that is assuming that earlier VMI Coatee's had VMI buttons and not Ball buttons like West Point. Anyway, my point is a VMI button from the Civil War is rare! Unfortunately, the D E Evans backmark does not isolate it to Civil War era only.
Congrats on your find!