The vxl motor comes with 3.5mm bullets, so the vxl-3S ESC is setup for 3.5mm bullets.

There should be females on the esc so you're saying that the motor you're trying to use has bullets that are smaller than 3.5mm?? You can either change to the matching size bullets on the motor, change the bullets on the esc to match the motor or get a set of adapters in the right size - something like these -

http://www3.towerhobbies.com/cgi-bin...?&I=LXPHD9&P=7

I use the adapters so when I sell a motor or esc, it has the orig bullets still installed on it.