Make sure there's no solder on the outside of the bullet connect. Then use a vice or something so that the plastic housing is supported, but the wires can dangle underneath. I use a hammer and a punch to tap the connector into the housing.

Also check the taper on the connector to make sure it presses in the way you think it should. Some press in from the front, some press in from the back.