I was just messing around with it again, and it seems to be much less of a problem (if not non-existent) with the high torque servo. (The other two servos I used to check it were just a couple I had lying around)

Maybe it's just something about the race servo then?

I'm running this in a Tamiya Mini, so I don't really think torque is much of an issue, especially while the car is on the bench with the wheels hanging in the air.

I'll install the high torque to see what happens.