A car with negative caster (king pin line tilting forward) will be hard to keep on a straight line -- if not impossible.
If the caster blocks are flopped left to right, you could have a negative setting.
If the caster is ok (positive), your problem is likely the servo.