Originally Posted by LVBaja

Sounds like your linkage is binding just enough to keep stress on the servo. Because you said it goes away after letting up on the brakes, I'm going to assume it's binding on the throttle side. If you don't already have a throttle return spring on the carb, try that so that your servo doesn't have to do the work of pushing the throttle back to idle position.