sounds like it could be something as simple as your linkage being wrong based on that statement. Sounds like at the neutral position your linkage has the carb pulled slightly open, then when you hit the brake it closes the carb further to it's idle position. Watch the carb slide and boot when you give it brakes, does it close slightly when brake is applied. If so the return spring and/or the blue collet is not set correctly. Should be a lsight bit of daylight between plastic swivel and blue collet, and there should be slight preload on return spring.