It may sound obvious but make sure to check that your throttle linkage is setup properly. If the return spring isn't tight enough to return it when you bring it in and hit the brake it comes to the "right" idle and when you let off it bumps the linkage just enough to give it just a tick of throttle. A quick test is when you stop it and the idle comes up hit the brakes again if you have let off and see if it returns to low idle. If it don't then try to richen the bottom a tick or check for a typical air leak, or exhaust gaskets that may be worn.