Alright now that we know what ESC it is lets see if this helps.
First set all your settings on your radio accordingly to the radio chart listed below.
Next, reprogram your ESC and if the ESC has drag-brake, disable it, if not then don't worry about it.
Now, check the mesh between your pinion and spur, are the teeth too close to each other? I don't use the paper trick but rather eye ball it to where there's about a 50/50 amount of teeth contact of both the spur and pinion. If there is not enough mesh(pinion fully flush with spur) then yes this will simulate drag brake and overheat a motor.
Next, check for freeness in the drivetrain to make sure everything is bind-free.
Other than that all I can think of is not running a diode on the motor/esc but this advice will void an esc warranty if you cook one. With mod motors, always at least run one diode on the motor-this is the black cylinder like thing with a silver strip near one end that solders onto the motor, not a capacitor like on an ESC.
M8 radio settings chart for novak esc's
http://www.teamnovak.com/How_to/tran.../trans_air.htm