Is it a sensored setup? If it is verify you connect the A-B-C wires from the ESC to the motor are right, if something is crossed over it will cause issues. Also unplug the sensor wire and seat it back down, if there is any hint of damage on it replace it. Still having issues just replace that and see what it does for you they are cheap and they do fail often.

As for the heat, that can be the A-B-C wire issue being out of sorts, but I would make sure the bullet connectors are nice and snug, if they have any slip or spin to them will get an arc and that will cause heat in the connector.

beyond that it does help to get another set of eyes on things. After you hit 10 post you can link in pictures or videos. But if you have a local hobby shop or track that may be good option for those extra eyes.