Just wondering if using the wrong wire size and type of wire on your motor can cause an esc to burn up?

No. any copper strain based wire between 14-10gauge would work reguardless of brand.

any wire works as long as you can solder it. 12 14 gauge noodle is the right way to go for flex. if you use stiff wire then it will break after flexing. if you plug you control for 1 mil a second in the wrong way, its toast. plug the battery in the wrong way its toast. plug the speed control in to the receiver the wrong way they are both toast. take your time. a to a b to b.

The wire wouldn't fry an ESC but a bad solder joint can by shorting out the ESC.

Too small a gauge of wire can also cause excessive wire heating, leading to drama if it gets too hot.

Depends on the load. In 1/12 scale we use 16ga wire in 17.5 with no ill effects. Also in crawlers.

^ This. Always match wire to the load.