Current ESC Designs
 

I have a question about ESC's. If you run a brushless system with the correct S rating on a lipo battery do the esc's fry or do they just shut down and turn back on when cooled? I may have worded the question weird to some people.

I assume you are referring to "S" as the battery voltage (1S, 2S, 3S....)? If you are running the within the specified input voltage, there shouldn't be any problems. If you went over the the maximum input, then you might have blown something up.

What exactly happened?

Depends on the ESC. Some have thermal protection, but as far as I know many don't. Even with thermal protection I wouldn't assume it will prevent all overheat damage. Best to get an inexpensive temp gauge and use it on both motor and esc, especially after making a change. One example, there are other brands too:

FlashPoint infrared temperature gauge

esc/lipo
 

As I'm understanding your question, you put a 2s lipo in a vehicle that came with a NIMH battery. The issue's I see are the gearing may need to be changed because of the power capability of the battery as well as how long your run time is. Your other problem is how or if the esc has a lipo low voltage cutoff, so you don't ruin the battery. If the power system was a brushed motor? You get more pop, so you have to reduce the gearing to keep the motor from overheating.