your batteries, motor, and esc will get hot if your buggy is over geared, meaning that your gearing is a tad too high. if you lower the buggies gearing, it should run cooler.
to do this, you can use a larger spur gear, or a smaller pinion gear, or vice versa. i definitely think that brushless is the way to go.(provided you have the funds for it) brushless motors are more efficient than their brushed counterparts, and will save you money in the long run because you wont have to buy replacment brushes for it.