Think the answer might be an outrunner motor like used on some airplanes. If I remember correctly they run much slower than the inrunner style used on most surface RCs, and still provide good power.

Brushless motor speed is typically specified by a kv value:

RPM = kv times voltage

One place that might be helpful is the Power Systems forum on RCGroups. Lots of custom power setups are discussed there.