One can set up the same drivetrain performance with 6cell and 4 cell batteries in the drivetrain. The key is picking the correct wind of motor to match the selected battery voltage. Where the experience will be different is the 4s setup will run hotter, and if the mAh are the same between the two packs the run time will be shorter.

In reality, usually for the same weight of pack, the 6S has a lower capacity than the 4s, but the higher voltage operation is more efficient so you still get longer run time.

No battery pack likes a lot of mechanical abuse, and I can't say I've noticed a difference in life between 4s and 6s in this regard. Enough hard landings eventually kill everything.