Sensored motors are generally wound in a different fashion that causes them to run poorly without sensors. Some faster winds(IE:lower turns), will run decent, but anything over 13.5 will almost always run very poor on a sensorless controller. There are exceptions, and I am not familiar with your motor, so dont take my word for it. If the manufacturer says it can run sensorless, and it performs the same way on a sensorless controller, then I would suspect it is a bad phase in the motor. The symptoms you described could be attributed to many possibilities, bad sensors, bad phase in the motor, blown MOSFET in the ESC, or a poor/failing connection between the ESC and motor. Without expensive testing equipment I dont think there is any way to test brushless components other than known good components.