Sensored type motors often use a different type of winding pattern than sensorless motor designs. With a lower resistance, high Kv motor, almost any ESC can power it up smoothly and keep the driver happy. But, when it comes to lower KV, high resistance motors, like a 17.5, most sensorless controllers will have a very hard time starting a sensored motor in this range without cogging. There has been some success stories floating around on the forum, but more often than that you will find many posts similar to yours. Just to be sure, I do suggest checking your connections very closely. The ESC settings can make a small difference, but it sounds like you are having a pretty big cogging issue. Also, make sure you watch temps closely, cogging can heat up a motor to dangerous levels before you know it.

I would suggest picking up a sensorless design motor. You can find them very cheap at places like HObbyPartz.com and they are proven to perform very well. On the proper motor these Hobbywing ESCs are some of the smoothest I have ever driven.