They say 8.5 is the maximum motor size you can use, but has anyone tried a 6.5 on it? Could you use a fan on the esc to do this? and how would you wire it in?

Don't. The esc is not designed to do it. It also has no external heatsink, so a fan would do next to nothing.

You need the rs pro or the new rsx

+1. You'll just be throwing your money away if you connect a 6.5 motor to it.