Never touch the sleeve.

I wouldn't go messing with port polishing either. The fuel/air needs to mix around together and if your ports are smooth surfaced then there is less chance of the fuel and air mixing together if you had rougher ports.

So basically, your engine would perform better without the port polish because of the fuel/air mixing together.

Sean