Are you sure its shimmed properly? If you dont have the axial support you might want to try placing a nylon washer and pinion on the arm while you cut so it does not move around. My guess (based on a assumption you do not have the axial support) is that during the cut, it is keeping tension on the arm so there is no play/wobble. As you return there is no tension and the arm is free to do as it pleases... producing a very small wobble/vibration that is 'clouding' your cut.
Keep in mind a mirror finish is not needed for max performance. Your mirror finish disappears on the comm about 4 seconds after you run it in!
I would pay more attention to measuring the comm at both ends and ensuring that it is a perfectly even cut, all the way through.