what is it doing to come off? Is the set screw just coming loose?

take a dremel with a cutting wheel and lightly score the set screw where it would touch the flat part of the shaft. This will create a slight serrated tip that will grab the shaft. You can also use blue loctite.