The collet is on as far as possible but im thinking I need a shim between the collet and bearing. The collet isn't thick enough to sit against the inner race and instead sits against a lip on the actual crankshaft, the collet will slide into the bearing Aswell.
5mm was probably a bit much but it slides back in at least a couple mm.
From what you say I gather any play isn't really acceptable so I think a shim would fix it. But how long will a tiny shim last in there?

Thanks for you input though mate! Everything helps