This is a possible solution, although I'd be weary as it may weaken the hub.
I use very thin shims either side of the castor block, careful not to overdo it as it will bind up the suspension if tight.
I agree most of the slop is in this joint, Ive also been keen to find a good way of removing the slop on the right side bellcrank, as its a combination of the slop in the trackrod ball cups, and the right side steering links cups.
One method I use to date is to snap some plastic from a parts bag over the ballstud then pop the ballcup on over the plastic, tearing away the excess it leaves a little plastic in there to pad out the extra space without binding it up.