Swapping bearings should be the exactly the same. Sounds to me like the flanged bearings inside the hub are dead. They die with the thrust washer on them, because they were not designed for lateral loads. Replace them every other time you rebuild your diff.
Also, make sure your bearings are all the same dimensions. Sounds silly, but I have accidentally used thinner bearings in things without realizing the dimensions were off.
doug