Why would it react slower one side and not the other then?
I think if you're turning the steering wheel straight and you have good electronics, then the command should be received and executed promptly and with no hesitation.
S9551 is a good servo, but they do age and start slacking off. Of course this assumes all else is peachy keen.
Tamiya servo savers are crap in my experience and this is one of the problems I found with them. They tend to not push back enough to fight the steering and bring it back home all the way. Coupled with the sloppy mini steering system, it can easily result in weird problems (difficult to track down). Or perhaps is all an accumulation of ill effects. But as Granpa said, I am sure one can make do with a bad saver.
Do you have equal toe out for instance? I mean really equal? If you say sit the car still and turn the steering left and release the steering wheel on the radio, does the steering centralise? Test this by moving the car forward slowly after you turned left and released. If the car doesn't go straight, see why not. Or even worse, if you make sure the steering is centralised and you drive the car slowly without any steering input does it track straight? If it doesn't, start there, see why not. And so on.