One of the big problems with latency, is that with a lot of systems it's quite variable.
Like GerryH said, humans can adapt quite well to latency and time things with extreme precision. You can adjust, and likewise, with slow servos, your brain will automatically just learn to turn the wheel a tenth of a second sooner to deal with the 100ms it may take for your servo to turn.
The problem is that with a lot of the higher latency radios, the latency is quite variable. Sometimes it will start to react in 10ms, sometimes it will take 50ms. Your brain can't adapt to this random latency, and it makes it literally impossible to run a totally consistent line on a fast car, because even if you time it exactly the same on 2 consecutive laps, The slow one has got an extra 1/20 of a second of travel before you hit the brakes, or start steering, as compared to the faster one. A car can go pretty far in that amount of time.
The cheap CPPM based transmitters are probably the absolute worst for this, just because of the way they work. They're asynchronously reading all of their pots, and then sending them in a continuous string of signal pulses one after another. CH1, 2, 3, 4.
What this means, is sometimes you hit the brake or steer right as it's about to send that channel, so it goes out the radio almost immediately.
Other times you hit it just after it has sent that channel, and it will have to send all of the other channels, and an end of frame delay, before it gets back to sending your signal again.
Add this to some similar timing variability on the receiver end, and you really just can't drive them well at high speeds.
I went from a cheap hobbyking gt3b to a sanwa mt4, and the difference was immediately obvious, even when I was just bashing around for fun. Not so much that it felt "faster", as that when I turn a corner, and aim for the apex, I would be reasonably close. (or miss it, but could feel what I did wrong, at least) With the gt3, I didn't actually notice the latency consciously, but I would overshoot a corner one time, and then run into it the next, and it just felt sloppy. I didn't know what I was doing wrong.
When I filmed it at 240fps, I was amazed to see just how much latency there was. (IIRC it was around 20 frames)