I finally manage the track the root cause of the issue. It seemed to be that the radio receiver was interfering with ESC when applying little throttle, only when the car is on the floor. I receiver was mounted next to the ESC originally.

When the receiver was mounted away from the ESC, the problem went away. How weird!