That red dirt is either clay or it has a high clay content. Therefore it packs together well and will in some cases, "bluegroove" meaning it gets rock hard and gets super grippy. So much so that the track will develop a blue shade where the racing line is. This is due to the tires wearing and leaving behind rubber.
Makes for a very fast track, but less forgiving in a crash
If you're having problems with dirt sticking, spray WD-40 on a toothbrush and give your car a rub-down with it. When the WD-40 evaporates, it leaves behind the oil, which the dirt won't stick to. In extreme cases, you may just wanna spray the WD-40 directly on the car.