I would put some basic drainage in - some drainage pipes (just that plastic tubing, buried even) to carry water away and then a layer of crushed rock (not gravel, the finer stuff) on top of that to let the water through but keep the dirt above it in place. Then you will have a firm foundation to put the clay on top of. We just did this for a walkway we installed in our yard and the crushed rocks keep everything level but let the water drain.
This will at least keep your track from washing away or having huge puddles sit on it forever. Also, it will keep grass and weeds out better. I'm not sure what to recommend for clay that will make up the actual track surface. It probably depends on where you live as to what is available. I think you want clay as opposed to just whatever dirt is in the ground to keep the track surface hard and consistent.
That all being said - how much effort are you going to be putting into this? And money?