Not sure if there is an area in mind, but once upon a time (probably about 25 years ago) SLO P&R allowed a track behind the main/stadium baseball field at Sinsheimer park. It was near the school district's bus parking lot/maintenance yard and train tracks. Noise was not an issue. Not sure if, with the housing developments, that would still be the case.
As I recall, P&R supplied the water and the 'club' paid for the rented chain link fence (to keep bicyclists off the track). Not a great location for visibility to attract new racers, but there were a couple of hobby shops at the time that had flyers promoting the track.