When I got my F104, it had music wire attached to a shitttty tamiya servo horn. I replaced those with turnbuckles and an RR servo saver I had in my spares box. Also make sure the servo tabs were trimmed off flush and the servo properly seated. There are no bolts touching the servo on a stock F104, so this is very important.

I noticed that having a little toe-out helps the car straighten itself. 1' is more than enough.