I don't have a F104, but I've seen someone at our track working on setup. I don't know how much tuning experience you have, so I don't want to setp on your toes, but I can give you some ideas.
Try stiffening the front up, and find ways to slow weight transfer. When you get off the gas the rear end will get light and come around on you. Use traction compound on the entire rear and half of inner tire or less on the front. Get the car on scales at each corner and try make sure the weight balance is even left to right. Also see what the current weight distribution is. Try shifting more weight to the back and make sure if you have any more aero adjustments to get the back planted more.
Usually as the day goes on traction goes up and the car oversteers badly, but I don't know why your car just snaps around when you get off the throttle going straight. Sounds like the chassis is very upset. My friends at the local track are switching to the lightest springs because the stock springs are way to stiff for the light lipo's they are using in the car.
TC6, Ten SCTE, Novak, Fantom, Savox, Spektrum