Will a 20v 1.5ah lithium battery work in rc vehicles?

Yes and no. The battery is probably frm a battery powered drill/saw?
It is probablt Li-Ion which cannot deliver huge powers some cars need.

Not enough mah.

Has anyone ever tried?

Yep. Don’t do it, it’s not worth your time and money, fitment will likely be pretty bad, and performance will be bad as well. Feel free to hook it up to a cheap RC from Walmart or something to see what will happen and get it out of your system. Don’t be surprised if/when things go “pop,” or stop working.

1.5Ah will hit LVC in what maybe 2-3 min if you're lucky, what's the point of even trying???