I have a basher sabertooth with a 100a esc and I'm trying to find out what's the biggest can size motor I can use before the esc gets to hot?

Most brands of motors will specify a recommended rating for the ESC... based on the picture you provided, it kinda looks like you are already maxed out on the can length, is that a 36mm diameter can and are you trying to increase the diameter to something like a 42mm can? If that's the case, do you have an 3mm of clearance under the existing motor?

Generally most 4274 cans on 4S will require a 150A ESC, chances are you might get away with running an over rated motor on a 100A ESC, but may be limited to gearing, roll out and duration of run time plus surface conditions before you start thermaling the ESC.... if it were me, I'd play it safe and get a 150A ESC.