Nope, dont remove the ends until you want to take them off to replace new ones. As you mention the BB25 part will fall off otherwise.
There will be a new direct coupling out soon for TA06/417/418 which will just fit on and allow you to use the swing shaft protectors.