It could reduce your download size not to include those if you are providing all the files anyway, but if you aren't going to include all the files which refer to them, then it just means extra files to require downloading (if you remove a bunch of units you might have to include Python, if any unit is specifically mentioned in that, probably the Events folders if mentioned there, and the Civilizations if there are UU versions of it, and probably Technologies if it is unlocked by a Tech...)
XML will never affect how the gamespeed is after loading though. Once the XML is loaded up, it is never touched again, all the data exists in the SDK and all functions are done with SDK or Python only.
So end of story: Cutting XML will speed up load times (marginally), and reduce download size IF you were already including all files which reference the data.