The only thing that does not scale to speed is movement. Growth of cities, improvement and building time cost are scaled. Research is scaled, whipping, chopping, GPP, etc, etc, etc.
Quick speed = More years per turn = slower units = + defense, -offense
In normal/quick speed, the defender has many years worth of hammers, etc (even if the same turns) to put into defenses.
Epic/Marathon (slow speed) = Less years per turn = faster units = + offence, -defense
Epic or slower means the defender gets little hammers/etc during the turns your troops move to engage.
Due to the cost of troop movement many games at standard+ size almost require epic or marathon speed, otherwise attacking 20-40 tiles away from cultural borders (domination/conquest victory) can result literally in obsoletion during march or insurmountable emergency defenses. While standard size/normal speed conquest games (1100ad) are easy at noble, on higher difficulty levels your 2nd, 3rd, or 4th opponent will be formidable (eventually someone else will have catapults or longbows).
As a warmonger I prefer small size + normal or even quick speed with a couple extra AIs to standard/large size and teleportation-like marathon/epic speed.
Total required playtime for the game is increased by larger map size and slower speeds. Great distances and many clicks = long games.
Overall difficulty is another matter altogether. If small is too easy for the warmonger, up the difficulty or increase the speed. If large is too easy for the expansive/builder, up the difficulty or slow the speed. When you are slamming deity level conquests on small maps (+2 AI?) at quick speed you can say small maps are too easy for warmongering. I dont like spending 3-4 hours moving multiple armadas and divisions across a standard size map for the endwar, I find challange on small maps (shorter games). I add the 2 AIs for 2 reasons: so that you cannot simply mow 3 people over and the game is done, and so that diplomacy is more intricate.
Normal and quick conquest victories at monarch/emporer: feel the speed.