Faster speeds are generally better for the defender, because the reinforcement times are shorter when you produce units that immediately start to defend. Also the window of opportunity for an attacker is shorter tech wise, because you can find yourself in a situation where your units have become obsolete faster, but you have no immediate way of upgrading them to the latest tech (whereas the defender can upgrade immediately within his own borders, which not only gives you better units but increases the combat strength of cities). Finally there are the combat mechanics themselves, for instance how a warrior moves 2 tiles regardless of speed, meaning that you are essentially traveling relatively "slower" at faster speeds, compared to at slow speeds (relative to the pace of the game otherwise, such as in teching etc.).
If you struggle with early attacks on faster speeds (or, in general rather), you need to scout ahead to the nearest neighbour to keep tabs on what he is doing with his army. I generally dont build a scout unit for this particular reason, because if you get invaded you need to have a warrior or slinger produced to stand a chance. Hence I usually build a warrior or slinger as my first unit (warrior is safer), and use my starting warrior to just do a quick lap around the capital to get vision on any barbarian camps. Then I use him or my newly produced warrior to clear out the camps, while the other one goes close to the nearest AI to keep tabs on what his army does. If he moves a bunch of warriors (generally 3 or more, as more might also hide in the fog of war) together in your direction, you can be pretty sure that he is trying to invade and you need to build more warriors asap. It helps if you know that he hasnt scouted your location yet, because he wont know where to invade then (hence why you also dont want to reveal your capital to him).
Once he does invade though, good tactical positioning can usually beat back the AI, assuming you recognized the invasion happening early (he usually only declares when he is right next to your capital, which gives you plenty time to build warrior).
