Then they take a look at your units, and just leave.
I had a huge Carthaginian army massing outside my city. Catapults, Swordsmen, archers, and LOTS of them. I quickly bought a bunch of units, changed my production, and got a lot of archers, etc, up to the front pretty quick.
So they took this huge army....and just left.
I love it when this happens. Because I know they'll be taking their huge army somewhere else and probably leaving their home cities unguarded.
Carthage did that and lost both their cities.