Sigh, another spearman-vs-tank thread. Just with a slightly different flavor, that's all.
Like the others have said, it all boils down to two possibilities:
1> The rifleman was actually typical of the units your opponent could field.
In this case, you were so far above him technologically that the game was already over, and you were just in that abstract "mopping up" phase that gets so boring. Try a higher difficulty level if you still want the AI to be competitive at this phase of the game.
Now, maybe the rifleman had exactly the right promotions for the circumstances. If he had Woodsman I and Woodsman II, for instance, and you attacked him in a forest, and he was fortified, those bonuses stack up very quickly. The numbers difference between a gunship and a unit of riflemen isn't THAT huge, especially if the rifleman is defending in good terrain. And there plenty of real world examples of this; see also: Afghanistan (vs Soviets) and Vietnam; before you start bragging about the night vision or thermal imaging capabilities of our helicopters, you should go try using one of those systems in something outside of ideal conditions.
2> The enemy actually had the technology for better units, but simply hadn't upgraded this specific one.
This was a big issue in Civ3, and it's one I hope a modder fixes in the near future.
The way I proposed to fix it was like this: When you unlock a new unit in a chain, the cost of upgrading each previous unit in the chain gets lower. The N-1 unit upgrades to the N at normal cost, the cost to go from N-2 to N-1 drops by half, and anything below N-2 automatically upgrades to N-2 for free (even if it's not in a town at the time).
That is, to take a Civ3 example, your infantry chain went Spear-Pike-Musket-Rifle-Infantry-MechInf. When you get the tech for Muskets, the cost to go from Spear to Pike is halved (although you can't actually upgrade halfway, of course). When you get the tech for Rifles, any spearmen automatically upgrade to pikes for free. So, by the modern era, you wouldn't see anything below riflemen still in play.