normal swordsmen only have 1 D.
and if you want forts, use workers with legions protecting them. If you want them quicker, use more workers. The fortify function is like digging trenches etc. so that part is in the game, like killer said. I don't think there's a problem.