Just losing one hit point can still make a great difference.
Using CivLacky's excellent "Combat Calculator", we have the following results:-
Cavalry (attack 6 with 4 HP) v.
unfortified Musketmen (def 4 with 4 HP) = success 66%, fail 34%
Cavalry (attack 6 with 4 HP) v.
fortified Musketmen (def 5.4 with 4 HP) = success 56%, fail 44%
But if the defending Musketmen have some sort of 'defensive bombardment' unit in their stack:-
Cavalry (attack 6 with
3 HP) v.
unfortified Musketmen (def 4 with 4 HP) = success 50%, fail 50%
Cavalry (attack 6 with
3 HP) v.
fortified Musketmen (def 5.4 with 4 HP) = success 39%, fail 61%
In effect, assuming that the 'defensive bombardment' unit gains a hit, unfortified Musketmen have a defence as if they
WERE fortified.
Still worth having I think.