I for one, like to upgrade units as it allows me to concentrate on buildings and development of my cities but I think that Padmewan makes a good point ( which will probably have me changing my strategy a bit)
Lets look at this as a best case/worst case analysis using a unit with 10-17 xp points.
At best all of the previous promotions are still useful and the unit had no xp loss (was at 10XP at the promotion time).
Therefore they get a 4th useful promotion in 7xp points time.
At worst none of the promotions are useful and the unit had 6XP loss (was at 16xp at promotion time).
Therefore they get 1 useful promotion for 13xp points (7 +6 wasted)
We can take an optimal new unit
with barracks, vassalage and theocracy one can get 8xp points (possibly 10 with West Point?)
for a total of 2-3 promotions all of which will be useful to the unit.
Therefore
(assuming no WestPoint) in 2xp one will get a 3th promotion or
(assuming WestPoint) in 7xp one will get a 4th promotion.
Ok so now the analysis.
Given the worst scenerio for the 10-17xp unit it is always better to create a new unit (money and production issues aside) as the 1st useful promotion is at best 7xp for the 10-17xp unit and the new unit only needs 2xp.
If one uses the best scenerio for the 10-17xp unit then it is better to upgrade as the unit will have no penalty accrued to its promotion progression.
Upon looking at this I noticed two key observations (never mind that these have been brought up before in this thread and in other threads

).
1/Upgrading will most often cause a penalty
-if the unit has excess xp points past the previous upgrade mark
-if the unit has obsolete upgrades (eg archery +25% when the enemy has all gunpowder units)
2/MAKE SURE you have a barracks and the correct civics in place when you make units!
and that folks is a restrospective Civ IV analysis
ps I did not regard the Agreesive trait since it would have been applied to any old unit that was being upgraded and to any new unit that the old unit could be upgraded into. This would not be the case if a unit could be upgraded across types (eg from archer to gunpowder) but I'm not sure if that can be done.