The obvious reason it was changed was to help the AI accurately determine war score.
In civ4 it didn't really matter if you partially worked on 10 units and then rush-bought them all on one turn because 10 units was 1/10th or 1/20th of your army.
In civ5, instantly rushing just 5 units can double your army and thus your war score, which the AI uses to determine whether it can or can not effectively wage a war against you. If partial rush-buys were possible in civ5, the AI would have to expect this to be the case and double it's forces before considering a war in every case. If that happened the poor joe who didn't exploit a rush-army would be crushed every time (assuming the AI could wage a war...) That creates an extremely narrow style of 'correct' play, which wouldn't be ideal.
The last thing civ5 needs is more ways to game the AI.
Thanks for raising an issue I hadn't considered. I'm sure there are ways to deal with this though without having to rule out partial rush-buys completely.
For example,
a) Hammer decay of units not at the top of the production queue for more than 10 turns. (i.e. civ4's approach). If the decay rate is not enough to deter those trying to abuse the system, make it more severe.
b) Disallow multiple partially completed units in the one production queue. If the user tries to switch away from a unit not yet completed, offer the user to rush-buy the unit or forfeit the hammers invested. Alternatively, allow the user to disband the unit being switched away from, converting the hammers directly to gold in a ratio equal to that used for 'build wealth' (this should probably only be allowed once build wealth is possible, or allow it before build wealth but at a much worse hammer to gold conversion ratio to avoid abuse).
c) Have partially completed units contribute partially or in full to the 'war score'.