Here's an idea - mobilization:
When not at war, you can demobilize troops - they can stack up to three, cost less maintenance, and gain extra movement. However, they get a significant (on the order of 50-60%) malus in combat.
When war breaks out, you can pay something like 3-5 gold apiece to mobilize individual units, or all at once (though the cost must still be paid). You can also demobilize individually.
That sounds like a good idea, though I'd rather have a more significant combat penalty (including no attacking and no pillaging) and a higher stack limit (maybe six, for the number of adjacent tiles?). With only a 50-60% combat strength reduction, you may be encouraged to take your demobilized units into combat since you could fill your front lines with 3 times as many units and the increased movement speed and lowered maintenance costs would be an extra bonus.
You want the penalty to be high enough to discourage that, since the real reason for the feature is just to prevent frustration and lost time.