Some good ideas here. Two loose thoughts and a different view on defence from me to make things more complicated.
1) At the moment commando troops aren't too useful because the AI always keeps several units in a city and sacrificing lots of commando troops to a surprise attack is just not a good tactic as these troops are too hard to get. If cities would get 0 or 1 defenders because they are considered 'safe' by the AI, then this might change. So if you consider commando troops during the era of railroads in your definition of safe cities, then not a lot of cities will be considered safe.
2) The human might turn a city into a killing zone. It bombs city defences, kills the major fully fortified defenders and waits for the AI reinforcements from other cities and slaughters them with its city raider III troops as they slowly trickle into the threatened city.
My view is that you can't defeat a serious (human controlled) city attack stack by defending the city. You need to create a stack that can defeat the city attack stack. The combination of collateral damage plus the strong city raider promotions makes stubbornly defending a city an inefficient use of troops. A city with its inherent defence bonus is great in slowing down an invasion force, but it's awful in stopping it once this defence bonus has been bombed away. The only way to defeat a serious city attack stack is by using collateral damage as the defender combined with a serious force in well promoted units that are good vs troops in the field.
So in my view, the AI should garrison its directly threatened cities with a decent city defence force (4-10 units) just to make a direct attack without bombardment on the city extremely costly. Next to that it should have a reaction force spread over several cities which should group together in case of an invasion. Hopefully, the city defence is good enough to stall the city attack stack until the reaction force is assembled and can destroy the city attack stack.
So the AI needs several categories of units
- city defenders
- reaction force
- city attack force
Units should be able to switch back and forth from the city attack force and the reaction force depending on the momentum of the war.
The hardest part (I think) in this type of defence would be to let the AI define a tile where it should assemble the reaction force so that this force can counterattack the city attack force but not be attacked by the city attack force. As reaction force, you want to be the one who uses collateral damage, not the one who suffers collateral damage. This probably means that you have to define an area around the city attack stack which is threatened by this stack (mainly its collateral damage attackers) and pick a spot outside this area that still allows you to attack the stack in one turn (the defender should have the advantage of roads). If such a spot is not available, then assemble outside the threatened area and then move into the threatened area in force. The AI of course should have a way to determine which stack will win, so this would benefit a lot from simulated combat discussed in another thread.
As promised, I have made things more complicated. However, if the AI would be able to do this, then it would finally be somewhat capable to counter a human attack. And that would be the greatest achievement ever in all the AI improvements which have been made since Blake started improving the AI years ago.