Interesting test, thanks for the effort! Without having any real experience modding, I suspect the core of the issue is that the AI is implemented entirely on a per-unit basis with absolutely no kind of unit coordination (outside of the fact that each unit knows if its at war or not).-- Japan did try to attack one of the two main, productive and protected, cities. However it did little more than cycle in a lone unit - attacked by my archers - and then cycled in another unit -- attacked by my archers. Japan made no attempt at a unit rush - even though it had the capability and numbers to do so.
If this is the case, I doubt a mod is going to be able to make any serious in-roads on military strategy. One way or another, units need to be able to take "orders" from an algorithm or process in charge of a coordinated strategy. Regardless of how smart each unit may be, I don't think an "every unit for itself" implementation is ever going to work out well.