A lot of interesting points

Thinking about it, the city ranged strike can be a challenge for the AI to deal with, as can the new siege mechanic. Perhaps like
@Jaybe mentioned some mechanics need to be changed/removed in order to give the AI a more reasonable chance at success. Right now when they attack cities, they have to deal with the player's units, the city's defense strength, the city's ranged attack, the city's healing (if not under siege), and any movement/positioning challenges. So they have to spend some time forming a team, decide to use it, deal with the challenges it takes to get that team in place, and then deal with all the above factors during the attack itself.
When we add all that to the "overbudget" mechanic that
@Josephias and
@Victoria mentioned, and add to it the "tunnel vision" that the AI sometimes has while on march to a target like
@PhilBowles mentioned, it seems like the AI has a lot on its plate when it decides to siege a city. Removing the city defense and strength mechanics might actually help a lot. From what I recall from Civ 4 and before, cities had no strength of their own, and depended completely on their garrison. Attackers wouldn't take damage from an un-garrisoned city.
I'll keep experimenting with giving melee units city strength bonuses, some solutions can be tested using mods.
Couple days ago Teddy attacked a city of mine. I had one archer present and no walls. He brought 4 archers and 3 warriors. He was so busy trying to get units into position for siege that I had time to build and bring more units and get a wall up and fending of the attack.
Yup that's sadly something I've noticed too, positioning before they arrive would help a lot with that.
Since the last patch I've seen a huge improvement in AI tactical play - it's still not at the Civ V stage where it can appropriately target specific units with that unit's counter, it's poor at focusing fire on a single target and will often randomly ignore a damaged unit it could kill in favour of something else, but in most other ways pre-air combat it's approaching Civ V levels of competence (it still can't use air units and even though it builds fighters it doesn't often deploy them and often simply randomly rebases them. After more than 250 hours in Civ VI, I had a bomber damaged by a fighter for the first time ever yesterday).
The issue that prevented units firing out of a city that has walls has finally been fixed, the AI attacks with large enough forces at the start of a war and frequently a reasonable mix of ranged and melee, and if anything it's now better than Civ V at avoiding the 'sit and loiter while embarked/embark in range of enemy ranged units' issue. It almost always escorts settlers now, a perennial pre-patch problem. It's also improved at both building cavalry and using their speed to good effect; although late game unit selection still needs a lot of work (no more anti-tank spam, please, and please build support units and anti-air - if possible, code the system so that it can respond by building counters to whatever it's being attacked by) that's a separate issue from the AI's actual use of units. It needs to prioritise taking out artillery with cavalry, but doesn't seem to have any response to 3-4 range units coded, so although it can now put up a reasonably creditable defence in earlier eras it will just collapse in the face of artillery and bombers, often while it has technologically superior units.
The major flaws I now see are mostly strategic. It does seem to follow the algorithm you outline, but its target selection is still "rush the capital" by default, even when it's not the closest or easiest target to attack - that can give the player a very long time to see the attack coming and prepare. It also exhibits no ability at all to adapt to changing situations once it's committed to an apparent attack (Civ V had a similar issue to this latter, but it wasn't as pronounced). I started a session yesterday when America, my closest neigbour, sent 5-6 warriors towards me - I was far enough away that I responded by giving Teddy a money gift and declaring friendship with him, to buy time while I teched to Early Empire to prevent a siege. Teddy didn't quite know what to do after that, so the warriors just sat around St. Petersburg not doing anything. This became a problem when Washington fell to the Sumerians - the large American army that could have defended it vanished a few turns later when America's other city was taken, as the army hadn't even started heading back until Washington was captured.
In general it doesn't know how to defend - it garrisons more frequently than it did, but not exclusively, but while that makes early cities reasonably challenging to take (especially with the AI tendency to spam encampments) and the AI will either reinforce with nearby units or rush-buy units to defend, it won't bring units from further afield to help, just as in my America example. An AI can be losing cities left and right and still have a reasonably large military that's just scattered randomly across its territory.
I'm glad to hear the tactical play has gotten better! And that the AI is going to war with a better unit mix. You handled that situation with Teddy really well

I'm guessing those warriors were part of a city attack team, and the AI didn't remove them from that team until it was too late. I wonder if reducing the number of teams of each type might help, or maybe allowing more city defense teams compared to other types. That might help with the issues you mentioned in the last paragraph too.
The strategic AI might be responsible for most of the issues we're encountering, hopefully they take another look at it soon.
The issue with capitals seems to be common, I've had two instances where I remember the AI gunning for my capital with enough forces to take it, but due to losses on the way and a beefier defense when they arrive, deciding to retreat or just hang around, causing more inevitable losses due to the city attack mechanic along with any garrisoned or otherwise safe ranged units. They could code them to attack the closest city to them. If they combine that with full dedication to take their target, even at the cost of losing 80% of their units, players would start to feel truly threatened. As it is now, with a few ranged units, I think most players feel safe enough to meet any invading AI army.
I think AI+ got really close to helping the AI actually commit to attacks, I'm not sure though how well it works with the latest patch, I know that issues with settling caused me to deactivate it for a while.
I wonder what would happen if units become much stronger but also more expensive. That might help a lot with the unit shuffling if the team unit limits are tweaked.
The AI I think will be a threat by the end of the Civ 6 cycle, and hopefully we can get it to be a threat through modding as well before then.
I noticed - more remarkably, it later even moved the fighter to directly defend the city I was attacking. It took me aback because I had no idea how interception worked in Civ VI as it simply hasn't come up - the AI never attacks with air units and has never previously damaged mine. I just use flghters as glorified observation balloons, and occasionally to snipe ground units.
That's great to hear! I'm trying to wait for Rise and Fall to play another game, but air combat always gets me excited to play. I still remember the fun and power of stealth fighters in Civ 2
