So I read that it is possible to adjust existing behavior by messing with the xml or create new processes but it is not possible to replace existing processes (i.e. Lua scripts)?
Basically. The game uses a Flavor system to weight the AI's choices. Take buildings, for example; if the building queue is empty, then the AI will pick from
every building it can build at that moment, but its choices will be weighted by Flavors. So if the AI really wants production buildings at the moment, then buildings with a positive FLAVOR_PRODUCTION will be relatively more likely to be chosen; the Factory, with a flavor value of 100, will nearly always be picked if the AI has Coal, but most of the buildings are in the 30-50 range and so aren't so absolute. This Flavor weighting will depend on the AI's leader choice (so Elizabeth loves naval stuff), the AI's Grand Strategy choice (so an AI going for a Domination win will favor military boosts), the AI's City Strategy choice for that city (the "Production Focus"-type things) and a bit of randomness thrown in for good measure. The only real cap on this is that the AI will never try to build anything that'd take more than 50 turns to complete, which mainly only applies to Wonders.
What it means is that you, the modder, can easily tweak the AI's choices by changing these Flavor values. Want the AI to REALLY want to build Walls in every city? Boost its Flavor. Want the AI to have puppet cities build the occasional growth and production buildings, instead of only ever making financial ones? Change the Flavor weightings on the puppet City Strategy.
What you can't do, however, is make the AI be more intelligent than that. You can't make the AI do things in an intelligent order (production stuff earlier, cheap stuff first, etc.), and you can't easily tell the AI that building X is a great thing to build once its cities get big but lousy to build when the city is still small. The same applies to other things, like Improvements; you can't teach the AI that building one of an improvement is great but that building two of them side-by-side is a bad idea. If something is a good choice, it'll be seen as ALWAYS a good choice.
Sounds like you can tweak the AI but not really mod it.
That's the basic idea. We can change the AI's biases, or add rudimentary extra behavior of our own, but we can't fundamentally alter the existing behaviors. For that, you'd need the DLL.