Can't tell you how to decide this, but here are some of my own thoughts. It's a mixed bag of pro and con and definitely not comprehensive.
On the pro side, programming Civ5 Lua/xml/sql is much more powerful than Civ4 python (without dll):
- Lua functions can actually remember things. In my little Civ4 mod that interacted with unique features, the script had to search every plot every turn to find these. There were many such situations where my mod had to figure stuff out in python logic every darn turn, rather than just putting the darn value in a variable and keeping it there (seems pretty basic, but you can't do it in Civ4 without dll). These values can be saved through game exit reload using SaveUtils or my own TableSaverLoader.
- Complex table construction. Very nice in Civ5, and pretty fast when you get the feel for sql.
- Total control of UI without dll (though a few gaps, like the ability to update terrain graphics).
Cons
- No documentation and some "not implemented" tags. Sucks, no doubt. I get around this pretty fast now by frequent use of Windows Grep on the core game and all the best/biggest mods.
- Worst of all from a fantasy mod point of view: no unit graphics and a pretty big hurdle to make them. There seem to be only 2 or 3 people in the world who have overcome this, and they seem to be pretty big WWII fans (no disrespect intended; I just wish they had watched more LOTR growing up rather than ... whatever it is WWII fans watch). I have not tried this myself yet, so I really don't know how high the hurdles are. But they seem pretty high. (This one issue gets me closer to believing anti-modder conspiracy theory than any thing else. However, even if not intentional, it's a pretty damning lack of interest in modders/modding.)
Unfortunately, after modding some in Civ5 with the DB/table functionality, I simply
can't go back to Civ4 (it would be an organizational nightmare). If the unit graphic issues are overcome, then I know I can get 90% of my mod working without dll. Since it's going to take me many months to get to that stage anyway, I decided I just won't worry about the dll for a while yet.
My own theory is that they are deep in expansion development (based on the theory that they want to make money and many folks like me will buy game play expansions but not buy yet-another-civilization DLCs) and that the base Civ5 code is now so thoroughly disorganized and undocumented that they can't release it. So perhaps we'll see it with the first expansion. It's just my uninformed theory though so don't put any weight on it.