Thoughts on a community DLL

[Hambil]

I've also created a real simple (for now) wiki off the git, I want it to be used for people to make requests for LUA functions to be looked up in the c++ source and documented better. This seems better done 'as needed' then making one giant effort.

https://github.com/Hambil/CommunityDLL/wiki/_pages

 

[Hambil]

Added the #defines file, and added Plot:GetCity(), closing two issues.

We have three defines in MyDefines.h, to match our three current milestones:

// Adds various missing LUA functions
#define MOD_GLOBAL_MISSING_LUA
// Adds various missing XML tags
#define MOD_GLOBAL_MISSING_XML_TAG
// Adds various new functionality
#define MOD_GLOBAL_NEW_FUNCTIONALITY

I think each fix should be accompanied by a lua test project, so I am making one now (simple unit test stuff). I'll add a place to put them on git-hub and put one for this in there to start.

 

[Hambil]

Lua unit test Repository added: https://github.com/Hambil/Mod-Unit-Tests

First change (Plot:GetCity()) successfully unit tested and project uploaded.

 

[Hambil]

Proposal:

DLL fix to keep free buildings from being destroyed on city capture. e.g.
Library -> Great Library
Wall -> Great Wall
Lighthouse -> Great Lighthouse
...

Since wonders can't be destroyed, and these buildings are granted free because of game mechanics (the Great Library is a library, etc...) then I think this is a bug we should fix. I want it done at the DLL level because the DLL does some odd stuff on city capture (it destroyed and recreates the entire city).

 

[Pazyryk]

You mean the conquering civ should get a Library (non-free I assume) after conquering a city with a free Library? Sounds fine to me but I'd call that an added feature rather than a fixed bug.

Something to keep in mind: There likely will be a lot of bug fixes with the new expansion, which will be applied also to vanilla and G&K dlls. (Doubtless there will be new bugs to replace these.) Just saying that your efforts may be better directed at cool new features.

 

[Hambil]

I'm saying the free library goes with the Wonder. If the wonder can't be destroyed the library should not be either. It's just a showy way of giving yield bonuses to the Wonder. By destroying the library you are destroying a part of the wonder. That's why I consider it buggy...

The focus of the community DLL is to provide the ability to modders to make cooler features in lua and xml. I would rather provide addition functions, access to missing data, better and fuller access to the games subsystems like building queue management... etc.

That said, it can be fixed with some effort in lua... and other than a building iterator on city, which is already reported as an alpha fix, I can't think of what to do to make it easy (in lua), so I don't know. Maybe I'll leave it with the iterator fix.

 

[Gedemon]

as long as it's an option, I suppose it's not a problem. Remember whoward's "what is a bug for some is a feature for others".

for me it's normal that capturing a Wonder does not give all the advantage of building it.

 

[Hambil]

We can certainly put it under features. If it's just a matter of how it's classified that people have concerns about then that's easy enough. It doesn't much matter to me

 

[Hambil]

As we know I am still a noob on the XML side of things, so before I add this to features if anyone know if it can already be done some other way, let me know:

Proposal:
Two new xml tags that allow you to specify religion specific beliefs, and civ specific religions (not religious preference). With matching database/dll changes needed.

Update: Just looked at the database tables and neither connection exists, so it's doubtful I'm missing anything and this can currently be done via xml. On the dll side not sure how complex this is yet, still looking into it. The hopeful part is all it's really doing is limiting the view into the beliefs data. Meaning, it should just work smoothly in the game, once implemented.

 

[whoward69]

allow you to specify religion specific beliefs, and civ specific religions (not religious preference).

Easiest and most flexible to do with new events and Lua - see http://www.picknmixmods.com/mods/48051461-097f-450d-bb49-fe16ad511583/mod.html

 

[Hambil]

One of these days I'll think of something you haven't done already A question though, about events;

You have stated that events add overhead even if not used. You also mentioned that your DLL uses a system that doesn't call parts of the code if those parts aren't being used by a mod. Assuming we do something similar do the unused events still cause overhead?

I ask because poking around it seems the XML/datebase part of this monster is in a DLL we don't have. That being the case fixing it the way I wanted is probably not doable. In addition, a number of missing tags I'd like to add just got harder...

 

[whoward69]

The ideas were Rob's

http://forums.civfanatics.com/showthread.php?t=491253 and http://forums.civfanatics.com/showthread.php?t=495185

 

[S3rgeus]

I ask because poking around it seems the XML/datebase part of this monster is in a DLL we don't have. That being the case fixing it the way I wanted is probably not doable. In addition, a number of missing tags I'd like to add just got harder...

You can still add new tags to the DB from XML/SQL (existing tables or adding new ones entirely) and manipulate them from the DLL. The syntax is a bit obtuse, but I've used it for a couple of things in my mod. I added two new columns to the Promotions table, TurnDamage and BlocksHealing. (These are penalty promotions for specific units.) TurnDamage causes the unit to take damage each turn, BlocksHealing prevents the unit from healing itself (like barbarians).

I added the new columns via SQL and then just queried them as a part of the CacheResults function in the PromotionEntry class. You need a new field in PromotionEntry to store each value so it can be checked while the game is running. (This may break saves when you add it, I'm not 100% sure, it depends on the workings of the bitfields in CvUnit that are used to flag which promotions a unit has.)

This commit shows how I did the changes for the above promotions on the DLL side. For an example of how to query the DB from scratch for a specific value, check out my change in CvMap.cpp from this commit. (Given you can execute SQL from within the DLL, you could make your table alterations/additions in there instead of in separate files if you didn't want the community DLL to have to include separate XML/SQL documents, just be careful about where the DB is accessed and make sure any changes are done before data is being queried from that table.)

If your difficulties were with the way religion is handled in the DLL then I may be slightly off topic. I haven't looked at how religion is handled there yet.

 

[Hambil]

You can still add new tags to the DB from XML/SQL (existing tables or adding new ones entirely) and manipulate them from the DLL. The syntax is a bit obtuse, but I've used it for a couple of things in my mod. I added two new columns to the Promotions table, TurnDamage and BlocksHealing. (These are penalty promotions for specific units.) TurnDamage causes the unit to take damage each turn, BlocksHealing prevents the unit from healing itself (like barbarians).

I added the new columns via SQL and then just queried them as a part of the CacheResults function in the PromotionEntry class. You need a new field in PromotionEntry to store each value so it can be checked while the game is running. (This may break saves when you add it, I'm not 100% sure, it depends on the workings of the bitfields in CvUnit that are used to flag which promotions a unit has.)

This commit shows how I did the changes for the above promotions on the DLL side. For an example of how to query the DB from scratch for a specific value, check out my change in CvMap.cpp from this commit. (Given you can execute SQL from within the DLL, you could make your table alterations/additions in there instead of in separate files if you didn't want the community DLL to have to include separate XML/SQL documents, just be careful about where the DB is access and make sure any changes are done before data is being queried from that table.)

If your difficulties were with the way religion is handled in the DLL then I may be slightly off topic. I haven't looked at how religion is handled there yet.

Here is my thinking/thoughts at the moment; if we can't handle something like this 'automagically' via allowing the xml to be extended, and it becomes a lua thing, we exclude some large percentage of modders who can't/won't use lua.

On the otherhand, not being able to handle data management at the source is a problem for all the reasons you stated above. I'd suggested perhaps we create a separate class that handles all the new database/xml additions (both reading and writing) perhaps driven off an xml config that tells it what is new. The class would use a generic getter and setter (e.g. CvDBExt.Get('MY_NEW_TAG')).

That way, if we find some timing or other such tweak we need to make, or uncover a bug, it's all in one place.

 

[Pazyryk]

@Hambil, I think you misread S3rgeus' post. Sounds like you can access DB (cached values) from DLL, so why would that involve Lua?

I would strongly advise against "query the DB from scratch for a specific value" except maybe in some specific (not time critical) code. In fact, this even causes a very dramatic slowdown for Lua (e.g., see Test 7 here). Why code in C++ if it's going to be 60x slower than what you could do with Lua?


Edit: Speaking of my Lua speed tests, I wonder if it would be possible (or practical) to recode GameInfo.<>() iterators to use cached values rather than (what I think they are doing) looking through the actual DB. These iterators are insanely slow, 60x slower than a Lua built iterator, and there are tons of them spread through the UI code. (And for Heaven's sake, I hope the DLL isn't using these...???)

 

[Hambil]

@Hambil, I think you misread S3rgeus' post. Sounds like you can access DB (cached values) from DLL, so why would that involve Lua?

I would strongly advise against "query the DB from scratch for a specific value" except maybe in some specific (not time critical) code. In fact, this even causes a very dramatic slowdown for Lua (e.g., see Test 7 here). Why code in C++ if its going to be 60x slower than what you could do with Lua?

You missunderstood/I miscommunicated

The database would/should be read once upon initialization whenever we can determine is a safe time. The generic new class knows what to read (what new columns, etc.) via an xml config file. The database is never read again for the length of the game being played.

The generic getter and setters do nothing but return the already read values.

 

[Pazyryk]

So, just out of curiosity, do iterations over DB info used cached values in DLL?

If so, why the heck is the iterator passed to Lua doing so (assuming my theory is correct) rather than iterating over DLL cached values?

And, Are these GameInfo.<>() type iterators passed to Lua accessible in the DLL we have?

 

[S3rgeus]

Here is my thinking/thoughts at the moment; if we can't handle something like this 'automagically' via allowing the xml to be extended, and it becomes a lua thing, we exclude some large percentage of modders who can't/won't use lua.

Definitely, the above changes I mentioned extend the XML capabilities of the promotions table.

On the otherhand, not being able to handle data management at the source is a problem for all the reasons you stated above. I'd suggested perhaps we create a separate class that handles all the new database/xml additions (both reading and writing) perhaps driven off an xml config that tells it what is new. The class would use a generic getter and setter (e.g. CvDBExt.Get('MY_NEW_TAG')).

As far as I've seen within the DLL, any gameplay DB changes are loaded explicitly in code that we have. Most relevant classes (Eg. PromotionEntry) have a CacheResults function which queries the database and loads the values then. I think it makes sense to store that changed data where it will be used. (Eg. in adding a new column to the promotions table, I extended the PromotionEntry class so it can be checked in other parts of the DLL where promotions are relevant.) Moving that information elsewhere would also necessitate re-running all of the queries from the original database to cache those specific values.

The existing CacheResults functions are already integrated into the hierarchy of when data should be loaded to prevent crashes/inconsistencies. I was warning against performing SQL table alterations after CacheResults for the relevant class had been called, which wouldn't work correctly.

I'm not sure what you mean with a generic getter and setter. Consolidating any new table reads into one helper class would be helpful for dealing with bugs within the DLL, but if the community DLL extends the functionality of the XML available to non-DLL, non-lua modders then they'll never be interacting with that getter. Given the potentially widely varying uses for entirely new tables, which kind of type would a single Getter return to give useful information elsewhere?

I would strongly advise against "query the DB from scratch for a specific value" except maybe in some specific (not time critical) code. In fact, this even causes a very dramatic slowdown for Lua (e.g., see Test 7 here). Why code in C++ if it's going to be 60x slower than what you could do with Lua?

Definitely, the speed penalty for doing DB queries from the DLL will be massive compared to the DLL's normal execution. I gave the above example for things like the helper class, which would be dealing with entirely new queries for new tables. (Side note: the code I quoted above which runs a DB query directly should only run once per game while the mapscript is generating the map. I'm not running DB queries all the time in my code, I promise!)

 

[Pazyryk]

I'm not running DB queries all the time in my code, I promise!)

Yeah, I did not think you were. However, I have less faith in the game developers. I think they are doing this for GameInfo.Unit() type iterators passed to Lua. Hence my line of questions above.

This is a tremendous burden on game runtime even in un-modded civ, given all the times these run each turn just for the Top Panel UI.

 

[Hambil]

I would have to check but based on something I think I remember whoward saying about overhead both the context switch and the db read are done, and it might depend on if you use GameInfoTypes (which I think was designed to memory cache those values).

They are values that shouldn't change once all mods are loaded. The only reason to keep/read them from the DB more than once would be if that part of the DB was too large to memory cache, and that seems unlikely.

However, I don't claim to know all the mechanics of this stuff yet - it can get confusing. I was hoping the sequence would go as follows (but I may be pinning too much hope of Fireaxes);
The db and all mods are loaded [we jump in here with our changes] and a memory cache is built for GameInfoTypes.

If that works, we wouldn't need to do much else. I'm doubtful though

And sure, the code in WhateverMyFunction needs to be changed to use those values, check them, etc. But it had to be either way. Data storage is pretty straight forward. I guess I'd have to see specific examples of what data we'd want to store and retrieve as part of extending the xml tag system, that wouldn't be just easily get/set paired (assuming we can't just use GameInfoTypes as above). In a slightly fancier way, that is all GameInfoTypes is doing isn't it?