DLL coding thread

[Kailric]

While we shouldn't hardcode those values in the DLL, adding them prior to debugging could be helpful because then instead of telling you that the TechTypes = 4, it will tell that it is MEDIEVAL_TECH_BLOCKPRINTING when you hover the mouse over the variable name. That makes it easier to figure out what goes on.
The extra enum values should be easy to clear too before committing. That gives an interesting assignment to the script, but the easier to read debugger would be worth this work.

I like this idea. I am always having to count the number of different types in the XML in order to figure out what is what. This would work great for debugging AI, and especially for things like Professions where you need to see what the AI thinks about each profession and why. With that you wouldn't have to count out the number of professions. Handy dandy.

 

[Nightinggale]

It turned out that it wasn't that hard to write a perl script to modify the enums. The C++ code now looks like this:

enum DllExport ProfessionTypes
{
	INVALID_PROFESSION = -2,
	NO_PROFESSION = -1,
	//// SCRIPT XML ENUM: Units/CIV4ProfessionInfos.xml
};

The script (creatively called enumXML.pl) will then trigger on the comment line. It will read the filename and then it will delete every single line it encounters until it reaches a line starting with };. On this line, if it is set to add, it will add the types from the filename in question. If it isn't set to add, then it will not insert them and everything after the comment line is deleted.

The setup for adding or not is automated. The first time the script encounters a comment line it triggers on, it will look at the lines before the end of the enum. If it encounters a ,, it knows that there are enum values and it sets itself to delete. If it fails to detect any, it will set itself to add.

The result is that if you double click the script, it will add. Double click again and it will detect the lines it added itself and it will set itself to remove them. In other words the script doesn't add the lines, it toggles them without any other setup than looking at the output from last run.

The script reads the types from all XML files and we can add whatever file we want in C++ without modifying the script. We can even add a new XML file and the script can handle that without being modified. Despite scanning all XML files at startup and scanning the entire CvEnum.h file, the script takes less than a second to complete.

 

[Kailric]

Awesome, ok, so if I wanted to add values from CIV4ProfessionInfos.xml what would need be done?

 

[Nightinggale]

if I wanted to add values from CIV4ProfessionInfos.xml what would need be done?

Since I already added the comment line for professions in CvEnums.h, all you have to do is to doubleclick enumXML.pl (located in sourceDLL\scripts). That's it. To remove the values again, you doubleclick again. It will automatically trigger on all such comment lines. I will not bother to go through all enums and adds this line in one go. However I'm currently removing a whole lot of typecasts and I improve the code around the lines I modify to make it more consistent and readable. At the same time whenever I touch an enum (to add the FIRST_* = 0 line), I add the comment line.

Slowly, but steadily we will get such a line in all enums, which mean we can hover the mouse over all types to get the XML values. The script will not go for a specific type, but rather it triggers on all comment lines starting with the magic words.

It's currently only in the CivEffects branch though. I figured I better add it to the branch where I mod the enums.

 

[Kailric]

So, what you are saying is you just need to add a line like //// SCRIPT XML ENUM: Units/CIV4ProfessionInfos.xml to any enum you would like to import XML values right?

 

[Nightinggale]

So, what you are saying is you just need to add a line like //// SCRIPT XML ENUM: Units/CIV4ProfessionInfos.xml to any enum you would like to import XML values right?

Yes, but you have to update the path to the file for each enum or it will add profession types everywhere, which would get messy

 

[Kailric]

Yes, but you have to update the path to the file for each enum or it will add profession types everywhere, which would get messy

Yeah, I understood that part. Sweet, this will be nice. I need to eventually learn this scripting stuff, nice addition for modders.

 

[Nightinggale]

Sadly I have encountered a bug

enum DllExport CivicTypes
{
	FIRST_CIVIC = 0,
	NO_CIVIC = -1,
	//// SCRIPT XML ENUM: CivEffects/CIV4CivicInfos.xml
	CIVIC_LOCALISM,

Both FIRST_CIVIC and CIVIC_LOCALISM are = 0 and when the debugger tells the value, it tells the first in the enum and we want the last one

I can't think of anything to do other than reversing the order in the enums, making them like

enum DllExport CivicTypes
{
	//// SCRIPT XML ENUM: CivEffects/CIV4CivicInfos.xml
	script output goes here
	
	FIRST_CIVIC = 0,
	NO_CIVIC = -1,

This mean correcting everything I have added so far (all 5 of them or so) and update the script to trigger on = instead of };. Good thing I didn't update all enums before discovering this

Another issue is that censures are defined both in an enum and in XML. The same strings, but they end up in different enums and the compiler throws an error. Censures appears to require a major cleanup as it is pretty much hardcoded as it is right now. For now, I will just skip adding the censure types and we can deal with this issue later.

EDIT: just pushed a fix for this issue.

 

[Kailric]

Before we add a new civic what will we need to do, remove the old values, add civic, then readd the enums?

 

[Nightinggale]

Before we add a new civic what will we need to do, remove the old values, add civic, then readd the enums?

Normally the values will not be in the enum, which mean there will not be a problem. After all if we leave them in, they will fit M:C only, not col2071 and whatever else would want our DLL. However if you add one while you debug for whatever reason change the xml files, you do have to remove and then add, which is to run the script twice.

When the script removes enum values, it removes everything between the comment line and the first line with =. This mean you can update the xml and it will still remove as intended. The script is fairly stupid and will not check the lines it removes.

 

[Nightinggale]

I just added

 void CvPlayer::launchChooseTechPopup(TechTypes ePreviousResearch, int iCityID)

Those arguments are easier to understand than iData1, iData2 and iData3. Also at the start of CvDLLButtonPopup::launchChooseTechPopup() (yes, renamed from inventions) it reads iData arguments and put them back into variables of the same names as the arguments.

This will hopefully prevent coders from confusing the arguments. Also by only set what is what in one function rather than for each caller, it becomes easier to modify to suit future needs. I'm thinking that we might need a function like this for each popup window. The undocumented usage of iData is really confusing

EDIT: I just tried to increase the warning level from 3 to 4, thinking that it would warn about issues, which is likely bugs. While that might be true, it generated 65069 lines of warnings
Looks like most of the warnings comes from boost. Not sure if we can do anything about that. Presumably this isn't the correct approach to hunt potential bugs.

 

[Lib.Spi't]

It is like trying to find everyone who is innocent to isolate the guilty party!

 

[Nightinggale]

I can't figure out how to get the compiler to warn when comparing enum values of different types. However interestingly GCC warns about that by default. This doesn't make it any less interesting if we could compile the DLL using GCC. GCC would be very tempting since the theory is that the DLL would be noteworthy faster. Also we all have GCC already. It's part of the strawberry perl installer meaning if you installed perl like I have linked to earlier, you have GCC.

However changing the source code to comply with a new compiler seems like a bit much right now. Judging from the ongoing project where this is attempted with BTS, I would say that it would be best if I just continue using the manual approach

 

[Nightinggale]

I made a major update to InfoArray. Now instead of telling how many ints should be read and how many of those are strings, which should be converted to ints, JIT_ARRAY_TYPES should be used instead. This mean the entire setup is done by setting a number of JIT_ARRAY_TYPES in the constructor arguments.

Examples:

m_info_AllowsProfessions                   (JIT_ARRAY_PROFESSION         , JIT_ARRAY_ALLOW)
m_info_ImprovementUpgradeRateModifiers     (JIT_ARRAY_IMPROVEMENT        , JIT_ARRAY_MODIFIER)
m_info_BuildingYieldChanges                (JIT_ARRAY_BUILDING_CLASS     , JIT_ARRAY_YIELD             , JIT_ARRAY_INT)
m_info_FreePromotionUnitProfession         (JIT_ARRAY_PROFESSION         , JIT_ARRAY_PROMOTION)

They fit the current XML layout, meaning the first is an array of professions and each of them have an allowed value.
BuildingYieldChanges is a list of building classes. For each building, there is an array of yields and for each yield there is an int.
The last one is a bit special. It's a list of professions and for each profession, there is a list of promotions. However there is no int or allow or anything like that. Since the rightmost type is a type from XML, the code goes into bool mode. It will then internally create a bool, which is false by default, but will be set to true for any profession/promotion combo present.

This system should be fairly easy to set up and adding more InfoArrays shouldn't be tricky. There has to be a minimum of 1 type and there is a hard limit of max 4 types. Adding more is near impossible because I wanted to use as little memory as possible. In fact the extra data only takes up an additional 4 bytes for each array. It shouldn't be a problem because 4 is much greater than it sounds like. Currently the player cache can't handle complexity higher than 3 anyway and it will likely stay that way because moving from 2D to 3D arrays in the cache adds too much overhead to make it really efficient.

To give full modder freedom, I added some "bogus" JIT arrays, such as JIT_ARRAY_INT, JIT_ARRAY_FLOAT, JIT_ARRAY_MODIFIER and a few others like that. I can't really think of any reasonable XML code, which InfoArray is unable to read when using this setup system.

Since the array is aware that the DLL expects a certain type, it only checks for strings of that type. Also because there is such a massive amount of information regarding what the DLL modder planned, the DLL will can be picky and assert on a whole lot more of different mishaps. Here is an example of a new assert message:

File: InfoArray.cpp
Line: 218
Expression: iValue != -1
Message: MEDIEVAL_TECH_ROADS tag AllowsYields: the string YIELD_STONE is read as UnitClass, but it isn't present in that XML file

YIELD_STONE really is present in the XML files, but it is a yield and my test run made it expect a UnitClass, hence the assert. Until now such a mistake wouldn't be detected by any mod I have looked at so far, but writing a string from the wrong file will surely mess up something ingame.

Notice that the string has 4 strings inserted (marked in red), which will are fetched from different places in the DLL in order to provide as much human readable info as possible about what went wrong. The assert message itself will be all black though.

I would like to give the precise subtag name where the string is read from, but that name is a well guarded secret by the exe. Tag names is a one way communication. The DLL can tell the exe to go to a specific tag, but if the DLL use the command to go to the next tag, it can't ask the exe for the name of that tag. I could have used that for configuring InfoArray (like tags starting with i is int, f is float). However without this info, I figured out a different way to solve that issue, but there is no way to fix the issue regarding the info being missing in the error/assert messages

I have a plan to use this new data to control the return type for the get function. If the constructor use JIT_ARRAY_YIELD, the get function will return YieldTypes instead of int. This will cause compile error or at least asserts if somebody tries to use that value as say TechTypes. In other words, it will become much harder to mix up types without being told about it right away.

 

[Kailric]

I made a major update to InfoArray.

Sweetness, I hope you are keeping up with all these changes somewhere so us layman modders can pick up what you are putting down. Would this have caught that one bug we had where I had something like a ModdersCode type named UNIT_SQUIRE and also a Unit name UNIT_SQUIRE and the code was confusing the two?

The DLL can tell the exe to go to a specific tag

I would rather the DLL to tell the exe to go to H-LL!!!

 

[Nightinggale]

Sweetness, I hope you are keeping up with all these changes somewhere so us layman modders can pick up what you are putting down.

I have written a number of lines on the top of the header file. However I'm considering rewriting it because it has evolved quite a bit since I wrote it.

I might end up having to write a wiki page on how CivEffects work at some point. I think I already wrote something like that at some point. I do remember something about writing that I would assume the reader not to be familiar with class inheritance and a class "tree". CivEffect is the first function to really use this type of C++ coding, which mean we can't assume even experienced modders to be experienced with this setup.

Would this have caught that one bug we had where I had something like a ModdersCode type named UNIT_SQUIRE and also a Unit name UNIT_SQUIRE and the code was confusing the two?

No, but back when we had this problem I added an assert to check that the same string isn't defined in more than one file. We should have this issue covered, not just for InfoArrays, but in general for all files and all types.

I would rather the DLL to tell the exe to go to H-LL!!!

I know that feeling, but sadly ditching the exe is not really a valid option

 

[Nightinggale]

I updated InfoArray to return correct types. Now if say you store starting techs in it and set it to use JIT_ARRAY_TECH, the get function will return TechTypes instead of int. This mean we will have a compiler error or an assert (both are possible depending on how the failure occurs) if somebody decides to use it as a UnitClassTypes or something like that. It also mean we will seriously reduce the amount of typecasting we have to do. Generally speaking, typecasting is evil as it hides bugs if not used correctly.

 

[Kailric]

By the way, could you explain Jit Arrays one more time (I'm sure you did that before), just what is going on there? Is this something newer in C++ programming?

 

[Nightinggale]

By the way, could you explain Jit Arrays one more time (I'm sure you did that before), just what is going on there? Is this something newer in C++ programming?

No, it's a class I wrote only with code which worked prior to C++03.

The idea is simple. It's a class containing a pointer. You have two functions, get and set, which you use to access data in the array. If you write to the array, it allocates automatically. If you read from an unallocated one, it returns the default value. In other words you don't have to consider if the array is allocated or not. Also the deconstructor frees the array meaning it will not leak memory even if you do nothing to free the array when freeing a class where it is a member variable. The length is set as an enum, which points to an XML file meaning it will automatically get the length of that XML file.

I added all sorts of extra features, such as buildin savegame handling and interaction with InfoArrays in order to make the player cache easier to use. Savegames are automatically upgraded to fit new XML layout.

I have major news regarding the CivEffect branch. I figured out how to modify the makefile to compile cpp files from subdirectories. I added Data and moved all data containers (including JIT arrays) into that directory. Now I want to clean up the code and comments and it should be movable to other mods simply by moving the subdirectory itself.

I started working on moving the project files to make a clean source (only cpp and header files, no project files or similar), but I ran into problems and got interrupted. I'm not sure if I can manage to get that working, but if I can't then it's not a big issue, but it would be nice to have.

I plan on adding an Info subdirectory as well where we can have a file for each XML file if we want. This mean if we modify a single file, we only have to recompile the files, which actively use that XML file, not all files.

 

[Nightinggale]

Not sure if this is the right place to write this, but with the lack of a project files thread, this will do.

I modified the makefile to handle not being placed in the same directory as the source files. It's now in "sourceDLL\Project Files". This mean the files in DLLsources is now purely DLL source files. I'm not completely done with this redesign, but since it's working right now, I decided this would be a good milestone to commit and possibly revert to if I break something.

Fastdep caused major issues. It turns out that if it encounters a .. in the path, it change to use absolute paths and then the dependency file will silently fail. It includes with no warnings, but modified header files will not cause cpp files to recompile. To solve this, I had to track down the source code for fastdep, which turned out to be an issue because it's from 2002 and the original server is long gone. I found the source, modified it to compile with MSVC 2010 and then I changed it to use relative paths even if it includes "..". I'm not completely done with this and haven't included it yet, but it appears to be working just fine locally.

For once I ended up doing something, which justifies a screenshot, which I attached to this post. I figured out how MSVC filters work and that project folders are called filters (which is a good start ). Using ordered files, it should be much quicker to find the file you are looking for.

I decided to sort them based on this:

• Cv files Self explaining. We will likely do most of our work in here
• Data Data structures. This is JIT, Bool and Info arrays.
• Misc Files not fitting the other categories. Often we will not need to modify those
• MOD specific The 3 files read from sourceMOD (yields and other stuff that mods don't have to share)
• Python All Cy files

Each folder has two subfolders called CPP and Header. Files can be moved around in the folders inside the project with actually moving the files. This makes resorting very easy.

Now that the project files are placed elsewhere and with Data actually being in anther directory, it makes it easier to write code, which can be copied to other mods (like RaRE). Since I decided that RaRE has to gain Data (FTTW will get it too eventually), I figured I might as well filter it out from the rest of the code ASAP.