DLL coding thread

[Nightinggale]

Yeah, I wondered about that, but since the vanilla coders didn't bother with anything less than an int most times I didn't either. I think you mentioned it before but memory isn't all that big an issue these days. Looking at my Task Manager with 9 Apps running including M:C it shows FireFox here using 290 MB while M:C only uses 230 MB. With all the other apps running it still shows 51% of my memory still available. The crazy thing is I don't even know what all that means, if anything at all.

51% mean you are only using half of your physical memory. If you run out, it will use the HD as well, which is much slower.

A 32 bit system can only use 4 GB including the HD, though a CPU with extended memory can use 6. Windows XP can only use 3 for unknown reasons.

On 64 bit systems (most computers today, except the one I'm using right now ), the limit is 4 GB for any 32 bit application (all civ4, including colo). However there is a bug report/support request for C2C claiming it crashes when reaching 2 GB. There are similar reports regarding running RaR on 32 bit systems, which can be problematic.

There is another part to this as well. Quite a lot of the time spent waiting for the CPU is actually spent by the CPU waiting for memory I/O. Whenever the code loops an array, it will actually be faster if the array uses less space in memory (read: less memory to transfer to the CPU).

Writing to something smaller than int could be slower though as the game has to load the other vars and then copya block of 32 bit to memory. This mean often written to and rarely looped arrays could benefit from being ints while read only arrays (like XML data storage) really benefits from being smaller.

Vanilla likely uses int everywhere to reduce development time (some parts of the code appears to be rushed to meet a deadline). The savegame code can't handle short or char. As a result JIT arrays converts to int and saves those. On read, it reads ints and copy those to the array. Come to think of it, it stores a bunch of unused bytes, which increases savegame size. Maybe I should look into saving more efficient to reduce file size. Like variable sizes, it isn't critical and can wait.

 

[Nightinggale]

I have a few words on DllExport
This keyword tells the compiler to make a function available for linking between DLL files. In other words it is a keyword, which allows the exe to access DLL functions.

We should never add new functions with DllExport as the exe will never access those anyway. If only vanilla functions use this keyword, then we can use it to identify which functions the exe might access. I also suspect that linking a function with DllExport is slower than linking one without it.

I plan to compare the source with vanilla at some point (not manually!). This will allow detection of incorrect usage of DllExport. It will also help identifying M:C code, which can break network games. This mean we shouldn't worry about the mistakes already made, but just remember this for future additions.

This isn't about blaming anybody for suboptimal coding. It looks like all modders touching our code (including authors modcomps we added) contributed to this, myself included. This is more about stating the correct approach, now that I figured out how DllExport should be used.

 

[Kailric]

I plead the 5! Or, is the 6th, or what ever number means "ignorance" I know I contributed to the Dllexport additions. I had know Idea what that meant so I just followed vanilla's example. I am glad to not be ignorant anymore, and in the future and will avoid this.

 

[Nightinggale]

I know I mentioned that I might be done with the JIT savegame conversion this weekend. I don't think I will make it. Part of the reason is that I figured out something, which makes it even better.

I decided NOT to convert data in CvUnit. It turns out that this file fails hard regarding tolerance to XML changes. Not only does it save arrays of length determined by XML, it also saves XML values.

Say for instance we have a unit with promotion 7. It gives +25% defense in hills. Somebody changes the XML file to 20% for balance reasons. Now the game will give 20% to units gaining promotion 7, while units, which gained it before the change will keep the 25% in the savegame. On top of that if the unit loses the promotion, it will lose 20% and it will stay with a 5% bonus without having any promotion supplying it. Even worse if the promotion was changed from 20% to 25%, the unit would end up with -5% and some of those numbers curses asserts when they are negative.

Rather than attempting to fix this broken promotion system, I completely removed it from the savegame. Anything related to promotions will no longer be saved. Instead CvUnit::read() will end up calling processPromotion() for all promotions the unit has. This way any XML change will apply correctly. Adding promotions to XML will work as promotions are reordered and removing promotions will also work. If the unit has a promotion, which is removed, it just loses a promotion.

A sideeffect of saving less data is.... saving less data. The savegames will decease in size by (all in bytes)
120 fixed
4 * number of domain types (we currently have 3 domain types)
12 * number of terrains
12 * number of features
8 * number of unit classes
4 * number of unit combat infos

Multiply this by the number of units in the game and it starts to add up. The last 4 lines have also been converted into JIT arrays meaning they will not use memory at runtime unless they contain non-zero data. This is a fair amount of memory as the majority of units will not have promotions. On top of that the arrays have been reduced in size to use only 1 or 2 bytes for each element instead of the vanilla 4.

Another bonus is that saving less data will presumably take less time. Nothing major, but still nice considering we all (hopefully) use autosaves frequently, possibly every single turn.

I also spotted an alt equipment yield array we missed when removing alt equipment. It was not used anymore and always 0, but it used 4 * number of yields in both memory and savegames. Naturally I removed that one right away.

 

[Nightinggale]

I decided to go all the way and make savegame conversion code for more or less any possible XML change. Looks like I'm almost done, but I left the hardest part for last. Also now that I completely ruined old savegames, I deleted the code to load them as all those now unreachable lines of code confused me when figuring out which lines needs conversion on load.

On top of that I plan to see if I can make more of the type of changes mentioned in the last post where promotion effects aren't saved. The more it reads XML files on load, the more modder friendly the DLL is. Also saving is faster and savegame files are smaller.

I ran into problems with CvStruct.

// XXX these should not be in the DLL per se (if the user changes them, we are screwed...)

At the same time not doing anything will ruin my efforts. Thinking about this for a bit made me realize one thing: the exe assumes certain stuff about the memory layout (.h file), but the exe is less picky about the cpp file. The outcome is a temporal JIT array on load, which handles the conversion, then the content of this array is copied into the memory structure of the header file and the exe can't tell the difference.
DO NOT TOUCH
This piece of code has a high probability of cursing a wreck due to close interaction with the exe file. Every single line of change here has to be carefully considered and hopefully we will never have to modify anything regarding CvStructs again.


The JIT array does decently as a temporal variable since it automatically inits the array and frees the memory when the variable goes out of scope. However as I once again used it as a short lived variable I started to think about memory allocation. The problem is that if we allocate say 2 arrays of 20 ints, they are placed after each other (presumably). We free the first and allocates one of 17 ints. We now have 3 ints between those two arrays, which is kind of useless unless we allocate a really short array (BoolArray can do that). As a result even though the new array is only 17 ints long, it effectively takes up 20 ints. This problem is known as memory fragmentation and the solution to this problem is writing a memory manager.

I came up with an idea for such a memory manager. Whenever an array frees the memory (current code), it should call a function to deal with it. This function takes the pointer and stores it in a vector of lists of pointers. The memory isn't freed and we keep a pointer to it meaning it isn't leaked either. Whenever a JIT array wants to allocate it calls a function, which goes into this vector and finds the list of arrays of the correct size. It then removes one and gives this pointer to the JIT array. It allocates one to give if the list is empty. This way we avoid the constant allocate and free, which causes memory fragmentation. We could also gain faster code as allocating memory is slow. This mean JIT arrays will be superior to vanilla arrays in yet another way.

This may not be the most advanced memory manager available, but it should do fine for our needs. I wrote it here now partly to remind myself of it when I'm done with more urgent tasks. As the memory manager will not affect savegames at all there is no reason to mess with it right now.


I have to say that my idea about saving a bit of memory has involved into something far beyond what it was originally intended to do. Luckily the core structure has a design, which allows adding more and more whenever the need is present

 

[ethernidee]

I came up with an idea for such a memory manager. Whenever an array frees the memory (current code), it should call a function to deal with it.

Usually C++ built-in memory manager does work well. It means that heap chunks are aligned to 8 byte boundary (no 17 bytes allocation ever) plus there could be possibly several double linked lists holding pointers to chunks of popular sizes (16, 64, 4096, etc). If you use malloc, free, new/delete then each allocation already requires 4-8 additional bytes. Additionally you store the pointer in global list to each chunk and try to implement memory defragmentation. Afair, there are lots of ready memory manager pluggable for C++ with sophisticated strategies, you could take stable one and use ^^.

 

[Kailric]

Finally, someone who actually understands what the hell Nightinggale is saying!

 

[Nightinggale]

Usually C++ built-in memory manager does work well. It means that heap chunks are aligned to 8 byte boundary (no 17 bytes allocation ever) plus there could be possibly several double linked lists holding pointers to chunks of popular sizes (16, 64, 4096, etc). If you use malloc, free, new/delete then each allocation already requires 4-8 additional bytes. Additionally you store the pointer in global list to each chunk and try to implement memory defragmentation. Afair, there are lots of ready memory manager pluggable for C++ with sophisticated strategies, you could take stable one and use ^^.

The main reason why I made one myself is actually quite simple: We use ancient software. We use a compiler and python from 2003 and boost from 2004. I have bad and uncommitted experience trying to add newer stuff than that and using 10 year old libraries isn't trivial either. This is why I think twice about adding external code.

As for C++ handling, I have to say that I don't fully trust MS VC++, at least not the 2003 edition.

The memory manager I implemented is actually quite simple.
https://github.com/Nightinggale/Medieval_Tech/blob/master/DLL_Sources/MemoryManager.cpp
https://github.com/Nightinggale/Medieval_Tech/blob/master/DLL_Sources/MemoryManager.h


Speaking of old libraries I managed to compile the thread lib for our version of boost as it isn't available online anymore. I could share it if anybody is interested. I also learned that threadpool 0.2.3 is compatible with our setup.

 

[ethernidee]

Ok, studied the memory management code and at last understood the idea. Just a few notes:

iLength += 3;
iLength /= 4;

No 64 bit support, sizeof(int) instead of 4 and sizeof(int) - 1 instead of 3?

-) what is reasonable m_vector size?
-) why don't you allocate new record in the list on demand? And if you do, that would be a waste of memory, because deallocation is not freeing memory in fact. So if you had many arrays of 8000 bytes and then they were resized to 8004 size, lots of nodes in 8000-list will be pending almost forever. In general waste of memory can be a problem, while fragmentation is severely reduced on game restart (cheap cure).

Maybe I'm wrong? ) Btw, I have 2 GB RAM and original colonization 2 becomes laaagy after 200 turns. Town screen is updated in 2-3 seconds after you move any colonist.

 

[Nightinggale]

No 64 bit support, sizeof(int) instead of 4 and sizeof(int) - 1 instead of 3?

The game exe is 32 bit meaning we can't gain 64 bit support even if we want to. If we could I wouldn't be concerned about memory.

-) what is reasonable m_vector size?

That will totally depend on XML settings. It will change itself to fit whatever is needed by current XML setup.

So if you had many arrays of 8000 bytes and then they were resized to 8004 size, lots of nodes in 8000-list will be pending almost forever.

That will not happen. An array could be of say ints and length would be of number of yields or professions or some other XML file. This is fixed and will only change by quitting the game, modify XML and then loading the game. This mean we will get requests for the same length of arrays over and over. This is the whole point in this memory strategy. If array sizes could change over time, then the whole concept would be ruined.

However I have been thinking of monitoring the list size more closely to catch if one of the lists grows uncontrollably. Maybe it should free arrays when the list content exceeds a certain number.

Maybe I'm wrong? ) Btw, I have 2 GB RAM and original colonization 2 becomes laaagy after 200 turns. Town screen is updated in 2-3 seconds after you move any colonist.

I added much optimization, and we do some tasks noteworthy faster than vanilla. This is mostly related to caching often used, but rarely changed values, like profession yield costs. In fact JIT arrays was originally added to cache profession yield costs as it would skip allocating arrays for all the professions without yield costs (most of them).

 

[Kailric]

Hey, Night, we need a Jit array for Road types and Improvements that are allowed by the player. At the moment it looks like we still have to cycle through all the Civic types in order to find out which ones are allowed. If you want to set this up, that be great as you'd be faster than me.

Also, in the Medieval_Tech.vcxproj has vanished with this last pull. I see the Colonization Mod Collection.vcxproj, but when I open it all the MEDIEVAL_CONQUEST yields are showing as undefined and I don't see were they are defined, is this normal?

Edit: Ok, I see they are in MOD_Yields but these files where not shown when I complied. But the games seems to load just fine, whats happening here?

 

[Nightinggale]

Hey, Night, we need a Jit array for Road types and Improvements that are allowed by the player. At the moment it looks like we still have to cycle through all the Civic types in order to find out which ones are allowed. If you want to set this up, that be great as you'd be faster than me.

I will look into it. The git log should provide a decent guide on how to do that for later (it isn't that complex).

Edit: Ok, I see they are in MOD_Yields but these files where not shown when I complied. But the games seems to load just fine, whats happening here?

I need to update the include path in the project. It is set correctly in the makefile. I will do that after my next commit. It worked at one point, but then I broke it by fixing another path related issue.

However before doing any of this, I will have to finish my uncommitted changes or it turns messy

 

[Kailric]

I will look into it. The git log should provide a decent guide on how to do that for later (it isn't that complex).


I need to update the include path in the project. It is set correctly in the makefile. I will do that after my next commit. It worked at one point, but then I broke it by fixing another path related issue.

However before doing any of this, I will have to finish my uncommitted changes or it turns messy

ok, My "just in time" is limited right now, so I was attempting to delegate some tasks I am not as familiar with and work on something that I am familiar with. But I do need to practice doing the jits.

What I was trying to do was make it so that city tiles auto update their roads when the player gains a new road tech. This is how Civ4 does it.

Source has an "issue" system we can use right?

 

[Nightinggale]

ok, My "just in time" is limited right now, so I was attempting to delegate some tasks I am not as familiar with and work on something that I am familiar with. But I do need to practice doing the jits.

It shouldn't be that hard, but then again I designed it and never went though the learning process

What I was trying to do was make it so that city tiles auto update their roads when the player gains a new road tech. This is how Civ4 does it.

Sounds like we need both available routes(JIT) and best available route (int).

Source has an "issue" system we can use right?

https://sourceforge.net/p/colonizationmodcollection/sourcetickets/

Maybe we should call it something else though. I didn't really think of the name. Would it make sense to called it "shared resources" and use it for DLL and python?

I just added it. There are a bunch of setup options, which I haven't looked into yet. Presumably we should make one for each mod. However first I need to figure out if I can move ticket from one mod to another as people will likely post M:C bugs in M:C even if they are DLL issues.

 

[Nightinggale]

Also, in the Medieval_Tech.vcxproj has vanished with this last pull. I see the Colonization Mod Collection.vcxproj, but when I open it all the MEDIEVAL_CONQUEST yields are showing as undefined and I don't see were they are defined, is this normal?

Looks fine to me. Try pull and then select sourceDLL and click log. You should hopefully see that I just pushed a JIT update (save related). Also make sure it says "sourceDLL (master).

I need to make a guide with screenshots as I think it will be quite common to run into not pulling from the right branch, or the default, which is not to pull source at all after it was pulled the first time

 

[Nightinggale]

Hey, Night, we need a Jit array for Road types and Improvements that are allowed by the player. At the moment it looks like we still have to cycle through all the Civic types in order to find out which ones are allowed. If you want to set this up, that be great as you'd be faster than me.

Done.
http://sourceforge.net/p/colonizati.../ci/c77d0e39b78c55277691df177ff2927769bc891d/
It's not used anywhere yet, but the data is now easy to access and it is virtually instant.

I used BoolArray instead of JIT. The commands for those two are intentionally identical. The difference is that BoolArray knows each array element is just a single bit. This mean it will do some other stuff internally and use less memory.

Writing this makes me remember something about hardcoding some values to max 32 (EuropeTypes I believe). That gave me the idea to use the 32 bit pointer as storage instead of allocating 32 bit. I might do that eventually, though it isn't important. As it is an internal change, the calls would still be the same.

 

[Kailric]

Done.

See there, pretty quick

I am attempting to push all the Art files in 100 mb paks. There are a couple of Unit and Leaderhead paks that I left with lower MB so we can add to those, they are labeled with the term "Edit". So, when they fill up to 100mb we can another Edit file.

It's currently pushing the changes. Slow go

 

[Nightinggale]

There are a couple of Unit and Leaderhead paks that I left with lower MB so we can add to those, they are labeled with the term "Edit".

The general idea is that once they are packed, we don't modify the file as a modified file might mean a brand new file. The reason why I said small files is if we have to modify anyway. We shouldn't plan to edit any of them if we can avoid it.

 

[Kailric]

The general idea is that once they are packed, we don't modify the file as a modified file might mean a brand new file. The reason why I said small files is if we have to modify anyway. We shouldn't plan to edit any of them if we can avoid it.

What are we doing with new art? If I add ten units and a couple leader heads am I suppose to create a new pak file each time? I figured it would be easier to just add the files to the lowest mb pak file then repush that one file. I should just name them all by numbers and add files just to the highest number until it fills up then create a new file.


What was your plan for adding new art?

 

[Nightinggale]

What are we doing with new art? If I add ten units and a couple leader heads am I suppose to create a new pak file each time? I figured it would be easier to just add the files to the lowest mb pak file then repush that one file. I should just name them all by numbers and add files just to the highest number.

What was your plan for adding new art?

Just adding new files unpacked. Eventually there are plenty of unpacked filed and then we repack or pack them. While this will not eliminate the need for repacking, it will greatly reduce the number of times it will be needed. The main problem is that each time somebody clones, they not only get the current version, they also get all the outdated versions and while that isn't a problem with text based files, it is a problem with often updated 200 MB files. Git has some clever packing technique to reduce the storage needed for text changes, but they will not work on binary files.



New hardcoding problem
Colonization 2071 has just uncovered a new problem. It turns out that CONTROL_CIVICS_SCREEN is hardcoded in the DLL. Col2071 lacks any reference to CONTROL_CIVICS_SCREEN in XML and gives weird assert messages, which tells nothing of what the problem really is. We could add checks in the DLL like we have with yields, but then we should likely do it with other hardcoded data too and we will end up with a bunch of testing code, which can be annoying to update.

Hardcoded XML stuff. There might be more as I didn't really search for it.

Spoiler :

]int CvGlobals::getNumDirections() const { return NUM_DIRECTION_TYPES; }
int CvGlobals::getNumResourceLayers() const { return NUM_RESOURCE_LAYERS; }
int CvGlobals::getNumUnitLayerOptionTypes() const { return NUM_UNIT_LAYER_OPTION_TYPES; }
int CvGlobals::getNumGameOptions() const { return NUM_GAMEOPTION_TYPES; }
int CvGlobals::getNumMPOptions() const { return NUM_MPOPTION_TYPES; }
int CvGlobals::getNumSpecialOptions() const { return NUM_SPECIALOPTION_TYPES; }
int CvGlobals::getNumGraphicOptions() const { return NUM_GRAPHICOPTION_TYPES; }
int CvGlobals::getNumTradeableItems() const { return NUM_TRADEABLE_ITEMS; }
int CvGlobals::getNumBasicItems() const { return NUM_BASIC_ITEMS; }
int CvGlobals::getNumTradeableHeadings() const { return NUM_TRADEABLE_HEADINGS; }
int CvGlobals::getNumCommandInfos() const { return NUM_COMMAND_TYPES; }
int CvGlobals::getNumControlInfos() const { return NUM_CONTROL_TYPES; }
int CvGlobals::getNumMissionInfos() const { return NUM_MISSION_TYPES; }
int CvGlobals::getNumPlayerOptionInfos() const { return NUM_PLAYEROPTION_TYPES; }
int CvGlobals::getMaxNumSymbols() const { return MAX_NUM_SYMBOLS; }
int CvGlobals::getNumGraphicLevels() const { return NUM_GRAPHICLEVELS; }

A semi related problem is that the old setup could script copying XML schema files from M:C to the other mods. The current setup lacks this ability. I'm not sure what to do about that either.