DLL coding thread

[Nightinggale]

Don't hardcode the index into python. If you need to hardcode, then do it in the DLL and give python access to those numbers. However the best would be to generate the numbers when loading XML and the problem would be gone.

"Techs, Trade Perks, and Censures". Then why not have a variable with one of these values?

Imagine this: we have CivicCategory with one of those names (which are handled as ints).

We then have civic option type to split each of those categories into subtypes, like we do now, except it would be tech categories if CivicCategory is tech.

It will need closer inspection and thinking if I were to declare what I think would be best in this case. I'm just brainstorming based on previous post alone.

 

[Kailric]

Don't hardcode the index into python. If you need to hardcode, then do it in the DLL and give python access to those numbers. However the best would be to generate the numbers when loading XML and the problem would be gone.

Well, like I said I am trying to steer clear of hardcoding altogether, so XML loading sounds best.

"Techs, Trade Perks, and Censures". Then why not have a variable with one of these values?

Imagine this: we have CivicCategory with one of those names (which are handled as ints).

We then have civic option type to split each of those categories into subtypes, like we do now, except it would be tech categories if CivicCategory is tech.

It will need closer inspection and thinking if I were to declare what I think would be best in this case. I'm just brainstorming based on previous post alone.

I am not quite following you here. Sounds like you want to go back to making new CivicOptions for the categories, but remember why I removed CIVICOPTION_INVENTION in the first place. So much of the CivicOption code builds an array based on the Number of CivicOptions. So, I would have to go back to manually removing those extra CivicOptions from the arrays so there would not be errors.

 

[Nightinggale]

Say we have CivicCategory. If we set it to Censures, then CivicOptions stay the way they are now.
If we set CivicCategory to Invention, then CivicOptions would be XML indexes, but indexes in another XML file. In other words CivicCategory becomes an XML file choice.

Alternatively we could have a CivicOptions variable for each XML file, but then we could end up with 3 or more variables using a one hot system. That isn't ideal considering we would need 3 times the code for a whole lot of stuff. Having just one saves a lot of coding, but then if we need to split, we have to use a combo of CivicOptions and CivicCategory.

 

[Nightinggale]

I made a new branch called JITarray_save_rewrite. It branches off the civic branch instead of master as it relies heavily on the type variable I added to JIT arrays in that branch. Would that make it a branch-branch like mods of mods are called modmods

I realized we can get several bonus exploits from this change on top of keeping savegames during XML changes.

JIT arrays no longer needs a class of their own. All they need is the base class and an enum value in constructor. I added two functions in CvGlobals.cpp with switch-cases where each type should be present. These functions tells array length and string from get*Info(index).getType().
It shouldn't be hard to add new array types using this system.

As the load code will fill in default data for indexes not present in savegames, we can skip saving a lot of data. Imagine a unit having the 5th promotion as the only one. Saving will then save the first 5 and then the rest will be filled out with the default false. I suspect savegame file size have a potiential to shrink somewhat due to this.

The whole arrayBitmap when loading and saving can be removed as this feature moves into JIT. This makes saving arrays a lot easier. All it will take is using the array member functions read and write with pStream as the only argument.

More improvements to come (hopefully). Feedback and ideas are welcome.

WARNING this new branch will not upload the DLL for bandwidth reasons (it isn't needed) and it is likely to break savegame loading entirely once in a while. It isn't meant to be played until it is declared finished and merged back into playable code.

 

[Nightinggale]

Looks like I'm approaching the end of coding this new branch, which mean I coded it much faster than I expected. That is presumably to make up for the slow progress of the language feature

What's left is to ensure that all saved arrays are JIT arrays. If we add a unit and just a single saved array isn't a JIT array, then the savegame will be broken. This mean quite a lot of code reviewing. After this reviewing phase, the actual testing can take place (that will be interesting).

Right now I butchered the ability to load old savegames to an extend never seen before and I will not try to correct it. This mean right now would be an exelent time to go through all load code and delete savegame conversion code as it will only make the code more messy. We will never need it again.

I'm sorry that my progress reports doesn't look as interesting as Kailric's, but it would be rather pointless to include screenshots for a savegame format

 

[Nightinggale]

Progess on converting array types into JIT arrays

• JIT_ARRAY_BONUS,
• JIT_ARRAY_BUILDING,
• Building class
• JIT_ARRAY_EUROPE,
• improvements
• JIT_ARRAY_PLAYER,
• JIT_ARRAY_PROFESSION,
• JIT_ARRAY_PROMOTION,
• JIT_ARRAY_UNIT,
• JIT_ARRAY_UNIT_CLASS,
• JIT_ARRAY_UNIT_COMBAT,
• JIT_ARRAY_YIELD,

(green is yet to be added JIT arrays)

I have been thinking of adding improvements to the list as well as I assume adding improvements wouldn't be an unlikely thing to do. Any ideas for other XML files where it would be beneficial to keep savegames despite XML changes?

I have run into another problem: player traits. One effect is free buildings, which is a building array. However I wonder if we really need to save the effect of the traits. If we instead recalculate the effect on load (like civics), then savegames will not only update to XML changes, but also be more resistant to changes (increased change of correctly loading old savegames). This will need further investigation to see if calculating everything on load is a doable solution.

 

[Kailric]

I have been thinking of adding improvements to the list as well as I assume adding improvements wouldn't be an unlikely thing to do. Any ideas for other XML files where it would be beneficial to keep savegames despite XML changes?

Yes, improvements will be good addition as well as I plan to modify the Improvements. I'll keep this in mind as I go about my playtesting. Thanks for these additions

 

[Kailric]

One thing, I tried adding a new cached defined int with my Anarchy civic DEFAULT_GLOBAL_EFFECT_ANARCHY. But the XML_DEFAULT_GLOBAL_EFFECT_ANARCHY returns a random "47", which is Dikes tech. I followed your examples but thats what come of it. For the time being I am just using the vanilla way of getGlobalDefine.

I looked over the code and can't see where I went wrong, I know I am missing something...

 

[Nightinggale]

I will take a look later. I can't right now because

1. I out of coding time right now
2. github's git server is down and I can't pull your changes

#2 is also the reason why I have a local commit, which I can't push. I will write about it when it is pushed as it is a pretty good example on how to use Just-In-Time arrays.

I also located bug regarding enabling trade screens (I think it's pre branch), which can result in accessing an array at index -1. That can lead to semi-random crashes. Moving to a JIT array made it assert on this error, which made detection and localization of the bug very easy.

 

[Nightinggale]

Github is back online. I pushed my latest commit and here it is: https://github.com/Nightinggale/Medieval_Tech/commit/674d1bfd376a6bdb6340ed2dcb99df3e9987a81a
It pretty much tells how to use JIT arrays. Read it, ask if you don't get it and then use it

 

[Nightinggale]

One thing, I tried adding a new cached defined int with my Anarchy civic DEFAULT_GLOBAL_EFFECT_ANARCHY. But the XML_DEFAULT_GLOBAL_EFFECT_ANARCHY returns a random "47", which is Dikes tech. I followed your examples but thats what come of it. For the time being I am just using the vanilla way of getGlobalDefine.

I took a quick look and it looks like it will need some closer examination to figure out this issue. Strange that it should cause such an issue because the code should be simple enough
I don't know precisely when I will do that, but it isn't urgent considering we have a working workaround.

Add comments to your vanilla approach, like

// TODO needs to be cached

That way we can quickly find all those places once the cache works correctly.

You do know that TODO is a reserved keywork in VC, right? You can autogenerate a TODO list with every TODO comment, which looks like it could be a nice tool. I haven't used it much, but I think we should, specially in cases like this one.

 

[Kailric]

You do know that TODO is a reserved keywork in VC, right? You can autogenerate a TODO list with every TODO comment, which looks like it could be a nice tool. I haven't used it much, but I think we should, specially in cases like this one.

That is very good info, I will start using that.

I looked over that jit code and it looks easy enough. But, soon as I touch it I'll probably get a return of the temperature of some lake in Alaska

 

[Nightinggale]

I looked over that jit code and it looks easy enough. But, soon as I touch it I'll probably get a return of the temperature of some lake in Alaska

If you do that, then you can't figure out regular arrays either.

JIT|vanilla
array.get(index)|array[index]
array.set(value,index)|array[index] = value
array.add(value,index)|array[index] += value

First argument in the constructor is the default value (0/false if you skip it). Any other details can be read in comments in the header files on each function.

 

[Nightinggale]

I passed the 50% marker
In addition to that, I have so much new in the savegame now that I can easily make at least simple tests. I did locate a single bug (adding to the end of vector instead of overwriting the right index). Now it seems like it loads the savegame correctly and determines the values based on type strings rather than index. Removed types will be given the value -1. This makes sense as say a plot with a removed improvement will then get the improvement -1, which is NO_IMPROVEMENT. In other words it is just removed and the game is fully playable anyway.

Seeing that nobody actually proposed any new XML files for upgrading, I went through the files and picked improvements and building classes. Maybe terrain and features could be beneficial as well.

While I work on the code, I also make small adjustments, like using 8 bit for storage if possible. As vanilla uses 32 for the same values, the memory used on those arrays will be reduced by 75%.

Speaking of memory reduction, I have started to wonder if I can make a class wrapper for JIT arrays, which have an array of ints and then it use each bit as a bool. This mean a bool can be stored in one byte instead of 32. If it stores the bools as ints, it can then also save as one byte instead of 32, making savegames somewhat smaller. I haven't figured out the details for this and maybe it isn't a great idea once the grim details appears.

 

[Nightinggale]

I have come up with a new strategy for implementing XML change acceptance. Rather than looking at it by one XML file at a time, I decided to go with one C++ file at a time. Here I will go through the read() functions line by line. That way we should be able to end up with being able to load savegames more or less regardless of what happened in XML. There are a few limitations, such as it will not behave nicely to getting a unit deleted when it is present in a savegame.

I also added a new read function.

1. pStream->Read((int*)&m_eProfession);
2. pStream->Read(JIT_ARRAY_PROFESSION, (int*)&m_eProfession);

The first is the current one (still functioning) while the latter reads the same and then it converts according to JIT_ARRAY_PROFESSION rules. This way a single int can be converted when needed. A good example of this would be current profession for a unit.
EDIT: scratch that. I came up with a better idea. Just do

pStream->Read(&m_eProfession);

The read engine will then detect that it is a ProfessionTypes pointer and automatically apply profession conversion rules to the int it reads.

I think I will try to store bools in singe bits before going through all the classes. That way I can convert for both bool size and XML conversion at the same time.

Changing to coding approach now mean it will likely take more time to finish this, but if we end up with a better result, then it should be worth it.

 

[Kailric]

Changing to coding approach now mean it will likely take more time to finish this, but if we end up with a better result, then it should be worth it.

Sounds like you are having a blast with this I wish I could be of more help, best I can do is stay out of your way

There is one area I plan on working with that may could use your additions and that is the AI, mostly CvPlayerAI. I plan on adding new StrategyTypes and new MemoryTypes. Working with the AI is actually one of my favorite parts but I rarely get to do it as I spend so much time with graphics and adding new features, but eventually I will get to it.

 

[Kailric]

How about a jit for Traits or will that be included into JIT_ARRAY_PLAYER? I am adding a new trait and just thought about this.

 

[Nightinggale]

How about a jit for Traits or will that be included into JIT_ARRAY_PLAYER? I am adding a new trait and just thought about this.

It isn't as JIT_ARRAY_PLAYER is just max num players (32). It isn't linked to any XML file.

The new approach I came up with mean I will go through all read() functions and then add the JIT array types needed. I'm not sure traits needs to be added as maybe the leader is saved and then it uses the leader to look up traits. This approach is used in some places meaning some XML changes will not affect savegames even in vanilla.
This approach mean I haven't planned which XML files, which should end up as JIT arrays. Adding more later on is fairly strait forward meaning it will not be a horrible disaster if we miss one or two.

Sounds like you are having a blast with this

I came up with a new approach because I got sick of the repetitive tasks involved in the first approach.
Having a blast reminds me of the translation where visiting a friend and having a blast ended up being translated like "I visited X. It was very windy there"
Still doesn't beat the Finnish major blunder where "I'm going to see magic Johnson play" became "I'm going to see magical Jonathan play music"

 

[Nightinggale]

While looking at the code I noticed that several arrays use int regardless of the size needed. We have the available sizes:
int(4): +- 2 billion
short(2) +- 32k
char(1) -128 to 127
unsigned char(1) 0 to 255
Numbers in () mean number of bytes used for each element in the array.

There is nothing horrible in using a too big size. It will work, but it will increase memory usage and memory I/O.

Looking atCvCity, I found m_aiConnectedTradeBonus and m_aiConnectedMissionBonus. I suspect the ability to assign values in the million range is a game breaking bonus size

It may be a good idea to go through all headers and evaluate all array sizes when I'm done with array conversion. It should then be possible to search for Array<int> to find all of them.

 

[Kailric]

Looking atCvCity, I found m_aiConnectedTradeBonus and m_aiConnectedMissionBonus. I suspect the ability to assign values in the million range is a game breaking bonus size

Yeah, I wondered about that, but since the vanilla coders didn't bother with anything less than an int most times I didn't either. I think you mentioned it before but memory isn't all that big an issue these days. Looking at my Task Manager with 9 Apps running including M:C it shows FireFox here using 290 MB while M:C only uses 230 MB. With all the other apps running it still shows 51% of my memory still available. The crazy thing is I don't even know what all that means, if anything at all.