Current and Future Developments

[Nightinggale]

I just had another idea. Usually a C++ class is split into one header file and one cpp file, both with the classname as filename. However that is to make the code human readable and we can place the cpp code wherever we like as long as we include the header. I want to exploit this by moving all the readWrite functions into a single cpp file.

This file will not focus on classes, but rather savegames. By moving everything savegame related into a single file and not using that file for anything else, we can get git to provide a log for that file, that is a log of changes in the savegame format. Keeping everything in one place will also help getting an overview of how and what is saved.

I started thinking in those lines when it hit me that the savegame version string should be the first to be saved. If we have savegame compatibility issues and a savegame is unreadable for whatever reason, getting the string before it gives up on loading the savegame would be a good idea. However that approach naturally came with a new question: which read/readWrite function is called first? CvGame is a likely candidate, but I'm not sure. I realized I would have to debug to tell the load order of all read functions, which then came with the question: where are "all" those functions? They are spread out all over the code with usually only a single one in each file. If we are to make savegame compatibilty a priority, we better conquer the savegame format once and for all.

Looks like adding a version string to savegames had become somewhat bigger than I first planned, but at the same time I really like this approach. It will be much easier to figure out what we can and can't do. Having a single file would also be a valid place to document the overall savegame structure and other savegame related documentation. The top part with comments might become a big bigger than the average cpp file

I just pushed another batch of textmgr additions. That's a pain in the rump. There is still a bunch to go so if you want to knock some of that out it be great. The getAllowsImprovementslooks the most confusing and I've been skipping it

Yeah I can look into those. In fact maybe I should take over. When I asked you to look into it, I had problems getting CivEffects to run stable and had too much to do for one person. Now CivEffects seems to work and there is little dll work left. At the same time we reached the xml verification stage. I think it would make more sense if you start to focus on xml now to fix the glitches the CivEffect conversions script made.

I remember the improvement stuff as being more complex than the rest. Normal allow/modifiers would be 2D as it has say UnitClass and modifier. Improvements ahve ImprovementType, TerrainType, and modifier. Routes are somewhat the same and I can't remember if there are more tags using 3D. I need to read more on how they are set up as I remember them as something not that tricky to set up for the xml modders, but I can't remember what I came up with.

 

[Kailric]

I want to exploit this by moving all the readWrite functions into a single cpp file.

Man, you are a beast, no job to big

Yeah I can look into those. In fact maybe I should take over. When I asked you to look into it, I had problems getting CivEffects to run stable and had too much to do for one person. Now CivEffects seems to work and there is little dll work left. At the same time we reached the xml verification stage. I think it would make more sense if you start to focus on xml now to fix the glitches the CivEffect conversions script made.

It will take me a while to finish those as I just punch at it every once in a while and you'd be way faster. The majority are the infoarrays now. One of the big pains is having to track down the correct XML text keys. We need to coordinate who is doing what and when. I'll look for you online tomorrow. I could hunt down the XML text keys and past them as a comment over the commented function. If you haven't noticed I went through and added the functions as comments so I would know which ones still need to be done. When it is done, I delete the comment.

 

[Nightinggale]

I went through the help text and fixed some issues. Sadly before I committed it, you did something too and I ended up with 5 conflicts

I handled those and rebased my changes to get a linear history again. The only issue is that I mentioned fixing civic and tradescreens in the log and you did too. I guess that's really a minor issue. On top of those two issues, I changed some loops and stuff to become more strongly typed.

Also I made the prospecting level change depend on prospecting being enabled in the game or not. Next is the goody version of prospecting. I plan on something simple like skipping prospecting ones when making the list of possible goodies to get.

If you haven't noticed I went through and added the functions as comments so I would know which ones still need to be done. When it is done, I delete the comment.

I noticed. The real question is if they appear in the same order as in xml and is the list complete? My plan was to do that to make it easier to locate a specific one in the future. Each group is sorted alphabetically in xml, meaning nothing in the order is random.

 

[Kailric]

I noticed. The real question is if they appear in the same order as in xml and is the list complete?

That I don't know. As far as being complete I don't remember how I even come up with the list, copied them from somewhere or did I even do that

Anyway, when it comes to wondering did we get them all, we'll just have to reevaluate the Civeffects when testing to be sure they state what they are doing and if they could be improved or balanced different.

 

[Nightinggale]

We will have a testing period to see if the pedia shows whatever we put in xml. While we should focus a lot on it now, it is something we should always keep an eye on as even a perfect code now can be rendered outdated.

 

[Nightinggale]

I got tired of help text and went for perl instead. I figured if we should run the header generator each time the compiler is started to provide debugger with the xml tags and possibly for hardcoding optimization later on, then it needs to be fast. I wrote a disk cache to save the output between runs, meaning it only examines an xml file if it is modified after the cache was generated. As people usually modify 0-5 files between runs and it caches round 50, it greatly reduce the amount of file content it needs to look through. The enum file is only generated if the cache was updated and the enum file is only written to if the content actually changed (as in something changed in xml, which doesn't change the header).

I managed to get the perl script to use all CPU cores instead of just one, like the fast compile. The execution time for when nothing changed in xml is reduced from 0.6s to 0.16s. I think that is low enough to accept it to run each time we compile.

Using this script and the multiple CPU cores we can make it do all sorts of precompile tasks, providing us with much greater freedom for compiler options and setup.

 

[Nightinggale]

I started moving all the savegame functions into Savegame.cpp. It now contains 24 readWrite() functions, which is all of them. However as it turns out, there are functions, which was never converted to this new format, but they should be. The task just suddenly got bigger, but at the same time, converting the rest makes the savegame less likely to be broken by xml changes. The goal is to give as much freedom to the xml modders as possible.

I'm wondering about making some more generic approach to make it easier to identify and read savegames from any version. I got my eye on uiFlag. It's saved in every instance of a class. However I want to start each savegame with a string telling the git tag and revision and then the uiFlags for all classes. This will allow us to set a breakpoint after reading those, which would allow us to closely examine how the savegame identifies itself even if it fails to load later on. Precisely how to do this is still unknown, but I'm thinking something in the line of making an enum for each class and then a global (to that file) instance of the variable. This will allow setting it in one class and reading from it in another. It will also allow setting a human readable enum value rather than just a number.

I think I will try that approach with plots. Saving the variable first also mean it will be saved once instead of once for each plot which naturally affects savegame size. Not by a great deal, but still some.

I wondered about making a single global uiFlag shared by all classes, but figured it would be problematic. If I change CvCity and Kailric changes CvUnit, then we would conflict. With a value for each class, we will have to modify the same class in order to conflict. There are a few classes where sharing the uiFlag could make sense, like unit and unitAI as they are always saved together anyway.

Now that I have written this in CivEffect, I realized it will conflict if we have changes to the savegame format in any of the other branches. I will just have to deal with that when we merge, but it brings up an interesting question? When should we merge? We are past the time where we call CivEffect experimental, right? It's playable and while there could be bugs, it's not a flawed design we will have to discard. It's sort of the same for the other branches, which mean why delay merging? The only thing I can think of is because the CivEffect xml files still needs to be filled out.

EDIT: btw I'm not caring about preserving savegames while working on this. I think I already broke them in CivEffects already and I have nothing I really want to keep, at least not important enough to bother coding to preserve it. This mean I will break savegames in CivEffects branch without warning and possibly multiple times!

 

[Kailric]

Merging sounds like a good idea. That way I can play test the Civeffect branch and make my changes and adjustments there while also adding in the last of the xml texts.

 

[Nightinggale]

Ok, I will merge once I'm done with this savegame stuff and is done breaking savegames.

I have learned some stuff from working on this and not all good. The first read file to be called in CvInitCore and it contains some variables, which should be converted according to xml edits in the meantime. However I saved the conversion data in CvGame, which isn't read yet. This mean either I have to teach the computer to predict the data it hasn't read yet or I should move the whole conversion into CvCoreInit. I don't feel like doing either
Still it should be done as not doing it mean savegames can break even though I made the rest of the savegame code handle the change. Stopping now with just a single class left seems pointless.

The good news is that I added a call to readWrite as the first thing in CvInitCore for both reading and writing. This mean I now have a line, which will be the first thing in the savegame. That is the original goal for doing this, which mean I unlocked the ability to handle a string to contain the git revision.

Also I'm planning on reviewing all xml files vs what is put in the savegames vs what is converted on load. We should know if any of them can pose a danger for savegames if they are changed.

 

[Nightinggale]

Done getting CvInitCore to store the conversion table. However something weird is going on. It seems that CvCoreInit is saved twice

I debugged loading and the second load does in fact overwrite the memory for the first time. Thinking it would be a waste of space, I made the second call return, ensuring only the first one does anything. However this makes the exe crash. There is something spooky going on here. I decided to open the savegame in a hex editor and it turns out savegames aren't compressed at all. While bad for disk space, it's good for this purpose as I can read all the strings used in the savegame. The strings for the conversion table are only present once. This mean for some really odd reason, the game loads the first class, then it discards that one and loads everything from the start.

I can only imagine the exe wants to preview something, like game name, but why start over when it has the first data?

The hex editor reveals that saving all the strings to make the conversion table use 29592 bytes, or around 30 kB. It makes up almost half of the savegame in my newly started game. However this size doesn't depend on the game size, but only the number of xml types and their string lengths. This mean when you have played for a while, the conversion part of the savegame will be 30 kB out of 500 kB and suddenly it's no longer big. The largest savegame I have is a stunning 1.9 MB and appears to be a RaR game. I knew the conversion table would take up some space, but I had no idea how much. I feared it would be worse than this and it is clearly acceptable.

To a reminder of when I made the conversion table, or just before I did, we talked about it and we talked about C2C already having one. I looked at the code and decided to make one from scratch instead of copying their code. The main reason was that I didn't want to maintain their code as I had a hard time following how it works. Now our code is easier to use as it is integrated into JIT arrays and Read() (yes it's triggered by type in arguments, not special code). However our code comes with the bonus of being faster and C2C claims that adding the strings increase filesize by around 30% and it scales with the savegame size. I'm suddenly even more happy that I decided not to just copy paste some code

Now that I can see what the game is doing with the savegame format and the hex editor, I will try to experiment and see how it reacts to saving different data. For instance will it just save byte by byte or is there some alignment? Will saving a char use just one byte or 4? Will saving 4 chars individually use more space than using the function to save an array of 4 chars. Knowing data like this mean I can more efficiently code to reduce the savegame size. JIT arrays already do math when saving and prevents saving default data as unsaved data will be default on load. If a codechange in this one place can affect all saved JIT arrays, then it should be noticeable on the resulting filesize.

Oh and I should do it right now. I just broke savegame compatibility and doing something like this will likely break it again.

 

[Kailric]

Heh, it seems the devs are haunting us once again. It is like they layed booby traps all over and you just never know when you will trip on one

 

[Nightinggale]

I modified the dll to save a bunch of bogus data and then I read it with the hex editor to see how it would react to different inputs. It turns out that the savegame format is quite simple. The exe writes the data first, like the name of the mod used to load the savegame. After that it just writes whatever the dll tries to write and it is a simple byte stream. It has no alignment or anything. It writes in little endian, which hints it just dumps the content of the memory.

Simplified: it just writes the variables provided to Write() with no extra data. If it's a char, it will write one byte. If it is an int, it will write 4, the same as those use in memory.
There is no data to tell where it belongs in memory. This mean it works purely by the principle that you write 5 ints, you read 5 ints and you know what goes where based on the order. The game has no way of knowing if you switch the reading order of two on load. That's the job of the modder to prevent (hence the readWrite() function).

Using this newfound knowledge of savegame disk usage, I can shrink saved JIT arrays even more. Let's make an example. We have an int array of length 50. index 15, 17, 18 and 35 are set to 1, the rest the default 0. I also added the disk space to save the empty array, as in all data is 0.

Vanilla saving:
200 bytes (4*50)
Empty: 0 or 200 bytes (likely the latter, depends on implementation)

JIT array (current):
148 bytes (4*36+4 overhead)
Empty: 4 bytes

JIT array (optimized with my old, yet unimplemented idea of skipping the leading defaults):
88 bytes (4*21+4 overhead)
Empty: 4 bytes

JIT array (block based concept):
28 bytes (16+4 overhead and 4+4 overhead)
Empty: 4 bytes

New compressed concept based on knowledge of savegame format combined with the block concept:
10 bytes (4*1+3 overhead and 1*1+2 overhead)
Empty: 1 byte

Considering that JIT arrays makes out a fair amount of the savegames and their share of the savegame grows as we add more and more content to the game, I would say that it's worth the time to dig deeper into this part of the code. Now I'm actually happy that CvInitCore is saved twice because without it, I wouldn't have started investigating the savegame format and it looks like it was quite rewarding to do so. Now I just need to make a well defined format for the overhead system, which use as few bytes as possible, but at the same time is able to store all the information for every single situation that can show up. Incidently this task is likely the modding task closest to my engineering training and I'm actually looking forward to it

How is it that I trigger the autoplay feature? I want to make the AI build up a game with lots of data and then modify write to see how much the savegame shrinks, then load to make sure it can read it as well. Since I might break savegame compatibility multiple times in the process, I better be able to make new testgames fast.

In semi related notes we need to make a developer cheat menu with either access to all those features or at least a list of hotkeys to remember. I have no overview of what we can do and I always forget the hotkeys, which aren't on the screen.

 

[Lib.Spi't]

I think autoplay is ctrl + shift + X

that's what it is for bts.

 

[orlanth]

I think it may be Ctrl shift z per rays thread here
http://forums.civfanatics.com/showthread.php?t=475319

Also there is a small bits modcomp that enhances the auto play feature slightly, this may be potentially useful for debugging if it's feasible to add to c4c.
http://forums.civfanatics.com/showthread.php?t=174812

 

[Kailric]

How is it that I trigger the autoplay feature? I want to make the AI build up a game with lots of data and then modify write to see how much the savegame shrinks, then load to make sure it can read it as well. Since I might break savegame compatibility multiple times in the process, I better be able to make new testgames fast.

The autoplay feature I designed is always on the helpful hint part of the pedia. There is several options so just check there.

 

[Nightinggale]

It's actually ctrl V. It shows the need for a debug menu as 2 people proposed two different things, I considered a 3rd and it turned out to be the 4th. Ctrl shift Z gives you control of player (yourplayer+1). Pressing it multiple times will eventually let you end up with the player you want to switch to.

I like the autoplay in RaR better. It asks how many turns it should autoplay for. We are stuck with 25 turns each time the key combo is activated.

Anyway I made "big" games with autoplay and they ended up as not very big. Even on the biggest map and playing 200 turns, the savegame is just under 200 kB, or 10% of RaR savegames. Looking at a huge RaR savegame with the hex editor didn't reveal anything, which could indicate the size difference. It's not just writing a bunch of 0 arrays. There are some, but they are like a few kB. Sure there is a huge difference in map size, but the map isn't 10 times bigger.


I changed all the uiFlags to be saved in the start of the savegame rather than at each instance of a class. This reduce filesize by a few kB (16 kB from plots alone on a 50x80 map) and it provides the version of the savegame before loading the savegame itself. This mean if it fails to load, a debugger can read the version before it fails to load.

I wrote my own read/write code for strings and it's surprisingly simple.

Spoiler :

void FDataStreamBase::ReadShortString(CvString& szName)
{
	szName.clear();
	char iTemp = 0;
	Read(&iTemp);
	while (iTemp != 0)
	{
		szName += iTemp;
		Read(&iTemp);
	}
}

void FDataStreamBase::WriteShortString(const char* pString)
{
	const char* it = pString;
	while (*it != 0)
	{
		Write(*it);
		it++;
	}
	char iTemp = 0;
	Write(iTemp);
}

It just saves the string char by char and in the end, it adds a 0, or to use the correct technical term, the strings are null terminated. This mean it use one byte for each character and one byte to separate two strings. This saves 3.8 kB. One has to wonder why the exe applies 14.5% extra bytes when saving precisely the same thing

I added the ability to make a JIT array get a pointer to another (normal) array while read/write. The trick here is that if it gets a pointer, it will read/write with that pointer instead of itself (has to be the same length as itself). This allows saving arrays with JIT array layout in savegames without actually converting to JIT arrays in the code. I made PlayerAI use it for the 4 arrays storing UnitAI data and in the debugger, I learned that it works and that the array I looked at used 10 bytes less in the savegame. Not a big save, but the main goal here is that using the JIT array read/write code will not just save space, but also convert to the new xml layout on load. This mean adding a new UnitAI type will no longer break savegames.

I think things are going well with the redesign of the savegame format. Remember the goals are the following in this priority:

1. DLL updates should not break the ability to load existing savegames (starting from next release)
2. XML modifications should not break existing savegames (removing units or similar, which are in use could be problematic. Adding new units or similar should be safe)
3. Make them easy to maintain/easy to modify without introducing bugs
4. Read from xml/recalculate instead of saving data if possible. This makes xml changes apply to existing savegames
5. Make the savegame format use as little disk space as possible as this is the best weapon against exploding savegame sizes in the future

For the first time ever I feel like I'm actually close to reaching all those goals

One thing I haven't mentioned before is another reason for keeping savegames small. When joining a network game, the server pause, saves the game, transfers it to the joining client and unpause once the client has read everything and has joined. Using half the savegame size means the "player joining freeze" wait is cut in half. If possible, I will not include the strings for the conversion table in savegames if (and only if) I can 100% sure identify the savegame as a multiplayer joining savegame because in that case the server saving and the client loading should have the same xml and no conversion should be needed. In other words, working on the savegame now will hopefully result in very quick multiplayer joining in the future, particularly on slow internet connections (quick being relative to how it could have been). Sure I would have to fix all the desyncs first or it would be pointless, but I want to do that as well... someday.

Also because I completely wreck the ability to load existing savegames, right now is the time to make all the changes. Once we have stable releases out with players wanting to keep their savegames and multiple mods using the DLL, improving the savegame format becomes problematic. This mean this is most likely the only time I have the chance to get the savegame format to be just right, which mean I better do it right this time

 

[Kailric]

Splendid work there.

Did you look at the help section of the pedia? There is actually three different keys or so for autoplay: 1 turn, 25 turn, 100 turns, and then a key to play till someone declares independence, or something like that. I need to rebuild the dll and can't start the game at the moment.

 

[Nightinggale]

Did you look at the help section of the pedia? There is actually three different keys or so for autoplay: 1 turn, 25 turn, 100 turns, and then a key to play till someone declares independence, or something like that

So you are saying that when I forget a hotkey for a debug feature, I should look in help? How am I to know that help could actually be helpful?

I figured out what to do about saving enum types. The compiler is from 2003, which mean it applies to the 2003 standard. Instance of enum variables was not added to the C++ standard before 2011 and before that the compiler could do what it wants. MS decided to add keywords to control this is 2005, but back in 2003, it's 4 bytes, like it or not.

This mean the solution is to copy the enum into a short or char and then save that one, This will allow us to control the size. This mean if we save CivicTypes and have 30 civics, it should use char and if we have 150 civics, it should use short as char would overflow when using numbers higher than 127. I knew that right away, but the problem is how to code that while making it easy to use and not prone to bugs. Not using the same number on load as was used on save will break the savegame.

The answer is to build an array of lengths of arrays and this should be done while creating/reading the conversion table strings. This mean the read/write code later for the same savegame will use the same lengths as the number of strings in the savegame and that way ensure the same size is used on both load and save. Presumably most xml files, if not all will be within the byte range (max 127), but supporting 32k types in each file seems like a good idea, particularly because that is the limit elsewhere in the code.

For this to work, we will need a reliable way to convert from EnumTypes to JIT_ARRAY enum values. I figure we should do something like this:

static inline JITarrayTypes getJITtype(CivicTypes eType)
{
    return JIT_ARRAY_CIVIC;
}

Making a bunch of those as overloaded functions mean it will provide the enum value in template functions.

Result: activating the code is done by using Read(), Write() and readWrite(). Calling it with say CivicTypes will then use the savegame size for CivicTypes and convert to the new civic xml on load. If adding a variable to a savegame is just adding a call to readWrite and the size/conversion happens automatically, it should be fairly hard to introduce bugs due to this functionality
The uiFlag system still applies and new variables should be loaded conditionally on this, but other than that, it should be piece of cake.

Once this system is working, I can use the functionality to expand JIT savegame code, or rather shrink the disk space. For instance CvPlayerAI saves 4 UnitAI arrays using ints. The current count cache will like contain between 0 and 100 of each unitAI, which mean the JIT array code should detect this and save using bytes, effectively cutting the size of that array by 75%. Doing this automatically to all JIT arrays may or may not shrink savegame size noteworthy. However we will know that we have done what we can to keep the size down and that we have countered future increase in size as much as possible.

 

[Nightinggale]

I noticed a problem. Unit automation xml is hardcoded in the dll (which seems ok considering how hardcoded the AI is towards each value). However there is no check to see if dll and xml is out of sync. Even worse, the xml has no type, meaning it isn't possible to just code it in the dll. Sure we could just add the type (which will likely be needed), but it might be a good idea to review every single hardcoded xml file to ensure we have a system, which isn't prone to undetected bugs before other mods starts to copy the xml files.

UnitAI types adds nothing to xml other than the type, yet the types are hardcoded in the dll. It might be a good idea to simply remove the xml and generate the code in the dll at runtime. That way they will never go out of sync. It's not a high priority because removing it later isn't really an issue. Worst case would be to leave an unread xml file in a mod, which does nothing.

We might want to review which xml files to save the strings for in the savegames. Adding too little makes the savegames fragile to change while adding too many makes savegames grow greatly in size.

EDIT: I added a "mod name" detector to RaRE. It displays a warning if people rename the mod, hence making it harder for me to load the game to look at bugs. I think CMC should do the same, but due to the shared dll code, it should compare to a string in xml rather than dll hardcoding.

 

[Nightinggale]

I ended up with the "savegame can't be read" issue due to not reading and writing the same amount of bytes. Without a clue to where it went wrong, I added a new function and added it all over the savegame code.

AddSavegameTag(bRead, pStream);

Like asserts, the compiler will remove it unless it is enabled by a flag. In this case the flag is SAVEGAME_DEBUG.
What it does is it saves the 3 chars TAG and then an unsigned int. each time it does that, the int is incremented by one, making each write unique. On load, it does the same, except this time it knows what it should read and it asserts if it reads something else. The result is that if it triggers an assert, you know it went wrong between the current AddSavegameTag and the previous one. With one in the start and end of each class and even more spread around in large readWrite functions, the error can be pinpointed quite easily. Once I was done with this, I just left all the debug code in and disabled SAVEGAME_DEBUG. It's now ready to be used if we need it in the future.

It turned out that the error was caused by typecasting. As I have said before, typecasting is evil. The type tells what kind of data the variable contain and with the more advanced objective code I have added now, we have overloaded functions acting on the different types, meaning typecasting a FeatureTypes into an int will make the game do something incorrectly. Once again I have to say that we should get rid of all/most typecasts as they are prone to cause bugs and/or hide them.

What I have made now is that I have made our code use ReadStringMod and WriteStringMod instead of the vanilla versions and it works for normal char and wide char strings. Both work on the NULL terminated string approach, meaning they take up less space in the savegame. When wide char strings are saved, it checks if the string fits in a normal char string (it does 99% of the times). If so, it will be saved as a normal char string, meaning it use one byte for each character instead of two, cutting the space needed in half.

I'm now done with strings

I finished the code to handle enum types for read/write. If the code writes say a FeatureType, it will use one byte for it, not 4. If there is more than 127 Features in xml, it will use two bytes, allowing room for 32k features. The same goes for all the other xml enum types in the list (around 40). On top of that, some types are now always saved in a single byte instead of 4. One good example is GameType. It's a vanilla enum we can't change and it has like 10 different values.

I added an assert for SAVE_VERSION to be 2. Some modders think they should use 3 because they have added new content to the savegame. However it turns out that the version is actually about the format in the exe and it really shouldn't be accessible to the modders. It's not mentioned in the dll at all, so why add it to xml? It's not like xml modders are more careful not to break the game than the dll modders.

I noticed despite my efforts to shrink the savegame size, the size has actually gone up. It turns out that it is because I added more xml files to the conversion table, which makes the table use something like 40 kB. That made me think. If we give each file a type prefix and then remove the prefix when saving then we should be able to save quite a lot of space. It would also require an assert if people do not follow the naming convention, which in itself would be a good thing. Techs would have the prefix TECH_, not MEDIEVAL_TECH_. A number of the strings would be reduced significantly, like CIVIC_OPTION_PIETY, which would be reduced from 19 to 6 bytes.

I'm also considering if we really need the diplo comments in the savegame. It would seem that some are hardcoded into the DLL, which brings up the question: how are they used? Would it make sense not to have the same types in all mods? If they are hardcoded in the dll and we only add to the end, then there would not be a need to have a conversion table. In fact if xml modders only add to the end (and never remove), then we could skip the table as well.

Looking at the savegame code, it looks like I'm at the end of what I can do towards keeping savegame compatibility. Looks like the exe prevents keeping savegames unless MAX_PLAYERS is kept constant.

The xml files currently being converted:

Spoiler :

JIT_ARRAY_CENSURE,
JIT_ARRAY_CIVIC,
JIT_ARRAY_EVENT_CIVEFFECT,
JIT_ARRAY_PERK,
JIT_ARRAY_TECH,
JIT_ARRAY_CIVEFFECT_DEFAULT,
JIT_ARRAY_BONUS,
JIT_ARRAY_BUILD,
JIT_ARRAY_BUILDING,
JIT_ARRAY_BUILDING_CLASS,
JIT_ARRAY_BUILDING_SPECIAL,
JIT_ARRAY_CIVIC_OPTION,
JIT_ARRAY_CULTURE,
JIT_ARRAY_DIPLO,
JIT_ARRAY_ERA,
JIT_ARRAY_EMPHASIZE,
JIT_ARRAY_EUROPE,
JIT_ARRAY_EVENT,
JIT_ARRAY_EVENT_TRIGGER,
JIT_ARRAY_FATHER,
JIT_ARRAY_FATHER_POINT,
JIT_ARRAY_FEATURE,
JIT_ARRAY_GOODY,
JIT_ARRAY_HANDICAP,
JIT_ARRAY_HURRY,
JIT_ARRAY_IMPROVEMENT,
JIT_ARRAY_LEADER_HEAD,
JIT_ARRAY_PROFESSION,
JIT_ARRAY_PROMOTION,
JIT_ARRAY_ROUTE,
JIT_ARRAY_TECH_CATEGORY,
JIT_ARRAY_TERRAIN,
JIT_ARRAY_TRAIT,
JIT_ARRAY_UNIT,
JIT_ARRAY_UNIT_AI,
JIT_ARRAY_UNIT_CLASS,
JIT_ARRAY_UNIT_COMBAT,
JIT_ARRAY_UNIT_SPECIAL,
JIT_ARRAY_VICTORY,
JIT_ARRAY_YIELD,

Unit automation should be in the list well. That or adding a note in the enum to only add new ones in the end. They are hardcoded in the dll anyway. Regardless of what we do, we should add types to xml. Currently dll and xml order has to match, but there is no way to confirm this without types. With types, at least the xml modder can get an assert if they go out of sync.

Maybe domains should be added at some point, but there is no need right now. We can add more files to the list later and still be backward compatible. They can be added once it makes sense for xml modders to add more domains without modifying the dll as well.