Kailric
Jack of All Trades
I'm reading what you're saying and as soon as I get my bachelors in applied science I'll give you a proper reply
But it all sounds good.
But it all sounds good.
Current and Future Developments
- Thread starter Kailric
- Start date
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
I finally reached the point where I could start working on JIT array save code. A block consist of a header byte, as well as a start and end. If the code thinks it's a good idea, it will save an array as multiple blocks and merge them on load.
Example. Say we have an array where we have written to elements 4, 5, 6 and 18. It will make one block starting at 4, containing 3 elements and another one starting at 18 containing a single element. Everything not mentioned in the savegame will be set to the default value (usually 0, but it can be whatever the C++ coder wants it to be, like NO_CIVIC).
I ran this on some actual game arrays to see if it works as intended and to see how well it performs. Saving a specific array I looked into:
Vanilla: 88 bytes
JIT (old): 78 bytes
JIT (new) 9 bytes
I would call that a success
Part of the reason why the saving is this good is because each block look at the min and max values of the numbers in the block. It then figure out the smallest variable able to hold all the numbers. In this case it was an int array (4 bytes), but all the numbers were so low that they could fit in a single byte. That alone gave a 3 byte or 75% size reduction for each element. The other reason is that the array contains 3 elements different from 0. They are so far apart that it makes it 3 blocks, each with a single element. The old approach was to also save the 0s in between the important elements.
I then started wondering how to test if it always behave as expected. It appeared to work just fine from testing, but 98% sure of working code isn't good enough if the remaining 2% corrupts savegames at random. After thinking long and hard about how to become as close to 100% sure as we can get, I came up with what I would consider a brilliant solution. The new code is written from scratch, leaving the old read and write functions intact in JIT arrays. Now if SAVEGAME_DEBUG is defined, it will save with both systems and on load, it will load the new one into a temporally JIT array and then it will assert check the two arrays to have loaded precisely the same data. If two different functions tries to save the same data and it loads the same data, then we can be pretty sure it went well. I defined SAVEGAME_DEBUG in the mod define header and we will use that for quite a while now. It should be removed prior to making a stable release, but if it can be used through testing without asserting even a single time, we should be able to trust the new code. Also it adds some counters in the savegame as well to assert on to see if it reads and writes the same amount of bytes in each class.
DLL version
I added makefile code (technically Makefile.project as it is unique to our project). It will run git to get the revision ID and forward it to a perl script, which then generates a .h file. Here the revision string is a static const char pointer (that is, a read only string using minimal resources). A CvString copies the content and then it calls readWrite(). This mean the string enters the savegame and whatever is in the savegame will enter the string on load. It isn't used at all, but it can be read using a debugger, which mean dll modders can identify the version of DLL, which would be really helpful if somebody comes with a buggy savegame in the future while saying "I don't know which version it is from".
The header is ignored by git, which mean it will never be committed. Instead it is generated just prior to running fastdep. This naturally adds having git and perl installed. The reason why it isn't added is because it would always be outdated. It will contain the current revision when you reach the point where you commit, but once the file goes into git, a new revision is made, but the header file still contains the old one. Because of this, it wouldn't make any sense to store the revision. We should figure out what to do about released source code and this requirement. Right now it wouldn't compile because such a release wouldn't be in a git repository. It should likely have some release flag to make it optional and the release should have a header file telling it's the release.
XML value conversion on load
I use the JIT conversion to convert even more arrays on game load, meaning savegames will be preserved if even more xml files are changed. To do this, I just added a whole bunch of xml file data to savegames and I suspect I might have overdone it. The savegames are now full of strings for the table and I lost track what is used and what isn't
The savegame appears to be too big and too complex to keep track of such types manually. I'm not considering just adding access to all xml files and then make counter for how many times each one is used. This will allow adding assert checks for unwanted numbers, like if it detects saving the strings for an xml file and the type is used 0 times.
We should have some on/off switch to tell if we want xml files to be included in a savegame. Or rather, we should decide if we want the conversion table for the type. If we happen to save something, which lacks a conversion table, then we can simply write the type string. On load, since the table wasn't in the savegame, it knows it should read a string and then loop the types of the xml file in question to find a matching ID. While I prefer to save ints when saving a number, it would make sense to save them as strings if there are less variables saved than there are types in the xml file. It wouldn't surprise me if some types are only saved once. Also even with say 30 diploEvents in a savgame, it would pay off to save the strings rather than the (currently) 154 strings needed for the conversion table.
Also I looked into the ability to set a prefix to a type and then not save the prefix in the table. This would reduce the disk space needed to store some files by more than 50%. Now I can't think of any reason not to do this, other than it requires some xml type cleanup.
Even though I have done quite a lot to the savegame now, it seems that there are still quite a lot of potential improvements. At least I'm not bored
Example. Say we have an array where we have written to elements 4, 5, 6 and 18. It will make one block starting at 4, containing 3 elements and another one starting at 18 containing a single element. Everything not mentioned in the savegame will be set to the default value (usually 0, but it can be whatever the C++ coder wants it to be, like NO_CIVIC).
I ran this on some actual game arrays to see if it works as intended and to see how well it performs. Saving a specific array I looked into:
Vanilla: 88 bytes
JIT (old): 78 bytes
JIT (new) 9 bytes
I would call that a success
Part of the reason why the saving is this good is because each block look at the min and max values of the numbers in the block. It then figure out the smallest variable able to hold all the numbers. In this case it was an int array (4 bytes), but all the numbers were so low that they could fit in a single byte. That alone gave a 3 byte or 75% size reduction for each element. The other reason is that the array contains 3 elements different from 0. They are so far apart that it makes it 3 blocks, each with a single element. The old approach was to also save the 0s in between the important elements.
I then started wondering how to test if it always behave as expected. It appeared to work just fine from testing, but 98% sure of working code isn't good enough if the remaining 2% corrupts savegames at random. After thinking long and hard about how to become as close to 100% sure as we can get, I came up with what I would consider a brilliant solution. The new code is written from scratch, leaving the old read and write functions intact in JIT arrays. Now if SAVEGAME_DEBUG is defined, it will save with both systems and on load, it will load the new one into a temporally JIT array and then it will assert check the two arrays to have loaded precisely the same data. If two different functions tries to save the same data and it loads the same data, then we can be pretty sure it went well. I defined SAVEGAME_DEBUG in the mod define header and we will use that for quite a while now. It should be removed prior to making a stable release, but if it can be used through testing without asserting even a single time, we should be able to trust the new code. Also it adds some counters in the savegame as well to assert on to see if it reads and writes the same amount of bytes in each class.
DLL version
I added makefile code (technically Makefile.project as it is unique to our project). It will run git to get the revision ID and forward it to a perl script, which then generates a .h file. Here the revision string is a static const char pointer (that is, a read only string using minimal resources). A CvString copies the content and then it calls readWrite(). This mean the string enters the savegame and whatever is in the savegame will enter the string on load. It isn't used at all, but it can be read using a debugger, which mean dll modders can identify the version of DLL, which would be really helpful if somebody comes with a buggy savegame in the future while saying "I don't know which version it is from".
The header is ignored by git, which mean it will never be committed. Instead it is generated just prior to running fastdep. This naturally adds having git and perl installed. The reason why it isn't added is because it would always be outdated. It will contain the current revision when you reach the point where you commit, but once the file goes into git, a new revision is made, but the header file still contains the old one. Because of this, it wouldn't make any sense to store the revision. We should figure out what to do about released source code and this requirement. Right now it wouldn't compile because such a release wouldn't be in a git repository. It should likely have some release flag to make it optional and the release should have a header file telling it's the release.
XML value conversion on load
I use the JIT conversion to convert even more arrays on game load, meaning savegames will be preserved if even more xml files are changed. To do this, I just added a whole bunch of xml file data to savegames and I suspect I might have overdone it. The savegames are now full of strings for the table and I lost track what is used and what isn't
The savegame appears to be too big and too complex to keep track of such types manually. I'm not considering just adding access to all xml files and then make counter for how many times each one is used. This will allow adding assert checks for unwanted numbers, like if it detects saving the strings for an xml file and the type is used 0 times.
We should have some on/off switch to tell if we want xml files to be included in a savegame. Or rather, we should decide if we want the conversion table for the type. If we happen to save something, which lacks a conversion table, then we can simply write the type string. On load, since the table wasn't in the savegame, it knows it should read a string and then loop the types of the xml file in question to find a matching ID. While I prefer to save ints when saving a number, it would make sense to save them as strings if there are less variables saved than there are types in the xml file. It wouldn't surprise me if some types are only saved once. Also even with say 30 diploEvents in a savgame, it would pay off to save the strings rather than the (currently) 154 strings needed for the conversion table.
Also I looked into the ability to set a prefix to a type and then not save the prefix in the table. This would reduce the disk space needed to store some files by more than 50%. Now I can't think of any reason not to do this, other than it requires some xml type cleanup.
Even though I have done quite a lot to the savegame now, it seems that there are still quite a lot of potential improvements. At least I'm not bored
And I was optimistic and expected a proper reply this centuryI'm reading what you're saying and as soon as I get my bachelors in applied science I'll give you a proper reply
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
Thinking on how to exploit the newcome ability to add perl script to the build process, I have realized the following:
I would like to be able to generate enum headers for xml types on each build, but currently the check to see if the header is outdated is a bit too slow. I would need to improve performance before using it. It would appear that string compare and other string related functions slows down the script. Renaming the xml files to match the naming convention would be an improvement as it would seriously reduce the need for string operations. Also checking the last modification date on a file should be done at the start and cached instead of checking the same file once for each xml file to compare against. There are a number of ways to get a faster script, which mean it does look realistic.
- perl scripts should be extremely quick because they are executed when starting every build.
- We need a new build target for xml modders (the dll without hardcoding)
- We need a new build target for final release (or whatever we call it)
- fail to compile if debug code is left in the dll (like the savegame debug overhead)
- Check that all variables are either saved or commented as no save
- Possibly some xml checks
- Possibly md5 checksum generation of xml schema files
- Possibly moving schema files between mod and dll repositories (beats doing it manually)
- More similar tasks
I would like to be able to generate enum headers for xml types on each build, but currently the check to see if the header is outdated is a bit too slow. I would need to improve performance before using it. It would appear that string compare and other string related functions slows down the script. Renaming the xml files to match the naming convention would be an improvement as it would seriously reduce the need for string operations. Also checking the last modification date on a file should be done at the start and cached instead of checking the same file once for each xml file to compare against. There are a number of ways to get a faster script, which mean it does look realistic.
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
I added statistics to conversions on load and save. I got the following:
JIT arrays: 14222
Variables: 19323
Num types (xml files) in conversion table: 44
Strings in conversion table: 1662
It looked at each enum type (read: each xml file) and revealed that:
9 files are completely unused
1 file is used for a single variable
If that variable is stored as a string rather than int, then removing those from the savegame will reduce the amount of strings by 390, or 23.5%. The variable being saved as a string still contains the string information and can be converted if needed.
Note: the numbers might be inaccurate because I forgot to include BoolArray. I need to redo the test
Next goal
Add the ability to simply skip adding the strings from an xml file. If the strings for a file isn't implemented, saving the enum type for that file should save the string rather than an int value.
The statistics should be used to assert if the setup about which files to make a conversion table for is incorrect. Being checks, which stays in the code, it will detect if some future change alters the requirements for what to add to the savegame.
Savegames should be able to be read even if which tables it includes is outdated. In fact it should make a table for civs if MAX_PLAYERS > GC.getNumCivilizationInfos(). This mean some tables may or may not be included depending on xml alone even without recompiling the dll. Naturally this is only useful if the savegame can still be read even if the test condition switched to the opposite setting.
XML data reading
All this turns out to be useful for reading from xml as well. If we say read the default profession for a unit, we should do it with a function, which take two arguments, (T&, tag). The tag string being "DefaultProfession" and T being the variable. It will then go "oh, it's a ProfessionTypes", read the string from the tag and then compare it to everything in the professionInfos. This allows tag reading and conversion to enum type to be done in a single line rather than two and it will not accept using say a unit type because it is told to look at professions.
DLL hardcoding
I figured out how to handle the xml hardcoding in the dll and turn it on and off. We simply add code like this:
Now whenever we use NUM_YIELD_TYPES, the value will be known at compile time for hardcoded dlls and optimized accordingly. If DONT_HARDCODE_XML_TYPES (or whatever we call it) is set by the makefile, then the precompiler will replace all NUM_YIELD_TYPES with runtime checks of the same value. Simple to code, simple to read and it will get the job done. The enum header generator can add code like that, meaning adding this for all xml files should be trivial.
JIT arrays: 14222
Variables: 19323
Num types (xml files) in conversion table: 44
Strings in conversion table: 1662
It looked at each enum type (read: each xml file) and revealed that:
9 files are completely unused
1 file is used for a single variable
If that variable is stored as a string rather than int, then removing those from the savegame will reduce the amount of strings by 390, or 23.5%. The variable being saved as a string still contains the string information and can be converted if needed.
Note: the numbers might be inaccurate because I forgot to include BoolArray. I need to redo the test
Next goal
Add the ability to simply skip adding the strings from an xml file. If the strings for a file isn't implemented, saving the enum type for that file should save the string rather than an int value.
The statistics should be used to assert if the setup about which files to make a conversion table for is incorrect. Being checks, which stays in the code, it will detect if some future change alters the requirements for what to add to the savegame.
Savegames should be able to be read even if which tables it includes is outdated. In fact it should make a table for civs if MAX_PLAYERS > GC.getNumCivilizationInfos(). This mean some tables may or may not be included depending on xml alone even without recompiling the dll. Naturally this is only useful if the savegame can still be read even if the test condition switched to the opposite setting.
XML data reading
All this turns out to be useful for reading from xml as well. If we say read the default profession for a unit, we should do it with a function, which take two arguments, (T&, tag). The tag string being "DefaultProfession" and T being the variable. It will then go "oh, it's a ProfessionTypes", read the string from the tag and then compare it to everything in the professionInfos. This allows tag reading and conversion to enum type to be done in a single line rather than two and it will not accept using say a unit type because it is told to look at professions.
DLL hardcoding
I figured out how to handle the xml hardcoding in the dll and turn it on and off. We simply add code like this:
Code:
#ifdef DONT_HARDCODE_XML_TYPES
#define NUM_YIELD_TYPES (GC.getNumYieldTypes())
#endifKailric
Jack of All Trades
Ok, let me get this straight, XML values where not saved before right, but you have added code to save some of the values to help preserve save games? And you are also waving your magic wand around to make builds more bug free with perl scrips
Anyway, when you get the branches merged I'll start testing all the changes and adding the XML text as I go.
Edit: Will I ever catch up.. as soon as I reply you've done posted again
Anyway, when you get the branches merged I'll start testing all the changes and adding the XML text as I go.
Edit: Will I ever catch up.. as soon as I reply you've done posted again
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
Yeah something like that.
Vanilla assumes xml to be the same when loading a game as when it was saved. This mean if you have UNIT_A, which is 18, then it will save 18 and on load it will become whatever is unit 18 and it just hopes that it is still UNIT_A. Also if it saves an array of units, it saves one int for every unit in xml. If you add a unit, it will read one int more than the savegame contains and the savegame fails to load as the number of read bytes and the bytes in the savegame mismatch.
What I did is to start by saving all the unit strings. On load, it makes a vector, where it goes through the saved strings and for each, it stores the index of the current location of that type. This mean whenever UnitTypes is read, it will read the int stored in the savegame and then use that int as index to read the actual value from the vector. This is done for all 44 types currently implemented.
JIT arrays use the conversion as well, except in this case it reads what is saved as index 18 and writes it to index 12 as this is what the conversion vector says. If the UnitArray stores say ProfessionTypes, then it will use both approaches for conversion as in change the content of the profession and then store it at the converted index.
JIT arrays are saved with a bit of overhead to tell how many bytes as saved. This mean (unlike vanilla) that adding a new unit or whatever will not force it to read more bytes. Instead it still reads precisely what was written and the new (unsaved) data will be set to the default value (0 unless specified otherwise).
This makes it much harder for xml modders to break savegames, but I wouldn't say impossible. What happens if you remove a unit and the unit is in use in the game. The city defending unit suddenly becomes of UnitTypes NO_UNIT and that is presumably bad, particularly when it will use the index to get UnitInfo, which will likely cause the game to crash. Adding new types and reordering existing ones should be fairly safe though, that is once I'm completely done. I need to do a systematic sweep of all readWrite functions as I keep detecting more breaking vanilla code.
One other bonus from this is the conversion array tells how many of each type was present when the game was saved. If it contain 15 terrain strings, TerrainTypes will contain a number between -1 and 14. Vanilla saved that as int (4 bytes), but the new code detects this as being able to fit in a single byte (-128 to 127). If somebody adds 130 terrains to xml, it will no longer fit in a byte and it will use two bytes to save the data. This mean we use 50%-75% less disk space to save the same amount of enum types. As the game grows in size (more units and stuff like that), it mean the savegame will increase in size much slower. Once the game is big enough (future bigger maps and whatever), I hope this space saving method will save more space than the strings use and the resulting savegame file will actually be smaller due to having added several kB of strings
The idea to save the strings comes from C2C, but I recoded everything from scratch and by using overloaded Read() and Write() you will not have to think about this while coding. Just call readWrite() and the conversion will be triggered and controlled by the enum type used in the argument. The same goes for JIT arrays as they already have the type stored internally. After all they need to be linked to an xml file to figure out how long they are. This makes our code much easier to use than C2C's and it looks like our code is faster and use less disk space as well.
Not yet. So far all the perl does is to add the DLL version string. However the modified makefile and the now added build requirement of having perl installed does allow us to add more perl scripts. Adding a target, which accepts slow scripts truly unlocks lots of interesting features and automated quality checks here. Ideas are welcome and the more wicked they are, the betterAnd you are also waving your magic wand around to make builds more bug free with perl scrips
If a perl script use the function die, it should end with an error, which the makefile detects like an error from the compiler and it will go "failed to build". Such errors are usually made together with a print of the error and should be quite useful for telling the modder what went wrong.
My idea is to look through the header of a class and store all words starting with m_. It then goes through the readWrite function for that class and checks each line for the stored words. In the end, it has a list of words starting with m_, which were not mentioned in the savegame code and it will cause an error. I made something like that at some point, which is why there are a bunch of "// m_(something) not saved" lines in the savegame code. It's simply to tell the script that it's a runtime cache and intentionally not saved. The purpose isn't to save everything, but to make sure the modder considers every single variable to figure out if it should be saved or not.
Maybe I should just merge right away. I'm not done with the savegame code, but there is no point in preventing you from modding. I will break savegames with more or less every commit for a while though. Maybe it would make sense to merge all branches and then make a new savegame dll branch
Actually I think I will do that. "Real programmers" often complain that people use branches too little. Since the work is in CivEffects, it's like CivEffects turned into develop, which wasn't the plan.
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
I have no words for what just happened. I spent hours merging the branches (we should have merged earlier) and now I can't commit or push because my local checkout is corrupted 
I guess there is nothing other to do than to start over
I guess there is nothing other to do than to start over
Kailric
Jack of All Trades
Ugh, sorry to hear thatWell, brite side, you don't have to start over with civeffects
Well, brite side, you don't have to start over with civeffects
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
Windows failed to book at I started disk repair. It fixed some HD issues and it's now working fine, but I guess that explains the corrupted git repository. I made a new clone from SF and merged the branches there. With the experience I gained from the first attempt, I did it with a cleaner git log this time, which mean it's not a total loss. I decided to split the CivEffect branch into CivEffect and Savegame where the latter is what is called CivEffect on SF and CivEffect is the last revision before I added Savegame.cpp. Never again should we move functions to another file while we modify those functions in multiple files. Git merge is awesome for merging changes, but it doesn't work well with moved code.
I see traderoutes now save how many units use them. While it seems like a good idea to cache this number, I wonder if it can be recalculated on load. I'n doing a lot of work to reduce savegame size and this one adds 4 bytes for each traderoute. Not really a major issue, but...
It seems that I lost two revisions I committed locally and forgot to push. I should be able to recreate them from the log through.
I see traderoutes now save how many units use them. While it seems like a good idea to cache this number, I wonder if it can be recalculated on load. I'n doing a lot of work to reduce savegame size and this one adds 4 bytes for each traderoute. Not really a major issue, but...
It seems that I lost two revisions I committed locally and forgot to push. I should be able to recreate them from the log through.
I have no idea how or why, but it is all your fault
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
I managed to recreate those two lost revisions by looking at the log in the corrupted repository and requesting the line changes (aka diff). Recovery is now complete.
I added prefix to types from each file. The prefix is removed when saving a type string and restored on load. This reduce the savegame by 11 kB, leaving 26 kB of strings for the conversion table. Naturally this adds the demand that the prefix is always used and I added an assert for this. It will print something like this:
I hope xml modders can figure out how to read that one. It will test all types on startup. I skipped 6 files because the xml files doesn't comply with the prefix and I don't feel like either fixing it right now or click ignore on the assert. Naturally this should be solved before next release.
I notice to my horror that I never fully converted traits to CivEffects. I better do that once I'm done with the savegame stuff. It's half done, but not completely. Also there are quite a number of "interesting" traits, like it contains an outdated copy of all techs.
I added prefix to types from each file. The prefix is removed when saving a type string and restored on load. This reduce the savegame by 11 kB, leaving 26 kB of strings for the conversion table. Naturally this adds the demand that the prefix is always used and I added an assert for this. It will print something like this:
Code:
Unit: SUPERUNIT_PEDDLER has to have prefix UNIT_I notice to my horror that I never fully converted traits to CivEffects. I better do that once I'm done with the savegame stuff. It's half done, but not completely. Also there are quite a number of "interesting" traits, like it contains an outdated copy of all techs.
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
I'm done with the savegame "engine" ![[party]](/images/smilies/partytime.gif "Party [party]")
It will no longer save all arrays in the conversion table. Instead it will ask a function if it should (to keep the decision in one place). If the table isn't included, whenever a variable of that type is saved, it will save the string rather than a number. Right now it isn't saving the strings for CivilizationInfos. Instead it will save the string for each player, which has a type other than NO_CIVILIZATION as this will always save less or (in rare cases) the same amount of strings as if we add the table.
It's easy to figure which xml files to make a conversion table for. If you can save or load without asserting, then it is set up correctly. If not, it will tell which file the DLL will recommend to be set differently. This check is always on and will inform us if future changes to the savegame will result in recommendations for a different setup.
The conversion table is now down to 17 kB and will be significantly smaller once all files have prefixes and traits have been cleaned up.
I'm seriously considering taking the list of XML files and make a JIT array type for each. We will not really lose anything from it considering we can just skip them from being saved and if it's complete now once and for all, then we can just rely on it more easily in the future. It's used for lots of other tasks in addition to JIT arrays. For instance they are used for setting types in InfoArrays and converting strings when reading xml files (like converting PROMOTION_VETERAN into a PromotionTypes int).
I have like 5 arrays I need to do something about to reach the level of savegame stability vs xml changes. Sadly I left the hardest for last, but at least the goal is within reach.
It will no longer save all arrays in the conversion table. Instead it will ask a function if it should (to keep the decision in one place). If the table isn't included, whenever a variable of that type is saved, it will save the string rather than a number. Right now it isn't saving the strings for CivilizationInfos. Instead it will save the string for each player, which has a type other than NO_CIVILIZATION as this will always save less or (in rare cases) the same amount of strings as if we add the table.
It's easy to figure which xml files to make a conversion table for. If you can save or load without asserting, then it is set up correctly. If not, it will tell which file the DLL will recommend to be set differently. This check is always on and will inform us if future changes to the savegame will result in recommendations for a different setup.
The conversion table is now down to 17 kB and will be significantly smaller once all files have prefixes and traits have been cleaned up.
I'm seriously considering taking the list of XML files and make a JIT array type for each. We will not really lose anything from it considering we can just skip them from being saved and if it's complete now once and for all, then we can just rely on it more easily in the future. It's used for lots of other tasks in addition to JIT arrays. For instance they are used for setting types in InfoArrays and converting strings when reading xml files (like converting PROMOTION_VETERAN into a PromotionTypes int).
I have like 5 arrays I need to do something about to reach the level of savegame stability vs xml changes. Sadly I left the hardest for last, but at least the goal is within reach.
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
Finally done converting all arrays in the savegame to use the conversion code. There is no longer any known issues regarding adding xml data or changing order, providing the change is in xml only. There are a few hardcoded xml files (like UnitAI), which are more picky and MAX_PLAYERS is hardcoded by the exe. I just hope DLL modders are more careful than xml modders
I'm not completely done yet though. I noticed that map coordinates are saved using 8 bytes (two ints). This needs investigation, but I suspect I can save them using 2-4 bytes instead. This matters more than one might think because of replay data. It saves events such as "player x took over plot x,y at turn z). With that in mind, the amount of coordinates saved seems to grow into something, I would almost call obscene. After that I think I'm done breaking savegame compatibility.
I started looking into converting Founding Fathers into CivEffects and my biggest concern is the amount of xml work, which would be required. I searched around for a good xml editor, but it turns out that the schema is some Microsoft specific version. I may end up converting the schema to a widespread standard to allow xml editors to work correctly, as in keeping both. I don't plan on rewriting the xml reading code to use a new schema file format (I think I could if I really wanted to though).
It looks like MSVC has some xml+xml schema features, though I'm not quite sure if they are useful for us.
I'm not completely done yet though. I noticed that map coordinates are saved using 8 bytes (two ints). This needs investigation, but I suspect I can save them using 2-4 bytes instead. This matters more than one might think because of replay data. It saves events such as "player x took over plot x,y at turn z). With that in mind, the amount of coordinates saved seems to grow into something, I would almost call obscene. After that I think I'm done breaking savegame compatibility.
I started looking into converting Founding Fathers into CivEffects and my biggest concern is the amount of xml work, which would be required. I searched around for a good xml editor, but it turns out that the schema is some Microsoft specific version. I may end up converting the schema to a widespread standard to allow xml editors to work correctly, as in keeping both. I don't plan on rewriting the xml reading code to use a new schema file format (I think I could if I really wanted to though).
It looks like MSVC has some xml+xml schema features, though I'm not quite sure if they are useful for us.
Kailric
Jack of All Trades
Well I would suggest holding off on the FFs and any more additions to Civeffects at the moment as we probably going to get burnt out on that xml work. I know after finishing up what we already have to do I am going to be pretty toast.
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
I have run into an unexpected problem. Turning all victory conditions on means somebody wins diplomatic victory at turn 2. However that seems fixable. Just debug check victory to see how it locates the winner. The real problem is once there is a winner, the replay runs. When it's done it saves the replay and then it loads it. Regardless of what I do, it always read garbage, even if I save and load using vanilla code 
I have debugged, changed, debugged changed and so on and nothing makes sense. I even made a version, which reads byte by byte and then I can see what it reads. It always starts by reading the number 4, even if I save a different number Difficulty level is always 0, yet the following strings seems to be correct... most of the time. After that it reads gigantic numbers and it will get stuck in a loop because it reads an int and runs a read loop the number of times set by the int, which is many millions. I have reached the point where I seriously consider starting to debug vanilla just to see what a clean vanilla does and if it even works as intended.
Naturally this setback has already taken some time and... well I ended up with 0 commits since last post
I have debugged, changed, debugged changed and so on and nothing makes sense. I even made a version, which reads byte by byte and then I can see what it reads. It always starts by reading the number 4, even if I save a different number
Difficulty level is always 0, yet the following strings seems to be correct... most of the time. After that it reads gigantic numbers and it will get stuck in a loop because it reads an int and runs a read loop the number of times set by the int, which is many millions. I have reached the point where I seriously consider starting to debug vanilla just to see what a clean vanilla does and if it even works as intended.Naturally this setback has already taken some time and... well I ended up with 0 commits since last post
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
I needed a break with some time to think. I have decided to screw replay for now and then get back to it later. Right now it stalled me and at the same time I'm not fixing the problem, which mean I'm in a deadlock. Just admitting it to be broken for now will be better. Actually any progress will be better.
It would appear that we have a lot of xml work to do and I really hate editing xml files. It's not that I can't, it's more that there is absolutely no fun in it and the challenge is annoying rather than interesting. Thinking there has to be a better way, I went brainstorming and researching. I found this screenshot:
It's written in wxPerl, which mean it is a GUI written in a perl script. It looks nice and browsing through the demo software I found all sorts of great features. I'm thinking if we make the left list show xml files, right of that a new list showing the types in the selected file and to the right the properties of the selected type. The "welcome text" is a help text, which mean it should be able to print a text for whatever tag you just pressed on, which could then be the documentation for the tag. It could fill out based on the xml schemas, be generic and that way work on any mod, colo or bts. Even better, it would be cross platform and work on windows, mac and linux.
It all sounds great so.... so why can't I figure out how to do anything but open a window and set size and name for that window
So progress since last post is 0 commits and some lines in a perl test script, which doesn't work. Looks like it will turn into a horrible week regarding progress
It would appear that we have a lot of xml work to do and I really hate editing xml files. It's not that I can't, it's more that there is absolutely no fun in it and the challenge is annoying rather than interesting. Thinking there has to be a better way, I went brainstorming and researching. I found this screenshot:
Spoiler :
It's written in wxPerl, which mean it is a GUI written in a perl script. It looks nice and browsing through the demo software I found all sorts of great features. I'm thinking if we make the left list show xml files, right of that a new list showing the types in the selected file and to the right the properties of the selected type. The "welcome text" is a help text, which mean it should be able to print a text for whatever tag you just pressed on, which could then be the documentation for the tag. It could fill out based on the xml schemas, be generic and that way work on any mod, colo or bts. Even better, it would be cross platform and work on windows, mac and linux.
It all sounds great so.... so why can't I figure out how to do anything but open a window and set size and name for that window
So progress since last post is 0 commits and some lines in a perl test script, which doesn't work. Looks like it will turn into a horrible week regarding progress
Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
IT'S WORKING! IT'S WORKING! IT'S WORKING! IT'S WORKING! IT'S WORKING! 
After only 729 attempts, I actually made a perl GUI, which actually does what I wanted it to do
The script is fairly simple. You fill out a type in the From field and what you want it to be named in the To field and hit rename. It will then go through all xml files and rename every reference to that type. Not just the type itself, but also all the other classes referring to it, like renaming a unitClass will also change the name in all the units using that class. wxPerl aims to feel as native as possible, which mean the usual features work, such as copy paste and tab to reach the next field.
If you think the behaviour sounds familiar, then it is because I copy pasted the content of rename.pl into click event handling. I decided to try to do something fairly simple as the first GUI and this task is simple and at the same time still useful.
I haven't actually installed wxPerl in windows yet (haven't even tried), which is why it's a mac screenshot. There is absolutely no reason why it shouldn't work on windows, or linux for that matter. I placed the new script in GUI_scripts because unlike the other scripts, it's not enough to have perl installed. This one needs wxPerl as well.
It turns out that wxPerl is somewhat simple to use once you have figured it out. The problem is.... the documentation is nearly non-existing and if you search the net, you either encounter people getting stuck or people saying "ignore the tools and just write the code yourself. It's so easy" without telling how to actually write the code.
Now it's back to pondering on how to make a general xml editor. That is, unless any of you have a request for something, which could be very useful to have here and now
After only 729 attempts, I actually made a perl GUI, which actually does what I wanted it to do
The script is fairly simple. You fill out a type in the From field and what you want it to be named in the To field and hit rename. It will then go through all xml files and rename every reference to that type. Not just the type itself, but also all the other classes referring to it, like renaming a unitClass will also change the name in all the units using that class. wxPerl aims to feel as native as possible, which mean the usual features work, such as copy paste and tab to reach the next field.
If you think the behaviour sounds familiar, then it is because I copy pasted the content of rename.pl into click event handling. I decided to try to do something fairly simple as the first GUI and this task is simple and at the same time still useful.
I haven't actually installed wxPerl in windows yet (haven't even tried), which is why it's a mac screenshot. There is absolutely no reason why it shouldn't work on windows, or linux for that matter. I placed the new script in GUI_scripts because unlike the other scripts, it's not enough to have perl installed. This one needs wxPerl as well.
It turns out that wxPerl is somewhat simple to use once you have figured it out. The problem is.... the documentation is nearly non-existing and if you search the net, you either encounter people getting stuck or people saying "ignore the tools and just write the code yourself. It's so easy" without telling how to actually write the code.
Now it's back to pondering on how to make a general xml editor. That is, unless any of you have a request for something, which could be very useful to have here and now
Attachments
Well done!hmmm....
This sounds like it could be very useful... My brain is just having a hard time of soldifying thoughts on the subject..
I will throw out a few thoughts, see if any of them fit into a realm of possibility.
So one problem I have when editing my xml (this one makes me so tired sometimes!)
Is adding in a tag to an existing entry. Trying to make sure I put it in the right place so that the order does not break. (Especially when using entries that do not need to have every tag.) So maybe something that lets you choose an entry like UNIT_INFANTRY, then a list of all unitinfo tags, then you select the one(s) you want and it builds or adds it in the right place(s). Something like that.
Perhaps another one for tech tree type situations is something that 'auto fills' the placement order. I don't know exactly what I am thinking, but for example I am thinking of building a tech tree from scratch, and having some way to put them in order, or just an easier way to organise them in the right order, click a button and it then builds that order.. Don't know my brain has gone a bit squishy right now..
Perhaps a similar thing for Unit/Building Upgrades. The ability to say input a set of units and have it auto generate the Upgrade path between them. Things like that..
Just some random thoughts...
