Mod load time

[Koshling]

Anyone else finding that this is reaching unacceptable levels??

It now takes me approximately 15 minutes to start the mod on my laptop. Desktop is much better (SSD there), but at about 3 minutes it's still a massive pain in the backside when testing since I am continually having to make small code changes, and restart to test - I wind up spending half my time just waiting for the mod to load.

I've added this as an issue to the tracker: http://caveman2cosmos.comule.com/view.php?id=28

If anyone has ideas or opinions on how important this is please let me know.

 

[ls612]

Anyone else finding that this is reaching unacceptable levels??

It now takes me approximately 15 minutes to start the mod on my laptop. Desktop is much better (SSD there), but at about 3 minutes it's still a massive pain in the backside when testing since I am continually having to make small code changes, and restart to test - I wind up spending half my time just waiting for the mod to load.

I've added this as an issue to the tracker: http://caveman2cosmos.comule.com/view.php?id=28

If anyone has ideas or opinions on how important this is please let me know.

Yes! It can take ten minutes sometimes for me, but I figured it was just my machine. I also didn't know if the mod load was handled in the DLL, so I didn't know if it was fixable. If there is something to be done about it, by all means, it would make things so much better for us modders, who have to load it repeatedly to test things.

 

[dusckr87]

I'm using a laptop, and it's 3-4 minute for me (granted, my working copy is only SVN2937 so the more recent one may be slower). And it's barely a middle-of-the-road laptop at that (Dual core 2.2GHz, 4GB RAM, Vista 64, Intel GMA4500 graphics).

 

[AIAndy]

Yes! It can take ten minutes sometimes for me, but I figured it was just my machine. I also didn't know if the mod load was handled in the DLL, so I didn't know if it was fixable. If there is something to be done about it, by all means, it would make things so much better for us modders, who have to load it repeatedly to test things.

I guess we should do some profiling on the game start up to see if it is some part of the XML loading (which is for a good part in the DLL) or some preliminary unpacking or looking at graphics or whatever that is taking that long.
Some simple time logging should at least give as the information how long the XML reading part of the DLL takes in total. There is quite some potential optimization that is possible there but I fear a lot of the delay happens before the DLL is even loaded.

 

[ls612]

I guess we should do some profiling on the game start up to see if it is some part of the XML loading (which is for a good part in the DLL) or some preliminary unpacking or looking at graphics or whatever that is taking that long.
Some simple time logging should at least give as the information how long the XML reading part of the DLL takes in total. There is quite some potential optimization that is possible there but I fear a lot of the delay happens before the DLL is even loaded.

Well, I don't know if the load screen is accurate, but the part where it says "Load XML [uncached]" Takes ~75% of the total load time for me.

 

[Koshling]

Well, I don't know if the load screen is accurate, but the part where it says "Load XML [uncached]" Takes ~75% of the total load time for me.

It's reasonably accurate. Its because we have reams and reams of XML, distrbuted amongst a very large number of files. It really brought it oome to me, working on the parallel maps mod recently (total load time less than 10 seconds).

 

[AIAndy]

It's reasonably accurate. Its because we have reams and reams of XML, distrbuted amongst a very large number of files. It really brought it oome to me, working on the parallel maps mod recently (total load time less than 10 seconds).

At least if it is the XML that means we can improve it, a lot, if needed.

 

[Koshling]

At least if it is the XML that means we can improve it, a lot, if needed.

Please add any idea you might have as notes on the tracker issue (I already added a possible suggestion)

 

[AIAndy]

Please add any idea you might have as notes on the tracker issue (I already added a possible suggestion)

I have been stepping through the early loading on a debug DLL a bit. It took a long time before it even loaded the DLL but it seems that was because I had for some reason DisableFileCaching in the Civ ini set to 1.
Then the major time was definitely loading all the XML files. Do you know how to activate the caching? The code seems to already write the caches but the call to the reading function returns false.
Maybe we should just write our own cache files that we can then also deploy.

 

[ls612]

OK, so I got scientific about this, which means that I checked my watch repeatedly during loading, and found that without any caching turned on in the .ini the breakdown of load time is;

• Pre-Load: 230 seconds (yikes)
• Check XML: 3 seconds
• Init Python: 3 seconds
• Init XML (uncached): 170 seconds (yikes)
• Init Engine: 18 seconds
• Total: 424 seconds

The other functions had negligible time consumption.

Now, with both Caching options turned on in the INI, the time breakdown is as follows.

• Pre-Load: 59 seconds (this is a MAJOR improvement)
• Check XML: 1 second
• Init Python: 3 seconds
• Init XML (uncached): 148 seconds (yikes)
• Init Engine: 5 seconds
• Total: 215 seconds

Both of these are very bad load times, so if any improvements can be made to the XML init or the pre-load processes, the effect would be immense.

 

[AIAndy]

Both of these are very bad load times, so if any improvements can be made to the XML init or the pre-load processes, the effect would be immense.

The pre-load stuff is without even the DLL loaded so we can't do anything about that (but the file caching option already deals with most of it).
For loading the main XML files (the big ones) there is a caching scheme in there but it seems it is deactivated somehow.
I think the best would be to cache them ourselves. The serialization code is in there but I am not sure if everyone has properly added every extension to it (I have with the stuff I have added). We could even cache it within the mods folder so deployed versions would get the fast loading right away.

 

[Thunderbrd]

I understand caching about as well as I understand the genetic code. I brought up to Koshling a while back that things seemed to work a bit better when caching was turned off in the core ini file (both faster loads and improved processing in game, particularly on less powerful computers.)

But I can say that from watching the debug dll run a huge part of it is not just the volume of our xml files but the diversity. I think we've got a bit too much modularization going on and we could speed things up a bit by moving as much as possible to cores and leaving as little as we can in modular form. It takes time for the xml loading to load the core then edit what it has module by module.

 

[AIAndy]

I understand caching about as well as I understand the genetic code. I brought up to Koshling a while back that things seemed to work a bit better when caching was turned off in the core ini file (both faster loads and improved processing in game, particularly on less powerful computers.)

But I can say that from watching the debug dll run a huge part of it is not just the volume of our xml files but the diversity. I think we've got a bit too much modularization going on and we could speed things up a bit by moving as much as possible to cores and leaving as little as we can in modular form. It takes time for the xml loading to load the core then edit what it has module by module.

But that is what manual caching would avoid. Instead of reading all that XML files it would just read one binary file that has the final content. Far faster.
Currently it does not seem to use the included caching on the main XML files.

 

[ls612]

I understand caching about as well as I understand the genetic code. I brought up to Koshling a while back that things seemed to work a bit better when caching was turned off in the core ini file (both faster loads and improved processing in game, particularly on less powerful computers.)

But I can say that from watching the debug dll run a huge part of it is not just the volume of our xml files but the diversity. I think we've got a bit too much modularization going on and we could speed things up a bit by moving as much as possible to cores and leaving as little as we can in modular form. It takes time for the xml loading to load the core then edit what it has module by module.

I agree on the management of Modular files, but I think that there is another issue going on.

We now have 7(?) FPKs for C2C, and I have a feeling that the number of FPKs and the amount of unPAKed art has a large influence on load times. This is just my empirical guess, as I don't have a clue how the PAKs work, but I've noticed that right after a freeze, after all of the unPAKed art has been moved to the PAKs, that the load times are better. This is to be expected, as the PAKs were designed to improve load and retrieval times. However, I also noticed a slight drop in load times when the REVDCM PAK was merged into the C2C PAK, so I wonder if having so many PAKs could be part of the problem.

 

[Dancing Hoskuld]

I have all caching off because I get MAFs and graphical glitches too often when they are on. Plus when I am adding stuff it is the only way to be sure that the changes will activate.

Moving the XML into the core area is a proven way of reducing load speed but it reduces flexibility for choosing what you the player want in game and how the modders build stuff.

If we had a C2C options screen at the the time of "Custom Game" like some of the map scripts have then we could get rid of a few more modules.

Others wont because the addition or removal of the folder uses the WoC system to include the effects of the chosen option. Maybe what I am saying here is that the MLF system needs to be looked at also.

We are having other problems because of the large amount of XML. Movies for example. Some get lost and just show a blue dialog with out the movie or OK button.

Question Is it faster to load XML which has all the tags 90% of which are the default values or should we only have the bits in the XML that are not the default values?

Conclusion (no coffee had yet) so this may not follow what has gone before.

1) any XML which is not optional should go into the core files after one whole version release.

2) optional XML should start using the WoC (and other) tags so they don't appear when they should not. An example here would be that culture units would only appear if their culture is also in the game.

3) what is optional and what is optimal. Currently there are no options on Subdued Animals but they are kept in a separate module which contains units, buildings, promotions (iirc) and their art. This is because it makes it easier for me to modify, and add new animals. New animals are added as individual modules them merged into the core and main XML files after a version. Should I be doing this.

This gets back to what is optional. The extra religions are all modules because some people turn some of them off.

 

[AIAndy]

Question Is it faster to load XML which has all the tags 90% of which are the default values or should we only have the bits in the XML that are not the default values?

It should not matter too much. The code starts the reading of each entry by mapping the children so it should be quite cheap to determine if a child with a certain name is in there or not independent of if it is in there or not. As it won't actually need to read what is not in there, it is probably faster to leave out whatever default values you have.

I will implement some manual caching of the bigger XML files as then it should not matter too much if we have a lot of modular content or not.

 

[AIAndy]

As there does not seem to be a way to grab a proper instance of an implementation of FDataStreamBase, I have now implemented it myself based on fstream.
The code that does the caching and checks if the cache for a type of XML file is up to date is also finished.
Part of the serialization/deserialization code in CvInfos is disfunctional though or works badly (mainly the array stuff that can also be a NULL pointer) so that is what I am updating now.

 

[Koshling]

As there does not seem to be a way to grab a proper instance of an implementation of FDataStreamBase, I have now implemented it myself based on fstream.
The code that does the caching and checks if the cache for a type of XML file is up to date is also finished.
Part of the serialization/deserialization code in CvInfos is disfunctional though or works badly (mainly the array stuff that can also be a NULL pointer) so that is what I am updating now.

When you say 'checks if the cache for a type of XML file is up to date' what are your units of up-to-datedness? Don't forget that types massively cross-refer so an addition of (say) a unit can change tech infos etc., due to the unitTypes-space changes.

 

[AIAndy]

When you say 'checks if the cache for a type of XML file is up to date' what are your units of up-to-datedness? Don't forget that types massively cross-refer so an addition of (say) a unit can change tech infos etc., due to the unitTypes-space changes.

I use the normal enumeration to find all XML files of that type and XOR all write timestamps on them into a checksum. Then I compare it to the checksum I write at the beginning of the cache file. If it is the same, then I use the cache, otherwise I read the XML files and write a new cache file at the end.
But you are right, I did not think of the cross reference and array space invalidation from an addition to one of the other files. So I guess I have to invalidate all caches if anything in the XMLs has changed (unless I want to change all that serialization to something like the savegame format).

 

[Paprikan]

so by turning off caching i can reduce MAFs? and if yes where can i find the file to turn them off?