Anyone else finding that this is reaching unacceptable levels??

It now takes me approximately 15 minutes to start the mod on my laptop. Desktop is much better (SSD there), but at about 3 minutes it's still a massive pain in the backside when testing since I am continually having to make small code changes, and restart to test - I wind up spending half my time just waiting for the mod to load.

I've added this as an issue to the tracker: http://caveman2cosmos.comule.com/view.php?id=28

If anyone has ideas or opinions on how important this is please let me know.

Anyone else finding that this is reaching unacceptable levels??

It now takes me approximately 15 minutes to start the mod on my laptop. Desktop is much better (SSD there), but at about 3 minutes it's still a massive pain in the backside when testing since I am continually having to make small code changes, and restart to test - I wind up spending half my time just waiting for the mod to load.

I've added this as an issue to the tracker: http://caveman2cosmos.comule.com/view.php?id=28

If anyone has ideas or opinions on how important this is please let me know.

Yes! It can take ten minutes sometimes for me, but I figured it was just my machine. I also didn't know if the mod load was handled in the DLL, so I didn't know if it was fixable. If there is something to be done about it, by all means, it would make things so much better for us modders, who have to load it repeatedly to test things.

I'm using a laptop, and it's 3-4 minute for me (granted, my working copy is only SVN2937 so the more recent one may be slower). And it's barely a middle-of-the-road laptop at that (Dual core 2.2GHz, 4GB RAM, Vista 64, Intel GMA4500 graphics).

Yes! It can take ten minutes sometimes for me, but I figured it was just my machine. I also didn't know if the mod load was handled in the DLL, so I didn't know if it was fixable. If there is something to be done about it, by all means, it would make things so much better for us modders, who have to load it repeatedly to test things.
I guess we should do some profiling on the game start up to see if it is some part of the XML loading (which is for a good part in the DLL) or some preliminary unpacking or looking at graphics or whatever that is taking that long.
Some simple time logging should at least give as the information how long the XML reading part of the DLL takes in total. There is quite some potential optimization that is possible there but I fear a lot of the delay happens before the DLL is even loaded.

I guess we should do some profiling on the game start up to see if it is some part of the XML loading (which is for a good part in the DLL) or some preliminary unpacking or looking at graphics or whatever that is taking that long.
Some simple time logging should at least give as the information how long the XML reading part of the DLL takes in total. There is quite some potential optimization that is possible there but I fear a lot of the delay happens before the DLL is even loaded.

Well, I don't know if the load screen is accurate, but the part where it says "Load XML [uncached]" Takes ~75% of the total load time for me.

Well, I don't know if the load screen is accurate, but the part where it says "Load XML [uncached]" Takes ~75% of the total load time for me.

It's reasonably accurate. Its because we have reams and reams of XML, distrbuted amongst a very large number of files. It really brought it oome to me, working on the parallel maps mod recently (total load time less than 10 seconds).

It's reasonably accurate. Its because we have reams and reams of XML, distrbuted amongst a very large number of files. It really brought it oome to me, working on the parallel maps mod recently (total load time less than 10 seconds).
At least if it is the XML that means we can improve it, a lot, if needed.

At least if it is the XML that means we can improve it, a lot, if needed.

Please add any idea you might have as notes on the tracker issue (I already added a possible suggestion)

Please add any idea you might have as notes on the tracker issue (I already added a possible suggestion)
I have been stepping through the early loading on a debug DLL a bit. It took a long time before it even loaded the DLL but it seems that was because I had for some reason DisableFileCaching in the Civ ini set to 1.
Then the major time was definitely loading all the XML files. Do you know how to activate the caching? The code seems to already write the caches but the call to the reading function returns false.
Maybe we should just write our own cache files that we can then also deploy.

OK, so I got scientific about this, which means that I checked my watch repeatedly during loading, and found that without any caching turned on in the .ini the breakdown of load time is;


Pre-Load: 230 seconds (yikes)
Check XML: 3 seconds
Init Python: 3 seconds
Init XML (uncached): 170 seconds (yikes)
Init Engine: 18 seconds
Total: 424 seconds


The other functions had negligible time consumption.

Now, with both Caching options turned on in the INI, the time breakdown is as follows.


Pre-Load: 59 seconds (this is a MAJOR improvement)
Check XML: 1 second
Init Python: 3 seconds
Init XML (uncached): 148 seconds (yikes)
Init Engine: 5 seconds
Total: 215 seconds


Both of these are very bad load times, so if any improvements can be made to the XML init or the pre-load processes, the effect would be immense.

Both of these are very bad load times, so if any improvements can be made to the XML init or the pre-load processes, the effect would be immense.
The pre-load stuff is without even the DLL loaded so we can't do anything about that (but the file caching option already deals with most of it).
For loading the main XML files (the big ones) there is a caching scheme in there but it seems it is deactivated somehow.
I think the best would be to cache them ourselves. The serialization code is in there but I am not sure if everyone has properly added every extension to it (I have with the stuff I have added). We could even cache it within the mods folder so deployed versions would get the fast loading right away.

I understand caching about as well as I understand the genetic code. I brought up to Koshling a while back that things seemed to work a bit better when caching was turned off in the core ini file (both faster loads and improved processing in game, particularly on less powerful computers.)

But I can say that from watching the debug dll run a huge part of it is not just the volume of our xml files but the diversity. I think we've got a bit too much modularization going on and we could speed things up a bit by moving as much as possible to cores and leaving as little as we can in modular form. It takes time for the xml loading to load the core then edit what it has module by module.

I understand caching about as well as I understand the genetic code. I brought up to Koshling a while back that things seemed to work a bit better when caching was turned off in the core ini file (both faster loads and improved processing in game, particularly on less powerful computers.)

But I can say that from watching the debug dll run a huge part of it is not just the volume of our xml files but the diversity. I think we've got a bit too much modularization going on and we could speed things up a bit by moving as much as possible to cores and leaving as little as we can in modular form. It takes time for the xml loading to load the core then edit what it has module by module.
But that is what manual caching would avoid. Instead of reading all that XML files it would just read one binary file that has the final content. Far faster.
Currently it does not seem to use the included caching on the main XML files.

I understand caching about as well as I understand the genetic code. I brought up to Koshling a while back that things seemed to work a bit better when caching was turned off in the core ini file (both faster loads and improved processing in game, particularly on less powerful computers.)

But I can say that from watching the debug dll run a huge part of it is not just the volume of our xml files but the diversity. I think we've got a bit too much modularization going on and we could speed things up a bit by moving as much as possible to cores and leaving as little as we can in modular form. It takes time for the xml loading to load the core then edit what it has module by module.

I agree on the management of Modular files, but I think that there is another issue going on.

We now have 7(?) FPKs for C2C, and I have a feeling that the number of FPKs and the amount of unPAKed art has a large influence on load times. This is just my empirical guess, as I don't have a clue how the PAKs work, but I've noticed that right after a freeze, after all of the unPAKed art has been moved to the PAKs, that the load times are better. This is to be expected, as the PAKs were designed to improve load and retrieval times. However, I also noticed a slight drop in load times when the REVDCM PAK was merged into the C2C PAK, so I wonder if having so many PAKs could be part of the problem.

I have all caching off because I get MAFs and graphical glitches too often when they are on. Plus when I am adding stuff it is the only way to be sure that the changes will activate. ;)

Moving the XML into the core area is a proven way of reducing load speed but it reduces flexibility for choosing what you the player want in game and how the modders build stuff.

If we had a C2C options screen at the the time of "Custom Game" like some of the map scripts have then we could get rid of a few more modules.

Others wont because the addition or removal of the folder uses the WoC system to include the effects of the chosen option. Maybe what I am saying here is that the MLF system needs to be looked at also.

We are having other problems because of the large amount of XML. Movies for example. Some get lost and just show a blue dialog with out the movie or OK button.

Question Is it faster to load XML which has all the tags 90% of which are the default values or should we only have the bits in the XML that are not the default values?

Conclusion (no coffee had yet) so this may not follow what has gone before.;)

1) any XML which is not optional should go into the core files after one whole version release.

2) optional XML should start using the WoC (and other) tags so they don't appear when they should not. An example here would be that culture units would only appear if their culture is also in the game.

3) what is optional and what is optimal. Currently there are no options on Subdued Animals but they are kept in a separate module which contains units, buildings, promotions (iirc) and their art. This is because it makes it easier for me to modify, and add new animals. New animals are added as individual modules them merged into the core and main XML files after a version. Should I be doing this.

This gets back to what is optional. The extra religions are all modules because some people turn some of them off.

Question Is it faster to load XML which has all the tags 90% of which are the default values or should we only have the bits in the XML that are not the default values?
It should not matter too much. The code starts the reading of each entry by mapping the children so it should be quite cheap to determine if a child with a certain name is in there or not independent of if it is in there or not. As it won't actually need to read what is not in there, it is probably faster to leave out whatever default values you have.

I will implement some manual caching of the bigger XML files as then it should not matter too much if we have a lot of modular content or not.

As there does not seem to be a way to grab a proper instance of an implementation of FDataStreamBase, I have now implemented it myself based on fstream.
The code that does the caching and checks if the cache for a type of XML file is up to date is also finished.
Part of the serialization/deserialization code in CvInfos is disfunctional though or works badly (mainly the array stuff that can also be a NULL pointer) so that is what I am updating now.

As there does not seem to be a way to grab a proper instance of an implementation of FDataStreamBase, I have now implemented it myself based on fstream.
The code that does the caching and checks if the cache for a type of XML file is up to date is also finished.
Part of the serialization/deserialization code in CvInfos is disfunctional though or works badly (mainly the array stuff that can also be a NULL pointer) so that is what I am updating now.

When you say 'checks if the cache for a type of XML file is up to date' what are your units of up-to-datedness? Don't forget that types massively cross-refer so an addition of (say) a unit can change tech infos etc., due to the unitTypes-space changes.

When you say 'checks if the cache for a type of XML file is up to date' what are your units of up-to-datedness? Don't forget that types massively cross-refer so an addition of (say) a unit can change tech infos etc., due to the unitTypes-space changes.
I use the normal enumeration to find all XML files of that type and XOR all write timestamps on them into a checksum. Then I compare it to the checksum I write at the beginning of the cache file. If it is the same, then I use the cache, otherwise I read the XML files and write a new cache file at the end.
But you are right, I did not think of the cross reference and array space invalidation from an addition to one of the other files. So I guess I have to invalidate all caches if anything in the XMLs has changed (unless I want to change all that serialization to something like the savegame format).

so by turning off caching i can reduce MAFs? and if yes where can i find the file to turn them off?

Its the _Civ4Config file. In your bts file, you'll see a shortcut to it... double click on that and it'll open it in notepad. Scroll down to where it says:


; Disable caching of xml and file system (may slow initialization)
DisableCaching = 0

and change the 0 to a 1. It will improve performance but slow loading a bit. AIAndy is apparently changing some things in regards to this, most of which I'm not understanding fully. My only problem with caching MORE, especially if the mod makes this compulsory, is that from my limited understanding, isn't that going to make it harder for computers with less ram to play at all? I understand I could be completely wrong, but doesn't caching generally mean that we're loading more fixed assumptions into the ram that needs to be held there, limiting ram for other application?

Its the _Civ4Config file. In your bts file, you'll see a shortcut to it... double click on that and it'll open it in notepad. Scroll down to where it says:


; Disable caching of xml and file system (may slow initialization)
DisableCaching = 0

and change the 0 to a 1. It will improve performance but slow loading a bit. AIAndy is apparently changing some things in regards to this, most of which I'm not understanding fully. My only problem with caching MORE, especially if the mod makes this compulsory, is that from my limited understanding, isn't that going to make it harder for computers with less ram to play at all? I understand I could be completely wrong, but doesn't caching generally mean that we're loading more fixed assumptions into the ram that needs to be held there, limiting ram for other application?

The RAM increase should not be an issue, as the Viewports will reduce memory usage by 30-60%.

Its the _Civ4Config file. In your bts file, you'll see a shortcut to it... double click on that and it'll open it in notepad. Scroll down to where it says:


; Disable caching of xml and file system (may slow initialization)
DisableCaching = 0

and change the 0 to a 1. It will improve performance but slow loading a bit. AIAndy is apparently changing some things in regards to this, most of which I'm not understanding fully. My only problem with caching MORE, especially if the mod makes this compulsory, is that from my limited understanding, isn't that going to make it harder for computers with less ram to play at all? I understand I could be completely wrong, but doesn't caching generally mean that we're loading more fixed assumptions into the ram that needs to be held there, limiting ram for other application?
No, the information that ends up in the process is the same as what would be read from the XML. It only caches the resulting data in a binary format on disk for faster loading when nothing in the XML has changed.
That is something that BtS does itself but for some reason it ends up deactivated regardless of the setting of that switch.

So that sounds really good. But I'm wondering then why caching currently causes so many MAFs on weaker computers. Could you offer me some insight on that matter?

So that sounds really good. But I'm wondering then why caching currently causes so many MAFs on weaker computers. Could you offer me some insight on that matter?
We don't know what exactly Civ is caching in those cases. It can only be some kind of file caching as caching the XML has never been working for C2C.

In other words, the problem is in the core and we don't know but we can resolve it for C2C without worrying about adding further problems then?

In other words, the problem is in the core and we don't know but we can resolve it for C2C without worrying about adding further problems then?
I just write and read my own cache so I have full control over that part. The Civ exe will still do the file caching it does (which probably involves some FPK caching) but it is no more involved in the attempts at XML caching.

Ok. Is there anything I need to keep in mind when creating tags then? I wonder if I'm going to do a merge here soon and get really confused by how you've changed something in regards to this caching stuff.

Ok. Is there anything I need to keep in mind when creating tags then? I wonder if I'm going to do a merge here soon and get really confused by how you've changed something in regards to this caching stuff.
Well, the important thing is to add the tags to the serializations in the info classes that have them (the read and write methods that take a stream).

Well, the important thing is to add the tags to the serializations in the info classes that have them (the read and write methods that take a stream).

Not all are currently I'm pretty sure.

Not all are currently I'm pretty sure.
Probably, but hopefully those will surface fast as bugs when a cached load is done.

Probably, but hopefully those will surface fast as bugs when a cached load is done.

Very fast in many cases, because I don't think that when I changed the storage of many of the arrays (modifiers etc) to just be a NULL pointer if all elements are default, that I changed the C2C-unused cache serialisation routines, so they will likely just fault. Since that was one of the first things I did on C2C (and therefore Civ) it was well before I understood the serialization, so I just wouldn't have noticed those routines needed changing at the time I suspect.

Very fast in many cases, because I don't think that when I changed the storage of many of the arrays (modifiers etc) to just be a NULL pointer if all elements are default, that I changed the C2C-unused cache serialisation routines, so they will likely just fault. Since that was one of the first things I did on C2C (and therefore Civ) it was well before I understood the serialization, so I just wouldn't have noticed those routines needed changing at the time I suspect.
Yes, that was one of the major changes I had to make to the serialization routines (halfway through I realized I should have made a macro given just how many occurances of that there are).

A short update: I am currently testing the caching code and discovered some really weird behaviors in old code like bools that were declared as ints but were used as bools and somehow did not call the int version of the serialization functions but the overloaded bool version. On deserialization on the other hand ...
Anyway, while it is nearly finished, I won't get to pushing it to the SVN until Sunday or Tuesday evening. And I also want to write some more read and write methods for the info classes that are split to many small module files but do not have them yet.

Would it help if I merged all of my Wonder files together? I've got three dozen Wonders plus a few units and miscellaneous buildings and I could easily merge them into one set of files. Right now, they are all individual modules.

Would it help if I merged all of my Wonder files together? I've got three dozen Wonders plus a few units and miscellaneous buildings and I could easily merge them into one set of files. Right now, they are all individual modules.

Once aiandy has done the caching it won't matter, so don't worry about the module structure.

Would it help if I merged all of my Wonder files together? I've got three dozen Wonders plus a few units and miscellaneous buildings and I could easily merge them into one set of files. Right now, they are all individual modules.

With wonders I would leave them in modules for now as people may want to remove some of them when they play. However any which have been around for five versions or so without complaint can be merged into one file. ;)

I am going to get around to merging all those in my files into core one of these days. They have only been around for 10 versions or more.:D

Once aiandy has done the caching it won't matter, so don't worry about the module structure.

You should have told me this before I merged my 40-some Culture Units into one set of files.:rolleyes:

Once aiandy has done the caching it won't matter, so don't worry about the module structure.

So what about the stuff which is not cached? And what is it?

So what about the stuff which is not cached? And what is it?

The only stuff that won't be cached will be the audio and graphics files. It may still be beneficial to consolidate those.

You should have told me this before I merged my 40-some Culture Units into one set of files.:rolleyes:

I still think for the benefit of the modding, its much easier to have as many merges as is possible. The main reason to maintain modular form is if players might want to not play with a particular thing and have it easily removed. But then again, I don't think too many actually utilize this ability either. I wonder how many do.

I still think for the benefit of the modding, its much easier to have as many merges as is possible. The main reason to maintain modular form is if players might want to not play with a particular thing and have it easily removed. But then again, I don't think too many actually utilize this ability either. I wonder how many do.

We get complaints if we do something which relates to the extra religions and don't make it modular. Not sure which religions they are removing but perhaps the latest tech tree changes will draw them out since if there is no religion to found why is there a tech to found it. ;)

For me:
Jainism
Mormon
Ngaiism
Scientology
Sikh
Tengriism
Voodoo
Yoruba.

Which reminds me, they keep XML erroring on me for the female missionaries.
Yes, I can see the technologies to research them even when they are not activated.

Cheers

For me:
Jainism
Mormon
Ngaiism
Scientology
Sikh
Tengriism
Voodoo
Yoruba.

Which reminds me, they keep XML erroring on me for the female missionaries.
Yes, I can see the technologies to research them even when they are not activated.

Cheers

How can I fix stuff if people don't tell me it is broken. :p

I've just forgotten to tell you. Doesn't take long to click those errors away anyway.

Cheers

I've just forgotten to tell you. Doesn't take long to click those errors away anyway.

Cheers

It was all the latest female missionaries. Which means I was slack when I merged them in.:blush: I will do a quick test and put the fix in the SVN.

I have pushed the caching to the SVN.
For now it is deactivated by default as bugs are to be expected. You can activate it with the global define DO_CACHING in A_New_Dawn_GlobalDefines.xml.
First run will create the caches and they are then used afterwards unless one of the XML files has been changed. The loading time is reduced from 85s on the first run to 40s on the second run for me (fast test run, not repeated).

I have written some additional serialization methods but not all info classes for which it would be useful have them yet so further speed increases are possible.

The important thing now is for you to test it and especially check for missing stuff on buildings, units and the like when the game is loaded from the caches. The serialization functions have not been tested for quite some time and it is not unlikely that some modders added stuff to the XML but not to the serialization functions (although most seem to have updated them fine).

I have pushed the caching to the SVN.
For now it is deactivated by default as bugs are to be expected. You can activate it with the global define DO_CACHING in A_New_Dawn_GlobalDefines.xml.
First run will create the caches and they are then used afterwards unless one of the XML files has been changed. The loading time is reduced from 85s on the first run to 40s on the second run for me (fast test run, not repeated).

I have written some additional serialization methods but not all info classes for which it would be useful have them yet so further speed increases are possible.

The important thing now is for you to test it and especially check for missing stuff on buildings, units and the like when the game is loaded from the caches. The serialization functions have not been tested for quite some time and it is not unlikely that some modders added stuff to the XML but not to the serialization functions (although most seem to have updated them fine).

Cool. Thanks for picking up on this, where I was too lazy to do more than whinge about it ;)

Cool. Thanks for picking up on this, where I was too lazy to do more than whinge about it ;)

I don't think any of us are 'lazy'... quite the opposite. We just have our skill sets and priorities that vary is all. You guys are making this mod overcome ALL the problems that AND has been plagued by and we would have continued to have been plagued by to such an extent that you're paving the way for this mod to be not only one of the best mods on the site, but one of the best games ever made. We can all dream and some of us can begin to implement those dreams. But you guys make it possible for all those implementations to run correctly and smoothly and keep the game playable. I'm impressed with EVERYTHING you two do!

Well done AIAndy! Keep us posted as to whether you find and work out any bugs. My god, between you and Koshling, my imminent merge is looking more and more intimidating by the day! lol! I'll just have to be VERY careful I think.

Keep us posted as to whether you find and work out any bugs.
I have to rely on people that play a lot to find the bugs as I would not recognize missing stuff on buildings given how rarely I actually play C2C.
So if there are no reports in the next days I will just activate caching by default which then should hopefully get feedback soon.

Is it recommended to have this in the CivilizationIV.ini

; Disable caching of file system (may slow initialization)
DisableFileCaching = 1
; Disable caching of xml and file system (may slow initialization)
DisableCaching = 1

While enabling DO_CACHING in A_New_Dawn_GlobalDefines.xml ?
_________________

Disabling caching in the .ini speeds up gameplay generally doesn't it?

Is it recommended to have this in the CivilizationIV.ini

; Disable caching of file system (may slow initialization)
DisableFileCaching = 1
; Disable caching of xml and file system (may slow initialization)
DisableCaching = 1

While enabling DO_CACHING in A_New_Dawn_GlobalDefines.xml ?
_________________

Disabling caching in the .ini speeds up gameplay generally doesn't it?
No to both questions.
Especially file caching has a serious effect on the start time of C2C that is independent of the new caching I added.

I recommend having both those options at 0 (meaning that the caching the exe does is active as well) and then setting DO_CACHING to 1.

I recommend having both those options at 0 (meaning that the caching the exe does is active as well) and then setting DO_CACHING to 1.

But will it "hurt" if the first two are set to 1, and the third is set to 1 as well? Because I would like to leave the first two at 1 for other purposes. If so, then I will just change the first two, as it would be easier to leave them at 0 then to change the third to 0 each time the SVN is updated (when you actually change it in future SVNs).

@AIAndy:

I did a test of load times on my crappy rig with caching on, and the results were;


Pre-load: 119 seconds (not much we can do about that)
Init XML: 40 seconds (Yay!)
Other functions: 10 seconds


So 170 seconds total. Not bad, considering how big of a mod we have.

But will it "hurt" if the first two are set to 1, and the third is set to 1 as well? Because I would like to leave the first two at 1 for other purposes. If so, then I will just change the first two, as it would be easier to leave them at 0 then to change the third to 0 each time the SVN is updated (when you actually change it in future SVNs).
They are completely independent so any combination works.
Btw, if you make any changes in your local SVN copy and update then, it should merge in your changes and only rarely cause conflicts.

@AIAndy:

I did a test of load times on my crappy rig with caching on, and the results were;


Pre-load: 119 seconds (not much we can do about that)
Init XML: 40 seconds (Yay!)
Other functions: 10 seconds


So 170 seconds total. Not bad, considering how big of a mod we have.
Since I have made sure that DisableFileCaching is set to 0, I do not get a lot of Pre-load time.

They are completely independent so any combination works.
Btw, if you make any changes in your local SVN copy and update then, it should merge in your changes and only rarely cause conflicts.

Sounds good. BTW, I was speaking of my "working" copy of the SVN, not the one I update daily. I update delete and replace my working copy every week or so with the latest SVN, and that is the one I would change. I may forget changes that I made previously when I do so.

They are completely independent so any combination works.
Btw, if you make any changes in your local SVN copy and update then, it should merge in your changes and only rarely cause conflicts.


Since I have made sure that DisableFileCaching is set to 0, I do not get a lot of Pre-load time.

I have both caching options set to zero already, my computer is just old and crappy, so that probably contributes to the issue.

We'd spoken a bit earlier about MAFs taking place, particularly on weaker systems, if file caching is set to 1. Has anything you've done given us any cause to suspect this would no longer take place?