v143 save load times
- Thread starter Zaldron
- Start date
Thalassicus
Bytes and Nibblers
A savegame Pep provided revealed some useful information. The "Demographics: Dequeuing demographics" line of output is when loading's done. This finishes within a minute, and the rest of the time is spent doing something else. In other words, it's not technically taking a long time to load, it's taking a long time to do something after loading. This means the load-game code I was searching through is not the problem. It might be some "start of turn" code instead.
Thalassicus
Bytes and Nibblers
For those encountering long load times, what map size & speed do you play on?
whoward69
DLL Minion
Completely off-the-wall thought regarding this. Try temporarily disabling anti-virus checking (or better if you know how to do it, exclude the cache, Saves and ModdedSaves sub-directories from your anti-virus on-access scans) and load the game to see if that makes any speed difference.
I play on standard map size and Marathon speed. I suppose Marathon has an effect over long loading times. At the beggining of the game, loading time is normal. It increases with the number of turns elapsed. I think it may be related to the number of units or cities on the map, because it increases over time.
gdwitt
Prince
I've experienced super long load times for the last 2 weeks or so. It has always been long.
I am the one that also has 5 minute turn times after turn 140 or so.
I just take it as a matter of course and never reload if I miss an opportunity.
It's about 10 to 15 minutes, depending on complexity.
Attached is a lua of the latest game.
How else can I help you?
My computer: 2009 HPPavilion, 6GB Ram, 1GB ram video card.
I play with AVP turned off and game booster turned on.
Nothing seems to help much. It's something in the coding.
I am the one that also has 5 minute turn times after turn 140 or so.
I just take it as a matter of course and never reload if I miss an opportunity.
It's about 10 to 15 minutes, depending on complexity.
Attached is a lua of the latest game.
How else can I help you?
My computer: 2009 HPPavilion, 6GB Ram, 1GB ram video card.
I play with AVP turned off and game booster turned on.
Nothing seems to help much. It's something in the coding.
Torvald
Prince
I too have lately been experiencing very long load times for saved games midway through a game, sometimes as long as five minutes.
I normally play at Prince difficulty level, standard map size (8 players), and with either Continents or Pangea map type.
I thought at first that it might be because of the very large number of added Wonder mods I've been using on top of UP/VEM, but on second thought it seems that the long load time has only been happening since approximately v137 of UP/VEM.
At any rate, my next UP/VEM game will be played WITHOUT any added Wonder mods, to see if that makes any difference. (Plus it will give me a chance to go for a Science victory, as too many Wonders - although fun - really skews the game towards cultural victory.)
I normally play at Prince difficulty level, standard map size (8 players), and with either Continents or Pangea map type.
I thought at first that it might be because of the very large number of added Wonder mods I've been using on top of UP/VEM, but on second thought it seems that the long load time has only been happening since approximately v137 of UP/VEM.
At any rate, my next UP/VEM game will be played WITHOUT any added Wonder mods, to see if that makes any difference. (Plus it will give me a chance to go for a Science victory, as too many Wonders - although fun - really skews the game towards cultural victory.)
Dunkah
Emperor
I have been playing Standard map, Epic Speed, Continents Plus, Prince.
Definately getting longer as the game gets older. 13 - 15 minutes.
Definately getting longer as the game gets older. 13 - 15 minutes.
That's very odd because my load time for a standard-size map at turn 100 is about 30 seconds. In pep's log, 98% of the load time is coming from an unreported source. I've started a thread on the main modding forum asking what that might be.
Out of curiosity, did you delete the cache before loading these unusually-long loadtime saves?
I'll just reply for myself: No, I clear the cache *only* when updating to a new version of VEM. The only other addon I use is infoaddict.
Standard/standard.
keepitsimpleeng
Chieftain
I play on standard/standard.
I also noticed late in the game, extremely long lags between user input and game response, 10 seconds or longer. Further more, an extremely large number of page faults for the Civilization_VDX11.exe, like 25,000,000+. Each user input will produce a 100 or more page faults.
I checked for page faults during a long load, and they were a tad high, nothing unusual. Will test setting priority to high... .. .
SysInfo: Windows Vista Ultimate, 64-bit (SP2); DirectX 11.0; GeForce GTX 560 Ti 1024 MB GDDR5; EVGA__132-YW-E178-FTW; Processor: Intel Core 2 Quad CPU @ 2.66GHz (4 CPUs); Memory: 8190MB DDR2 1066MHz RAM System Disk: SATA 150 GB 10Krpm; Steam Program Disk SATA 320 GB 7.2Krpm
I also noticed late in the game, extremely long lags between user input and game response, 10 seconds or longer. Further more, an extremely large number of page faults for the Civilization_VDX11.exe, like 25,000,000+. Each user input will produce a 100 or more page faults.
I checked for page faults during a long load, and they were a tad high, nothing unusual. Will test setting priority to high... .. .
SysInfo: Windows Vista Ultimate, 64-bit (SP2); DirectX 11.0; GeForce GTX 560 Ti 1024 MB GDDR5; EVGA__132-YW-E178-FTW; Processor: Intel Core 2 Quad CPU @ 2.66GHz (4 CPUs); Memory: 8190MB DDR2 1066MHz RAM System Disk: SATA 150 GB 10Krpm; Steam Program Disk SATA 320 GB 7.2Krpm
Thalassicus
Bytes and Nibblers
Page faults are part of normal memory access, not really an error. Games like Civ are memory intensive.
keepitsimpleeng
Chieftain
True, and these are soft page faults thus far less time and resource consuming than hard page faults.
On the other hand, the interrupt from a soft page fault can cause a context switch (where a processor L2 cache is flushed) which happens in processes with a large number of threads. I recall a maximum of 25 threads per processor (which maybe low for a hyper-threaded processor) and CivilizationV_D11 is running 255 threads.
The time required is for the cache to be filled for the processors next task. CPUs with larger cache sizes will suffer more when context switches increase (my cpu uses 2x4MB) and different CPU configurations will also have a big effect. The P4 hardware context switching overhead was 995ns, newer processors are faster. Context switching is the evil twin of SMP.
In my case mentioned earlier, the 25 million+ page faults were twenty-five time higher than the next process with the highest page fault count, and it was using >,1% of CPU time compared to CivilizationV_D11's ≈ 33%.
Another factor may at work here, programming methods for SMP are different than for uni-processor, code optimized for uni-processors may not take full advantage of SMP system, and vice versa.
The time required is for the cache to be filled for the processors next task. CPUs with larger cache sizes will suffer more when context switches increase (my cpu uses 2x4MB) and different CPU configurations will also have a big effect. The P4 hardware context switching overhead was 995ns, newer processors are faster. Context switching is the evil twin of SMP.
In my case mentioned earlier, the 25 million+ page faults were twenty-five time higher than the next process with the highest page fault count, and it was using >,1% of CPU time compared to CivilizationV_D11's ≈ 33%.
Another factor may at work here, programming methods for SMP are different than for uni-processor, code optimized for uni-processors may not take full advantage of SMP system, and vice versa.
In any case, this would affect game play more than load times.
Note: My background is in Unix style systems, not Windows systems so I may not be spot on with this.
Thalassicus
Bytes and Nibblers
How does it compare alongside other games like Skyrim or Starcraft 2?
keepitsimpleeng
Chieftain
I downloaded & installed StarcraftII demo (a full version with limited scenarios), then ran a small test. The number of threads was around 55, and CPU utilization was approximately the same as CivilizationV_D11. There were no delays or lags in video or game·play.
Vista has redesigned the way the CPU handles interrupts/memory management and makes available the information. I'm mucking around trying to find out how to measure the processor times dealing with the hardware interrupts with regards to context switching. (It was easy to find on XP).
My suspicion is the large number of threads combined with the large amount of memory is inducing a choke point tied to hardware interrupts for context switching. This is a casual thing inspired by boredum, the bane of the retired 
.
.Other things I`m considering doing is try Civ V on 64bit XP and 32 bit XP (if Gigabyte ever gets my faulty main·board back to me). Some folks have reported getting Civ V to run on Linux/wine, and I may try that too.
Dunkah
Emperor
FYI I am running on a 32 bit machine with Windows 7 and experiencing the long load times!
I am abandoning my latest game and going to start with 149.?? when I start up again.
I am abandoning my latest game and going to start with 149.?? when I start up again.
keepitsimpleeng
Chieftain
Thalassicus
Bytes and Nibblers
My suspicion is the large number of threads combined with the large amount of memory is inducing a choke point tied to hardware interrupts for context switching. This is a casual thing inspired by boredum, the bane of the retired
This is likely true. Also, the Lua language we use for modding is not nearly as efficient as c++, so the work I do is probably much more memory intensive. It's difficult to track down problems in more details since ModBuddy has no debugging tools... it doesn't even have a message logger (I had to create one myself). These aren't factors we can control, however, so I just do the best I can with the tools available.
keepitsimpleeng
Chieftain
This is likely true. Also, the Lua language we use for modding is not nearly as efficient as c++, so the work I do is probably much more memory intensive. It's difficult to track down problems in more details since ModBuddy has no debugging tools... it doesn't even have a message logger (I had to create one myself). These aren't factors we can control, however, so I just do the best I can with the tools available.
Personally, I think you do a great job. Please point your elbow at the ceiling and pat yourself on the back. 
Now for the not so good news. I loaded a 280 turn standard/standard (the same as before) and it took the customary 12min18sec. In that time there were 864,001 context switches. After one turn of play there were 2,382,537. Curiously, the I/O during most of the load was a steady 53,000+ bytes per second.
I also loaded a 380 turn save of the same game. This time there were 40,720,123 context switches. I don't know how long this one took, as I popped down to the pub for a few. At this point, game play is excruciatingly slow.
I note that the thread count is 25. Earlier I saw (or think I saw) 250+ threads, but this was after several hours of play. I could be wrong. In any case, it appears to be a result of excessive context switches. Note that there are more context switches than page faults. The efficiency of the hardware in handling context switches most likely varies widely.
These were measured using Process Explorer from MS Windows Sysinternals. You have to hunt around to find the columns for context switching. The measurement delta here is one second.
I think I'll try an unmodded game and test.
gdwitt
Prince
My long load times seemed gone on 3 different tries in the v149.
The games loaded in under 5 minutes each.
I only went to turn 110 however
I timed my v146 load times at between 15-21 minutes each for games around turn 140.
Would seeing my lua file help?
The only change I did was set the task manager priority to high.
That has never helped in other games.
The games loaded in under 5 minutes each.
I only went to turn 110 however
I timed my v146 load times at between 15-21 minutes each for games around turn 140.
Would seeing my lua file help?
The only change I did was set the task manager priority to high.
That has never helped in other games.
Similar threads
