Ridiculously long save and load time

[syllogism]

Ouch my save is 32mb

e: it was 9mb and then 5 turns later it's 32mb

e2: turns out 30mb of that was FINAL_RELEASE repeated over and over again

e3: I see someone already figured this all out

 

[sbq]

I'm on turn 333 of a huge continents map, year 1565 AD, and my quicksave and save game file is at 818KB. I never load form in game, I always exit out to Windows then restart the game, and I've loaded *a lot* to do the reseed and for testing. I'm betting if I didn't have reseed turned on and just played through without loading much the size would be much smaller. This seems to confirm that the bug is in the load function, that it's adding junk to the game state that gets saved, and if you load a lot from in game it multiplies exponentially.

After I finish this game I'm going to try a game with auto-save going every turn and use that for loads, see how big they get.

P.S. Siam is going to kick my behind all over the place in this game, went with Prince difficulty for the first time and he's already conquered his continent and hit industrial era.

 

[kuwazome]

I'm going to guess that its not just the FINAL_RELEASE crap, since the file size increases vs how many of these are added would not correspond. For example adding two of these would not add 2kb, but from the file size change thats what happens when I load/save from in game for the first time. I'm assuming that although the FINAL_RELEASE crap is evident of a broken save, its adding more than just that and obviously doubling something.

I've also figured out why my turns take so long, and its is precisely do to the larger saves filled with crap in them, since I have this 30-60 second delay AFTER all the AIs have made their turn. And this was with every 5 turn autosave, yet it lags EVERY turn.

The best procedure it seems is that if you need to load, close the game, start the game again and then load your game, or else it will start bloating. Any load or save procedure, even from the main menu seems to effect the size of the saves, I'm guessing its saving previous games that were in the memory and not dumped get saved into the same file. Saving should just be autosave every turn, make it like keep max 1000 or something, you have enough harddrive space for it.

You'll notice the save once its "corrupted" that when you load, Civ5 in window mode becomes unresponsive for a bit, rather than just the loading cursor. Windows puts a (not responding) into the top of the program title. Or you can go into a hex editor and confirm how many lines of "201080 FINAL_RELEASE" you have; in the good save file I've just tested, I have only one of these. The first autosave (@ 4000BC) of this correct save is ~500kb, vs the first autosave of a "corrupted" save is ~900kb. One second turn, the correct save is ~ 650kb, vs the "corrupted" save is ~1mb.

 

[Tekener]

I'm plaing in snail on a large map, having turn 935 and you can guess it: savegames having tons of tons of FINAL_RELEASE200405 in it, it's nearly impossible anymore to save or even load the game. On my 3 Ghz Cure2Duo with 4GB RAM on Win7 64bit loading takes about 5 minutes - if it doesn't freeze or crash, saving only about 2 minutes.

This is ridiculous, did nobody at Fireaxis ever played the game longer than 30 turns?

I agree with kuwazome, loading the game seems only to work when you start CivV new, loading it from an existing game or by going just back to the main menu will freeze it. Reduzing the auto saves from a save each 4 turns to a save each 20 turns helps keeping the play, but is of course no solution or even good protection against bad things happening (CTD, freezes, AI decalring war, ...)

Does anybody already have an idea how to tweak this or if Firaxis is working on this?

 

[abtomat74]

They are well aware of the problem and said last week that a patch was in the works for this problem. That was posted before version .20 arrived. I would expect the next patch to address this, if not the one after.

In the meantime, use the methods laid out here by myself and the others in regards to a fresh game. I plan to stick with loading only from desktop, and allowing my auto saves to do all my saving, each round. Since I have only played quick and normal paced games on normal map sizes, I don't know how it will affect a fresh Epic or large game.

If you guys already have tons of repeated "Release Note" garbage, use the file linked by the guy named Arak here halfway down the page:
http://forums.2kgames.com/forums/showthread.php?p=1158897#post1158897

It will automatically trim nearly all of the garbage out, leaving hopefully a 2MB file for you guys that have larger than 2MB files already. That is better than nothing.

I have not needed that file since using my method though.

 

[Tekener]

I haven't seen this thread, thanks for psoting the link again.

About the issue itself: I started a fresh game using auto save every round. It still seems to happen with large epic games.

Actually I have a dual boot XP 32 bit and Win7 64 bit. Once started a game on the Win7 I can't get the save game to run on XP's Steam, even though the game file HD is of course the same for both systems. The other way, starting and saving with XP works fine when loading it from Win7.

Seems like the game somehow stores the information if it can use 64 bit with 4 GB or not, just a guess to explain the behavior - or it's just that the save files are garbled only on 64 bit system and therefore cannot be used anymore on 32 bit?

Sorry if this is all nonsens, but I just try to find a reason why 32 bit saves are working on 64 bit but not the other way.

 

[kuwazome]

Seems like the only thing that doesn't add the junk to your files is if you don't load at all. I played a game well into 600 turns with a working save, but the catch is that I never restarted my game, it was on the whole time.

 

[Tekener]

I don't let my computer running when going to sleep

Anyway, I cleaned my save files with the java tool from the other thread, that worked fine, but loading/saving times didn't decreased. It still takes several minutes to laod/save. I hope they will fix this also with the save file garbage thing.

 

[kuwazome]

I don't let my computer running when going to sleep

Anyway, I cleaned my save files with the java tool from the other thread, that worked fine, but loading/saving times didn't decreased. It still takes several minutes to laod/save. I hope they will fix this also with the save file garbage thing.

Yeah it won't fix it as its not just that final release crap, theres obviously other junk in there that shouldn't be in there.

Probably the only companies that have yet to disappoint me with buggy crap is Valve and Blizzard...

 

[JayRyan]

Save time isn't bad for me. Load time is bad, especially since it always crashes now if I load from the game. I have to quit and restart the game to load a game.

I am having the exact same problem...

 

[Hormagaunt]

I just got the game and installed it (that's a whole different annoyance).

After having the game freeze during a load, I ran the following test:

1) Created a new scenario, Settler, Huge, 1 opponent, 0 city-states. I also set that the random number seed changes. Saved it, exited the game.

2) Copied Test01 to Test02 and Test03.

3) Started the game, loaded Test01, ran my warrior in a straight line until I found a Ruins. Entered the Ruins, noted what I got, saved, exited the game.

4) Started the game, loaded Test02, ran my warrior in a straight line to the same ruins. It wasn't exactly on my line, so when I moved off, I went through the same hexes as previous. I DID NOT enter the Ruins.

5) Saved the scenario, entered the Ruins. Didn't get what I got the first time. Reloaded the scenario, tried again. Repeated this about 10 times until I got the same thing. Note that I DID NOT exit the game, merely used ^L to reload after each failure.

6) Saved the scenario, exited the game.

Examination of Test01 versus Test02 was interesting. Test01 was about 450 KB. Test02 was 3.7 MB.

Preliminary thought is that it's saving some sort of memory leak, and it adds to the leak EVERY time you reload a scenario. So if you find a Ruins, save, then hit it multiple times, reloading until you get a result you like, and you do this several times, well, that file's gonna get LARGE.

I saved Test03 for another run (which I haven't done yet). Go to the Ruins, save, enter the Ruins. If I don't get what I want, exit the GAME, restart, reload, try again until I get the same thing, then save the scenario. The objective will be to see how big the file gets on the final save if it's always from a fresh load of the game and scenario.

 

[kuwazome]

Bottom line is for now, just restart the game entirely if you want to load. Even that adds a bit to the file, but its barable. For now I just keep my game on and put my computer to sleep so I don't have to load.

 

[DaveShack]

This is ridiculous, did nobody at Fireaxis ever played the game longer than 30 turns?

It sounds like this bug is strictly from reloading while the game is running. Speaking in general terms, there is a tradeoff when testing between doing what players should do and what players will do. I have no knowledge of who tested this time around, but many of the Civ4 testers were people who play what they have instead of reloading to see if the odds will change.

This will make it much easier to check GOTM and HOF games for reloading.

 

[Baleur]

The save/load times are horrid for me. It takes 2 minutes to save, and about 3 or 4 minutes to load. Except that when I load it crashes, so I have to keep restarting it which is annoying as hell.

Woah really? For me it saves in like 3sec..
Im running with a slow old Core 2 Duo E6600. Its so wierd how these issues seem randomly spread out across systems. They definetly didnt test this properly. Thats the point of QA, to test on as many different hardware setups as possible. >_>

 

[abtomat74]

Woah really? For me it saves in like 3sec..
Im running with a slow old Core 2 Duo E6600. Its so wierd how these issues seem randomly spread out across systems. They definetly didnt test this properly. Thats the point of QA, to test on as many different hardware setups as possible. >_>

On the contrary, I'm sure they tested the hell out of it. The problem is usually the publisher trying to push the game out by a certain date, in this case just prior to all the games coming out in the next day(tomorrow) and several weeks following.

This isn't a system spec problem, it's a save file problem. I also load in a few seconds now that I have tamed my save file sizes...otherwise I was in the several minutes to load and save category before learning the tricks to avoid the bloating, cancerous save file growth.

 

[dannythefool]

It will automatically trim nearly all of the garbage out, leaving hopefully a 2MB file for you guys that have larger than 2MB files already. That is better than nothing.

2 MB is probably right, I'm trying to figure out the file format and now I think they have accidentally (or just being sloppy) coded saving as serializing objects that all contain a huge amount of duplicate data.

I looked at a turn 0 autosave from a duel game that was 280k, which sounded almost okay-ish until I realized that most of it really is compressed data. Which expanded to more than 7 megabytes once I figured out what compression it uses (hint: the game ships with zlib...) 7 megabytes of uncompressed data for a game on a 52x32 size map that hasn't even started yet.

When I looked at the decompressed data, I found lots of lists of all building types (BUILDING_FLOATING_GARDENS BUILDING_COURTHOUSE BUILDING_MUGHAL_FORT etc.) over and over again, 190 times in my duel game, and the same for promotions, 78 times. And then some blocks with the same strings but partially overwritten, possibly some previously used but now unused memory (in the best case). Basically, this is what would happen if you had a class for buildings that knows all those strings from the XML, and then saved all the data every single instance of anything that could remotely be related to a building has access to...

 

[last skunk]

The only way to play is to put auto saving on "every round"...and pray you don't want to try a new seed because as you all know this means crash.

Load once and play, let the auto save do it's thing while we await the patch for the not fully tested product.

I am affraid that the issue has little to do with memory, resources etc though on "older" hardware the game has far more problems(64bit 7, 1gig ddr3 nvidia, 8 gig ram), these errors seem to be in code as I have taken 8 games past turn 350 where the errors become critical. In my case it is the loading-playing of worker and unit selection animations despite map size, unit numbers, tile improvements etc. If it were simple a resource issue, and this game was designed with the efficiency of a wood burning stove, I wouldn't see 90sec delays for every unit selected or sound effect, with audible glitches-loops, this is all bad code and for some reason it increases late game irrespective of complexity.

I tried a small map 2 civs 4 city states, avoided a early end to game and the same thing is happening past turn 300. Auto saves or loads do NOTHING to address these issues though the first 200 turns fly, yeah. It's like they beta tested for for first 100 turns and decided everything was fine. Who releases a pc game that doesn't have proper saving features? 2k support for mp says just save and reload, oh if that were only possible. good god, nothing like investing 12-24 hours into a dead end.

Who here can't wait to see a group of workers doing the synchronized swimming routine or halo for every for every action? don't get me wrong the combat animations actually work fine for me but come on after two hours I don't need to see a long bowman scratch his balls or a worker shovel after their turn has ended think of how great the game would WORK if I could select single unit graphics, animations only during combat, and no ridiculous fireworks and clunks every time I select a unit.

Has anyone taken a game past turn 275 and completion without issue?

 

[abtomat74]

I am affraid that the issue has little to do with memory,

I mentioned "memory leak" because the game uses over 3GB of ram for me(see my reply after the one you qouted). There is no way in hell a game of this scale needs even close to 2GB of system ram to run smoothly, let alone 3+. GTA 4 doesn't even come close to that amount.

That certainly smells like a memory leak, especially when the GB used increases over time(watch your task manager). However, I have noticed it not using as much ever since taming my save files...so they are related.

 

[dannythefool]

There is no way in hell a game of this scale needs even close to 2GB of system ram to run smoothly, let alone 3+.

You underestimate the power of object oriented programming

 

[noas5]

Turn 133 - saved Modded game last night. Now crashing 30sec into loading. Will not do anything so I use Task manager to get out of game. Says runtime error.... so I close down. Have tried this several times.