MAF! Who wants the saved game functions in the SDK?

[seady]

I was thinking about the way to fix the memory allocation functions that are crashing large mods. Firaxis doesn't want to spend money to fix it, which is fair enough, they have budgets and bills to pay, and if we want something for free (a better save game system!), well, we could do it ourselves if they let us!

Nev seems to have found a solution and I have been playtesting huge maps without a hitch so far. I'm going to try to get Firaxis interested.

 

[rockinroger]

@Seady, hey good idea. the MAf is a major annoyance. I usually get them at about the industrial era. Also they save size is about 1 mb. Then if i switch to low graphics i can get bigger saves. The low graphics saves, i have gotten as large as 1.4 mb. then as usual poof ctd. Lets get some people fixing this.

 

[Tigerclaw]

Seconded. It's a major problem for me, I can't play much past rifling on my preferred Huge maps (which my system can otherwise handle, albeit with a little slowdown). I've also had it a couple of times on super large/long marathon UNMODDED games, so b*****ks to those who are saying it's only a modding problem. Don't have the saves still though, as autosave saved over them...

 

[TAfirehawk]

It is certainly NOT a modding issue, but BIG mods make it worse.

Nobody I found on this forum that claimed a MAF in a non-mod game would answer my PM when I had TWO FIRAXIAN'S PM'ing me about the problem so they could investigate....so in the end Firaxis wanted to look at it but nobody here could come up with a non-mod save game that MAF's.

 

[Tigerclaw]

Apologies for not being able to send you anything, but like I said, in unmodded games, this problem only appears towards the very end of VERY long, and VERY large games. It's not something that can be produced out of a hat, it would require hours (more likely days) of time invested in producing a save featuring this problem. And then that's only one save, and most likely the testers would want several, at least. In the absence of an autorun feature (although a partial system seems to exist in the Barbarians scenario.), and the absence of days worth of free time for "working" at playing a game, there's little most people can do.

 

[seady]

I actually asked one of the programmers about an auto run feature. I thought that that would be used to test the system out and it looked like it was there already from what i could see.

It seems likely if i make a junk xml file that's large enough and syntactically correct, and make a mod with valid but stupid code, i can make a saved game file that might meet their requirements. You know, a thousand variations of warrior with identical stats except the name (warrior1, warrior2 .... warrior1000). That might do it.

I know MrGenie is busy now but I'd love to know what he found.

In any case, what we can do as a community is sign a petition. The more people we get interested, the more likely they are to take notice. Companies exist to make money, and we are saying we will fix your code for free.

 

[TAfirehawk]

Apologies for not being able to send you anything, but like I said, in unmodded games, this problem only appears towards the very end of VERY long, and VERY large games. It's not something that can be produced out of a hat, it would require hours (more likely days) of time invested in producing a save featuring this problem. And then that's only one save, and most likely the testers would want several, at least. In the absence of an autorun feature (although a partial system seems to exist in the Barbarians scenario.), and the absence of days worth of free time for "working" at playing a game, there's little most people can do.

Actually I did play for about 3 days of just plain Warlords....well past the end at 2050 with no MAF. But my machine has never had an MAF so it was inconclusive...

 

[seady]

@ Tafirehawk how much ram do you have? Which OS?

I had an idea while hanging out my washing that made me rush inside. I suspect that the problem with mods is that civ4 loads every bit of artwork and xml into ram; even though in a mod it might only use 1% of it. If you have 100+ civilizations with levels of unique buildings and units, this is an incredibly wasteful way to do it. It makes sense to choose what races etc are in the game... before you load the artwork and so on.

This is not a problem in vanilla because you only have a few races, maybe one unique building and unit, a few religions, and so on, and the percentage of 'fat' is low. In Visa it is enormous. Maybe a pre-loader is needed? Since i doubt firaxis is going to open up the code enough for us to fix it?

I'm going to post this in the main area. I doubt anything will be done, but i am pretty sure that's the answer. I wish i was bright enough to write the code for this solution.

 

[Gyathaar]

@ Tafirehawk how much ram do you have? Which OS?

I had an idea while hanging out my washing that made me rush inside. I suspect that the problem with mods is that civ4 loads every bit of artwork and xml into ram; even though in a mod it might only use 1% of it. If you have 100+ civilizations with levels of unique buildings and units, this is an incredibly wasteful way to do it. It makes sense to choose what races etc are in the game... before you load the artwork and so on.

This is not a problem in vanilla because you only have a few races, maybe one unique building and unit, a few religions, and so on, and the percentage of 'fat' is low. In Visa it is enormous. Maybe a pre-loader is needed? Since i doubt firaxis is going to open up the code enough for us to fix it?

I'm going to post this in the main area. I doubt anything will be done, but i am pretty sure that's the answer. I wish i was bright enough to write the code for this solution.

If that is the case.. have you tried to disable this cacheing and see if that helps the problem?
I believe the 3 settings in the ini file you would need to change to disable all this would be : the following (set all 3 values to 1)

; Disable PAK memory mapping (May affect performance)
DisablePAKMemoryMapping = 0

; Set to 1 to page units out when non-visible
DynamicUnitPaging = 1

; Set to 1 to page unit anims out when the unit is non visible
DynamicAnimPaging = 0

The PAK memory mapping would be that part that cache all the units into memory I believe. If you change this you probably want to extract all the art files with the PAKBuild program, or you will get a big performance hit.

 

[seady]

If that is the case.. have you tried to disable this cacheing and see if that helps the problem?
I believe the 3 settings in the ini file you would need to change to disable all this would be : the following (set all 3 values to 1)

; Disable PAK memory mapping (May affect performance)
DisablePAKMemoryMapping = 0

; Set to 1 to page units out when non-visible
DynamicUnitPaging = 1

; Set to 1 to page unit anims out when the unit is non visible
DynamicAnimPaging = 0

The PAK memory mapping would be that part that cache all the units into memory I believe. If you change this you probably want to extract all the art files with the PAKBuild program, or you will get a big performance hit.

Hmm. Does pak cache just the 2d stuff or 3d as well? DisablePAKMemoryMapping = 0 is the only thing i don't use. I have 512 meg and do see MAFs occasionally with Visa. I was going to get a new pc but my oven blew up and whoosh goes my entertainment budget.

 

[Gyathaar]

Hmm. Does pak cache just the 2d stuff or 3d as well? DisablePAKMemoryMapping = 0 is the only thing i don't use. I have 512 meg and do see MAFs occasionally with Visa. I was going to get a new pc but my oven blew up and whoosh goes my entertainment budget.

Assume it would affect the 3d files too.. since they are in the pak files. You would need to set the value to 1 btw to disable it.. personally I have never gotten an out of memory error (assume the bug happens when zlib tries to allocate memory for the temporary buffer before it compresses the save file, and it fails because the computer is out of memory), so I dont think I am the right person to test.

 

[TAfirehawk]

No seady, all the art is not loaded....just as was posted in the G Era thread.

ViSa by itself has almost 700 MB of art and all of Warlords runs under 500 MB for me.

My system is 'old', 2.4GHz with 1GB RAM.

 

[seady]

No seady, all the art is not loaded....just as was posted in the G Era thread.

ViSa by itself has almost 700 MB of art and all of Warlords runs under 500 MB for me.

My system is 'old', 2.4GHz with 1GB RAM.

Yeah, i got the offical answer... oh well.

does civ load unused artwork into ram?

No, it loads artwork at the time it's about to be used.

I assume that applies to animations sound files and xml data?? If so, i have no idea how civ 4 works; i had assumed it took the xml 'tokens' and assembled them into a database (including sounds, animations and 2d art), and then created a map based on your selections (e.g aztec, indian, chinese) and left the other civs still in memory.

So does it load the art on the fly? That seems slow? Or does the map creation screen also determine which 'tokens' are read into RAM? That's the way I'd have done it, if i were a programmer, which i am not (and I have the certification to prove it)!

 

[barty911]

from what ive noticed MAF appears when a huge map, on marathon speed is at a high level of tech advancement... so cant you just manafacture a map like that with world builder, give each civ two dozen towns and loads of units, ten gazillion workers?

I might have a messed up autosave file on my old pc, no mods at all.

 

[seady]

from what ive noticed MAF appears when a huge map, on marathon speed is at a high level of tech advancement... so cant you just manafacture a map like that with world builder, give each civ two dozen towns and loads of units, ten gazillion workers?

I might have a messed up autosave file on my old pc, no mods at all.

If you find it happens with no mods, please keep those saves, as it looks like they will only accept responsibility if the MAF occurs without the use of a mod. Since modders cannot alter the save game function, there's no logical reason for that, but there you go. Send a private message to Alexman and hopefully a copy of the game as well; if it's too large to do in one chunk, maybe the file can be split.

 

[ERLoft]

My most recent Visa game has been getting the MAF error in the early industrial era on marathon speed with only a large map and 16 civs. Hardly deep into a huge game with lots of civs. And my system specs are FAR beyond anything that this could/SHOULD require.

 

[TAfirehawk]

My most recent Visa game has been getting the MAF error in the early industrial era on marathon speed with only a large map and 16 civs. Hardly deep into a huge game with lots of civs. And my system specs are FAR beyond anything that this could/SHOULD require.

As said before, it is a bug in the save routine in the Civ EXE...nothing to do with any mod. The point the MAF happens seems to vary and makes no difference what hardware you have....my laptop has 512MB RAM and has no problems with the MAF.

Basically an array overflows on save and crashes but nothing we can do as only Firaxis has access to the EXE.

 

[ERLoft]

As said before, it is a bug in the save routine in the Civ EXE...nothing to do with any mod. The point the MAF happens seems to vary and makes no difference what hardware you have....my laptop has 512MB RAM and has no problems with the MAF.

Basically an array overflows on save and crashes but nothing we can do as only Firaxis has access to the EXE.

So, is it random based on a particular game? If I start a new game, am I likely to have it start crashing at about the same point?

 

[CuriousG]

I am running a laptop with a dual core 2ghz processor with 2gb of ram and I cant manage to finish a single game using the giant earth map or Visa Mods. This really blows and I think this would and should be addressed by Firaxis as this appears to be a bug in the code itself and has nothing to do with mods but more to do with the savegame function in their code.

 

[TAfirehawk]

So, is it random based on a particular game? If I start a new game, am I likely to have it start crashing at about the same point?

It will probably crash around the same point in file size for you...but others will crash at different points with different computers.

I am running a laptop with a dual core 2ghz processor with 2gb of ram and I cant manage to finish a single game using the giant earth map or Visa Mods. This really blows and I think this would and should be addressed by Firaxis as this appears to be a bug in the code itself and has nothing to do with mods but more to do with the savegame function in their code.

It seems older computers have less trouble....strange....

EDIT
Oh and in ViSa we specifically address the changes to settings to reduce this....first is don't play giant maps or with full civs....