Memory footprint

I've drilled down into this and unfortunately my conclusion is that there is not much we can do that has no other downside :(

The key scaling factor (as you'd expect) is map area. Unfortunately the game engine just isn't very scalable in this regard, and since we don't have access to its code there is not much we can do to address this.

In detail it turns out that after loading the game state the bulk of the memory allocation (and load time actually) occurs while the game engine scans the game state and sets up plot graphics. It builds (and appears to retain in memory always) tile graphics for every tile, which involve a number of layers if the tile has any features, improvements, routes, yeild values (the plot yield graphics are kept for every tile for example even if you have their display turned off). The game engine also allocates an updater object (no way to tell how large this is - all we ever see is a virtual function table) UNIQUELY for every tile that has any feature or improvement on it, and binds that instance to that plot (no sharing of the updater objects between plots of an equivalence class for instance).

The upshot of all this is substantial memory usage that is proportional to 'not entirely plane' plot count (which in turn is roughly proportional to map area, but increases somewhat as games progresses and routes/improvements and so on get laid down).

There are only two things that can be done, given the code and assets we have control over:

1) Reduce map size - that's in the hands of the player, but C2C really benefits from large maps so it rather sucks as an option

2) Reduce plot graphical detail. I believe a less detailed (i.e. - smaller texture graphic) tile set would help significantly. It really requires some experimentation to see how much this might help. Since I'm not skilled in the use of any graphical editing tools it's beyond me, but what I will do is just substitute a base BtS-simple plot texture (grassland say) for ALL the terrain types and see what the effect is on memory usage.

If the experiment on simplifying terrain graphics works out then we'll need someone who IS skilled in grahpics editing (Hydro?? SO??) to produce a less detailed (well smaller in file size/memory usage anyway, however it is achieved) terrain graphics set. I suspect they can be reduced a fair bit without major compromise to the visual impact.

Edit - ok I'm stuck. Where do I get tools that can deal with the current BtS format for PAK files? Any tutorials on how to pack and unpack things you can point me at (that are up to date, but not so up to date that they don't apply to BtS but only to Civ5)?
 
I've drilled down into this and unfortunately my conclusion is that there is not much we can do that has no other downside :(

The key scaling factor (as you'd expect) is map area. Unfortunately the game engine just isn't very scalable in this regard, and since we don't have access to its code there is not much we can do to address this.

In detail it turns out that after loading the game state the bulk of the memory allocation (and load time actually) occurs while the game engine scans the game state and sets up plot graphics. It builds (and appears to retain in memory always) tile graphics for every tile, which involve a number of layers if the tile has any features, improvements, routes, yeild values (the plot yield graphics are kept for every tile for example even if you have their display turned off). The game engine also allocates an updater object (no way to tell how large this is - all we ever see is a virtual function table) UNIQUELY for every tile that has any feature or improvement on it, and binds that instance to that plot (no sharing of the updater objects between plots of an equivalence class for instance).

The upshot of all this is substantial memory usage that is proportional to 'not entirely plane' plot count (which in turn is roughly proportional to map area, but increases somewhat as games progresses and routes/improvements and so on get laid down).

There are only two things that can be done, given the code and assets we have control over:

1) Reduce map size - that's in the hands of the player, but C2C really benefits from large maps so it rather sucks as an option

2) Reduce plot graphical detail. I believe a less detailed (i.e. - smaller texture graphic) tile set would help significantly. It really requires some experimentation to see how much this might help. Since I'm not skilled in the use of any graphical editing tools it's beyond me, but what I will do is just substitute a base BtS-simple plot texture (grassland say) for ALL the terrain types and see what the effect is on memory usage.

If the experiment on simplifying terrain graphics works out then we'll need someone who IS skilled in grahpics editing (Hydro?? SO??) to produce a less detailed (well smaller in file size/memory usage anyway, however it is achieved) terrain graphics set. I suspect they can be reduced a fair bit without major compromise to the visual impact.

Edit - ok I'm stuck. Where do I get tools that can deal with the current BtS format for PAK files? Any tutorials on how to pack and unpack things you can point me at (that are up to date, but not so up to date that they don't apply to BtS but only to Civ5)?

I believe DH has some terrain lower graphics, but i got ALL kinds of complaints on the looking like plastic grassland etc, so i changed them back, also, DH and i are having a discussion right now, about static stuff in the MAIN mod and everyone else can use the others as an option??

I just put a detailed PAK usage in the Help Wanted Ads, i believe?? If not there you can do a search on CFC and look for the one that has Kael as the author.
 
I believe DH has some terrain lower graphics, but i got ALL kinds of complaints on the looking like plastic grassland etc, so i changed them back, also, DH and i are having a discussion right now, about static stuff in the MAIN mod and everyone else can use the others as an option??

I just put a detailed PAK usage in the Help Wanted Ads, i believe??

Can you make me a private FPK that literally replaces EVERY terrain with one really plane one (literally a one-colour wash of the tile)? I know this is of NO game use at all, but it'll allow me to make best-case measurements on just how much memory could be freed up along these lines as a best case - would at least show if its worth pursuing at all...
 
Can you make me a private FPK that literally replaces EVERY terrain with one really plane one (literally a one-colour wash of the tile)? I know this is of NO game use at all, but it'll allow me to make best-case measurements on just how much memory could be freed up along these lines as a best case - would at least show if its worth pursuing at all...

I could make one that has the Vanilla terrain, best "I" can do. But it does not happen for the NEW terrain features that are included, because they are made specifically for Civ4 BtS (I believe):dunno:

Once you get the hang of PAK build, it is easy as 1,2, 3.
 
Can you make me a private FPK that literally replaces EVERY terrain with one really plane one (literally a one-colour wash of the tile)? I know this is of NO game use at all, but it'll allow me to make best-case measurements on just how much memory could be freed up along these lines as a best case - would at least show if its worth pursuing at all...

Would it work if you replaced all of the art defines in the XML/Terrain terrain infos with the same one? And have just that one replaced? Eg all sea and use the default BtS sea definition?
 
Would it work if you replaced all of the art defines in the XML/Terrain terrain infos with the same one? And have just that one replaced? Eg all sea and use the default BtS sea definition?

Yes that's what I'm trying to do, but I can't find a single ONE small one to replace them all with! The basic BtS assets don't make any sense to me - they appear to reference non-existant files.
 
In detail it turns out that after loading the game state the bulk of the memory allocation (and load time actually) occurs while the game engine scans the game state and sets up plot graphics. It builds (and appears to retain in memory always) tile graphics for every tile, which involve a number of layers if the tile has any features, improvements, routes, yeild values (the plot yield graphics are kept for every tile for example even if you have their display turned off). The game engine also allocates an updater object (no way to tell how large this is - all we ever see is a virtual function table) UNIQUELY for every tile that has any feature or improvement on it, and binds that instance to that plot (no sharing of the updater objects between plots of an equivalence class for instance).
Actually we do have control when these objects are created and destroyed. That code is in CvPlot.cpp which then calls different engine functions. Check CvPlot::setupGraphical() and all the methods it calls.

So if we want we could create and destroy the graphical representations of the symbols (features, improvements, roads, yield symbols) and the plot builder dynamically. They are currently created for all plots, regardless of if they are hidden or not.
 
Actually we do have control when these objects are created and destroyed. That code is in CvPlot.cpp which then calls different engine functions. Check CvPlot::setupGraphical() and all the methods it calls.

So if we want we could create and destroy the graphical representations of the symbols (features, improvements, roads, yield symbols) and the plot builder dynamically. They are currently created for all plots, regardless of if they are hidden or not.

I don't think we can because the game engine only calls us through setupGraphical() once per plot per turn and then assumes everything is set up for its display purposes. However, theer may be some experimenation to be done here.
 
I don't think we can because the game engine only calls us through setupGraphical() once per plot per turn and then assumes everything is set up for its display purposes. However, theer may be some experimenation to be done here.
Well, those functions work outside of that context as well. For example if you change a feature type you use those functions and the graphics are changed right away.

The Civ4 graphics engine does not care for turns much anyway.
You can even swap the entire map for a different one as shown in these experiments to investigate tactical combat possibilities: http://forums.civfanatics.com/showthread.php?t=441280&page=5
That has some drawbacks but none that would apply in our context as we don't want to swap the entire map for a different sized one. And the symbols are far easier to manipulate compared to the entities (units, cities).
 
Based on a very rough experiment (but one that should have been enough to provide measurable changes), it looks like we can't actually save that much with smaller textures, so don't worry about that approach for now.

I am experimenting with more dynamic generation of the graphical entities as AIAndy suggested currently...

Edit - I have things working so that the yield symbols are only created (the graphics for them that is) when they are displayed, and are freed once they are not displayed. This basically mean that if you don't run with 'show plot yields' turned on you save about 35M bytes on a gigantic map. However, there aren't really any saving beyond that to be had for the most important cases (i.e. - late game, when basically all plots will have been revealed). None of the other elemmts (plotBuilder, feature symbols, route symbols) can be dynamically loaded/created, because if the plot is revealed the game engine doesn't call us across a map scroll to bring it into the field of display. We **could** load things dynamically based on whether or not they are revealed, but it wouldn't help for the everything-revealed late-game case, which is actually when we need the memory the most (so IMO it's not worth it).

Still - the 35M from not pre-loading the yield symbols is something at least, so that aspect I'll be pushing to SVN after a bit more testing.
 
Following on from some experiemnts AIAndy did with city graphical entities (from a perspecitive of trying to pin down some CTDs), I have taken a look at the same area (and also unit graphics) from a perspective of memory usage.

Turns out there are some things we can do:
  1. While it's not possible to eliminate city graphical entities entirely (at least for revealed cities, which might as well be all in the problem cases, which are the late game ones), we can reduce the graphical intensity of each city. In particular, it's extremely simple to turn off unique building graphics. The result is that all cities look rather similar (bland collections of era-appropriate urban sprawl), but that has no impact on playability/functionality, and saves around 31Mbytes in my test game (which is a gigantic map, early modern, has about 500 cities)
  2. It is also possible to load unit graphical entities only when the units in question are visible (to the active human player). Furthermore we can load the entities just for the unit stack head unit (assuming no formations). This saves us another 30M. It's not more because late in the game things like espionage tend to mean you have visibility almost everywhere

I have some debugging to do before it's ready for prime time, but I plan to do the following:

  • Add a global define for city building display level (all, wonders only, none). 'All' is what we have now. 'None' is the generic no-unique building graphics look talked about above, 'wonders only' displays unique graphics for wonders but not other buildings. Default will be 'All' (so same as now)
  • Add a second global define for whether unit graphical entities are dynamically loaded or not. This one will default to dynamic loading (reduced memory usage), but I want to leave it as an option because it will have a small performance impact

Just as a note for people experiencing memory issues, some of the graphical options can also make a significant difference.

Choosing the following options knocked off another 86M:
  • Single unit graphics
  • Animations Frozen
  • Effects off
  • Low resolution textures
None of these (IMO) really have a significant (negative) visual effect.

I also tried setting 'Graphics quality' to 'Low' (which sets 'Render quality' to low also), but that only made a further 1M (so almost nothing).
 
Animations Frozen

This unfornately has a more serious sideeffect graphics wise. All units are displayed as their BtS vanilla unit they are based on. Examples are Arabian Archers as normal Archers and Iron Frigate as a normal Frigate. On a short test I've also seen crocodiles as some kind of knight. Additionally some of the new resources like pumpkins and melons are a pink blob.
 
Following on from some experiemnts AIAndy did with city graphical entities (from a perspecitive of trying to pin down some CTDs), I have taken a look at the same area (and also unit graphics) from a perspective of memory usage.

Turns out there are some things we can do:
  1. While it's not possible to eliminate city graphical entities entirely (at least for revealed cities, which might as well be all in the problem cases, which are the late game ones), we can reduce the graphical intensity of each city. In particular, it's extremely simple to turn off unique building graphics. The result is that all cities look rather similar (bland collections of era-appropriate urban sprawl), but that has no impact on playability/functionality, and saves around 31Mbytes in my test game (which is a gigantic map, early modern, has about 500 cities)
  2. It is also possible to load unit graphical entities only when the units in question are visible (to the active human player). Furthermore we can load the entities just for the unit stack head unit (assuming no formations). This saves us another 30M. It's not more because late in the game things like espionage tend to mean you have visibility almost everywhere

I have some debugging to do before it's ready for prime time, but I plan to do the following:

  • Add a global define for city building display level (all, wonders only, none). 'All' is what we have now. 'None' is the generic no-unique building graphics look talked about above, 'wonders only' displays unique graphics for wonders but not other buildings. Default will be 'All' (so same as now)
  • Add a second global define for whether unit graphical entities are dynamically loaded or not. This one will default to dynamic loading (reduced memory usage), but I want to leave it as an option because it will have a small performance impact

Just as a note for people experiencing memory issues, some of the graphical options can also make a significant difference.

Choosing the following options knocked off another 86M:
  • Single unit graphics
  • Animations Frozen
  • Effects off
  • Low resolution textures
None of these (IMO) really have a significant (negative) visual effect.

I also tried setting 'Graphics quality' to 'Low' (which sets 'Render quality' to low also), but that only made a further 1M (so almost nothing).
i know that you are just experimenting but i thing a medium map and better graphics is better that gigantic map with bad graphics
 
i know that you are just experimenting but i thing a medium map and better graphics is better that gigantic map with bad graphics

Yeh, was just pointing out the numbers, and testing in larger maps shows the more extreme cases is all. The changes I propose will work for smaller maps too, but chances are you don't have memory issues anyway.
 
Back
Top Bottom