Memory usage

[Koshling]

Memory usage is starting to become an issue again, with the recent rate of growth of the assets (they are no the dominant factor in total usage, but they represent a low tide mark which shoves everything progressively up, which is causing issues to occur earlier and on smaller maps).

As such I plan to bring forward the investigatory (and subsequent change) work on (large scale) memory usage I was planning to do in a month or so. Right now I'm a bit behind on course work (I have another course starting later next month, so I need to finish the current one pronto!), so after finishing work-work, that will have to come before C2C for a few days at least. Hence I will likely be later next week before I am able to get started on this.

 

[strategyonly]

Memory usage is starting to become an issue again, with the recent rate of growth of the assets (they are no the dominant factor in total usage, but they represent a low tide mark which shoves everything progressively up, which is causing issues to occur earlier and on smaller maps).

I have noticed this QUITE a bit lately, but are you saying the more WE are adding its messing up the memory usage?? Even if they are just empty slots being built?

IF we could get the GEM map down to the usage of turn time to less than 3/4 of what it is right now, i might even play this map for once, but being its so turn dynamic, there is no way i can play on it this current way

 

[Koshling]

I have noticed this QUITE a bit lately, but are you saying the more WE are adding its messing up the memory usage?? Even if they are just empty slots being built?

IF we could get the GEM map down to the usage of turn time to less than 3/4 of what it is right now, i might even play this map for once, but being its so turn dynamic, there is no way i can play on it this current way

Every buildings we add, every unit we add, every promotion we add, every civilization we add, every leader we add, etc., etc. adds to the static base memory footprint before you even start to consider maps etc., so yes, it is continuously going up, and lately has been doing so quite fast.

 

[Maxima7]

I have noticed this QUITE a bit lately, but are you saying the more WE are adding its messing up the memory usage?? Even if they are just empty slots being built?

IF we could get the GEM map down to the usage of turn time to less than 3/4 of what it is right now, i might even play this map for once, but being its so turn dynamic, there is no way i can play on it this current way

I have a question SO. When you're near the modern era, how much time do the turns take to process for you?

 

[Koshling]

I have a question SO. When you're near the modern era, how much time do the turns take to process for you?

For me it rises fast in the Renaissance era. When I entered it turn times were circa 10 seconds (Large map). At the start of industrial it's more like 40 seconds. Once you have visibility of most of the map (and so do the AIs) and once trade networks become global things take a lot longer.

 

[strategyonly]

I have a question SO. When you're near the modern era, how much time do the turns take to process for you?

For me it rises fast in the Renaissance era. When I entered it turn times were circa 10 seconds (Large map). At the start of industrial it's more like 40 seconds. Once you have visibility of most of the map (and so do the AIs) and once trade networks become global things take a lot longer.

Lately in the past month, the turn times have gone up

12 Civ, only a Large map:

something like 20-39ish seconds for Classical
30-40ish sec for Medieval
Indus 49-58ish seconds
Modern - over 1 1/2 minutes

For me the GEM map will need to be less than 20 seconds all the way to the Industrial era, and right now its past 30 seconds from the start(PreH), waaay to much for me, sorry.

 

[Maxima7]

Lately in the past month, the turn times have gone up

12 Civ, only a Large map:

something like 20-39ish seconds for Classical
30-40ish sec for Medieval
Indus 49-58ish seconds
Modern - over 1 1/2 minutes

For me the GEM map will need to be less than 20 seconds all the way to the Industrial era, and right now its past 30 seconds from the start(PreH), waaay to much for me, sorry.

Hmm. Can one of you provide save and tell me how long the turn takes to process? I want to see how well it scales, since I believe both of you have newer Intel CPUs. The turn times are what kills it for me which is a shame. That's why I was rooting for starting from scratch.

 

[n47]

@strategyonly, I'm in Medieval and at a turn end, I'm going to make a tea or read a book. But I have 2GB of memory, so it is disk-memory shuffling.

That's why I was rooting for starting from scratch.

I quite believe the engine should be started from a scratch.

 

[strategyonly]

@strategyonly, I'm in Medieval and at a turn end, I'm going to make a tea or read a book. But I have 2GB of memory, so it is disk-memory shuffling.


I quite believe the engine should be started from a scratch.

I sure hope you have the 3GB switch.

 

[DRJ]

Every buildings we add, every unit we add, every promotion we add, every civilization we add, every leader we add, etc., etc. adds to the static base memory footprint before you even start to consider maps etc., so yes, it is continuously going up, and lately has been doing so quite fast.

Perhaps every civ should have like a treshold of available building slots. Technologies could allow you to have more buildings. You could have to raze old stuff to make room for new buildings (like in a real life city).

As I understand a max unit treshold is already in place? Or was at elast in vanilla Civ4...

"Tresholded Buildings" could be thaught to the AI quite easy, I think (at least for real coders like yourself)

For example, if the AI faces a case of "out-of-building-slots" scenario but initially decided to build building X it will look what other building in the city brings the less revenue and will sell it. As it has a routine to figure out what building it would profit MOST by it would also have a routine to do the excact opposite. Of course the exception must be made that buildings that allow other, later buildings should be excluded from sell-off.

The main idea behind this is to

a) receive an oversight about how many civs A have how many buildings B in what stage C also related to map size D and gamespeed E
This oversight could then be visualized with graphs and be used for optimizing gamespeed, perhaps by either scaling/tweaking the code or tweaking the rules (so bottlenecks that lead to too many buildings in one era for example can be avoided)

b) in terms of gameplay I didn't like the +35% for buildings new gameoption that is recently available as I play an already very low hammer eternity deity game and waiting even longer for the starting buildings sucks. But if I had only available a certain number of building slots, that would be unlocked by tech along the way, by resources (think of stone or wood allowing more building slots in cities), or by traits (excessive leader could build 1 building more in each city or something), I would quite gladly use it.

The building slots are, of course, not the only place to scale, as the same may apply to units. What about having a unit-limit tied to city size and number of cities? (As cities represent the population the number of soldiers should realistically be tied to that population!) How the AI could understand this? Quite easy: the closer the unit count comes to the limit the more expensive the units would become. The AI "knows" that "too expansive = not good, better take other stuff in same field, if possible" and would perhaps build a balista turret in a besieged city instead of a 20th longbow.

Well this wouldn't necessarily help game performance directly as AI just would build a building instead of a unit but with the building limit in place, after all there would be some situations where it could only sell a building or produce wealth science or espionage with the hamemrs which especially in evolved modern era games might be a resolution to not suffer too immense turn times.

tl,dr;

Building slot limits + unit slot limits as possible intermediate solutions for scaling + later gameplay?

 

[Koshling]

@strategyonly, I'm in Medieval and at a turn end, I'm going to make a tea or read a book. But I have 2GB of memory, so it is disk-memory shuffling.


I quite believe the engine should be started from a scratch.

Been extensively discussed (see http://forums.civfanatics.com/showthread.php?t=480285 and also the AXXXE blog linked from that thread). Bottom line is that it would probably be 6 months to a year before we had anything significantly usable, 2+ years to get equivalence with where we are now, and we lack at least one critical skill set on the team (3D graphics programming). Furthermore, copyright would prevent the reuse of any asset graphics from the current game.

 

[Koshling]

Perhaps every civ should have like a treshold of available building slots. Technologies could allow you to have more buildings. You could have to raze old stuff to make room for new buildings (like in a real life city).

As I understand a max unit treshold is already in place? Or was at elast in vanilla Civ4...

"Tresholded Buildings" could be thaught to the AI quite easy, I think (at least for real coders like yourself)

For example, if the AI faces a case of "out-of-building-slots" scenario but initially decided to build building X it will look what other building in the city brings the less revenue and will sell it. As it has a routine to figure out what building it would profit MOST by it would also have a routine to do the excact opposite. Of course the exception must be made that buildings that allow other, later buildings should be excluded from sell-off.

The main idea behind this is to

a) receive an oversight about how many civs A have how many buildings B in what stage C also related to map size D and gamespeed E
This oversight could then be visualized with graphs and be used for optimizing gamespeed, perhaps by either scaling/tweaking the code or tweaking the rules (so bottlenecks that lead to too many buildings in one era for example can be avoided)

b) in terms of gameplay I didn't like the +35% for buildings new gameoption that is recently available as I play an already very low hammer eternity deity game and waiting even longer for the starting buildings sucks. But if I had only available a certain number of building slots, that would be unlocked by tech along the way, by resources (think of stone or wood allowing more building slots in cities), or by traits (excessive leader could build 1 building more in each city or something), I would quite gladly use it.

The building slots are, of course, not the only place to scale, as the same may apply to units. What about having a unit-limit tied to city size and number of cities? (As cities represent the population the number of soldiers should realistically be tied to that population!) How the AI could understand this? Quite easy: the closer the unit count comes to the limit the more expensive the units would become. The AI "knows" that "too expansive = not good, better take other stuff in same field, if possible" and would perhaps build a balista turret in a besieged city instead of a 20th longbow.

Well this wouldn't necessarily help game performance directly as AI just would build a building instead of a unit but with the building limit in place, after all there would be some situations where it could only sell a building or produce wealth science or espionage with the hamemrs which especially in evolved modern era games might be a resolution to not suffer too immense turn times.

tl,dr;

Building slot limits + unit slot limits as possible intermediate solutions for scaling + later gameplay?

The issue (with buildings specifically) is not INSTANCES of buildings (in cities), its the number of building DEFINITIONs

 

[DRJ]

Oh I see. So would it, as Dancing Hoskuld suggested today make sense to have something like a buildings definition convertable save system that loads the next set of buildings definition when its needed and not all at the beginning... Perhaps there could be a way that only the definitions available to the most advanced civ are loaded initially?
Or there'd be sets of definition and once the most advanced civ reaches a new set of definition is loaded when a convertable save that is created will be loaded... I really don't mind reloading my game 4 times when most advanced civ reaches new era.
You even could make the system a game option perhaps

The misunderstanding in my previous posting aside I think the idea is good even if instances are no matter because it helps to even out the game and make AI smarter.

 

[ls612]

Been extensively discussed (see http://forums.civfanatics.com/showthread.php?t=480285 and also the AXXXE blog linked from that thread). Bottom line is that it would probably be 6 months to a year before we had anything significantly usable, 2+ years to get equivalence with where we are now, and we lack at least one critical skill set on the team (3D graphics programming). Furthermore, copyright would prevent the reuse of any asset graphics from the current game.

We also lack 3d content creation skills, which is IMO a bigger concern (unless we were to make a graphics engine ourselves from scratch).

 

[Thunderbrd]

I could do content creation and teach others how. And I think Hydro has a good start on that. However, I would warn that it's extremely time consuming - we'd need a lot more help. Also, would hope that we could use something a bit improved over what we're using now for animation mechanisms. Even today's standards of game design though would make the 3d graphic design process a little frustrating because I'd always want the design to be better than it should be - the pros are masters of minimalism where polygons are concerned. I'm not as much. But I could try my hand at it anyhow.

I would hope the programs I have would interface well but usually they can, even if they require going through another program to convert the file type thereafter. What I'm NOT good at is skinning. I've never had a good program for that so I've only really used onboard texture generators on the programs I use which usually don't translate into having a skin distinct from the model and some file types don't attempt to capture that information and ones that do often tweak it unsatisfactorily.

 

[n47]

Been extensively discussed (see http://forums.civfanatics.com/showthread.php?t=480285 and also the AXXXE blog linked from that thread). Bottom line is that it would probably be 6 months to a year before we had anything significantly usable, 2+ years to get equivalence with where we are now, and we lack at least one critical skill set on the team (3D graphics programming). Furthermore, copyright would prevent the reuse of any asset graphics from the current game.

I wouldn't say graphics is such a big deal. Games like this have their point in something else. If class interface would be well thought, the graphics could be even 2D (but thinking flat 3D is better) at the beginning and swapped to (full) 3D later. I have some basic skills in OpenGL and some experience in the 1st person 3D -- quite sure 3rd person is easier as there is no need of tricks about far objects -- so can do some basic 3D engine. But for anything pro I wouldn't count.

@ls612, it is even smaller problem. In such game, objects can be very very low-poly, so there is no skills of making beautiful models needed. And not beautiful objects aren't hard to do.

 

[strategyonly]

And not beautiful objects aren't hard to do.

Yeah, where'd you get that picture of me??

 

[Koshling]

After some experimentation this weekend, I have had something of a break-through, and as a result I am about to push to SVN a version with a new graphics paging capability that greatly reduced memory usage on large maps. For all but the largest games this should typically remove the need to use viewports (GEM users should be OK without viewports, at least until much much later in the game than before - I can even load and run Talin's huge game, which is often what gets used for very large game performance testing, without viewports now).

Details on the SVN thread shortly. Note that (for now) the new mode will not be enabled by default, and will require you to modify some global defines. This will likely change (or at least move to more accessible BUG options) once it's had a bit more widespread testing, so those that are forced into viewport usage due to memory constraints are encouraged to try it and let me know how you get on.

 

[strategyonly]

For all but the largest games this should typically remove the need to use viewports (GEM users should be OK without viewports, at least until much much later in the game than before - I can even load and run Talin's huge game, which is often what gets used for very large game performance testing, without viewports now).

 

[n47]

@Koshling,

If I only will have some time to play, I'll surly try it.