Hard Crash at end of turn - 100% reproducible

[JosEPh_II]

I guess automatic cleanup of RAM after software crashed was already a thing in Windows XP if not Windows 95 or even DOS.
Or even you could misremember something from back then and automatic cleanup of RAM was part of any OS even earlier before PCs were a thing.

That is RAM being locked out from use if software crashed likely was not a problem even in 60's or 70's.

Sorry Raxxo but what Toffer says is correct. And your information or understanding is not. And what Toffer has posted is the same thing an then IBM Virtual Machine programmer told me several years ago. This person now runs ( is the Head of) the IT dept. of a Local County Hospital. Has been working in computer programming, maintenance, and building since 1997.

 

[Harrier]

References to RAM locations are explicitly tied to the processes that reference the location, when the process terminates, for whatever reason, the kernel will automatically refresh its "registry" that keeps track of RAM location references.
If a RAM location is not referenced by any running processes the memory in that location is "freed" and thus available when another process need to store something in RAM.
The RAM location is only occupied as long as a reference to it exist.

This has been the case for all normal OS's for time immemorial.

True. The problem was with the programming, at least it was in the 60/70s when I was a programmer. Inexperienced programmers would declare a variable and not assign a value (null, zero or whatever). When the program ran the variable was allocated a part of the RAM. That variable would then contain what ever was in that part of the ram when previously used. Result - program fail or if lucky crash (so you could debug it.). Of course turning the computer on and off (clearing the RAM) would solve this problem, initially only.


Real-time operating system (RTOS) are an exception to what I said here, but they are not normal OS's, no personal computers would have one of those.

Memories.

In the early 1990's - I used to train very experienced programmers how to write RTOS programs for banks , hospitals etc. At the time I worked for an American company that sold those type of machines.

 

[Toffer90]

True. The problem was with the programming, at least it was in the 60/70s when I was a programmer. Inexperienced programmers would declare a variable and not assign a value (null, zero or whatever). When the program ran the variable was allocated a part of the RAM. That variable would then contain what ever was in that part of the ram when previously used.

It's dependent on the programming language used, but what you say there is an important aspect for programmers to be aware of regardless of what language they know and use.

I have limited experience with programming, but I would think that most of the "modern" programming languages, when declaring a variable without a value, automatically set's the initial value to a default that is appropriate for the variable type in question.
Java automatically sets defaults.
Lower level programming languages, like assembler and such, does not do much, if anything, automatically. ^^
Higher level languages like python does so much automatically for you that it is hard to make major programming mistakes, e.g. it's actually impossible to declare a variable without giving it a value in python.

 

[Thunderbrd]

It's dependent on the programming language used, but what you say there is an important aspect for programmers to be aware of regardless of what language they know and use.

I have limited experience with programming, but I would think that most of the "modern" programming languages, when declaring a variable without a value, automatically set's the initial value to a default that is appropriate for the variable type in question.
Java automatically sets defaults.
Lower level programming languages, like assembler and such, does not do much, if anything, automatically. ^^
Higher level languages like python does so much automatically for you that it is hard to make major programming mistakes, e.g. it's actually impossible to declare a variable without giving it a value in python.

You can get stack overflows in C++ by not properly assigning a value to a variable. Some variables more likely to create a problem than others. Generally a NULL value is a valid default, but sometimes problems can still exist if the variable is unassigned - you can even get OOS errors from it oddly enough.

Thanks for clarifying some things on that Toffer. I'll admit, I was taught some fundamentals of computer science before Windows was a thing. Wasn't even working with DOS in those classes in High School. Apple 2e. How's that for a platform of initial knowledge huh? So what Harrier said is something I was never fully sure had been 'solved'. You explained it well how and why it was fixed but would still often be a good idea to restart the computer now and then, particularly after crashes. Been habitual ever since.

 

[raxo2222]

Sorry Raxxo but what Toffer says is correct. And your information or understanding is not. And what Toffer has posted is the same thing an then IBM Virtual Machine programmer told me several years ago. This person now runs ( is the Head of) the IT dept. of a Local County Hospital. Has been working in computer programming, maintenance, and building since 1997.

I just meant, that restarting computer more than once per day isn't really needed, if you have more than 8 GB of RAM.

I also said I wasn't sure when it stopped being real problem - I never had problems with other applications if others crashed before.
Or with memory hungry game when it crashed before.

Only problem with RAM was that sometimes games used HDD as RAM (virtual memory) - then computer could get laggy when I had 4 GB RAM.
I don't think Civilization 4 can use virtual memory though.

Modded 64 bit games can easily make computer lag even if you have 8 or 16 GB of RAM, since they aren't limited in memory consumption practically.

 

[sorcdk]

T

References to RAM locations are explicitly tied to the processes that reference the location, when the process terminates, for whatever reason, the kernel will automatically refresh its "registry" that keeps track of RAM location references.
If a RAM location is not referenced by any running processes the memory in that location is "freed" and thus available when another process need to store something in RAM.
The RAM location is only occupied as long as a reference to it exist.

This has been the case for all normal OS's for time immemorial.
Real-time operating system (RTOS) are an exception to what I said here, but they are not normal OS's, no personal computers would have one of those.

Much software have memory leaks, also some of the typical processes that are always running on many computers, so it is not that rare to see a computer chugging more and more RAM the longer it has continually been powered on.
Sometimes a computer restart is the easiest way to deal with this, but as long as one have 2-3 GB or more free RAM, there is little reason to restart the comp. before launching C2C.
Back in the days when having only 512 MB of RAM was ordinary, or even when 4 GB was all one had, it was far from strange that a computer got all laggy and problemeatic after running for 2-4 days continually, especially for those who had many different processes running in the background all the time.

OS's can have memory leaks between version x and version y, and the same is true for graphics drivers, Motherboard software, fan controlling software, mp3 players, printer related processes, P2P clients, etc. stuff that often is running in the background.
The abundance of RAM in today's computer, like 16GB, makes this problem a very rarely noticeable issue.

As far as I am aware that isn't quite how it works nowadays. Quite a long while ago OSs change to have a layer of abstraction between programs memory use and physical RAM locations, with the OS doing the translation through the page table (I might be mixing the name up). This means that a program can run as if it was the only thing running on the computer (in the memory abstraction, practice is different of cause). When a program is declared more or less dead by the OS it can the free up the memory it has allocated for the program from is memory allocation data structure (which is similar to thee structure behind malloc I C).
Whether what happens at the physical RAM is up to the OS to decide, but it is fairly common to set it to 0 on allocation (if memory serves me correctly malloc do this, I at least seem to recall someone complaining about performance problems related to this). It can therefore be quite hard to get a program to see a non-zero default value in an unassigned variable, especially on simple programs, since the easiest way would be to have the stack expand back to previously used space.
I don't see how unassigned variables would directly cause a stack overflow (insufficient room on the stack to contain something), but it it is a pointer or equivalent, then it can easily cause you to attempt to access unassigned memory (or other parts of the program), which the OS/memory system will tend to detect and crash you for.
Most memory leaks comes in one of two flavours:
1) [only without garbage collection] When a price of memory is allocated, but never deallocated. This is mostly for things placed on the heap with malloc or new.
2) When some part of the program will keep growing. This could be things like a log of everything happening (maybe so you could undo something), a cache a paths and so on.

The problem is most likely in the C++ code, and it could easily be related to synchronisation in multiplayer, where you would need to keep track of a lot of details and whether you are sure that others know them too.

 

[Toffer90]

T

As far as I am aware that isn't quite how it works nowadays. Quite a long while ago OSs change to have a layer of abstraction between programs memory use and physical RAM locations, with the OS doing the translation through the page table (I might be mixing the name up).

It still work that way with modern OS's, when a process stops, the OS will make the kernel refresh so that ram locations that were flagged as in-use will lose that flag, meaning that that RAM location is ready to be overwritten by another process.
I'm not sure how the rest of your response is related to my post that you quoted.
In that post I was only telling someone who thought it necessary to restart their computer every time C2C CTD, due to concerns about memory not cleaning up after a CTD, that such measures are not necessary.
Even if one restarts C2C the nanosecond after C2C have CTD'd the new C2C process is not counted as the same process as the previous one that is no longer running and the computer will automatically mark any memory used by the old C2C process as available for other processes even if there is a C2C process running at the time when the kernel refreshes.

You are in your post complicating my simple answer that was meant for a newbie to understand by mixing in subjects such as what may be stored in the physical RAM location after it has been freed; and the subject of "used-but-available" RAM (also known as reclaimable RAM). Words like page tables, (page file space, and physical page frames mapped to logical pages ^^) and malloc will not help in assuring a regular dude(tte) that a comp. restart after a CTD is unnecessary.
It's not my intention to offend you here, I'm just worried that you may confuse the individual that my post - that you quoted - was intended for.

 

[sorcdk]

It still work that way with modern OS's, when a process stops, the OS will make the kernel refresh so that ram locations that were flagged as in-use will lose that flag, meaning that that RAM location is ready to be overwritten by another process.
I'm not sure how the rest of your response is related to my post that you quoted.
In that post I was only telling someone who thought it necessary to restart their computer every time C2C CTD, due to concerns about memory not cleaning up after a CTD, that such measures are not necessary.
Even if one restarts C2C the nanosecond after C2C have CTD'd the new C2C process is not counted as the same process as the previous one that is no longer running and the computer will automatically mark any memory used by the old C2C process as available for other processes even if there is a C2C process running at the time when the kernel refreshes.

You are in your post complicating my simple answer that was meant for a newbie to understand by mixing in subjects such as what may be stored in the physical RAM location after it has been freed; and the subject of "used-but-available" RAM (also known as reclaimable RAM). Words like page tables, (page file space, and physical page frames mapped to logical pages ^^) and malloc will not help in assuring a regular dude(tte) that a comp. restart after a CTD is unnecessary.
It's not my intention to offend you here, I'm just worried that you may confuse the individual that my post - that you quoted - was intended for.

I might not have been sufficiencly clear, but my intent was more along the lines of saying that there is more to the story, and my audience wasnt so much the person who had the problem as it was the other modders and programmers active in the thread.
I wasnt so much trying to point out that the OS doesnt keep track of the memory used, as I was trying to point out that the memory one programs sees behaves as if it was not interfered with by other programs (there are some technical details I will gloss over here, which arent relevant in the RAM model). There is a point to be made about memory fragmentation, but that is mostly relevant if you have many things running in parallel that keep requesting new pieces of memory from the OS, so it shouldnt be a problem for a decent modern computer, unless you are doing a LOT of other stuff on the computer in the mean time.

 

[Lochlann]

I mean larger maps generally will suffer them. My wife is on a Huge map game and crashes every 2 rounds in mid-Atomic era. Let me know how far you can go before you start experiencing them, if ever, on your size. I'm looking for a memory leak that should exist somewhere.

Sorry to butt in, but should this be interpreted as you (Thunderbrd) looking for large/complex games that are starting to run into these MAF errors? If so, I'd like to volunteer mine. It's a Gigantic/Eternal game, and I'm starting to encounter MAFs more than once per turn. (I'm midway through the Renaissance era, some 4300 turns in.) I've been fine reloading the game between turns--that has been my standard procedure for a while now, and it pretty reliably allowed me to play unhindered one turn at a time, which I didn't mind--but having to do so in the middle of turns is tiring, given the inexact timing (and given the pesky bug where units loaded on ships lose their movement upon a mid-turn reload). I've lavished a lot of love and time on this game, and it'd pain me to abandon it due to needing to reload multiple times per turn. I assume it will only get worse.

Anyway, if your post above was an open call for such games, let me know and I'll upload a save. Thanks.

 

[Thunderbrd]

Sorry to butt in, but should this be interpreted as you (Thunderbrd) looking for large/complex games that are starting to run into these MAF errors?

No. I'm completely surrendered to this as a reality that cannot be fixed for now. The correction is not to be found in individual games but in ways we need to go through and make the overall code a lot more memory efficient. There's a LOT of work that can and needs to be done down that road and NONE of it will make any visible difference except to extend the length of games. Furthermore, there are major AI issues that could be addressed that would further help the situation. If we can get the AI to limit its unit counts to more strategic precision and have no over-training protocols, and that's particularly true for the NPC (barbarian) player AIs, then we may be able to keep things sane enough to enable a lot more late game experiences.

At the moment, there's simply a variable limit to how far a game can go and that limit is farther on smaller maps and fewer players in the game. Space maps often go the distance because the starting Earth is tiny by comparison to a full Earth planetary map on Large or larger size - and most of those maps are inaccessible entirely for the AI player(s).

I assume it will only get worse.

Unfortunately I can only agree. And the likelihood is that by the time things are significantly improved, you will either have fallen entirely out of familiarity with that game and/or so much will have changed that the game may not even be compatible any longer. I'm sorry to offer this prognosis but the 'fix' is a conglomeration of hundreds or thousands of smaller fixes, some of which may be quite code-shaking to the point of incompatibility forcing.

 

[Lochlann]

Thanks for the frank assessment. I fully understand; I had done a bit of research before I launched the game, back on Christmas Day, and I figured this would be the inevitable result. Nothing but respect on my end for the tremendous efforts you and the rest of the team have put forth in creating the mod.

Could you perhaps shed a little light on the macro-level processes that are causing this to happen? For example, is it the growth of my own empire that is causing this--proliferation of cities, or of units (or of active units), etc.--or the growth of AI civs, or both, or something else? Might there be something I can control on my end to counteract some of this? I'm not above going on a killing spree and wiping out a continent or two if it would ensure greater longevity, or paring down my military forces, etc.

 

[Thunderbrd]

Could you perhaps shed a little light on the macro-level processes that are causing this to happen? For example, is it the growth of my own empire that is causing this--proliferation of cities, or of units (or of active units), etc.--or the growth of AI civs, or both, or something else? Might there be something I can control on my end to counteract some of this? I'm not above going on a killing spree and wiping out a continent or two if it would ensure greater longevity, or paring down my military forces, etc.

The conglomeration of all of the above. Particularly on the current codeset I believe the AI overproduces units. The volume of active units in the game (and somewhat inefficient memory allocations to each one) would be the primary culprit for running out of memory at this time. These can be your units, their units, just however you get a ton of active units on the map.

 

[Lochlann]

The conglomeration of all of the above. Particularly on the current codeset I believe the AI overproduces units. The volume of active units in the game (and somewhat inefficient memory allocations to each one) would be the primary culprit for running out of memory at this time. These can be your units, their units, just however you get a ton of active units on the map.

Aha. That is good to know. Time to thin the herd. Where's Thanos when you need him...

Thanks for your time and help. I'll release this hijacked thread now.

 

[Thunderbrd]

Only now do we understand why Thanos thought we should all appreciate him for his actions...

 

[Lochlann]

Only now do we understand why Thanos thought we should all appreciate him for his actions...

The beauty of the multiverse is that somewhere, in some universe, that's exactly what happened. Also, the Snap would make for a helluva random event/quest...

 

[Thunderbrd]

I've often thought we should do a Marvel MODMOD for C2C - would definitely include that now.

 

[raxo2222]

I've often thought we should do a Marvel MODMOD for C2C - would definitely include that now.

It sounds like event idea. In last era MCU and other multiverses should be easy to recreate because cultural victory is as important as scientific victory

Spoiler contains Endgame spoiler :

Is civilization in Prehistoric era, billions light years away from actual Marvel action.
Get snapped in half (pop shrinks by half).
Turn later population doubles again.
Myths and religions are influenced by this mysterious event as proof that god or extremely advanced aliens are real.

Is civilization in Transcedent era, its capital is trillions light years from Marvel mess.
Sees as snap shockwave is rushing trough hyperspace and behind it goes second shockwave to undo snap.
Teleports to source of those shackwave and undoes all those Marvel actions and takes back all those stones.
Shouldn't give toys to barbarians from Galactic era, who gave toys to apes from Information era.

Is civilization in Atomic/Information era millions years away from Marvel mayahem.
Gets snapped in half. In the end loses 70% of population before stabilizing.
Snapped half comes back. This time loses 85% of total population before stabilizing.
Barely manages to maintain infrastructure and tech level - didn't collapse back to Medieval or earlier era tech level like other 95% Atomic and Information era worlds.
Everyone is like: God or Aliens are real - that mess came from outer space. We need to explore and colonize space and hope we won't get dusted again.

In summary civs closer to Atomic/Information era are easier to break.
Primitive civs will just slowly grow back.
Advanced civs have technology to hold them together.

Looks like Thanos hit hardest least sustainable civs after all

 

[Lochlann]

That'd be fantastic. You've already got the heroes mechanic in play; civs could have to join forces and trade culture resources to assemble (er, Assemble) a team in order to trigger the second Snap or something. Or make the Avengers national heroes and have civs race to trigger it. ...if they want to!; you could go all meta and have the Snap triggered by the computer's detection of memloss and leave it up to the player(s) to determine if they actually want to do anything about it.

 

[tmv]

I think the fighting abilities would be the smallest problem (although accurately portraying someone like the Hulk would be interesting). Some of those heroes (cough ... Iron Man) carry a large part of their nation's research pretty much on their own, and I don't think there is any mechanic for units producing any kind of commerce, unless it is by self destruction.

Then there is vibranium - the way it is portrayed it should be a map resource or depend on a Natural Wonder (otherwise it would be much harder to limit access to it).

Then we would need something like worldviews or perhaps civics to portray the way the people think of the superheroes, perhaps an event (Sokovia Accord) to influence that.

The most interesting part might be the "Guardians of the Galaxy" part of the map.

 

[Thunderbrd]

The trick would be to let the characters come into play in a way that lets the rest of the game become mostly a backdrop to their story.