C2C - UEM - Ultimate-Earth-Map 100% MOD and SVN update compatible by Pit2015

[Pit2015]

I bet it is failing of something else, there is enouth memory, game crashes sometimes if it cant allocate the memory correctly.

 

[Somebody613]

I bet it is failing of something else, there is enouth memory, game crashes sometimes if it cant allocate the memory correctly.

More civs => more cities and more units => more memory consumption => inevitable crashes.
As of right NOW, though.

 

[Pit2015]

Indeed, it is still likely to be an exe limitation, where e.g. a 1024 KB memory buffer is exceeded for light effects or something along that line. These smaller RAM buffers that can be exceeded are mostly handled by the exe as they are typically related to sound and graphics. If MAF is caused by the dll then it will most likely be a MAF where one runs out of RAM (~2.8 GB) because otherwise the exe wouldn't post a MAF message, but the game would CTD in the dll due to unexpected behavior from the bad memory handling there and would not post any message at all unless attached to VS or something.

In late game maybe, but the game MAFs also if there is enough RAM left. So with 100 civs you will only start with 100kb more RAM usage or so. You can see that by causing the plot error in WB then the game will MAF but it has many RAM left.

So the future is to save more stuff to the HDD, not to memory. That will give a lot more free memory and the usage will be very low.

Same like it is done now with the graphics paging, only load the units to the memory in the region where you are currently viewing, maybe that can be done.

@Toffer90 Also if the exe still CTD at 2.8 GB read this, looks like the dll is not LAA flagged correctly then? Maybe that can be improved, read FAQs on this page:

2-4GB game memory limits and solutions - Nexus Mods Wiki

wiki.tesnexus.com

FAQs​

"Do you need to LAA patch DLLs? The answer is a qualified 'yes'. See the Wikipedia articles Dynamic-link library and Virtual Memory for the full background, but the relevant point is this:

"With the introduction of 32-bit libraries in Windows 95 every [DLL] process runs in its own address space." [This would be physical address space. As far as the application is concerned, the DLL is loaded into it's virtual address space.]

DLLs share the same file format as an EXE. As with EXEs, DLLs can contain code, data, and resources..."

Looks to me if it still CTD at 2.8 GB that it is not set correctly to use the full 4 GB currently. Maybe because the dll is not LAA patched.

Also maybe you want to test that tool on the exe if it will help to free full 4 GB:

4GB Patch – NTCore

ntcore.com

I want to see if it will CTD later with my 24 GB graphics card now. But hard to go there with the memory usage updates now.

Next possibility - Read here: https://accu.org/journals/overload/21/113/oldwood_1875/

(Maybe that can be adapted to the mod code)​

Page-files​

The hard disk can also act as a temporary store for memory pages that are not currently in use. This is called virtual memory because it can only be used for page storage – the pages still have to be present in physical memory to be accessed.
This total size of all paging files defines the virtual memory limit for the entire machine; this can be smaller or larger than the per-process 4GB limit.

Page-file backed shared memory​

There is another way to access all that extra memory using the existing Windows APIs in a manner similar to the AWE mechanism, but without many of its limitations: Shared Memory. Apart from not needing any extra privileges the memory allocated can also be paged which is useful for overcoming transient spikes or exploiting the paging algorithm already provided by the OS.

Allocating shared memory under Windows is the job of the same API used for Memory Mapped Files. In essence what you are mapping is a portion of a file, though not an application defined file but a part of the system’s page-file. This is achieved by passing INVALID_HANDLE_VALUE instead of a real file handle to CreateFileMapping() . Listing 1 creates a shared segment of 1MB.

const size_t size = 1024 * 1024; HANDLE mapping = CreateFileMapping(INVALID_HANDLE_VALUE, NULL, PAGE_READWRITE, 0u, size, NULL); if (mapping == NULL) throw std::runtime_error ("Failed to create segment");

 

[Pit2015]

Hmm... yes if the game still crash at 2.8 GB then there is something wrong, a thought... is the mod overwriting the steam downloaded exe? Otherwise a LAA falgged civ4 bts exe has to be provided to be put in manually.

Or the DLLs are not LAA patched.

Someone check that, maybe its as simple as this to get more free memory.

 

[tmv]

The exe is LAA.

 

[raxo2222]

Hmm... yes if the game still crash at 2.8 GB then there is something wrong, a thought... is the mod overwriting the steam downloaded exe? Otherwise a LAA falgged civ4 bts exe has to be provided to be put in manually.

Or the DLLs are not LAA patched.

Someone check that, maybe its as simple as this to get more free memory.

Looks like 1024 MB is reserved for VRAM
That is 3 GB of RAM + 1 GB of VRAM - it was done on old engine with limitations.

 

[Thunderbrd]

I bet it is failing of something else, there is enouth memory, game crashes sometimes if it cant allocate the memory correctly.

You do realize our BIGGEST challenge in this entire mod is trying to keep it within memory limits to reach the end of the game right? That and speed. And then wanting to do so much more still with rules and combat depth and trade and supply issues and... and... and... There's no way I'm putting 100 civs as a priority over other projects that would make things so much more cool once we get it in place.

However... there have been some truly genius suggestions of late as to how we might be able to get rid of much of our concerns about memory.

 

[Pit2015]

You do realize our BIGGEST challenge in this entire mod is trying to keep it within memory limits to reach the end of the game right? That and speed. And then wanting to do so much more still with rules and combat depth and trade and supply issues and... and... and... There's no way I'm putting 100 civs as a priority over other projects that would make things so much more cool once we get it in place.

However... there have been some truly genius suggestions of late as to how we might be able to get rid of much of our concerns about memory.

Its C2C! That mod goes where no one has gone before. Also 100 civs are good as memory test, recommend 40 civs but 100 civs can be tested/tried.

Really try to flag the DLLs to LAA, maybe it will free even more memory. "However... there have been some truly genius suggestions of late as to how we might be able to get rid of much of our concerns about memory." Good to hear.

 

[Toffer90]

Its C2C! That mod goes where no one has gone before. Also 100 civs are good as memory test, recommend 40 civs but 100 civs can be tested/tried.

Multiple mods have gone there before, 100 civ dll is a very old thing in the Civ4 community.

 

[Pit2015]

Multiple mods have gone there before, 100 civ dll is a very old thing in the Civ4 community.

But not working or? Ok then C2C should go 192/195 all world civs. I meant it in general, but no one got it to work on the C2C/UEM scale. Is it much work or only increase the civ slot numbers to increase the usable civs?

Btw, maybe the civ mega pack can be integrated into C2C, so more choice.

 

[Thunderbrd]

Multiple mods have gone there before, 100 civ dll is a very old thing in the Civ4 community.

And we had that until we realized we needed to step it back until we could re-introduce it later, if we could ever justify it again.

 

[Thunderbrd]

Btw, maybe the civ mega pack can be integrated into C2C, so more choice.

I'd rather we wait until we have designed out a much larger selection of cultures and moved to a more civic selection basis of what civilization, among all your possessed cultures, that you as a country identify as. That may only be fulfilled by expanding out the massive civ pack perhaps but we'll see... might be easier as a breakdown of fundamental parts of what a civ is defined as and the ability to mix and blend those things more with a 'cheaper' shell culture adoption system.

 

[Toffer90]

And we had that until we realized we needed to step it back until we could re-introduce it later, if we could ever justify it again.

Yeah, I thought I remembered C2C had a 100 civ dll back sometime before I joined the team (like 8 years ago or something).

 

[Thunderbrd]

Yeah, I thought I remembered C2C had a 100 civ dll back sometime before I joined the team (like 8 years ago or something).

To think there was ever a time before... lol So glad you did!

 

[Somebody613]

I'm still conflicted between some very much opposite directions I want C2C to go.

1. Generic regional civs (Native-type, literally) growing and expanding in-game to become more specific and more localized civs.
Basically, you don't START as "100 civs" - you UNLOCK them in the game itself.
Kinda making "vassals" but in reverse - splitting them off the main civ in the very process of the game play.
I think that there even IS a similar method of "colonization" already existing in C2C, but I never got a chance to test that one.
Like, you start as "Asian", and then you unlock "Chinese" by fulfilling specific RFC-ish CONDITIONS for "Chinese civ", creating it similarly to how we now get new civs from Barbs.
So it's not that we start with EXISTING 100/200 civs (whether on-map or not), but we get them by "creating" them under specific conditions.
In practice, this could even be still limited to the current 40 civs (though I dislike that), but you could "give birth" to civs you WANT to be there, not just see them appear RANDOMLY.
Ties well with the multitude of existing C2C cultures, which could be expanded even further.
Basically, this option isn't even "increase the number of TOTAL on-map civs", but instead a method to rework the BIRTH of civs in-game.

2. Straightforward literal increase of the number of allowed civs.
This causes memory and turn speed issues, but also allows for Real Earth Full World scenarios that have their own "sentimental" value for myself, lol.
Unlike the first option, this shouldn't be done right now, because of the memory/speed issues killing all the joy in any bigger game.
But the idea is still there, anyways.

3. Unrelated to the other two, mostly.
C2C-RFC hybrid, once again literally.
No change in civ numbers, no big changes in civ mechanics (maybe, maybe not), but a huge focus on the Earth Map itself.
C2C is so much broader than Vanilla for RFC-type purposes, we could have dozens of content more compared to the original RFC.
Tons and tons of new features, even more tons of new civs, all those cultures, and so on - it's literally a gold mine for RFC-ing C2C.
But it does require a strict Earth Map focus, and I see that not all C2C modders would approve that, sadly for me.
Ah, well.

 

[Pit2015]

100 civ dll is justified now as memory usage is improved or later if it improves further, 40 civs can stay recommended but more are possible to use on own risk, its only not prioritized currently.

 

[Thunderbrd]

100 civ dll is justified now as memory usage is improved or later if it improves further, 40 civs can stay recommended but more are possible to use on own risk, its only not prioritized currently.

Ah, see there's the problem with it. Much of the memory usage is allocated when you open it up that far even if you aren't using all those player slots in your game. So its limiting even for those who choose more wisely.

And yes, we've relaxed, with great efforts, our memory limits, and work continues to do so, but we didn't do it to just recreate the problem in this direction - there are other more pressing projects to attend to that will need that memory.

 

[Pit2015]

Yeah someone has to come up with cool code to save 90% to the HDD. 2023 soon and no solution for 32bit games? Or maybe crush that VRAM somehow or use it more.

Someone try to flag the DLLs LAA, and let me know if it conserves more memory or shout out and i test it, put it into latest svn, nothing to loos. But report here i want to know. On my other mod i managed to conserve about 600kb.

 

[Toffer90]

Someone try to flag the DLLs LAA, and let me know if it conserves more memory or shout out and i test it, put it into latest svn, nothing to loos. But report here i want to know. On my other mod i managed to conserve about 600kb.

Our dll is compiled with the large address aware flag set.
The BtS process can use more than 2 GB of RAM which proves that it is large address aware as 32 bit programs can only use 2 GB of RAM without being large address aware.
The exe decides how to partition/allocate the 4GB of RAM it has available, and it has chosen to reserve 1.2 GB for other things (graphics, audio, misc), and has thus only allocated 2.8 GB to the actual game code.
It's an exe thing, there is nothing we can do, so please stop beating this dead horse.
All we can do is to find ways for the BtS process to use less RAM, and that is something we are constantly working on one way or another when we see ways to do so.

 

[Pit2015]

Dll's don't have LAA flag, it's an exe thing. Also, the BtS process can use more than 2 GB of RAM which proves that it is large address aware as 32 bit programs can only use 2 GB of RAM without being large address aware.
The exe decides how to partition/allocate the 4GB of RAM it has available, and it has chosen to reserve 1.2 GB for other things (graphics, audio, misc), and has thus only allocated 2.8 GB to the actual game code.
It's an exe thing, there is nothing we can do, so please stop beating this dead horse.
All we can do is to find ways for the BtS process to use less RAM, and that is something we are constantly working on one way or another when we see ways to do so.

Read this under FAQs: DLLs should be able to LAA flagged, try to patch them LAA.

2-4GB game memory limits and solutions - Nexus Mods Wiki

wiki.tesnexus.com

FAQs​

"Do you need to LAA patch DLLs? The answer is a qualified 'yes'. See the Wikipedia articles Dynamic-link library and Virtual Memory for the full background, but the relevant point is this:

"With the introduction of 32-bit libraries in Windows 95 every [DLL] process runs in its own address space." [This would be physical address space. As far as the application is concerned, the DLL is loaded into it's virtual address space.]

DLLs share the same file format as an EXE. As with EXEs, DLLs can contain code, data, and resources, in any combination, within their own address space. Conceptually in abstract they are the same type of file: separate but cooperating programs or applications ('apps'). The LAA flag is a 'Large Address AWARE' flag (my emphasis). It enables a DLL (which is loaded as needed by the calling program, and thus relatively unpredictable by the player) to load into physical memory locations greater than the 4 GB virtual space addressable by the 32-bit game. Without the flag, they will only be loaded into the same 4 GB space as the game. The more code you can get to utilize the higher memory above 4 GB, the more room for the code (and data) which can't (not LAA enabled). It's about memory management in loading code, which reduces the chances of exceeding available memory.

LAA flagging does not prevent a program from gobbling up all available memory until it crashes, nor is it a magic bullet that makes the code 'better'. There is no guarantee that a DLL will perform correctly when loaded to upper memory if it was not LAA flagged by the developer. But it should manifest problems quickly in that case if the registry test setting suggested above is used, and has proven to successfully improve the amount of available memory for some games (i.e. The Elder Scrolls: Oblivion).

Two Considerations​

1. As pointed out above: in a game that is written for a 32-bit environment, 'virtual memory' addressing only goes up to 4 GB. If the game does not handle 'out of memory' conditions well, you get CTDs when it attempts to write data beyond the bounds of it's allocated memory. This is the OS protecting itself from 'rogue apps'. While each 32-bit app gets it's own 'allocated' memory space, they all must share the lower 4 GB of memory because they can't address memory above 4 GB. The more loaded in the lower 4 GB, the less available memory is allocated to each process. All have to fit into the same 4 GB 'bag'.

2. In order to get the maximum use out of that lower 4 GB of memory for a 32-bit game, you need to reduce the 'reserved footprint' of the OS and anything else occupying that space as much as possible. That means loading everything you can above 4 GB with a 64-bit OS, and shutting down any non-essential programs and services with a 32-bit OS. Programs like ioBit's (now Razor's) GameBooster help you determine and decide which apps and services are 'non-essential' as well as provide a simple mechanism to stop and start them. How well you manage this determines how close to the 4 GB game virtual memory limit you can get, even on a 64-bit system. If DLLs are not flagged as LAA, then they won't get loaded above 4GB. They still might not, even with the flag. The only way you can tell is to track each process and find the starting address, but what can you do if it isn't loading high? (Nothing: it's out of your control.) But if the LAA flag is not set, it's guaranteed the EXE or DLL will be loaded into the lower 4 GB of memory and reduce the available memory.

So: flagging a DLL as LAA enables it to be loaded above the 4 GB boundary, and thus make more space available to your game. But it is not absolutely essential, of only minor effect with a 32-bit OS, and any particular DLL might not have any practical effect. But having a chance to reduce your lower memory footprint is commonly regarded as preferable to having no chance. Whether these efforts are worth it to you is a personal decision."

They tell crap or is it possible? So as i understand the DLLs currently go into the 2.8 GB memory the exe is using now, if you flag the DLLs LAA then it will not go into that 4 GB range and free more memory.

But dont ask me how to patch DLLs LAA, you have to research or to find out. But should be the same like for exe files if they similar.

May not work, but is worth a try. If it saves 50-100 kb is better than nothing.

Btw my current memory usage is 1.250 MB at UEM start what is pretty cool, but nothing is impossible in this universe.