It's a point, I could try caching large amounts in python and see if I can go beyond 3 GB usage that way.
Have an array of arrays, and cache a 1 million big array into the first array every new turn I play should probably do it as a test.
Beside this i did some RAM testing...
1. I placed 20.000 - 30.000 units on the map, works! Memory usage only 100 MB more. 1.250 MB to 1.350 MB now (After complete game restart). If i move to all regions of the map memory usage is still under 2 GB, as all of the map is loaded with thousands of units outstanding.
2. Closed WB and got this crash: "Failed to allocate video memory..." screenshot RAMTEST1 Line:321 there is a problem but it is fixed if you reload the complete game. Look into there if something can be done there. Increase the outside video memory or something. Tells something about texture, maybe only a bugged texture...
3. After reload i did a game turn, worked! Conclusion 30.000 units only 100 MB more, so 100 civ dll should be no problem.
4. If i restart the game my memory usage is 1.250/1.350 MB also with 30.000 units on map, if i go to every region on world map its 1.900 MB (But the 30.000 units still only take 100 MB more, its the map stuff), so try to fix the unloading of the graphics paging, should clear the memory again if you go to another region, like you restart the game completely, that will save a lot of memory.
Conclusion a long game will be possible now and the CTDs are mostly by game bugs as the memory usage stays far under 2.8 GB.
Interesting if you tab out of the game in fullcreen sometimes the preloaded regions on the minimap disappear, maybe it works to tab out of game and then turn without crashing. Sometimes you need to tab out 3 times before the minimap is not blue anymore everywhere. But ram usage still higher.
Check out and do some testing:
1. Load RAMTEST1 scenario, open WB and close WB, turn = crash maybe.
2. Load RAMTESTSAVE after complete game restart and turn it, should work most times. Watch the combat to ping/view every region. Turn combat animations on. (To be able to start a turn put a production in every placed city)
Looks like someone has to fix the graphics paging to unload the loaded graphics again correctly if you look another region on the map. Game restart does this, in game changing the view to another region its not doing it.