Taurus 1.02

[alberts8]

Players who don't do so will no longer be burdened with the wasteful memory use of SmartMap and the visual clutter of scripts they never use.

The reason why SmartMap never releases the memory is by design in Python 2.4 and that memory issue also affects every other line of Python code in Civ4.

Until version 2.5 Python's internal small-object allocator never released memory from it's pool of memory arenas.

This was changed from Python 2.5 onwards and it would be possible to port that change back to Python 2.4.

- Patch #1123430: Python's small-object allocator now returns an arena to
the system ``free()`` when all memory within an arena becomes unused
again. Prior to Python 2.5, arenas (256KB chunks of memory) were never
freed. Some applications will see a drop in virtual memory size now,
especially long-running applications that, from time to time, temporarily
use a large number of small objects. Note that when Python returns an
arena to the platform C's ``free()``, there's no guarantee that the
platform C will in turn return that memory to the operating system. The
effect of the patch is to stop making that impossible, and in tests it
appears to be effective at least on Microsoft C and gcc-based systems.
Thanks to Evan Jones for hard work and patience.

Merge the tim-obmalloc branch to the trunk. · python/cpython@cf79aac

This is a heavily altered derivative of SF patch 1123430, Evan Jones's heroic effort to make obmalloc return unused arenas to the system free(), with some heuristic strategies to make it more l...

github.com

 

[f1rpo]

Thanks for this background info. I take it that would mean recompiling python24.dll – which gets loaded from the BtS install directory, so it's not something a mod could fix. Without really having looked into the implementation of SmartMap, I would assume that there is some way to delay the bulk of its memory allocations until the map becomes selected. My understanding (from Toffer's C2C Git commit, which made me aware of the problem in the first place, and iirc also a quick test of my own) is that, currently, all the memory gets allocated upon just opening one of the game setup screens.

 

[alberts8]

My understanding (from Toffer's C2C Git commit, which made me aware of the problem in the first place, and iirc also a quick test of my own) is that, currently, all the memory gets allocated upon just opening one of the game setup screens.

SmartMap does some processing to setup some of it's options. It's probably necessary to do that once the setup screen gets opened the first time. At least that's what i recall but maybe there's room for improvement.

 

[f1rpo]

There is a function firstTimeInit. That's actually only called once the script is selected. The 100 MB get allocated already when the module is loaded, which happens upon selecting Single Player on the opening menu, i.e. even before the game setup screen. There's an enormous global array "blueMarbleHeightData". Well, it's perhaps 300k integers. That should result in maybe a few MB? But, apparently, it's responsible for the full 100. Turning it into a local variable is no help. The data can be moved into a separate (.csv) file that only gets parsed in generatePlotTypes. However, I'm not sure if a map script can easily locate a file in its own folder (i.e. in PrivateMaps or PublicMaps). __file__ seems to point to the BtS "Assets" directory. The data also has a great deal of redundancy. And I haven't yet figured out precisely what it gets used for. With the default options, it doesn't seem to be used at all.

I also noticed a bug that causes the script's config file to be created directly under My Games: fileName = civFilePath() + "smartmap90.cfg"
Lack of a path separator. Fixing that moves the config file into My Games\Sid Meier's Civilization 4 rather than into the corresponding BtS directory. Apparently the registry keys that the script consults are outdated. Another hint for me that I probably shouldn't try to mess with this monstrous script given my inexpertise in Python.

 

[f1rpo]

I've pulled myself together a little and fixed the early memory allocation by replacing those big int arrays (it's actually two 300x600 arrays) with strings. Probably, just putting quotation marks around the array data (and later parsing that of course) would've been enough, but I've also applied a primitive RLE compression – as Python 2.4 apparently has no string compression onboard. The big arrays using the small-object allocator, if that's indeed what happens, actually seems strange. Well, preventing allocation so long as the script isn't used is an improvement in any case. As far as I understand it now, that data encodes plot types and terrain types for Earth maps, a bit like a WB scenario inside a map script, but with flexible map dimensions.

Also managed to improve the location of the file where SmartMap stores the most recently used settings. At least on my system, My Games\Beyond the Sword is now successfully located.

Git commits