The reason why SmartMap never releases the memory is by design in Python 2.4 and that memory issue also affects every other line of Python code in Civ4.
Until version 2.5 Python's internal small-object allocator never released memory from it's pool of memory arenas.
This was changed from Python 2.5 onwards and it would be possible to port that change back to Python 2.4.
- Patch #1123430: Python's small-object allocator now returns an arena to
the system ``free()`` when all memory within an arena becomes unused
again. Prior to Python 2.5, arenas (256KB chunks of memory) were never
freed. Some applications will see a drop in virtual memory size now,
especially long-running applications that, from time to time, temporarily
use a large number of small objects. Note that when Python returns an
arena to the platform C's ``free()``, there's no guarantee that the
platform C will in turn return that memory to the operating system. The
effect of the patch is to stop making that impossible, and in tests it
appears to be effective at least on Microsoft C and gcc-based systems.
Thanks to Evan Jones for hard work and patience.
This is a heavily altered derivative of SF patch 1123430, Evan Jones's heroic effort to make obmalloc return unused arenas to the system free(), with some heuristic strategies to make it more l...
Thanks for this background info. I take it that would mean recompiling python24.dll – which gets loaded from the BtS install directory, so it's not something a mod could fix. Without really having looked into the implementation of SmartMap, I would assume that there is some way to delay the bulk of its memory allocations until the map becomes selected. My understanding (from Toffer's C2C Git commit, which made me aware of the problem in the first place, and iirc also a quick test of my own) is that, currently, all the memory gets allocated upon just opening one of the game setup screens.
My understanding (from Toffer's C2C Git commit, which made me aware of the problem in the first place, and iirc also a quick test of my own) is that, currently, all the memory gets allocated upon just opening one of the game setup screens.
SmartMap does some processing to setup some of it's options. It's probably necessary to do that once the setup screen gets opened the first time. At least that's what i recall but maybe there's room for improvement.
There is a function firstTimeInit. That's actually only called once the script is selected. The 100 MB get allocated already when the module is loaded, which happens upon selecting Single Player on the opening menu, i.e. even before the game setup screen. There's an enormous global array "blueMarbleHeightData". Well, it's perhaps 300k integers. That should result in maybe a few MB? But, apparently, it's responsible for the full 100. Turning it into a local variable is no help. The data can be moved into a separate (.csv) file that only gets parsed in generatePlotTypes. However, I'm not sure if a map script can easily locate a file in its own folder (i.e. in PrivateMaps or PublicMaps). __file__ seems to point to the BtS "Assets" directory. The data also has a great deal of redundancy. And I haven't yet figured out precisely what it gets used for. With the default options, it doesn't seem to be used at all.
I also noticed a bug that causes the script's config file to be created directly under My Games: fileName = civFilePath() + "smartmap90.cfg"
Lack of a path separator. Fixing that moves the config file into My Games\Sid Meier's Civilization 4 rather than into the corresponding BtS directory. Apparently the registry keys that the script consults are outdated. Another hint for me that I probably shouldn't try to mess with this monstrous script given my inexpertise in Python.
I've pulled myself together a little and fixed the early memory allocation by replacing those big int arrays (it's actually two 300x600 arrays) with strings. Probably, just putting quotation marks around the array data (and later parsing that of course) would've been enough, but I've also applied a primitive RLE compression – as Python 2.4 apparently has no string compression onboard. The big arrays using the small-object allocator, if that's indeed what happens, actually seems strange. Well, preventing allocation so long as the script isn't used is an improvement in any case. As far as I understand it now, that data encodes plot types and terrain types for Earth maps, a bit like a WB scenario inside a map script, but with flexible map dimensions.
Also managed to improve the location of the file where SmartMap stores the most recently used settings. At least on my system, My Games\Beyond the Sword is now successfully located.
Arboria: yeah, both selections can affect it. high vs low sealevel changes the landcount from ~1700 to ~2300 and rocky climate noticeably increases peak count.
Tectonics: having to change the player count isn't an effective solution when the dll playercount is still 18, and it also makes random selection worthless. I play huge marathon maps with 22+ players at 3k+ land count and the Mediterranean setting in this map is over 5k. I think it also makes no sense in general for earth maps to be smaller than regional maps and firaxis did increase the grid size for their Terra map. But now I've started to think tectonics is just a bad map script. The plate novelty is about all it has going for it and it has poor climate generation and it usually bugs around the poles.
City bar changes: very opposed to. Noticed them immediately, tried playing with them; looked in the commit log for how to reverse them (to the old even smaller settings) and immediately was more comfortable. I've never understood why Bug pushed the wide city bars as default, but at least they were disableable. Now that the resource bubbles are shrinkable, (I set them to 25) I don't understand why I would want to have tiny resources bubbles yet be forced to have giant growing bars littering the screen.
Should I expect this to happen frequently if I just put myself in slot 0 and an AI teammate in slot 1? Or does it hinge on special settings/ circumstances?
Arboria: yeah, both selections can affect it. high vs low sealevel changes the landcount from ~1700 to ~2300 and rocky climate noticeably increases peak count.
So all maps that use HintedWorld or directly use FractalWorld and disable the sea level setting probably have this problem. There's also MultilayeredFractal for generating continents, which leaves the sea level up to the map script. The map script failing to do so is the problem with Big and Small. As for Arboria, I guess it'll have to subtract the sea level change upfront in order to counter the addition in FractalWorld. And will then also have to repreat the threshold-192 thing (that Firaxis themselves didn't understand anymore, judging by the comment) because the water_percent argument will no longer be be at its default value of -1. A little convoluted, but should do the trick, i.e. fix Arboria just through the script file. Seems that a map script can't just set the sea level to Medium (while keeping the menu hidden). (Edit: Well, Python's reflection capabilities do allow this to an extent.) The DLL could allow map scripts to do that, but then the script would no longer be portable to other mods.
Team_Battleground seems to have largely the same problem; also uses HintedWorld. (Typing this as I go through the scripts, also for my own later reference.) Boreal and Rainforest use FractalWorld, but fully override generatePlotTypes. So that looks good, - but the overridden methods do apply the sea level. So that should either be removed or the sea level menu should be unlocked. At least easy to fix within the map script files.
Donut, Highlands, Mirror, Maze and Great Plains also handle the plot type generator on their own, and they ignore the sea level as they should. Oasis is based on MultilayeredFractal and does not add any sea level impact.
As for climate, all the plot type generators in CvMapGeneratorUtil.py take into account the climate for Hills and Peak generation, and the TerrainGenerator and FeatureGenerator in that file also account for the climate. So any map script disabling the climate menu is dubious.
Arboria apparently handles terrain and features on its own, so that part is fine, but the plot generator is a problem. Which aligns with your observation of Peaks being affected by climate. I think the map script can overwrite the climate-adjusted variables of FractalWorld. So this should be fixable too. Ice Age has a similar problem, and also doesn't override climate-based Jungle placement. Boreal, Rainforest, Highlands, Great Plains, Oasis and Fantasy Realm seem to do plots, terrain and features on their own; fine. Donut too, but it does use the climate for Jungle placement, so that needs to be corrected.
Tectonics doesn't use a stock plot generator for any of its options, but it does use the climate-based feature generator from CvMapGeneratorUtil.py. Confusingly, the addFeatures function is defined twice, meaning that the first definition, with a "no-Ice" generator for the Mediterranean setting, has no effect, and the second definition counts, which uses custom latitudes for Mediterranean. I suppose that will also result in no Ice, so probably working as intended and the first definition should just be removed for clarity. Probably a remnant from an old version (specifically v3.6).
Tectonics: having to change the player count isn't an effective solution when the dll playercount is still 18, and it also makes random selection worthless. [...] But now I've started to think tectonics is just a bad map script. The plate novelty is about all it has going for it and it has poor climate generation and it usually bugs around the poles.
Tectonics has quite varied options, maybe some that no other map script does as well/ at all, e.g. Mediterranean. The geo-simulation idea ought to be much better executed by the PerfectWorld family. Well, the precipitation patterns at least; the plate tectonics aspect is not all that complex in PW iirc. (On a side note, I've recently searched for planet generator apps outside the context of Civ and wasn't able to find something similar – and more sophisticated – than PW. The scientific models all seem to rely heavily on real-Earth data and will essentially only reproduce our planet's actual history rather than allowing a random topology.) Getting all Tectonics options to work well seems like a big project, and, therefore, getting random choices to work well also seems unattainable.
City bar changes: very opposed to. Noticed them immediately, tried playing with them; looked in the commit log for how to reverse them (to the old even smaller settings) and immediately was more comfortable. [...] Now that the resource bubbles are shrinkable, (I set them to 25) I don't understand why I would want to have tiny resources bubbles yet be forced to have giant growing bars littering the screen.
Attaching a screenshot; how the city billboards look over here. (Balloon size is 40 btw, via the "Auto" choice.) Maybe the fader settings for the billboards don't interact well with your combination of screen size, field of view and camera distance. Or do they also seem gigantic in my screenshot? As to why – to be able to read city name and production strings, especially when the orange food bar fills up. But, if the fader approach can't be made to work for (most) anyone, I guess the billboard text will have to be left alone – like the rest of the text elements, which could be enlarged too, but not in a way that is customizable from the BUG options screen.
BUG enables almost everything by default. I don't like that either, but, if I start disabling stuff, BUG users may well think the feature isn't present at all. (Happens a lot with K-Mod, which disables most of the BUG options by default.) I did change the default for a select few, e.g. "Train military units forever: Selecting a military unit in the Choose Production popup sets the city to train that unit repeatedly." Still can't quite believe that this was enabled by default. Hm, forum search indeed finds nearly a dozen users asking why their cities keep producing units. Edit: Actually, the wide city bars weren't optional at all in BUG/BULL. I had to employ a hack in the DLL to make them optional. Forgot about that myself.
That doesn't seem to be the cause, but I did manage to reproduce the issue while trying it that way. Still getting neither a splash screen nor a popup from my savegame when I turn NoTechSplash off, but I do get the splash screen again on later techs, so the setting is apparently not stored in savegames. Anyway, I'll investigate.
By the way, I found this doc comment for isAdvancedMap (in CvMapScriptInterface.py):
"Advanced maps only show up in the map script pulldown on the advanced menu.
Return 0 if you want your map to show up in the simple singleplayer menu"
I suppose Play Now is the simple menu and Custom Game the advanced one. Although this doesn't seem to align with our experience of certain scripts (don't recall which) not showing up under Play Now regardless of what their isAdvancedMap function says. But I guess it might work as intended, and there are just other (mystery) conditions that can block a script from Play Now.
Got them all fixed fairly surgically (the map scripts): Git commit
About ready then to finally release v1.03. I've even played - and thereby tested - it for a few hours.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
github.com
