[PYTHONCOMP] Python utility library (includes INI file reading)

D

Dr Elmer Jiggle

King
Joined
Feb 25, 2003
Messages
811
While developing my alerts mod, I had occasion to implement several Python modules that I think will be useful to a wider audience. The modules attached here are also available within that mod, but I've packaged them separately here to keep out the code that's unique to the alerts mod. This thread will also serve as a more appropriate place for questions about these modules.

I've included a brief description of each module below, but the code is extensively documented, so I'm not going to duplicate that here.

CvPath.py

Exposes variables that point to various interesting directories within the Civilziation 4 hierarchy. For example, there are variables for the install directory, the user directory, and the complete Python search path.

CvConfigParser.py

Provides convenient INI file handling. Usage is as simple as

Code: 
config = CvConfigParser.CvConfigParser("Foo.ini")
# default to 5 if not specified in the .INI file
myInt = config.getint("My Section", "My Int Option", 5)

Advantages of INI files over other approaches such as embedding the settings directly in Python or using an XML file include

  • Users tend to be less intimidated by INI files
  • Users tend to be less likely to completely mess up an INI file
  • The game's anti-cheating mechanism prevents editing of mod source files but not INI files, so this method is compatible with Game of the Month

CvCustomEventManager.py

Implements an extensible event manager. If multiple mods exist that all override the default event manager, you can simply register their needs with calls such as

Code: 
em1.addEventHandler("cityGrowth", em1.onCityGrowth)
em1.addEventHandler("cityDoTurn", em1.onCityDoTurn)
em2.addEventHandler("cityGrowth", em2.onCityGrowth)

Enjoy.

Changes:
  • 2006-Aug-11
    • Updated for Warlords compatibility.
  • 2006-May-1, v1.4
    • Added validity checking for event types in the add, remove, and setEventHandler functions. An exception is thrown if the event type string is incorrect.
    • Added a setPopupHandler function for defining new popup dialog handlers.
  • 2006-Mar-12, v1.3
    • Fixed a bug with determining the user directory when the installation directory has been renamed. Thanks to 12monkeys
  • 2006-Feb-28, v1.2
    • Fixed the search path for .INI files. The parser was searching the Assets directories instead of their respective parent directories. Thanks to jray
    • Restored the 6 Boolean state flags to the event handler front end. Thanks to jray
  • 2006-Feb-2, v1.1
    • Added the CvModName feature to CvConfigParser to allow discovery of the active mod name under certain circumstances.
    • Fixed exception handling in INI file reader.

DrEJlib.zip
 
FYI, I highly recommend using this library and I am going to implemented in my MODCOMPs that provide configuration options. One key learning I have to give to the group that I learned though adding INI parsing in one of my MODCOMPs:

If you want to use INI files but don't want to have them in the install dir you should change the line in CvPath.py:
assetsPath = [userAssetsDir, installAssetsDir]

to

assetsPath = [userAssetsDir, installAssetsDir, installModsDir]

This way all of the mod component INI files can live in the mod directory until we can figure out a way to get the active mod name.
 
TheLopez said:
If you want to use INI files but don't want to have them in the install dir you should change the line in CvPath.py:
assetsPath = [userAssetsDir, installAssetsDir]

to

assetsPath = [userAssetsDir, installAssetsDir, installModsDir]

This way all of the mod component INI files can live in the mod directory until we can figure out a way to get the active mod name.
Click to expand...

Good point. I've actually had an equivalent change at various points, and I keep going back and forth on whether I like it or not. If you want it to work that way though, I think I'd suggest a few slight alterations in your approach.

1) You should probably include userModsDir as well, since users would typically be installing their mods there. To clarify, userModsDir is C:/Documents and Settings/User/My Documents/My Games/Sid Meier's Civilization 4/Mods. installModsDir is C:/Program Files/Firaxis Games/Sid Meier's Civilization 4/Mods.

2) The mods directories should maybe come first in the list, before the other directories. In general, when loading Python and XML files, the mods directories override the regular Assets and CustomAssets directories, so I think that would make sense for INI files too. The only real problem with that is since there's no way of knowing when a mod is active, your Mod INI file would override the main one even when you aren't actually running a mod.

3) Finally, another thought is that it might be better to make the change in the CvConfigParser code instead of in CvPaths. By changing assetsPath directly, you impact anything else that uses assetsPath, including, for example, the pythonPath variable. With your change, pythonPath will include all Python directories in every mod. One way to get around this would be to make the following change around line 63 in the CvConfigParser constructor.

Code: 
            iniPath = [userModsDir, installModsDir] + assetsPath
            filenames = [os.path.join(dir, filename) 
                         for dir in iniPath]

I haven't tested that, but I think that code is correct and would do what you want.
 
It seems it would not take much to turn this into an extensions type of framework. For example:

  1. Mods don't mess with the events manager directly at all
  2. A standard events manager in the 'framework' looks at all files in a standard named directory say 'exts' and creates and instance of any classes it finds there that extends correct base class.
  3. As part of the init for these classes they register their events.
That would make merging of python mods very simple without any toe stepping. It's a pretty common design and I'm sure I'm glossing over any number of issues needing consideration but it seems a reasonable approach.
 
Dr Elmer Jiggle said:
3) Finally, another thought is that it might be better to make the change in the CvConfigParser code instead of in CvPaths. By changing assetsPath directly, you impact anything else that uses assetsPath, including, for example, the pythonPath variable. With your change, pythonPath will include all Python directories in every mod. One way to get around this would be to make the following change around line 63 in the CvConfigParser constructor.

Code: 
            iniPath = [userModsDir, installModsDir] + assetsPath
            filenames = [os.path.join(dir, filename) 
                         for dir in iniPath]

I haven't tested that, but I think that code is correct and would do what you want.
Click to expand...

Dr Elmer Jiggle, ok I tried making the suggested changes and tested them. There were a couple of issues. Here is the fixed code:

Code: 
            iniPath = [CvPath.userModsDir, CvPath.installModsDir] + CvPath.assetsPath
            filenames = [os.path.join(dir, filename) 
                         for dir in iniPath]

Another issue I ran into is when a key=value pair does not exist in the INI file the game still throws the exception and the default value does not get set. Any ideas?
 
rcuddy said:
A standard events manager in the 'framework' looks at all files in a standard named directory say 'exts' and creates and instance of any classes it finds there that extends correct base class.
Click to expand...

That's how I'd prefer for this to work, actually. There are a few other areas where you could take a similar approach. For example, the options screen could be extended to allow mods to plug in a tab with their custom settings.

SimCutie has actually implemented something like that for event managers already. I'll see if I can find a link for that.

The main problem (really the only problem I can think of right now) with that is there's no known way to determine the active mod. So the idea of searching through all files doesn't work, because you don't know where to search. For example, should you look in Mods/Foo/Assets/Python, Mods/Bar/Assets/Python, or neither because this is just a regular non-modded game?

SimCutie's code has an interesting way of dealing with that problem. Basically it attempts to load Python modules from the Mod directories, and when one succeeds it assumes that means it found the active mod directory. At first glance that seems like an incredibly devious but also ingenous solution, but I'm convinced that it doesn't work.

The problem is, what happens if several mods contain a Python module with the same name? This is actually quite likely. Consider for example the number of mods that will contain a CvCustomEventManager and/or a CvEventInterface module. Now when you load "CvEventInterface" successfully, how do you know which mod that came from or if it came from a mod at all? So for this idea to work, you would need each mod to contain at least one uniquely named Python module that isn't present in any other mod.

I tried running with that idea by implementing some nasty code that dynamically creates a uniquely named Python module in each mod directory. I made files like a.py, aa.py, etc. They wouldn't load, though. I think the game's module loader caches the list of available files, so if you create one while the game is running, it doesn't pick up on that.

Anyway, the point of this long essay is that it's a good idea, but I think it needs a little help from Firaxis before it can really work the way it should. If Firaxis provides an API function to give the active mod name, or if someone figures out a way to get that, then I think we've got something to work with.

Edit: SimCutie's event manager code is in this thread: http://forums.civfanatics.com/showthread.php?t=147018
 
Dr Elmer Jiggle said:
Anyway, the point of this long essay is that it's a good idea, but I think it needs a little help from Firaxis before it can really work the way it should. If Firaxis provides an API function to give the active mod name, or if someone figures out a way to get that, then I think we've got something to work with.
Click to expand...

Dr Elmer Jiggle, Do you know what gets loaded first? XML files or python files?
 
TheLopez said:
Another issue I ran into is when a key=value pair does not exist in the INI file the game still throws the exception and the default value does not get set. Any ideas?
Click to expand...

I think I've tested that, but I'll have to see. Maybe I never verified that it actually works. :blush: Do you have a copy of the exception backtrace from a PythonErr.log file? That might help give me an idea of what's wrong. Just looking at the code it seems like it should be working.
 
TheLopez said:
Dr Elmer Jiggle, Do you know what gets loaded first? XML files or python files?
Click to expand...

I think it can depend on the situation, but definitely some Python files get loaded before the first XML file. I noticed this at one point when I was trying to configure some settings using GlobalDefinesAlt.xml. The settings in the XML weren't being picked up properly unless I delayed looking for them until after the onInit event handler was called. But as far as I know, Python files are loaded whenever you hit the first "import Foo", so if that happens later, then the XML files have probably already been loaded.
 
TheLopez said:
Another issue I ran into is when a key=value pair does not exist in the INI file the game still throws the exception and the default value does not get set. Any ideas?
Click to expand...

That does seem to be working for me. I temporarily moved my INI file, and the code still ran successfully and without exceptions.

I don't think my mod actually uses anything besides the getint function. Are you having problems with a different one? They all work mostly the same way, but there are slight variations (especially in the boolean one) that might be causing an unexpected problem since they haven't been tested carefully.

Do you have a code snippet you can post? You need to pass 3 arguments to each get* function if you want to include a default value.

Code: 
get(sectionName, optionName, default = None)
getint(sectionName, optionName, default = None)
getfloat(sectionName, optionName, default = None)
getboolean(sectionName, optionName, default = None)

where sectionName and optionName are strings that refer to the INI file value you want. default is whatever type you expect as the result of the get method (ex. int for getint, boolean for getboolean, etc.).
 
Dr Elmer Jiggle said:
I think I've tested that, but I'll have to see. Maybe I never verified that it actually works. :blush: Do you have a copy of the exception backtrace from a PythonErr.log file? That might help give me an idea of what's wrong. Just looking at the code it seems like it should be working.
Click to expand...

I don't have access to upload it right now, my laptop doesn't want to connect to our corporate network for whatever reason. I will add it

<HERE>

when I get home
 
Dr Elmer Jiggle said:
I think it can depend on the situation, but definitely some Python files get loaded before the first XML file. I noticed this at one point when I was trying to configure some settings using GlobalDefinesAlt.xml. The settings in the XML weren't being picked up properly unless I delayed looking for them until after the onInit event handler was called. But as far as I know, Python files are loaded whenever you hit the first "import Foo", so if that happens later, then the XML files have probably already been loaded.
Click to expand...

Ok, then I know how we can get the active mod name added. What if we added the active mod name in an XML info file like you are doing so in your Civ4AlertTextInfo.xml file and extract it using the CyTranslator class and its getText method?
 
Dr Elmer Jiggle said:
That does seem to be working for me. I temporarily moved my INI file, and the code still ran successfully and without exceptions.

I don't think my mod actually uses anything besides the getint function. Are you having problems with a different one? They all work mostly the same way, but there are slight variations (especially in the boolean one) that might be causing an unexpected problem since they haven't been tested carefully.

Do you have a code snippet you can post? You need to pass 3 arguments to each get* function if you want to include a default value.

Code: 
get(sectionName, optionName, default = None)
getint(sectionName, optionName, default = None)
getfloat(sectionName, optionName, default = None)
getboolean(sectionName, optionName, default = None)

where sectionName and optionName are strings that refer to the INI file value you want. default is whatever type you expect as the result of the get method (ex. int for getint, boolean for getboolean, etc.).
Click to expand...


Here is the code i am using:
bHighlightForcedSpecialists = config.getboolean("Specialist Stacker", "HighlightForcedSpecialists",True)

The key=value pair does not exist in my INI file.

EDIT:
Here is another couple of lines. I tried:
SPECIALISTS_STACK_WIDTH = config.getint("Specialist Stacker", "SpecialistStackWidth", default = 15)
and this:
SPECIALISTS_STACK_WIDTH = config.getint("Specialist Stacker", "SpecialistStackWidth", 15)

neither worked.
 
TheLopez said:
Ok, then I know how we can get the active mod name added. What if we added the active mod name in an XML info file like you are doing so in your Civ4AlertTextInfo.xml file and extract it using the CyTranslator class and its getText method?
Click to expand...

Yes! I think that will work. I'm not sure if you're suggesting that the mod author should create an XML file with the correct setting in it or that we should create one dynamically at startup for each mod and see which one gets loaded.

The first approach is obviously much easier, but it will only work if the mod author goes through the trouble of creating the right XML file. At that point, there are lots of ways you could do it. You could, for example, try to "import ActiveModName" which would be a trivial Python file that does nothing except set a variable to the active mod name. Once you're willing to force mod authors to cooperate with the protocol, it's easy. The problem is making it also work with mods like "American Revolution" that you have no control over.

Creating a special XML file for each mod during startup is godawful ugly and at the same time absolutely brilliant. ;) It might take some experimentation to figure out the best way of making sure the XML files get created before the game tries to read the XML files, but it seems like it might do the job.

I'm going to see if I can get a prototype of this working tonight.
 
TheLopez said:
Here is another couple of lines. I tried:
SPECIALISTS_STACK_WIDTH = config.getint("Specialist Stacker", "SpecialistStackWidth", default = 15)
and this:
SPECIALISTS_STACK_WIDTH = config.getint("Specialist Stacker", "SpecialistStackWidth", 15)
Click to expand...

I'm not sure if the first way is supposed to work or not (I don't know enough Python to be sure), but the second definitely should. That's exactly how I'm using it in my code. I also realized that I'm using both getint and getboolean, so as far as I know both of those work correctly.

I have a theory. How/where are you initializing config? Are you sure it has a value at the point where you're using it? Do you perhaps need to use self.config or some other variation to make it work? What kind of exception are you getting?
 
Dr Elmer Jiggle said:
I'm not sure if the first way is supposed to work or not (I don't know enough Python to be sure), but the second definitely should. That's exactly how I'm using it in my code. I also realized that I'm using both getint and getboolean, so as far as I know both of those work correctly.

I have a theory. How/where are you initializing config? Are you sure it has a value at the point where you're using it? Do you perhaps need to use self.config or some other variation to make it work? What kind of exception are you getting?
Click to expand...

Here's the exception trace:
Traceback (most recent call last):
File "CvScreensInterface", line 63, in showMainInterface
File "CvMainInterface", line 190, in interfaceScreen
File "CvConfigParser", line 83, in getint
File "CvConfigParser", line 102, in _wrappedGet
File "D:\main\Civilization4\Assets\Python\System\ConfigParser.py", line 321, in getint
File "D:\main\Civilization4\Assets\Python\System\ConfigParser.py", line 318, in _get
File "CvConfigParser", line 78, in get
File "CvConfigParser", line 102, in _wrappedGet
File "D:\main\Civilization4\Assets\Python\System\ConfigParser.py", line 520, in get
ConfigParser.NoOptionError: No option 'specialist stack width' in section: 'Specialist Stacker'

Dr Elmer Jiggle said:
Yes! I think that will work. I'm not sure if you're suggesting that the mod author should create an XML file with the correct setting in it or that we should create one dynamically at startup for each mod and see which one gets loaded.
Click to expand...
The mod author should create an XML file with the correct setting in it. It really isn't that much to ask from mod authors, epecially when you balance the benefits of being able to provide them with the ability to let people configure their mods using INI files instead modifying python files.
 
I think I see what the problem is. Try changing the _wrappedGet function in CvConfigParser as shown below. The change is the addition of parentheses in the except clause. Apparently if you leave out the parentheses it's still correct syntax, but the meaning is completely different.

Code: 
    def _wrappedGet(self, getter, section, option, default, *args, **kwargs):
        """Wraps the specified getter function with an exception handler
        and returns a default value if NoSectionError or NoOptionError
        is raised.

        """
        try:
            return getter(section, option, *args, **kwargs)
        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
            if (default != None):
                return default
            else:
                raise

Assuming this fixes it, I'll upload a revised version (much) later tonight.
 
Dr Elmer Jiggle said:
I think I see what the problem is. Try changing the _wrappedGet function in CvConfigParser as shown below. The change is the addition of parentheses in the except clause. Apparently if you leave out the parentheses it's still correct syntax, but the meaning is completely different.

Code: 
    def _wrappedGet(self, getter, section, option, default, *args, **kwargs):
        """Wraps the specified getter function with an exception handler
        and returns a default value if NoSectionError or NoOptionError
        is raised.

        """
        try:
            return getter(section, option, *args, **kwargs)
        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
            if (default != None):
                return default
            else:
                raise

Assuming this fixes it, I'll upload a revised version (much) later tonight.
Click to expand...

Dr Elmer Jiggle, I have validated the code fix and it works just like it should. Thanks for your help.
 
TheLopez said:
The mod author should create an XML file with the correct setting in it. It really isn't that much to ask from mod authors, epecially when you balance the benefits of being able to provide them with the ability to let people configure their mods using INI files instead modifying python files.
Click to expand...

Yeah, I agree with you. I keep reminding myself that since the perfect solution isn't possible, the goal is to create the least bad solution, and this would clearly be less bad than having no way at all of getting the active mod name.

I tried some experiments with using an XML file, and they weren't working out well. There are timing issues that make the system very fragile. As I mentioned before, the XML files seem to be loaded fairly late in the startup process, so if you try to initialize the active mod name too early, you get the wrong value.

I'm thinking of using a Python module instead. For example, I can do something like (this code is untested, just roughed in now as an example):

Code: 
try:
    import CvModName
    activeModName = CvModName.modName
except:
    pass

So if the module loads successfully (and contains a variable named modName), the active mod name is set. Otherwise it just fails quietly, and activeModName stays set to None just like it does now. An example CvModName module would be as simple as:

Code: 
# CvModName.py
modName = "My Mod Name"

I find this approach kind of ugly and klugy looking, but like I said before, it might be the least bad option available. At least it would give you some way of getting an INI file to load from the Mods/My Mod Name directory. If I can verify that it works, I'll probably fire something up at lunch or later today.
 
Dr Elmer Jiggle,

Following your example I tested your code and it seems to work.

Basically, I created created the file CvModuleName.py
Code: 
# CvModName.py
modName = "Specialist Stacker"

I added this code after line 98 in the CvPath.py file to use the active mod name.
Code: 
try:
    import CvModName
    activeModName = CvModName.modName
except:
    pass

Changed the CvConfigParser __init__ method so it will only use the CvPath.installActiveModDir to look for the variables.

To test this change I left my "Stacked Specialists Config.ini" file in the "Stacked Specialists" mod directory and did two tests, one with the highlight forced specialists set to true and with it set to false, both providing the expected results.

Anyone using or planning to use the "Stacked Specialists" mod component, expect a new release tonight using this new methodology.
 
You must log in or register to reply here.

Similar threads

A
[SCENARIO] Incoherent starting locations with the one precised in the .CivBeyondSwordWBSave file.
Replies
0
Views
94
arkantoslegrand
A
<Nexus>
BUG REPORT, F.A.Q. and known issues
4 5 6
Replies
109
Views
5K
<Nexus>
<Nexus>
L
  • Sticky
Workarounds and suggestions for making Civ III run well in 2023
2 3
Replies
54
Views
6K
WildWeazel
WildWeazel
S
  • Sticky
Tweak Mod for Xtended + Master of Mana Updates
10 11 12
Replies
224
Views
15K
Guru-Meditation
Guru-Meditation
L
Stub/Skeleton for CvPythonExtensions, for auto-complete in PyCharm
Replies
5
Views
2K
Ramkhamhaeng
R
Top Bottom