Stub/Skeleton for CvPythonExtensions, for auto-complete in PyCharm

[lfgr]

Background

Advanced features of python IDEs like PyCharm, such as content assist/auto-complete, do not work very well for CivIV python modding, since the IDE doesn't know Civ's C++-generated module "CvPythonExtensions". This is not an unknown problem in the python world, since many python packages have substantial portions written in C/C++. A usual solution is to provide stubs or skeletons (I'm honestly unsure what the correct terminology is). These are basically modules written in pure python that mirror the member variables and functions in the original module, but omit the implementation. Often, they additionally contain type annotations.

Stub/Skeleton for CvPythonExtensions

The attached file is a skeleton for Bts's CvPythonExtensions, with type annotations. It is generated via a script (see below), so some errors and missing information are to be expected. I'll describe how to use it with PyCharm, which is the only IDE I know that properly supports type annotations. PyCharm does have a mechanism to add stubs to existing modules, but since we don't actually have a CvPythonExtensions module that is usable outside of CivIV, we'll just add the skeleton as a normal module. We'll never run from PyCharm, so we don't care about the missing implementation.

The easiest way would be to set up a new project in your mod's python directory, and drop the attached file there. I don't recommend that, since you might have to remove the skeleton every time before running the mod. Also, PyCharm's project files would pollute your mod directory. Here's how I do it:

1. Install PyCharm (choose the black Download button above "free, open source", to get the, well, free version).
   
2. Install Python 2.7 (windows download, linux users probably already have it).
   
3. Set up a python project somewhere, outside the mod directory. Use your installation of Python 2.7 as the python interpreter (Civ uses 2.4, but PyCharm only supports 2.7).
   
4. Download the attached CvPythonExtensions.py.zip file and extract it into some directory, say "blah/skeleton".
5. Go to File -> Settings -> Project: ... -> Project Structure
6. Click "+ Add Content Root" at the top right, and choose your mod's python directory. Do the same for "blah/skeleton", where you put the attached file.
7. Below the "+ Add Content Root" button, you should now see the two folders and their subfolders. Select your mod's python directory. Select all folders (including subfolders!) in the middle column and click mark them as "Sources", with the button above. This is because Civ imports modules from the python directory and all its subdirectories.
8. Click "blah/skeleton" on the right side, and mark it as a "Sources" directory, if it isn't already.
9. Start enjoying content assist, as in the screenshot!

Feel free to ask questions here if this doesn't work for you.

Generating stubs for your own mod

If your mod exposes new python classes/methods/functions in the DLL, these will obviously not be available in the attached stub file. You can easily generate your own stub file, though. The source code and documentation for this is found here.

Basically, there are two steps: First, you add merge a small modcomp into your mod. Then you start the game. The modcomp automatically scrapes the CvPythonExtensions module and writes the obtained data into the log. Afterwards, you shoud remove the modcomp. You then put the scraped data into a python script that generates the stub. The type annotations are partly hardcoded with regular expressions (see generate/config_default.json), but mostly obtained by parsing the docstrings that the DLL provides. This is quite complicated, but works surprisingly well. Still, there are a couple of erroneous docstrings which translate to wrong type annotations.

Again, if you have any questions, feel free to ask.

 

[Ramkhamhaeng]

Hi lfgr,

yes I found such stub file helpful, too.

Just as reference: We wrote a similar file for Pylint:
https://github.com/civ4-mp/pbmod/tree/master/tests/Pylint_for_Civ4/Civ4PythonApi

It contain at least one class not included in your version: CyPitboss.

 

[lfgr]

Nice! I was wondering if somebody else tried something like this. Did you generate it by parsing the Cy...Interface.cpp files somehow?

 

[Ramkhamhaeng]

I used several methods but later found out, like you, that this methods all can be superseded by an extraction like

import pydoc, CvPythonExtensions
pydoc.doc(CvPythonExtensions)
# More stuff to parsing Types.

Some classes are hidden in the Exe binary and can be found over this way, but not by reading the SDK interfaces

 

[Nightinggale]

This looks great. One thing, which bugs me about python is precisely the lack of a proper IDE. Well that and the lack of a proper debugger.

I used several methods but later found out, like you, that this methods all can be superseded by an extraction like

Sounds good. Any chance you would share precisely how you did that for people who would like to use it elsewhere, like with say the Colonization exe?

 

[Ramkhamhaeng]

Sounds good. Any chance you would share precisely how you did that for people who would like to use it elsewhere, like with say the Colonization exe?

Lfgr, above project can be used for this case. His script contain filters to bring the output in an usable form.

Another way: Take a look into BTS/Assets/Python/EntryPoints/CvAppInterface.py

        # for PythonExtensions Help File
        PythonHelp = 0          # doesn't work on systems which haven't installed Python
                        
        # dump Civ python module directory
        if PythonHelp:          
                import CvPythonExtensions
                helpFile=file("CvPythonExtensions.hlp.txt", "w")
                sys.stdout=helpFile
                import pydoc                  
                pydoc.help(CvPythonExtensions)
                helpFile.close()

=> Documentation will be written in CvPythonExtensions.hlp.txt
Probably the same code is already contained in Colonization.