1. Firaxis celebrates the "Asian American and Pacific Islander Heritage Month", and offers a give-away of a Civ6 anthology copy (5 in total)! For all the details, please check the thread here. .
    Dismiss Notice
  2. We have selected the winners of the Old World random draw and competition. For the winning entries, please check this thread.
    Dismiss Notice
  3. Old World has finally been released on GOG and Steam, besides also being available in the Epic store . Come to our Old World forum and discuss with us!
    Dismiss Notice

Stub/Skeleton for CvPythonExtensions, for auto-complete in PyCharm

Discussion in 'Civ4 - SDK/Python' started by lfgr, May 1, 2021.

  1. lfgr

    lfgr Emperor

    Joined:
    Feb 6, 2010
    Messages:
    1,021

    Background


    Advanced features of python IDEs like PyCharm, such as content assist/auto-complete, do not work very well for CivIV python modding, since the IDE doesn't know Civ's C++-generated module "CvPythonExtensions". This is not an unknown problem in the python world, since many python packages have substantial portions written in C/C++. A usual solution is to provide stubs or skeletons (I'm honestly unsure what the correct terminology is). These are basically modules written in pure python that mirror the member variables and functions in the original module, but omit the implementation. Often, they additionally contain type annotations.

    Stub/Skeleton for CvPythonExtensions

    The attached file is a skeleton for Bts's CvPythonExtensions, with type annotations. It is generated via a script (see below), so some errors and missing information are to be expected. I'll describe how to use it with PyCharm, which is the only IDE I know that properly supports type annotations. PyCharm does have a mechanism to add stubs to existing modules, but since we don't actually have a CvPythonExtensions module that is usable outside of CivIV, we'll just add the skeleton as a normal module. We'll never run from PyCharm, so we don't care about the missing implementation.

    The easiest way would be to set up a new project in your mod's python directory, and drop the attached file there. I don't recommend that, since you might have to remove the skeleton every time before running the mod. Also, PyCharm's project files would pollute your mod directory. Here's how I do it:
    1. Install PyCharm (choose the black Download button above "free, open source", to get the, well, free version).
    2. Install Python 2.7 (windows download, linux users probably already have it).
    3. Set up a python project somewhere, outside the mod directory. Use your installation of Python 2.7 as the python interpreter (Civ uses 2.4, but PyCharm only supports 2.7).
    4. Download the attached CvPythonExtensions.py.zip file and extract it into some directory, say "blah/skeleton".
    5. Go to File -> Settings -> Project: ... -> Project Structure
    6. Click "+ Add Content Root" at the top right, and choose your mod's python directory. Do the same for "blah/skeleton", where you put the attached file.
    7. Below the "+ Add Content Root" button, you should now see the two folders and their subfolders. Select your mod's python directory. Select all folders (including subfolders!) in the middle column and click mark them as "Sources", with the button above. This is because Civ imports modules from the python directory and all its subdirectories.
    8. Click "blah/skeleton" on the right side, and mark it as a "Sources" directory, if it isn't already.
    9. Start enjoying content assist, as in the screenshot!
    Feel free to ask questions here if this doesn't work for you.

    Generating stubs for your own mod

    If your mod exposes new python classes/methods/functions in the DLL, these will obviously not be available in the attached stub file. You can easily generate your own stub file, though. The source code and documentation for this is found here.

    Basically, there are two steps: First, you add merge a small modcomp into your mod. Then you start the game. The modcomp automatically scrapes the CvPythonExtensions module and writes the obtained data into the log. Afterwards, you shoud remove the modcomp. You then put the scraped data into a python script that generates the stub. The type annotations are partly hardcoded with regular expressions (see generate/config_default.json), but mostly obtained by parsing the docstrings that the DLL provides. This is quite complicated, but works surprisingly well. Still, there are a couple of erroneous docstrings which translate to wrong type annotations.

    Again, if you have any questions, feel free to ask.
     

    Attached Files:

  2. Ramkhamhaeng

    Ramkhamhaeng Warlord

    Joined:
    Feb 24, 2014
    Messages:
    192
    lfgr likes this.
  3. lfgr

    lfgr Emperor

    Joined:
    Feb 6, 2010
    Messages:
    1,021
    Nice! I was wondering if somebody else tried something like this. Did you generate it by parsing the Cy...Interface.cpp files somehow?
     
  4. Ramkhamhaeng

    Ramkhamhaeng Warlord

    Joined:
    Feb 24, 2014
    Messages:
    192
    I used several methods but later found out, like you, that this methods all can be superseded by an extraction like
    Code:
    import pydoc, CvPythonExtensions
    pydoc.doc(CvPythonExtensions)
    # More stuff to parsing Types.
    
    Some classes are hidden in the Exe binary and can be found over this way, but not by reading the SDK interfaces :)
     
    Last edited: May 3, 2021
    lfgr likes this.
  5. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,977
    This looks great. One thing, which bugs me about python is precisely the lack of a proper IDE. Well that and the lack of a proper debugger.

    Sounds good. Any chance you would share precisely how you did that for people who would like to use it elsewhere, like with say the Colonization exe? :)
     
  6. Ramkhamhaeng

    Ramkhamhaeng Warlord

    Joined:
    Feb 24, 2014
    Messages:
    192
    Lfgr, above project can be used for this case. His script contain filters to bring the output in an usable form.

    Another way: Take a look into BTS/Assets/Python/EntryPoints/CvAppInterface.py

    Code:
            # for PythonExtensions Help File
            PythonHelp = 0          # doesn't work on systems which haven't installed Python
                            
            # dump Civ python module directory
            if PythonHelp:          
                    import CvPythonExtensions
                    helpFile=file("CvPythonExtensions.hlp.txt", "w")
                    sys.stdout=helpFile
                    import pydoc                  
                    pydoc.help(CvPythonExtensions)
                    helpFile.close()
    
    => Documentation will be written in CvPythonExtensions.hlp.txt
    Probably the same code is already contained in Colonization.
     
    Nightinggale likes this.

Share This Page