1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

Stub/Skeleton for CvPythonExtensions, for auto-complete in PyCharm

Discussion in 'Civ4 - SDK/Python' started by lfgr, May 1, 2021.

  1. lfgr

    lfgr King

    Joined:
    Feb 6, 2010
    Messages:
    931

    Background


    Advanced features of python IDEs like PyCharm, such as content assist/auto-complete, do not work very well for CivIV python modding, since the IDE doesn't know Civ's C++-generated module "CvPythonExtensions". This is not an unknown problem in the python world, since many python packages have substantial portions written in C/C++. A usual solution is to provide stubs or skeletons (I'm honestly unsure what the correct terminology is). These are basically modules written in pure python that mirror the member variables and functions in the original module, but omit the implementation. Often, they additionally contain type annotations.

    Stub/Skeleton for CvPythonExtensions

    The attached file is a skeleton for Bts's CvPythonExtensions, with type annotations. It is generated via a script (see below), so some errors and missing information are to be expected. I'll describe how to use it with PyCharm, which is the only IDE I know that properly supports type annotations. PyCharm does have a mechanism to add stubs to existing modules, but since we don't actually have a CvPythonExtensions module that is usable outside of CivIV, we'll just add the skeleton as a normal module. We'll never run from PyCharm, so we don't care about the missing implementation.

    The easiest way would be to set up a new project in your mod's python directory, and drop the attached file there. I don't recommend that, since you might have to remove the skeleton every time before running the mod. Also, PyCharm's project files would pollute your mod directory. Here's how I do it:
    1. Install PyCharm (choose the black Download button above "free, open source", to get the, well, free version).
    2. Install Python 2.7 (windows download, linux users probably already have it).
    3. Set up a python project somewhere, outside the mod directory. Use your installation of Python 2.7 as the python interpreter (Civ uses 2.4, but PyCharm only supports 2.7).
    4. Download the attached CvPythonExtensions.py.zip file and extract it into some directory, say "blah/skeleton".
    5. Go to File -> Settings -> Project: ... -> Project Structure
    6. Click "+ Add Content Root" at the top right, and choose your mod's python directory. Do the same for "blah/skeleton", where you put the attached file.
    7. Below the "+ Add Content Root" button, you should now see the two folders and their subfolders. Select your mod's python directory. Select all folders (including subfolders!) in the middle column and click mark them as "Sources", with the button above. This is because Civ imports modules from the python directory and all its subdirectories.
    8. Click "blah/skeleton" on the right side, and mark it as a "Sources" directory, if it isn't already.
    9. Start enjoying content assist, as in the screenshot!
    Feel free to ask questions here if this doesn't work for you.

    Generating stubs for your own mod

    If your mod exposes new python classes/methods/functions in the DLL, these will obviously not be available in the attached stub file. You can easily generate your own stub file, though. The source code and documentation for this is found here.

    Basically, there are two steps: First, you add merge a small modcomp into your mod. Then you start the game. The modcomp automatically scrapes the CvPythonExtensions module and writes the obtained data into the log. Afterwards, you shoud remove the modcomp. You then put the scraped data into a python script that generates the stub. The type annotations are partly hardcoded with regular expressions (see generate/config_default.json), but mostly obtained by parsing the docstrings that the DLL provides. This is quite complicated, but works surprisingly well. Still, there are a couple of erroneous docstrings which translate to wrong type annotations.

    Again, if you have any questions, feel free to ask.
     

    Attached Files:

    Louis the XIV, f1rpo and Nightinggale like this.
  2. Ramkhamhaeng

    Ramkhamhaeng Warlord

    Joined:
    Feb 24, 2014
    Messages:
    189
    lfgr likes this.
  3. lfgr

    lfgr King

    Joined:
    Feb 6, 2010
    Messages:
    931
    Nice! I was wondering if somebody else tried something like this. Did you generate it by parsing the Cy...Interface.cpp files somehow?
     
  4. Ramkhamhaeng

    Ramkhamhaeng Warlord

    Joined:
    Feb 24, 2014
    Messages:
    189
    I used several methods but later found out, like you, that this methods all can be superseded by an extraction like
    Code:
    import pydoc, CvPythonExtensions
    pydoc.doc(CvPythonExtensions)
    # More stuff to parsing Types.
    
    Some classes are hidden in the Exe binary and can be found over this way, but not by reading the SDK interfaces :)
     
    Last edited: May 3, 2021
    lfgr likes this.
  5. Nightinggale

    Nightinggale Deity

    Joined:
    Feb 2, 2009
    Messages:
    4,859
    This looks great. One thing, which bugs me about python is precisely the lack of a proper IDE. Well that and the lack of a proper debugger.

    Sounds good. Any chance you would share precisely how you did that for people who would like to use it elsewhere, like with say the Colonization exe? :)
     
  6. Ramkhamhaeng

    Ramkhamhaeng Warlord

    Joined:
    Feb 24, 2014
    Messages:
    189
    Lfgr, above project can be used for this case. His script contain filters to bring the output in an usable form.

    Another way: Take a look into BTS/Assets/Python/EntryPoints/CvAppInterface.py

    Code:
            # for PythonExtensions Help File
            PythonHelp = 0          # doesn't work on systems which haven't installed Python
                            
            # dump Civ python module directory
            if PythonHelp:          
                    import CvPythonExtensions
                    helpFile=file("CvPythonExtensions.hlp.txt", "w")
                    sys.stdout=helpFile
                    import pydoc                  
                    pydoc.help(CvPythonExtensions)
                    helpFile.close()
    
    => Documentation will be written in CvPythonExtensions.hlp.txt
    Probably the same code is already contained in Colonization.
     
    Nightinggale likes this.

Share This Page