1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

Mod for Pitboss Games

Discussion in 'Civ4 - PitBoss Games' started by Ramkhamhaeng, Sep 4, 2014.

  1. Ramkhamhaeng

    Ramkhamhaeng Chieftain

    Joined:
    Feb 24, 2014
    Messages:
    140

    The prefix is free selectable and should reflect your environment. In your example case I would suggest
    Code:
    {server}=d.docs.live.net
    {prefix}=[USERID]/Test
    
    Now, I want to point out that Civ4 tries to load the save at the same path as the server stores it! Thus, if you start the game normally, it will use
    Code:
    C:\Users\[USERNAME]\My Games\Beyond the Sword\Saves\multi   (DirectIP game, assumed variant in the following text)
    or
    C:\Users\[USERNAME]\My Games\Beyond the Sword\Saves\pitboss\auto (Pitboss game)
    
    and that is probably a bad path to use at the other machine :)
    Moreover, we need the path to encode the server url. The trick is the usage of the ALTROOT-Argument!

    Let's say the drive letter Z: is free and [PATH] is an arbitrary folder on your system…
    Create the subfolder [PATH]\_http_d.docs.live.net\[USERID]\Test
    and map [PATH] to Z: with
    Code:
    [i]subst  Z: PATH[/i]
    Spoiler :

    Keep in mind that the ALTROOT-Argument forces Civ4 to move files from your "My Games\Beyond the Sword"-folder to the new root folder. (Don't ask my why :)) Thus, it would be a good idea to backup the Beyond the Sword folder....


    Then start the game with the following arguments (Altroot had to be first argument and space after mod= required):
    Code:
    /ALTROOT="Z:\_http_d.docs.live.net\[USERID]\Test" mod= "[Your mod name]" 
    If it works, Civ4 copies its files into the new folder...
    Close the game and continue with the red marked step of your description. We had to reflect the path structure of the game becaus we would only share the subfolder with the saves:
    Create "Test/Saves/multi" on your OneDrive location and use mklink
    to combine it with [PATH]\...\Test\Saves\multi

    Now, if an player left the game the VOTE_...-Saves should automatically be uploaded to your web space.
    Check, if you can download it with your browser. (For testing please follow my test path and use DirectIP>Load game to re-load that VOTE-Save)

    Let's assume the player tries to reconnect...
    He joins and the server asks him to load
    Code:
    Z:\_http_d.docs.live\[USERID]\Test\Saves\multi\VOTE_...
    
    The BTS_Wrapper.exe detects the _http_ keyword and tries to download http://d.docs.live\[USERID]\...




    The net code is hidden in the executable and there are no public sources available. I've used disassemble/monitoring tools like Ida, Api Monitor or Wireshark for my analysis.
    Currently, I'm study the Windows API and under the assumption IPv4-Address + Good configured firewall at host side I probably could provide a much simpler solution for Non-Pitboss DirectIP games. :)
     
  2. 45°38'N-13°47'E

    45°38'N-13°47'E Chieftain

    Joined:
    Jun 7, 2008
    Messages:
    5,748
    Location:
    Just wonder...
    @Ramkhamhaeng, it's unbelievable, it works for DIRECT IP too. Hard enough to set up for the first time, but once it's done... Only thing to pay attention to when using this method for Direct IP is to switch off autosaves for all players but one, or you get an error because autosaves are being saved by multiple players at the same time. Also, probably due to slow upload speed in my case, I disable autosaves completely : I can save the game manually and I do it locally and only upload the save when needed or the connection between players will timeout when large saves are being uploaded. Thank you Ramkhamhaeng, that's been an amazing trick.
     
  3. DarkLunaPhantom

    DarkLunaPhantom Chieftain Supporter

    Joined:
    Feb 4, 2013
    Messages:
    348
    Clicking "Manage game" on Zulan's server currently gives "Server Error (500)".

    EDIT: Nevermind, everything is ok now.
     
    Last edited: Apr 8, 2017
  4. Ramkhamhaeng

    Ramkhamhaeng Chieftain

    Joined:
    Feb 24, 2014
    Messages:
    140
    @45°38'N-13°47'E: Wohoo, the path fiddling works. Nice to hear :)
    If your network had an stable IP you could also replace the webdrive with an tiny local http server at your machine. Then you could enable the autosaves again, because only a few saves will be transfered over the wire.
     
  5. 45°38'N-13°47'E

    45°38'N-13°47'E Chieftain

    Joined:
    Jun 7, 2008
    Messages:
    5,748
    Location:
    Just wonder...
    Yeah, I was thinking of using my Nas but it doesn't support webdav. But I can find other methods, it's OK. Thanks again!
     
  6. Ramkhamhaeng

    Ramkhamhaeng Chieftain

    Joined:
    Feb 24, 2014
    Messages:
    140
    @45°38'N-13°47'E
    I've finished my update of the Civ4:BTS modification. It should now supports DirectIP games, too. :)

    How it works: At startup, a minimalistic webserver will be listen at port 8080 (The port is selectable with -P argument; Civ4:BTS save games are downloadabe, but no other files). One network package will be modified and informs the client about the port where it could fetch the save. The (modified) client tries to load the save over http and fallback into the normal mode if it fails.

    Installation/Usage:
    1. Extract the content of the attachment and copy all files from 'Release' into the Civ4:BTS installation folder.
    2. Start the game with
    Code:
    [b]BTS_Wrapper.exe[/b] -P [some port] mod= "Your mod"
    3. Give users access to the selected port, if required (Port forwarding).

    Source Code (+Visual Studio 2017 project files):
    https://github.com/YggdrasiI/PBStats/tree/master/tests/GetSaveOverHttp/sources_v3

    Notes:
    • It only works for save games in one of the Civ4-"Saves" folders.
    • I did not tested it in internet lobby games yet.
    • Win XP/Vista is not supported anymore.
    • The included file Civ4BeyondSword2015.exe is optional. It fixes the connection issues due the Gamespy server shutdown.
    Work in progress Some cases (internet lobby games) wasn't tested.
     

    Attached Files:

    Last edited: Apr 21, 2017
  7. 45°38'N-13°47'E

    45°38'N-13°47'E Chieftain

    Joined:
    Jun 7, 2008
    Messages:
    5,748
    Location:
    Just wonder...
    Thanks, I will try it as soon as I can! It's awesome!! :)
     
  8. Ramkhamhaeng

    Ramkhamhaeng Chieftain

    Joined:
    Feb 24, 2014
    Messages:
    140
    Unfortunately it seems not work in all situations. I've tested it today, with a Win7 + Linux/Wine combination, but the save was not transfered (http request url is fine, but request fails) :-(
    I will start more tests and inform you again later.

    Regards
     
  9. Ramkhamhaeng

    Ramkhamhaeng Chieftain

    Joined:
    Feb 24, 2014
    Messages:
    140
    It should work, now :)
    I've replaced the Zip in the above post.
     
  10. 45°38'N-13°47'E

    45°38'N-13°47'E Chieftain

    Joined:
    Jun 7, 2008
    Messages:
    5,748
    Location:
    Just wonder...
    Is it me or loading a save with version 7 does not work? I've merged it into my mod and I could not load any save, so I've tried plain pitboss mod v7 and still I cannot load pitboss savegames I've just saved. Anyone having the same problem?
     
  11. 45°38'N-13°47'E

    45°38'N-13°47'E Chieftain

    Joined:
    Jun 7, 2008
    Messages:
    5,748
    Location:
    Just wonder...
    I've tried again and again but it looks like there's something wrong with the loading of the savegame.
    Using plain Pitboss Mod v7, I get the following python error

    Code:
    Traceback (most recent call last):
    
      File "C:\Giochi\PBs\PB2\..\Python\v7\PbWizard.py", line 698, in OnPageChanging
        (iResult,filepath) = loadSavegame(path, -1, adminPwd)
    
      File "C:\Giochi\PBs\PB2\..\Python\v7\PbWizard.py", line 120, in loadSavegame
        matchingPwd = checkSavegame(filepath, pbPasswords)
    
      File "C:\Giochi\PBs\PB2\..\Python\v7\PbWizard.py", line 75, in checkSavegame
        hSave = FindHash.get_admin_hash(filename)
    
    TypeError: get_admin_hash() takes exactly 2 arguments (1 given)
    Either I use a password or not, when I load the game it hangs there after "Shutdown Graphics" (after I've entered the password if I used one, after I selected the savegame to load if I didn't use a password). Any hint?


    Edit: it looks like there's an error here

    Code:
    def checkSavegame(filename, adminPwds):
        """ Return correct password of given list for a savegame.
        filename - The save
        adminPwds - List of passwords which md5 sum should compared
    
        return: Password ("" if save not password protected) or None
        """
        hSave = FindHash.get_admin_hash(filename) 
    As a workaround I've changed the last line to

    hSave = FindHash.get_admin_hash(filename, "")

    and now I can load any save, regardless it's password protected or not. Edit: correction, I can load the game in pitboss, but I still need the password to connect as a player.
    Also, some lines above the code reads

    pythonDir = os.path.join(gc.getAltrootDir(),'..','Python','v6')

    I suppose it should be

    pythonDir = os.path.join(gc.getAltrootDir(),'..','Python','v7')
     
    Last edited: Apr 29, 2017
  12. Ramkhamhaeng

    Ramkhamhaeng Chieftain

    Joined:
    Feb 24, 2014
    Messages:
    140
    Thanks for both bugfixes!

    It exists three different types of passwords:
    1. Admin password. Needed to load save as Host/Pitboss host.
    2. Game password. Needed to connect to DirectIP/Pitboss game.
    3. Player password. Needed as Player in Login Screen

    The PbWizard class just get in touch with the first type. The line
    Code:
                iResult = PB.load(filepath, str(matchingPwd)) # should be 0
    
    should not affect the passwords of the players. (To change a password of player you could use the 'manage game' page of the webinterface.)
    Moreover, this code line will be hang in an infinite loop, if the password is not correct!
    Thats why I've created the checkSavegame function to pre-check if the password will generate the correct hash. :)
     
  13. Ramkhamhaeng

    Ramkhamhaeng Chieftain

    Joined:
    Feb 24, 2014
    Messages:
    140
    Hello PB-guys,

    I've released a few more helper tools for the Pitboss/Civ4 world and would like to describe them here :) Tool 2 only works if you integrate the latest version of this mod, PB Mod_v7, into your mod. Tool 1 will work with older versions of this mod, too.


    1. I've replaced the startup scripts, startPitboss.bat on Windows and startPitboss.sh on Linux) with a more robust Python variant:
    PBStats/PBs/startPitboss.py

    Example usage:
    a) List available saves for game id 1
    Code:
    python startPitboss.py list 1 [filename pattern]
      Load local environment
      Youngest saves for pattern 'None':
      Nb                Timestamp Mod name        Path (without extension)
       1 Sat Dec 16 00:37:32 2017 PB Mod_v7       pitboss/auto/AutoSave_BC-3960
       2 Sat Dec 16 00:35:41 2017 PB Mod_v7       pitboss/auto/AutoSave_BC-3880
       3 Sat Dec 16 00:33:59 2017 PB Mod_v7       pitboss/auto/AutoSave_BC-3920
       4 Mon Dec  4 12:19:59 2017 PB Mod_v7       multi/Example_v7
       5 Mon Dec  4 12:19:59 2017 PB Mod_v6       multi/Example_v6
    

    b) Start server. Load newest save, matching the pattern or, without argument, the save stored in pbSettings.json
    Code:
    python startPitboss.py 1 [filename pattern] [password]
    
    Fill your environment variables into the template startPitbossEnv.py.example to define
    your PB games. The script checks the header of the given save to load the correct mod. If you doesn't use the automatic load of the save, it will show you a list of installed mods.


    2. The PB Mod settings containing now a section for an interactive remote shell:
    Code:
    # Part of pbSettings.json. Use 0.0.0.0 to allow connections from other devices
     "shell": {
      "enable": true,
      "ip": "127.0.0.1",
      "port": 3333
     },
    
    If enabled, you could send arbitrary python commands to a running PB game. If your familiar with Civ4 modding, this gives you full control over your PB games.
    Here are some examples:

    a) Toggle Pause in PB game.
    Code:
    > cd PBStats/PBs
    > python Pyconsole 3333
    >       pause
    
    b) Call Python code directly to finish turn for Playerid X .
    Code:
    > python Pyconsole 3333
    >       print(gc.getPlayer(X).getName())
    >       gc.getGame().setActivePlayer(X, False)
    >       CyMessageControl().sendTurnComplete()
    >       gc.getGame().setActivePlayer(-1, False)  # Default for PB Host is -1
    >       bye
    
    (Note that above code is valid for Civ4:BTS, but had some side effects. I've added the
    function CvInitCore:sendTurnCompletePB(PlayerTypes ePlayer)
    to avoid this effects.)

    c) Use predifined macro for above task. (Type 'help' to get a list of all commands)
    Code:
    > python Pyconsole 3333
    >       pb_end_turn X
    >       bye
    
    d) List saves, edit config and restart server with new save.
    Code:
    > python Pyconsole 3333
    >       list PlayerX
    >       list Logoff.*P0
    >       config edit save/filename=Logoff_P0_PlayerX_T1507924088.CivBeyondSwordSave
    >       config show
    >       pb_start
    >       bye
    
    e) List 10 saves, and restart with second save of list
    Code:
    > python Pyconsole 3333
    >       list 10
    >       load 2
    >       pb_start
    >       bye
    
    The shell based on Pythons Cmd.cmd class and could easily extended by further commands.

    Here a screenshot where I've used both tools under Windows 7:



    3. New PB executable (tests/Civ4BeyondSword_Pitboss_Zulan.exe):
    Based on Civ4BeyondSword_Pitboss2014.exe, but fix save loading issue under Wine.
    If some of your saves not load, give this a try.


    4. Pylint_for_Civ4:

    You probably know the website PythonAPI.
    Some functions declarations on this page are wrong and a few (i.e. CyPitboss class) are missing. Moreover I could not use the web information for static code analysis of my text editor.

    Thus, I've created a packages of python files with stubs of all related functions of Civ4.
    You could use this with Pylint on Linux, Mac or Windows.
    Provides helpful list of classes/function stubs for Pylint. This gives Pylint
    or your text editor the capability to detect existing Civ4:SDK functions, etc.

    5. The code base of PBSpy web interface was updated on Django 1.11 and the installation description contain more details. (Installation tested with Ubuntu 16.04)
     
    Last edited: Dec 19, 2017
    45°38'N-13°47'E likes this.
  14. 45°38'N-13°47'E

    45°38'N-13°47'E Chieftain

    Joined:
    Jun 7, 2008
    Messages:
    5,748
    Location:
    Just wonder...
    Amazing, I wish I had more time to try everything. Thanks for your work Ramkhamhaeng.
     
  15. Ramkhamhaeng

    Ramkhamhaeng Chieftain

    Joined:
    Feb 24, 2014
    Messages:
    140
    Due the Intel-Bug patches, the PBSpy server will be rebooted. Thus, the interface will be down for a small amount of time.

    Edit: Done.
     
    Last edited: Jan 10, 2018
  16. DarkLunaPhantom

    DarkLunaPhantom Chieftain Supporter

    Joined:
    Feb 4, 2013
    Messages:
    348
    The number of players in a game on pbspy web (on the list of all games) is wrong. I think that it currently displays all players the game has stored (including previous games if the id was reused) instead of the number of players in the current game.
     
  17. Ramkhamhaeng

    Ramkhamhaeng Chieftain

    Joined:
    Feb 24, 2014
    Messages:
    140
    Well, the displayed number scaled by the number of admin's defined for each game :( (ugly join of some sql tables…).
    Zulan found the reason and fixed it :)
     
    DarkLunaPhantom likes this.
  18. Ramkhamhaeng

    Ramkhamhaeng Chieftain

    Joined:
    Feb 24, 2014
    Messages:
    140
    I've published a new version, PB Mod_v8, of the Mod. Main change is an integrated ingame updater for mods. :)
    Lets say, you want share your latest changes of a mod. Normally, you would give your users an zip archive with the changed files. Now, the user could ingame search for such update zips and Civ4 will unzip them.
    Well, a few years to late to be an useful feature but I still like it ;)

    As minimal example, I've created an extra Mod, which can be found here: https://github.com/YggdrasiI/PBStats/tree/master/tests/Updater
    'Mods/Updater' contain the mod changes and 'server' shows the required structure of the backend.

    Known Issues/Limitations:
    • Sometimes, the Updater screen is not visible at startup. (Reselecting the Window solves is.)
    • Unzipping under wine fails (Reason unknown.)
    • https is not supported (Problem of Civ4's Python 2.4 version, I assume)


    P.S. An other interesting coding part is the implementation of CyGame().getModPath(). For years, I had problems to detect the correct mod folder (i.e. [BTS]/Mods/[Mod name] or [My Games].../[Mod name]), because I does not want guess which folder could be the right one...
    Other Modder solve it by editing an Python-file, but this changes the checksums...
    Finally I realise that I just could ask the OS from which destination CvGameCoreDLL.dll was loaded :)
     
  19. Ramkhamhaeng

    Ramkhamhaeng Chieftain

    Joined:
    Feb 24, 2014
    Messages:
    140
  20. DarkLunaPhantom

    DarkLunaPhantom Chieftain Supporter

    Joined:
    Feb 4, 2013
    Messages:
    348
    pbspy supports only a limited range of in-game "years", i.e. month displayed can only be January or July as seen in:
    Spoiler :

    Code:
    def parse_year(year_str):
        try:
            (year, qual) = year_str.split()
            year = int(year)
        except ValueError:
            (month, year, qual) = year_str.split()
            if month.lower().find('jan'):
                imonth = 1
            elif month.lower().find('jul'):
                imonth = 7
            else:
                raise ValueError('Failed to parse month part of date')
            year = int(year) + 10000 * imonth
    
        if qual == 'AD':
            return year
        elif qual == 'BC':
            return -year
        else:
            raise ValueError('invalid year suffix')
    

    However, marathon game speed in BTS uses other months resulting in erroneous information being displayed. A lot of mods also use other months or some completely different calendars.
    Is there a reason why the year info is transmitted like this instead of using the full calendar string displayed in game?
     
    Last edited: Oct 10, 2018

Share This Page