Mod for Pitboss Games

[Ramkhamhaeng]

Hello Pitboss players and especially Pitboss hosters,

I've played Pitboss games for years and was bugged by some issues like the 'pause bug' and the 'upload bug' which probably all of you know.
Moreover, there existing minor issues like the high cpu usage of the Pitboss server and the manual reloading of saves.

Now, we currently testing a solution for most if these problems with a small drawback: The solution requires a mod.
The sources and install instructions can be found on https://github.com/YggdrasiI/PBStats

Please note that my modding colleague Zulan currently work on a Django based webfrontend, which would replace the php stuff, but this does not affect the Civ4 mod component of the project. The webfrontend is not required for all features but would be strongly recommended. You can use https://civ.zulan.net/pbspy if you do not want host it on your own webserver.

Fixed bugs and new features:

• Webfrontend which communicates to your PB servers over TCP/IP. The webfrontend was inspired by civstats.com, but allows the
  administration of games.
  
• Unbreakable pause during open diplomacy screens.
  Solutions:
  A) The pause can now be removed by the player over
  the F1 Screen. ( This screen can be reached per mouse if the diplo menu is open.)
  B) The web interface allows to pause/unpause the game, too.
  
• Reload of last savegame after server restart.
  (No boring clicking in the wizard required anymore.)
  
• Remove window freeze of Pitboss wizard due Gamespy shutdown.
  (This was solved by python in this mod. A more general solution is the usage
  of a modified executable, see test/Civ4BeyondSword_Pitboss2014.exe )
  
• Fix of the upload bug: It's now possible to close open network connections to clients which
  has already leaved the game. (This bug can cause extremely high traffic!)
  Look into test/fix_upload_bug for more instructions. The solution runs independently from this mod.
  
• Headless Mode. Disables the GUI of your Pitboss server and reduce the CPU usage. (Almost every cpu load (>90%)
  is caused due the redrawing of the GUI...) Please setup your game correctly before you disable the
  GUI. This option assumes that a savegame will be loaded at startup. Thus, enable and test the reloading of save games.
  
• Increase maximal number of players to 52.
  
• Add extra nation/player color combinations.
  
• Player password could be changed over webinterface.
• Extra-Executable which minimize the login time into PB games, see https://github.com/YggdrasiI/PBStats/tree/master/tests/GetSaveOverHttp/sources_v4
  Prebuild binary: https://pb.zulan.net/BTS_Wrapper_v9.zip
  More information (GER) at https://civ-wiki.de/wiki/Pitboss_(Civ4)#Login_der_Spieler_beschleunigen

If you are hoster and are interested in this solution please note that normal savegames without mod can be converted into saves with mod. Thus, if you like the solution you can use it for running games, too.


If you find a bug please inform us over Github (or here, but your EMail notifications are broken... )


Happy Civving,
Ramkhamhaeng

 

[Caledorn]

Thank you very much for this! As soon as I saw this, and that you have created a fix for the upload bug, I installed it right away on my hosting machine. I suspect much of the problems I've had with my hosting server crashing/hanging is because of those open sockets, so having a utility that monitors it and fixes those leftover open sockets is just brilliant.

 

[Ramkhamhaeng]

Hello Caledorn,

I'm glad to hear that this program can help you.
This anGnoying bug can kill the strongest server because the load increases linear. One other Pitboss hoster has give me feedback and wrote that it helps him to reduce the hosting problems.
(Example image of increasing network load)

Just one note related to some posts of you in other threads: An other problem still exists and it would probably bugging you. On Zulans Server the Save-Function fails sometimes and throws a python exception. Even if he disable the Python error pop-ups this window appears and blocks the whole program. Could you confirm this type of errors for your hosted games, too? I'm interested to know if this problem (or the probability of occurrence) is related to the changes of my mod.

While the detection program is handy, be aware of the fact that it is not an excuse to not Exit to Main Menu before you close Civ in the future.

Zulan and I strongly assume that this is just a long-time rumor which isn't true Alt+F4 is ok. Only a hard kill with the task manager (or unplugging the cable/disable the wireless lan) produces the upload bug.

 

[Caledorn]

That does indeed happen from time to time, yes. I get an error dialogue stating that a save error has occurred, and the Pitboss in question is unconnectable for the players. In those cases when this happens I have to manually quit the Pitbosses that crashed and reload the game from the latest functioning autosave (most of the time that isn't the last autosave generated causing the crash error, because the autosave that is generated when this error occurs has a tendency to be corrupt, or even size 0KB).

I suspect that a lot of the technical issues I've encountered with server freezes/hangs are related to the socket error that this program fixes. I have moved my games from the main OS on my hosting computer into a VMWare Virtual Machine to try to avoid the entire server hanging (I've had to resolve to using the power button to reset the server, which is less than ideal...).

One thing I noticed in the program though is that it sometimes takes a lot of attempts before it successfully closes the open socket down - I've had some outputs in the program where the same IP and connection is reported 30 times or more, over a span of 30 minutes or so before it finally closes the connection. However, it doesn't seem to be a problem except for the log file "spam", as the traffic used is a mere few bytes per second as opposed to thousands of kilobytes per second.

 

[Caledorn]

Oh, and as for your edit. I asked all players that play in my Pitbosses over at Realms Beyond to exit Civ without using Exit to Main Menu, and several players who have never (to my knowledge) given me the open socket issue generated it. They used various ways of exitting Civ badly, like Alt+F4, Exit to Desktop and even playing windowed mode and just closing Civ using the X button in the upper right corner.

So, I am 99% certain that not using Exit to Main Menu increases the risk of the error happening quite dramatically. However, sometimes even players who do close Civ by first doing Exit to Main Menu leaves an open socket.

There was one player in particular who caused me a great deal of headache more than a year ago, because it didn't matter what the player did - there was an open socket left after the player consistently in like 99/100 connections. The extremely few times there wasn't left an open socket was always a surprise to me. This suggests that router configuration, internal NIC drivers, and so forth may also play a part. However, in that specific player's case, the player was playing Civ inside a Virtual Machine environment (I believe the player used a Mac, and ran a Windows installation inside Virtualbox), so I wouldn't be surprised if that was the reason in that specific case.

 

[Ramkhamhaeng]

One thing I noticed in the program though is that it sometimes takes a lot of attempts before it successfully closes the open socket down - I've had some outputs in the program where the same IP and connection is reported 30 times or more, over a span of 30 minutes or so before it finally closes the connection. However, it doesn't seem to be a problem except for the log file "spam", as the traffic used is a mere few bytes per second as opposed to thousands of kilobytes per second.

This is a new information for me. Was other players connected to the PB server during the phase where the disconnection fake fails?(*) The analysis of the UDP packages was done only for one connected player.
It could be possible that the syntax for the fake package changes if other players are connected because the packages includes some type of message counter.

This suggests that router configuration, internal NIC drivers, and so forth may also play a part.

Oh, thanks for your experience. It could be a race condition between threads. Difficult to track.



(*) Well, it would be possible to add this information into in the log of my program, but this would require some bigger changes of the code. Thus, I ask you firstly

 

[Caledorn]

Keep in mind that I have 6 Pitbosses running at the moment. In the active hours I more or less always have someone connected to one of the games - frequently two or more to one Pitboss, especially in the case of Realms Beyond Pitboss 18 with it's 33 players. In one case, which I was paying attention to while this was happening though, there was no one else connected to any of the games.

I have found absolutely no discernible pattern to the open socket issue, except that it happens more frequently when players disconnect without going to the Main Menu first, so that's about all I can contribute on the subject. Your program is vastly helpful though, so I am really grateful for the effort you have gone through to provide it to us Pitboss hosters!

 

[Ramkhamhaeng]

Hello pitboss hosters,

I want share some information about a connection problem some users reported.

Issue: A user log into the game, but the leader selection screen is empty and the user can not pick a nation. Sometimes, the leader list appears after a few minutes.

Reason: Zulan and I checks the network traffic between server and client and notice a very big UDP packet, which contains the leader names and civ descriptions

If this package exceeds 1500 Bytes (this value can differ), it will be fragmented. Some ISP filtering out fragmented packages and the sending of the list fails!


Solutions:
• Use fewer players
• Reduce the length of player names and civ descriptions. For 52 player we limit the sum of both values on 5. (You can translate this to your player size.)
Note that it's not possible to use an empty civ description. It will be replaced by the default value.
This solution has the disadvantage that alle Names in the Game (i.e. Log) will be very short, too.

• Both solutions was no real option for our current 52-players-game and we decide to reduce the number of characters if and only if a player connects to the server.

This approach can be found the new version of this Mod (Branch: PB Mod_v2).

With Mod, the login screen will be looks like that:
Not good, but solves the issue

 

[2metraninja]

WOW, man, where is this 52-players game? Can I follow this somehow? And where is this community with 52 active civ4 pitbossers? It is worth checking for sure.

And to not be 100% off-topic, I will say I never encountered such a problem, given I only played in 18 players games at most.

 

[Ramkhamhaeng]

Oh, the admins fixes the db error . Now, I can reply without creating a new account.

WOW, man, where is this 52-players game? Can I follow this somehow? And where is this community with 52 active civ4 pitbossers? It is worth checking for sure.

Yes, it's optimistic to start a game with this huge number of players. The organization overhead and probability of quitters is high.

You can follow, but it's played by Germans (many Germans ) and all stories of this game are written in German
Moreover, the score list is very boring/uniform because we're only in round 6.

And to not be 100% off-topic, I will say I never encountered such a problem, given I only played in 18 players games at most.

Yes, the error is very improbable. In normal games I was only affected a few times in many years.

 

[civliam]

Thank you very much for your mod! I tried setting it up because of the 100% cpu usage when I'm using PitBoss on my server, but there seem to be issues:

The PitBoss UI won't show up, there's an immediate "PB Server was quitted" and the server keeps on restarting because of the loop:

D:\PBStats\PBs>startPitboss.bat
..........................................................
 Select Game
..........................................................
 1 - Pitboss A
 2 - Pitboss B
..........................................................

Type 1, 2,... to select the game and then press ENTER: 1
App::Active
new mod name is Mods\PB Mod_v2\
Init Progress - step 1 / 9 - Init File Catalog
Deleting file CACHE
Done reading INI
Init Progress - step 2 / 9 - Check XML
[...]
Init Progress - step 8 / 9 - Init Fonts (4.47s)
Init Progress - step 9 / 9 - DONE
App Init start elapsed time = 4.48, uncached XML<BR>

CvEngine::shutdownGraphics

CVRESMGR: shared 0, loaded 4, copied 0, cloned 0, failedLoads 0
CvEngine::shutdownGraphics

Cleaning up Properties:
PB Server was quitted. Restart server.
App::Active
new mod name is Mods\PB Mod_v2\
Init Progress - step 1 / 9 - Init File Catalog
Deleting file CACHE
Done reading INI
Init Progress - step 2 / 9 - Check XML
Init Progress - step 3 / 9 - Init MP / Voice
[...]

Unmodified PitBoss works just fine...

I did not enable headless mode or auto loading of a save file in pbSettings.json.
I also tried using the modified .exe Files, no change.

Configuration of startBitPoss.bat:

rem ### Begin of configuration ###

SET ALTROOT_BASEDIR=D:\PBStats\PBs
SET CIV4BTS_PATH=D:\SteamLibrary\SteamApps\common\Sid Meier's Civilization IV Beyond the Sword\Beyond the Sword\

I tried using the latest version from the Git repsitory as well as the tagged V1 release version.

Operating system is Windows 8.1U1 and Civ4 BtS is latest version available on Steam.

Can I provide additional information (log files, etc..) to help solving this issue?

 

[Ramkhamhaeng]

Hello Civliam,

yes, the first setup can be frustrating
After the setup, it's very comfortable. Currently, manage a game on a linux host over ssh (without X) and can reload a game in 10 Seconds Nevertheless some manual operations are still necessary because the server still crashs sometimes...


Did you write the path of the ALTROOT-Folder into the Civilization.ini?

It's necessary to give my mod the information were it can find pbSettings.json
Edit PitbossSMTPLogin and add your folder.
Example

PitbossSMTPLogin=Z:\opt\Civ4\Pitboss\PBs\PB1

It's a good idea to use the latest version (master branch/ PB Mod_v2).
Note that the PB server can hang up (100% cpu usage) if he can not load the given save. I've prepend a check to search for the file, but other loading issues can not detected by my mod. Thus, be careful and wrote the right admin password for your savegame into pbSettings.json.

 

[Caledorn]

Hey, Ramkhamhaeng.

How does hosting Pitboss games work with Linux? What packages need to be installed? How does it work with various DLL mods?

In RB Pitboss 18 we use a custom Civstats portal (seen at http://pitboss.watto.no ) which is hosted on IIS. That might be a problem for us. But I assume the Pitbosses are more stable under a Linux environment, as Linux is far better at jailing individual processes unlike Windows?

 

[Ramkhamhaeng]

Hey, Ramkhamhaeng.
How does hosting Pitboss games work with Linux? What packages need to be installed?

Hello Caledorn,

the shortest answer would be 'sudo apt-get install wine' and install Civ 4 + BTS and update on 3.19'. Moreover, you can install a newer DirectX version and skip this step during the installation of Civ.

Look on https://appdb.winehq.org/objectManager.php?sClass=version&iId=10158 go get more information

It's hard to post a complete answer because it depends on the OS and wine version and almost everyone gets some error messages. But for each problem there is someone in the net which has already solved it
Moreover, I can not give a precice answer because I've setup the game a few years ago and my knowlege is out-dated

One useful (out-dated/obsolent?) hint could be the installation of msxml3 with winetricks.

wget http://www.kegel.com/wine/winetricks
sh winetricks msxml3

The user Redarg wrote here some notes: http://forums.civfanatics.com/showthread.php?p=6547872#post6547872

If Civ4 runs well, the PB Server can be started, too. To run the PB Server remotly, their exists several approaches. The simpliest would be the usage of VNC. Here is a description: http://www.civforum.de/showthread.p...-quot-Thread&p=2890020&viewfull=1#post2890020
An other way is Xvfb, but the setup depends more knowlege (I do not know all steps).

Note that the end of Gamespy also affects the PB-Server wizzard. (Ops, why did I wrote this. YOU hosted a NATNEG server )

How does it work with various DLL mods?

It depends. Normal Mods can be start without problems. Unusual DLL mods could require some 3rd party libraries. (Which can install in wine, too.)
Note the unintuitive notation for the mod-argument!

wine "${CIV4BTS_EXE}" mod= "${MODNAME}"\" /ALTROOT="${ALTROOT_W}"

In RB Pitboss 18 we use a custom Civstats portal (seen at http://pitboss.watto.no ) which is hosted on IIS. That might be a problem for us. But I assume the Pitbosses are more stable under a Linux environment, as Linux is far better at jailing individual processes unlike Windows?

Nice web-interface. It's funny to see other civstats clones.
To push the log file for the 'Civstats portal' to your webserver there exists
four ways:
1) Made the logfile accesable over Lan/Net (samba or rsync-Loop)
2) Write the log directly on your Win machine (i.e. netcat )
3) Replace your ISS with a linux webserver (i.e. Apache2)
4) Change your mod and push the data directly over some kind of interface ( thats the approach of our mod )

 

[Ramkhamhaeng]

Hello guys,

we've updated the Mod to version 3 (PB Mod_v3). The webserver now
supports the following operations:

Pause/unpause game
Save game
Load games (with same admin password as the running game)
Force end of turn
Change turn timer
Set timer of current round
Send game chat message
Change message of the day (MotD)
Change number of characters in PB login screen (Fix network issue)
Reduce number of characters in signs (Fix save corruption under wine in your current game)
Automatic save game loading at startup (Yes/No)
Hide Pitboss window (Yes/No)
Change password of player
Change colorset of player
Display information for up to 52 players
Several filtering options for log events.
E-mail notifications for new rounds (only in the python based webinterface)

The Github code still contains two webinterfaces which can be connected with
the modded PB server.
The second/newer webinterface, called PBSpy, is written in Python/Django and supports all
operations of the first interface, now. It can be found in the civdj-folder.
Example page: http://civ.zulan.net/pbspy.

The first approach, called PBStats was written in PHP can be found in the web-folder.
Example page: http://civ.zulan.net/pb/games.php


If your interested to setup a PB game with this mod please inform your players
to install the mod into [Game Installation Dir]\Beyond the Sword\Mods\ and not [...]\My Games\Beyond the Sword\Mod
Only the first variant will be accept by Civ same mod version like the PB Server one.

 

[Caledorn]

Is there a problem with the NATNeg server? I see several people who are reporting problems in multiple games.

 

[Ramkhamhaeng]

Hello Caledorn,

yes, there was a problem with the server. Now, the NatNEG is running again.

 

[Caledorn]

Hello Caledorn,

yes, there was a problem with the server. Should be solved by Zulan, now.

Thanks!

 

[Ramkhamhaeng]

Ops, posting too early...

 

[Ramkhamhaeng]

Hello guys,

if you contributing in an advanced Pitboss game with long loading times, this could be interesting for you. Edit The solution does not require any mod.

Normally, Civ4 just sends ten packages with 10kB payload per second. This limits the bandwidth during the save game transfer to an ancient rate
We developed a solution which release the file transfer from Civ4's internal mechanism to libcurl (http downloader).

As player, you just need to use the new executable and some DLLs. Both can be found here: https://github.com/YggdrasiI/PBStats/tree/master/tests/GetSaveOverHttp
The source code is also included.

As pitboss host, you have to setup a webserver which provides the save games over http(s).
Moreover, the ALTROOT-Argument of your PB Server requires a special syntax (see Readme.txt).


Regards Ramkhamhaeng

P.S. Linux could replace the given libcurl.dll with a version from http://curl.haxx.se/download.html which uses openssl.

P.P.S. It exists an other solution which does not need any changes, but a bond of trust between you and your pitboss host:
Assume that the pitboss host uses the folder Z:\PBs\myPB\Saves\pitboss\ for the save games.
If you mount this folder on your machine with exaclty the same path Civ4 will handle the saves like locale files.
The mounting could be done over several ways. Linux users could use sshfs or ftpfs.
Windows users could mount the pitboss folder with webdav (i.e to Drive X: ) and use mklink to create a link with the desired path ( mklink /D Z:\PBs\myPB\Saves\pitboss X: ).