[BTS] Help with Project Structure and Enums

[SKRYMr]

I am just starting to mod Civ4 in order to implement a Reinforcement Learning agent from a 2009 paper and try to improve upon it as a project for an exam. The code for the paper was written in C++ and embedded in the SDK instead of using the Python API so for now I am mostly working with the C++ codebase.
I would consider myself an expert Python user and I have used C++ sporadically in the past with Cython bindings but due to time limitations and my unfamiliarity with older versions of C++ and Python I have some questions that I'd rather have answered by an expert modder than trying to figure everything out by myself.
My main concerns are as follows:

1. Is there a cheat sheet or a way to generate one for the various Enums in the game and how they relate to actual human-readable names? I am struggling to fully understand the logs I'm producing because everything is encoded as an int and not very readable, I know that these are filled in at runtime by reading the corresponding XML files but does anyone have a cheat sheet for the base BTS civs and leaders?
2. On the same note, I'm trying to use some available functions like

   GC.getInitCore().getLeaderName(PlayerTypes eId)

   that I expect would produce the name of the leader of the PlayerID I pass to it, but instead gives me the letter corresponding to the team name for some reason. Other similar functions baffle me with their responses. Am I interpreting/using these wrong?
3. Finally I've seen the Cython wrappers and the corresponding Python files but I can't find where they are called in the C++ code. Also all the tutorials show how you can implement Python scripts that make use of the exposed C++ variables and functions but is it possible to do the other way around? Not only is Python quicker to develop in and more suited to write Machine Learning algorithms but also a lot of the interface functions that I would like to use for code readability like the ones mentioned above are already implemented in the Python API because they are used to generate the interface. How would I go about calling Python functions and classes from the C++ SDK?

Thank you all for the work you've already done on this game, it's amazing to see this level of support and community still active more than 15 years after the game's release.

 

[Nightinggale]

1:
enums are defined in CvEnums.h. Some are filled out to match types from xml and others aren't. Those, which are filled out are way more friendly towards the debugger as UNIT_WARRIOR(0) is way more readable than just 0. I added a perl script to WTP to read xml data at runtime and build C++ headers to fill out data like this. It is however not designed to be easily moved to another mod, but you are welcome to do so if you want to. You could also write a similar script from scratch in python if you like. It's just reading all Type tags so it's not like it's a super complex task.

2:
It returns a CvWString. Each character in that one is 2 bytes. If you read it as CvString, then it's assumed characters are 1 byte. Being little endian, it means say George Washington becomes ['G', NULL, 'e', NULL, ....] and as such you end up with the name G. Common pitfall when you work with both CvString and CvWString. You can likely get around it by doing

CvString name = CvString(GC.getInitCore().getLeaderName(PlayerTypes eId));

3:
Look at CvDLLPython.cpp. DLLPublishToPython() is a DllExport function, which means it can be called from the exe. The exe will call it once during startup (prior to main menu) and then again if you change a .py file while the game is running.

I'm not sure how much of a tutorial exist for exposing variables, but copying the approach of vanilla seems to work quite nicely. One thing to be aware of is the return policy. If you return a pointer to python, then you need to set if python should take ownership for it or not. If python takes ownership, then python is responsible for freeing the memory once it is done using it. If not, then python assumes C++ to keep a pointer to it. Both cases are useful, like C++ has ownership of the classes with xml info while you can generate say an instance of CyCity, which is given to python without anything in C++ to keep it.

As for the other way:

#include "CvGameCoreDll.h"
#include "CyArgsList.h"
void Python::XML::editorScreenDragOn(int source1, int source2, int dest1, int dest2)
{
    CyArgsList argsList;
    argsList.add(source1);
    argsList.add(source2);
    argsList.add(dest1);
    argsList.add(dest2);
    gDLL->getPythonIFace().callFunction(PYScreensModule, "editorScreenDragOn", argsList.makeFunctionArgs());
}

That's some code I wrote at some point. It calls "editorScreenDragOn" in PYScreensModule. PYScreensModule is defined in CvDefines.h and it's just the string of the python module/file/class in quetion. You obviously don't have editorScreenDragOn as it's my own custom code.

 

[SKRYMr]

Perfect! This is exactly what I needed, thank you so much.
As far as the enums are concerned, is there a way to tell which ones are filled out and which aren't without just trying them all? I couldn't find the part of the code responsible for this.
If I understand correctly, you wrote a script to read the XML data while the game is running and then write headers for C++ with the Enums prefilled, then built the mod again with those headers?

 

[Nightinggale]

As far as the enums are concerned, is there a way to tell which ones are filled out and which aren't without just trying them all? I couldn't find the part of the code responsible for this.

The filled ones are filled manually and there is no way to know which ones that is without looking through the header file. Even worse if you alter xml without updating the header, then you have introduced a bug. Vanilla is full of "just keep these two unrelated parts in sync. Failure will cause bugs". It's not really a design choice I would have made, but vanilla is vanilla and that is what we got to work with.

If I understand correctly, you wrote a script to read the XML data while the game is running and then write headers for C++ with the Enums prefilled, then built the mod again with those headers?

No, the script is called by the makefile, which means it's a compile time setup. I did that mainly for two reasons. One is to help the debugger while the other is to ensure that the enums, which are always set will not go out of sync with xml. Considering that I wrote a not insignificant part of the makefile used by most mods (I did base it on a previous one) it shouldn't be a huge surprise I'm one to do all sorts of tricks in the makefile to trigger even before the C++ precompiler triggers.

 

[SKRYMr]

No, the script is called by the makefile, which means it's a compile time setup. I did that mainly for two reasons. One is to help the debugger while the other is to ensure that the enums, which are always set will not go out of sync with xml. Considering that I wrote a not insignificant part of the makefile used by most mods (I did base it on a previous one) it shouldn't be a huge surprise I'm one to do all sorts of tricks in the makefile to trigger even before the C++ precompiler triggers.

That makes more sense! I'll take a look at your script and see if I can easily make it work for me but thankfully I don't plan on making any changes to the XML files.

 

[f1rpo]

I am struggling to fully understand the logs I'm producing because everything is encoded as an int and not very readable [...]

When I write to log files from the DLL, I lookup the CvInfoBase instance corresponding to the XML IDs I encounter, and call CvInfoBase::getDescription for their display text (loaded ultimately from the game text files in XML\Text). That's also how the logging function added by the Better BtS AI mod operates, e.g. in CvCityAI.cpp#L7885:

logBBAI(
		"      City %S (%d) hurries %S. %d pop (%d) + %d gold (%d) to save %d turns. (value %d) (hd %d)",
		getName().GetCString(), getPopulation(),
		GC.getUnitInfo(eProductionUnit).getDescription(0), iHurryPopulation,
		iPopCost, iHurryGold, iGoldCost, getProductionTurnsLeft(eProductionUnit, 1),
		iValue, iHappyDiff
);

The unit ID eProductionUnit gets mapped to a wide-character string here. Not everything in CvEnums.h corresponds to a Cv...Info class. There is some pretty-printing code in CvGameCoureUtils.cpp for some internals that lack proper game text but are sometimes useful in log files, e.g.:

void getDirectionTypeString(CvWString& szString, DirectionTypes eDirectionType)
{
	switch (eDirectionType)
	{
	case NO_DIRECTION: szString = L"NO_DIRECTION"; break;

	case DIRECTION_NORTH: szString = L"north"; break;
	// ...

Just to offer an alternative to approaches involving a script analyzing the XML files.

 

[SKRYMr]

I have one more quick question that I can't seem to solve, is there a way to specify which civilizations and leaders each player should get when playing a quickstart game?

To expand on why I need this: The authors of the original paper gathered data only for Frederick vs. Washington and Gandhi vs. Genghis Kahn. In order to reproduce their results and build upon them I would need to perform the same experiments which require playing approximately 100 games with each of these pairs, where the RL Agent is playing as the first leader and the computer AI is playing as the second one. Even if I remove all leaders except the two that I want from the XML the game will still assign civilizations randomly so approximately 50% of the time my model will play as the wrong civilization. I would need a way to tell the game to always assign CvilizationTypes 0 to PlayerTypes 0 and 1 to 1 but I couldn't find the code that does this at game start. Note that I have autorun = 1 in my .ini file so both players are considered AI players and setting the "bPlayable" tag to 0 in the XML file won't do anything.

 

[f1rpo]

Maybe you could hack it into CvInitCore::setLeader and setCiv. The game setup screens in the EXE call those functions. Other code might, too, but, if, in your mod, player 0 is only ever supposed to be leader X and civ Y, I guess it can't hurt to always change eCiv and eLeader through a switch(eID) block in those functions, i.e. to set those variables based on the player ID.

Hm, I've never actually tried AutoRun via CivilizationIV.ini; I've always been using the AI Auto Play mod instead. The standard version of that mod has the flaw that the handicap of the human player does not get set to the AI handicap when Auto Play is started, i.e. in a game started on Prince difficulty, the handicap of player 0 will remain Prince when Auto Play is started while the other civs play on the default AI player handicap Noble. A lesser problem is that player options set on the Options screen (Ctrl+O or via the opening menu) continue to apply – only to player 0 – once AI Auto Play starts. This is relevant e.g. for the "Workers Leave Forests" option. I've just run a quick test with AutoRun, and it seems to have the same issues. I've set a breakpoint in CvInitCore::setHandicap to check which handicaps get set. The handicaps are not an issue, I think, if Noble is chosen during game setup. Then all player handicaps will be Noble (except for the Barbarian player at Chieftain) and the game handicap (which applies AI freebies and discounts and affects Barbarian activity) is also Noble. (When playing on Noble, AI Auto Play works fine too.) I guess player options affecting AI behavior aren't really a problem either if player 0 gets completely novel AI behavior anyway (RL agent). Which is to say, the RL code probably doesn't check PLAYEROPTION_LEAVE_FORESTS anyway. I guess you could just uncheck all "behavioral" options on the options screen to be on the safe side; or perhaps you have done so already. Can't think of other pitfalls with AutoRun.

Quick Start – isn't that a Civ 3 thing? Civ 4 has "Play Now!", but that doesn't immediately launch a game. Edit: Oh, it's in the INI file, I see. Another one I've never tried. Maybe the QuickStart handicap will only ever be Noble anyway; or, with QuickStart, the proper player handicap will actually get assigned even if that's not Noble. Presumably, when avoiding the game setup screens through QuickStart, player 0 never gets set to 'human' in the first place. I did just set a breakpoint in setLeader, and that function does get called when using QuickStart. So setting the desired leaders there might indeed work.

 

[SKRYMr]

Hacking the enums directly into CvInitCore::setLeader and CvInitCore::setCiv worked! I was afraid it would mess up something later on but it looks fine.

As far as the rest is concerned, inside the .ini there is a bunch of settings that are applied for QuickPlay games, one of which is the handicap but I'm not sure if this is applied only to human players or also to the AI. The RL Agent actually makes use of a lot of the default AI's code to shrink the action space and perform only macro-decisions so it might be relevant enough to keep an eye on it, thanks for the heads-up.