python how-to

[davidlallen]

So far, except for a few tweaks by koma13, the python for Dune Wars is controlled and maintained by me. This thread is to explain how you can tweak some things in the python. I will try to keep it up to date, but if there are later rewrites, it may be that the instructions are slightly out of date.

At this time (1.5.3 release, mid-September 2009), 95% of the python changes are in one file, assets/python/DuneWars.py. There are some small hooks in CvGameUtils.py, Screens/CvMainInterface.py, and EntryPoints/CvRandomEventInterface.py. Koma13 has also added the homeworld advisor screen, which involves larger changes in CvMainInterface.py, new file MercenaryUtils.py, and a few lines in DuneWars.py. I will not try to describe those, but he may.

General architecture

DuneWars.py is built on top of the BUG event manager. So there are no changes in CvEventManager.py. Instead, DuneWars::__init__ calls BUG functions to register event handlers; see lines like "customEM.addEventHandler("improvementBuilt", self.onImprovementBuilt)".

One key event is onEndGameTurn. A lot of things happen within the python at the end of the turn. See the code for the latest information, but in general I have tried to make high level entry points with comprehensible names. Here is the current code.

Spoiler :

	def onEndGameTurn(self, argsList):
		self.Initialize()
		iGameTurn = argsList[0]
		self.Dequeue() # results of AI actions
		(iUnowned, iWormCount, iSpiceCount, iMaxStr) = self.Count()
		self.BlowAdd(iSpiceCount, false)
		self.SpiceDecay()
		self.SpiceBlow()
		self.WormAdd(iUnowned, iWormCount, iMaxStr)
		self.StormRun()
		self.AtreidesDraft()
		self.UnitLoop()
		self.PlagueCities()
		self.Terraform()
		self.Dequeue() # results of end of turn actions

Hopefully you can sort of guess from the names what is happening, and then go into the related function.

Sandworms and sandstorms

One easy python change is to make sandworms and sandstorms more or less frequent, or maybe remove them altogether. See routines WormAdd and StormAddSubtract. They are similar. Both maintain a relatively constant number of worms or storms. If worms or storms are killed or deleted for whatever reason, then these routines will add more to make sure the right number is maintained. You can change the goal number of worms or storms.

In WormAdd, find the line "iWormWant = (iUnowned / (3 * iUTPGA)) - iWormCount". This says the goal number of worms is the number of unowned tiles, divided by three times the handicap value, UnownedTilesPerGameAnimal. If you want more worms, make 3 a smaller number or edit the handicap xml file to decrease UTPGA. If you want no worms, edit this line to set iWormWant to zero.

In StormAddSubtract, find the line "iStormWant = (xmax * ymax / 400) - len(lStorms)". This says that the number of storms is equal to the number of plots on the map, divided by 400. If you want more storms, make 400 a smaller number. If you want less, make 400 a larger number. If you want none, set iStormWant to 0.

Spice frequency and decay

You may want to increase or decrease the amount of spice. You can make spice blows more or less frequent by editing function BlowAdd. Find the line "iBlowWant = (xmax * ymax / 50) - (iSpiceCount / 8)". Each tile of existing spice counts as 1/8 of a spice blow, and we want roughly one spice blow for every 50 tiles on the map. If you want more spice, make 50 a smaller number. If you want less spice, make 50 a larger number.

To change the spice decay rate, see function SpiceDecay. This function is a little more complicated. Basically it sets the decay rate to either 0.2% (plot is owned by a civilization following the spice industry civic), 2% (if there is a spice harvester) or 0.5% (default). This is the chance that a given spice tile will be removed, per turn.

Fresh water

Suppose you want to add/change which improvements spread fresh water. Today certain improvements and city buildings spread fresh water in adjacent plots. One building, the Reservoir of Liet, spreads fresh water in all the plots of the city BFC. (This particular part is not released yet, but planned for 1.5.3.)

To add or subtract an improvement, which spreads fresh water, edit the list dFreshI which is defined in the Init routine:

Spoiler :

		self.dFreshI = {
			self.GetCheckInfo("IMPROVEMENT_WINDTRAP")     : 1,
			self.GetCheckInfo("IMPROVEMENT_AQUABORE") : 1,
			self.GetCheckInfo("IMPROVEMENT_SHALLOW_WELL") : 1,
			self.GetCheckInfo("IMPROVEMENT_DEEP_WELL")    : 1
		}

Basically, just add another line with your improvement, or delete an improvement line if you don't want it to spread fresh water. You can see how this gets used in the event function onImprovementBuilt. It adds an invisible feature called WINDTRAP_WATER. This feature is basically a copy of the oasis feature from vanilla, but the art for it has fScale 0.001, which makes it really tiny and effectively invisible. So when an improvement is built, this invisible feature is added; when an improvement is destroyed or overbuilt, the invisible feature is removed.

 

[davidlallen]

Adding a new contract

There are a number of offworld contracts in the game. You can add your own, but there are a few steps to follow. I am writing this how-to some time after writing the actual code, so hopefully I did not miss any steps. You can follow any particular contract, such as CALADANIAN_WINE, by searching all the XML for it.

1. Create the bonus art and XML. You do not need any in game art for the bonus itself, since it never appears. But you do need a button and the XML.

2. Create the buildingclass and building and art. The key point is that the building should give three instances of the bonus, using the lines:

	<FreeBonus>BONUS_CALADANIAN_WINE</FreeBonus>
	<iNumFreeBonuses>3</iNumFreeBonuses>

3. The dialog to let you choose one available contract is actually created by the random event manager, using files assets/xml/events/Civ4EventInfos.xml and CivEventTriggers.xml. In EventTriggerInfos, find event EVENTTRIGGER_LANDING_STAGE and find its <Events> tag underneath. Add your event, EVENT_CONTRACT_whatever, just like the other events there.

4. In EventInfos, copy one of the sections <EventInfo> <Type>EVENT_CALADANIAN_WINE... </EventInfo>, about 80 lines, and paste it at the end of the file. Change the type and description. Also, the convention I have used for the iAIValue field is 100 for normal, boring bonuses, 500 for bonuses which give access to special units, and 1000 for bonuses which are highly valuable since they give access to special units *and* can only be chosen by certain civs. There is a race for these bonuses, so the AI should try to pick them first.

5. At the end of the EventInfo section, find the lines:

	<PythonCallback>DuneWarsWineDo</PythonCallback>
	<PythonExpireCheck/>
	<PythonCanDo>DuneWarsWineCan</PythonCanDo>

This is the connection to the next layer, the random event python. Change the "Wine" part to some short, obvious identifier which is not already used in the file.

6. In file assets/python/EntryPoints/CvRandomEventInterface.py, you will find all the functions like DuneWarsWineDo and DuneWarsWineCan. Hopefully it is obvious how to paste the four new lines you will need. Here are the lines for wine:

def DuneWarsWineCan(argsList):
	return DuneWars.ContractCan(argsList, "CALADANIAN_WINE")
def DuneWarsWineDo(argsList):
	return DuneWars.ContractDo(argsList, "CALADANIAN_WINE")

Copy these four lines and paste; change "Wine" to match the short identifier you already used in step 5, and fill in the correct name of the bonus. Note you do not need the BONUS_ prefix of the string, the next layer of code fills that in.

7. Seventh inning stretch. Almost there.

8. In file assets/python/DuneWars.py, search for function ContractStart near the end of the file. If your new contract is available to any civ, just add a line to the dContracts data structure matching Kindjal or Sapho Juice:

		self.dContracts = {
			iWine : -1, iThink : -1, iSardau : -1,
			iEcaz1 : -1, iEcaz2 : -1,
			self.GetCheckInfo("BUILDING_KINDJAL_CONTRACT") : -1,
			self.GetCheckInfo("BUILDING_SAPHO_JUICE_CONTRACT") : -1,

However, if you want to restrict the contract to just one civ, you need to do a little more. If you know python, it should be fairly obvious to follow the code of ContractStart; follow the example of Caladanian Wine and its related class variable, self.iCAtrei. The class variable is set in Init.

I can see the detail is a little lacking in this last step, but if you have made it this far, you should be able to modify the python OK. If anybody actually tries this and gets stuck, I will be happy to answer questions and eventually add more detail to the step.

 

[davidlallen]

As of 1.6 or so (I forget when exactly) I made some changes for fresh water. The design is in the "arrakis terraforming" thread. Basically, spice blows, sandworms and spice are forbidden to be near fresh water. The game does not have a good mechanism for storing information about fresh water on plots, so I had to fake it a little.

Certain improvements and buildings create fresh water as described in the first post above. When these appear, the python function RemoveSpiceNearby is called with a radius value, so different improvements or buildings may have a different radius.

The python routines to place spice blows and move sandworms call a subroutine NearFresh which looks in the nine adjacent plots. The sdk has a function pPlot.isFreshWater, which itself checks in the nine adjacent plots. So calling NearFresh calls isFreshWater nine times which redundantly checks 81 tiles for fresh water. The result from the function is true if there is a fresh water source within a two tile radius.

The Reservoir of Liet prevents spice within a three tile radius. This is an even worse hack. When the reservoir is built, function ReservoirFresh is called. This adds four sources of fresh water in the four vertical/horizontal adjacent tiles. If you plot this out on graph paper, you will see that because of RemoveSpiceNearby and NearFresh, this prevents spice and sandworms within a three plot radius. While it is inefficient, it is not fatal; a few redundant calls to check fresh water are needed, but the runtime is not too bad.

One bug relates to razing cities with a reservoir. When this happens, the four plots "might" have their fresh water sources removed. Of course they might have had a fresh water source like a well on their own, in which case the fresh water source should not be removed. The first problem is how to detect the destruction of a reservoir. The game provides onCityRazed, which obviously is called when a city is razed; but sometimes when a city is captured, buildings are destroyed. I do not think the game provides a way for the python code to know that. So, probably in almost all cases, the wrong thing happens when a reservoir is destroyed.

Here is what "should" happen. This is not a huge rewrite, but it does require a new sdk feature and then changes to the python code.

An *integer* field should be added to the basic plot structure in the sdk. CvPlot. This field records how many times fresh water is added to the plot. In many data structures in sdk and other programs, an int field is used where you would think a boolean is enough. Use an int. This way, you can add fresh water twice, remove it once, and see there is still fresh water there. The field should initialize to zero, and should provide function CvPlot::changeFreshWater(int iChange); call with +1 to add fresh water, -1 to remove it. Modify the existing function CvPlot::isFreshWater() to check the field on its tile instead of checking all adjacent tiles for a fresh water feature. Modify the function to add and remove features to add and remove the flag from adjacent tiles. Test, by removing a feature and making sure the fresh water goes away.

Once the base sdk changes are made, NearFresh will be slightly more efficient, but it will not need to be changed. The key point will be that you can now write python to add a well, which only puts fresh water in its own plot, or to add a reservoir which adds fresh water to all plots within a radius of 2, and everything else will work. There may still be a problem with removing fresh water when a reservoir building is individually destroyed, but razing will work correctly.

 

[keldath]

awesome idea david,

you put a lot into these posts, i salute you for the effort.

 

[Ahriman]

Absolutely, very nice documentation. I just hope it doesn't mean you're planning on leaving any time soon

Watch out for buses!

 

[davidlallen]

I am not planning on leaving, but as you notice I have not spent much time adding things since December. Now that deliverator is "armed and dangerous" on sdk changes, it may be helpful information.

 

[Deliverator]

"armed and dangerous"

Dangerous being the operative word.

 

[Deliverator]

An *integer* field should be added to the basic plot structure in the sdk. CvPlot. This field records how many times fresh water is added to the plot. In many data structures in sdk and other programs, an int field is used where you would think a boolean is enough. Use an int. This way, you can add fresh water twice, remove it once, and see there is still fresh water there. The field should initialize to zero, and should provide function CvPlot::changeFreshWater(int iChange); call with +1 to add fresh water, -1 to remove it. Modify the existing function CvPlot::isFreshWater() to check the field on its tile instead of checking all adjacent tiles for a fresh water feature. Modify the function to add and remove features to add and remove the flag from adjacent tiles. Test, by removing a feature and making sure the fresh water goes away.

I've got this solution working. The code needs some tidy-up/refactoring, but it works. This is how I did the "Modify the function to add and remove features to add and remove the flag from adjacent tiles" bit.

CvPlot::setFeatureType:
// Deliverator
// If changing from no fresh water feature to having one, then add fresh water to this tile and surrounding
		if (((eOldFeature != NO_FEATURE) && (GC.getFeatureInfo(eOldFeature).isAddsFreshWater())) && 
			  ((getFeatureType() == NO_FEATURE) || !(GC.getFeatureInfo(getFeatureType()).isAddsFreshWater())))
		{
			for (iDX = -1; iDX <= 1; iDX++)
			{
				for (iDY = -1; iDY <= 1; iDY++)
				{
					pLoopPlot = plotXY(getX_INLINE(), getY_INLINE(), iDX, iDY);

					if (pLoopPlot != NULL)
					{
						pLoopPlot->changeFreshWater(-1);
					}
				}
			}
		} 

// If changing from having a fresh water feature to not having one, then remove fresh water from this tile and surrounding		
		if (((eOldFeature == NO_FEATURE) || !(GC.getFeatureInfo(eOldFeature).isAddsFreshWater())) && 
			  ((getFeatureType() != NO_FEATURE) && (GC.getFeatureInfo(getFeatureType()).isAddsFreshWater())))
		{
			for (iDX = -1; iDX <= 1; iDX++)
			{
				for (iDY = -1; iDY <= 1; iDY++)
				{
					pLoopPlot = plotXY(getX_INLINE(), getY_INLINE(), iDX, iDY);

					if (pLoopPlot != NULL)
					{
						pLoopPlot->changeFreshWater(1);
					}
				}
			}
		} 		
		// Deliverator

The key point will be that you can now write python to add a well, which only puts fresh water in its own plot, or to add a reservoir which adds fresh water to all plots within a radius of 2, and everything else will work.

I'm a bit confused by how I can put fresh water just in a single plot purely from Python with this solution. If I add the FEATURE_WINDTRAP (Water Access) to a plot then this new code set up will mean that there is Fresh Water in that plot and the 8 surrounding tiles.

Do you mean exposing the changeFreshWater function to Python, and then adding or removing the Fresh Water from individual plots that way?

 

[davidlallen]

Do you mean exposing the changeFreshWater function to Python, and then adding or removing the Fresh Water from individual plots that way?

Yes. Also, maybe add an sdk function like changeFreshWaterInRadius, with a radius defaulting to 1. You could pass zero for a well, or 2 for a reservoir.

 

[Deliverator]

I want to try and make harvesters being eaten by worms more dramatic. To do this, I'd like the worm to fortify in the tile when it kills a harvester and then perhaps be removed the following turn.

I have tried to change WormAI in DuneWars.py to set the worm to fortify like this:

		if self.DamageImprovement(pPlot, "sandworm", 100):
			# 1.4.8: die after destroying improvement
			#self.Queue(pUnit, -1)
			pGroup = pUnit.getGroup()
			pGroup.clearMissionQueue()
			pGroup.pushMission(MissionTypes.MISSION_FORTIFY, 0, 0, 0, false, false, MissionAITypes.NO_MISSIONAI, pPlot, pUnit)
			pUnit.finishMoves()
			return true

I got the fortify code from here. This code doesn't seem to do anything though. The worm just sits in the tile in sand-lump form. Any ideas?

 

[davidlallen]

It looks ok to me. If you add a worm to your own team with WB and use the fortify button, does it change? If you have chipotle debug mode on, then some combination of shift, alt, ctrl while hovering over any unit will tell you its current mission.

I have thought of using the actual pillage action. Since DamageImprovement is doing basically the same thing as pillaging, you could trying pushing a pillage mission instead of doing DamageImprovement and then fortifying. I'm not sure if there is a pillage animation for the worm already, but I am sure there could be

Another thing we discussed is that the first time in a game that this happens to a player, they could get a special popup. The code is there to create the popup, and firing it is easy. You'd need to store something on player scriptdata to make sure it only happens once.

These popups are associated with random events, that normally come from xml/events/Civ4EventTriggerInfos.xml. DW uses a couple already, for choosing mentat specialty and choosing the offworld resource when a landing stage is built. Please search for initTriggeredData in the python to see this. The events in the xml can have image files associated with them, so you could put a nice dramatic screencap of a worm destroying a harvester from the movies.

 

[Ahriman]

Another thing we discussed is that the first time in a game that this happens to a player, they could get a special popup. The code is there to create the popup, and firing it is easy. You'd need to store something on player scriptdata to make sure it only happens once.

These popups are associated with random events, that normally come from xml/events/Civ4EventTriggerInfos.xml. DW uses a couple already, for choosing mentat specialty and choosing the offworld resource when a landing stage is built. Please search for initTriggeredData in the python to see this. The events in the xml can have image files associated with them, so you could put a nice dramatic screencap of a worm destroying a harvester from the movies.

I still like this idea. I'm not sure that anything else is necessary.

 

[Deliverator]

It looks ok to me. If you add a worm to your own team with WB and use the fortify button, does it change?

Yes, and it looks cool because the fortify animation is the one where the worm breaks through the sand, hence why I'd like to use it.

If you have chipotle debug mode on, then some combination of shift, alt, ctrl while hovering over any unit will tell you its current mission.

Using chipotle CTRL-SHIFT, the worms show up with MISSION_MOVE_TO when they are close to harvesters, but even with the fortify code added, I never see MISSION_FORTIFY. Seems to be something that works in theory but not practice. What I really want is for the worm to be set to the fortify mission on the turn it enters the harvester tile so that it fortifies in the same turn that it destroys the harvester.

I have thought of using the actual pillage action. Since DamageImprovement is doing basically the same thing as pillaging, you could trying pushing a pillage mission instead of doing DamageImprovement and then fortifying. I'm not sure if there is a pillage animation for the worm already, but I am sure there could be

AFAIK there isn't a separate Pillage animation. The game engine uses the same Strike animation that is used in combat. For worm harvester attack, I guess this would be better than the current situation where the worm doesn't even break the surface, but I still think that fortify would look better if we can make it work. The strike animation has the worm already above the sand, whereas the fortify one features the worm breaking through the sand as in this screenshot:

Spoiler :

I'm all for having the one-time popup too. I've had candidate images ready for the popup for a while.

Spoiler :

If the in-game action could mirror the popup that would be great.

 

[Ahriman]

My purely speculative guess is the problem is, you can only fortify when you have spare movement points. If you don't have any movement points spare, you can't fortify (you can issue a fortify command, but it actually gives a sleep command, followed by fortification next turn). So maybe in between somehow you need to add a "hunting" promotion that gives +1 movement point to the worm, so that it can use that movement point to fortify, and then disappear next turn?

 

[davidlallen]

Using chipotle CTRL-SHIFT, the worms show up with MISSION_MOVE_TO when they are close to harvesters, but even with the fortify code added, I never see MISSION_FORTIFY.

Well, you could do a bigger experimental hack by just giving a 90% chance for the worm to fortify, instead of moving at all. Then after a few turns, you should see most of the worms on the map are fortified. But Ahriman is probably right; the worm moves into the plot before it destroys the harvester, so its turn is over. I don't think the fortify is queued, it may be just discarded. If that is the problem, you could try calling something like CyUnit.setMovesRemaining(1) after the move, and then push the fortify mission. Or, maybe there is a way to just play the fortify animation, regardless of whatever else is happening? I am not sure how to do that.

 

[Deliverator]

Got this working. The worm now fortifies in the tile the same turn that it destroys the harvester and is removed the following turn. Previously, I was using the wrong technique for force fortifying - the NotifyEntity way works. I had to use a bit of a hack to flag the unit for deletion - there may be a nicer way.

		# delete UnitAITypes.NO_UNITAI units
		if (pUnit.getUnitAIType() == UnitAITypes.NO_UNITAI):
			self.Queue(pUnit, -1)	
			pUnit.finishMoves()
			return true			
		if self.DamageImprovement(pPlot, "sandworm", 100):
			# fortify after destroying improvement - flag for deletion using UnitAITypes.NO_UNITAI
			pGroup = pUnit.getGroup()
			pGroup.clearMissionQueue()
			pUnit.setUnitAIType(UnitAITypes.NO_UNITAI)
			pGroup.setActivityType(ActivityTypes.ACTIVITY_SLEEP)
			pUnit.NotifyEntity(MissionTypes.MISSION_FORTIFY)		
			pUnit.finishMoves()
			return true

A small change but I quite like it. If I make a better Fortify Idle animation then we can have the worm swaying and moving a little bit which will look even better.

 

[davidlallen]

Ahriman has requested changes to the tleilaxu plague mechanism many times, and the bene gesserit training mechanism may need changing too. The code "may" be obvious, but here are some pointers anyway.

For plague, at startup, the class variable iPTlei is set to the player index of the Tleilaxu player. Already, we are in a little bit of trouble, because there could be several Tleilaxu players in a large game. It may be worthwhile to consider this in a *total* redesign, but a point redesign could ignore it.

The main routine is SpreadPlague. This is called from onCombatResult to spread as a result of combat. It is also called from onEndGameTurn calling UnitLoop, for each saboteur unit and each unit with plague. When SpreadPlague is called, it means that every unit in the plot is exposed, and will catch it under some conditions. If the owning player is not at war with Tleilaxu, plague is removed; otherwise it has a 25% chance to catch it. This routine is where the bulk of Ahriman's proposed changes would go, for checking unitcombats.

Some plague handling is also done in onEndGameTurn calling PlagueCities. This loops all the cities. If there is a cure building (hospital) all units are cured. If not, and any unit has plague, it puts the invisible plague building that causes unhealth. If no unit has plague then the unhealth building is removed. There is no provision for removing the unhealth building in the event of peace; but since peace removes plague for all units, that should happen automatically.

One suggestion is that a unit should have a chance of losing plague on its own and being immune for 10 turns. This is a good idea, but a little hard to implement. The easy part is to create a new promotion called "Plague Immunity" or similar. It is also relatively easy to add the promotion. The problem is removing it after ten turns. I have used pUnit.scriptdata for different random things, and there is no other convenient place to hang this information in python. The best approach may be an sdk change to add a field for promotion countdowns. Some modpacks have this; basically you would initialize a counter on the unit, decrement all the counters once per turn, and when the counter reaches zero, remove the promotion.

For BG training, onEndGameTurn calls UnitLoop which calls WeirdExperience on every Sayyadina type unit. This builds a candidate list of all the appropriate units in the plot; a dictionary of allowable unitcombats is kept in class variable dSayyTrain. One candidate unit is picked. Then there are a bunch of if-statements to find what may be an appropriate promotion to add; if the unit has combat 3, then combat 4 is appropriate, etc. There may be 0, 1 or 2 appropriate promotions possible (combat or drill, either, or possibly neither for a maxxed out unit). One promotion is picked and then added.