Proposal: Custom Barbarian Spawn Python Module

[Baldyr]

Since there are several active Barbarian modding threads on these board I guess people really wanna be able to script this stuff. What if we designed our own custom barbarian spawn Python module as a community? It would be as easy as any XML file to edit by adding lines into a data structure at the top of the module.

What are the specifications? What parameters do you need? Like type of unit, in what game age, how many units, interval between spawns, random vs repeated intervals, in what type of terrain, that sort of thing. Do you need to be able to spawn Barbarian cities also?

Without promising too much, I could try to find the time to code this thing - if you guys take the time to test everything. What do you say? (At this point this is only a proposal, so now would be the time to come up with those bright ideas.)

 

[Baldyr]

This is what a unit spawn entry could look like:

#era         type        num     interval    random  
"Ancient",   "Wolf",     1,      20          False

So this data would cause the proposed module to spawn one Wolf unit every 20 turns in the Ancient era. Adding a line like this would be all there would be to designing the actual spawns, but chances are that you want more setting to play with. So tell me what they should be.

You would also have to turn off the pre-existing Barbarian spawns with some XML setting and also add a couple of lines of code into the CvEventManager module of your mod. (If your mod doesn't have one the download would supply you with a ready-to-use copy.) This would all be documented in any case.

 

[Dancing Hoskuld]

I am currently mostly interested in animal units and locusts (which can't be animals since they need to enter nations lands to ruin crops).

There is a DLL mod that spawns sea animals but it does them too often. We may need to figure out how to disable it also.

It would be nice if the units spawned were based on Era or Str. In the case of my locusts mod I have a swarm size for all eras, one per era. Where the era is the common/ average calculated one for all civs at that turn. BTW where is that information defined. This is part of your example structure isn't it

Would it be possible to have spawning points. For example:- an elephant unit spawns every 20 turns from 5% of the ivory resources in unclaimed territory.

#era         type        num     interval    random     resource     chance  
"Ancient",   "Wolf",     1,      20          False      None          0
"Ancient",   "Elephant", 1,      5,          False      Ivory         5

 

[Baldyr]

I am currently mostly interested in animal units and locusts (which can't be animals since they need to enter nations lands to ruin crops).

I think that the actual behavior is based on the default UnitAIType for the UnitType, as defined in the Unit Infos XML. So you can have whatever you like. Or there could even be a setting for custom Unit AI in our spawning code.

There is a DLL mod that spawns sea animals but it does them too often. We may need to figure out how to disable it also.

Just don't use it. You should be able to spawn whatever (Barbarian) units you like with the Python module.

It would be nice if the units spawned were based on Era or Str. In the case of my locusts mod I have a swarm size for all eras, one per era. Where the era is the common/ average calculated one for all civs at that turn. BTW where is that information defined. This is part of your example structure isn't it

I'm not sure I understand what you mean by Str (Strength?) and there is already a game era defined within the game. It would only add lag to the game to constantly keep calculating the "average" era for all players. This is why I propose we use game era instead of player eras.

Would it be possible to have spawning points. For example:- an elephant unit spawns every 20 turns from 5% of the ivory resources in unclaimed territory.

I don't think there needs to be both a turn interval and a random chance - the same value could be used for both. (The control I added in my sample data was the random value, which is set to True for one in interval chance every turn, or False for a static interval of turns.) So 5% every 20 turns equates to 1% chance per turn then, right?

It would of course be possible to spawn units on/near resources, but this pretty much requires the module to create an index of all resources at startup. Which of course is doable but will add one second or so the game start sequence. This would also happen on load game.

So you'd have this:

#era         type        num     interval    random     resource
"Ancient",   "Wolf",     1,      20,         False,     None
"Ancient",   "Elephant", 1,      100,        True,      "Ivory"

 

[Baldyr]

For scenarios it would probably be necessary to define map areas also. Like you want ethnic units to appear in some specific location or divide the map into climate zones.

One way to achieve this is to use rectangular areas where you define the area with the lower left corner tile and the upper right corner tile. Example:

#era         type        num     interval    random     area1      area2
"Ancient",   "Wolf",     1,      20,         False,     (23, 34),  (78, 89)

So this would restrict the spawned Wolf units to unowned land tiles between the X coordinates 23 and 78, and the Y coordinates 34 and 89.

Default values for "no area" could be a couple of None values. To only have one target tile you could only use area1 and set area2 to None.

 

[Dancing Hoskuld]

Just don't use it. You should be able to spawn whatever (Barbarian) units you like with the Python module.

I will have to check to see if it is can be disabled, it used to be but it may now just be active if you sea animals - DOMAIN_SEA and bAnimal = 1.

I don't think there needs to be both a turn interval and a random chance - the same value could be used for both. (The control I added in my sample data was the random value, which is set to True for one in interval chance every turn, or False for a static interval of turns.) So 5% every 20 turns equates to 1% chance per turn then, right?

I really don't like using a field to mean two different things it makes taking up maintenance and enhancement much more difficult for someone who comes late to a piece of code. (I used to be a maintenance programmer so I speak from experience here. We even have two fields already, "interval" and "random" we could just make random the percentage chance rather than a true/false field.

It would of course be possible to spawn units on/near resources, but this pretty much requires the module to create an index of all resources at startup. Which of course is doable but will add one second or so the game start sequence. This would also happen on load game.

Except that my mod "SubdueAnimals', based on Roamty's, mod has a small chance that it will add resources to the map. So I will need a way of adding resource plots to the index. Very doable.

So you'd have this:

#era         type        num     interval    random     resource
"Ancient",   "Wolf",     1,      20          100      None
"Ancient",   "Elephant", 1,      20,        5       "Ivory"

 

[Baldyr]

I will have to check to see if it is can be disabled, it used to be but it may now just be active if you sea animals - DOMAIN_SEA and bAnimal = 1.

What I mean is that you skip that SDK mod altogether, so you don't have to "disable" it. Instead you can use the proposed Python module for all your barbarian spawning needs. Including sea creatures.

I really don't like using a field to mean two different things it makes taking up maintenance and enhancement much more difficult for someone who comes late to a piece of code. (I used to be a maintenance programmer so I speak from experience here. We even have two fields already, "interval" and "random" we could just make random the percentage chance rather than a true/false field.

I think you're right on this.

Except that my mod "SubdueAnimals', based on Roamty's, mod has a small chance that it will add resources to the map. So I will need a way of adding resource plots to the index. Very doable.

Yeah, we could add a function call for your specific mod.

So you'd have this:

Except every value needs to be separated with a comma. And I just realized that each line has to be enclosed in a parenthesis, like this:

( "Ancient",   "Wolf",     1,      20,        100,     None ),
( "Ancient",   "Elephant", 1,      20,        5,      "Ivory" ),

Because each line represents a data structure in itself.

 

[strategyonly]

One thing to think about also is that most mods use BUG, therefore do not use any CvEventManager, and the likes, also try to completely stay away from any PythonCallBacks junk. IMO

 

[Dancing Hoskuld]

One thing to think about also is that most mods use BUG, therefore do not use any CvEventManager, and the likes, also try to completely stay away from any PythonCallBacks junk. IMO

Doesn't matter how we build it I would need to convert it to BUG/WoC anyway to use it myself. Something I do all the time. So I would expect we would go the way Orion Veteran has gone and release it in both standard and BUG/WoC compatible versions.

Suggested very broad design events:-

• _init__ - convert the data structures from human readable form into game values for speed in game by getting all the function calls done only once.
• onGameStart - create "special" plot array(s)
• onGameLoad - as onGameStart plus figure out what the game turn is
• onEndTurn - place the new barbarian units

Some ideas on the internal data structures ie those we build from the human readable ones.

• having arrays for each era may speed things up by reducing array size when choosing what to place

How were you thinking of deciding where to spawn things? When doing my locust mod I went through all plots every turn and did not notice any slow down. A crude function for deciding what to place may be something like:- (Note: this is not code but first cut design English)

[B]onEndTurnEvent;[/B]
Identify array for the game era
For each plot:-
   If plot visible to non barbarian unit then go to next plot
   else
       Get terrain, feature and bonus information for this plot
       If a unit defined in the array for this domain (sea/land)
            "magic happens here"

Or you may have better ways of doing it using array or dictionary functions.

 

[Baldyr]

Regarding the actual Python design I don't think that you can do anything without a callback from the DLL. So the turn-to-turn checks would be done at beingGameTurn or whatever game event. Since there is a game era setting it would make sense to do what Dancing Hoskuld suggested and put all the unit spawn data into a dictionary indexed by era. (Unless, of course, we allow for a None era setting. Then the benefit of doing that would be less.)

There would not be any going over the entire map plot by plot action going on. Instead it would be possible to select random plots by plot ID and then evaluate whether or not its a valid plot. (Invalid plots are peaks and water tiles for land units - and land plots for water units. But also plots adjacent to units or cities.) The question is how many tries a spawn will get before going on to the next on. One way to do it would be to only give each spawn only one chance per turn. So the further the game develops, the less valid plots there will be, and the chance of spawn probability decreases. Which mostly makes sense.

If all resources are to be indexed at startup it would also be possible to index all peak and water tiles, as to prevent unnecessary checks for the same plots. Or it would however also be possible to store this data once the validity of the plot is checked, for future reference. So the code would actually become faster over time.

But before any plots are evaluated at all, the current game turn will have to be validated with a modulus operation. So if the Wolf units in the initial example are to be spawned every 20 turns, then the game turn would have to be evenly dividable with 20. We can add a random denominator for each game session also, so that the actual game turn will still vary from game to game. (So that the Wolf unit isn't always spawned at turn 20, for example.)

The next condition would be the random one, and if that passes will there be a plot selected to be evaluated for validity, as above.

 

[Dancing Hoskuld]

Regarding the actual Python design I don't think that you can do anything without a callback from the DLL.

Callbacks are to the dll for specific stuff. It is rare that you really need it, none of Tenstom's python wonders I have looked at require a callback for example. Nor does anything I have done. The reason callbacks are to be avoided if possible is that they are slooooow, very slow.

So the turn-to-turn checks would be done at beingGameTurn or whatever game event. Since there is a game era setting it would make sense to do what Dancing Hoskuld suggested and put all the unit spawn data into a dictionary indexed by era. (Unless, of course, we allow for a None era setting. Then the benefit of doing that would be less.)

I am fairly sure that the documentation suggests that onEndGameTurn is better than onStartGameTurn but then one says it is at the end of start game and the other says it is at teh end of the game turn before the player moves or something. So it could have just been my dyslexia leading me astray (again).

There would not be any going over the entire map plot by plot action going on. Instead it would be possible to select random plots by plot ID and then evaluate whether or not its a valid plot. (Invalid plots are peaks and water tiles for land units - and land plots for water units. But also plots adjacent to units or cities.) The question is how many tries a spawn will get before going on to the next on. One way to do it would be to only give each spawn only one chance per turn. So the further the game develops, the less valid plots there will be, and the chance of spawn probability decreases. Which mostly makes sense.

Ah! So you thought that (Wolf 5%) means 5% chance for a wolf to spawn this turn whereas I thought it meant that a wolf spawns on 5% of the valid tiles for a wolf! Where the valid tiles for a wolf is defined in the unitinfos XML terrain and feature natives.

If all resources are to be indexed at startup it would also be possible to index all peak and water tiles, as to prevent unnecessary checks for the same plots. Or it would however also be possible to store this data once the validity of the plot is checked, for future reference. So the code would actually become faster over time.

But before any plots are evaluated at all, the current game turn will have to be validated with a modulus operation. So if the Wolf units in the initial example are to be spawned every 20 turns, then the game turn would have to be evenly dividable with 20. We can add a random denominator for each game session also, so that the actual game turn will still vary from game to game. (So that the Wolf unit isn't always spawned at turn 20, for example.)

The next condition would be the random one, and if that passes will there be a plot selected to be evaluated for validity, as above.

The turn counter is only for the period it is in which means that we need to know what turn the change of era happens so we can rebase our counts. (TurnCounter for era = Turn - Start of Era turn). We may also need to store the information in save games so when the game is loaded we have the information. Unless you were planing on the simpler function of pretending that change of era has no effect on the turn counter function. Which is OK too.

If you want to randomise the actual turn that units are generated we could follow a totally different plan. At the start of an era generate a list of units to place and a turn number based on the specified turn number. Then place the unit on the map at the correct time. You could generate lots of turns or generate some and then more when you run out. The initial list could be done on load or start.

If fact this looks like a faster way of doing stuff. I need to think a bit more on this and its consequences....

Edit: A new idea

[B]_init__, onGameLoad, onEmptyList[/B]
   For fake turn number 1 to 100
       For each element in the list of barbarian units for this game era
            if unit possible this turn
                 roll to see if happens
                 if happens
                     adjust/randomise the turn for this unit to next occur
                     save in List
[B]place barbarian unit[/B]
   For all barbarian units to be placed this turn in List
       Find a valid plot
          place the unit
          remove from List
   If end of list
      generate more using onEmptyList
   (If generates units for this turn then do this again.)

 

[Asaf]

Regarding the actual Python design I don't think that you can do anything without a callback from the DLL. So the turn-to-turn checks would be done at beingGameTurn or whatever game event.

I am fairly sure that the documentation suggests that onEndGameTurn is better than onStartGameTurn but then one says it is at the end of start game and the other says it is at teh end of the game turn before the player moves or something.

There is already the createBarbarianUnits() callback (no need to even enable it in the XML, since it is always called, at least for unmodded BTS). You also have createBarbarianCities().

Callbacks are to the dll for specific stuff. It is rare that you really need it, none of Tenstom's python wonders I have looked at require a callback for example. Nor does anything I have done.

Whether callbacks are calls to the DLL which are made from Python or vice versa is kinda a semantic issue. I guess both can be considered callbacks. Although probably you're technically right.

The reason callbacks are to be avoided if possible is that they are slooooow, very slow.

If you're talking about calls from the DLL to Python - they're not slower than any other Python code. If you're talking about calls from Python to the DLL, well, they're much faster.


There's existing code in the DLL for spawning barbarians, which can be tweaked to receive all those parameters from an XML file. I understand that part of the issue here is to do it in Python, but I do think that it would fit better, run faster and will be easier to be used by modders in C++ and XML.

There are, of course, advantages to do it in Python as well.

 

[Baldyr]

I'm on my lunch break so I thought I'd just clarify what I mean about "game era": CyGame.getCurrentEra() - nothing else. It requires only invoking this method at the start of the barbarian spawn sequence.

And I'll look into the built-in barbarian spawns also. I'll get back with more later.

 

[Baldyr]

I don't have time to reply to everything right now, but could someone perhaps describe how the createBarbarianUnits callback works? It returns False by default - so? Should one add CyPlayer.initUnit() invocations into the method? Change the return value to True? What?

 

[Dancing Hoskuld]

All I know is that by making all refs in PythonCallbackDefines.xml 0 the time between turns was greatly improved. iirc 75% or more faster, so 4 min wait would become a 1 min wait.

 

[Baldyr]

All I know is that by making all refs in PythonCallbackDefines.xml 0 the time between turns was greatly improved. iirc 75% or more faster, so 4 min wait would become a 1 min wait.

I'm no expert, but the callbacks from Python to the SDK aren't that big of a problem in themselves, but the fact that those that are disabled or can be disabled are called repeatedly during game-play. Like hundreds of times during a given player's turn. That will eventually add up to full seconds and become noticeable.

My original proposal (sans any built in support for the Barbarian spawn callbacks mentioned by Asaf) wouldn't use any calls back to the DLL. What I was referring to was instead the fact that Python is used as an extension of C++ in CivIV, and doesn't achieve anything on its own. All of the Python "hooks" are calls coming from the DLL. So you could disable all those callbacks all you like.

As a rule of thumb, the calls in CvEventManager are merely calls from the DLL to Python, while the callbacks in CvGameUtils return some value back to the DLL. As Asaf said - this isn't a real problem unless this happens a lot.

I am fairly sure that the documentation suggests that onEndGameTurn is better than onStartGameTurn but then one says it is at the end of start game and the other says it is at teh end of the game turn before the player moves or something. So it could have just been my dyslexia leading me astray (again).

I actually don't know. You might be correct and I'd love to see the documentation you're referring to. Because I don't feel at all comfortable with doing anything simply because it is how I've "always" done it in the past. I'm doing the Python scripting thing because it amuses me, and a big part of the fun is learning.

Ah! So you thought that (Wolf 5%) means 5% chance for a wolf to spawn this turn whereas I thought it meant that a wolf spawns on 5% of the valid tiles for a wolf! Where the valid tiles for a wolf is defined in the unitinfos XML terrain and feature natives.

And my ignorance about the XML portion of the game is once again obvious. Those settings would totally have to be utilized, if at all possible. And looking at the CvUnitInfo class in the API gives me two boolean methods for exactly this.

If you want to randomise the actual turn that units are generated we could follow a totally different plan. At the start of an era generate a list of units to place and a turn number based on the specified turn number. Then place the unit on the map at the correct time. You could generate lots of turns or generate some and then more when you run out. The initial list could be done on load or start.

This idea actually occurred to me also... Pre-generating what spawns happen on what turn might be faster, but I'm not sure that speed is even an issue with the proposed module. Maintaining and indexing huge arrays might be both slow and wasteful with memory, but I'm not computer scientist...

There is already the createBarbarianUnits() callback (no need to even enable it in the XML, since it is always called, at least for unmodded BTS). You also have createBarbarianCities().

See earlier post - I have no idea what to do with these? Disabling the default Barbarians spawns would disable this callback also, not? Rendering it practically useless, then?

There's existing code in the DLL for spawning barbarians, which can be tweaked to receive all those parameters from an XML file. I understand that part of the issue here is to do it in Python, but I do think that it would fit better, run faster and will be easier to be used by modders in C++ and XML.

There are, of course, advantages to do it in Python as well.

You're of course right on everything. But the point of this proposal is, as you pointed out, to do it in Python. Partly because C++ is currently out of my own reach, but mainly because any non-programmer modder would be better off simply dumping a CustomBarbarianSpawns.py file into their mod and adding a couple of lines (not blocks of code) into the event manager, than to merge some SDK mod with the other mod-comp they're already utilizing and compiling the whole thing. 9/10 non-programmers wouldn't even attempt it, and 9/10 who would try it would probably fail, at least at their first few attempts. 9/10 probably wouldn't be motivated enough to see it through. Do the math...

So making this entirely in Python, no matter what lag it ends up causing or how much "easier" working with XML is (yeah, right), makes it accessible. This would be the real benefit of Python over SDK modding.

 

[Dancing Hoskuld]

The doco is the Modiki.

def onBeginGameTurn() 	self, argsList (iGameTurn) 	Called at the beginning of the end of each turn
def onEndGameTurn() 	self, argsList (iGameTurn) 	Called at the end of the end of each turn

I saw pregeneration as a better way of varying the actual turn - ie not having a wolf turn up every 20th turn.

 

[Baldyr]

The doco is the Modiki.

Ah, that basically told me nothing. beginGameTurn has served me well in the past, but with especially Barbarian spawning I suppose endGameTurn might be even better. (I guess the difference would be that the Barbarians don't get to move their units on the same turn they spawn?)

I saw pregeneration as a better way of varying the actual turn - ie not having a wolf turn up every 20th turn.

The interval thing is just some design I copied from RFC, where all spawning is periodical instead of random in nature. It would be optional anyway. (It strikes me as a faster way to execute the code however.)

 

[Asaf]

The createBarbarianUnits() callback is called whenever the game checks for spawning barbarians units (once a turn), unless the 'No Barbarians' options is turned on.

This is the first thing that happens after testing whether this option exists, and if it returns True then the process of spawning barbarians units doesn't continue.

So you could effectively turn barbarians off by doing nothing and returning True even if the 'No Barbs' option is turned off, but you can just enter there any code to spawn your barbarians.
It will fully replace the game's barbarians spawning.

The same goes for createBarbarianCities().

 

[Baldyr]

So I can basically use the createBarbarianUnits callback instead of beginGameTurn/endGameTurn - and changing the return value to True disables the default barbarian spawns. Ok, got it.