Lua Scripting Possibilities

[Prof. Garfield]

@Prof. Garfield this is an interesting topic. Personally, I feel like using Lua events to dynamically change or swap files within the scenario directory itself (and any of its subdirectories) is "acceptable behavior", but doing so within any higher-level directory (such as the main TOT installation directory, or anywhere else on the user's computer) is strictly "off limits". Note that I'm basing this not on what I'd like to do as a designer in my own projects, but on how I feel about an appropriate boundary for someone else's scenario that I download, install, and run. I'm willing to give a designer control over his own scenario's folder, since bad decisions there would only affect his own work, but I feel pretty strongly that he has no "rights" outside of that.

This seems like a perfectly reasonable restriction. Occasionally, when I've written quick and dirty scripts that have had to write files, I've written directly to the TOT directory, since it is easier. However, that stuff hasn't been meant to be used by the end users of scenarios.

Unfortunately you're right that any Lua script opens up the possibility of error or malicious behavior, or someone not abiding by this boundary, which could have severe consequences. TNO added a warning popup about loading Lua events... I'm not sure if enforceable rules (to limit the power of Lua in this regard) are possible or not. I do agree that the risk is very low at the moment, but it's something to be mindful of if the community grows. If a technical limitation isn't feasible, eventually I could envision some sort of "certification" program in which one or more experienced Lua programmers, who are known to the community, review the code in a project and publicly confirm that it's safe for general usage.

At the moment, if I'm paranoid, I can search for things in the code like 'os.execute' and 'io.write.' If the code is supposed to be writing stuff, then I have to figure out if what is being written is advertised, which could possibly be a bit more involved.

What we might do, is write a library with all the possible things you could want to overwrite (or, that are considered 'allowed' to be overwritten). For example, overwrite.rules(newRulesName), overwrite.movpiece(newSoundName).

 

[Civinator]

At the moment, if I'm paranoid...

Prof. Garfield, of course you are not paranoid, but very, very helpful and -in contrary to myself - aware about the dangers that such codes can contain. Thank you very much for your answers. Nevertheless it would be nice, if the problem of the very simple Civ 2 'tac...tac...tac- sound' could be upgraded to individual sounds for Civ 2 ToT units, while an 'onmovement' trigger is missing in ToTPP.

 

[JPetroski]

Is there no way to add a count, veteran, and randomize to this:

civ.createUnit(object.uCenturion, object.tEurope,object.lHKInvasionPoint1)

I need to use civ.createUnit vs. civlua.createUnit (which allows those options) because this is a naval invasion and I want the ground force to spawn at sea. Randomize isn't critical here but it is a shame I can't do count or vet as there are several units to get through.

 

[Prof. Garfield]

You (or someone else) will have to write equivalent code to civlua.createUnit, with the difference being that placing land units on water is allowed.

Have you had a look at the code for civlua.createUnit and tried to understand it? There are a couple helper functions, and then the loop that generates units. Or, here is some relevant code from my munition library:

        for i=1,numToGenerate do
            local newUnit = civ.createUnit(specification.generatedUnitType,
                generatingUnit.owner,generatingUnit.location)
            newUnit.veteran = specification.giveVeteran or false
            newUnit.veteran = newUnit.veteran or (specification.copyVeteranStatus and generatingUnit.veteran)
            newUnit.homeCity = nil
            if specification.setHomeCityFunction then
                newUnit.homeCity = specification.setHomeCityFunction(generatingUnit)
            end
            unitTable[i] = newUnit
        end

 

[JPetroski]

Is there a way to set parameters around what game years an event can randomly fire? I know how to use random turn, but I want there to be a floor and ceiling based on turn numbers for events to fire. How would I do that?

Basically say I wanted there to be a 25% chance an event would fire once on any turn between 10 and 20. How would I define the range?

 

[Prof. Garfield]

If you want there to be a 25% chance to fire on any given turn between turn 10 and 20, use this condition (and a justOnce)

if turn >=10 and turn <=20 and math.random() < 0.25 then

If you want a 25% chance for an event to ever happen, but guarantee that if it happens, it has an equal chance of happening on any turn between turn 10 and turn 20 (both inclusive) then you want something like this

-- define this counter in appropriate location
counter.define("turnToTriggerEvent",-2)

-- on the turn you want to decide whether the event will happen or not, and on which turn

if turn == decisionTurn then
     if math.random() < 0.25 then
        counter.setValue("turnToTriggerEvent",math.random(10,20))
    end
end

-- Also, in onTurn (or, maybe, afterProduction) you check if the turn is the same as the counter value

if turn == counter.value("turnToTriggerEvent") then
    -- do event stuff
end

 

[JPetroski]

I'm playing around with a theory that with lua, it's possible to have "unlimited units" in a sense that if you have a scenario taking place over a certain number of years, and you know certain units become obsolete or destroyed after a portion of those, you can recycle the old unit into a new doing something like this, where we might expect Hurricanes to eventually be replaced by Spitfires and the Hurricane slot to then be used for Mustangs under the control of an entirely different civ.

object.uHurricane =civ.getUnitType(38)
object.uMustang =civ.getUnitType(38) -- Turn 25 onward

object.uSpitfire =civ.getUnitType(39) -- all object.uHurricane needs to be deleted on turn 24 and and replaced with object.uSpitfire on that same turn, on that same tile, with the same home city and the same veterancy

The theory seems sound to me, but I'm having trouble figuring out how to put it into practice. I figured I'd start by trying to take the damaged B-17 mechanism in OTR because it has all of the functionality I need (same home city, same vet status, same tile, etc.) but this is a unitKilled trigger which doesn't work for this event. Instead I need this to be a function that works on a turn. Unfortunately, my attempt below does absolutely nothing. Any idea why? (I put this in "onKeyPress" for now by the way as that is easier to check and trouble shoot than onTurn).

local swapUnitTypes ={
["HMS Hood to Battleship"] = {unitType=object.uHood, replacingUnit = object.uBattleship , preserveVetStatus = true, preserveHome=true},}


local function unitSwap(unitType)
for unit in civ.iterateUnits() do
if unit.type == swapUnitTypes.unitType then 
if civ.getTile(unit.location.x,unit.location.y,unit.location.z) ~= nil then
        local tile = unit.location
        for __, unitSwapInfo in pairs(swapUnitTypes) do
            if unit.type == unitSwapInfo.unitType then
                local quantityToProduce = unitSwapInfo.replacingQuantity or 1
                if math.random() <= (quantityToProduce - math.floor(quantityToProduce)) then
                    quantityToProduce = math.ceil(quantityToProduce)
                else 
                    quantityToProduce = math.floor(quantityToProduce)
                end
                local replacingHome = nil
                if unitSwapInfo.preserveHome then
                    replacingHome = unit.homeCity
                end
                local replacingVetStatus = unitSwapInfo.replacementVetStatus or false 
                if unitSwapInfo.preserveVetStatus then
                    replacingVetStatus = unit.veteran
                end
                for i=1,quantityToProduce do
                    local newUnit = civ.createUnit(unitSwapInfo.replacingUnit,unit.owner,unit.location)
                    newUnit.homeCity = replacingHome
                    newUnit.veteran = replacingVetStatus
                    replacementUnit = newUnit
                end --1st instance for i=1,quantityToProduce
                if unitSwapInfo.bonusUnit then
                    quantityToProduce = unitSwapInfo.bonusUnitQuantity or 1
                    if math.random() <= (quantityToProduce - math.floor(quantityToProduce)) then
                        quantityToProduce = math.ceil(quantityToProduce)
                    else 
                        quantityToProduce = math.floor(quantityToProduce)
                    end       
                    for i=1,quantityToProduce do
                        local newUnit = civ.createUnit(unitSwapInfo.bonusUnit,unit.owner,unit.location)
                        newUnit.homeCity = nil
                        newUnit.veteran = false
                    end --2nd instance for i=1,quantityToProduce   
                end -- end if unitSwapInfo.bonusUnit       
            end -- unit.type == unitSwapInfo.unitType
        end -- for unitSwapInfo in pairs(swapUnitTypes)
        --civ.deleteUnit(unit)
    end--civ.getTile(...
   
    end 
end 
end

local function doOnKeyPress(keyCode)

--I snipped out a lot
  
 if keyCode == keyboard.two then
 unitSwap(unitType)
 return
 end 
end

 

[Knighttime]

Unfortunately, my attempt below does absolutely nothing. Any idea why?

Well, without trying to run this, I noticed a couple things.

First, based on he way you defined local swapUnitTypes, this reference isn't valid:
swapUnitTypes.unitType
swapUnitTypes is a table, but it doesn't have a key of unitType -- the (only) key that you defined is "HMS Hood to Battleship". The value of that key is also a table, and that table is the one with a key of unitType.

It looks to me like you can simply delete this line entirely:
if unit.type == swapUnitTypes.unitType then
and its corresponding end statement later on. Your line that says
for __, unitSwapInfo in pairs(swapUnitTypes) do
will take care of looping through the swapUnitTypes table, but that first if check means you're never getting to that part of the code. The line that says
if unit.type == unitSwapInfo.unitType then
is correct.

Then in onKeyPress(), you're calling the unitSwap function with unitType as a parameter, but that seems unnecessary. Did you find a unit type somewhere within the "I snipped out a lot" section to populate this variable?
Within the function itself, though, I don't see you using it -- you're just iterating over all units in the game. So I think you could eliminate that parameter both from the function definition and the call to the function.

Does that get you back on track to implementing this feature?

I'm curious, though, what name you plan to assign to civ.getUnitType(38) in Rules.txt, if you want it to represent a Hurricane at one point in the game and a Mustang at another. unittype.name is not a writeable field in Lua, so while you can use the type differently, you can't change the name of it dynamically. (The only way to do that is by swapping Rules.txt files.)

 

[JPetroski]

Thank you a million times over! Yes, that got it working perfectly! I had to play around a little with where i put civ.deleteUnit(unit) to get the Hood to disappear, but I seem to have found a place to tuck it in where it works well.

I'm curious, though, what name you plan to assign to civ.getUnitType(38) in Rules.txt, if you want it to represent a Hurricane at one point in the game and a Mustang at another. unittype.name is not a writeable field in Lua, so while you can use the type differently, you can't change the name of it dynamically. (The only way to do that is by swapping Rules.txt files.)

Yes, the plan is to have a batch file with rules changes that addresses the different names (as well as what tribes can build them). I think it's an acceptable trade off to have this kind of versatility, especially since the units need to swap out at approximately summer/winter sequences, anyway. There will be some housekeeping issues to deal with (such as dealing with the fact that the AI from tribe1 might still produce the unit even though only tribe2 should be able to at this point), but that should be a pretty simple fix.

 

[Prof. Garfield]

I spent some time today trying to write a script to determine where the 'fish' and 'whale' resource tiles are. After spending some time trying to reverse engineer the resource seed pattern, I realized that someone probably did that years ago, and that I should probably just look up that work. However, I couldn't find it, but several references were made to no longer existing apolyton web pages.

My next idea was to programmatically determine what tiles have specials, and produce a table in a file that could then be referenced later. This would also work when special resources are placed by hand.

The procedure was for the scenario maker to set desert production to 1 of each resource. The desert "fish special" would be set to 10 food, and the desert "whale special" would be set to 10 shields. By setting the city size to 0 (so only the centre tile is worked), changing terrain to desert and teleporting (or creating/deleting, I tried both) a city to each square and checking production via totalSheild and totalFood to determine if a special exists, and which one.

However, this doesn't work. I think the reason is that the city is not "calculated" unless either a human looks at the city or the city is processed for taxes/production/etc.

Does anyone have any ideas? The only one I'm left with is having the scenario creator manually compile the information in some way (I'm thinking use keypress to place one of two units on each of the specials, then looping over units). We've got this far without lua knowing where the specials are, so I suppose it isn't that critical, but I did think this was going to be a relatively easy thing to add...

 

[Knighttime]

We've got this far without lua knowing where the specials are, so I suppose it isn't that critical, but I did think this was going to be a relatively easy thing to add...

I think I poked at this in similar ways, awhile ago, and didn't figure out a good solution. I ran into the same issue you did:

I think the reason is that the city is not "calculated" unless either a human looks at the city or the city is processed for taxes/production/etc.

I may have made my own analysis more difficult because I also wanted to be able to detect specials on new, randomly-generated maps -- not just document ones on a static map used by a scenario designer. I gave up, eventually, but I would certainly be interested in a solution if you can find one.

Within a scenario, the fact that TOTPP allows special resources to be placed manually makes this more challenging -- it basically means that the documentation related to resource seeds and patterns isn't sufficient. You might be able to calculate where specials ought to be, but I don't know how you could calculate where they actually are, if manual placement is involved. Everything below assumes we're only dealing with the game's native specials.

Although there isn't any quick way to tell whether a tile contains a special, there is one related piece of information available. If an ocean tile would be a special but is more than two tiles away from any land tile, the special is "suppressed" and this shows up in tile.terrainType. TNO said we should always reference the terrain type as tile.terrainType & 0x0F but if you don't use bitwise operators and just look at the integer value stored in this field, an ocean tile is 10 but an ocean tile with a suppressed special is 74 (it adds 64 to the base value of 10 -- does that mean tile.terrainType & 0x40 would be appropriate to check this?). So you might be able to detect the pattern of all specials based on the locations of these suppressed specials. I don't think it tells you which resource was suppressed though (fish or whales).

Then I found two pages with formulas that might be useful:

1. TNO's formula for how the resource seed determines placement of specials: https://forums.civfanatics.com/thre...is-add-resources-on-map.518649/#post-13002282
2. If the resource seed isn't known (from the map editor) then here's a potential formula for calculating it based on a hut location: http://www.codehappy.net/apolyton/threads/68481-1.htm

BTW, many of the old Apolyton threads are preserved on that codehappy site, but they're not easy to find or navigate.

Hope this is helpful. Good luck!

 

[Prof. Garfield]

Thanks @Knighttime , this is very helpful.

Since TNO mentioned how he stores custom resource locations with a byte for each tile, I'm inclined to think that it would make more sense to figure out how to read a saved game file to extract the information into a table, rather than mess around further trying to figure out how to read it from inside a game. Unless someone comes up with a cool concept that would benefit from identifying special squares, I'm inclined to think that for now it is best to do other stuff and hope that TNO shows up and provides us with the functionality directly.

Although there isn't any quick way to tell whether a tile contains a special, there is one related piece of information available. If an ocean tile would be a special but is more than two tiles away from any land tile, the special is "suppressed" and this shows up in tile.terrainType. TNO said we should always reference the terrain type as tile.terrainType & 0x0F but if you don't use bitwise operators and just look at the integer value stored in this field, an ocean tile is 10 but an ocean tile with a suppressed special is 74 (it adds 64 to the base value of 10 -- does that mean tile.terrainType & 0x40 would be appropriate to check this?). So you might be able to detect the pattern of all specials based on the locations of these suppressed specials. I don't think it tells you which resource was suppressed though (fish or whales).

I think you would use

tile.terrainType & 0x40 == 0x40

to check that, but in the General Library, I have functions like

gen.isBit1(tile.terrainType,7)

to do that kind of job, without worrying directly about bitwise operations. Incidentally, I've been using %16 on terrain type instead of &0x0F, though I suppose the latter might technically be faster, now that it is pointed out to me.

I think that with enough ocean squares, we could narrow the seed down to 4 possibilities, which would tell us where all the resources are, but not their type.

 

[Prof. Garfield]

I've moved the conversation here, so as not to clutter up the Function Reference thread.

I was looking into the issue of corruption, and realized that the fields/values provided in the city object aren't ideal.

city.totalTrade is actually the net trade produced by the city, after arrows lost to corruption are subtracted.

There doesn't seem to be any field that provides the true total trade generated by the city, or -- for that matter -- the amount of corruption experienced.

city.totalTrade is made up of city.baseTrade (arrows generated from worked tiles) plus the value of arrows from trade routes. But it turns out that this is also affected by corruption. I haven't worked on the formula, but some of the corruption seems to be taken from the "baseTrade" portion, and some from the "trade routes" portion. So city.baseTrade is also the "net trade" generated by the worked tiles, not the actual quantity they generate.

Based on the corruption formula I found at https://apolyton.net/forum/miscellaneous/archives/civ2-strategy-archive/62524-corruption-and-waste, I worked on a way to reverse-engineer the amount of corruption based on the net trade value that we have. "What would the total trade need to be, in order for the corruption it generates to leave this amount of net trade?" This approach can get close, but it won't hit the target every time. Based on the way the formula works, if a city has 4 net trade, there are situations where it's possible for this to result from 8 with 4 corruption, or 9 with 5 corruption, or 10 with 6 corruption. I can't tell which one of those is actually the case. Even in larger cities with more trade and multiple trade routes, it's possible for 37 net trade to result from 41 with 4 corruption or 42 with 5 corruption.

The same problem likely exists with waste (I expect that city.totalShield is the net value after waste is subtracted) but I haven't confirmed or tested this yet.

First, an answer to your waste question. I found this in the explanation I gave for an event that rehomes units about to be disbanded for a lost city.

What we'd like to do is to determine if a city can support more units, and if not, exclude it from the list of cities that we're considering transferring the unit to. Unfortunately, there isn't an easy way to do this. While the command city.totalShield excludes waste from the result it returns, it does include shields paid for supporting units. Hence, we can't simply check if the result of that function call is 0. Also, since there isn't a command to find how many units the city is supporting, we don't have an easy way to determine if the city can support extra units.

For your question about computing total trade, you will probably have to compute it manually. I have some of the work completed.

I wrote the following function for the General Library

-- gen.cityRadiusTiles(cityOrTileOrCoordTable) --> table
--  returns a table of tiles around a center tile, the
--  size of a city 'footprint'.  The indices are listed below
--  and are based on how city.workers determines which tiles
--  are worked
--
--    
--
--      #       #       #       #       #
--          #       #       #       #       #
--      #       #       #       #       #
--          #       20      13      #       #
--      #       12      8       9       #
--          19      7       1       14      #
--      #       6       21      2       #
--          18      5       3       15      #
--      #       11      4       10      #
--          #       17      16      #       #
--      #       #       #       #       #
--          #       #       #       #       #
--
--

If you write a function

getTradeProduction(city,tile)

then the following should work

function totalCityTrade(city)
    local workerAllocation = city.workers
    local tradeProduction = 0
    for index, tile in pairs(gen.cityRadiusTiles(city)) do
        if gen.isBit1(workerAllocation,index) then
            tradeProduction = tradeProduction + getTradeProduction(city,tile)
        end
    end
    return tradeProduction

 

[Knighttime]

Thanks for confirming what I suspected about city.totalShield and waste.

For your question about computing total trade, you will probably have to compute it manually. I have some of the work completed.

I wrote the following function for the General Library
...
If you write a function

getTradeProduction(city,tile)

then the following should work
...

Hmm... what would that function do, exactly? Return the number of trade arrows produced by a particular tile? I could include a table mapping terrainId to trade arrows generated (duplicating the data from Rules.txt) as the basis for this, and I'm assuming that the "city" parameter would be used to determine if the city owner's government is Despotism (could mean one fewer trade arrow) or Republic/Democracy (could mean one more trade arrow). I guess the Colossus wonder would be applicable here too. But aren't we back to the problem of not knowing which tiles have terrain specials, since those frequently add extra arrows?

Best case, this lets us know how many trade arrows are generated from worked tiles. What about trade arrows generated by trade routes, though? city.numTradeRoutes contains the quantity of routes (0 through 3) but not the number of arrows generated by them. I don't see a field with that information, or one with the IDs of the cities that are connected (which would be required in order to attempt calculating that.)

I guess I'm still not seeing a path through this other than the approach I was taking, which was to reverse-engineer the total trade based on the corruption formula and the net trade.

 

[Prof. Garfield]

I forgot about special terrain, but you could in principle keep all special locations in a giant table, or use the standard resource seed if the total is important. You're also right that we can't get the raw value of trade routes without knowing at the very least what cities are at the other end of the routes.

What are you trying to accomplish that you must know how much corruption a city has? I'm nearly certain that trade route bonuses are computed based on after corruption trade values, probably obtained from the field that city.baseTrade reads from.

 

[Knighttime]

Well, some of it is just the principle of it -- it bothers me that "totalTrade" is actually "net trade" and that corruption (which feels like a pretty important concept in Civ) is seemingly ignored from a Lua perspective.

I had two applications in mind. The first was trying to do comparisons between the effectiveness of each government type when designing a scenario, in order to tweak their cosmic settings and try and achieve better balance or progression. The second was a quick way to determine, as a player, which cities would derive the biggest benefit from building a Courthouse. There's no summary screen with corruption info, and paging through every city in a big empire to review this info seemed needlessly cumbersome.

In the end, my reverse-engineering approach is probably generating results that are "close enough" to be useful for both of those purposes, even if they're not perfect.

 

[Prof. Garfield]

Well, some of it is just the principle of it -- it bothers me that "totalTrade" is actually "net trade" and that corruption (which feels like a pretty important concept in Civ) is seemingly ignored from a Lua perspective.

I can understand and sympathize with that. I suspect that the values lua accesses are the values that the game needs for producing reports or similar things (baseTrade is probably used to compute caravan bonuses, for example), and would (or would have in the 90's) caused lag to recalculate all cities all the time.

I had two applications in mind. The first was trying to do comparisons between the effectiveness of each government type when designing a scenario, in order to tweak their cosmic settings and try and achieve better balance or progression. The second was a quick way to determine, as a player, which cities would derive the biggest benefit from building a Courthouse. There's no summary screen with corruption info, and paging through every city in a big empire to review this info seemed needlessly cumbersome.

What we can do if we really want to is calculate the trade that a city 'harvests' from terrain. This might mean restricting ourselves to known resource seeds or manually inputting special squares (or, figuring out where they are stored in the saved game and extracting them with a program), but it can be done. With that and baseTrade, the corruption not associated with trade routes can be determined. We also have access to the net trade from trade routes (subtract totalTrade from baseTrade), you can probably get a decent idea of the part of trade route trade lost to corruption based on the corruption rate for base trade.

We might actually be able to build a list of cities that are connected with a trade route.

When a caravan is activated, cycle through the cities and gather the numTradeRoutes field for each city. When the next unit is activated, cycle through the cities again, and check if any cities have incremented the numTradeRoutes field. If any have, those two cities have been joined by the trade route, so you can store that information in the state table. For those two cities, find the 3 trading partner cities that have the highest base trade, and those are the ones that it should be trading with (I think). From there, you can use the trade route formula to determine gross trade from trade routes. I'm sure there would be some kinks to work out (caravans contributing to a wonder, etc.), but you might get a decent enough picture.

 

[Knighttime]

Whew. Yes, I can see how all of that might work, and although it sounds complicated, both of us have probably tackled other issues that were equally involved. I don’t think I’ll attempt this immediately, but I’ll file it away as something to consider in the future. In the meantime I’ll try to clean up the code I wrote to estimate total trade based on net trade and post that here, in case it’s useful.

 

[Knighttime]

Here is what I put together (so far) for corruption. As you mentioned in this thread, my formula for "distance from capital" is based upon flat maps and doesn't handle for maps that wrap around, so that enhancement would need to be made in order for this to be more broadly useful.

The way this code works, whenever there are multiple values for "total trade" that would yield the same amount of "net trade", it will always return the lowest amount of corruption possible. In other words, if 9 net trade could result from either 15 total with 6 corruption, or 16 total with 7 corruption, or 17 total with 8 corruption, then the code will calculate the corruption as 6.

 

[Prof. Garfield]

Looks good. Maybe I read your code wrong, but it looks like your code assumes the civ has only one capital. While this is usually the case, you can use cheat mode to give a civ multiple capitals, so it might make sense to try to find the nearest one.

I'll have to put the 'in game distance' into the General Library, so we can reference distances without re-creating the function all the time, and let the General Library handle the world shape (currently, setting the world as round is just running a function once).