Linking Lua with XML, implimenting Classes, and jump tables

Thalassicus

Thalassicus

Bytes and Nibblers
Joined
Nov 9, 2005
Messages
11,057
Location
Texas
This is just a reference for anyone trying to figure out how to do classes in Lua, link Lua with XML and SQL, or create function jump tables. It's some code I put together while learning Lua. I started with existing algorithms as a basis while figuring out how to do these topics, since I'm just getting my feet wet and anything more complex would probably be over my head. Some parts are based on work by Afforess (an immensely useful StringUtils library for serializing tables) and killmeplease (a great Emigration Mod that helps with the ignore-happiness problem in the vanilla game). If either author would prefer me to use different algorithms as a basis though, I will try and find something.

First I created a queue class to track average happiness, stored in savegame files. This comes in several parts.

The class has all the standard queue methods: new, push, pop, size. I did add something nonstandard to the push method: automated size management, useful for this particular application since the upper bound is fixed and I never pop manually. I also created an average() method which isn't a typical part of queues, but again was useful here. Then comes main, load, and save methods using instantiated objects of this queue.

Spoiler :
PHP: 
--
--
-- Queue class
--

Queue = { }
 
function Queue:new(o)
    o = o or {};
    setmetatable(o, self);
    self.__index = self;
    return o;
end
 
function Queue:push( value )
    table.insert(self, value);
    if (self:size() >= QUEUE_CAPACITY) then
        self:pop();
    end
end
 
function Queue:pop( )
    if self:size() == 0 then
        return nil;
    end
 
    local val = self[self.first];
    table.remove(self, 1);
    return val;
end
 
function Queue:size( )
    return #self;
end

function Queue:average()
    local sum = 0;
    for id,val in pairs(self) do
        --printDebug(self[i]);
        sum = sum + val;
    end    
    --printDebug("sum = " .. sum);
    --printDebug("self:size() = " .. self:size());
    return math.floor(sum / self:size());
end


--
-- Globals
--

DebugMode = true;
QUEUE_CAPACITY = GameInfo.Emigration["HappinessAverageTurns"].Value * GameInfo.GameSpeeds[PreGame.GetGameSpeed()].GrowthPercent / 100;
iProbCityEmigration = {};
AverageHappiness = {};
HappinessTable = nil;

--
-- Main
--

-- Based off the Emigration Mod.

function doEmigration()
    -- Thank you to Afforess for making his code open source. This save/load procedure is based off his work.

    if not HappinessTable then
        doGameInitialization();
    end

    updateHappinessInfo();

    -- for each major civ
    for iPlayerID,pPlayer in pairs(Players) do
        if isValidPlayer(pPlayer) and not pPlayer:IsMinorCiv() then
            printDebug("=== " .. getPlayerName(pPlayer) .. " ===");
            local iAverageHappiness = HappinessTable[iPlayerID]:average();
            printDebug("iAverageHappiness = " .. iAverageHappiness);
            
            -- if unhappy
            local bShouldEmigrate = (pPlayer:GetExcessHappiness() < 0 and iAverageHappiness < 0);
            if pPlayer:GetNumCities() == 1 then
                for pCity in pPlayer:Cities() do
                    if (pCity:IsCapital()) and (pCity:GetPopulation() == 1) then
                        bShouldEmigrate = false;
                    end
                end
            end
            
            local iNumEmigrants = math.floor(-iAverageHappiness / GameInfo.Emigration["NumEmigrantsDenominator"].Value) + 1;

            -- do emigration
            if bShouldEmigrate then    
                doEmigratePlayer(pPlayer, iNumEmigrants);
            end
        end
    end
    saveGameData();
end

--
-- Utility Functions
--

-- Thank you to Afforess for providing  his code open source. This save/load procedure is based off the Revolutions Mod.

function doGameInitialization()
    HappinessTable = {};
    for iPlayerID,pPlayer in pairs(Players) do
        if (isValidPlayer(pPlayer)) then
            local SaveData = pPlayer:GetScriptData();
            --printDebug("Attempting to Load Save Data");
            if SaveData == "" then
                HappinessTable[iPlayerID] = Queue:new();
            else
                --printDebug("Loading Save Data");
                --printDebug(SaveData);
                SaveData = stringToTable(SaveData, QUEUE_CAPACITY, 1, {","}, 1);
                HappinessTable[iPlayerID] = Queue:new(SaveData);
            end
        end
    end
end
------------------------------------------------------------------------------
function saveGameData()
    for iPlayerID,pPlayer in pairs(Players) do
        if isValidPlayer(pPlayer) then
            --printDebug("Attempting to Save Data");
            local tbl = tableToString(HappinessTable[iPlayerID], QUEUE_CAPACITY, 1, {","}, 1);
            --printDebug("Save Data Created");
            --printDebug(tbl);
            pPlayer:SetScriptData(tbl);
        end
    end
end

The next part is where I tied in an XML file with Lua. The second half of the XML file is actually a jump table: a list of function names the program calls depending on input (C++ programmers can think of this as a function pointer table). The reason for doing a jump table is the function names can be stored externally. This increases modding flexibility, the user doesn't have to recompile the Lua to change things, and it allows the Lua code to have a very simple loop structure. Following the XML are two methods I created using this external data.

Spoiler :
PHP: 
<GameData>
    <!-- Table definition -->
    <Table name="Emigration">
        <Column name="Type" type="text"/>
        <Column name="Value" type="variant" default="0"/>
    </Table>
    <Table name="EmigrationWeights">
        <Column name="Type" type="text"/>
        <Column name="IsCityStatus" type="boolean" default="false"/>
        <Column name="Value" type="variant" default="0"/>
    </Table>
    <!-- Table data -->
    <Emigration>
        <Row>
            <Type>HappinessAverageTurns</Type>
            <Value>10</Value>
        </Row>
        <Row>
            <Type>MaxPasses</Type>
            <Value>3</Value>
        </Row>
        <Row>
            <Type>NumEmigrantsDenominator</Type>
            <Value>10</Value>
        </Row>
    </Emigration>
    <EmigrationWeights>
        <Row>
            <Type>EmigratedOnceAlready</Type>
            <Value>0.5</Value>
        </Row>
        <Row>
            <Type>IsRazing</Type>
            <IsCityStatus>true</IsCityStatus>            
            <Value>0.0</Value>
        </Row>
        <Row>
            <Type>IsResistance</Type>
            <IsCityStatus>true</IsCityStatus>
            <Value>1.0</Value>
        </Row>
        <Row>
            <Type>IsPuppet</Type>
            <IsCityStatus>true</IsCityStatus>
            <Value>2.0</Value>
        </Row>
        <Row>
            <Type>IsOccupied</Type>
            <IsCityStatus>true</IsCityStatus>
            <Value>1.5</Value>
        </Row>
        <Row>
            <Type>IsBlockaded</Type>
            <IsCityStatus>true</IsCityStatus>
            <Value>0.5</Value>
        </Row>
        <Row>
            <Type>IsCapital</Type>
            <IsCityStatus>true</IsCityStatus>
            <Value>0.5</Value>
        </Row>
        <Row>
            <Type>GetGarrisonedUnit</Type>
            <IsCityStatus>true</IsCityStatus>
            <Value>0.5</Value>
        </Row>
    </EmigrationWeights>
</GameData>
Spoiler :
PHP: 
function updateEmigrations(pPlayer)
    -- Based off the Emigration Mod.
    local cityWeight = {};
    local cityProb = {};
    local totalCulture = 0;
    local totalWeight = 0;
    local cityID = 0;
    
    -- figure out weight
    for city in pPlayer:Cities() do
        cityID = city:GetID();
        cityWeight[cityID] = 1;
        
        -- retrieve function and weight from jump table stored with XML
        for row in GameInfo.EmigrationWeights() do
            if row.IsCityStatus and city[row.Type](city) then
                cityWeight[cityID] = cityWeight[cityID] * row.Value;
            end;
        end
        
        -- factor in culture
        cityWeight[cityID] = cityWeight[cityID] / city:GetJONSCultureThreshold();
        totalWeight = totalWeight + cityWeight[cityID];
        
        cityID = cityID + 1;
    end
    
    -- probability = contribution to total weight
    for city in pPlayer:Cities() do
        iProbCityEmigration[city:GetID()] = 100 * cityWeight[city:GetID()] / totalWeight;
    end    
end
------------------------------------------------------------------------------
function shouldCityEmigrate(passNum, city)
    -- Based off the Emigration Mod.
    local p = Map.Rand(100, "City emigration probability - Lua")
    printDebug("  " .. iProbCityEmigration[city:GetID()] * 2 ^ (passNum - 1) .. " >= " .. p .. "?");

    if ((iProbCityEmigration[city:GetID()] * 2 ^ (passNum - 1)) >= p) and (city:GetPopulation() > 1) then
        -- less likely to emigrate from a city multiple times
        iProbCityEmigration[city:GetID()] = iProbCityEmigration[city:GetID()] * GameInfo.EmigrationWeights["EmigratedOnceAlready"].Value;
        return true;
    end;
    return false;
end

Anyway, hopefully this is helpful for someone's modding efforts out there so you don't have to figure out things from scratch. The code is all tested and works properly, you can download it below.
 

Attachments

Thank you much. I caught your conversation with Afforess the other day about this and was picking it apart trying to figure out where the string functions were hiding. Expect to see your name in my Special Thanks. :)
 
Would you be willing to create a wrapper function for SetScriptData() and GetScriptData() so that multiple objects could be stored and retrieved by key-name?

Ie:
Code: 
setScriptDataItem( plot, "key1", string );
setScriptDataItem( plot, "key2", integer );
setScriptDataItem( plot, "key3", boolean );
setScriptDataItem( plot, "key4", object1 );
setScriptDataItem( plot, "key5", object2 );

if getScriptDataItem( plot, "key3" ) then
  do something...
end

If you don't, I will, but I suspect your code would be cleaner.
 
I'm having trouble getting this to work. I suspect it has something to do with size and or depth on my call to tableToString. I don't understand how those values are being used. Or it might be my syntax in SaveData:set.

Code: 
--===========================================================================
-- SaveDataUtils
--===========================================================================
--[[]]

include( "StringUtils.lua" );

-------------------------------------------------
-- SaveData Class
-------------------------------------------------
SaveData = { }
 
function SaveData:new(o)
	o = o or {};
	setmetatable(o, self);
	self.__index = self;
	return o;
end
 
function SaveData:set( key, value )
	self[key] = value;
end
 
function SaveData:get( key )
	return self[key];
end
-------------------------------------------------
-- END SaveData Class
-------------------------------------------------

function setData( target, key, value )
	local saveData = target:GetScriptData();
	if saveData == "" then
		saveData = SaveData:new();
	else
		saveData = stringToTable( saveData, 3, 1, {","}, 1 );
		saveData = SaveData:new( saveData );
	end
	saveData:set( key, value );
	saveData = tableToString( saveData, 3, 1, {","}, 1 );
	target:SetScriptData( saveData, "test" );
end

function getData( target, key )
	local saveData = target:GetScriptData();
	if saveData ~= "" then
		saveData = stringToTable( saveData, 3, 1, {","}, 1 );
		saveData = SaveData:new( saveData );
		return saveData:get( key );
	end
end

--===========================================================================
--END SaveDataUtils
--===========================================================================
 
I believe I've narrowed the problem down to the sizeOfTable function in the StringUtils. I found the following info on the web:

myTable = {}
myTable["foo"] = 12
myTable["bar"] = "blah"
print(#myTable) -- this prints 0

Instead I have to iterate through the items in the table to get the number of keys.

numItems = 0
for k,v in pairs(myTable) do
numItems = numItems + 1
end
print(numItems) -- this prints 2
Click to expand...

My test show that this proves true. So because I'm using keys instead of numbers, the size of the table is misinterpreted and the tableToString function is returning an empty string.

If you can tell me how to fix the sizeOfTable function, that would be great. Otherwise I'm gonna keep beating my head against this wall. :)
 
I'm working off of KillmePlease's new serialization package right now. It's almost ready for use. My old one was just a pile of hacks that worked, lol.
 
To be honest I usually ponder about a subject for a few days to think it through before really starting work on it, so you could probably finish itemized data storage/retrieval faster than I can. :)
 
Yeah I've been at it all day. I have it mostly working. Will hopefully post it tonight.
 
I'm so close I hate to quit for the night. Just working out one final bug with unserializing multiple nested tables. On the bright side, no need for 'depth' or 'count'. Instead it uses open and close delimiters when serializing a nested table.

Hopefully will post the code within another day.
 
Here it is: version 1.

Many thanks to killmeplease for the serialize(), deserialize() functions, and the idea of spanning mods.

It allows for ITEMIZED data storage across mods. It does not currently load to cache, but I'm working on it. I assume it works across saves, but I haven't tested that.

Currently, the load function will return the entire save table if no mod name is designated, and additional class functions for retrieving different kinds of data groupings can be easily implemented.

SaveUtils.lua
Spoiler :
Code: 
-- vymdt.01.2010.10.19.0000
-- Created by: Whys -Open source
--===========================================================================
-- SaveUtils
--===========================================================================
--[[
Credits: killmeplease -- serialize() and deserialize().
Special Thanks: Thalassicus, Afforess.
]]

-----------------------------------------------------------------------------
-- Save Class
-----------------------------------------------------------------------------
--[[
Allows SetScriptData() and GetScriptData() to behave as a hash-table of
tables while maintaining seperation of individual mod data.
ie:
	save( "myMod",   pPlayer, key1, table );
	save( "myMod",   pPlayer, key2, table );
	save( "someMod", pPlayer, key1, table );
	save( "myMod",   pPlot,   ...., ..... ); --same as above.
	save( "myMod",   pUnit,   ...., ..... ); --same as above.

	load( "myMod",   pPlayer, key1 );
	load( "myMod",   pPlayer, key2 );
	load( "someMod", pPlayer, key1 );
	load( "myMod",   pPlot,   .... ); --same as above.
	load( "myMod",   pUnit,   .... ); --same as above.
	load( nil, pPlayer ); --all data for pPlayer.
	load( nil, pPlot );   --all data for pPlot.
	load( nil, pUnit );   --all data for pUnit.
]]
Save = {};

function Save:new( o )
	o = o or {};
	setmetatable( o, self );
	self.__index = self;
	self:initKeys( o );
	return o;
end

-------------------------------------------------
-- Save:initKeys()
-------------------------------------------------
--[[
Deep assignment of keyed values.  Required for
duplication of table data assigned by key.
]]
function Save:initKeys( o )
	for k,v in pairs( o ) do
		self[k] = v;
		if type( v ) == "table" then
			self:initKeys( v );
		end
	end
end
 
-------------------------------------------------
-- Save:set()
-------------------------------------------------
--[[
Sets the given value to the given key for the
given mod.
]]
function Save:set( mod, key, value )
	self[mod] = self[mod] or {};
	self[mod][key] = value;
end
 
-------------------------------------------------
-- Save:get()
-------------------------------------------------
--[[
Returns value for given mod and key.  Returns
self when mod not given.
]]
function Save:get( mod, key )
	local r = self;
	if mod ~= nil and self[mod] ~= nil and key ~= nil then
		r = self[mod][key];
	end
	return r;
end

-----------------------------------------------------------------------------
--END Save Class
-----------------------------------------------------------------------------

-------------------------------------------------
-- save()
-------------------------------------------------
--[[
Saves the given value to the given key for the
given target of the given mod.
]]
function save( mod, target, key, value )
	local pSave = load( mod, target );
	pSave:set( mod, key, value );
	target:SetScriptData(serialize( pSave ));
end

-------------------------------------------------
-- load()
-------------------------------------------------
--[[
Loads the value for the given key for the given
target of the given mod.
]]
function load( mod, target, key )
	local pSave = Save:new(deserialize( target:GetScriptData() ));
	return pSave:get( mod, key );
end

-------------------------------------------------
-- serialize()
-------------------------------------------------
--[[
Created by: killmeplease.
]]
function serialize( tbl )
	local r = "{ ";
	local num = 0;
	for k,v in pairs( tbl ) do
		if num > 0 then
			r = r.." ";
		end
		r = r..k.."=";
		if type( v ) == "table" then
			r = r..serialize( v );
		else
			if v == false or v == true or type( v ) == "number" then
				v = tostring( v );
			else
				v = v:gsub('"', "\[QUOTE\]");
				v = v:gsub('{', "\[LCB\]");
				v = v:gsub('}', "\[RCB\]");
				v = "\""..v.."\""; -- string
			end
			r = r..v;
		end
		num = num +1;
	end
	return r.." }";
end

-------------------------------------------------
-- deserialize()
-------------------------------------------------
--[[
Created by: killmeplease.
]]
function deserialize( str )
	local tbls = {}; -- nested tables
	local topTbl = {};
	repeat
		topTbl = {}; -- clear variable for top tbl
		-- find first top-level table position in str, its values list and key.
		local strStart, strEnd, topTblKey, topTblVals = str:find("([%w]*)={%s([^{}]*)%s}");
		if topTblKey == nil then
			strStart, strEnd, topTblVals = str:find("{%s([^{}]*)%s}");
		end
		if topTblVals == nil then
			topTblVals = "";
		end
		--print(topTblVals);

		-- parse values:
		-- save string values of top table to a temp table
		local strings = {};
		for vstr in topTblVals:gmatch('"([^"]*)"') do -- match "..." string values
			vstr = vstr:gsub("%[QUOTE%]", '"');
			vstr = vstr:gsub("%[LCB%]", '{');
			vstr = vstr:gsub("%[RCB%]", '}');
			table.insert(strings, vstr);
		end
		-- cut string values from top table string and replace them with "@" symbol
		topTblVals = topTblVals:gsub('"[^"]*"', "@");

		-- parse key=value pairs
		stringNum = 1;
		for k,v in topTblVals:gmatch("(%S+)=(%S+)") do
			-- parse value
			if v == "#" then
				v = tbls[k];
				-- tbls[k] = nil;
			elseif v == "@" then
				v = strings[stringNum];
				stringNum = stringNum +1;
			else
				-- parse simple value, a number or a boolean
				local n = tonumber(v);
				if n ~= nil then
					v = n;
				elseif v == "true" then
					v = true;
				elseif v == "false" then
					v = false;
				end
			end
			-- parse key:
			local n = tonumber(k);
			if n ~= nil then
				k = n;  
			end
			-- add value to the table:
			topTbl[k] = v;
		end
		if topTblKey ~= nil then -- key is nil when it is a base table
			tbls[topTblKey] = topTbl;
			str = str:sub(1, strStart -1)..topTblKey.."=#"..str:sub(strEnd +1);
		else
			str = "#";
		end
		--print(str);
	until str == "#";
	return topTbl;
end

--===========================================================================
--END SaveUtils
--===========================================================================
-- Created by: Whys -Open source


Spoiler :
Code: 
include( "SaveUtils" );

function testThis()
	
	local pPlayer = GetPlayer();
	local t =  { 1, 2, 3, t={6, {7, 7.5}, 8}, a=55, b=56, zz={"alpha", "beta \"or\" {gamma}", false} };
	
	save( "myMod",   pPlayer, "key1", t );
	save( "myMod",   pPlayer, "key2", {"some value"} );
	save( "someMod", pPlayer, "key1", { 3, 2, 1, true } );
	
	print( "test1: "..serialize(load( "myMod",   pPlayer, "key1" )) );
	print( "test2: "..serialize(load( "myMod",   pPlayer, "key2" )) );
	print( "test3: "..serialize(load( "someMod", pPlayer, "key1" )) );
	print( "test4: "..serialize(load( nil, pPlayer )) );
end
Events.ActivePlayerTurnStart.Add( testThis );

Output:
Spoiler :
Code: 
test1: { 1=1 2=2 3=3 a=55 t={ 1=6 2={ 1=7 2=7.5 } 3=8 } zz={ 1="alpha" 2="beta [QUOTE]or[QUOTE] [LCB]gamma[RCB]" 3=false } b=56 }

test2: { 1="some value" }

test3: { 1=3 2=2 3=1 4=true }

test4: { myMod={ key1={ 1=1 2=2 3=3 a=55 t={ 1=6 2={ 1=7 2=7.5 } 3=8 } zz={ 1="alpha" 2="beta [QUOTE]or[QUOTE] [LCB]gamma[RCB]" 3=false } b=56 } key2={ 1="some value" } } someMod={ key1={ 1=3 2=2 3=1 4=true } } }
 
The only thing necessary for access is an UpdateDatabase statement in the project actions like other xml files. I've attached two projects where I create xml tables, the Emigration one and also Units balance (which uses a promotion swap table). Hopefully looking at these files might help you identify where the problem might be.
 

Attachments

killmeplease, I'm currently working on a resources from buildings mod and have had luck adding a new table to the database with the following XML.
Spoiler :

Code: 
<?xml version="1.0" encoding="utf-8"?>
<!-- Created by ModBuddy on 10/18/2010 10:36:36 AM -->
<GameData>
	<!-- Table definition -->
  <Table name="Building_ResourceChange">
		<Column name="BuildingType" type="text" reference="Buildings(Type)"/>
		<Column name="ResourceType" type="text" reference="Resources(Type)"/>
		<Column name="ResourceChange" type="integer"/>
	</Table>
	<!-- Table data -->
	<Buildings>
		<Row>
			<Type>BUILDING_ALCHEMIST</Type>
			<BuildingClass>BUILDINGCLASS_ALCHEMIST</BuildingClass>
			<Cost>1</Cost>
			<GoldMaintenance>1</GoldMaintenance>
			<Help>TXT_KEY_BUILDING_ALCHEMIST_HELP</Help>
			<Description>TXT_KEY_BUILDING_ALCHEMIST</Description>
			<Civilopedia>TXT_KEY_BUILDING_ALCHEMIST_PEDIA</Civilopedia>
			<Strategy>TXT_KEY_BUILDING_ALCHEMIST_STRATEGY</Strategy> <!-- not used. -->
			<ArtDefineTag>ART_DEF_BUILDING_FORGE</ArtDefineTag>
			<MinAreaSize>-1</MinAreaSize>
			<ConquestProb>0</ConquestProb>
			<HurryCostModifier>-1</HurryCostModifier>
			<IconAtlas>BW_ATLAS_1</IconAtlas>
			<PortraitIndex>28</PortraitIndex>
		</Row>
	</Buildings>
	<Building_ResourceChange>
		<Row>
			<BuildingType>BUILDING_ALCHEMIST</BuildingType>
			<ResourceType>RESOURCE_GOLD</ResourceType>
			<ResourceChange>1</ResourceChange>
		</Row>
	</Building_ResourceChange>
</GameData>

Of course this requires adding something to the building classes as well.

Spoiler :
Code: 
<?xml version="1.0" encoding="utf-8"?>
<!-- Created by ModBuddy on 10/18/2010 10:47:13 AM -->
<GameData>
	<BuildingClasses>
		<!-- Table data -->
		<Row>
			<Type>BUILDINGCLASS_ALCHEMIST</Type>
			<DefaultBuilding>BUILDING_ALCHEMIST</DefaultBuilding>
			<Description>TXT_KEY_BUILDING_ALCHEMIST</Description>
		</Row>
	</BuildingClasses>
</GameData>

Like Thalassicus says, just the usual actions will do it.

OnModActivated, UpdateDatabase, CIV5BuildingClasses.xml
OnModActivated, UpdateDatabase, CIV5Buildings.xml

Hope this helps.
 
Whys said:
I'm currently working on a resources from buildings mod and have had luck adding a new table to the database with the following XML.
Click to expand...

Does this actually let building provide resources, though?
 
Supercheese said:
Does this actually let building provide resources, though?
Click to expand...

I did not post the necessary Lua code, so just this, no.

But as for my Lua code, strategic, yes. Haven't tested luxuries yet. It just needs a couple more lines of code, but real life keeps interrupting. The issue is the only commands for increasing or decreasing a players resources are cumulative, so I have to keep a list of which buildings have already been applied and then look for any that have disappeared so as to subtract.

With luck, I'll have it uploaded sometime tonight.
 
Supercheese said:
Does this actually let building provide resources, though?
Click to expand...

You can find my full work on this here:
BuildingResources

Technically, it works, but with a couple issues I'm asking for help resolving. Mostly it requires an extra turn for certain things to update properly. But I'm not sure how to attach the function to a building created or building destroyed event. <shrug>
 
You must log in or register to reply here.

Similar threads

D
Custom properties on objects
Replies
2
Views
217
DeckerdJames
D
LilBudyWizer
Fun with Databases
Replies
2
Views
112
LilBudyWizer
LilBudyWizer
A
Need Help Disabling Ancient Ruins Reward
Replies
2
Views
165
Ashurean
A
B
Using python to interface with civ 5 using the FireTuner Port?
Replies
0
Views
103
Battary
B
B
Created a custom mod, it's shows up in the mod panel, I can enable it, but it's not changing anything. What are the steps I need to take?
Replies
0
Views
94
BruceLover69
B
Back
Top Bottom