Self-aware units

[Hambil]

So many edge cases... head going to explode... sleep now

 

[Hambil]

It's working. Now guess what I've been doing all day (and likely for the next day or two also)... here's a hint...

Serialize = function(self)
    -- do magic stuff keeping in mind we can only serialize simple values
	-- call save
	TableSave(gT, "Escort_Selfaware_Units")
  end,

  Load = function(self)
	TableLoad(gT, "Escort_Selfaware_Units")
	-- do magic stuff keeping in mind we have to convert some simple values
  end,

Object orientation does not *always* make things easier. My head hurts worse than working out the edge cases.

Attaching project just in case anyone wants a look at the alpha results of combining Self Aware Units, Utils - Modular Unit Panel, Whoward's plot interators, and a lot of my own glue and spit and sweat. TableSaverLoader will find it's way into eventually, too. I may need more sleep first though.

This is actually only half the project. I am going to likely take a break from this before doing the serialization and write the second half, which is Worker Control (workers can be assigned a city, or told to only build roads/railroads and will consider friendly combat units valid hiding places when in panic mode (they won't move if they have cover).

 

[Pazyryk]

I wouldn't necessarily recommend TableSaverLoader for small modular stuff that you want to add in a sort of pick & mix way. The main problem is that you have to plug into the save system (overwriting SaveMenu.lua is the way I do it). It's really good for managing 1000s of variables in a large conversion mod where other folks are forced to mod to the main mod.

Probably SetValue and GetValue are your best options here. In theory, these might cause a slowdown if you are doing dozens of SetValues per turn, since each DB transaction has a certain overhead (2 rotations of your hard drive, in fact). But I've never timed it before. It's possible that Civ5 does this in the background without much effect (but if that's true, then I'm not sure how GetValue could run before SetValue has successfully written to the DB).

TableSaverLoader gets around the transaction overhead by packing many 100s of inserts/updates into a single transaction. But if you call it more than once a turn (& on save) you are sort of defeating its purpose. It isn't fast per run (takes ~0.5 sec per run for the ~2000 data points that my mod persists).

The fastest and simplest way to persist data is still the ScriptData functions. Not all work and some that do are used by SaveUtils, unfortunately making this system hard to use modularly (unless everyone uses SaveUtils). IIRC, plot is not used by that. If you could figure out exactly which objects SaveUtils uses for ScriptData, you could uses another one for your needs. I use ScriptData for plots since there can be 10000 plots in a huge map (and that many data points does cause slowdown using TableSaverLoader).

 

[Hambil]

The AI currently takes care of all it's own escorting. This is a human player mod only so it would at most save once per turn. However, I thought the call every turn I was making was going to be ignored unless it was the turn before an autosave turn or something. I mean, I haven't looked into the code but why else say "save every turn for autosave" in the instructions?

So, overhead should be low no matter what util I use to save. I just think it makes sense to use a util rather than do it from scratch everytime.

If I am saving at the start of everyturn (blind to autosave turns) then I don't understand why I'd use a save util at all. I mean, the real savings in effort and consistency across my mods in using a util like TableSaverLoader would be in that it handles autosaves and handles other save events. The table serialization/deserialization is nice, but not half as important to me as not wanting to hunt down all Firaxes' strange save edge-cases.

Am I misunderstanding it's usage?

Edit: BTW TableSaverLoader is not really my issue here anyway My issue is just figuring out what to save and restore, and how and where to access it. I'm pretty sure I've got the where correct (base class serialization function) but the what is not so clear. Especially since I'm new to this style of object oriented programming. I'm use to stronger languages for OO like Java and C++. Back "in the day" we were just starting to use these kind of (what we called at the time psuedo methods) to mimic object oriented programming in Javascript. I see things have come a long ways - but that leaves me with catching up to do.

 

[Pazyryk]

Am I misunderstanding it's usage?

Perhaps. The intended use of TableSave is to run it just before a Save, and at no other time. It won't hurt to run at other times. But there is no reason to do it and it does have some overhead (even if no data is changed so no DB update). I guess you could say it's intended as a "bulk save" function for your Lua tables.

Here is what happens when you call TableSave(gT, "MyMod"):

• If there is not a MyMod_Data table in SavedGameDB (i.e., the very 1st time you use it in a game):
  • Create MyMod_Data table in SavedGameDB
  • Create a parallel Lua representation of MyMod_Data
  • Loop through all fields in gT, recursively looking through all fields in nested tables (skip tables that were already recorded in cases where we have multiple pointers to the same table).
  • Write each item as a single line in MyMod_Data in both the DB and the Lua representation
• If there is already a MyMod_Data table in SavedGameDB:
  • Loop through all fields in gT, recursively looking through all fields in nested tables.
  • Check each item against a Lua representation of MyMod_Data
  • If changed, update MyMod_Data in the DB and the Lua representation
• Write feedback info to MyMod_Info (create if it doesn't already exists)

For TableLoad(gT, "MyMod"), which you need only run at game init:

• Read each line in MyMod_Data in the DB
• Create Lua representation of MyMod_Data
• "Build out" gT back to it's original structure (works even if it is complex with multiple pointers to the same tables)

So, even if there isn't a single change in any data, you still have one insert transaction going to MyMod_Info. And if you have 1700 data items to persist (as I do) then Lua has to check all those against its internal representation of the DB each time you call TableSave (which takes ~0.3 sec).


Edit: one limitation here for OO is that TableSave doesn't deal with functions. It will throw an error if any nested data is not type: number, boolean, string or table.

 

[Hambil]

Edit: one limitation here for OO is that TableSave doesn't deal with functions. It will throw an error if any nested data is not type: number, boolean, string or table.

That's really an SQLite limit. Any save util would have the same issue. And yeah, a lot of objects being tossed around that I'm going to have to figure out how to serialize and then recreate.

BTW: I like the way you are caching and handling data. Good stuff! But, you are probably right, rolling my own will make more sense in this case. Maybe we need a TableSaverLoaderLite

 

[Pazyryk]

I like it because it is "install and forget". So now I have ~1700 persisted data items and I didn't have to worry about altering table structures, and not too bad on added turn time (though I'd love to make it even less).

If I were to re-write now (almost 2 yrs later), I'd do it a little differently:

1. Use metamethods to know whenever a persisted table was changed and what index exactly was altered. No need for the Lua representation of DB data.
2. Force modder to register all tables that needed to be saved (could be done during run time). This would sacrifice some flexibility but reduce code complexity.

Btw, after discussion on the other thread, I spent 2 days building a fully cacheable GameInfo replacement. It worked! When I attached it to my mod, it cut turn time by an average of about 1%. (To be fair, I worked very hard in my mod to not do a lot of GameInfo iterations, so there wasn't too much there to optimize anyway.)

 

[Hambil]

I think I've untangled this. From those base class serializers I can create a single table of the structure [iEscort][iWorker] and then also rebuild from that. Where both are unit IDs. It's worth noting this doesn't cover every way the self aware units could be used, but simply how I am using them. Choices are ultimately driven in this case by the user interface you choose for your mod. Mine still focuses on Combat units escorting Civilians. The base code is more flexible than that and could be used (by someone else if they desire - it was given to us all by Whoward) to have many to one follow relations ships and to have the ability for Combat units to be followers as well as leaders.

The example of this would be wanting multiple combat units to follow a single great general. This mod will not do that. The base code will. Feel free to build the UI for it

Edit: I'm also aware that I'm not saving blockers. That's okay. By the very nature of what a blocker is and how the code handles them, I don't need to save them.

 

[Hambil]

Working final product! Self aware units live!

I must say the work was worth it, I'm pleased with the outcome.

Once I complete the second half of this mod collection (Workzone) I'll publish.