SaveUtils.lua -- Itemized Data Storage Across Mods

[robk]

I just noticed some weird behavior by the Tuner console that was tripping me up a bit: I guess it won't print out strings over a certain length. I had a 3 dimensional array that I was trying to save and the load function didn't appear to work in testing. However, after poking around a bit, I realized that wrapping a string.len() around the load function told me that the length was 10k+ characters, just that the tuner was quietly refusing to print it. I did manage to get it to print out a string of length 690 or so characters so the cut off is somewhere in between.

Anyway, just a warning in case someone else sees the same thing.

 

[Ryan F. Mercer]

I wonder what the ScriptData character limit is. I've designed the serialize function to be very space efficient, but I assume ScriptData has a built-in limit. Possibly system specific?

 

[lemmy101]

Hello! awesome tool , but I've yet to manage to get it to work

I've got a table structure as follows:

PlayerData[nNumPlayers].buildings[nNumBuildings]
PlayerData[nNumPlayers].piety
PlayerData[nNumPlayers].etc

with a whole bunch of data in PlayerData[x] that my mod uses.

I'm doing:

save(pPlayer, "playerdata", PlayerData[pPlayer:GetID()]);

and...

PlayerData[pPlayer:GetID()] = load(pPlayer, "playerdata");

and I'm getting so random and strange results I can't figure out where to start figuring out what's gone wrong.

It seems like at best I get PlayerData[pPlayer:GetID()] as a blank table. And worse I seem to be getting this BEFORE I've put any data into there. Does it not return nil if the key isn't present in the save data? Other times it seems like I get the results as a child entry in the returned table, I may be reading it wrong but I've yet to get any useful data from the system.

I'm sure it's me being thick, however, but I did follow the instructions in the comments.

Thanks again!

lemmy

 

[lemmy101]

I wonder what the ScriptData character limit is. I've designed the serialize function to be very space efficient, but I assume ScriptData has a built-in limit. Possibly system specific?

I doubt it. In this day and age they'll be using std::string in the game code. I'm pretty sure they did that in Civ 4 too.

 

[lemmy101]

I just tested something, and the problem seems to be down to load getting different data back between Lua States (which is the primary reason I'm using it to do).

[ Lua State = TopPanel ]
> print(load(Players[0], "playerData").piety)
 TopPanel: nil
[ Lua State = ISDoctrineEntry ]
> print(load(Players[0], "playerData").piety)
 ISDoctrineEntry: 1

In my main lua file's state:

print(load(Players[0], "playerData").piety) == 1

In the TopPanel.lua state:

print(load(Players[0], "playerData").piety) == nil

I thought that the script data that load/save use should be consistent between Lua states?

EDIT: AHHH I disabled the cache and that seems to have fixed it. SORRY. Awesome tool btw: Sorry I just noticed I've posted separately instead of editing.

YAY! <3 Thanks again for the awesomely useful script!

 

[robk]

lemmy: Did you make sure to set MY_MOD_NAME in the TopPanel state? I kept forgetting that in my testing and, when I did, the load() function would give some strange results. It doesn't do any checking for that and, unlike save(), won't spit out any errors when the mod name is nil.

Edit: oops, just noticed you posted again. I agree, great tool!

 

[Ryan F. Mercer]

lemmy, was the cache working improperly or was it just a misunderstanding?

load() can return different things depending on how you use it. That's intentional. For example, if you supply 'false' for the optional third parameter, load(player,nil,false), it will return the entire save table stored in ScriptData. That includes data saved by other mods to that target. If you supply nil for the key, load(player), then it returns your mod's data table, which includes all key-value pairs saved to the target by your mod. This way you can either grab a single item, all of your items, or all items for all mods.

 

[Ryan F. Mercer]

robk, is there some additional checking I should be doing? I'm doing some work on a new version right now anyway, so tell me if I need to add something.

 

[lemmy101]

lemmy, was the cache working improperly or was it just a misunderstanding?

I'm using it to serialize data between different Lua States, so when I was setting it in one it didn't seem to update the cache in the other somehow so I was getting old values out of it. It didn't seem correct behaviour but perhaps I've misunderstood how it works.

All sorted now anyway, it's working a treat with cache turned off.

Thanks!

 

[Ryan F. Mercer]

Okay, yeah, this is because the cache is a local global, not a superglobal. Concurrent mods can only see their own cache. Right now I'm working on some LuaEvents that will make it possible for each mod to inspect each other mod's cache. The tricky part is making it work generically, rather than having a unique LuaEvent for each mod that has to be called by name. We'll see what I can do.

 

[robk]

robk, is there some additional checking I should be doing? I'm doing some work on a new version right now anyway, so tell me if I need to add something.

My comment was more of an observation than a critique. However, if you want to prevent confusion, you could put something like this at the beginning of the load and save functions.

if (mod == nil and MY_MOD_NAME == nil) then
  print "Error: mod name not provided nor is MY_MOD_NAME set"
  return nil
end

 

[Ryan F. Mercer]

Version 5.

Fixed a bug with clearing cache.

Implemented ShareData functionality for sharing object and function references across concurrent mods as super-globals. When ShareData.lua is added to the InGame.xml, all mods using SaveUtils will operate on the very same super-global cache. This will avoid much deserialization while still remaining synchronized. Not to mention, the share() function allows you to share anything else you want in the same way, including object and function references.

If you were disabling caching before, now is the time to add ShareData to the InGame.xml and reenable caching.

I also added ample checking.

 

[Ryan F. Mercer]

Version 6.

InGame.xml no longer required!

Normally I wouldn't version so rapidly, but discussions with smellymummy and Onni in another thread have made it possible to implement shared data without the concern for execution order of individual mods.

 

[Valkrionn]

How was that done? Anything that could be applied to other mods using InGame?

 

[Ryan F. Mercer]

Can apply to many of the mods currently using InGame. Read the following thread.
http://forums.civfanatics.com/showthread.php?t=398450

 

[alpaca]

Hmm I wonder if it's possible to add your own mod entry point. The entry point type just looks like a normal string so if you use InGameUIAddin for a mod and add an addin entry point for other mods to it, you can determine loading order and so on automatically

 

[Ryan F. Mercer]

Load order is mostly resolved with proper use of events. You do things according to the order of events and that determines the order of code execution. But I like your thinking. Custom entry points sounds like functionality worth pursing in any case.

 

[Valkrionn]

Alright, question for you.

I'm attempting to convert my Dark Ages mod to use SaveUtils, in preparation for a general release (ensure as much compatibility as possible). However, I'm having a few problems.

Basically, I need to store three values. DarkAgeProgress, DarkAgeThreshold, and DarkAgeTurns. So I save as follows: save( pPlayer, "DarkAgeProgress", darkAgeProgress );

And then load: local darkAgeProgress = load( pPlayer, "DarkAgeProgress" );

From your example, it looks like it should work. However, when I try to use the value it says it's a table, rather than int. What exactly am I doing wrong?

Very tired atm, so it's not jumping out at me.

 

[Ryan F. Mercer]

Did you include the following line with the include statement?

include( "SaveUtils" ); MY_MOD_NAME = "???";

And did you set that to your own mod's name? If not, load will return the entire save table, which includes all items for all mods on the given target.

 

[Valkrionn]

Yep, I'd seen that part. "include( "SaveUtils" ); MY_MOD_NAME = "DarkAges--vmdy.02.11.25.2010";"