DLL - Serialization

[S3rgeus]

Has anyone given adding new members/objects into the process of serializing and deserializing the game into/from save files? If so, any tips? This is obviously full of pitfalls, and I expect to crash my game many many times before I get it working, but I'm making my first attempts now.

 

[Hambil]

From lua? There are several, the most recent and probably complete is TableSaveLoader. The limitation is that SQLlite can only handle ints, strings, bools, and I think nil. So, storing anything other than tables is pretty impossible. But being able to store units and players and such and retrieve them would be hella cool if we could do it.

Perhaps I'm talking about something completely different. I usually am.

 

[whoward69]

For the C++ DLL code the easiest way is if you're extending an existing class/method - eg I added a m_bPassive flag to the espionage data to indicate if the spy was just in a city and not actually stealing tech - in which case you just need to a) decide where in the serialisation process you're going to inject your data, b) make sure you extract it at EXACTLY the same point when deserialising the object and c) remember that you have now broken every previous save, cloud save, autosave, etc that used your old DLL

Tips? Due to c, try and do it as infrequently as possible. If you can't avoid that, then look how Firaxis handle versions of their saved games (eg CvCity.cpp) and implement something similiar

 

[S3rgeus]

From lua? There are several, the most recent and probably complete is TableSaveLoader. The limitation is that SQLlite can only handle ints, strings, bools, and I think nil. So, storing anything other than tables is pretty impossible. But being able to store units and players and such and retrieve them would be hella cool if we could do it.

Perhaps I'm talking about something completely different. I usually am.

The ModMapData (MapModData?) that TableSaveLoader uses is something Firaxis implemented and handles the saving and loading from the actual files for us. TableSaveLoader is awesome, I'm really impressed with what it can do, but I'm looking at the serialization in the DLL that writes the files to disk by individually putting variables into a file stream. It would be great to be able to store units and players directly from Lua, but there are a lot of issues involved with that, and it's easier to store IDs and look up the objects after the game has been re-initialized, I'd say.

For the C++ DLL code the easiest way is if you're extending an existing class/method - eg I added a m_bPassive flag to the espionage data to indicate if the spy was just in a city and not actually stealing tech - in which case you just need to a) decide where in the serialisation process you're going to inject your data, b) make sure you extract it at EXACTLY the same point when deserialising the object and c) remember that you have now broken every previous save, cloud save, autosave, etc that used your old DLL

Tips? Due to c, try and do it as infrequently as possible. If you can't avoid that, then look how Firaxis handle versions of their saved games (eg CvCity.cpp) and implement something similiar

Awesome, I figured this was how it was working. Mostly I'm doing exactly that, adding additional member variables to existing classes and serializing/deserializing them in corresponding places in that class's read and write functions. It is definitely a hurdle that it breaks compatibility with all previous saves, but that "uiVersion" workaround Firaxis has used seems like it could work well for me. For now, since my mod isn't released and is in very early stages, I'm fine with breaking my own save files every now and again during development, but I'll need a save versioning approach soon.

 

[Pazyryk]

It seems like there are two choices if you really want to extend the player, unit, etc. objects in an advanced mod:

1. Build up parallel Lua objects and then make that data persistent.
2. Extend the actual dll objects.

There are many reasons why #2 is better. It's certainly more elegant. I went with #1 mostly because I needed this capability long before the dll was available. There is one remaining advantage for #1 though, which is that you don't have to do much when new dlls start magically dropping out of the steam cloud, which they will with the new expansion.

 

[S3rgeus]

I've just discovered FAutoVariable in the CvUnit.h file and after some reading and looking at how CvUnit.cpp handles reads/writes, I do believe this FAutoVariable thing may be magic.

Tell me if I'm getting this wrong, but I do so unbelievably hope I've got it right, but it looks like a member of a class declared as:

FAutoVariable<MemberClass,ContainerClass> MemberName;

Is automatically serialized as type MemberClass when the associated ContainerClass instance is serialized? That would be mean we don't need to manage the order of serialization manually if we use FAutoVariable wrappers. This would preclude you from using the uiVersion workaround to load additional members later, so the default reads/writes would still have to be used for those.

It seems like there are two choices if you really want to extend the player, unit, etc. objects in an advanced mod:

1. Build up parallel Lua objects and then make that data persistent.
2. Extend the actual dll objects.

There are many reasons why #2 is better. It's certainly more elegant. I went with #1 mostly because I needed this capability long before the dll was available. There is one remaining advantage for #1 though, which is that you don't have to do much when new dlls start magically dropping out of the steam cloud, which they will with the new expansion.

I went for elegance for now with serialization. I am very impressed by TableSaveLoader, I remember seeing it before the DLL source was made available and it was (still is) extremely powerful as a Lua-only solution to persisting data.

On another note, I did get the original serialization stuff working that I discussed in this topic, though I may do a quick overhaul if FAutoVariable works like it looks like it should.

 

[whoward69]

Mmmmm. I'm not sure that's what they're used for. To me a "sync archive" implies something that has to be sent over a network to ensure several machines are in the same state - as you would need for multiplayer. Also, the various auto variable classes that deal with vectors mention "smallest deltas" - which is a common techniques for only sending what has actually changed to reduce network traffic / increase speed. There would never be any reason for just using a delta in a game save file - you would always need the full vector to be persisted. Also, if you look at most of the auto-variables (eg m_iHolyCityID in CvPlayer) they are explicitly read from / written to the game save file (via kstream). Finally, if they do actually work like that, why on earth did Firaxis ignore them and implement explicitly versioning in their saved objects?

Of course I may be totally wrong

 

[S3rgeus]

Mmmmm. I'm not sure that's what they're used for. To me a "sync archive" implies something that has to be sent over a network to ensure several machines are in the same state - as you would need for multiplayer. Also, the various auto variable classes that deal with vectors mention "smallest deltas" - which is a common techniques for only sending what has actually changed to reduce network traffic / increase speed. There would never be any reason for just using a delta in a game save file - you would always need the full vector to be persisted. Also, if you look at most of the auto-variables (eg m_iHolyCityID in CvPlayer) they are explicitly read from / written to the game save file (via kstream). Finally, if they do actually work like that, why on earth did Firaxis ignore them and implement explicitly versioning in their saved objects?

Of course I may be totally wrong

That does make sense, but I'm still encouraged by some related comments in CvUnit.cpp. If you look at the write function in CvUnit, you'll see this above a lot of integer member serializations:

// slewis - move to autovariable when saves are broken

Also, none of the FAutoVariable members in CvUnit are serialized in CvUnit's write function. I'm thinking that some enterprising developer wrote the FAutoVariable class when they were part (most?) of the way through development as a way of automatically managing their save file system, but it wasn't adopted everywhere because the other one was already used by so many people.

I'll actually test this and see what it does tomorrow. Until then I can hope.

 

[whoward69]

We know Firaxis broke the save format around build 511, so makes me wonder why it wasn't done then. Also the date in FAutoArchiveClassContainer.h is 07/20/2009 - that's pretty inefficient implementation practices if they really haven't managed to fully utilise that code in almost 4 years!

Still, as you say, we can but hope

 

[S3rgeus]

As one would expect, this isn't quite as magic as it first looks. FAutoVariable data members have certain requirements of the structure of their container class. There are a couple of methods they need to implement (debugDump and stackTraceRemark) and their default constructors need to fit a certain format, which you can see if you look a the CvUnit constructor. I'd say this is why they didn't use it elsewhere, the default constructors for things like CvMap are already laid out in a way that 'works' and modifying them to work with FAutoVariable probably breaks pretty much everything.

 

[whoward69]

Civ 6 then