Maintaining backward Savedgame compatability

[Impaler[WrG]]

Theirs been some discussion on if and how we will maintain backward savegame compatability. Ofcorse were all for backwards compatability so long as its doable. I've resently spent some time beating my head against the problem made no progress and started wondering if anyone else has done work on it.

I have some Mods which add new data to the game which realy must be saved, I can manage to do this with the pStream->Write/Read functions. Bute these crash any game that was saved under a different DLL. A flag for skipping reads needs to be included for each variable which isn't hard to do, the trick is ofcorse getting what dll version a game was saved under. I dont think Civ saves any kind of DLL version number in its save games but it definatly saves information about Mods, thus alowing it to throw warnings when theirs a mismatch.

So what should our strategy for backwards compatability be, piggybacking the DLL on the Mod name and requiring both to match before loading. Or recording the dll version in the game so pStream->read can be conditionaly skipped and Mod options properly configued when you load up an old save. I would like to see the DLL version used but how to read it without crashing the game too? Any Ideas?

 

[SimCutie]

Theirs been some discussion on if and how we will maintain backward savegame compatability. Ofcorse were all for backwards compatability so long as its doable. I've resently spent some time beating my head against the problem made no progress and started wondering if anyone else has done work on it.

I have some Mods which add new data to the game which realy must be saved, I can manage to do this with the pStream->Write/Read functions. Bute these crash any game that was saved under a different DLL. A flag for skipping reads needs to be included for each variable which isn't hard to do, the trick is ofcorse getting what dll version a game was saved under. I dont think Civ saves any kind of DLL version number in its save games but it definatly saves information about Mods, thus alowing it to throw warnings when theirs a mismatch.

So what should our strategy for backwards compatability be, piggybacking the DLL on the Mod name and requiring both to match before loading. Or recording the dll version in the game so pStream->read can be conditionaly skipped and Mod options properly configued when you load up an old save. I would like to see the DLL version used but how to read it without crashing the game too? Any Ideas?

I alread implemented some backward compatible save system.
it will read original (Firaxis) save file, Moded save file of curret mod version and previous mod version (backward by only one version only)
This is done by following method.
It has a C struct to store new variables added to the mod.
First, before I read my struct from file, I save the file position in the stream.
Then I read the struct from file stream and check if it is modded save file or original firaxis save file. A magic maker is used to do this..
If the magic marker is missing, it is firaxis save file, so I rewind file stream to file position saved before. I stop reading stream and the C struct is filled with default values.
If the magic marker is found, it is modded savefile.
Then I check version number, it version number matches with current version, it is OK to go.
If it version number maches with old version number, then rewind to saved file position like in case of Friaxis, and re-read with old version C struct. and old date is converted to current C struct.
If both version failes, this save file is too old version, then the loading is aborted and exit the game.
Here is code..

// This is called by  CvGame::read() of "CvGame.cpp": time to load global data.
void CvExt_read(FDataStreamBase* pStream)
{
  struct old_global_ext_st    // global variables saved in savefile. 
  {
    int iGlobalExtSize;      // size of this struct global_ext_st
    int iSaveVersion;      // version
    int iSaveMagic;        // save file magic maker
    int iSaveFlags;        // Save flags : unused
    int reserved1[4];      // for future use

    int iExtSizeTable[8];    // size of various struct ext_st
    int iNumUnitCreated;    // Total created unit count; used for UID counting
    int reserved2[15];      // for future use
  } old_Ext;

  FAssertMsg(sizeof g_Ext == EXTSAVE_GLOBALEXT_SIZE, 
    __FUNCTION__ ": Global data extension save size" );
  FAssertMsg(sizeof CvUnit::unit_ext_st == EXTSAVE_UNITEXT_SIZE,  
    __FUNCTION__": Unit data extension save size" );
  
  srand((int)time(NULL));  
  memset(&g_Ext, 0, sizeof g_Ext);

  size_t SavePos = pStream->GetPosition();
  pStream->Read( sizeof g_Ext, (byte *) &g_Ext);

  if ( g_Ext.iSaveMagic != EXTSAVE_MAGIC )  // Sanity check
  {  // Firaxis original save
#ifdef SIMCUTIE_UNITEXT
    CvUnit::cm_eExtLoad = EXLD_BASE;
#endif // SIMCUTIE_UNITEXT
    pStream->SetPosition(SavePos);      // rewind
    g_Ext.iNumUnitCreated = 0;        // *TODO*

  }
  else 
  {
    FAssertMsg( sizeof g_Ext >= g_Ext.iGlobalExtSize, 
      __FUNCTION__ ": Loaded Global data extension size" );
#ifdef SIMCUTIE_UNITEXT
    FAssertMsg( sizeof CvUnit::unit_ext_st >= g_Ext.EXTSAVESIZE(UNIT), 
      __FUNCTION__ ": Loaded CvUnit extension data size" );
#endif // SIMCUTIE_UNITEXT

    if ( g_Ext.iGlobalExtSize == sizeof(g_Ext) 
      && g_Ext.iSaveVersion == EXTSAVE_VERSION )
    {  // OK. Extension save file, current version.
#ifdef SIMCUTIE_UNITEXT
      CvUnit::cm_eExtLoad = EXLD_LOAD;
#endif // SIMCUTIE_UNITEXT
    }
    else if ( g_Ext.iGlobalExtSize == sizeof(old_Ext) 
      && g_Ext.iSaveVersion == EXTSAVE_VERSION_OLD )
    {  // old version.
      pStream->SetPosition(SavePos);
      pStream->Read( sizeof old_Ext, (byte *) &old_Ext);
      g_Ext.iNumUnitCreated = old_Ext.iNumUnitCreated;
#ifdef SIMCUTIE_UNITEXT
      CvUnit::cm_eExtLoad = EXLD_CONVERT;
#endif // SIMCUTIE_UNITEXT
    }
    else
    {  // unsupported save version, die...
      FAssertMsg( 0, __FUNCTION__ ": Unsuppoted save version" );
      ReportError( "Unsupported save version: %d ", g_Ext.iSaveVersion);
      //  gDLL->getInterfaceIFace()->exitingToMainMenu();
      // GC.getGameINLINE().doControl(CONTROL_LOAD_GAME);
      // gDLL->MessageBox( "Loading Save file failed: version mismatc", "Loading Error" );
      throw std::runtime_error("Loading Save file failed: version mismatch");
      //  exit(1);
      return;
    }
  }
}

 

[Impaler[WrG]]

This is an interesting solution but it dosn't alow us to load old games, mearly exits us from the game with a nice message rather then a crash when theirs an incompatable version. If we can get a vershion number out of the game then we could set up all of the reads something like follows..

On our special header file each mod has a version number at which it was incorporated, so for example this mod is active in our 1.55 release but wasn't in prior version and shouldn't be activated in previous versions or have any data read from a save that predates that version.

int MyMod.version = 1.55

Throughout the code we can check the mods flag to either perform modified behavior or maintain the original game behavior.

bool isMyModActive()
return MyModStatus;

The options menu could use version numbers to prevent activation of a Mod that incompatable with a currently running game. It could also disable incompatable mods at the loading of on old save but then the user to reactivate them when a New game is created (new game would be stamped with the DLL's version and would thus allow all mod).

bool ActivateMod(Mod)
if (Mod.version >= CurrentGameVersion)
return false; //failure
else
setModflag(Mod, true);
return true; //success

When we try to load a save game Version numbers for the Mod can be checked before any new data is loaded. If the save is old we just skip the load and deactivate the Mod to maintain the original save games content and the game behavior underwhich is was created.

if (Mod.version >= SaveVersion)
pStream->read(MynewVariable)

 

[The Great Apple]

I think the only way we could get a version number out of the save game would be to include the mod version in with the savegame. That would fix any backward compatabilty issues within the project, but would mean that save-games from anything before the mod was initially installed would not be compatable.

Now - an idea. I'll preface it by saying that I don't think I am anywhere near as experiened at C++ as the rest of you... so it may be nonsence!

Why not just use C++ standard exception handling (try - catch). We try to load the save-game mod version from the save, and if it isn't there, then we initialise the mod version to version 0.01. This will mean that new users won't get anything that changes the save game version when they use an old savegame, but they will get anything that doesn't. As soon as they start a new game the proper mod version is saved into the save game, and everything works as it is meant to.

Would this work?

 

[Impaler[WrG]]

I think if we use SimCuties tecnice of saving and restoring the Readstream then it would work, if the try block fails we restore the Stream and set version to "Firaxis Virgin" and disable everything. The user gets to continue their saved game as if they were playing a totaly unaltered dll, asuming they also have the proper Assets they will have a totaly transparent and backward compatable game.

 

[The Great Apple]

Sounds great!

Correct me if I'm wrong, but the user will still benifit from improvements which don't rely on saved data right?

I will add a bit in the guidelines later today describing how to make things backwardly compatable.

 

[Impaler[WrG]]

If they desire to activate them then yes, things like AI improvments should just be on all the time, other modifications that run through XML will require that the end user keeps their XML assets consistent through out a game, the games regular load protection dose this for us. Mods that are togglable, not controled by XML and dont require saved data could be activated in games created outside the dll, though personaly I am more interested in maintaining the rule set a game was created under. If a person wants a new feature they generaly want to start a new game with it. I dont realy see lots of people loading up the new dll to change rules in a game thats half played, it seems rather un-sportsmanly.

This gets me thinking, perhaps we are being excessive here. The DLL is under Assets and will thus be an integrated part of a Mod, the game already prevents mixing of games created under different Mods and as long as the enduser dosn't intentialy get around this with Custom Assets we will have all the backwards compatability we need. Players must simply keep Mod files containing old DLL versions to continue playing legacy games (or load no Mod to play plain Firaxis games). Perhaps publicly poling the general community would be wise.

 

[The Great Apple]

The thing is this is the sort of thing that should be going in the custom assets directory. That's where I'm going to stick my copy. It saves all the mucking around loading a mod everytime I want to play a game.

Basically I'm in favour of keeping full backward compatablity, and advising people to put it in their custom assets as it is a no-hassle solution. If there is a long list of instructions then it'll put people off - not good. It seems to me that "put the file here and it'll work on any new game" is pretty simple. We could even make a .exe installer/unistaller to make it even easier.

We already are planning on including a global toggle to switch ALL aspects of the mod off at the press of a button, so if they don't want to see it in any new game, they won't.

 

[12monkeys]

With the above mentioned method from SimCutie, we should be able to detect that the loaded game is a standard save game which needs to be upgraded to a custom save game.
Also, we should be able to switch on/off standard save game behavior, by disabling certain new features.

So we can guide the user a bit like this :
When the user loads such a standard save game while our features are enabled, we'll pop up small dialog saying "hey user, attention! The game you load is a standard game which will be upgraded to a custom save game as soon as you save it." This is just an info, so the user can only press OK.
As soon as the user is going to save the game, we'll pop up another dialog saying : "hey, we'll convert the game to custom game. This may make the save game inoperable with the stadnard game. Do you want to continue?". User can answer yes/no.
If he selects "yes", we'll save it. If he say "no", we'll give him the message "Go to option sscreen, switch to stadnard save game mode (which disables certain features) and save the game again". This makes the save game working as a standard game again.

To do so, we need to rate each feature in a save-game-relevant and save-game-non-relevant switch. We also have to adapt the read/write stream methods and we have to add the dialogs.

I'm not sure, what impacts this will have on contains features when we do so, so please comment.

12m

 

[The Great Apple]

Hmmm - I hadn't thought about new save games being able to be run in vanilla Civ 4.

If they load a standard game with our dll installed then none of our new things will be loaded from the save game. Now there are two options from here:
1) We initialise our new bits of data to a default value, and save this value when a new save is made
2) We don't use our new bits of data, and they are not saved when a new save is made - only when a new game is loaded.

Seems to me 2) will cover all situations, while 1) may cause some glitches with some operations which require accurate past data. For this reason I'd go for 2).

This will mean that any game not started with the right files will never be able to have the mod running fully on it, but installing the mod won't break old save games, and you can install it in custom assets where you can forget about it.

 

[Impaler[WrG]]

I personaly avoid using Custom Assets so that I dont get any conflicts with content their over-riding portions of an explicitly loaded mod. For Convenience I use the config file to set mods that I want to load. I can open, edit and save it faster then Civ would require to reload the game from Advanced options and as I am usaly loading a mod many time in a row I am golden untill I want to explicitly change it.

I've been thinking about it more and more and I'm going to come out against taking any extra efforts to maintain backward compatability under the reasoning that it is.

A. A possible source of buggy loading
B. Waste of effort better spent on making mods
C. Of low desirability to the community at large
D. Not nessary for the success of the project
E. Useless as the Dll will be combined with other content that breaks back compatability

None of the larger more elaborate mods maintain backward compatability yet it has not prevented large scale usage of these mods even as frequent undated versions are released. I have yet too see any posts on the forum by people asking for this kind of capacity. The current system of seperate mod folders and Mod names being stamped to save games is more then adiquite to alow a person to continue playing on old game under the rules it was created under and to start new games under different mods. This is what the average user desires, virtualy none would see a use for converting a game to new rules after it is started.

I think your reasoning about the use of Custom Assets and the mannor in which the typical user actualy uses downloaded Mod content is flawed. We are making a Mod component rather then a whole Mod. The makers of mods will use our dll as a piece of their mod, combining it with modified Python and XML files. They will want to make the use of their mod as simple as possible (because atempting to run their mod without it will most definatly crash the game resulting in a lot of wining noobs) so rather then telling their end users to go get version 1.2345 of the CCP dll from some obscure site called Source Forge they will simply included it in their Assets directory. Then the end user just drops the downloaded file in the Mods directory and loads, a much more idiot resistant system and its the current standard way mods are packaged and distributed. Custom Assets folder is realy for elite users such as our selves who MAKE customizations rather then just use them and have the skill to prevent, identify and correct conflicts between custom assets and mods. Most users dont ever use Custom Assets or know what it dose.

Now all that said a simple Version stamp and sanity check is a nice idea and as SimCutie alreay has it basicaly done its no work for future moding, his changes should definatly be included but no further effort need be taken in this regard

 

[The Great Apple]

You definatly have a point with the incompatablilty of other mods. However Blue Marble is a very good example of the opposite of this. That is a terrain mod which, as far as I can tell, is universally accepted to be better than the standard civ graphics, and the way it loads is seamless - you double-click on an .exe file, and it all works (I think it loads in Custom Assets... not positive). Also - I think Custom Assets won't do anything if the same file exists in the mod folder, thus preventing conflicts.

E. Useless as the Dll will be combined with other content that breaks back compatability

Not neccessaily. People may download the mod as a standalone mod, and it should work fine as a standalone mod. They won't see the improvements made to the moddability, but everything else will have an effect. It can be a mod, or it can be a component in a mod.

Having said all this - you are right about it making the whole modding process just a bit more complicated. I think, as you said, a poll would be a good idea, though people tend to want everything so we have to be careful interpreting the results.

 

[The Great Apple]

Currently it seems we're not quite sure what direction we're going with this.

Where I stand:

1) Games need not be compatable with Firaxis default. They should give an error message when default games are loaded, but should not CTD.

2) Games should usually be compatable with newer versions of the project. While most versions should maintain compatablilty within the project, not all versions will maintain save game compatablity (for practical reasons).

 

[SimCutie]

Currently it seems we're not quite sure what direction we're going with this.

Where I stand:

1) Games need not be compatable with Firaxis default. They should give an error message when default games are loaded, but should not CTD.

2) Games should usually be compatable with newer versions of the project. While most versions should maintain compatablilty within the project, not all versions will maintain save game compatablity (for practical reasons).

I agree on both point. My current work is just that. It will read original game. But I am not sure that the new DLL features will work as intened just like game started with DLL installed. But it has facility to provide such compatibility if individual developer have a will to do so.
I agree that newer version of our project DLL should be largely compatible to save file from our old project version. So it needs careful design from the start for maintaining compatibility.

 

[Olleus]

Personaly, I don't mind if we break backwards compatibility with every release, as long as an error message comes up and it doesn't CTD.

 

[SimCutie]

Personaly, I don't mind if we break backwards compatibility with every release, as long as an error message comes up and it doesn't CTD.

Yes, it depends on personal taste. It is not important to players who enjoys quick or short game. But it will be essential for playes who loves long marathon game or PBEM/succession game. I too personally does not condiser it paramount importance for actual game playing (I do short game). But we should satisfy at least, say, 90% of players out there. Some players surely will complain on save file incompatibility on different version and consider it as indication of low quality or technical deficency of our project outcome.

 

[12monkeys]

Currently it seems we're not quite sure what direction we're going with this.

Where I stand:

1) Games need not be compatable with Firaxis default. They should give an error message when default games are loaded, but should not CTD.

2) Games should usually be compatable with newer versions of the project. While most versions should maintain compatablilty within the project, not all versions will maintain save game compatablity (for practical reasons).

OK by me!

 

[Impaler[WrG]]

The long marathon players simpley need to put the Dll in a Mod folder, clear the CustomAssets and hold onto said Mod folder and use it to run their long game. They dont need to download new versions if they want to complete their long games.

 

[The Great Apple]

The long marathon players simpley need to put the Dll in a Mod folder, clear the CustomAssets and hold onto said Mod folder and use it to run their long game. They dont need to download new versions if they want to complete their long games.

This is a point, although I still feel it would be nice to try to have some updates that don't break backward compatability. However, it certainly isn't a neccessity.

 

[The Great Apple]

We're agreed then. To say it again, to be extra sure (I've reworded #2):

1) Games need not be compatable with Firaxis default. They should give an error message when default games are loaded, but should not CTD.

2) Games should aim to usually be compatable with newer versions of the project. While some versions will maintain compatablilty within the project, not all versions will maintain save game compatablity (for practical reasons).