Nightinggale
Deity
- Joined
- Feb 2, 2009
- Messages
- 5,378
As referenced a few time on the forum, I came up with a new savegame format quite a while ago. Now it has finally reached a stage where it's usable (poorly tested though), though it won't be included in any releases prior to 2.8 because it's not backward compatible. Any 2.7.x release can read savegames from RaR 2.7 and any earlier version of WTP.
For most people this won't change anything. You still save and load like normal ingame and players will not have to learn how to do anything new. This thread is about how it indirectly affects players and more directly unlocks modding abilities without breaking existing savegames.
Less bugprone data layout
Vanilla (and all mods so far?) relies on data location to figure out what each byte is. This means the game expect all the variables to be stored in a specific order. This is problematic for two reasons. One is that programmers have to write code for writing and reading (two different locations) and that will break savegames if they mismatch. Another issue is if more data is added to the savegame. Savegames do have version numbers, but reading code, which reads one thing for version 1, another for 2, something else for 3 etc quickly becomes near impossible to maintain and can quickly become buggy. That's one reason why killing savegame backward compatibility is widespread.
The new savegame system doesn't do that. Instead each variable starts with a "header" telling what data is saved. This header is just 1-2 bytes (usually 1) and the reading code doesn't care which order the writing code has written the data. In fact it generally doesn't even care if all the data is present.
This will result in way easier modding and the mod is much less likely to break savegames when updating.
XML changes won't break savegames
Vanilla assumes the xml files to be identical when reading a savegame as they were while writing. We removed this assumption and while it is possible to break savegames (like removing all units in xml), generally speaking changing xml data will no longer break savegames. Particularly adding data (like a new unit) is safe. The order inside an xml file can in most cases be changed as well without issues.
Smaller savegames
When loading something, the game will set everything to the default values. Afterwards it will read the savegame and overwrite data. This means anything not present in the savegame will get the default value.
Example: a unit has 0 experience. Since 0 is the default experience, the savegame will not contain any mentioning of experience for that unit.
Removed a number of bytes, which always had the same value. For instance a bool was saved using 4 bytes, but 3 of them are always 0. Now it's saved using a single byte.
Those two changes combined have reduced the savegame size. My testing (new game on giant map) indicates it's a 40% file size reduction, though odds are that the difference will grow as the game progresses.
Allows compressed savegames
The dll now buffers the savegame meaning it has the data needed to compress the file. Actually doing this with a proper compression algorithm requires more work, but at least now it's possible (unlike vanilla). This means once it's released, the reduction in size is likely way better than 40%.
Faster network joining
When a player joins a network game, the server freezes the game, saves and transmits a savegame. Once done the game unfreezes. If the filesize is reduced by 40%, it would be reasonable to assume it affects the length of the freeze. However testing the effect here isn't trivial.
Unable to read big savegames issue
Some people have encountered an issue where the game fails to read savegames if they are bigger than a certain size. Working with the savegame format has revealed that this seems to be linked to a buffer issue inside the exe file. Interestingly if the debugger triggers a breakpoint just before it happens, once the debugger starts the game again, it can read the rest of the savegame. This is positive news as it opens op possibilities for workarounds, like maybe ask the game to sleep for 10 seconds after reading 2 MB to allow the exe to flush the buffer or something. Yeah I know waiting isn't awesome, but if the alternative is unreadable savegames, then it would enter the realm of acceptable. There are other tricks we can do, which might prevent this issue even without waiting.
The issue might be a non-issue though because it triggers after a certain amount of bytes. Now the savegames are smaller and if they are also compressed in the future, it might become unrealistic to save enough data to trigger this issue.
For most people this won't change anything. You still save and load like normal ingame and players will not have to learn how to do anything new. This thread is about how it indirectly affects players and more directly unlocks modding abilities without breaking existing savegames.
Less bugprone data layout
Vanilla (and all mods so far?) relies on data location to figure out what each byte is. This means the game expect all the variables to be stored in a specific order. This is problematic for two reasons. One is that programmers have to write code for writing and reading (two different locations) and that will break savegames if they mismatch. Another issue is if more data is added to the savegame. Savegames do have version numbers, but reading code, which reads one thing for version 1, another for 2, something else for 3 etc quickly becomes near impossible to maintain and can quickly become buggy. That's one reason why killing savegame backward compatibility is widespread.
The new savegame system doesn't do that. Instead each variable starts with a "header" telling what data is saved. This header is just 1-2 bytes (usually 1) and the reading code doesn't care which order the writing code has written the data. In fact it generally doesn't even care if all the data is present.
This will result in way easier modding and the mod is much less likely to break savegames when updating.
XML changes won't break savegames
Vanilla assumes the xml files to be identical when reading a savegame as they were while writing. We removed this assumption and while it is possible to break savegames (like removing all units in xml), generally speaking changing xml data will no longer break savegames. Particularly adding data (like a new unit) is safe. The order inside an xml file can in most cases be changed as well without issues.
Smaller savegames
When loading something, the game will set everything to the default values. Afterwards it will read the savegame and overwrite data. This means anything not present in the savegame will get the default value.
Example: a unit has 0 experience. Since 0 is the default experience, the savegame will not contain any mentioning of experience for that unit.
Removed a number of bytes, which always had the same value. For instance a bool was saved using 4 bytes, but 3 of them are always 0. Now it's saved using a single byte.
Those two changes combined have reduced the savegame size. My testing (new game on giant map) indicates it's a 40% file size reduction, though odds are that the difference will grow as the game progresses.
Allows compressed savegames
The dll now buffers the savegame meaning it has the data needed to compress the file. Actually doing this with a proper compression algorithm requires more work, but at least now it's possible (unlike vanilla). This means once it's released, the reduction in size is likely way better than 40%.
Faster network joining
When a player joins a network game, the server freezes the game, saves and transmits a savegame. Once done the game unfreezes. If the filesize is reduced by 40%, it would be reasonable to assume it affects the length of the freeze. However testing the effect here isn't trivial.
Unable to read big savegames issue
Some people have encountered an issue where the game fails to read savegames if they are bigger than a certain size. Working with the savegame format has revealed that this seems to be linked to a buffer issue inside the exe file. Interestingly if the debugger triggers a breakpoint just before it happens, once the debugger starts the game again, it can read the rest of the savegame. This is positive news as it opens op possibilities for workarounds, like maybe ask the game to sleep for 10 seconds after reading 2 MB to allow the exe to flush the buffer or something. Yeah I know waiting isn't awesome, but if the alternative is unreadable savegames, then it would enter the realm of acceptable. There are other tricks we can do, which might prevent this issue even without waiting.
The issue might be a non-issue though because it triggers after a certain amount of bytes. Now the savegames are smaller and if they are also compressed in the future, it might become unrealistic to save enough data to trigger this issue.
