That sounds like an interesting idea for being able to re-play the game. I definitely think it should be doable and save space over storing each file - but indeed, the "how" is not easy. Does it make sense to pull the save apart and store changes in the database, and if so, how? I've never dealt with the code of programs that deal with this sort of problem, source control coming to mind, but there's surely a way that's better than storing every file.
Oh heck yeah. I'm not using all the data from the file, so I'll only put what I might want in the DB. The easiest thing to do is probably to assume that files are imported in chronological order, and that a hash of world seed + game/world settings uniquely identifies a game, get that working and then deal with customized maps, replayed maps and edge cases. In that case, after I determine the map in the current save file matches seed & settings from one already imported, compare each tile and only store updated tiles. Then through some magic I haven't really figured out yet, have an index that can query the proper set of data upon demand.
I know existing analytics tools can have data tagged with from/to timestamps, so some permutation of that could work for slicing the 3D map (time/turn being the 3rd dimension) into a 2D map.
Once the basics work, dealing with edge cases might involve fingerprinting the base terrain at preset intervals, hashing some sort of table of civ IDs, names and start locations and other such things.
In the short term, I think the #1 thing necessary to get Civ3 Show-And-Tell to where it can be widely used is the ability to easily upload a save, and get back an SVG. Redirecting might solve it well enough for external sites to work, although IMO it would be nice to also have a "Upload from my computer" button. If you add a "Save SVG" button that will let the user save the SVG file locally, then they can re-upload it and it shouldn't be necessary to keep them forever on your server (although what's a good hosting site for SVG files? I don't know yet. So if space isn't prohibitive, being able to keep them and using a link might make it more attractive to Civ3 players).
I tend to over-plan and over-think things. I can probably store and serve everything myself and not run out of space or bandwidth even if every actively-posting Civ3 user on this site used it weekly or maybe even daily.
After some more thought today, an architecture is emerging:
File provider --> validator/decompressor --> importer --> storage
Map/data UI pulls from storage
The file provider in the case of the web app is the upload or URI fetch UI. In the case of a local version of the app, this would be the part that watches the autosave folder for updated files.
The validator/decompressor will have whatever sanity/security checks I want and ensure a decompressed save file gets fed to the importer.
The importer will have any compression concerns removed and simply turn uncompressed save game data into stored data. Today the storage is an SVG file. In the future it may be a MongoDB data store. I will need to include some logic to the current version to store separate SVGs without overwriting old ones. And if I put in a DB backend the importer may be the place to put the deduplication logic. (Hmm, actually I don't think I want those in the importer...I might need another program in the data flow. Maybe the importer should be split in two: a converter that outputs structured data from the save file and the importer that worries about how it needs to be stored.)
The UI will be a web browser for the foreseeable future, and probably forever. This will work whether the app is local or hosted. Currently it's just viewer.html, but anything I can think of doing can be done with HTML and Javascript. The "storage" is currently an SVG file, but even if it goes to a DB backend I'll have an API the browser can call to get the map data.
Regardless, I'm glad to see an update post, even if it's just thoughts on future direction. I still think this is one of the most promising utilities in development, particularly for the interactive, community-focused part of Civ3, and despite the disclaimer in the first post that you'd never follow through on it, I think you're very close to actually having followed through. I definitely understand the not having time to work on it part, though. I'm just glad that there probably is a future for it.
Thanks! It is close to being usable, isn't it? I want it to have some more visual polish and have the resource icon set complete, but it really doesn't
need that to be usable, does it?
And given my architecture thoughts, I could make the "file provider" (uploader), figure out how to handle the multi-file SVG storage and retrieval URIs, get it working as-is and then I can go the MongoDB route without even changing the upload UI or map view URL. Come to think of it, there must be plenty of existing projects or code to handle uploads and downloads of user-submitted content. Hmm....