Database Queries for Saved Game Data

[Whys]

Currently, the majority are handling saved game data using SaveUtils, or at the very least it's serialization routines. In short, it converts all saved data into a single cryptic string that can then be decrypted upon game reload. However, there currently exists the opportunity to move all saved data to the database, accessible by database query. While I will not argue against the benefits of such an improvement, I am curious just how many of you out there are actually clamoring for this.

If at some point while modding, you found yourself thinking, I really wish I could retrieve this data by query, please raise your hand.

Thank you.

[SamBC]

Not yet, but then I never thought of it. For instance, it would be useful to be able to say get an iterator over all units that have data label 'foo' set to value 'bar', or even any value between 10 and 20 (say). Ordinarily I'd probably handle that through iteration, but making use of the underlying SQL could be phenomenally helpful, making for simpler and more efficient code.

But with the question literally as stated, no.

[Whys]

Thank you for that very honest answer Sam.

It can be done and I can do it, but from a global scope it looks a bit like building the pyramids from the top, down. I mean fancy gadgets are great, but where is the grass roots need? More efficient? More efficient for what? Where's the fire?

Civ5 is sputtering and saved game data queries aren't going to put air under the wings. I think I might have a better modding project in mind, but I want to get some reaction to this first, because I don't disagree that it would be superior functionality. I just don't see the need.

[MouseyPounds]

My response would be similar to Sam's. I see it as something pretty useful, but not something I definitely "need" currently. In part that's because until there's DLL access, I'm focusing on UI and "informational" modding that doesn't need to do much data saving. But another part of it, is it's something that I haven't considered before. I've always taken a practical approach to modding: yes, it would be nice to have X, but if we don't have X, how can I work around it and still do approximately what I want? For anything that I have considered that needed to save information, I've previously only limited my thinking to trying to fit that information into the existing tools.

Take a look at InfoAddict, for example. It only saves an ID string in the actual save file and instead puts the bulk of its historical information into a completely separate ModUserData database, because that's how robk had to do it. While I can't speak for rob, I would think that having all the InfoAddict information within the actual save is a much cleaner implementation, and if he were starting the project today, he'd probably be using the SavedGameDatabase to store that stuff.

I'm also concerned that eventually DB access may be the only option. ScriptData was already completely removed once, and there have been some reports that it no longer works on some structures where it used to (Units, I think.) Despite any previous statements to the contrary, I don't really believe that it's something to rely on long-term, so anything that I design from this point on will be using the SavedGameDatabase to store its information. I'd obviously prefer if SaveUtils was handling the details to make it simpler and help avoid mod conflicts. But if, completely understandably, you don't see it as worth your time to add DB support to SaveUtils, I'll just roll my own methods.

[SamBC]

The most useful thing SaveUtils could do for the SavedGameDatabase is to retrofit the old association that ScriptData has with game objects (units, with (PlayerID,UnitID), players, plots, maybe cities directly as an enhancement). Of course, then to work really well you'd want an event to tie into to make it so the data got copied across when a unit was upgraded (as it seems that UnitIDs aren't preserved in the process)... after that, it wouldn't be much harder to add query functionality on specific named values (which would have columns of their own, I suppose) than it would be to add it in the first place, and there's something useful to be getting on with.

Have to agree with MouseyPounds, though, that I'll manage whatever I see a need for if I need to, though it might mean I do less. A clean, well-tested API will make it easier for more people to do things that are cross-compatible and maintainable, but it's not necessary per se

[Whys]

Originally Posted by MouseyPounds

I'm also concerned that eventually DB access may be the only option. ScriptData was already completely removed once, and there have been some reports that it no longer works on some structures where it used to (Units, I think.)

I guess my point is this. I wrote SaveUtils so that the developers could instead spend their time addressing other more debilitating issues that modders need addressed. Instead, the devs have chosen to focus on this unessential functionality, to SaveUtils detriment. By removing ScriptData, they are intentionally strong arming the modding community into implementing their unessential solution, rather than working with us so as to address things in an 'as needed' order. Instead, the community's concerns have gone entirely ignored for 8 months and the most significant thing they did was to kill ScriptData too early, without warning, and without an alternative. They reimplemented it after enormous complaint and (sort of) admitted their error. If they are now repeating that mistake, even in an incremental fashion, then it is, for me, quite literally, the final insult.

The only ones I see benefiting from such code at this time would be the DLC developers. The modding community has never asked for it, and at the moment, Cheeseburger leads.

I am increasingly convinced that not only is there zero commitement to the modding community by the devs, but that they are cynically using the commited few to make life easier for people getting paid!

Pure and unadulterated bull.

[Valkrionn]

I really wish Sseckman would actually reply to this; All I can say is he truly seems to want to do everything he can to make modding work.

[robk]

Originally Posted by MouseyPounds

... I've always taken a practical approach to modding: yes, it would be nice to have X, but if we don't have X, how can I work around it and still do approximately what I want? ...

Take a look at InfoAddict, for example. It only saves an ID string in the actual save file and instead puts the bulk of its historical information into a completely separate ModUserData database, because that's how robk had to do it. While I can't speak for rob, I would think that having all the InfoAddict information within the actual save is a much cleaner implementation, and if he were starting the project today, he'd probably be using the SavedGameDatabase to store that stuff.

Yep, I would much rather keep the data that I collect in the save file. In fact, looking at the new in game database is next on my list of things to explore. I tried using SaveUtils to save the infoaddict data table but it was way too much for saveutils to handle - I think I hit on a string length limit after the data was serialized or something like that. I think InfoAddict data is a corner case though, because most people aren't saving (relatively) large data structures like I am.

My 2 cents: Some modders are programmers by trade (or pretend to be, like me) and can deal with and take advantage of a robust datastore that we apparently have available to us now. I'm actually quite excited that I'm going to be able to create my own tables and structured queries, not that I have anything in mind now to take serious advantage of it other than shifting my main data storage to the save file. On the other hand, a lot of modders have a more casual acquaintance with programming (SQL specifically) or don't have the need for something that robust. SaveUtils could (and does) fill that gap, presenting an easy interface to save any arbitrary data structure to the save file in a non-destructive way, without having to understand how that works or dealing with the set up part of creating your own table, checking to see if already exists, etc.

Regarding the ScriptData functions, I think they were kinda weird to begin with. They're associated with objects that may or may not exist from game to game and were incredibly flat (just a string, no structure). Before SaveUtils, I'd say they were next to useless. SaveUtils made them useful but it still wasn't the best implementation because the underlying data structure is still just a string that everyone has to share. Re-implementing SaveUtils using the in game SQL database might be nice because, instead of forcing people to associate their data with an object, you can just present them an interface to save their data, period. Each mod gets their own table or, maybe, you just have a table that has 3 columns: modname, key, value (serialized data). You prevent collision for people who don't want or need to dig, which I think was the initial motivator for SaveUtils. Also, it'll continue to be a tool for people who want to save and extract LUA data structures.

Basically, advanced modders will use the in game database as they please, while those who just need some data saved for later access could use SaveUtils with the in game database as its backing store instead of ScriptData. I still think SaveUtils has its place.

[Whys]

Originally Posted by Valkrionn

I really wish Sseckman would actually reply to this; All I can say is he truly seems to want to do everything he can to make modding work.

I notice you voted for 'Yes, I want it!', placing it in a tie with Cheeseburger. Given that I have not seen your name attached to any mod, mod component, or tutorial, I really have to ask...

What is the nature of your relationship with the development team?

You've been around from the beginning, claim to have an inside ear, and act as advocate for the devs while attempting to shepherd the modding community. Now you take part in a poll directed specifically at the modders when you yourself clearly are not. I really think you need to recuse yourself from this vote.

Cheeseburger still leads, and that says more than Sseckman ever could.

I write code for the community, not for the devs. No one in this community asked for this solution, but the devs are forcing it upon us anyway. If that weren't enough, they are doing it at the detriment of one of the few truly useful things to come out of this community. If Sseckman truly believes these changes are so necessary, then he can write the code himself.

The direction of this game's development does not match your statement, so I ask again...

What is the nature of your relationship with the development team?

[Valkrionn]

Originally Posted by Whys

I notice you voted for 'Yes, I want it!', placing it in a tie with Cheeseburger. Given that I have not seen your name attached to any mod, mod component, or tutorial, I really have to ask...

What is the nature of your relationship with the development team?

You've been around from the beginning, claim to have an inside ear, and act as advocate for the devs while attempting to shepherd the modding community. Now you take part in a poll directed specifically at the modders when you yourself clearly are not. I really think you need to recuse yourself from this vote.

Cheeseburger still leads, and that says more than Sseckman ever could.

I write code for the community, not for the devs. No one in this community asked for this solution, but the devs are forcing it upon us anyway. If that weren't enough, they are doing it at the detriment of one of the few truly useful things to come out of this community. If Sseckman truly believes these changes are so necessary, then he can write the code himself.

The direction of this game's development does not match your statement, so I ask again...

What is the nature of your relationship with the development team?

I'm going to try not to be insulted by this...

First off, my name was attached to one of the early balance mods, incidentally one of the most successful, until I decided I simply did not have time to continue working on it.

Second, I am part of the Eden development team, which has extensive plans (with little accompanying code, as yet, waiting on the DLL), though you cannot see this as those forums are hidden.

Third, I have created multiple modcomponents, I simply have not released them publicly. They were for testing purposes. Hell, I could dig up screenshots I've posted on this forum; Dark Ages, happiness from excess luxuries, etc. Was likely one of the first modders to use SQL to add totally new xml, and create functionality for it in Lua. Both mods, btw, utilized your SaveUtils. And these would just be the mods I've discussed on these forums.

Simply because I do not have time to work on Civ5, instead devoting time to an existing civ4 mod which I want to bring to a finished state, does not mean I am some toady for the developers. I fully understand your frustration, but here's a few things for you to keep in mind.

• There are NOT dozens of developers; They MUST use their resources where they think best.
• There are many issues with the game which must be addressed before dealing with modding.
  • If you disagree that the base game must come first, please look at the number of people who actually use mods, even when you can download them in game; There are steam stats.
• Sseckman was unaware of SaveUtils until AFTER beginning on his database. He did not anticipate modders would hack a string object in order to save multiple values.
  • Without SaveUtils, that database would be a huge leap forward in modding capabilities. If you accept that Sseckman did not know of SaveUtils prior to working on it (EXTREMELY likely, given their statements that they did not think people were using ScriptData), then this is an example of "the best of intentions".
  • Even had he known of SaveUtils, the database would likely STILL have been necessary, for internal use with scenarios. Note that it was used by them as soon as it became available.

Becoming so disgruntled simply because the devs were unaware of your work, and created something that would allow much the same functionality, without (gasp) addressing you first, is absolutely ludicrous.

You can turn around and say that SaveUtils allows more advanced functionality; This is true. But how many modders use that functionality, and how many would be totally satisfied with the new database, had it been provided from the beginning?


Feel free to flame me all you wish. It doesn't negate what I've said.

BTW: Given that you created the poll with a clear agenda in mind, should you yourself not also abstain from it? Food for thought.

[Valkrionn]

Also, just to be clear: I don't necessarily want the ability to query my saved data.

I want a clean, easily used, data storage implementation that does not require a third party mod to be distributed with my own, nor should it cause compatibility issues between mods simply on the basis of which mods use SaveUtils and which do not. The database fits this requirement. The ability to query stored data is just icing.

[Whys]

I'm not the only one here with an agenda, Valkrionn.

Becoming so disgruntled simply because the devs were unaware of your work, and created something that would allow much the same functionality, without (gasp) addressing you first, is absolutely ludicrous.

First of all, they were aware of my work. You yourself assured me of that, or have you now forgotten?

Second, you think I want recognition from the devs? You think I write obscure components in the lua forum because I'm in it for the glory?? Allow me to reiterate, I wrote the SaveUtils functionality so the devs could focus on other more detrimental issues that matter to the community, not waste time duplicating functionality they knew already existed. To then yank scriptdata, as if it's hurting something, is only to add insult to injury.

Third, I'm not flaming you. I'm making it clear that I believe you have a conflict of interests in this matter, and that I no longer trust your reassurances regarding the developers "commitment" to the modding community.

I've already stated, I won't argue the merits of database functionality. What I strongly disagree with is that these needles actions, both in code creation and code deletion, somehow represent a step forward for modders, while all of our most simple and basic concerns have gone entirely ignored for 8 months.

It's not about me Valkrionn. I won't allow you to paint it that way.

[Valkrionn]

Originally Posted by Whys

I'm not the only one here with an agenda, Valkrionn.



First of all, they were aware of my work. You yourself assured me of that, or have you now forgotten?

Second, you think I want recognition from the devs? You think I write obscure components in the lua forum because I'm in it for the glory?? Allow me to reiterate, I wrote the SaveUtils functionality so the devs could focus on other more detrimental issues that matter to the community, not waste time duplicating functionality they knew already existed. To then yank scriptdata, as if it's hurting something, is only to add insult to injury.

Third, I'm not flaming you. I'm making it clear that I believe you have a conflict of interests in this matter, and that I no longer trust your reassurances regarding the developers "commitment" to the modding community.

I've already stated, I won't argue the merits of database functionality. What I strongly disagree with is that these needles actions, both in code creation and code deletion, somehow represent a step forward for modders, while all of our most simple and basic concerns have gone entirely ignored for 8 months.

It's not about me Valkrionn. I won't allow you to paint it that way.

Of course they became aware of your work... After myself and others pointed it out to them. Again, after the database was planned/started on (not really sure which, honestly).

...How does your statement not imply you desire recognition from the devs? Deciding that your work is "good enough" and moving on to other topics would be recognition, is it not? Or are we in some bizarro land where acknowledgement that one's work is enough for the devs to focus on other areas does not comprise recognition of one's work?

Frankly, I don't much care. I believe you are wrong; My own experience is enough to assure me of this. You do not share this experience, and come to a different conclusion; That's all well and good. I simply take offense at some of your other statements; Had you left it at "I believe you have a conflict of interest", I would not have been offended.

And again, I believe it does. It represents the ability to store any data you may find necessary, within the save game, without using a third party mod that you must be aware of before using, and must distribute with your own mod.

SaveUtils was extremely useful, and very well made. But it was nothing more than a stopgap, nor COULD it be more, IMO, so long as it was maintained separately from the game itself.

[Afforess]

The Querying data features seems to be an excellent example of Firaxis shooting themselves in the foot. Why?

Well, why do we need to use Firaxis's queries in the first place? Because Firaxis disabled I/O operations with Lua. Why were I/O operations disabled? For "safety" (I use the word "safety" in the loosest sense possible) reasons.

Databases are not any safer. I can still crash your game with SQL. Let's see... "DROP ALL TABLES", or "DROP DATABASE". Done.

So in essence, Firaxis has wasted time providing a feature for modders that is ultimately not any safer, and only marginally more advantageous, in place of I/O access (where I could do whatever I wanted) and giving us more patches.

[Valkrionn]

You can crash the game. You cannot crash the computer. Not as easily, at the very least. There is a huge difference.

[Afforess]

Originally Posted by Valkrionn

You can crash the game. You cannot crash the computer. Not as easily, at the very least. There is a huge difference.

Not as easily? Oh boy, that sent fear into hackers all over the world. Not.

This is why I didn't want any Modding Tools, just give us the code, and let us do what we please.

[Gedemon]

Originally Posted by Afforess

Databases are not any safer. I can still crash your game with SQL. Let's see... "DROP ALL TABLES", or "DROP DATABASE". Done.

I'm sure there are a lot of way to crash the game with the available SDK, but the computer himself ?

[Afforess]

Originally Posted by Gedemon

I'm sure there are a lot of way to crash the game with the available SDK, but the computer himself ?

Sure. Start saving tons of garbage in the database until the harddrive fills. Windows is not good with dealing with 0 hard drive space.

[Zyxpsilon]

What for?
No need for some Game Data, even at runtime.
And we already have a save_reload to handle when *our* objects require access for whatever reasons.
List the Events below, now.

Hold it - don't bother; i want a hook.

[Pazyryk]

I voted for the first option.

I did read the OP before voting, but not the whole thread. So now I feel some heat (not at me...not yet anyway...but potentially for my vote) and I feel I should explain.

I am a modest modder (my FFH modmodmod was downloaded 151 times). I write many long scripts (Lua, Python, and PERL for my job) but barely know a thread from a scope. I am not a professional programmer but I'm a scientist who has to program a lot to get what I need (and only that... my programs are nasty and inelegant). I have some minor Civ5 mods but only for my own testing. I have 25 page design document for my Civ5 mod, so my ambitions are quite large.

I have repeatedly asked for this since before the game was released:

Some specific Lua modding questions. How much access does it have to game internals? (See question #5.)
Ability to create new gamesave information (already there?/modable by Lua?/dll mod?)
Knowing what db elements are in gamesave file versus reloaded from xml/sql
Can Lua code (running during game) modify database values?

This is an important thing that I have been "clamoring for."

Yes, I understand that SaveUtils does everything I asked for above. I have used SaveUtils and deeply appreciate Whys' efforts. I also learned a lot from Whys' posts and am grateful for his answers to my questions. I am sorry that the game designers appear to be running roughshod over Whys' and other's efforts at Civ5 modding. It seems likely to me that this is not intentional (but I could be wrong).

I did not vote the way I did or post this to inflame anyone. I wanted to explain my position in voting for the first option. I like the new SavedGameDatabase system. I liked SaveUtils too but thought it was always intended to be a stopgap until the developers implemented some kind of database access that was stable in the gamesave file.

I continue to wait patiently for Civ5 modding to be a reasonable enterprise for someone with my level of dedication and programming ability. I am optimistic that this will be the case in 6 months or so. Of course, I thought the same thing 6 months ago, but there you go...