Ideas for Automatic Document Generator utility

I waked up this morning with this idea, so I thought I should share it with all of you. It's just a first draft; please go easy on me if you think I'm way over my head. Ok, here we go:

Since it has taken me a lot of time to keep and timeline for the QSC with enough detail so that someone else can be able to follow and to recreate the events, I thought it would be nice to have a utility program that do all that for us. Here is what I have in mind so far:

There shall be a Civ3 Diary Program running in the background of the game. This program will have a text window for us to enter a log for each turn. By a click of a button, it will automatically generate appropriate year index as well as other appropriate format to make the document easier to read. Also, there will be a button to retrieve information from the most recent save game as well as the entire directory of savegames. By saving our games at the end of each turn then click on this automatic document generator, it will retrieve those savegames and create appropriate logs of all our units relative position, current cities production settings, current gold in treasury, current treaties with the AIs, and many other goodies that you can think of. Now, we can choose to do this at the end of every turn or just do it once when we are ready to submit the QSC (GOTM, HOF, or whatever).

By combining this Automatic Document Generator utility with our manual log entries, it would be a piece of cake to recontruct any game from scratch. For those of us that are too lazy to keep a game journal, we don't have to. I think the Automatic Document Generator would provide a good enough information to follow and to recontruct any game from the beginning to the end (just as long as we remember to save our game at the end of each turn). Of course, this utility is not just for QSC; it can be use to document the entire game from 4000BC to the end. And best of all, it will provide the final document in the nice HTML format too, so the users don't even have to worry about learning HTML.

Of course, the big question is "who will take on this challenge to bring this ultility program to life?" Well, I'm sure there are many talented programmers here who like to give something back to the community, myself included; however, I will do it only if someone can provide me with the Window source code for the Mapstat program as well as the format of the savegame. Although I could take the time to research the file struture of the savegame, but I see no reason to re-invent the wheel when someone else has already done that. Anyway, since I already have a full time job, I can't take on this project alone; therefore, I'm hopping to join force with some of you on this adventure. Please let me know what you think.

Moonsinger, if Firaxis would have been serious about getting the bugs out of the game, they would have provided a way to turn on an audit trail of moves (automated play retesting anyone?). Would have solved alot of problems for them and for us.

Originally posted by Cartouche Bee
they would have provided a way to turn on an audit trail of moves (automated play retesting anyone?).

Click to expand...

Yes, that would be very nice! I hope they would consider doing that for the future civ3 patch or maybe for civ4.

Moonsinger@

A while back I posted a request exactly like this in the Creation & Customization/Utility forum. I wanted to know if a turnlog generator existed or if someone could make one. I offered my programming skills if only someone could give me the save game format. Blah, Blah Blah.

Anyway, I only got one response: Cheiftess said there is CivAssist. But never clarified what that was or where to get it.

Crackers work on QSC GOTM I think is generating a utility to grab the info from a save file for data entry purposes into the scoring calculator. So obviously it's possible to get this stuff into excel. I just wish I knew how to do it.

Turn logs triple the amount of time it takes me to play a turn. Sure would be nice to type less and play more.

C'mon someone, help us out, PLEASE!

The source code for MapStat (the older 1.41 Java version) is available.

Originally posted by chiefpaco
After some consideration I have decided to make this project open source. Why not! It is now bound by the GNU public license. Basically, you're free to download it, change it, use it, redistribute it, but you can't make any money off it.

Here is the source. My next post will have Mapstat version 1.41, which includes this information in the package.

Click to expand...

MapStat 1.41 Java code

CivLackey's CivAssistants is supposed to have a log/history utility as a part of it, but to my knowledge he hasn't released a working version of it... just his battle odds calculator.

I would agree that something along those lines would be VERY helpful!

BoB

Thanks to Gramphos and others over at Apolyton, a good portion of the BIC format is known. A lot of the same structure can be found in the save files.

There isn't as much information for the SAV format, but here is a starting point. PtW saves use slightly different formats, as do various patch versions, but for the most part all that changes are some easily figured out offsets.

I would be willing to help on this project as well. The simplest way would be to have a program running in the background watching the autosaves folder, and analyzing each autosave to generate a log. Wouldn't even need to deal with decompression. If anyone has experience with latching onto process id's and recording input that way, a replay program would be possible though.

One thing to keep in mind is that the file format is far and away the easiest part of such an undertaking.

--------------------------------

For PtW (from a post of mine hidden away, just for fun)...

I found the 'time played' portion of the PtW saves. Not too difficult to find. On a hunch I looked at some vanilla saves to try and find any similar data, but couldn't find anything like it. The time played doesn't show up when you load the vanilla civ saves in PtW either (and at least 1.17f saves don't load at all).

Anyways... To find the 'time played':

WRLD..
GAME..
GAME..
00: GAME..
604: 4 bytes (TimePlayed)

TimePlayed is stored in milliseconds. FF FF FF FF would be about 79 hours and 32 minutes, so it might be more than just those 4 bytes, but it seems surrounded by other data. Not sure how this would affect it and don't have any PtW saves with 80 hours or more on them yet.

Click to expand...

Originally posted by Aeson
The source code for MapStat (the older 1.41 Java version) is available.

Click to expand...

How about the source for the Windows version? I can do C++, Delphi, and VB very well, but I don't know much about Java. Oh well, may be it's about time for me to learn Java.

Aeson: Thanks for the link to the BIC and SAV format. I will review them and post my findings later.

I can give you C++ source for looking at the map data as it's part of what I've been working on, though I don't think it applies to what you are talking about. Ainwood would be the one to ask about citizen/city offsets and code. Lovro has C++ save decompression code you can ask to use. And Gramphos would have most of the save format info you are looking for probably, as his CopyTool handles just about everything.

Originally posted by Aeson
I can give you C++ source for looking at the map data as it's part of what I've been working on, though I don't think it applies to what you are talking about. Ainwood would be the one to ask about citizen/city offsets and code. Lovro has C++ save decompression code you can ask to use. And Gramphos would have most of the save format info you are looking for probably, as his CopyTool handles just about everything.

Click to expand...

Thank you! Thank you! I will definitely be back knocking on your door begging for code next, but first, I have other deadlines that need to be taken care of before I can be free to tackle this project (making a living has to come first). In the mean time, if you, Ainwood, or anyone want to take over this project right away, please go ahead. I will catch up with you later.

Moonsinger,

I'm a bit indesposed at the moment - certainly for the next month or two. But I would be happy to provide input where and when I can.

Awww! Writing the game logs is the best part of QSC. Heck, I spent 6 hours yesterday on a single turn (granted that turn was pretty much the pivotal point of the QSC portion of my GotM16), and half of that was the writeup! Over a page of notes!

Seriously, sounds like a good idea, as long as we had easy editing capabilities during the turn. I put in comments for every turn. BTW, the main page was inaccessible, I had to come in a back route. Anyone else experience this?

Originally posted by DANGERBOY
Awww! Writing the game logs is the best part of QSC. Heck, I spent 6 hours yesterday on a single turn (granted that turn was pretty much the pivotal point of the QSC portion of my GotM16), and half of that was the writeup! Over a page of notes!

Click to expand...

Can't wait to read that in a while and find out what on earth was so important.

This automatic logger utility thing is a great idea, IMO. I enjoyed writing up my timeline for QSC15, but of necessity (and courtesy to the readers who would be slogging through it) I left a lot out. In addition to helping players to recall specific details of the game, it would take care of recording the hard, dry facts, therefore freeing timeline-writers up to concentrate on the strategy behind their decisions in their writeups.

Of course, I wouldn't have the faintest idea how to create something like that, so I will bow to the expertise of ye programmingly minded and hope you take it upon yourselves to carry this project through. :worshp:

Originally posted by Aeson
One thing to keep in mind is that the file format is far and away the easiest part of such an undertaking.

Click to expand...

Things required by timeline:

1. When you locate a goody hut report its location, when you open it, and what you get out of the hut.
2. When you meet another civilization, describe the encounter and what you learn about your rival.
3. When you engage in >any trade event, describe the exchange whether it succeeds or fails.
4. If you establish embassy/conduct diplomacy describe the events, the costs, and anything you learn.
5. Describe any adjustments you make to research or luxury spending rates.
6. Describe any technology research decisions.

From the SAV format its possible to see the state of the current game. Positions of huts, units. I'm guessing you can see what techs you have, who has contact with whom, who you have embassies with and what your sliders are set at. We can compare two SAV files to see what changed from turn to turn. But how to determine what caused the change is a different story.

1. Huts - Settler if a new settler appears within 1 move radius of hut that disappeared. Conscript, same deal. Barbs, same deal. Tech and Maps are more complicated. And the dreaded "Deserted".
2. New civ contact should be easy to see. But how was contact made? Intrepid scout, or bought contact. Or did they find you? At this point it would be easy to list number of cities, techs know by them luxuries etc. But the log might be logging things the player didn't actually check into.
3. Trading is a fun one. SAV files will have the end result of ALL trades that turn. So if three civs are now at even tech and one of them owe you 28gpt, how did we get there? Selling maps is another fun thing to figure out. At least luxuries and resources should be easy to follow.
4. Embassies should be easy. But logging what you learn from the city screen has the same problem as initial contact: the log might be logging things the player didn't actually check into.
5. Change in slider values should be easy to grab from two SAV files. Log won't say why the change though.
6. Tech path. Can tell when research changes. It might be difficult to tell why the change. Did you buy a tech you were near to completing? Did you raise slider at the last minute to finish a turn earlier? Or go through all cities and make scientists?

One thing thats not on the required list is the founding of cities. This would be something that could be included fairly easily.

Any insights on how to infer the hows and whys regarding tech trade/research path?

I think the idea is to create a log of events that users can go back an annotate with insights on those decisions.

BoB

The only way around most of the points you've made would be to actually latch onto the Civ III process id and read input as it happens. That's most likely beyond the scope of what we can do.

I think it will just have to generate a rough sketch of what happened for the player to fill in the details. The really tedious tasks, like keeping track of unit movement, tile use, and city production should be where a program like this can shine though.

Originally posted by ControlFreak
Things required by timeline:

1. When you locate a goody hut report its location, when you open it, and what you get out of the hut.
2. When you meet another civilization, describe the encounter and what you learn about your rival.
3. When you engage in >any trade event, describe the exchange whether it succeeds or fails.
4. If you establish embassy/conduct diplomacy describe the events, the costs, and anything you learn.
5. Describe any adjustments you make to research or luxury spending rates.
6. Describe any technology research decisions.

From the SAV format its possible to see the state of the current game. Positions of huts, units. I'm guessing you can see what techs you have, who has contact with whom, who you have embassies with and what your sliders are set at. We can compare two SAV files to see what changed from turn to turn. But how to determine what caused the change is a different story.

Click to expand...

You are absolutely right about that! This utility won't be able to do those things. My intention is not to completely eliminate the manual log entries, but help make them easier to read and to keep track of. Just like Aeson said, keeping track of all unit position and laborer position of each city are the toughest part of the QSC; and I hope that this utility will solve that.

Right!

I guess the app running in the background would have a text entry field for comments and include those comments after each turns status update.

I wonder if it would speed log generation right now if there were Controls like drop down menus and trade, hut, tech tree, buttons. They could automate some of the typing. That might be an easier application. The reading of the autosave would be an added bonus to preselect some of the drop-down values so you just have to hit the "Made contact with..." button and add a comment.

Hmmm....might be able to start on this Toolbox for Word.

{CF alt-tabs away from his reallife job mumbling something about visual basic controls.}

[Edit:]
Also,
Wouldn't we need to filter out some of the unit movements. I'm expecting to Moonsinger to have 50 workers and 150 slaves by 1000BC. That timeline would be quite long if every unit movement was recorded.

Here is a small subset of the RAW DATA the program will collect:

4000BC autosave: 1 settler, 1 worker at tileA
3950BC autosave: 0 settler, 1 worker at tileB, cityA with 1 citizen at tileA, science slider = 100%
3900BC autosave: 0 settler, 1 worker buidling road at tile B, cityA with 1 citizen at tileA, science slider = 100%

------

In the users point of view, this program's display may be something like this (not worrying about the format at this time):

4000BC:

• Settler#1 founded cityA. AI: by comparing the different between 4000BC and 3950BC autosave, the program has come up with this conclusion because the settler at tileA has ceased to exist. Of course, it could be destroyed/disbanded/whatever or it has founded a city. According to the 3950BC autosave, a city just popuped from that location; therefore, the program safely assumes that the settler#1 has founded cityA.
  
• Move worker#1 East. AI: by noticing the displacement in position of worker#1 between the 4000BC and 3950BC autosave, the program come up with the conclusion that worker#1 has moved East.
  
• ....
  
• your note for this turn goes here!

3900BC:

• Worker#1 (East of CityA) start to build road. AI: By noticing the different in the task assignment of worker#1 between the 3950BC and 3900BC autosave, the program safely come to the conclusion that the user just order it to start building road there.
  
• ....
  
• your note for this turn goes here!

As you have seen in the above example, there will be zillion programming codes and logics that the program has to process before it can come up with a simple and friend description such as "Founded CityA" or "Worker#1 moved East". Therefore, don't be so surprise if you see me still working on this program a year from now.

Hey, I know this is an old post and I'm new here, I have been trying to work on this problem, first by trying to decipher the .sav file formats by using a hex comparison program and comparing two game saves with only one change in between each, but its not working out too well. Even when I don't change anything physically but just allow the game clock to run and compare those saves, there are still no fewer than 6 or 7 lengthy binary blocks between them. The only thing I have been able to pull from the comparisons is that at least in my saves the first 17,000 or so binary couplets describe the map. Other than that I'm at a complete loss.

-edit- I forgot to include what kind of experience I have with this kind of stuff. Way back in the day I was responsible for the first actual partial translation of the pokemon gold rom for gameboy, and was leading it until I lost interest and others surpased me.