Archiving Our Creations, Or, Making Sure We Still Have Culture in 2050 AD

[Quintillus]

From a technical standpoint, I added an archiveInfo.html page which is generated at the end of the archival process. This lists how many image downloads succeeded out of the overall number of attempts, and also lists out the URLs of the failed ones. This is quite useful for bringing the missing images back to life, if a copy can be found somewhere.

Edit: Also updated the web site, converting the list of uploaded stories into more-readable table format.

I will probably be a bit less active the rest of the week. But I'm pleased that to have reached the point where threads are consistently succeeding at downloading on the first try.

 

[Lanzelot]

Looks like you did some good progress, while I was asleep... (And lot's of stuff for me to incorporate into the UI, too...) I probably won't have much free time before the weekend, but then I'll be on vacation for a couple of days, so can do some real work again... (Only need to set up my wife's laptop with all the tools. But that should now be quicker than the first time... Especially as I've kept all the installers.)

And good detective work with restoring files from the Internet Archive! I remember that some time ago one of the CFC admins restored "Cracker's Opening Play" strategy article from some "Internet time machine". (It had also gotten lost during the forum upgrade -- or even before that?!?) Don't know whether this was the same as the "Internet Archive" that you were using, but I can try and dig up the discussion about it in the admin forum. Perhaps it'll give us an additional source for recovering lost files. (I think that also some of the early GOTM start saves had been recovered from that time machine archive.)

You mentioned it might be a good idea to split this thread into a "development thread" and another thread that reports the progress on the actual archiving. Well, I happen to be a Mod, so I could do it, if you like... For example, I could move one thread into the "Utility Programs" forum and the other one into the "Stories and Tales" forum?
(Even though we should not only archive Stories, but also, as you already noted, legendary HoF threads and perhaps even some of the old GOTM spoiler threads. I learned a lot from these old GOTM discussions, where the legendary pioneers like SirPleb, Moonsinger, Aeson, Bamspeedy, DaveMcW and many more first discovered and developed those tricks and strategies that are today part of every experienced Civ3 player's tools arsenal...)

 

[Quintillus]

It probably is the same archive (https://archive.org/index.php). This one's been around since 1996, and although there are others, is probably the most prominent and comprehensive. And though it doesn't have entire CFC stories, it does have the occasional CFC snapshot or image.

I like the Utility/S&T split idea. Sounds like a good way to do it; many of our current discussions would be in the Utility one, but new uploads would be in S&T.

(And on that note, I'm uploading The Sumerian Empire by Takeo. It is the least-complete one yet, with the last hundred+ images having been lost in the Great Hack)

 

[Lanzelot]

It probably is the same archive (https://archive.org/index.php).

Yes, indeed. I found the admin discussion about restoring Cracker's "Opening Play" article, and they referred to the "Wayback Machine", which is located at archive.org.

Really funny to look at how Civfanatics looked like in 2001...

 

[Quintillus]

I went ahead and created the S&T thread; this can become the Utilities one.

Restored one more story, Chieftain to Monarch: Game Two by CommandoBob. It was another one whose images had disappeared, but are now reunited with the text. In the future, I'll post updates such as that in the S&T thread.

I also added a change to fix formatting that was broken in the vBulletin -> XenForo migration, such as seen in the first post of that story. While there still may be a couple edge cases that aren't covered, it's definitely helping readability.

 

[Lanzelot]

Moderator Action: Ok, I've now moved and copied some of the posts of this thread to the other thread.
It'l still need a bit of house-keeping and stream-lining, though...

 

[Quintillus]

I haven't added any code for a few days, but I have opened an issue tracker here (you will likely need to be signed in to BitBucket to see it). It should help us avoid forgetting enhancement opportunities.

The GUI has been nice, by the way. It's certainly convenient to not have to change the code and restart every time I download a new story, and that speeds it up a bit. One minor enhancement idea is being able to set a base directory, so that the first time you select a save location after starting the program, it defaults to that folder. As an example, right now it defaults to My Documents, but I always save within D:\Civ Stories and Tales Archive. It's minor, but would save a little bit of time and be something else for GUI development practice.

 

[Lanzelot]

Ok, I submitted two small features now, and the endresult looks ok, but the way I got there (and the commits displayed now on bitbucket) looks a bit strange...

I pulled the latest changes, then started coding, and when I was about to commit, I noticed, that you had beaten me by 45 minutes... So I did another pull to get your last changes, then another commit and a push. It aborted and said I need to merge first, so I did that and pushed again. Then it created two commits: one with my changes and following that another one which just repeats your changes?!

No idea, whether this is the way it's supposed to work or whether I messed it up again, but at least we didn't lose anybody's changes...
I'm getting better at this...
More changes to follow later today.

 

[Quintillus]

That looks normal! The commits view (https://bitbucket.org/QuintillusCFC/storyarchiver/commits/all) gives a visual representation, and it's consistent with what you described - you started working with what I'd committed 5 days ago, and since my latest commit was before your most recent changes finished, we wound up with two diverging code paths. The merge brought them back together.

Sounds like some good changes. I knew there were some additional characters we'd need to sanitize, but hadn't looked that up, so I appreciate that addition.

One other thing I've thought of that would be handy is being able to download one specific post (such as post #28). It wouldn't be useful for archiving, but it would for testing new features such as attachments showing up, or a particular post that had issues.

 

[Quintillus]

I added a small update today - attachments (including images) from the CivFanatics.net Uploads system will now be included. An example of these being used is in one of superslug's HOF games, at https://forums.civfanatics.com/threads/superslug-a-pacifist-attempt-at-the-hof.73780/, but I'm sure there are old stories using this format as well. More generally, being able to include attachments (notably BIQs and SAVs) is desirable, and this is the first step in that direction.