As any long-term member (or many short-term members who have tried to download old mods) can tell you, one of the hazards of the sands of the time on the Internet is that links break, sites go down, and eventually what was accessible is not any more. Having been a member of CFC for more than a decade, I've seen that happen more than a few times:
- The Great Hack of 2008, wiping out about a year's worth of CFC Uploads
- MegaUpload going down, taking mods with it
- AtomicGamer going down, with the same result
- Photobucket putting up a paywall, making many images (particularly in Stories and Tales) unavailable
- Various ImageBucket links breaking over the years
- Various smaller sites (including personal sites hosting mods) going down
It's also true that the average age of our creations is increasing. And while there's a decent chance that graphical mods, such as unit graphics, are hosted here and still available, beyond that it's become increasingly likely over the years that any given mod or story is no longer available. For stories, I would not be at all surprised if more than half are no longer fully available; for mods the situation is probably better, but there are scores that were hosted on AtomicGamer and its affiliates.
But archiving things is hard, and takes time. So while there are a few posters, such as Ozymandias, who have done a good job with it, no one has been able to archive everything, and those who have made an effort have tended to focus on one area. It's also not sufficient just to rely on CFC being backed up (though that is important), since many resources, especially larger ones, are hosted externally.
So, largely, when an external site has gone down, we've lost content. Occasionally the original author is still here, and re-uploads it, but as I can attest to, that takes time. More often than not, they've already moved on, or don't have the time.
--------------------
I don't have a silver bullet for mods, particularly externally-hosted ones. But this week I got around to creating a program that automatically archives Stories and Tales threads, saving them - including external images - locally. I'm sure it will still need work to work with all the stories. But I wanted to post about it, to demonstrate that this sort of thing is possible. I wish I'd had the technical know-how to do this a decade ago -
my approach then was manual, and not scalable even with my greater amounts of free time then. While that involved manually downloading each image and putting them together by hand, my new approach can download an entire story automatically within a few minutes (depending on the size of the story and bandwidth).
My sample is based on Lanzelot's story
The Republicans go to War, which just wrapped up.
You can download an archive of the first page here. The program can download all pages, but the combined size (70 MB) is too large for my e-mail's file hosting. Unzip the archive, then open StoryArchive.html. If you turn off your network after downloading, you'll find that everything still works.
My main goal is to preserve the remaining stories before another popular image-hosting site goes down. But I can also see this being useful for those who will be offline for awhile, such as on an airplane.
The code for this program is
here. Not super user-friendly yet, but I want the focus to be on the archival discussion, not the tools to do so, beyond that they can be created.
The same approach can be applied to other threads, as well.
---------------------
Thinking about the long term, how many of our creations will still be available in 2050? That's farther off than the Internet is old, but based on past experience, the most likely answer is "not many." While I'm not too worried about CFC itself - there are multiple admins, Thunderfall is already much less active than he used to be, and there are enough active users to step up if need be - all bets are off for external sites. The amount of content lost over the past 10 years alone sets a poor precedent.
That's why I think a focus on preserving our past (using an approach that can also preserve our future) should be a focus. Here's what I'm thinking:
- Tools are created to help automatically archive resources where possible
- An external site is created for hosting archived resources. Both tool-archived resources and manually-archived ones (where tool-based archives are not possible, such as external sites having anti-bot technology) are stored there.
- A small team of individuals administers the archive - the inactivity of any one member should not render the resources inaccessible.
- The archive site itself is properly backed up. Hard drives are cheap these days.
- Ideally, a process is created where CFC resources can be updated when links break, using archived versions.
The last point is rather key. A common situation is that a mod link breaks, and the first post is stuck forever linking to a broken link. Even when someone re-uploads it, their post often eventually winds up a page or two from the end, and difficult to find. With stories, the images simply disappear over time. Having some way to restore these links significantly enhances the value of both mods and stories - although this will require CFC involvement. Having mods and stories archived externally, while not as convenient, would at least provide
an option if that is not possible - and some archive is necessary for reliable restoration of links.
---------
Thoughts? Anyone else interested in helping with Civ3 artifact preservation? Please feel free to link in C&C or S&T as well; I've posted this here as a central location.
