• Civ7 is already available! Happy playing :).

Download database updates

The_J

Say No 2 Net Validations
Administrator
Supporter
Joined
Oct 22, 2008
Messages
41,359
Location
DE/NL/FR
Hi everyone,

we recently noticed a weird behaviour in the download database:
If a download database entry was created, and linked to an external file, this would ONLY work if the link was deposted as https, and not if it was http. This is an effect of security measures within the browsers, and is mostly an organizational thing. But it lead to the fact that some of our downloads were not downloadable, and we want to fix this now.

We do have nearly 4600 links to external entries in our database.
A good part (~1400) is lost, since they have been hosted at defunct sites (megaupload, fileinformer, atomicgamer, etc). We hope to be able to address that later.
From the remaing ~3200, nearly 1900 need fixing.
This will be happening over the next days, automatically.
We hope this will not cause any disturbances. In theory this updating should work without any changes to the database, besides a temporal re-assignment of the uploads, which will be reversed after the update.
But if you see any weird behaviour, or anything unexpected in the database, please let us know, so that we can take a look at it :).
Otherwise: Happy modding :).
 
We should have fixed now all besides 60 links, which will probably require some contacting of users and re-hosting of material, all to be decided.
It seems that nothing weird happened on the way, so we're all good :).

In related news, we have decreased the limit for files to 30 mb (database and attachements), because apparently bigger uploads have never worked, due to a server setting which we don't seem to be able to locate. So while this seems like a drastic change (decrease from nearly 150 mb down to 30), there is effectively no change.
 
How do we address those that are lost due to defunct sites?

Does this include old mod files?
 
It's all more-or-less old mod files.
How to address... so I'll at some point (early next year, second weekend probably) go through the files, and organize this by civ version and by user. If there's any user who has many lost files, I'll try to contact them.
Otherwise, I'll most likely post a thread per C&C forum, asking if anyone has these files, so that we can replace them (not much hope there, but hey, who knows).
THen there's the consideration what to do with things which cannot be replaced. I'm considering asking the admins for a "defunct downloads" section, since I'm not sure if deleting all these entries would be the best option, but this is not yet thought through.
 
Often they will have an original thread that might be worth bumping where others may have the files and come forwards...
 
Well as I said, happy to help if you put the list together :)
 
It's all more-or-less old mod files.
How to address... so I'll at some point (early next year, second weekend probably) go through the files, and organize this by civ version and by user. If there's any user who has many lost files, I'll try to contact them.
Otherwise, I'll most likely post a thread per C&C forum, asking if anyone has these files, so that we can replace them (not much hope there, but hey, who knows).
THen there's the consideration what to do with things which cannot be replaced. I'm considering asking the admins for a "defunct downloads" section, since I'm not sure if deleting all these entries would be the best option, but this is not yet thought through.
We've already been through restoring files to the db once - years ago when there was a major db server malfunction. We did a pretty decent job of getting things uploaded again. There are definitely still people around this time who will be able to provide uploads of many if not the majority of missing files. However, many of us are getting to the stage of life where memory doesn't function quite as well as it used to, and sustained intensive tasks take longer to accomplish. And - sad to say - we're not gonna be around CFC forever. We've already lost a few key members within the last couple of years or so - Balthasar for one. I won't mention any names, but some of us are quite aware of - and discuss privately - our time being measured in years rather than decades. So better sooner rather than later that we have some idea what needs to be tracked down.

A pinned (& regularly updated) list of "defunct downloads" needs to be part of the restoration efforts, imho. It is very likely that files may be discovered further down the road. For many reasons. One of the many being that some of us sign in to the forum much less often than we used to, and even then only to check whatever alerts we have received. When we do check the entire forum, pinned threads are naturally the ones most likely to be noticed. Another reason among the many for maintaining a pinned list, is that having the list available for the foreseeable future will allow those of us with means of contacting those creators alive but no longer available through CFC channels to do so, and for those people to have the time needed to restore their own files.

One final note:

I am aware of at least two cases where individuals' extensive uploads to the C3 section of the db were removed to protect CFC from legal action regarding intellectual property. There is also at least one individual who has adamantly stated that none of their work is to be re-uploaded by anyone. Staff is well aware whom I refer to in all three cases.

Making it clear ahead of time to all of us what should not be re-uploaded will be far easier for staff than having to sift through the entire bulk of re-uploaded material.
 
I'm happy to see this thread, and that there are restoration efforts underway.

One additional thing I'll add is that especially for the older games such as Civ3, there are a lot of resources (both scenarios and components such as unit graphics) that have forum threads, but no entry in the Downloads Database, in many cases because the Downloads Database didn't exist yet when they were created. For a lot of these, the original link may have rotted, but at least among scenarios a lot them have been re-uploaded, with links buried near the end of their threads. The community has done a fairly good job of restoring access to link-rotted mods in the threads when someone asks if anyone has a copy, and I'd say there is probably better access to the historical corpus of CFC mods than there was a few years ago. But there are two significant challenges to this from a holistic perspective:

1. It's hard to discover which links are broken. This is something that the Downloads Database makes easy to enumerate in comparison. But it probably would be possible to query all the first posts in the completed scenario/unit graphics/etc. forums to look for broken links, and specifically broken links that aren't image embeds. There would be some false positives (links to research used for the mod, etc.), but I suspect a few external hosts would show up significantly in such a list, and indicate legitimate broken links.

2. If the original creator is not still active on CFC, than the links in the first post are never updated, and if there are subsequent replies to the thread, it becomes increasingly difficult to find the re-uploaded link over time. There have been some feel-good stories where the original creator came back, having lost their old work, and someone else had it and re-uploaded it, and links were restored; or where Thunderfall fixed a link of a prominent mod. But over 90% of the time, the link remains broken even if the file has been re-uploaded. Realistically, to fix this problem would require a couple of active users to have sufficient privileges to fix those links in first posts.

Once again, I'm glad there's discussion along these lines. The whole video game industry has been around long enough to see that it's pretty easy for bits to disappear over time, and we've seen that enough here to be aware that, as I like to analogize, without a plan for preservation, our cultural artifacts aren't going to stand the test of time and still be around at the end of the Civ game in 2050 AD. I've seen this even more in the Stories & Tales forum, where external image hosting has proven even less reliable than external file hosting, and it's less likely that someone has a copy on their hard drive. So kudos for taking steps to improve the situation now, while there's still a lot that's available, and while a lot of what's not available may plausibly be recovered. I'm happy to help in any way I can.
 
Thanks for bringing this to our attention (as well to BlueMonkey).
All very valid.
Checking broken first-post links is definitely doable, at least from a quick glance how many Civ3 modpacks/scenarios/maps there are (units and other graphics might be too much). I think I'll attempt this at some point, but I'm afraid I still have quite a huge backlog of work, including this here, the legacy Civ2 database migration (ugh), and compiling GOTM stats, so this might take a while :/.
For now, I'd recommend for any such case you see, that you report the post with the new link, pointing out that the link in the OP is broken and needs to be fixed. That should be the easiest way to take care of a small or important amount of links. If this happens to be too much, we'd need to consider if we need more moderators for this area.
 
I could probably swing the checking for broken first-post links. I've done some work on forum and download database parsing, particularly with links. The former mainly for restoration work on stories and tales, the latter mainly to enable downloading of units directly from the Downloads Database into Civ3 scenarios in my scenario editor. The story restoration work in particular already reports which links in the story are broken and not automatically restorable, and groups them by domain (ImageShack, VillagePhotos, Photobucket, GeoCities, etc.). So in theory, it wouldn't be a tremendous amount of work to point the story restoration code to a sub-forum and have it only look at first posts, and to both tabulate what's missing per-thread and tabulate aggregates. The other part that would make it more useful would be putting the results in some sort of a database.

Then there's the manual part of checking if there's a re-upload, and I like the report what's broken and where it can be fixed part. Being able to have it fixed is an important part of the motivation for the task of figuring out what's broken and checking if it's re-uploaded. I could probably start whack-a-mole style reporting what's already re-uploaded but has a broken link in 10 minutes, but having a proper inventory first seems like a good idea.

Of course I also tend to dip my toes into too many things, including too many Civ-related things. But I'll be sure to report back if I make good progress on the inventory-ing problem, and may start reporting case-by-case sooner than that if it isn't forthcoming.
 
An update on the inventory-ing. The inventory-taker program has progressed well, and basically works now; a few small enhancements continue to be made. It creates a database of missing links, and has been run against the Civ3 Completed Modpacks forum, finding 435 broken links by thread creators on the first page of 564 threads.

The greatest portion of the broken links (roughly 25% of the total) are to civfanatics.com, with the Uploads system followed by Downloads and Attachments being most of those. 3ddownloads and atomicgamers are next, followed by cdgroup.org, all with 30+ broken links.

Not all of these are to the actual mods; some are images; some are links to research, etc. Compared to the Downloads Database, there will be a lot more low-importance broken links. I've captured some stats to help with prioritization, but there will be some manual effort as well.

I'm currently inventory-ing Civ3 Completed Scenarios, which is a larger forum, and somewhat newer (opening in mid 2002). I expect broadly similar patterns, but we'll find out.

I should also note that I've only been testing for links that are well and truly broken; if they are http (not https), but still work, they are counted as "working" in this inventory, and will work for forum denizens using Firefox. But they probably won't for those using Chrome. That will have to be looked at in the future, but is lower priority as there's an easy workaround in using Firefox, and those files aren't truly lost.

I'll be working with The_J to coordinate posting thread search lists in C&C forums, and also will be running the inventory program against other Civ version forums to spot broken links there.

------------

It should also be noted that 381 of 564 threads have no broken links; a bit less than one-third have a broken link (of any variety, including image or by someone other than the thread author). So that gives us another benchmark by which to measure progress in restoration.
 
Last edited:
This is really great stuff as I worked with The_J on cleaning up and reorganizing the Civ2 downloads area and didn't realise you guys were looking at later Civ game section too. Once I'm finished up with all my Civilization 2 preservation work I was also going to turn my attention to Civ3, however you guys can do so much more so much better than I can with your advanced searching & cataloging tools so this is just brilliant! Despite the Civ2 focus some Civ3'ers will recognize me as I've already done a whole bunch of Civ3 rescues over the last 2 years, so there's probably a decent bunch of threads in Quintillus's results where the OP has a dead link, but towards the end of the thread in the the most recent posts you'll find one from me with a new working file link that I uploaded to CFC's FTP. Obviously in these situations I couldn't do much about the OP or associated CFC download page if one was originally created so they'd still be dead (although in a couple of cases like Drift's Master of Myrror the author came back and updated their OP links to my CFC link). I've already said this to The_J but once you guys have setup that thread (listing all the missing stuff you need) if someone can please let me know then I can hopefully help out and be useful as I've built a bit of a Civ3 scenario collection in preparation for my own preservation project and a lot of that stuff came from collectors who have way more stuff that we can chase down if needed. So it's highly likely we can bring back a lot of lost stuff and fix those dead links. :)

This is just so great guys.. well done! Hope we get to work together a bit! :)

.
 
Last edited:
Top Bottom