Download database updates

Discussion in 'Site Feedback' started by The_J, Dec 18, 2021.

  1. The_J

    The_J Say No 2 Net Validations Super Moderator Supporter

    Joined:
    Oct 22, 2008
    Messages:
    35,024
    Location:
    DE/NL/FR
    Hi everyone,

    we recently noticed a weird behaviour in the download database:
    If a download database entry was created, and linked to an external file, this would ONLY work if the link was deposted as https, and not if it was http. This is an effect of security measures within the browsers, and is mostly an organizational thing. But it lead to the fact that some of our downloads were not downloadable, and we want to fix this now.

    We do have nearly 4600 links to external entries in our database.
    A good part (~1400) is lost, since they have been hosted at defunct sites (megaupload, fileinformer, atomicgamer, etc). We hope to be able to address that later.
    From the remaing ~3200, nearly 1900 need fixing.
    This will be happening over the next days, automatically.
    We hope this will not cause any disturbances. In theory this updating should work without any changes to the database, besides a temporal re-assignment of the uploads, which will be reversed after the update.
    But if you see any weird behaviour, or anything unexpected in the database, please let us know, so that we can take a look at it :).
    Otherwise: Happy modding :).
     
  2. The_J

    The_J Say No 2 Net Validations Super Moderator Supporter

    Joined:
    Oct 22, 2008
    Messages:
    35,024
    Location:
    DE/NL/FR
    We should have fixed now all besides 60 links, which will probably require some contacting of users and re-hosting of material, all to be decided.
    It seems that nothing weird happened on the way, so we're all good :).

    In related news, we have decreased the limit for files to 30 mb (database and attachements), because apparently bigger uploads have never worked, due to a server setting which we don't seem to be able to locate. So while this seems like a drastic change (decrease from nearly 150 mb down to 30), there is effectively no change.
     
    Knightfall likes this.
  3. Aiken_Drumn

    Aiken_Drumn King

    Joined:
    Oct 16, 2021
    Messages:
    720
    Location:
    NES/FG/SF Activity:Arguing the toss
    How do we address those that are lost due to defunct sites?

    Does this include old mod files?
     
    Knightfall likes this.
  4. The_J

    The_J Say No 2 Net Validations Super Moderator Supporter

    Joined:
    Oct 22, 2008
    Messages:
    35,024
    Location:
    DE/NL/FR
    It's all more-or-less old mod files.
    How to address... so I'll at some point (early next year, second weekend probably) go through the files, and organize this by civ version and by user. If there's any user who has many lost files, I'll try to contact them.
    Otherwise, I'll most likely post a thread per C&C forum, asking if anyone has these files, so that we can replace them (not much hope there, but hey, who knows).
    THen there's the consideration what to do with things which cannot be replaced. I'm considering asking the admins for a "defunct downloads" section, since I'm not sure if deleting all these entries would be the best option, but this is not yet thought through.
     
    Quintillus and Knightfall like this.
  5. Aiken_Drumn

    Aiken_Drumn King

    Joined:
    Oct 16, 2021
    Messages:
    720
    Location:
    NES/FG/SF Activity:Arguing the toss
    Happy to help chase Civ3 mods/files :)
     
    Knightfall and The_J like this.
  6. Aiken_Drumn

    Aiken_Drumn King

    Joined:
    Oct 16, 2021
    Messages:
    720
    Location:
    NES/FG/SF Activity:Arguing the toss
    Often they will have an original thread that might be worth bumping where others may have the files and come forwards...
     
    Knightfall likes this.
  7. The_J

    The_J Say No 2 Net Validations Super Moderator Supporter

    Joined:
    Oct 22, 2008
    Messages:
    35,024
    Location:
    DE/NL/FR
    That is probably true, but we're talking about 1400 files, which... yeah... I can't do that on my own, I'm afraid.
     
    Knightfall likes this.
  8. Aiken_Drumn

    Aiken_Drumn King

    Joined:
    Oct 16, 2021
    Messages:
    720
    Location:
    NES/FG/SF Activity:Arguing the toss
    Well as I said, happy to help if you put the list together :)
     
    Knightfall and The_J like this.
  9. Blue Monkey

    Blue Monkey Archon Without Portfolio

    Joined:
    Jul 31, 2005
    Messages:
    11,303
    Location:
    Timeless Isle
    We've already been through restoring files to the db once - years ago when there was a major db server malfunction. We did a pretty decent job of getting things uploaded again. There are definitely still people around this time who will be able to provide uploads of many if not the majority of missing files. However, many of us are getting to the stage of life where memory doesn't function quite as well as it used to, and sustained intensive tasks take longer to accomplish. And - sad to say - we're not gonna be around CFC forever. We've already lost a few key members within the last couple of years or so - Balthasar for one. I won't mention any names, but some of us are quite aware of - and discuss privately - our time being measured in years rather than decades. So better sooner rather than later that we have some idea what needs to be tracked down.

    A pinned (& regularly updated) list of "defunct downloads" needs to be part of the restoration efforts, imho. It is very likely that files may be discovered further down the road. For many reasons. One of the many being that some of us sign in to the forum much less often than we used to, and even then only to check whatever alerts we have received. When we do check the entire forum, pinned threads are naturally the ones most likely to be noticed. Another reason among the many for maintaining a pinned list, is that having the list available for the foreseeable future will allow those of us with means of contacting those creators alive but no longer available through CFC channels to do so, and for those people to have the time needed to restore their own files.

    One final note:

    I am aware of at least two cases where individuals' extensive uploads to the C3 section of the db were removed to protect CFC from legal action regarding intellectual property. There is also at least one individual who has adamantly stated that none of their work is to be re-uploaded by anyone. Staff is well aware whom I refer to in all three cases.

    Making it clear ahead of time to all of us what should not be re-uploaded will be far easier for staff than having to sift through the entire bulk of re-uploaded material.
     
    Quintillus, tjs282 and The_J like this.
  10. Quintillus

    Quintillus Archiving Civ3 Content Supporter

    Joined:
    Mar 17, 2007
    Messages:
    6,927
    Location:
    Ohio
    I'm happy to see this thread, and that there are restoration efforts underway.

    One additional thing I'll add is that especially for the older games such as Civ3, there are a lot of resources (both scenarios and components such as unit graphics) that have forum threads, but no entry in the Downloads Database, in many cases because the Downloads Database didn't exist yet when they were created. For a lot of these, the original link may have rotted, but at least among scenarios a lot them have been re-uploaded, with links buried near the end of their threads. The community has done a fairly good job of restoring access to link-rotted mods in the threads when someone asks if anyone has a copy, and I'd say there is probably better access to the historical corpus of CFC mods than there was a few years ago. But there are two significant challenges to this from a holistic perspective:

    1. It's hard to discover which links are broken. This is something that the Downloads Database makes easy to enumerate in comparison. But it probably would be possible to query all the first posts in the completed scenario/unit graphics/etc. forums to look for broken links, and specifically broken links that aren't image embeds. There would be some false positives (links to research used for the mod, etc.), but I suspect a few external hosts would show up significantly in such a list, and indicate legitimate broken links.

    2. If the original creator is not still active on CFC, than the links in the first post are never updated, and if there are subsequent replies to the thread, it becomes increasingly difficult to find the re-uploaded link over time. There have been some feel-good stories where the original creator came back, having lost their old work, and someone else had it and re-uploaded it, and links were restored; or where Thunderfall fixed a link of a prominent mod. But over 90% of the time, the link remains broken even if the file has been re-uploaded. Realistically, to fix this problem would require a couple of active users to have sufficient privileges to fix those links in first posts.

    Once again, I'm glad there's discussion along these lines. The whole video game industry has been around long enough to see that it's pretty easy for bits to disappear over time, and we've seen that enough here to be aware that, as I like to analogize, without a plan for preservation, our cultural artifacts aren't going to stand the test of time and still be around at the end of the Civ game in 2050 AD. I've seen this even more in the Stories & Tales forum, where external image hosting has proven even less reliable than external file hosting, and it's less likely that someone has a copy on their hard drive. So kudos for taking steps to improve the situation now, while there's still a lot that's available, and while a lot of what's not available may plausibly be recovered. I'm happy to help in any way I can.
     
    The_J likes this.
  11. The_J

    The_J Say No 2 Net Validations Super Moderator Supporter

    Joined:
    Oct 22, 2008
    Messages:
    35,024
    Location:
    DE/NL/FR
    Thanks for bringing this to our attention (as well to BlueMonkey).
    All very valid.
    Checking broken first-post links is definitely doable, at least from a quick glance how many Civ3 modpacks/scenarios/maps there are (units and other graphics might be too much). I think I'll attempt this at some point, but I'm afraid I still have quite a huge backlog of work, including this here, the legacy Civ2 database migration (ugh), and compiling GOTM stats, so this might take a while :/.
    For now, I'd recommend for any such case you see, that you report the post with the new link, pointing out that the link in the OP is broken and needs to be fixed. That should be the easiest way to take care of a small or important amount of links. If this happens to be too much, we'd need to consider if we need more moderators for this area.
     
    Quintillus likes this.
  12. Quintillus

    Quintillus Archiving Civ3 Content Supporter

    Joined:
    Mar 17, 2007
    Messages:
    6,927
    Location:
    Ohio
    I could probably swing the checking for broken first-post links. I've done some work on forum and download database parsing, particularly with links. The former mainly for restoration work on stories and tales, the latter mainly to enable downloading of units directly from the Downloads Database into Civ3 scenarios in my scenario editor. The story restoration work in particular already reports which links in the story are broken and not automatically restorable, and groups them by domain (ImageShack, VillagePhotos, Photobucket, GeoCities, etc.). So in theory, it wouldn't be a tremendous amount of work to point the story restoration code to a sub-forum and have it only look at first posts, and to both tabulate what's missing per-thread and tabulate aggregates. The other part that would make it more useful would be putting the results in some sort of a database.

    Then there's the manual part of checking if there's a re-upload, and I like the report what's broken and where it can be fixed part. Being able to have it fixed is an important part of the motivation for the task of figuring out what's broken and checking if it's re-uploaded. I could probably start whack-a-mole style reporting what's already re-uploaded but has a broken link in 10 minutes, but having a proper inventory first seems like a good idea.

    Of course I also tend to dip my toes into too many things, including too many Civ-related things. But I'll be sure to report back if I make good progress on the inventory-ing problem, and may start reporting case-by-case sooner than that if it isn't forthcoming.
     
  13. The_J

    The_J Say No 2 Net Validations Super Moderator Supporter

    Joined:
    Oct 22, 2008
    Messages:
    35,024
    Location:
    DE/NL/FR
  14. Quintillus

    Quintillus Archiving Civ3 Content Supporter

    Joined:
    Mar 17, 2007
    Messages:
    6,927
    Location:
    Ohio
    An update on the inventory-ing. The inventory-taker program has progressed well, and basically works now; a few small enhancements continue to be made. It creates a database of missing links, and has been run against the Civ3 Completed Modpacks forum, finding 435 broken links by thread creators on the first page of 564 threads.

    The greatest portion of the broken links (roughly 25% of the total) are to civfanatics.com, with the Uploads system followed by Downloads and Attachments being most of those. 3ddownloads and atomicgamers are next, followed by cdgroup.org, all with 30+ broken links.

    Not all of these are to the actual mods; some are images; some are links to research, etc. Compared to the Downloads Database, there will be a lot more low-importance broken links. I've captured some stats to help with prioritization, but there will be some manual effort as well.

    I'm currently inventory-ing Civ3 Completed Scenarios, which is a larger forum, and somewhat newer (opening in mid 2002). I expect broadly similar patterns, but we'll find out.

    I should also note that I've only been testing for links that are well and truly broken; if they are http (not https), but still work, they are counted as "working" in this inventory, and will work for forum denizens using Firefox. But they probably won't for those using Chrome. That will have to be looked at in the future, but is lower priority as there's an easy workaround in using Firefox, and those files aren't truly lost.

    I'll be working with The_J to coordinate posting thread search lists in C&C forums, and also will be running the inventory program against other Civ version forums to spot broken links there.

    ------------

    It should also be noted that 381 of 564 threads have no broken links; a bit less than one-third have a broken link (of any variety, including image or by someone other than the thread author). So that gives us another benchmark by which to measure progress in restoration.
     
    Last edited: Jan 20, 2022 at 4:35 PM
  15. The_J

    The_J Say No 2 Net Validations Super Moderator Supporter

    Joined:
    Oct 22, 2008
    Messages:
    35,024
    Location:
    DE/NL/FR
    Thank you very much Quintillus, this is a big help :hatsoff:.
    I guess it'll be tough to get everything fixed, or even a majority, but we can only find out if we start with it.
    I hope it'll all go well :thumbsup:.
     

Share This Page