1. We have added a Gift Upgrades feature that allows you to gift an account upgrade to another member, just in time for the holiday season. You can see the gift option when going to the Account Upgrades screen, or on any user profile screen.
    Dismiss Notice

OT Crawling

Discussion in 'Site Feedback' started by Truthy, Oct 14, 2019.

  1. Truthy

    Truthy Ambulating

    Joined:
    Oct 9, 2010
    Messages:
    1,937
    I didn't have much going on this weekend, so I did a little information retrieval project--I wrote a short scraper that can do a traversal of the Off-Topic section to grab metadata. I wouldn't want it to be a burden on the website, so I added a random delay of several seconds between every request. My plan would be to do a traversal and then make a thread showing my findings (long-term posting/activity trends and community detection).

    I can't find anything in the rules prohibiting/discouraging crawling (unless it involves spam), nor can I find a Robots.txt file that lays out the policy. I'd argue this isn't any more invasive or disruptive than search engine indexing, which you guys have no problems with, I'm sure. Regardless, I worry site staff might be miffed if I just went ahead and did it. Or my IP would get banned or I'd annoy mods by using a proxy. So is this ok? Does the site have a robots/scraping/crawling policy?

    To be clear, I'm not trying to create bots to spam or DoS the site or anything like that. I'm just interested in gathering publicly available metadata purely out of curiosity, without overloading the site with excessive requests. I assume many OTers would find the results interesting.
     
    Last edited: Oct 14, 2019
  2. Petek

    Petek Alpha Centaurian Administrator Supporter

    Joined:
    Nov 8, 2001
    Messages:
    3,647
    Location:
    Berkeley, Calif., USA
    I doubt this proposal would burden the server. However, please PM me examples of the data that you would collect and post. Thanks!
     
    Truthy likes this.
  3. hobbsyoyo

    hobbsyoyo Deity

    Joined:
    Jul 13, 2012
    Messages:
    22,954
    Gender:
    Male
    Location:
    The pale blue dot.
    You should do it!
     
  4. Truthy

    Truthy Ambulating

    Joined:
    Oct 9, 2010
    Messages:
    1,937
    Alas, I was asked not to due to potential privacy or GDPR stuff.

    Is the OT composed of "roving like gangs" and cliques? How have posting trends changed over time? How severe is posting inequality? What percentage of all likes were given by @hobbsyoyo?

    We may never know :p
     
    Narz and hobbsyoyo like this.
  5. hobbsyoyo

    hobbsyoyo Deity

    Joined:
    Jul 13, 2012
    Messages:
    22,954
    Gender:
    Male
    Location:
    The pale blue dot.
    Aww that's too bad.

    Yeah I can't think of a way that you could anonymize that data and still have it be fun/meaningful for posters.
     
  6. Takhisis

    Takhisis daria dance party

    Joined:
    Jul 11, 2005
    Messages:
    45,681
    Location:
    up yours!
    Tell me so I can have my account deleted and prosecute you if I ever find out who you are.
     
  7. Truthy

    Truthy Ambulating

    Joined:
    Oct 9, 2010
    Messages:
    1,937
    This is why we can't have nice graphs
     
    hobbsyoyo likes this.
  8. Takhisis

    Takhisis daria dance party

    Joined:
    Jul 11, 2005
    Messages:
    45,681
    Location:
    up yours!
    Actually, the one who's not being nice is you.
     
  9. Timsup2nothin

    Timsup2nothin Quad B

    Joined:
    Apr 2, 2013
    Messages:
    39,587
    Location:
    Shadowy Fringe of the Candy Industry
    If the graph were made it would turn out to look like an outline of Argentina with an arrow pointing @Takhisis, that's the problem.
     
  10. Takhisis

    Takhisis daria dance party

    Joined:
    Jul 11, 2005
    Messages:
    45,681
    Location:
    up yours!
    I believe that the answer is ‘you don't have the right to make such a graph because my metadata is mine. That is why I block Facebook and similar companies from gathering my data.’

    Ergo, you're the one who's not being nice.
     

Share This Page