Web scraping continues to be a profitable and controversial practice:
The practice of Web ‘scraping’ is growing as many firms offer to collect personal, and potentially incriminating, data about users from their social networking profiles and discussions. Many companies even collect online conversations and personal details from social networks, job sites and forums where people might discuss their lives and even potentially sensitive data, such as health issues. These scrapers operate in a legal grey area leaving many users exposed.
— Slashdot
Slashdot notes: “We ban scrapers like this regularly here simply for not adhering to the rules spelled out in robots.txt.”
For more about robots.txt, see [categorySeeAlso slug=”robots-exclusion-standard”].
