Understanding Robots.txt and Legal Considerations in Web Scraping

In the US, the hiQ v. LinkedIn decision established that scraping publicly available data does not violate the CFAA. However, legality depends on multiple factors including the type of data collected, how you use it, whether you circumvent technical barriers, and the jurisdiction. Product pricing and specification data is generally lower risk than personal data like reviews with user information.

While robots.txt is technically advisory, ignoring it can have serious consequences. Your IP addresses may be blocked, your scraping infrastructure may be fingerprinted and banned, and in legal disputes, ignoring robots.txt is often cited as evidence of unauthorized access or bad faith. Using a residential proxy network can help maintain access, but courts have looked at robots.txt compliance when evaluating scraping lawsuits.

GDPR applies to the processing of personal data. Pure product data such as prices, descriptions, and stock levels are not personal data and fall outside GDPR scope. However, if your scraping collects reviewer names, seller contact details, or any information that could identify a natural person, GDPR obligations apply, including the need for a lawful processing basis, data minimization, and providing data subject rights.

If a site actively blocks your scraper, the ethical and legal approach is to first check if they offer an API, then consider reaching out to request data access. You may also want to explore the trade-offs between scraping and official APIs. If blocking persists after a good-faith attempt, using a managed data provider like DataWeBot can be a safer alternative, as they maintain compliant access to data sources and handle the technical and legal complexities on your behalf.

A safe starting point is one request every 2-5 seconds for mid-size sites, adjusting based on the site's Crawl-delay directive if present. Large marketplaces can often handle 1-2 requests per second, while small stores may require 5-10 second delays. Always monitor for 429 and 503 responses and reduce your rate if you receive them.

The Robots Exclusion Protocol is a standard that allows website owners to communicate crawling preferences to web bots through a plain-text file called robots.txt placed at the site root. The file contains directives specifying which paths bots are allowed or disallowed from accessing. While the protocol is advisory and not technically enforced, major search engines and reputable crawlers honor these directives as a matter of standard practice.

The hiQ v. LinkedIn case (2022) was a landmark Ninth Circuit decision establishing that scraping publicly available data does not violate the Computer Fraud and Abuse Act because there is no unauthorized access when information is visible to any web browser user. This case significantly reduced the legal risk of scraping public data in the United States, though it applies specifically to the Ninth Circuit jurisdiction and to publicly accessible information only.

Product prices, specifications, stock levels, and other non-personal product data fall outside the scope of GDPR because they are not personal data. However, if your scraping captures customer review content with reviewer names, seller contact details for sole proprietors, or any information that identifies a natural person, GDPR obligations apply. The safest approach is to collect only the product-level data you actually need.

The crawl-delay directive is a non-standard but widely supported robots.txt instruction that tells bots to wait a specified number of seconds between requests. While major search engines like Google ignore this directive, ecommerce sites often rely on it to protect their server infrastructure. Respecting crawl-delay demonstrates good faith and helps maintain long-term access to data sources without triggering blocks or legal action.

Scraping publicly available data that any visitor can see in a browser is generally considered lower legal risk. Circumventing access controls, such as bypassing login walls, CAPTCHAs, IP blocks, or rate limiters, introduces significant legal risk because it may constitute unauthorized access under computer fraud statutes. The distinction between accessing public information and bypassing technical barriers is a critical factor in legal analyses of web scraping cases.

Most websites include terms of service that prohibit automated data collection, but the enforceability of these provisions against scrapers remains a contested legal question. Courts have reached different conclusions depending on whether the user agreed to the terms via a clickwrap agreement, whether the terms were reasonably conspicuous, and the jurisdiction. The safest approach is to document your compliance efforts and consult legal counsel when scraping sites with restrictive terms.

A clickwrap agreement requires the user to actively click an 'I agree' button before accessing a service, creating a stronger contractual relationship. A browsewrap agreement states that merely using the website constitutes acceptance of the terms, without requiring any affirmative action. Courts generally enforce clickwrap agreements more readily, while browsewrap agreements are often found unenforceable against scrapers because bots never had an opportunity to read or agree to the terms.

The EU Database Directive grants sui generis rights to database creators who have made a substantial investment in obtaining, verifying, or presenting the contents of a database. This protection exists independently of copyright and can apply even when individual data points are not copyrightable. For web scrapers, this means extracting a substantial portion of a European database could violate these rights, even if the individual prices or product details are factual and non-copyrightable.

IP rotation involves distributing web scraping requests across multiple IP addresses rather than sending all requests from a single address. Scrapers use this technique to avoid triggering rate limits or IP-based blocks that websites deploy to prevent automated access. While IP rotation is a common technical practice, using it to circumvent explicit access restrictions after receiving a cease-and-desist letter could be viewed as evidence of intentional evasion of access controls in legal proceedings.

The CCPA gives California residents the right to know what personal information businesses collect about them and to request its deletion. If your scraping collects data that identifies California residents, such as names from product reviews or seller profiles, CCPA obligations apply regardless of where your business is located. You must be prepared to honor deletion requests and disclose your data collection practices in your privacy policy.

A cease-and-desist letter is a formal written notice from a website owner demanding that you stop scraping their site, typically citing terms of service violations, trespass to chattels, or computer fraud statutes. Receiving one does not mean you have broken the law, but ignoring it significantly increases legal risk. The recommended response is to immediately pause scraping the affected site, consult with a lawyer to evaluate your legal position, and explore alternative data access methods such as official APIs.

Trespass to chattels is a legal claim alleging that someone intentionally interfered with another party's personal property, causing harm. In web scraping cases, website owners have argued that excessive automated requests consume server resources and degrade site performance for legitimate users. Courts have required plaintiffs to demonstrate actual harm to their servers, making this claim most relevant when scraping is conducted at volumes that measurably impact website performance or availability.