---
title: "Is web scraping legal? robots.txt and terms explained"
url: https://www.datawebot.com/blog/robots-txt-legal-considerations-web-scraping
description: "Document the source and access conditions. Google describes robots.txt as a crawl-management mechanism, not a way to secure private information. Access and reuse questions need separate review for the specific source and jurisdiction."
updated: 2026-10-23
author: datawebot team
---

Extraction guide

# Review a source before collection

Document the source and access conditions.

Updated October 23, 2026 · datawebot team

[← All guides](https://www.datawebot.com/blog)Field map ↓Try the workflow ↗

01 / INPUT**Source URL + intended use**

02 / YOUR WORKFLOW**Review access → scope → retain decision**

03 / OUTPUT**A collection review record**

## Keep the right fields.

Workflow field map

| Field / step | Input | Rule |
| --- | --- | --- |
| Crawl rules | Source robots.txt | A crawler-control file is not an access license. |
| Terms and access | Applicable source terms and authorization | Review the actual intended collection. |
| Data use | Fields, retention and publication | Consider the relevant jurisdiction and data type. |

## Read a crawl rule in its own scope.

Illustrative robots.txt file for a hypothetical store. The rule below asks matching crawlers to avoid /account/ and /checkout/. It does not grant a license to reuse the rest of the site.

Example

```
User-agent: *
Disallow: /account/
Disallow: /checkout/
```

[Robots Exclusion Protocol · RFC 9309 ↗](https://www.rfc-editor.org/rfc/rfc9309.html)

## Crawling, indexing and authorization are separate checks.

RFC 9309 defines crawler rules, not access authorization. Google also distinguishes crawl blocking from keeping a URL out of its index.

| Keep | Record | Decision |
| --- | --- | --- |
| Crawl rule | Rules for the relevant host, path and user agent | Check the actual source file and retain when it was reviewed. |
| Search indexing | A URL may be discovered even when crawling is blocked | Site owners should use an appropriate indexing control or authentication for their objective. |
| Access control | Login requirements, account permissions and technical restrictions | A robots.txt file is not a mechanism for securing private content. |
| Reuse decision | Terms, intended purpose, fields and applicable jurisdiction | Public accessibility alone does not settle copyright, contractual or privacy questions. |

[Google's robots.txt guide ↗](https://developers.google.com/search/docs/crawling-indexing/robots/intro)

## Keep a review record with the collection request.

This is an operational checklist, not a legal conclusion about a particular collection. Escalate unresolved legal questions to someone qualified in the relevant jurisdiction.

| Keep | Record | Decision |
| --- | --- | --- |
| Source and scope | Exact hosts, public paths, required fields and intended use | Exclude private account data from a public product-data request. |
| Evidence checked | Dated crawl rules, applicable terms and any owner authorization | Keep enough context to revisit the decision when the source changes. |
| Collection limits | Request bounds, rate expectations and failure handling | Stop for unresolved access restrictions rather than treating a failure as permission. |
| Downstream handling | Storage period, recipients and publication rights | Review images, descriptions and personal data separately when included. |

[Acceptable use ↗](https://www.datawebot.com/acceptable-use) · [Choose the access method ↗](https://www.datawebot.com/blog/web-scraping-vs-official-apis-ecommerce)

Content reviewed 2026-10-23.

## Before you use the output

Google describes robots.txt as a crawl-management mechanism, not a way to secure private information. Access and reuse questions need separate review for the specific source and jurisdiction.

[Google: introduction to robots.txt ↗](https://developers.google.com/search/docs/crawling-indexing/robots/intro)

## Check one record first.

Start with the smallest useful scope. Compare the returned record with its source, confirm variant identity and inspect missing fields before applying the mapping to a larger dataset.

Keep extraction warnings and the retrieval timestamp with downstream output. When a required field is missing, leave it unresolved or return to the source; do not fill it from an assumption.

Working references [Product and variant schema ↗](https://www.datawebot.com/docs/data) · [Supported endpoints ↗](https://www.datawebot.com/scraper) · [Incomplete or unavailable results ↗](https://www.datawebot.com/docs/troubleshooting)

[Continue this workflow](https://www.datawebot.com/acceptable-use)