--- title: "Build a product price history" url: https://www.datawebot.com/blog/build-product-price-history-database description: "Keep observations without overwriting yesterday's evidence. Schedule collection in your application. A saved extraction does not create recurring monitoring. Compare the same variant, market and currency before calculating a change." updated: 2026-10-07 author: datawebot team --- Extraction guide # Build a product price history Keep observations without overwriting yesterday's evidence. Updated October 7, 2026 · datawebot team [← All guides](https://www.datawebot.com/blog)Field map ↓Try the workflow ↗ 01 / INPUT**Dated product/variant JSON** 02 / YOUR WORKFLOW**Append by source + variant + time** 03 / OUTPUT**Comparable observation history** ## Keep the right fields. Workflow field map | Field / step | Input | Rule | | --- | --- | --- | | Observation key | storefront_id + variant.id + retrieved_at | Use strings for identifiers. | | Price | amount + currency | Store decimals; preserve the original currency. | | Completeness | run completeness + warnings | Retain partial and missing-value flags. | Synthetic catalog data · not a retrieval result 01 / Example record Fictional product ### Cotton Crewneck T-Shirt `No source URL` Products 1 Variants 4 Record origin Synthetic Page visits Not applicable [Field definitions](https://www.datawebot.com/docs/data) 02 / Inspect the example | Variant | SKU | Example price | Example availability | | --- | --- | --- | --- | | Small / Navy | `DEMO-TEE-S-NV` | USD 32.00 | Example: available | | Medium / Navy | `DEMO-TEE-M-NV` | USD 32.00 | Example: available | | Medium / Sand | `DEMO-TEE-M-SD` | USD 34.00 | Example: unavailable | | Large / Navy | `DEMO-TEE-L-NV` | USD 32.00 | Example: unknown | All names, identifiers, prices and availability values are fictional. They do not describe a merchant’s catalog or inventory. Illustrative schema · synthetic records [Example CSV](https://www.datawebot.com/samples/catalogue-example.csv) · [Example JSON](https://www.datawebot.com/samples/catalogue-example.json) · [Use with an agent](https://www.datawebot.com/docs/agents) ## Store observations, not just the latest price. Keep a source namespace, stable variant identity and retrieval timestamp in every row. Re-importing the same observation must not create a second measurement. | Keep | Record | Decision | | --- | --- | --- | | Primary key | Storefront + product ID + variant ID + retrieval time | A retry of the same saved result becomes a no-op. | | Price representation | Source decimal text + ISO currency | Avoid binary floating-point storage; use decimal arithmetic for calculations. | | Comparison basis | Market + pack + condition + seller where applicable | Add these fields when adapting the recipe to marketplace offers. | | Evidence | Source URL + completeness + warning JSON | Retain the original export alongside the normalized observation table. | | History | Append a newly retrieved observation | Do not overwrite yesterday's record when today's value is unchanged. | ## Run the synthetic price-history exercise. Download the Python recipe and the two fictional JSON records below into one folder. The product, merchant URL, timestamps and prices are invented to demonstrate the import and comparison rules. The recipe uses Python’s standard SQLite and decimal libraries; no API key or live request is needed. Example ``` python price-history.py history.sqlite price-history-example-first.json price-history-example-later.json ``` [Python recipe ↗](https://www.datawebot.com/examples/price-history.py) · [Synthetic first record · JSON ↗](https://www.datawebot.com/samples/price-history-example-first.json) · [Synthetic later record · JSON ↗](https://www.datawebot.com/samples/price-history-example-later.json) · [Example comparison · CSV ↗](https://www.datawebot.com/samples/price-history-example.csv) ## One price changes; three stay the same. In this fictional exercise, Small / Navy changes from 32.00 to 34.00 USD between two invented timestamps. The other three prices stay the same. An unchanged price still produces a new observation, while re-importing either file must not duplicate it. | Variant | First observation | Later observation | | --- | --- | --- | | Small / Navy | 32.00 USD | 34.00 USD · change +2.00 | | Medium / Navy | 32.00 USD | 32.00 USD · unchanged | | Medium / Sand | 34.00 USD | 34.00 USD · unchanged | | Large / Navy | 32.00 USD | 32.00 USD · unchanged | [Download the synthetic comparison ↗](https://www.datawebot.com/samples/price-history-example.csv) ## Extend the recipe deliberately. The illustration uses one fictional merchant and a stable option tuple. Marketplace offers need additional identity and delivery fields before comparisons are meaningful. | Keep | Record | Decision | | --- | --- | --- | | Collection schedule | Your scheduler starts a bounded extraction | Save the run ID before yielding and keep a spending ceiling. | | Result retention | Retrieve all result pages, then persist your own copy | The extraction service is not a permanent historical-price archive. | | Corrections | Keep the original observation and append a correction record | Do not silently edit raw evidence after downstream decisions use it. | | Currency conversion | Separate rate, source and effective date | Keep the original price unchanged; do not mix conversions into raw history. | [API workflow ↗](https://www.datawebot.com/docs/api/quickstart) · [Monitoring rules ↗](https://www.datawebot.com/blog/price-monitoring-guide-for-ecommerce) Content reviewed 2026-10-07. ## Before you use the output Schedule collection in your application. A saved extraction does not create recurring monitoring. Compare the same variant, market and currency before calculating a change. ## Check one record first. Start with the smallest useful scope. Compare the returned record with its source, confirm variant identity and inspect missing fields before applying the mapping to a larger dataset. Keep extraction warnings and the retrieval timestamp with downstream output. When a required field is missing, leave it unresolved or return to the source; do not fill it from an assumption. Working references [Product and variant schema ↗](https://www.datawebot.com/docs/data) · [Supported endpoints ↗](https://www.datawebot.com/scraper) · [Incomplete or unavailable results ↗](https://www.datawebot.com/docs/troubleshooting) [Continue this workflow](https://www.datawebot.com/docs/api/quickstart)