Architecture

Core Architecture of a Production Web Scraping Pipeline

Reliable pipelines separate collection, processing, storage, and delivery into layers so
individual components can fail, scale, or change without breaking the entire system.

1. Source Discovery and Job Queue

A production workflow first determines which URLs or resources need collection. URLs may
originate from seed lists, category pages, sitemaps, pagination, an internal product catalog,
previous crawl results, or API requests.

Rather than processing everything immediately, jobs are usually tracked through states such
as:

• pending
• running
• completed
• failed
• retry scheduled

Separating discovery from execution makes it easier to scale worker capacity, retry failed
jobs, and understand exactly which sources were processed.

2. Acquisition and Fetch Layer

The fetch layer retrieves the actual page or endpoint. The architecture should choose the
lightest approach that reliably returns the required information.

Static pages may only require normal HTTP requests. Dynamic pages can require rendered
browser sessions because important information may appear only after JavaScript executes.

For higher-volume systems, the acquisition layer may also manage rate limits, request
pacing, concurrency, geography, sessions, and failed-request retries.

For more detail on large-scale collection architecture, see
large-scale web scraping.

3. Raw Source Capture

One of the most useful architectural decisions is separating raw source acquisition from
downstream extraction.

A raw capture may include:

• HTML
• JSON response
• source URL
• HTTP status
• collection timestamp
• source identifier
• request metadata

Keeping raw responses can make it possible to reprocess historical pages later if parsing
logic changes or a field was initially missed.

4. Parsing and Field Extraction

The extraction layer converts raw HTML or JSON into the fields required by the dataset.
Depending on the source, parsing may use DOM selectors, JSON paths, embedded structured
data, API fields, or other deterministic rules.

For example, a product pipeline might extract:

• source product ID
• title
• brand
• price
• seller
• availability
• rating
• review count
• promotion

5. Validation Layer

Extraction should not be considered successful merely because the HTTP request succeeded.
The resulting records must also satisfy the expected data contract.

Validation may check:

• required fields
• data types
• accepted value ranges
• identifier patterns
• record counts
• duplicate records
• unexpected null rates
• schema changes

6. Transformation and Normalization

Raw values often need to be standardized before different sources can be combined.
Typical transformations include:

• currency normalization
• UTC timestamp conversion
• category mapping
• unit standardization
• text cleaning
• identifier mapping
• product or entity matching

For multi-retailer datasets,
product matching
is often required before pricing or assortment data becomes directly comparable.

7. Storage and Versioning

The storage layer should reflect how the data will be used. A single database is not always
the right answer.

Mature pipelines often separate:

• raw storage — original HTML, JSON, or source responses
• processed storage — cleaned and validated records
• serving storage — datasets optimized for applications or analytics

8. Delivery and Monitoring

The pipeline ends when the data reaches the system that needs it—not when the scraper
finishes.

Delivery may use APIs, webhooks, CSV, JSON, Parquet, Amazon S3, databases, or data warehouses.
Monitoring should track both technical execution and the actual quality of delivered data.

Collection Strategy

Static HTTP Scraping vs Browser-Based Collection

Many production systems use both. URLs can be routed to the least expensive collection
method that reliably exposes the required fields.

Stable Delivery

Using Data Contracts in Web Scraping Pipelines

A data contract defines what downstream consumers can expect from the delivered dataset.
The internal scraping implementation can change without breaking users as long as the
external contract remains stable.

product_id: string        # required
source_id: string         # required
price: decimal            # required
currency: ISO-4217
availability: enum
seller_name: string
collected_at: UTC timestamp
source_url: URL
schema_version: string

Useful contracts define:

• field names
• field types
• nullable fields
• required identifiers
• date/time format
• currency conventions
• schema version
• expected refresh frequency

Storage

Where Should Scraped Data Be Stored?

Recovery

Retry Logic, Checkpoints, and Backfills

Production pipelines should assume that some jobs will fail. Reliability comes from how the
system recovers.

Example Stack

Example Technology Stack for a Web Scraping Pipeline

The exact stack should depend on source complexity, scale, engineering standards, and
downstream requirements. The tools below are examples rather than a required architecture.

Best Practices

Web Scraping Data Pipeline Best Practices