Sage

An end-to-end ML pipeline that scrapes product data, trains a Random Forest model to predict prices, serves predictions via a FastAPI REST API, and orchestrates everything with Prefect + MLflow.

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                        SCRAPING PIPELINE                                │
│                                                                         │
│  fetcher.fetch_page() ──► parser.parse_book_listing()                   │
│       │                    (title, price, rating, URL)                  │
│       ▼                                                                │
│  fetcher.fetch_page() ──► parser.parse_book_detail()                   │
│  (per product)             (UPC, genre, description)                    │
│       │                                                                │
│       ▼                                                                │
│  storage.save_to_csv() / save_to_sqlite()                               │
│       │                                                                │
│       ▼                                                                │
│  data/products.csv                                                      │
└───────────────────────────┬─────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                    TRAINING PIPELINE (Prefect)                          │
│                                                                         │
│  load_data ──► clean_data ──► split_and_train                          │
│                                    │                                    │
│                         ┌──────────┴──────────┐                        │
│                         ▼                     ▼                        │
│                  log_to_mlflow           save_model                     │
│                  (params, MAE, R2,     (data/model.pkl)                │
│                   model artifact)                                      │
│                                                                         │
│  Scheduled: cron("0 2 * * *") — nightly retraining at 2 AM             │
└───────────────────────────┬─────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                      SERVING PIPELINE (FastAPI)                         │
│                                                                         │
│  GET  /health                  ──► {"status": "ok"}                    │
│  POST /predict                 ──► {"predicted_price": 42.50}          │
│       Body: {rating, genre, description_len, title_len}                │
│                                                                         │
│  Model source: MLflow registry (book-price-predictor/Production)       │
│  Fallback:     data/model.pkl                                           │
└─────────────────────────────────────────────────────────────────────────┘

Demo

What It Does

Stage	Description
Scrape	Crawls product listings (title, price, rating, availability, UPC, genre, description) from books.toscrape.com with rate limiting, retry logic, and CAPTCHA/block detection
Train	Trains a RandomForestRegressor on scraped data using a ColumnTransformer pipeline (StandardScaler for numerics, OneHotEncoder for genre) — produces a price-prediction model
Orchestrate	Wraps training in Prefect tasks/flows with MLflow tracking and nightly cron scheduling
Serve	Exposes the model via FastAPI for real-time price predictions, loading from MLflow registry or local pickle

Project Structure

2-amazon-scraper/
├── scraper.py                   # CLI entry point for scraping
├── train.py                     # Standalone training script
├── train_flow.py                # Prefect + MLflow orchestrated training pipeline
├── app.py                       # FastAPI inference server
├── Dockerfile                   # Container: train at build, serve at runtime
├── requirements.txt             # Python dependencies
├── README.md
│
├── src/
│   ├── __init__.py
│   ├── config.py                # Defaults, user agents, blocked keywords, rate limits
│   ├── fetcher.py               # HTTP requests + retry + block detection
│   ├── parser.py                # BeautifulSoup selectors (listing + detail pages)
│   ├── scraper.py               # Scraper orchestrator class
│   └── storage.py               # CSV / SQLite output with deduplication
│
├── data/
│   ├── products.csv             # Scraped product data
│   └── model.pkl                # Serialized trained model
│
└── mlruns/                      # MLflow tracking database & model artifacts
    ├── mlflow.db
    └── 1/models/                # 4 registered model versions

Components

Scraping (src/)

Module	Responsibility
config.py	Default URL (books.toscrape.com), rate-limit constants (DELAY_MIN=1s, DELAY_MAX=3s), 6 hardcoded User-Agents, 7 blocked keywords for CAPTCHA/block detection
fetcher.py	fetch_page(url) — random delay → random UA → HTTP GET (30s timeout) → block detection (< 200 chars or keyword match) → 3 retries with exponential backoff (1s, 2s, 4s). Raises BlockedException on detection.
parser.py	parse_book_listing() — extracts title, price, rating, availability, product_url from article.product_pod elements. parse_book_detail() — fetches individual product pages for UPC, genre, description.
storage.py	save_to_csv() — appends with headers. save_to_sqlite() — creates products table, upserts by URL. deduplicate() — removes duplicate rows by URL.

Training

Standalone (train.py):

• Reads data/products.csv → cleans (parse price strings, compute text lengths, fill null ratings)
• ColumnTransformer(StandardScaler + OneHotEncoder) → RandomForestRegressor(n_estimators=100)
• 80/20 train/test split → evaluates MAE & R2 → pickles model to data/model.pkl

Orchestrated (train_flow.py):

• Same logic decomposed into 5 Prefect tasks composed into a training_pipeline flow
• Logs params, metrics, and model artifact to MLflow (registered model: book-price-predictor)
• Deployable as nightly cron: cron="0 2 * * *"

Serving (app.py)

FastAPI server on port 8000:

Endpoint	Method	Input	Output
/health	GET	—	{"status": "ok"}
/predict	POST	{rating: int, genre: str, description_len: int, title_len: int}	{"predicted_price": float}

Model loading priority: MLflow registry (models:/book-price-predictor/Production) → local fallback (data/model.pkl).

Pipeline Orchestration

Prefect (Workflow Management)

train_flow.py defines:

@task load_data       → reads CSV
@task clean_data      → cleans and engineers features
@task split_and_train → trains RandomForest, returns model + metrics
@task log_to_mlflow   → logs params, metrics, and model to MLflow
@task save_model      → pickles model locally

Tasks are composed in a training_pipeline flow, which can be deployed as a scheduled job (default: nightly at 2 AM). Prefect handles retries, task state, and execution monitoring.

MLflow (Experiment Tracking & Model Registry)

Every training run logs:

• Params: data_path, n_rows, n_features, model_type, n_estimators, random_state
• Metrics: MAE (Mean Absolute Error), R2 (R-squared)
• Artifact: Full model pipeline registered as book-price-predictor

The model registry tracks 4 versions with staging/production aliases. FastAPI loads from the Production alias, enabling zero-downtime model promotion.

Setup

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# (Optional) Install fake-useragent for dynamic UA rotation
pip install fake-useragent

Usage

Scraping

# Scrape 5 pages (default), store as CSV
python scraper.py

# Scrape 10 pages from a different category
python scraper.py --pages 10 --url "https://books.toscrape.com/catalogue/category/books_2/index.html"

# Store in SQLite
python scraper.py --storage sqlite

# Store in both CSV and SQLite
python scraper.py --storage both

Training

# Standalone training
python train.py

# Prefect-orchestrated training with MLflow tracking
python train_flow.py

Serving

# Start the prediction API
uvicorn app:app --host 0.0.0.0 --port 8000

# Test it
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"rating": 4, "genre": "Fiction", "description_len": 350, "title_len": 25}'

Docker

# Build (runs train.py at build time)
docker build -t amazon-scraper .

# Run the FastAPI server
docker run -p 8000:8000 amazon-scraper

Swapping to Amazon

To target Amazon, change these selectors in src/parser.py:

Field	books.toscrape.com	Amazon (example, subject to change)
Title	article.product_pod h3 a	h2 a.a-link-normal span
Price	.price_color	span.a-price-whole
Rating	p.star-rating class name	i.a-icon-star / span.a-icon-alt
URL	h3 a[href]	h2 a.a-link-normal[href]

Note: Amazon changes its DOM frequently, injects dynamic content, and aggressively blocks scrapers. Using an official API (Amazon Product Advertising API, Keepa, Rainforest) is strongly recommended for production.

Rate Limiting & Anti-Block

• Random delay 1–3 seconds per request (configurable in src/config.py)
• 3 retries with exponential backoff (1s, 2s, 4s)
• Random User-Agent rotation across 6+ desktop browser profiles (dynamic via fake-useragent if installed, hardcoded fallback otherwise)
• Block detection: response < 200 characters or matching any blocked keyword (captcha, access denied, blocked, etc.) raises BlockedException

Tech Stack

Layer	Technology
Language	Python 3.10+
Scraping	requests + BeautifulSoup (lxml parser)
Data	pandas (CSV), sqlite3 (SQLite)
ML	scikit-learn — RandomForestRegressor, ColumnTransformer, StandardScaler, OneHotEncoder
Orchestration	Prefect — tasks, flows, cron scheduling
Experiment Tracking	MLflow — params, metrics, model registry, staging/production aliases
API	FastAPI + uvicorn + pydantic
Containerization	Docker (python:3.11-slim)

License

This project is for educational purposes only. Review target site Terms of Service before use.