Bookstore Data Pipeline — Docker Exercise

A hands-on exercise to build a mini data engineering pipeline using Python, PostgreSQL, and Docker.

What You Will Learn

• Ingest CSV data into PostgreSQL using pandas and SQLAlchemy
• Containerize a Python application with Docker
• Orchestrate services with Docker Compose
• Persist data using Docker volumes
• Connect services over Docker networks
• Use pgAdmin to inspect the database visually
• Manage configuration with environment variables

Project Structure

docker-book-project/
├── data/                   # Raw CSV datasets
│   ├── books.csv
│   ├── orders.csv
│   └── northwind.csv
├── notebooks/              # Jupyter notebooks
│   └── northwind_ingest.ipynb
├── exercises/              # Exercise instructions and solutions
│   ├── exercise.md
│   ├── exercise_solution.md
│   └── activity_1_postgresql_execution_inside_container.md
├── pgadmin/
│   └── servers.json        # Pre-configured pgAdmin server
├── ingest.py               # Main ingestion script
├── Dockerfile
├── docker-compose.yml
├── .env                    # Environment variables (credentials)
├── pyproject.toml
└── uv.lock

Prerequisites

• Docker Desktop
• Python 3.11+
• uv (pip install uv)
• Jupyter (for notebooks)

Setup — Python Environment with uv

uv is a fast Python package manager used in this project.

1. Install uv

pip install uv

2. Initialize the project (already done — pyproject.toml exists)

uv init --no-workspace

3. Add dependencies

uv add pandas sqlalchemy psycopg2-binary

4. Run a script

uv run ingest.py

uv automatically creates a virtual environment and installs dependencies — no manual venv activate needed.

---

Exercise Steps

Follow the instructions in exercises/exercise.md. The steps are:

Part	Description
1	Local setup — run ingestion script against a local PostgreSQL
2	Dockerize the Python ingestion app
3	Run a PostgreSQL container manually
4	Fix container networking
5	Wire everything together with Docker Compose
6	Verify data persistence across restarts
Bonus	Add a second dataset, pgAdmin, and .env config

Quick Start

# Start the full stack
docker compose up --build

# Open pgAdmin
open http://localhost:5050
# Login: admin@admin.com / admin
# Server bookstore is pre-configured — enter password: admin

# Connect to the database directly
docker exec -it bookstore-db psql -U postgres -d bookstore

Environment Variables

Credentials are managed via .env:

Variable	Default	Description
POSTGRES_USER	postgres	PostgreSQL superuser
POSTGRES_PASSWORD	admin	PostgreSQL password
POSTGRES_DB	bookstore	Database name
PGADMIN_EMAIL	admin@admin.com	pgAdmin login email
PGADMIN_PASSWORD	admin	pgAdmin login password

Running the Notebook

cd notebooks
jupyter notebook northwind_ingest.ipynb

Ensure bookstore-db is running first (docker compose up -d db), then use Kernel → Restart & Run All.