BioSeq Data Platform

A production-grade, cloud-native data engineering platform for bioinformatics sequencing metadata.

Built to demonstrate real-world data engineering skills for biotech environments - ingesting, validating, transforming, and serving NCBI SRA sequencing metadata through a multi-zone AWS data lake, PostgreSQL warehouse, REST API, and analytics dashboard.

---

Architecture

NCBI SRA API
     |
     v
Python Ingestion Service (boto3, tenacity, structlog)
     |
     v
S3 Bronze Zone (raw JSON, gzip, date-partitioned)
     |
     v
AWS Glue Crawler --> Glue Data Catalog
     |
     v
Validation & Quality Engine (Pydantic, rule engine)
     |-- Passed --> S3 Silver Zone (Parquet, Snappy)
     |-- Failed --> S3 Quarantine (JSON + rejection reason)
          |
          v
     Amazon Athena (serverless SQL over S3)
          |
          v
     RDS PostgreSQL (star schema warehouse)
          |
          v
     FastAPI (ECS Fargate) --> Power BI Dashboard

---

Technology Stack

Category	Technologies
Languages	Python 3.11, SQL, HCL
Data Engineering	AWS Glue, Apache Parquet, Snappy compression
Storage	Amazon S3 (multi-zone data lake), Amazon RDS PostgreSQL 15
Querying	Amazon Athena (serverless SQL)
API	FastAPI, SQLAlchemy 2.0 (async), Pydantic v2
Infrastructure	Terraform, AWS IAM, VPC, CloudWatch, SNS
Containers	Docker (multi-stage builds), Amazon ECR, ECS Fargate
CI/CD	GitHub Actions (lint -> test -> build -> deploy)
Testing	pytest, moto, Great Expectations
Monitoring	CloudWatch Logs, Metrics, Alarms, SNS alerts

---

Data Quality

The platform enforces 9 validation rules across four categories:

• Identity - run_accession non-null, matches SRR/ERR/DRR regex
• Dates - ISO 8601 format, not in the future
• Metrics - bases and spots non-negative integers
• Completeness - organism, platform, study_title present

Failed records are routed to a quarantine prefix with rejection reason codes. They are never silently dropped.

From 300 records ingested: 298 passed (99.3%), 2 quarantined (0.7%)

---

API Endpoints

Endpoint	Description
GET /health	Liveness check with DB connectivity and run count
GET /studies	Paginated study list with filter support
GET /studies/{accession}	Full study detail with run count
GET /organisms	Organism list with run counts
GET /runs/summary	Aggregate run statistics
GET /runs/{accession}	Individual run detail
GET /platforms/summary	Platform distribution
GET /quality/report	Data quality KPIs

---

Running Locally

# Clone and set up
git clone https://github.com/YOUR_USERNAME/bioseq-platform.git
cd bioseq-platform
python -m venv .venv

# Windows
.venv\Scripts\activate

# Mac/Linux
source .venv/bin/activate

pip install -r requirements.txt
pip install -e .

# Configure environment
cp .env.example .env
# Edit .env with your AWS credentials and RDS endpoint

# Run ingestion
python -m ingestion.src.pipeline

# Run validation
python -c "from validation.src.pipeline import ValidationPipeline; ValidationPipeline().run()"

# Load warehouse
python api/models/loader.py

# Start API
uvicorn api.main:app --reload --port 8000
# Visit http://localhost:8000/docs

---

Infrastructure

All AWS resources are provisioned via Terraform:

cd infra
terraform init
terraform apply

Estimated cost: ~20 GBP/month with RDS running continuously.

To pause RDS when not in use:

aws rds stop-db-instance --db-instance-identifier bio-platform-postgres

To restart RDS:

aws rds start-db-instance --db-instance-identifier bio-platform-postgres

To destroy all resources:

terraform destroy

Infrastructure can be recreated from zero in under 10 minutes.

---

Project Structure

bioseq-platform/
├── infra/                   # Terraform - all AWS resources
│   ├── main.tf
│   ├── variables.tf
│   ├── s3.tf
│   ├── rds.tf
│   ├── iam.tf
│   ├── glue.tf
│   ├── monitoring.tf
│   └── outputs.tf
├── ingestion/               # NCBI SRA ingestion pipeline
│   ├── src/
│   │   ├── config.py        # Centralised settings (Pydantic)
│   │   ├── ncbi_client.py   # NCBI API client with retry + rate limiting
│   │   ├── writer.py        # S3 Bronze zone writer (gzip, partitioned)
│   │   ├── checkpoint.py    # Idempotency checkpoint store
│   │   └── pipeline.py      # Pipeline orchestrator
│   └── tests/
├── validation/              # Data quality engine
│   ├── src/
│   │   ├── rules.py         # 9 validation rules
│   │   ├── validator.py     # Rule engine + completeness scoring
│   │   ├── silver_writer.py # Parquet writer + quarantine handler
│   │   ├── metrics.py       # CloudWatch metrics emitter
│   │   └── pipeline.py      # Validation orchestrator
│   └── tests/
├── glue_jobs/               # AWS Glue ETL scripts
│   └── bronze_to_silver.py
├── api/                     # FastAPI application
│   ├── main.py
│   ├── models/
│   │   ├── database.py      # SQLAlchemy async + sync engines
│   │   ├── tables.py        # ORM models (star schema)
│   │   ├── schemas.py       # Pydantic response models
│   │   └── loader.py        # Silver zone -> PostgreSQL loader
│   └── routers/
│       ├── health.py
│       ├── studies.py
│       ├── organisms.py
│       ├── runs.py
│       ├── platforms.py
│       └── quality.py
├── alembic/                 # Database migrations
├── docs/
│   ├── architecture.md      # Architecture Decision Records (ADRs)
│   ├── data_dictionary.md   # Column-level definitions
│   └── screenshots/         # AWS console evidence
├── .github/
│   └── workflows/
│       └── ci.yml           # GitHub Actions CI pipeline
├── Dockerfile               # Multi-stage, non-root user
├── docker-compose.yml       # Local full-stack setup
├── requirements.txt
├── setup.py
├── pyproject.toml
└── .env.example

---

Data Model

Star schema optimised for analytical queries:

dim_organism ----+
                 |
dim_platform ----+--> fact_sequencing_run --> fact_metadata_quality
                 |
dim_study -------+

Dimensions: dim_organism, dim_platform, dim_study

Facts: fact_sequencing_run (298 rows), fact_metadata_quality (298 rows)

---

Test Coverage

ingestion/src/checkpoint.py    100%
ingestion/src/logger.py        100%
validation/src/validator.py     98%
validation/src/rules.py         83%
ingestion/src/writer.py         79%
ingestion/src/config.py         92%

Run tests:

pytest ingestion/tests/ validation/tests/ -v --cov

---

Architecture Decisions

Key decisions documented in docs/architecture.md:

• ADR-001 - Medallion lake (Bronze/Silver/Gold) over single-zone
• ADR-002 - Quarantine-not-delete for failed validation records
• ADR-003 - PostgreSQL alongside Athena for low-latency API queries
• ADR-004 - FastAPI over Flask for native async and auto OpenAPI docs
• ADR-005 - Terraform for full infrastructure reproducibility
• ADR-006 - Architecture portable to GCP (BigQuery, GCS, Cloud Run)

---

GCP Portability

This platform is designed to be portable. AWS to GCP mapping:

AWS	GCP Equivalent
Amazon S3	Google Cloud Storage
AWS Glue	Dataflow / Dataproc
Amazon Athena	BigQuery
RDS PostgreSQL	Cloud SQL
ECS Fargate	Cloud Run
CloudWatch	Cloud Monitoring
SNS	Pub/Sub