feat: resilient background job retry & monitoring (bounty #130)

[alexanderxfgl-bit]

Bounty Claim: $250 — Issue #130

Implementation

Production-ready background job queue with retry, exponential backoff, and dead-letter queue support.

Backend Changes

Model (app/models.py):

• JobStatus enum: pending → running → succeeded/failed → retrying → dead
• BackgroundJob model with: task_type, payload (JSON), status, attempts, max_attempts, last_error, result, next_retry_at, scheduled_for

Service (app/services/jobs.py):

• enqueue_job() — create new jobs with optional scheduling
• mark_running/succeeded/failed() — lifecycle transitions
• _compute_backoff() — exponential backoff (30s base, 2x multiplier, 1hr cap)
• get_jobs_due_for_retry() / get_pending_jobs() — query helpers for workers
• mark_pending() — manual retry of dead/failed jobs
• Dead-letter queue: jobs exceeding max_attempts move to DEAD status

Routes (app/routes/jobs.py):

• POST /jobs — enqueue job with validation
• GET /jobs — list with pagination + status/task_type filters
• GET /jobs/<id> — get job details
• POST /jobs/<id>/retry — manual retry of failed/dead jobs
• DELETE /jobs/<id> — remove job
• GET /jobs/dead — dead-letter queue listing
• GET /jobs/stats — job statistics (total/pending/running/succeeded/failed/dead)

Schema (app/db/schema.sql):

• background_jobs table with job_status enum type
• Optimized partial indexes for retry and pending queries

Tests

11 tests covering:

• CRUD operations (create, list, get, delete)
• Input validation (missing task_type, invalid types)
• Scheduled job creation
• Failed job retry workflow
• Retry validation (pending jobs cannot be retried)
• Job statistics
• Filtering by task_type
• Dead-letter queue listing

Closes #130