Skip to content

Backfill legacy jobs_job.logs into JobLog and drop the column #1278

Description

@mihow

Background

PR #1259 introduces the JobLog child table for per-job log persistence. Marks the legacy jobs_job.logs JSONB column as deprecated and leaves it in place as a read-only fallback for jobs created before the table existed.

Job.logs should not be removed yet — older jobs still surface their logs in the UI through the legacy fallback in serialize_job_logs. Removing the column outright would erase pre-migration job logs from the UI.

Approach

Two-step removal once #1259 has soaked:

  1. Backfill existing jobs_job.logs.{stdout, stderr} entries into JobLog rows. One-off Django management command (or data migration), batched, idempotent. Each legacy stdout line becomes one JobLog row with level parsed from the [ts] LEVEL ... prefix where present, raw line otherwise. created_at parsed from the line's prefix when available, falling back to the Job.created_at of the parent.
  2. Drop the column. Schema migration removing Job.logs SchemaField + the JobLogs Pydantic class. After this lands, serialize_job_logs collapses to the JobLog-only path and _legacy_logs_shape goes away.

Acceptance

  • Management command (or data migration) that backfills legacy JSON entries into JobLog rows. Idempotent — running twice does not duplicate rows.
  • Tested against a copy of the demo DB with a representative spread of legacy job rows.
  • Schema migration dropping Job.logs (separate, ships after the backfill is verified in production).
  • serialize_job_logs and friends in ami/jobs/models.py simplified — drop the legacy fallback branch.
  • PR body links Refactor job logging to use separate table #1259 and the soak summary.

Blocked by

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions