Skip to content

App runs stopped by a server restart show Failed with no reason, and finished RDF runs stay Failed #33900

Description

@harshach

Problem

When the server executing an app run stops (restart, crash or redeploy), the run is left without any explanation:

  • The next server to start runs ApplicationHandler.cleanupStaleJobs(), which flips every running run record to failed with a bare SQL update. It writes no failure message and no end time, so the UI shows Failed with nothing to explain it, the Logs link returns the same record, and the duration keeps growing because the UI falls back to Date.now() when endTime is missing.
  • The search indexer's recovery (JobRecoveryManager) syncs its run record with the same kind of bare update, dropping the reason it already has ("Job abandoned due to server crash or shutdown").
  • Distributed RDF indexing: when the coordinating server stops, the other servers reclaim its partitions and the job still reaches COMPLETED, but only the coordinator writes the run record. The run stays Failed even though every entity was indexed.

Seen on devrel on 2026-09-23: an RdfIndexApp run started at 13:41 UTC showed Failed with 10,590 of 11,985 records and no reason. The graph actually held all 5,695 tables.

Expected

  • A run marked failed because its server stopped records why and when.
  • A distributed RDF run whose coordinator stopped records the job's real outcome once the other servers finish it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    • Status
      In Review / QA 👀

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions