Add external_job_uri as a first-class proto field

[dejanzele]

Summary

The external job URI already gets special treatment in Armada (its own DB column, its own query API endpoint), but clients have to know the magic annotation key armadaproject.io/externalJobUri to set it. There is nothing in the API that tells you this annotation exists or matters.

This promotes it to a proper proto field on JobSubmitRequestItem and SubmitJob so clients get autocompletion, type checking, and generated docs for free.

Backward compatibility

• Old client, new server: Client sets only the annotation. Server reads it via fallback, populates the proto field on the outgoing event, and mirrors it back to the annotation.
• New client, old server: Client sets the proto field. Old server ignores the unknown proto field but the annotation still works as before.
• Old Pulsar events, new ingester: Ingester reads proto field first, falls back to annotation for events that predate the field.
• Independent component updates: The resolved URI is mirrored back into the annotation on the outgoing event, so components that only read annotations continue to work even if updated independently.

Changes

• pkg/api/submit.proto, pkg/armadaevents/events.proto - Added external_job_uri field
• internal/server/submit/conversion/conversions.go - Resolve from proto field first, fall back to annotation, mirror into annotation
• internal/lookoutingester/instructions/instructions.go - Read proto field first, fall back to annotation
• internal/common/constants/constants.go - Shared constant for the annotation key
• client/rust/src/builder.rs - Added field to Rust client builder
• Tests for both conversion and ingester fallback logic

No DB migration needed since the external_job_uri column already exists.

Test plan

• go test ./internal/server/submit/conversion/... passes
• go test ./internal/lookoutingester/instructions/... passes
• mage lintCheck passes
• mage buildPython generates the field in the Python client
• cargo build and cargo test pass for Rust client
• Local e2e: submitted job with proto field, verified external_job_uri column populated in lookout DB
• Local e2e: submitted job with annotation only (old client), verified fallback works and column populated

[greptile-apps]

Greptile Summary

This PR promotes external_job_uri from a magic annotation to a first-class proto field on both JobSubmitRequestItem (submit API) and SubmitJob (internal Pulsar event). The server-side conversion and lookout ingester both implement a "proto field wins, annotation fallback" strategy, preserving full backward compatibility with old clients and old Pulsar events.

Confidence Score: 5/5

Safe to merge — all three compat scenarios are handled, field numbers are correct, and test coverage is thorough.

No P0 or P1 findings. Proto field numbers don't conflict with reserved fields. The proto-wins/annotation-fallback logic is consistent in both the server conversion and the ingester. The previous concern about mutating the caller's annotations map is resolved. Test suite covers all edge cases including nil annotations.

No files require special attention.

Vulnerabilities

No security concerns identified. The new field is a plain string URI with no execution or privilege implications. It is stored in the DB column that already exists and is subject to length truncation (1024 chars) in the ingester.

Important Files Changed

Filename	Overview
pkg/api/submit.proto	Adds external_job_uri = 13 to JobSubmitRequestItem; field 12 is reserved so 13 is the correct next number. No conflicts.
pkg/armadaevents/events.proto	Adds external_job_uri = 16 to SubmitJob; fields 1, 13, 15 are reserved, so 16 is correct. No conflicts.
internal/server/submit/conversion/conversions.go	Correctly resolves externalJobUri by preferring the proto field and falling back to the annotation; no map mutation; sets the resolved value on SubmitJob.ExternalJobUri.
internal/lookoutingester/instructions/instructions.go	Same proto-field-first / annotation-fallback logic as the server conversion; includes truncation to maxAnnotationValLen before storing.
internal/server/submit/conversion/conversions_test.go	Adds TestExternalJobUri covering proto-wins, annotation-fallback, neither-set, and nil-annotations cases — good coverage.
internal/lookoutingester/instructions/instructions_test.go	Adds TestExternalJobUriProtoFieldPreferred with identical three-case coverage; existing TestConvert updated to use the named constant.
internal/common/constants/constants.go	Adds ExternalJobUriAnnotation constant to replace the magic string "armadaproject.io/externalJobUri" used in four places; clearly documented as legacy.
client/rust/src/builder.rs	Adds external_job_uri field and builder method to the Rust JobRequestItemBuilder; correctly wired into both build() and build_from_pod_spec() paths.

Sequence Diagram

sequenceDiagram
    participant C as Client
    participant S as Server (conversions.go)
    participant P as Pulsar
    participant I as Ingester (instructions.go)
    participant DB as Lookout DB

    C->>S: JobSubmitRequestItem\n(ExternalJobUri proto field OR annotation)
    Note over S: Prefer proto field\nFall back to annotation
    S->>P: SubmitJob event\n(ExternalJobUri field set)
    P->>I: Consume SubmitJob event
    Note over I: Prefer ExternalJobUri field\nFall back to annotation\n(for pre-field events)
    I->>DB: CreateJobInstruction\n(external_job_uri column)

Loading

_{Reviews (9): Last reviewed commit: "Merge branch 'master' into external-job-..." | Re-trigger Greptile}

[dejanzele]

@greptileai

[dejanzele]

@greptileai