Skip to content

Add per-request image count to logs and metrics - #201

Merged
bdehamer merged 2 commits into
mainfrom
bdehamer-request-image-count
Aug 19, 2026
Merged

Add per-request image count to logs and metrics#201
bdehamer merged 2 commits into
mainfrom
bdehamer-request-image-count

Conversation

@bdehamer

Copy link
Copy Markdown
Contributor

Summary

When Gatekeeper's admission controller validates a pod, it sends all of the pod's image references in a single external-data request. Today we have no visibility into how many images ride along in each request. This PR captures that count to support two goals:

  1. Distribution analysis — are we primarily validating single-image pods, or large multi-image pods?
  2. Timeout troubleshooting (primary driver) — when an attestation fetch fails with canceled/timeout (context_cancelled), trace that specific failure back to determine whether it was a solo validation or one image in a larger, multi-image request that ran out of time.

The provider processes images sequentially in one loop that shares the request context/deadline, so later images in a big request are the ones most likely to time out — which is exactly the pattern these fields surface.

Changes

  • pkg/metrics/prom.go — new histogram aaop_attestations_request_images (buckets 1, 2, 3, 5, 10, 20, 50), observed once per request. A histogram (not a counter or a label) is the right tool: it yields the distribution (_bucket{le="1"} / _count for the single-image fraction, histogram_quantile for percentiles, _sum/_count for the average) without exploding metric label cardinality.
  • pkg/provider/provider.go — in Validate, generate a per-request request_id (UUID) and thread request_id + image_count + image_index through the per-image log lines via slog.With. Every per-image line (success and failure) now self-describes its request context, so a single reason="canceled"/"timeout" line reads e.g. image_count=17, image_index=14 → "image 14 of a 17-image request ran out of time."
  • go.mod — promote github.com/google/uuid from an indirect to a direct dependency (already in the module graph).
  • pkg/provider/provider_test.go — a histogram test (_count +1, _sum +N) and a log-correlation test asserting a failure line's request_id matches the request entry line and carries image_count/image_index.
  • README.md — document the new metric and log fields.

Pure observability — no change to the provider's request/response contract or verification behavior.

Example log line after this change

{"level":"ERROR","msg":"validate: error fetching bundles","request_id":"",
 "image_count":17,"image_index":14,"image":"","reason":"canceled",}

Validation

  • go build ./...
  • go test ./... (all packages pass)
  • golangci-lint run on changed packages — clean

Notes / trade-offs

  • Buckets are a tunable starting point; add a higher bucket if very large pods appear in real traffic.
  • The entry log is INFO, one line per request; per-image lines already existed and simply gain fields.

Gatekeeper sends all of a pod's images in a single external-data request.
Capture that count so we can (1) analyze the distribution of images per
request and (2) trace a failed image fetch back to whether it was a solo
or a large multi-image request.

- metrics: add aaop_attestations_request_images histogram, observed once
  per request.
- provider: generate a per-request request_id and thread
  request_id/image_count/image_index through the per-image log lines via
  slog.With, so a single failure line (e.g. a canceled/timeout fetch)
  self-describes its request context.
- promote github.com/google/uuid to a direct dependency.
- add tests and document the metric and log fields in the README.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c1b5c800-5c7c-466d-83f4-c1aaa75967e5

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds per-request image-count observability for validation and timeout troubleshooting.

Changes:

  • Adds an image-count histogram.
  • Correlates request and per-image logs using UUIDs and image indexes.
  • Adds tests and documentation for the new telemetry.
Show a summary per file
File Description
README.md Documents metrics and log fields.
go.mod Promotes the UUID dependency.
pkg/metrics/prom.go Defines the image-count histogram.
pkg/provider/provider.go Records metrics and structured request context.
pkg/provider/provider_test.go Tests histogram and log correlation behavior.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

  • Files reviewed: 5/5 changed files
  • Comments generated: 2
  • Review effort level: Balanced

Comment thread pkg/provider/provider_test.go Outdated
Comment thread pkg/provider/provider_test.go
- Promote github.com/prometheus/client_model to a direct dependency; the
  test imports it directly for the DTO type, so it should not be marked
  indirect (go mod tidy would otherwise rewrite go.mod).
- Assert the per-image failure line's image_index is exactly 2 (the
  second image) instead of merely non-nil, covering the 1-based position
  semantics.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c1b5c800-5c7c-466d-83f4-c1aaa75967e5

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 5/5 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

@bdehamer
bdehamer marked this pull request as ready for review August 19, 2026 15:28
@bdehamer
bdehamer requested a review from a team as a code owner August 19, 2026 15:28
@bdehamer
bdehamer merged commit e5e9404 into main Aug 19, 2026
12 checks passed
@bdehamer
bdehamer deleted the bdehamer-request-image-count branch August 19, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants