fix(container-cache): consistent-hash relay lacks connection reuse and local caching, and is unobservable

[balajinvda]

Summary

Consistent-hash routing in the container-cache proxy tier relays roughly two
thirds of requests to a peer pod. That relay path has two defects that add
latency on every relayed request, and the relay itself is invisible in metrics,
so the cost cannot currently be measured.

Background

cc-route.lua selects an owner with md5(cache_key) mod N and relays to that
peer when the receiving pod is not the owner:

local owner = tonumber(string.sub(ngx.md5(ngx.var.cc_hash_key), 1, 8), 16) % n
if self ~= nil and owner ~= self then ngx.var.cc_owner = "cc_owner_" .. owner

With a 3 pod tier and clients arriving uniformly through the Service, 2 of every
3 requests take the relay hop. Because the hash key is the per-range cache key,
a single large object's byte ranges deliberately spread across all owners, so
one client downloading one object relays most of its chunks.

Defect 1: the relay has no connection reuse

proxy-common.conf sets proxy_set_header Connection ""; at server scope to
enable upstream keepalive. @cc_relay then defines its own proxy_set_header
directives (Host, Range, X-NVCF-CC-Relayed). nginx cancels inheritance of
proxy_set_header as soon as a level declares any of its own, so the relay
never receives the server-level Connection "" and falls back to the nginx
default of Connection: close.

The cc_owner_* upstream blocks also declare no keepalive directive, so there
is no connection pool to reuse even if the header were correct.

Result: every relayed request opens a new TCP connection and performs a new TLS
handshake to the peer. Range chunks make this worse, since the hash spreads them
across owners and produces many small relayed requests where handshake cost
dominates.

Defect 2: the relay never caches, so hot objects relay indefinitely

location @cc_relay {
  proxy_cache off;

An object requested repeatedly through a non-owner relays on every request for
its entire lifetime. There is no mechanism by which frequently requested objects
stop paying the hop.

Defect 3: relay is not observable

Relayed and locally served requests are indistinguishable in
proxy_cache_request_duration_seconds and in the throughput histograms. The
host label only ever carries the upstream origin, never a peer. There is no
counter for relayed versus local, so the relay fraction and its latency cost
cannot be measured from metrics today.

Two related histogram problems surfaced while investigating:

• proxy_cache_request_duration_seconds has a largest finite bucket of 10s,
  while about 43 percent of observed requests exceed it. histogram_quantile
  clamps at the last finite bucket, so any reported p99 is an artifact rather
  than a measurement.
• proxy_cache_response_size_bytes jumps from 100MB to 1GB to 10GB, so
  essentially all traffic lands in one bucket and the size distribution is
  invisible.

Observed impact

Measured on one production cluster, 3 pod tier, roughly 41 hours of counters:

• about 520 TB served to clients in aggregate
• cache hit rate 99.98 percent, average object about 528 MB
• with a 2/3 relay fraction, each relayed byte crosses the network twice, so
  per pod NIC traffic is about 2.3x what it would be without relay
• storage utilisation is even across pods, which is the intended benefit and is
  working correctly

The tier trades network bandwidth for storage efficiency. Where storage is not
the constraint and latency is, that trade currently runs the wrong way.

Proposed fix

Configuration only, no architecture change:

1. Add proxy_set_header Connection ""; inside @cc_relay and keepalive to
   each cc_owner_* upstream, restoring connection reuse on the peer hop.
2. Cache on the relay with a use threshold, so objects repeatedly requested
   through a non-owner replicate locally and stop relaying, while one-off
   objects continue to live only on their owner. Bounded by the existing
   free space based eviction.
3. Add a local versus relayed label to the request and throughput metrics,
   and widen the duration and response size histogram buckets so the resulting
   latency is measurable.

Follow-up options, not in scope here

• Replication factor of two in the owner selection, reducing the relay fraction
  from 2/3 to 1/3 at 2x storage.
• Returning a redirect to the owner instead of relaying, so the body crosses the
  network once. This removes the bandwidth amplification entirely but requires
  solving client side trust for the per pod peer certificates.
• Dropping TLS on the internal peer hop, which already runs with peer
  verification disabled and therefore provides encryption rather than
  authentication.

Testing

Chart render tests plus a load comparison of relayed versus local request
latency once the new label is in place.