[tmpnet] Enable externally accessible URIs for kube-hosted nodes

[maru-ava]

Why this should be merged

Previously, access from outside of a kube cluster was enabled by port-forwarding through the kube API. This approach turned out incompatible with load testing because it greatly limited the rate at which transactions could be sent.

The port-forwarding is replaced with nginx+ingress to ensure minimum overhead when running outside of a kube cluster.

How this works

• deploy kind cluster with a static NodePort
• deploy the nginx ingress operator with helm and configure to use the NodePort
• update kube runtime to create a service and ingress for each node to configure external accessibility via nginx
• replace GetLocalURI with simplified GetAccessibleURI
• enable the configuration of ingress host and secret via the tmpnet defaults configmap

How this was tested

CI

Need to be documented in RELEASES.md?

N/A

TODO

• Perform self-review
• Ensure attribution for kind-with-registry.sh
• Merge [tmpnet] Source tmpnet defaults from a configmap #4057
• After merge, update the test cluster's tmpnet rbac configuration

[Copilot]

Pull Request Overview

This PR replaces port-forwarding with an externally accessible URI approach for kube-hosted nodes in order to improve load testing performance. Key changes include refactoring functions from GetLocalURI to GetAccessibleURI (and staking address equivalents), adding new logic to create Kubernetes Service and Ingress resources, and updating tests to use the new accessible URI methods.

Reviewed Changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/fixture/tmpnet/utils.go	Updated error handling in CheckNodeHealth and simplified Node URI retrieval
tests/fixture/tmpnet/start_kind_cluster.go	Added functions to deploy and verify the nginx ingress controller via Helm
tests/fixture/tmpnet/process_runtime.go	Replaced GetLocalURI with GetAccessibleURI and updated staking address method
tests/fixture/tmpnet/node.go	Updated Node interface methods to use the accessible URI versions
tests/fixture/tmpnet/network.go	Updated network URI retrieval to use GetAccessibleURI
tests/fixture/tmpnet/kube_runtime.go	Refactored URI retrieval for nodes to support external access and added Service/Ingress
tests/fixture/tmpnet/flags/kube_runtime.go	Introduced new flag for base accessible URI configuration
tests/e2e/*	Updated tests to use GetAccessibleURI (and staking address) throughout
scripts/kind-with-registry.sh	Added a new script to set up kind with a local registry
flake.nix	Removed kind-with-registry derivation and updated PATH configuration

[RodrigoVillar]

Ran this locally with the load test passing.

I tried using this against the remote cluster with no success (this is expected). However, I believe you mentioned that this PR should refuse deployment to a context other than kind-kind. Do you think we should add the refuse logic to this PR?

[maru-ava]

Ran this locally with the load test passing.

I tried using this against the remote cluster with no success (this is expected). However, I believe you mentioned that this PR should refuse deployment to a context other than kind-kind. Do you think we should add the refuse logic to this PR?

I meant to say that the test should only target kind-kind, not that tmpnet should refuse to deploy to a context other than kind-kind.

Was the failure something like failed to wait for Ingress...? It may make sense to log a warning if the ingress status suggests that the reason is the lack of an ingress controller.

[RodrigoVillar]

@maru-ava I'm getting the following error:

[06-26|09:29:52.291] ERROR avalanchego-load-test tests/simple_test_context.go:33
        Error Trace:    /Users/rodrigo.villar/go/src/github.com/ava-labs/avalanchego/tests/fixture/e2e/helpers.go:304
                                                /Users/rodrigo.villar/go/src/github.com/ava-labs/avalanchego/tests/fixture/e2e/env.go:185
                                                /Users/rodrigo.villar/go/src/github.com/ava-labs/avalanchego/tests/load/c/main/main.go:63
                                                /Users/rodrigo.villar/.goenv/versions/1.23.9/src/runtime/proc.go:272
                                                /Users/rodrigo.villar/.goenv/versions/1.23.9/src/runtime/asm_arm64.s:1223
        Error:          Received unexpected error:
                        failed to query node health: failed to issue request: Post "http://localhost:30791/networks/cb4ff442-bea0-401b-b5b5-5d7c38fc1e9c/NodeID-PWaVaDZqkNCfsYBtdCsXBSBkdas6Uj3Fu/ext/health": dial tcp 127.0.0.1:30791: connect: connection refused
        Messages:       failed to bootstrap network

Going back to the original question, logging a warning would be great here.

[maru-ava]

Rebased

[maru-ava]

Going back to the original question, logging a warning would be great here.

I added more checks, but when I was testing this, I realized that our test cluster actually has an ingress controller so the checks didn't actually help.

Instead, I've changed how the base accessible URI is configured so that it is no longer defaulted to the kind-kind value. Now if you try to deploy outside a kube cluster and the target cluster is not the kind cluster we deploy, it will complain that --base-accessible-uri must be provided and exit without deploying anything.

Edit: Updated to source the ingress configuration from a configmap in the cluster. That way it can be zero-config for the end-user.

[maru-ava]

Dropped basicauth protection - the guid and node id should be sufficient.

[Elvis339]

What kind of transaction rates are we hitting the wall with using port-forwarding? And what are we seeing now with nginx ingress? Just curious about the actual numbers that drove this change.

I see from @RodrigoVillar's testing that this doesn't work against remote clusters. Makes sense since it's tied to the kind setup. But what would it take to make this work for remote clusters too? Could we deploy a similar nginx ingress setup to a remote cluster and reuse the same accessible URI logic, or are there other blockers?

Trying to understand the scope and whether this could be extended beyond kind.

[maru-ava]

What kind of transaction rates are we hitting the wall with using port-forwarding? And what are we seeing now with nginx ingress? Just curious about the actual numbers that drove this change.

My understanding is that port-forwarding through the kube API limited throughput below what would be seen for a process-based network, and that with nginx removed that limitation.

I see from @RodrigoVillar's testing that this doesn't work against remote clusters. Makes sense since it's tied to the kind setup. But what would it take to make this work for remote clusters too? Could we deploy a similar nginx ingress setup to a remote cluster and reuse the same accessible URI logic, or are there other blockers?

Rodrigo's testing is outdated. This works for any cluster that a) has an nginx ingress controller deployed b) provides the tmpnet-defaults configmap that provides the necessary ingress configuration.