Skip to content

feat: HTTPProvider with AutoTLS#11333

Draft
lidel wants to merge 17 commits into
masterfrom
poc/wss-fallback-trustless-gateway
Draft

feat: HTTPProvider with AutoTLS#11333
lidel wants to merge 17 commits into
masterfrom
poc/wss-fallback-trustless-gateway

Conversation

@lidel
Copy link
Copy Markdown
Member

@lidel lidel commented May 16, 2026 •
edited
Loading

Important

Depends on:

That ☝️ PR must land and ship in a tagged go-libp2p release before this one can merge, but this is ready for early review.

A new experimental, opt-in HTTPProvider feature. When enabled with AutoTLS, kubo exposes its local trustless gateway over plain HTTPS on the same TCP swarm port as the /tls/ws libp2p listener, reusing the AutoTLS-issued Let's Encrypt cert. Browsers, curl, and any HTTP/2-over-TLS client can fetch verifiable blocks without a libp2p stack. Port-sharing depends on libp2p/go-libp2p#3509 (websocket.WithFallbackHTTPHandler).

User-facing details: v0.43 changelog highlight (likely to be pushed to later release), HTTPProvider config section (read-only, NoFetch, raw blocks only; mental model vs Gateway.*).

Test coverage

Four layers, each catching regressions a different way:

  • Unit tests (core/node/libp2p/): the settable HTTPProvider handler (unset/set/reset), the h2-required-over-TLS wrapper (h1/h2 x TLS/cleartext matrix), and the ?dial=/?dns= URL-overrides parser (used in CLI and E2E tests below).
  • CLI integration tests (test/cli/): real kubo daemon spawned per test. TestHTTPProvider covers h2c + HTTP/1.1 on the swarm port, TestHTTPProviderAutoTLS covers h2-over-TLS with h1-rejected and .well-known/libp2p/protocols served, TestHTTPProviderOverLibp2p covers the libp2p-stream variant.
  • End-to-end canary (test/autotls/): TestACMEEndToEnd brings up an in-process Pebble + p2p-forge and drives a real kubo daemon through registration, DNS-01 validation, cert install, and an HTTP/2 fetch via the announced /tls/http. ~6 seconds.
  • Gateway-conformance CI (three jobs): the full recursive Gateway suite, plus trustless-only conformance over HTTPProvider's libp2p-stream and h2c-on-swarm-port transports.

TODO

  • Wait for libp2p/go-libp2p#3509 to merge and ship in a tagged go-libp2p release; then bump the kubo go.mod off the pseudo-version.
  • Audit the metrics inherited from the old Experimental.GatewayOverLibp2p paths and rename anything that still carries the old name.
  • Decide whether to split the HTTPProvider metric (or add a label) so the libp2p-stream and native-HTTP transports show up separately. We currently cannot tell how much traffic reaches the gateway over plain HTTP/2, WebTransport, or the libp2p stream.
  • Audit the request timeouts and other safety limits that landed in boxo/gateway over the last two years; wire the ones that apply to the HTTPProvider handler (today it inherits whatever the existing trustless-gateway path uses, which may be incomplete).
  • Decide if we ship this enabled by default (so if you have AutoTLS, you also expose /tls/http), or opt-in for now.
    • 👉 Probably opt-in for N releases, then flip to enabled by default.
    • I am torn:
      • On one side, leaning towards enabling this asap, so inbrowser.link can hit over HTTP/2 and not /tls/ws, because /ws in libp2p is janky, it does not work over http/2 and modern browser hitting it as h2 fails, and has to retry as 1.1. If the browser hit HTTP handler as h2, we would not waste time for second handshake.
      • On other side, I am also leaning towards doing this slow, until we have means for signaling raw vs car vs webseed support per /tls/http endpoint in routing-system agnostic way that can be returned by /routing/v1 endpoints

Refs

Kubo's Relativity (1953), Lithograph

One hall, three staircases, three gravities. Each figurine knows its own staircase is the right one. The staircases are Bitswap, native HTTPS, and HTTP-over-libp2p. None is. All deliver. The block arrives.

lidel added 4 commits May 16, 2026 01:57
@lidel
chore(deps): pin go-libp2p to libp2p/go-libp2p#3509
1e424c5 
Picks up websocket.WithFallbackHTTPHandler so a /ws or /tls/ws listener
can serve a caller-supplied http.Handler for non-upgrade requests.
Required by the HTTPProvider feature in subsequent commits.
@lidel
refactor(httpprovider): rename WSSFallback, add config, deprecate old…
fd7c935 
… flag

Renames the lazy fallback handler from WSSFallback to HTTPProvider across
the kubo tree. The new name reflects the role this peer plays on the
network: an HTTP-native source for trustless-gateway block retrieval,
discoverable through identify/DHT alongside libp2p transports.

- new HTTPProvider config struct with Enabled/Libp2p/Cleartext/
  AnnounceMultiaddrs flags and matching Default* constants
- IpfsNode.WSSFallback field, FX param, file/struct/test renames; the
  libp2p-stream gateway function becomes serveHTTPProviderOverLibp2p
- Experimental.GatewayOverLibp2p marked deprecated; daemon refuses to
  start when it is set, with a one-line migration error pointing at
  HTTPProvider.Enabled and HTTPProvider.Libp2p
- libp2p-stream gateway gating switches from the deprecated flag to
  HTTPProvider.{Enabled,Libp2p}
- test/cli/http_gateway_over_libp2p_test.go renamed to
  http_provider_over_libp2p_test.go, subtests + config keys updated to
  match the new naming
@lidel
feat(httpprovider): expose trustless gateway over AutoTLS HTTPS
8b71576 
Mounts the same trustless-gateway handler that already serves the
libp2p-stream transport on the AutoTLS WebSocket port via the new
go-libp2p WithFallbackHTTPHandler hook, so any HTTP/2 client can fetch
blocks directly with the AutoTLS-issued cert and no libp2p stack.

Two pieces:

- RequireHTTP2OverTLS wraps the handler so HTTPS requests must speak
  HTTP/2; HTTP/1.1 over TLS gets 426 Upgrade Required, with an
  Upgrade: h2,websocket hint. HTTP/1.1 stays reserved for the
  WebSocket upgrade dance, and bitswap-httpnet gets the multiplexing
  it needs. Cleartext requests pass through regardless of HTTP
  version, so reverse-proxy deployments are unaffected.
- The HTTPS mux also serves /.well-known/libp2p/protocols (libp2p
  Gateway spec) via a fresh WellKnownHandler, mirroring what
  p2phttp.Host already auto-mounts on the libp2p-stream path. A
  bitswap-httpnet client hitting the AutoTLS hostname can discover the
  /ipfs/gateway protocol with one HTTPS GET, no libp2p stream needed.

The serveHTTPProviderOverLibp2p doc comment now points at
https://specs.ipfs.tech/http-gateways/libp2p-gateway/ and replaces the
stale FIXME on gatewayProtocolID.
@lidel
feat(httpprovider): announce /http multiaddrs and auto-append clearte…
dc07518 
…xt /ws

Two changes that turn this peer into a discoverable HTTP source for any
client following the libp2p+HTTP discovery model.

Announcement: HTTPProvider.AnnounceMultiaddrs (default true) derives an
HTTP-flavored sibling for every announced WebSocket multiaddr:
/ws becomes /http, /tls/ws becomes /tls/http,
/tls/sni/<host>/ws becomes /tls/sni/<host>/http. The /http entries
share the same TCP port and TLS cert as their /ws siblings, so this is
purely an announcement; no extra socket is opened. The derivation
runs after p2p-forge resolves the wildcard SNI and before the
Addresses.NoAnnounce filter, so derived addresses inherit the same
filtering as everything else. boxo/bitswap/network/httpnet picks them
up through identify, the DHT, and IPNI.

Cleartext: HTTPProvider.Cleartext (default false) auto-appends a
plaintext /ws listener to each /tcp/N already in Addresses.Swarm,
unless one is configured. Intended for reverse-proxy deployments where
TLS is terminated upstream and the proxy forwards either HTTP/1.1 or
h2c to kubo. Off by default because a node running AutoTLS does not
need a public cleartext path. The check excludes /wss and /tls/ws
(both TLS forms), so flipping it on alongside AutoTLS still adds
cleartext where the operator wants it.

The FX gate for the HTTPProviderHandler relaxes from
"AutoTLS && HTTPProvider" to just "HTTPProvider", because cleartext
/ws (manual or Cleartext-derived) needs the handler too.
@lidel lidel added the status/blocked Unable to be worked further until needs are met label May 16, 2026
@lidel lidel self-assigned this May 16, 2026
This was referenced May 16, 2026
Open
Merged
lidel added 7 commits May 16, 2026 20:58
@lidel
chore(deps): bump p2p-forge to v0.8.1
220d327 
Picks up client.WithHTTPClient and client.WithResolver, used by the new
?dial=/?dns= AutoTLS.RegistrationEndpoint overrides in a follow-up commit.
@lidel
feat(autotls): add test/debug config affordances
99eac13 
Three knobs that make end-to-end tests against a private ACME/forge stack
practical without polluting production paths:

- AutoTLS.TrustedCARootsPEM: trust a private CA bundle for the ACME
  directory HTTPS call. Already documented as accepting a Pebble or
  self-hosted root.
- AutoTLS.AllowPrivateForgeAddrs: lift the public-IP gate on cert requests
  so a loopback-only test node can complete a forge handshake.
- AutoTLS.SelfSignedForTests: skip the forge handshake entirely and hand
  the WebSocket transport an in-memory self-signed cert. Pure test escape
  hatch; off by default.

The SelfSignedForTests path lives in core/node/libp2p/autotls_selfsigned.go
behind a constructor wired into the libp2p host setup only when the flag
is on. Production builds with the flag off are unaffected.
@lidel
feat(autotls): add ?dial=/?dns= URL overrides
05cf8a9 
AutoTLS.RegistrationEndpoint now accepts two optional query parameters:

- ?dial=host:port reroutes the TCP dial of the registration POST while
  leaving the URL Kubo advertises (and signs over for PeerID-auth)
  unchanged. Lets a local p2p-forge instance on a non-privileged loopback
  port answer requests whose Host header must still match the production
  registration hostname.
- ?dns=host:port points the DNS-01 propagation pre-flight check at this
  DNS server instead of the system resolver. Needed when the forge's
  domain suffix is not in public DNS, e.g. a private .test suffix used
  in end-to-end tests.

The daemon strips both parameters before handing the URL to p2p-forge;
unrelated query parameters are preserved. Parsing and dispatch live in
parseForgeOverrides + forgeDialOverrideClient + forgeDNSOverrideResolver.
Covered by TestParseForgeOverrides; documented under AutoTLS.RegistrationEndpoint.
@lidel
test(autotls): add E2E canary sub-module
51350d1 
Adds test/autotls/ as a dedicated Go sub-module with its own go.mod, so
the heavy test dependencies (Pebble, CoreDNS, p2p-forge) do not leak into
kubo's main module or vulnerability-scanning surface.

The canary brings up an in-process Pebble (ACME test server) and the full
p2p-forge service (CoreDNS-based DNS + HTTP registration endpoint),
points a real kubo daemon at them via the new AutoTLS knobs and the
?dial=/?dns= overrides, and walks the chain end-to-end: registration POST,
DNS-01 validation, cert install, /tls/http announcement, HTTP/2 fetch of
a real block. One canary covers every link.

Run with: make test_autotls (or `cd test/autotls && go test ./...`).
Wired into CI as the autotls-tests job in gotest.yml.
@lidel
test(httpprovider): CLI tests + migrate flag
c05a053 
Adds two CLI test files exercising the HTTPProvider transports:

- TestHTTPProvider covers the plain-HTTP and h2c paths on the swarm port
  (HTTP/1.1, h2c, WSS upgrade still works, well-known/libp2p/protocols,
  refusal to serve deserialized responses).
- TestHTTPProviderAutoTLS covers the AutoTLS HTTPS path: h2 over TLS,
  rejection of h1 over TLS, well-known served, /tls/http multiaddr
  announced.

Also migrates the two pre-existing tests that still set the removed
Experimental.GatewayOverLibp2p flag (which the daemon now refuses to
start with) over to HTTPProvider.Enabled + HTTPProvider.Libp2p, and
renames the GatewayOverLibp2p subtest in content_blocking_test.go to
HTTPProviderLibp2p.
@lidel
fix(httpprovider): always install AutoWSS handler
552aea9 
serveHTTPProviderOverLibp2p early-returned when HTTPProvider.Libp2p was
off and so skipped installing the trustless gateway handler into the
AutoWSS-port placeholder. Any node with HTTPProvider.Enabled=true and
Libp2p=false (typical for HTTPProvider.Cleartext-only setups) ended up
serving permanent 503 Service Unavailable on /ws and /tls/ws even though
the listener was up and the placeholder was wired.

Restructure so the function does two independent jobs:

1. Always install the handler on the AutoWSS-port placeholder when
   HTTPProvider.Enabled=true. Covers AutoTLS, manually configured /ws,
   and HTTPProvider.Cleartext-derived /ws.
2. Then optionally start the libp2p-stream transport when
   HTTPProvider.Libp2p=true.

Also renames the local error channel from p2pGwErrc to httpProviderErrc
and replaces the now-stale "add trustless gateway over libp2p" comment.
@lidel
docs(httpprovider): clarify role and changelog
04b7c9f 
Documents HTTPProvider in docs/config.md (it had no section before) and
adds a v0.43 changelog highlight. Two related cleanups:

- Spell out the HTTPProvider vs Gateway mental model: server side (raw
  blocks via ?format=raw, NoFetch, swarm port, no recursion) versus the
  recursive loopback Gateway on 127.0.0.1:8080. Mirror "client side"
  framing onto HTTPRetrieval.
- Replace stale "raw blocks + CARs only" doc strings in
  config/httpprovider.go, core/corehttp/gateway.go, and
  core/node/libp2p/httpprovider.go with "raw blocks via ?format=raw only".
  HTTPProvider does not serve CARs.

Adds the gateway-conformance-http-provider-cleartext CI job and migrates
the existing gateway-conformance-libp2p-experiment job off the removed
Experimental.GatewayOverLibp2p flag. Updates docs/experimental-features.md
to point at HTTPProvider.Libp2p (the GraphSync section's cross-reference
too).
@lidel lidel changed the title feat(httpprovider): expose trustless gateway over AutoTLS HTTPS feat: HTTProvider with AutoTLS May 16, 2026
@lidel lidel changed the title feat: HTTProvider with AutoTLS feat: HTTPProvider with AutoTLS May 16, 2026
@lidel lidel force-pushed the poc/wss-fallback-trustless-gateway branch from f7864a1 to 04b7c9f Compare May 16, 2026 20:02
lidel added 3 commits May 16, 2026 22:07
@lidel
fix(ci): unbreak autotls-tests + drop noise from conformance jobs
23f5253 
Three small CI cleanups surfaced by the first PR run:

- mk/golang.mk: collapse the test_autotls recipe to a single line.
  The backslash line continuations worked locally with bash-as-sh but
  failed under dash on the GHA runner, which executed each line
  separately and tried to run the trailing backslashes as commands.
  Matches the single-line style already used by test_examples.

- .github/workflows/gateway-conformance.yml: drop
  Gateway.PublicGateways from the libp2p-experiment and cleartext jobs.
  Both run only the trustless-gateway conformance subset, which does
  not exercise subdomain routing, so the example.com / localhost public
  gateway config was unused.

- .gitignore: ignore test/autotls/autotls-tests.json (gotestsum
  jsonfile output), matching the existing rules for the cli and fuse
  test artifacts.
@lidel
ci(gateway-conformance): pin unique ports per job
53c71ca 
Each of the three conformance jobs now sets Addresses.Swarm, Gateway,
and API explicitly to ports unique across the matrix:

- full gateway suite: 14001 / 18080 / 15001
- libp2p (gateway node): 24001 / 28080 / 25001
- libp2p (proxy node):   24002 / 28081 / 25002, http-forward 28092
- cleartext HTTPProvider: 34001 / 38080 / 35001

Leading 1/2/3 identifies the job at a glance. No job inherits the kubo
defaults, so every port in a CI log unambiguously names the job it
belongs to and the jobs can safely share a runner if matrix scheduling
ever puts them there.
@lidel
fix(ci): commit zone file, gofmt nits, and stylecheck nit
19107ed 
Three follow-ups to the first PR run:

- test/autotls/zones/libp2p.test was used by the canary locally but
  never committed; the p2p-forge ipparser plugin reads it at CoreDNS
  startup. Without the file CI's canary failed before any test ran.
  The root .gitignore's *.test rule (intended for Go test binaries)
  matched the zone file, hence the per-directory .gitignore exception.
- gofmt: config/autotls.go const block alignment, trailing blank line
  at the end of test/cli/http_provider_test.go, harness.go const block,
  canary_test.go trailing blank line.
- stylecheck ST1005: the GatewayOverLibp2p migration error string ended
  with a period; reworded so it no longer does.
@lidel lidel mentioned this pull request May 16, 2026
Closed
lidel added 3 commits May 27, 2026 17:26
@lidel
chore(deps): bump p2p-forge to v0.9.0
35acb9b 
Pulls in pebble v2.10 API changes. The autotls harness now mirrors what
p2p-forge's own e2e suite did for v0.9.0: pebble VA dials CoreDNS's TCP
listener (pebble v2.10 forces TCP for ACME DNS lookups), and
pebbleCA.New uses keyAlg="rsa" since pebble's GetRootKey only handles
RSA.
@lidel
fix(httpprovider): default /http announcement to off
23e144b 
HTTPProvider.Enabled stays off by default; once enabled, the
libp2p-stream transport comes up automatically (it is the in-band path
.well-known already advertises) while Cleartext and the /http multiaddr
announcement stay off until the operator opts in explicitly. The const
block carries the default rationale; field godocs no longer restate
default values.
@lidel
Merge branch 'master' into poc/wss-fallback-trustless-gateway
0263193 
Pulls in Boxo v0.40.0, go-libp2p-kad-dht v0.40.0, cheggaaa/pb v3, and
the dag --local-only / migration fetcher work from master.

Dep conflicts resolved by keeping our HTTPProvider-required pins on top
of master's baseline:

- certmagic v0.25.3 (pulled in by p2p-forge v0.9.0)
- p2p-forge v0.9.0
- go-libp2p v0.48.1-0.20260515215300-a72c0588b088 (pin for libp2p/go-libp2p#3509,
  still open; needed for websocket.WithFallbackHTTPHandler)

Other transitive bumps follow from `make mod_tidy`. AutoTLS canary +
HTTPProvider CLI tests pass on the merged tree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@lidel lidel

Labels

status/blocked Unable to be worked further until needs are met

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lidel