Skip to content

feat(kube): replace broken kube/ with working Kustomize manifests - #278

Open
dnplkndll wants to merge 4 commits into
hcengineering:mainfrom
ledoent:feat/kustomize-kube
Open

feat(kube): replace broken kube/ with working Kustomize manifests#278
dnplkndll wants to merge 4 commits into
hcengineering:mainfrom
ledoent:feat/kustomize-kube

Conversation

@dnplkndll

@dnplkndll dnplkndll commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Replaces the existing broken kube/ manifests (see #181) with a Kustomize-based deployment structure, aligned with the merged Helm chart (#280, #286).

What changed

kube/
  base/
    config/         # ConfigMap — placeholder domain: huly.example
    infra/          # CockroachDB v24.2, Redpanda v24.3, ES 7.14.2, MinIO
    app/            # 9 Huly services with init containers + readiness/liveness probes
    ingress/        # 6 nginx Ingress resources (path-based routing)
  overlays/
    example/        # Template overlay users copy & customize
  deploy.sh         # One-liner deploy script
  README.md         # Complete deployment guide

Services (13 pods total)

Service Port Protocol
front 8080 HTTP
account 3000 HTTP
transactor 3333 WebSocket
collaborator 3078 WebSocket
workspace background
fulltext 4700 HTTP
kvs 8094 HTTP
rekoni 4004 HTTP
stats 4900 HTTP
cockroach 26257 SQL
redpanda 9092 Kafka
elasticsearch 9200 HTTP
minio 9000 S3

Key design decisions

  1. Path-based routing (/_accounts, /_transactor, etc.) — matches compose.yml nginx config
  2. Elasticsearch 7.14.2 (not OpenSearch) — Huly's elasticsearch-js rejects non-ES backends
  3. Explicit env vars on backend services — prevents internal services from getting external URLs
  4. Init containers for startup ordering — busybox nc -z wait loops
  5. Readiness + liveness probes on all services — aligned with Helm chart defaults
  6. No secrets in repo — imperative kubectl create secret
  7. huly.example placeholder — users customize via overlay
  8. CockroachDB --insecure mode with root user (no password) — matches Helm chart

Alignment with Helm chart

The second commit brings Kustomize manifests to parity with the merged Helm chart:

  • KVS service added (pinned to s0.7.392hulykvs does not publish a :latest tag)
  • Liveness probes on all services, readiness initialDelaySeconds aligned with Helm
  • CI workflow fixed: kubectl apply -k, correct deployment names, namespaced commands
  • Account: optional GitHub OAuth, OIDC, and DISABLE_SIGNUP env vars
  • Transactor: SERVER_CURSOR_MAXTIMEMS
  • ConfigMap: DESKTOP_UPDATES_CHANNEL
  • Redpanda: pandaproxy + schema-registry args, external listeners, superuser name fix, init container for data dir ownership
  • Elasticsearch: 1024m heap, exec-based readiness probe, BITNAMI_DEBUG, removed sysctl init container
  • MinIO: liveness probe, PVC increased to 50Gi
  • CockroachDB: use root user for --insecure mode (COCKROACH_USER/PASSWORD removed — they have no effect in insecure mode)
  • deploy.sh / README: updated secret creation, documented auth providers

Renders

kubectl kustomize kube/overlays/example/ produces 36 resources:

  • 13 Deployments (9 app + 4 infra)
  • 12 Services
  • 6 Ingresses
  • 4 PVCs
  • 1 ConfigMap

Testing

  • kubectl kustomize kube/overlays/example/ renders valid YAML (36 resources)
  • Deployed to Hetzner K8s cluster — 13/13 pods Running, 0 restarts
  • TLS certificate issued via cert-manager
  • Front responds (200), Account API responds (405), createAccount RPC returns token
  • All Huly images use latest in base except hulykvs (pinned, no :latest published)
  • No provider-specific values in base (storageClassName, domains, emails)

🤖 Generated with Claude Code

@huly-github-staging

Copy link
Copy Markdown

Connected to Huly®: SELFH-277

@ArtyomSavchenko

Copy link
Copy Markdown
Member

Hi @dnplkndll
Could you also check failed Huly deploy step in CI?
https://github.com/hcengineering/huly-selfhost/actions/runs/22860243485/job/66909781849#step:7:63

resource mapping not found for name: "" namespace: "" from "base/ingress/kustomization.yaml": no matches for kind "Kustomization" in version "kustomize.config.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "" namespace: "" from "base/kustomization.yaml": no matches for kind "Kustomization" in version "kustomize.config.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "" namespace: "" from "overlays/example/kustomization.yaml": no matches for kind "Kustomization" in version "kustomize.config.k8s.io/v1beta1"
ensure CRDs are installed first

dnplkndll and others added 4 commits March 21, 2026 10:20
The front service needs the GitHub App Client ID to build the OAuth
authorization URL. Without it, the redirect has client_id= (empty)
and GitHub returns 404.

Signed-off-by: Daniel Kendall <dkendall@ledoweb.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Don Kendall <kendall@donkendall.com>
The front service needs GITHUB_APP (app slug) to build the GitHub App
installation URL. Without it, the URL becomes /apps//installations/new
which returns 404.

Signed-off-by: Daniel Kendall <dkendall@ledoweb.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Don Kendall <kendall@donkendall.com>
Replace the existing flat kube/ manifests (broken for v7, see hcengineering#181) with a
battle-tested Kustomize-based deployment structure.

Structure:
  kube/base/        - config, infra, app, ingress layers
  kube/overlays/    - example overlay users copy & customize
  kube/deploy.sh    - one-liner deploy script

Key design decisions:
- Path-based routing (/_accounts, /_transactor, etc.) matching compose.yml
- Elasticsearch 7.14.2 (not OpenSearch) - Huly's elasticsearch-js rejects
  non-ES backends
- Explicit env vars on backend services (no bulk envFrom leaking external URLs)
- Init containers for startup ordering (busybox wait loops)
- Readiness probes on all services (TCP for WS/POST-only, HTTP for REST)
- Imperative secret creation (no secret YAML in repo)
- huly.example placeholder domain - users must customize via overlay

Tested on K3s with v0.7.382. Renders 34 resources:
12 Deployments, 11 Services, 6 Ingresses, 4 PVCs, 1 ConfigMap.

Closes hcengineering#181
Addresses hcengineering#20
Supersedes hcengineering#215

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add KVS service (deployment + service, pinned to s0.7.392 since
  hulykvs does not publish a :latest tag)
- Add liveness probes to all services, align readiness initialDelaySeconds
  with Helm defaults
- Fix CI workflow: use kubectl apply -k, correct deployment names,
  path-based routing, namespaced commands
- Add optional auth env vars to account (GitHub OAuth, OIDC, DISABLE_SIGNUP)
- Add SERVER_CURSOR_MAXTIMEMS to transactor
- Add DESKTOP_UPDATES_CHANNEL to configmap
- Align Redpanda args with Helm (pandaproxy, schema-registry, external
  listeners), fix superuser username, use init container for data dir
- Align Elasticsearch with Helm (1024m heap, exec readiness probe,
  BITNAMI_DEBUG, remove sysctl init container)
- Add MinIO liveness probe, increase PVC to 50Gi
- Fix CockroachDB: use root user for --insecure mode (no password auth)
- Fix deploy.sh and README secret creation to match

Tested on Hetzner cluster: 13/13 pods running, 0 restarts,
all endpoints verified including account creation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Don Kendall <kendall@donkendall.com>
@dnplkndll
dnplkndll force-pushed the feat/kustomize-kube branch from 662ca3b to 81b22b9 Compare March 22, 2026 16:13
@ArtyomSavchenko

ArtyomSavchenko commented Apr 3, 2026

Copy link
Copy Markdown
Member

Could you please verify that changes in .github/workflows/main.yaml match the Huly selfhost configuration?
https://github.com/hcengineering/huly-selfhost/actions/runs/23407104276/job/68087805849?pr=278 is failed due to this error:

* Connected to huly.example (127.0.0.1) port 80
> POST /_accounts/ HTTP/1.1
> Host: huly.example
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 107
> 
} [107 bytes data]
< HTTP/1.1 308 Permanent Redirect
< Date: Sun, 22 Mar 2026 16:16:24 GMT
< Content-Type: text/html
< Content-Length: 164
< Connection: keep-alive
< Location: https://huly.example/_accounts
< 
{ [164 bytes data]

100   271  100   164  100   107   109k  73087 --:--:-- --:--:-- --:--:--  264k
* Connection #0 to host huly.example left intact
jq: parse error: Invalid numeric literal at line 1, column 7
Error: Process completed with exit code 5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants