Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 15 additions & 12 deletions .github/workflows/main.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
[ $(uname -m) = aarch64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.26.0/kind-linux-arm64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

- name: "Setup k8s cluster"
run: |
cat <<EOF | kind create cluster --wait=3m --config=-
Expand Down Expand Up @@ -62,22 +62,25 @@ jobs:
working-directory: ${{ github.workspace }}/src/github.com/${{ github.repository }}/kube
run: |
echo 127.0.0.1 huly.example | sudo tee -a /etc/hosts
echo 127.0.0.1 account.huly.example | sudo tee -a /etc/hosts
echo 127.0.0.1 transactor.huly.example | sudo tee -a /etc/hosts
echo 127.0.0.1 collaborator.huly.example | sudo tee -a /etc/hosts
kubectl apply -R -f .
kubectl wait --for=condition=Available deployment/cockroachdb-single-node --timeout 3m
kubectl wait --for=condition=Available deployment/transactor --timeout 3m
kubectl delete pod -l app=account
kubectl wait --for=condition=Available deployment/account --timeout 3m
kubectl create namespace huly || true
kubectl -n huly create secret generic huly-secret \
--from-literal=SERVER_SECRET=secret \
--from-literal=STORAGE_CONFIG='minio|minio?accessKey=minioadmin&secretKey=minioadmin' \
--from-literal=REDPANDA_SUPERUSER_PASSWORD=redpanda \
--from-literal=CR_DB_URL='postgres://root@cockroach:26257/defaultdb?sslmode=disable'
kubectl apply -k overlays/example -n huly
kubectl -n huly wait --for=condition=Available deployment/cockroach --timeout 3m
kubectl -n huly wait --for=condition=Available deployment/transactor --timeout 3m
kubectl -n huly delete pod -l app.kubernetes.io/name=account
kubectl -n huly wait --for=condition=Available deployment/account --timeout 3m
- name: "Check login"
run: |
token=$(curl -v -H 'Content-Type: application/json' \
-d '{"method": "signUp","params": {"email": "user3","password": "1234","firstName": "Test","lastName": "User"}}' \
-X POST \
http://account.huly.example/ | jq -r '.result.token')
http://huly.example/_accounts/ | jq -r '.result.token')

curl -v http://account.huly.example/ \
curl -v http://huly.example/_accounts/ \
-X POST \
-d '{"method":"getUserWorkspaces","params":[]}' \
-H 'Content-Type: application/json' \
Expand All @@ -87,5 +90,5 @@ jobs:
- name: Cleanup resources
if: ${{ success() || failure() || cancelled() }}
run: |
kubectl describe pods
kubectl -n huly describe pods
kind delete cluster
3 changes: 3 additions & 0 deletions helm/huly/templates/front/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ spec:
{{- include "huly.envConfig" (dict "name" "LOVE_ENDPOINT" "key" "LOVE_ENDPOINT" "root" .) | nindent 12 }}
{{- if .Values.githubIntegration.enabled }}
{{- include "huly.envConfig" (dict "name" "GITHUB_URL" "key" "GITHUB_URL" "root" .) | nindent 12 }}
{{- include "huly.envSecret" (dict "name" "GITHUB_CLIENTID" "key" "GITHUB_APP_CLIENT_ID" "root" .) | nindent 12 }}
- name: GITHUB_APP
value: {{ .Values.githubIntegration.appSlug | quote }}
{{- end }}
{{- if .Values.aibot.enabled }}
{{- include "huly.envConfig" (dict "name" "AI_URL" "key" "AI_URL" "root" .) | nindent 12 }}
Expand Down
2 changes: 2 additions & 0 deletions helm/huly/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,8 @@ githubIntegration:
replicas: 1
# Bot name shown on GitHub (must match the GitHub App's slug + [bot])
botName: ""
# GitHub App slug (from the app URL: github.com/apps/<slug>)
appSlug: ""
# GitHub App credentials (stored in the shared secret)
appId: ""
clientId: ""
Expand Down
101 changes: 0 additions & 101 deletions kube/QUICKSTART.md

This file was deleted.

198 changes: 185 additions & 13 deletions kube/README.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,198 @@
# Huly Kubernetes Deployment
# Huly on Kubernetes (Kustomize)

This folder contains a sample configuration for Huly Kubernetes deployment.
Deploy [Huly](https://huly.io) to any Kubernetes cluster using Kustomize.

## Requires
## Architecture

Requires a working kubernetes cluster with min one node. Each node should have at least 2 vCPUs and 8 GB RAM.
```
┌─────────────────────────────────────────────┐
│ nginx ingress │
│ / → front:8080 │
│ /_accounts → account:3000 │
│ /_transactor → transactor:3333 (WS) │
│ /_collaborator → collaborator:3078 (WS) │
│ /_rekoni → rekoni:4004 │
│ /_stats → stats:4900 │
└────────────────────┬────────────────────────┘
┌──────────┬──────────┬────────┴───────┬──────────┬──────────┐
│ │ │ │ │ │
front account transactor collaborator rekoni stats
│ │ │ │
│ └────┬─────┘ │
│ │ │
│ ┌──────────┴──────────┐ │
│ │ │ │
│ cockroach redpanda │
│ │ │ │
│ ├── workspace │ │
│ ├── fulltext ─────────┤ │
│ │ │ │ │
│ │ elasticsearch │ │
│ │ │ │
└────┴─────── minio ──────┴──────────┘
```

> **Note:** Elasticsearch 7.14.2 is used (not OpenSearch) because Huly's
> fulltext service bundles `elasticsearch-js` which rejects non-Elasticsearch backends.

If you don't have any k8s cluster, consider using the [kind setup](QUICKSTART.md).
## Directory layout

## Check and update configuration
```
kube/
base/
config/ # ConfigMap with URLs and app settings
infra/ # CockroachDB, Redpanda, Elasticsearch, MinIO
app/ # 8 Huly services
ingress/ # 6 nginx Ingress resources (path-based routing)
overlays/
example/ # Copy this, set your domain + image tags
```

Huly deployment configuration is located in [config.yaml](config/config.yaml) and [secret.yaml](config/secret.yaml) files.
The sample configuration assume that Huly is available on huly.example hostname with dedicated hostname per service.
## Prerequisites

## Deploy Huly to Kubernetes
- Kubernetes cluster with:
- [ingress-nginx](https://kubernetes.github.io/ingress-nginx/)
- [cert-manager](https://cert-manager.io/) with a `ClusterIssuer` named `letsencrypt-prod`
- `kubectl` configured for your cluster
- DNS A record pointing your domain to the ingress load-balancer IP
- (Optional) Google OAuth credentials for login

Deploy Huly with `kubectl`.
## Quick start

```bash
kubectl create namespace huly-v7
# 1. Create your overlay (copy the example)
cp -r kube/overlays/example kube/overlays/my-site
# Edit kube/overlays/my-site/kustomization.yaml:
# - Uncomment the patches section
# - Replace huly.yourdomain.com with your actual domain
# - Replace you@yourdomain.com with your admin email
# - Pin image tags to a specific version (e.g., v0.7.382)

# 2. Create namespace
kubectl create namespace huly

kubectl apply -R -f . --namespace huly-v7
# 3. Create secrets
SERVER_SECRET=$(openssl rand -hex 32)
RP_PASS=$(openssl rand -hex 16)

kubectl -n huly create secret generic huly-secret \
--from-literal=SERVER_SECRET="$SERVER_SECRET" \
--from-literal=STORAGE_CONFIG='minio|minio?accessKey=minioadmin&secretKey=minioadmin' \
--from-literal=REDPANDA_SUPERUSER_PASSWORD="$RP_PASS" \
--from-literal=CR_DB_URL='postgres://root@cockroach:26257/defaultdb?sslmode=disable' \
--from-literal=GOOGLE_CLIENT_ID='<your-google-client-id>' \
--from-literal=GOOGLE_CLIENT_SECRET='<your-google-client-secret>'

# 4. Deploy
kubectl apply -k kube/overlays/my-site/

# 5. Watch pods come up
kubectl -n huly get pods -w
```

Now, launch your web browser and enjoy Huly!
## One-liner deploy

If you have `envsubst` available, you can deploy with a single command after
creating the namespace and secret above:

```bash
HULY_DOMAIN=huly.yourdomain.com \
ADMIN_EMAIL=you@yourdomain.com \
HULY_VERSION=v0.7.382 \
bash kube/deploy.sh
```

> See the [deploy.sh script](#deploy-script) section below for details.

## Authentication

### Google OAuth

1. Go to [Google Cloud Console](https://console.cloud.google.com/apis/credentials)
2. Create an OAuth 2.0 Client ID (Web application)
3. Add authorized redirect URI: `https://<your-domain>/_accounts/auth/google/callback`
4. Set `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET` in `huly-secret`

### GitHub OAuth (optional)

1. Create a GitHub OAuth App at https://github.com/settings/developers
2. Set callback URL: `https://<your-domain>/_accounts/auth/github/callback`
3. Add `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` to `huly-secret`

### OpenID Connect (optional)

Add the following keys to `huly-secret`:
- `OPENID_CLIENT_ID`
- `OPENID_CLIENT_SECRET`
- `OPENID_ISSUER` (e.g. `https://accounts.google.com`)

### Admin bootstrap

The first user to sign up creates their workspace and becomes the workspace owner.
Set `ADMIN_EMAILS` in the ConfigMap to grant system-wide admin privileges (comma-separated).

### Restricting sign-ups

After creating admin accounts, add `DISABLE_SIGNUP=true` to the account and
front deployments. New users can then only join via workspace invites.

## Upgrading

Edit your overlay's `kustomization.yaml` — change all `newTag` values, then:

```bash
kubectl apply -k kube/overlays/my-site/
```

## Verify

```bash
# All 13 pods should be Running (including kvs)
kubectl -n huly get pods

# Front responds
curl -I https://<your-domain>/

# Account responds (405 = healthy, POST-only API)
curl -I https://<your-domain>/_accounts

# CockroachDB admin UI (port-forward)
kubectl -n huly port-forward svc/cockroach 8080:8080
```

## Troubleshooting

| Symptom | Check |
|---------|-------|
| Pod stuck in `Init` | Infra not ready — `kubectl -n huly logs <pod> -c wait-cockroach` |
| Account CrashLoop | DB auth — `kubectl -n huly logs deploy/account` |
| Front CrashLoop | Missing env — check logs for `please provide <var>` |
| Redpanda won't start | Needs ≥1Gi memory limit |
| Elasticsearch permission denied | Pod spec needs `securityContext.fsGroup: 1000` |
| Fulltext `ProductNotSupportedError` | Must use ES 7.14.2, not OpenSearch |
| WebSocket 502 | Check ingress `proxy-read-timeout: "3600"` annotation |

## Services

| Service | Port | Protocol | Needs DB | Needs Redpanda |
|---------|------|----------|----------|----------------|
| front | 8080 | HTTP | - | - |
| account | 3000 | HTTP | yes | yes |
| transactor | 3333 | WebSocket | yes | yes |
| collaborator | 3078 | WebSocket | - | - |
| workspace | - | background | yes | yes |
| fulltext | 4700 | HTTP | yes | yes |
| kvs | 8094 | HTTP | yes | - |
| rekoni | 4004 | HTTP | - | - |
| stats | 4900 | HTTP | - | - |

## Teardown

```bash
kubectl delete -k kube/overlays/my-site/
kubectl -n huly delete secret huly-secret
kubectl delete namespace huly
# PVCs are retained — delete manually if needed:
kubectl -n huly delete pvc --all
```
Loading
Loading