Move Staging to run on OVH with k3s #3476

yuvipanda · 2025-10-20T23:48:58Z

Changes existing OVH terraform code to instead be able to support OVH registry only, as we use k3s for kubernetes
Setup a new kubernetes k3s cluster, following our existing k3s docs, for running the staging cluster
Deploy to this new staging cluster
Have the staging cluster's launch events come to the same analytics pipeline as our prod cluster for now, instead of setting up two separate ones as we have so far. This adds a tiny amount of noise to our pipeline (potentially - i see it's a different log name) but is worth it
Turn off the event publisher related code for now. This means we'll only have this in prod.
Turn off matomo, as we use plausible now in prod instances.
Setup k3s on a manually created VM on OVHCloud. I added an ssh key with access to the ubuntu user (which has full sudo) to this PR as a secret.
DNS is currently set to https://staging.mybinder.2i2c.cloud/ because i no longer have access to mybinder.org DNS. It's on cloudflare but it seems to require 2fa now and i don't have access. Maybe @minrk does?

TODO

Fix the DNS to point to staging.mybinder.org instead of the 2i2c.cloud domains
Test that this works in our CI
Rip out the existing staging cluster and everything with it, say 'thank you' and gently bury it with respect

See #3477 (keep it open until we've cleaned up the old staging)

manics · 2025-10-25T12:54:45Z

The mybinder.org domain is managed by Jupyter: jupyterhub/team-compass#424 (comment)

yuvipanda · 2025-10-25T16:21:04Z

i no longer have access to that cloudflare account. It seems to require 2fa and i don’t have 2fa for it :(

manics · 2025-11-20T15:14:27Z

@Carreau has very kindly updated the DNS for staging.mybinder.org to point to this new host.
Next step is to update this PR with the new hostnames for the K8S ingress and certificates

With our k3s based setups, terraform on OVH is primarily managing *just* the harbor setup, removing all the k8s stuff. I'd like for us to use one harbor across multiple smaller VPS instances.

- Use 2i2c.cloud as domain because I don't have access to mybinder.org domain yet

for more information, see https://pre-commit.ci

manics · 2025-11-22T16:40:49Z

/test-this-pr

jupyterhub-bot · 2025-11-22T16:41:11Z

This Pull Request is now being tested 🎉 See the test progress in GitHub Actions.

jupyterhub-bot · 2025-11-22T16:53:24Z

Job status: failure
Branch 'test-this-pr/3476' has been deleted

manics · 2025-11-22T17:26:42Z

The deployment failed because events-archiver-secrets isn't available for staging. It's not configured on our current staging, so I'll revert the extraVolumes config added in 4d2594f

Edit: reverted the revert in a subsequent commit, it's needed

manics · 2025-11-22T17:28:19Z

/test-this-pr

jupyterhub-bot · 2025-11-22T17:28:37Z

This Pull Request is now being tested 🎉 See the test progress in GitHub Actions.

jupyterhub-bot · 2025-11-22T17:40:27Z

Job status: failure
Branch 'test-this-pr/3476' has been deleted

manics · 2025-11-22T17:40:56Z

/test-this-pr

jupyterhub-bot · 2025-11-22T17:41:15Z

This Pull Request is now being tested 🎉 See the test progress in GitHub Actions.

This reverts commit 4737811. Revert "staging: revert addition of `events-archiver-secrets` config" This reverts commit 38bec04.

jupyterhub-bot · 2025-11-22T17:59:00Z

Job status: failure
Branch 'test-this-pr/3476' has been deleted

staging on gke presumable has automatic persmissions to push events to a staging location? OVH won't, but to unblock this push events to prod for now

manics · 2025-11-22T18:11:17Z

/test-this-pr

jupyterhub-bot · 2025-11-22T18:11:35Z

This Pull Request is now being tested 🎉 See the test progress in GitHub Actions.

manics · 2025-11-22T18:37:25Z

@yuvipanda do you have some unpushed changes? secrets/config/staging.yaml is missing the updated OVH registry credentials

jupyterhub-bot · 2025-11-22T18:39:28Z

Job status: failure
Branch 'test-this-pr/3476' has been deleted

manics · 2025-11-23T11:26:40Z

I've updated the ovh registry secrets using the terraform outputs from

mybinder.org-deploy/terraform/ovh/harbor.tf

Lines 104 to 121 in 02b3755

    
           output "registry_builder_name" { 
        
             value     = harbor_robot_account.builder.full_name 
        
             sensitive = true 
        
           } 
        
           output "registry_builder_token" { 
        
             value     = harbor_robot_account.builder.secret 
        
             sensitive = true 
        
           } 
        
           output "registry_user_puller_name" { 
        
             value     = harbor_robot_account.user-puller.full_name 
        
             sensitive = true 
        
           } 
        
           output "registry_user_puller_token" { 
        
             value     = harbor_robot_account.user-puller.secret 
        
             sensitive = true 
        
           }

/test-this-pr

jupyterhub-bot · 2025-11-23T11:26:59Z

This Pull Request is now being tested 🎉 See the test progress in GitHub Actions.

jupyterhub-bot · 2025-11-23T11:31:09Z

Job status: success
Branch 'test-this-pr/3476' has been deleted

manics · 2025-11-23T12:17:04Z

test-this-pr has successfully deployed this to staging! Be aware the node is quite small, I suspect this causes more complicated images to fail e.g. https://github.com/manics/jupyter-desktop-mate/ will build and launch, but the desktop hangs.

I think another review would be good, then merge!

yuvipanda · 2025-11-24T20:52:12Z

Thank you for taking this through, @manics! I think you should just merge and see how it goes :D

manics · 2025-11-25T14:48:45Z

@yuvipanda Thanks! YOLO....

yuvipanda · 2025-11-25T16:28:27Z

@manics how did this go?

manics · 2025-11-25T17:06:19Z

Seems to have worked 😀

jupyterhub-pr-triage-board-bot bot added this to PR triage (experimental) Oct 21, 2025

minrk approved these changes Oct 21, 2025

View reviewed changes

yuvipanda force-pushed the ovh branch 2 times, most recently from 91ec4fd to fa19883 Compare October 21, 2025 23:48

yuvipanda changed the title ~~Revamp OVH terraform to support k3s based setups~~ Move Staging to run on OVH with k3s Oct 21, 2025

yuvipanda mentioned this pull request Oct 22, 2025

Move staging to OVH #3477

Open

5 tasks

yuvipanda mentioned this pull request Oct 28, 2025

[Timebox: 12h] Add support for running a single node cluster on OVH 2i2c-org/infrastructure#7035

Open

3 tasks

jupyterhub-pr-triage-board-bot bot removed this from PR triage (experimental) Nov 5, 2025

jupyterhub-pr-triage-board-bot bot added this to PR triage (experimental) Nov 5, 2025

yuvipanda and others added 6 commits November 20, 2025 10:17

Revamp OVH terraform to support k3s based setups

0d4a675

With our k3s based setups, terraform on OVH is primarily managing *just* the harbor setup, removing all the k8s stuff. I'd like for us to use one harbor across multiple smaller VPS instances.

Turn locals into vars, and add a staging setup

47fd1eb

Run staging off a k3s cluster

4d2594f

- Use 2i2c.cloud as domain because I don't have access to mybinder.org domain yet

[pre-commit.ci] auto fixes from pre-commit.com hooks

29aeef2

for more information, see https://pre-commit.ci

Turn off matomo

0144d75

Fix grafana on staging

c21fc57

yuvipanda force-pushed the ovh branch from 8a8d672 to c21fc57 Compare November 20, 2025 18:18

Carreau mentioned this pull request Nov 20, 2025

Need access to cloudflare 2FA, please share it via 1password. jupyter-governance/ec-team-compass#164

Open

Replace staging.mybinder.2i2c.cloud with staging.mybinder.org

47e20f4

staging: revert addition of events-archiver-secrets config

38bec04

Add GOOGLE_APPLICATION_CREDENTIALS back

4737811

Revert "Add GOOGLE_APPLICATION_CREDENTIALS back"

c90b428

This reverts commit 4737811. Revert "staging: revert addition of `events-archiver-secrets` config" This reverts commit 38bec04.

Allow staging to push events to prod

21e4d9d

staging on gke presumable has automatic persmissions to push events to a staging location? OVH won't, but to unblock this push events to prod for now

manics added 2 commits November 23, 2025 11:19

ovh terraform s32 backend: update for 1.6.0

cf99693

Update staging registry secrets

02b3755

manics requested a review from minrk November 23, 2025 12:17

ovh terraform lock.hcl

6f909cb

manics merged commit d12eb26 into jupyterhub:main Nov 25, 2025
8 checks passed

github-project-automation bot moved this to Done in PR triage (experimental) Nov 25, 2025

jupyterhub-pr-triage-board-bot bot removed this from PR triage (experimental) Nov 25, 2025

Move Staging to run on OVH with k3s #3476

Move Staging to run on OVH with k3s #3476

Uh oh!

Conversation

yuvipanda commented Oct 20, 2025 • edited by manics Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

manics commented Oct 25, 2025

Uh oh!

yuvipanda commented Oct 25, 2025

Uh oh!

manics commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manics commented Nov 22, 2025

Uh oh!

jupyterhub-bot commented Nov 22, 2025

Uh oh!

jupyterhub-bot commented Nov 22, 2025

Uh oh!

manics commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manics commented Nov 22, 2025

Uh oh!

jupyterhub-bot commented Nov 22, 2025

Uh oh!

jupyterhub-bot commented Nov 22, 2025

Uh oh!

manics commented Nov 22, 2025

Uh oh!

jupyterhub-bot commented Nov 22, 2025

Uh oh!

jupyterhub-bot commented Nov 22, 2025

Uh oh!

manics commented Nov 22, 2025

Uh oh!

jupyterhub-bot commented Nov 22, 2025

Uh oh!

manics commented Nov 22, 2025

Uh oh!

jupyterhub-bot commented Nov 22, 2025

Uh oh!

manics commented Nov 23, 2025

Uh oh!

jupyterhub-bot commented Nov 23, 2025

Uh oh!

jupyterhub-bot commented Nov 23, 2025

Uh oh!

manics commented Nov 23, 2025

Uh oh!

yuvipanda commented Nov 24, 2025

Uh oh!

manics commented Nov 25, 2025

Uh oh!

Uh oh!

yuvipanda commented Nov 25, 2025

Uh oh!

manics commented Nov 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yuvipanda commented Oct 20, 2025 •

edited by manics

Loading

manics commented Nov 20, 2025 •

edited

Loading

manics commented Nov 22, 2025 •

edited

Loading