Skip to content

docs: clarify how to set backup encryption key#3675

Open
netrome wants to merge 1 commit into
mainfrom
3674-update-tdx-node-migration-docs-after-recent-migration-hiccups
Open

docs: clarify how to set backup encryption key#3675
netrome wants to merge 1 commit into
mainfrom
3674-update-tdx-node-migration-docs-after-recent-migration-hiccups

Conversation

@netrome

@netrome netrome commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

closes #3674

Copilot AI review requested due to automatic review settings June 25, 2026 11:19
@netrome netrome linked an issue Jun 25, 2026 that may be closed by this pull request
@netrome netrome self-assigned this Jun 25, 2026
@claude

claude Bot commented Jun 25, 2026

Copy link
Copy Markdown

Pull request overview

Documentation-only PR clarifying how to configure the backup encryption key during node migration, especially the divergence between TDX/CVM nodes (set in user-config.toml) and non-TEE nodes (set via MPC_BACKUP_ENCRYPTION_KEY_HEX env var). Closes #3674.

Changes:

  • Adds per-deployment-type guidance in Step 3 of the migration guide, distinguishing TDX/CVM placement (TOML) from non-TEE placement (env var).
  • Adds a callout that auto-generated keys cannot be recovered from a CVM (no filesystem access), making such a node impossible to migrate off.
  • Expands Step 5 with a TOML snippet for the new node, a redeploy reminder, and a warning about the silent auto-generation behavior.
  • Documents secrets.secret_store_key_hex and secrets.backup_encryption_key_hex in the TDX external guide's configurable-variables list.

Reviewed changes

Per-file summary
File Description
docs/node-migration-guide.md Steps 3 and 5 rewritten to distinguish CVM vs non-TEE configuration; adds explicit warnings about unrecoverable auto-generated keys inside CVMs and the need to redeploy.
docs/running-an-mpc-node-in-tdx-external-guide.md Adds bullet entries for secrets.secret_store_key_hex and secrets.backup_encryption_key_hex to the configurable-variables list.

Findings

Verified the substantive claims against the codebase:

  • [mpc_node_config.secrets].backup_encryption_key_hex field name and placement matches crates/node-config/src/start.rs:121-129 (SecretsStartConfig).
  • 32 bytes / 64 hex chars and "fails to start on malformed" matches crates/node/src/config.rs:180-188 (hex_to_binary_key) and the AesKey256 type at crates/node/src/config.rs:177.
  • Silent auto-generation when unset matches crates/node/src/run.rs:118-120 (writes backup_encryption_key.hex to home_dir).
  • "No env-var pathway into the CVM" is consistent with deployment/cvm-deployment/deploy-launcher.sh and default.env — neither references MPC_BACKUP_ENCRYPTION_KEY_HEX, and only user-config.toml is mounted into the CVM (USER_CONFIG_FILE_PATH).
  • Template placeholder behavior matches deployment/cvm-deployment/user-config.toml:85-88 (non-hex placeholder; clearly documented in template header).
  • secret_store_key_hex 16-byte / 32-hex-char framing matches crates/node/src/cli.rs:78-81 and the template comment.
  • Step-anchor cross-references (step-5-prepare-the-new-node, step-7-transfer-keyshares-to-new-node) match the headings in the file.

No blocking issues — the documentation is accurate and resolves a real operator footgun.

✅ Approved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates operator documentation to reduce migration failures by clarifying how the backup encryption key must be configured—especially for TDX/CVM deployments where environment variables and filesystem access differ from non-TEE nodes.

Changes:

  • Document secrets.secret_store_key_hex and secrets.backup_encryption_key_hex requirements in the TDX external guide.
  • Update the node migration guide to distinguish TDX/CVM vs non-TEE key configuration paths and add stronger warnings about CVM unrecoverable auto-generated keys.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
docs/running-an-mpc-node-in-tdx-external-guide.md Adds explicit guidance for required secret keys in user-config.toml, including generation commands and CVM-specific constraints.
docs/node-migration-guide.md Clarifies how/where to set the migration transport encryption key for TDX/CVM vs non-TEE nodes; adds CVM-specific warnings and redeploy note.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

### Retrieve a key from an existing node.

**Note:** If your node has been running without the `MPC_BACKUP_ENCRYPTION_KEY_HEX` environment variable set, the node automatically generates an encryption key and stores it in a file called `backup_encryption_key.hex` in your `$MPC_HOME_DIR` directory. You can retrieve it with:
**Note:** If your node has been running without an encryption key configured, the node automatically generates one and stores it in a file called `backup_encryption_key.hex` in your home directory. On a **non-TEE** node you can retrieve it with:
Comment on lines +139 to +143
**How you configure the key on a node depends on the deployment type** (see [Step 5](#step-5-prepare-the-new-node) for the exact placement):
- **TDX / CVM node:** set `backup_encryption_key_hex` under `[mpc_node_config.secrets]` in `user-config.toml`. There is **no** `MPC_BACKUP_ENCRYPTION_KEY_HEX` environment-variable pathway into a CVM — the `.env` used by `deploy-launcher.sh` is not read for this key.
- **Non-TEE node:** set the `MPC_BACKUP_ENCRYPTION_KEY_HEX` environment variable.

**Important:** The backup encryption key must be the same between the backup-cli and the node it is currently communicating with (e.g., the old node when running `get-keyshares`, and the new node when running `put-keyshares`). The node requires exactly 64 hex characters (32 bytes) with no surrounding whitespace, and **fails to start** if the configured value is malformed.

@gilcu3 gilcu3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!


Copy this key and set it as the `BACKUP_ENCRYPTION_KEY` environment variable for the backup-cli when running `get-keyshares`.

> **⚠️ TDX / CVM source nodes:** you **cannot** retrieve an auto-generated key this way — there is no filesystem access into the CVM. If a CVM node was deployed without `backup_encryption_key_hex` set in its `user-config.toml`, the key it generated internally is unrecoverable and the node **cannot be migrated off**. Always set `backup_encryption_key_hex` explicitly when deploying a CVM node so you know the key. (The shipped `user-config.toml` template makes this field a required, non-hex placeholder so a forgotten value fails loudly at startup rather than silently auto-generating.)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh this seems really dangerous. Shouldn't we fail instead if no key is provided?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that would be better, iirc this was just introduced for backwards compatibility when we introduced the field for the 3.0.0 release since we could not add new env variables at that time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update TDX node & migration docs after recent migration hiccups

3 participants