New CC token verification mechanism (#3829)

### Summary

This PR introduces a new confidential computing token verification
mechanism to replace the previous job-based verification approach.

Previously, the verification mechanism was tied to specific jobs, which
required generating a new set of tokens for each new job. This approach
was inefficient and error-prone. The new mechanism provides a
persistent, cross-site token validation system that ensures secure and
consistent communication between components.

### Implementation Details
1. Client Registration

When a client sends a registration request to the server:

  - The client includes its token in the request.

  - The server validates the client’s token.

  - The server responds with its own token.

  - The client validates the server’s token.

2. Periodic Cross-Site Validation

Each site (server or client) periodically triggers a cross-site token
validation event (e.g., every 5–10 minutes):

  - The initiating site (e.g., siteA) starts the validation event.

  - All sites, including siteA, generate new tokens for this event.

  - siteA validates tokens from all participating sites.

3. Failure Handling

If any token validation fails:

The affected site will shut itself down.

Optionally, it may attempt to trigger a system-wide shutdown to prevent
inconsistent states.

4. Benefits

  - Removes dependency on per-job token generation.

- Enables periodic, automated validation to detect and isolate
compromised sites.

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Quick tests passed locally by running `./runtest.sh`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated.

Author: YuanTingHsieh

docs/user_guide/confidential_computing/attestation.rst

@@ -76,43 +76,61 @@ You can configure the CC attestation components during the provision step. See t
 Runtime Behavior
 ================
 
-The attestation workflow consists of several phases during job lifecycle:
+
+The Confidential Computing (CC) attestation workflow establishes continuous, system-wide trust between all federated learning participants.
 
 1. System Bootstrap
 -------------------
 
-When a participant (server or client) starts up, the ``CCManager`` responds to the ``EventType.SYSTEM_BOOTSTRAP`` event by generating its own CC token using the configured ``CCAuthorizers``.
+When the system starts, each CC-enabled site (server or client) initializes its confidential computing components and generates a CC token that identifies its trusted environment.
+
 
 2. Client Registration
 ----------------------
 
-When a client registers with the server, it includes its CC token as part of the registration data. If the registration is successful, the server collects and stores the client's CC token.
+During client registration:
+
+    - The client sends its token to the server.
+
+    - The server verifies the client’s token and responds with its own.
+
+    - The client then validates the server’s token.
+
+This mutual verification ensures both sides trust each other before participating in any job.
 
-The server's ``CCManager`` maintains both its own CC token and the tokens of all registered clients.
 
-3. Job Deployment Verification
--------------------------------
+3. Continuous Cross-Site Validation
+-----------------------------------
 
-Once a job is submitted and scheduled for deployment, the server verifies the CC tokens of the clients listed in the job's deployment map, using its own security policy.
+After startup, all sites periodically perform cross-site token validation:
 
-If all client tokens in the deployment map pass verification, the server sends the verified tokens to those clients for peer verification.
+Each site generates new CC tokens at regular intervals.
 
-4. Peer Verification
---------------------
+Sites exchange tokens through a secure communication channel.
 
-Each client evaluates the received CC tokens (including the server's token and other clients' tokens) against its own security policy to decide whether it trusts the other participants.
+Every participant validates the tokens of all others.
 
-Based on this evaluation, the client may choose to accept or reject participation in the job.
+If any CC-enabled site fails token validation, the system will shut down to maintain a trusted environment.
+Sites that are not CC-enabled are skipped during attestation checks.
 
-If a client declines to join the job, the server excludes it from deployment.
 
-5. Job Scheduling
+4. Job Scheduling
 -----------------
 
-Finally, the server's job scheduler determines whether the job has sufficient resources to proceed. It finalizes the job's status based on:
+Before jobs run, the server confirms that all CC-enabled participants have valid, verified tokens.
+If validation fails, the system shuts down to prevent untrusted operation.
+Jobs involving untrusted code (for example, BYOC) are blocked in CC mode.
+
+5. Summary
+----------
+
+The attestation workflow provides:
+
+    - Continuous, system-wide token verification
+
+    - Mutual trust between server and clients
+
+    - Automatic shutdown on attestation failure
 
-- Resource availability
-- Number of participants that passed verification
-- Any defined retry policies
+This ensures that all confidential computing participants operate only within secure and attested environments.
 
-This multi-stage verification process ensures that all participants in a federated learning job operate in trusted, attested environments.

docs/user_guide/confidential_computing/cc_deployment_guide.rst

@@ -23,6 +23,7 @@ This guide covers the following deployment configuration:
 
 Prerequisites
 =============
+For a complete and thorough setup guide covering Hardware IT, Host OS Administration, and VM Administration, please refer to [NVIDIA's Deployment Guide for SecureAI](https://docs.nvidia.com/cc-deployment-guide-snp.pdf) 
 
 Hardware Requirements
 ---------------------

examples/advanced/cc_provision/README.md

@@ -140,3 +140,9 @@ tensorflow
 safetensors
 nv_attestation_sdk
 ```
+
+## 8. Notes on re-building initramfs with CVM image builder
+
+1. Before re-building the initramfs for the CVM, remove the ``initrd.img`` file from the ``image_builder/base_images/`` directory.
+   This ensures the Image Builder regenerates a fresh initramfs during the build process.
+

examples/advanced/cc_provision/docker/Dockerfile

@@ -11,5 +11,5 @@ COPY code/ /local/custom
 COPY requirements.txt .
 RUN pip install -r requirements.txt
 
-ENTRYPOINT ["/user_config/nvflare/startup/sub_start.sh", "--verify"]
+ENTRYPOINT ["/user_config/nvflare/startup/sub_start.sh", "--once", "--verify"]
 

examples/advanced/cc_provision/jobs/hello-pt_cifar10_fedavg/app_server/config/config_fed_server.json

@@ -5,7 +5,7 @@
             "id": "controller",
             "path": "nvflare.app_common.workflows.fedavg.FedAvg",
             "args": {
-                "num_clients": 1,
+                "num_clients": 2,
                 "num_rounds": 2
             }
         }

examples/advanced/cc_provision/jobs/hello-pt_cifar10_fedavg/app_site-1/config/config_fed_client.json

@@ -8,7 +8,7 @@
             "executor": {
                 "path": "nvflare.app_opt.pt.in_process_client_api_executor.PTInProcessClientAPIExecutor",
                 "args": {
-                    "task_script_path": "/local/custom/hello-pt_cifar10_fl.py"
+                    "task_script_path": "/local/custom/client.py"
                 }
             }
         }

examples/advanced/cc_provision/jobs/hello-pt_cifar10_fedavg/meta.json

@@ -1,13 +1,14 @@
 {
     "name": "hello-pt_cifar10_fedavg",
     "resource_spec": {},
-    "min_clients": 1,
+    "min_clients": 2,
     "deploy_map": {
         "app_server": [
             "server"
         ],
         "app_site-1": [
-            "site-1"
+            "site-1",
+            "site-2"
         ]
     }
-}
+}

examples/advanced/cc_provision/project.yml

@@ -40,15 +40,6 @@ builders:
       # if not set, no app_validator is included in fed_server.json
       # app_validator: PATH_TO_YOUR_OWN_APP_VALIDATOR
 
-      # download_job_url is set to http://download.server.com/ as default in fed_server.json.  You can override this
-      # to different url.
-      # download_job_url: http://download.server.com/
-
   - path: nvflare.lighter.impl.cert.CertBuilder
   - path: nvflare.lighter.cc_provision.impl.cc.CCBuilder
   - path: nvflare.lighter.impl.signature.SignatureBuilder
-packager:
-  path: nvflare.lighter.cc_provision.impl.onprem_packager.OnPremPackager
-  args:
-    # this needs to be replace with the real path of the image build scripts
-    build_image_cmd: build_cvm_image.sh

nvflare/apis/fl_constant.py

@@ -73,6 +73,7 @@ class ReservedKey(object):
     WORKSPACE_ROOT = "__workspace_root__"
     APP_ROOT = "__app_root__"
     CLIENT_NAME = "__client_name__"
+    CLIENT_TYPE = "__client_type__"
     TASK_NAME = "__task_name__"
     TASK_DATA = "__task_data__"
     TASK_RESULT = "__task_result__"
@@ -125,6 +126,7 @@ class FLContextKey(object):
     EVENT_SCOPE = ReservedKey.EVENT_SCOPE
     EXCEPTIONS = ReservedKey.EXCEPTIONS
     CLIENT_NAME = ReservedKey.CLIENT_NAME
+    CLIENT_TYPE = ReservedKey.CLIENT_TYPE
     WORKSPACE_ROOT = ReservedKey.WORKSPACE_ROOT
     CURRENT_RUN = ReservedKey.RUN_NUM
     APP_ROOT = ReservedKey.APP_ROOT