Update cc provision (#3719)

YuanTingHsieh · web-flow · commit 6cd1816a12be · 2025-09-24T14:51:13.000-07:00
Fix several bugs and add enhancements. ### Description Changes: - Remove using of pre-install and just build a docker image workload with custom code inside - Move startup_kit outside of CVM image and then use @IsaacYangSLA 's verify_startup_kit method - Undo changes to pre-install tool - Move the update of logging config (to /applog) from Packager to Builder so it can be signed by "SignatureBuilder" - Move "SignatureBuilder" to the last item of "project.yml" to sign all contents ### Types of changes  - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Quick tests passed locally by running `./runtest.sh`. - [ ] In-line docstrings updated. - [ ] Documentation updated.
diff --git a/examples/advanced/cc_provision/README.md b/examples/advanced/cc_provision/README.md
@@ -65,6 +65,8 @@ builders:
   - path: nvflare.lighter.cc_provision.impl.cc.CCBuilder
 ```
 
+Note that this CCBuilder needs to be placed **after** the "StaticFileBuilder" and
+**before** the "SignatureBuilder".
 This builder sets up all CC-related configurations and assets.
 
 ## 4. Add the OnPremPackager
diff --git a/examples/advanced/cc_provision/cc_server1.yml b/examples/advanced/cc_provision/cc_server1.yml
@@ -21,7 +21,8 @@ cc_issuers:
   - id: snp_authorizer
     path: nvflare.app_opt.confidential_computing.snp_authorizer.SNPAuthorizer
     token_expiration: 100 # seconds, needs to be less than check_frequency
-    snpguest_binary: "/host/bin/snpguest"
+    args:
+      snpguest_binary: "/host/bin/snpguest"
 
 cc_attestation:
   check_frequency: 120 # seconds
diff --git a/examples/advanced/cc_provision/cc_site-1.yml b/examples/advanced/cc_provision/cc_site-1.yml
@@ -24,7 +24,8 @@ cc_issuers:
   - id: snp_authorizer
     path: nvflare.app_opt.confidential_computing.snp_authorizer.SNPAuthorizer
     token_expiration: 100 # seconds, needs to be less than check_frequency
-    snpguest_binary: "/host/bin/snpguest"
+    args:
+      snpguest_binary: "/host/bin/snpguest"
   - id: gpu_authorizer
     path: nvflare.app_opt.confidential_computing.gpu_authorizer.GPUAuthorizer
     token_expiration: 100 # seconds, needs to be less than check_frequency
diff --git a/examples/advanced/cc_provision/docker/Dockerfile b/examples/advanced/cc_provision/docker/Dockerfile
@@ -4,13 +4,12 @@ FROM ${BASE_IMAGE}
 
 ENV PYTHONDONTWRITEBYTECODE=1
 ENV PIP_NO_CACHE_DIR=1
-ENV PATH="/host/bin:${PATH}"
 
-RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get -y install zip
-RUN pip install -U pip
-RUN pip install git+https://github.com/NVIDIA/NVFlare.git@main
-COPY application_code.zip application_code.zip
-RUN nvflare pre-install install -a application_code.zip
+RUN pip install -U pip &&\
+    pip install nvflare~=2.7.0rc
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+COPY code/ /local/custom
 
 ENTRYPOINT ["/user_config/nvflare/startup/sub_start.sh", "--verify"]
 
diff --git a/examples/advanced/cc_provision/docker/README.md b/examples/advanced/cc_provision/docker/README.md
@@ -1,36 +1,23 @@
-This is a simple example to show how to build NVFlare docker image.
+## Overview
 
+This is a simple example to show how to build an application Docker image for NVFlare.
 
-# NVFlare application code package
-We can pre-install custom codes inside each docker image.
-We utilize our nvflare pre-install command to do that.
+### Prerequisites
 
-First, we need to prepare the `application_code.zip` folder structure:
+Before proceeding, ensure you have Docker installed and running on your machine. Additionally, make sure you have access to the required NVFlare code and resources.
 
-```bash
-application_code_folder
-├── application/                    # optional
-│   └── <job_name>/
-│               ├── meta.json       # job metadata
-│               ├── app_<site>/     # Site custom code
-│                  └── custom/      # Site custom code
-├── application-share/              # Shared resources
-|   └── simple_network.py           # Shared model definition 
-└── requirements.txt       # Python dependencies (optional)
-```
+### Build the Docker Image
+
+We have provided a `Dockerfile` to build the NVFlare Docker image.
 
-We have already prepared application-share folder and requirements.txt in this example.
-We run the following command to create a zip folder so we can use that to build the CVM:
+To build the NVFlare Docker image, run the following command in your terminal:
 
 ```bash
-python -m zipfile -c application_code.zip application_code/*
+./docker_build.sh Dockerfile nvflare-site
 ```
 
-# NVFlare docker file
+Important notes:
+- Code directory: The local ``code`` folder is copied into the Docker image under ``/local/custom``
+- Job configuration: In the [job configuration](../jobs/hello-pt_cifar10_fedavg/app_site-1/config/config_fed_client.json) we configure the task script to be ``/local/custom/hello-pt_cifar10_fl.py``
 
-We have prepared a `Dockerfile`.
-Please run the following command to build the NVFlare docker image:
-
-```bash
-./docker_build.sh Dockerfile nvflare-site 
-```
+Make sure to adjust the paths and filenames in your configurations accordingly.
diff --git a/examples/advanced/cc_provision/docker/code/hello-pt_cifar10_fl.py b/examples/advanced/cc_provision/docker/code/hello-pt_cifar10_fl.py
diff --git a/examples/advanced/cc_provision/docker/code/simple_network.py b/examples/advanced/cc_provision/docker/code/simple_network.py
diff --git a/examples/advanced/cc_provision/docker/requirements.txt b/examples/advanced/cc_provision/docker/requirements.txt
diff --git a/examples/advanced/cc_provision/jobs/hello-pt_cifar10_fedavg/app_site-1/config/config_fed_client.json b/examples/advanced/cc_provision/jobs/hello-pt_cifar10_fedavg/app_site-1/config/config_fed_client.json
@@ -8,7 +8,7 @@
             "executor": {
                 "path": "nvflare.app_opt.pt.in_process_client_api_executor.PTInProcessClientAPIExecutor",
                 "args": {
-                    "task_script_path": "/vault/packages/share/hello-pt_cifar10_fl.py"
+                    "task_script_path": "/local/custom/hello-pt_cifar10_fl.py"
                 }
             }
         }
diff --git a/examples/advanced/cc_provision/project.yml b/examples/advanced/cc_provision/project.yml
@@ -56,12 +56,8 @@ builders:
           sp_end_point: server1:8002:8003
 
   - path: nvflare.lighter.impl.cert.CertBuilder
-  - path: nvflare.lighter.impl.signature.SignatureBuilder
   - path: nvflare.lighter.cc_provision.impl.cc.CCBuilder
-  - path: nvflare.lighter.impl.docker_image_builder.DockerImageBuilder
-    args:
-      base_dockerfile: Dockerfile.base
-      requirement: git+https://github.com/NVIDIA/NVFlare.git@main
+  - path: nvflare.lighter.impl.signature.SignatureBuilder
 packager:
   path: nvflare.lighter.cc_provision.impl.onprem_packager.OnPremPackager
   args:
diff --git a/nvflare/lighter/cc_provision/impl/onprem_cvm.py b/nvflare/lighter/cc_provision/impl/onprem_cvm.py
@@ -17,12 +17,31 @@
 
 from nvflare.lighter import utils
 from nvflare.lighter.cc_provision.cc_constants import CC_AUTHORIZERS_KEY
-from nvflare.lighter.constants import PropKey
+from nvflare.lighter.constants import PropKey, ProvFileName
 from nvflare.lighter.ctx import ProvisionContext
 from nvflare.lighter.entity import Entity, Project
 from nvflare.lighter.spec import Builder
 
 
+def _update_log_filenames(config, new_log_root: str = "/applog"):
+    handlers = config.get("handlers", {})
+    for handler_name, handler_cfg in handlers.items():
+        filename = handler_cfg.get("filename")
+        if filename:
+            handler_cfg["filename"] = os.path.join(new_log_root, filename)
+    return config
+
+
+def _change_log_dir(log_config_path: str):
+    with open(log_config_path, "r") as f:
+        config = json.load(f)
+
+    updated_config = _update_log_filenames(config)
+
+    with open(log_config_path, "w") as f:
+        json.dump(updated_config, f, indent=4)
+
+
 class OnPremCVMBuilder(Builder):
     """Builder for On-premises Confidential Computing VMs."""
 
@@ -56,3 +75,10 @@ def _build_resources(self, entity: Entity, ctx: ProvisionContext):
                 json.dumps({"components": [authorizer]}, indent=2),
                 "t",
             )
+        log_config_path = os.path.join(dest_dir, ProvFileName.LOG_CONFIG_DEFAULT)
+        if not os.path.exists(log_config_path):
+            raise RuntimeError(
+                "The OnPremCVMBuilder requires StaticFileBuilder to run first. "
+                "Please ensure that StaticFileBuilder is included in the 'builder' section of your configuration before OnPremCVMBuilder."
+            )
+        _change_log_dir(log_config_path)
diff --git a/nvflare/lighter/cc_provision/impl/onprem_packager.py b/nvflare/lighter/cc_provision/impl/onprem_packager.py
@@ -12,13 +12,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import json
 import os
 import re
 import shutil
 import subprocess
 import tempfile
-import time
 from pathlib import Path
 
 import yaml
@@ -47,15 +45,6 @@ def _extract_docker_tar_path(output):
     return None
 
 
-def update_log_filenames(config, new_log_root: str = "/applog"):
-    handlers = config.get("handlers", {})
-    for handler_name, handler_cfg in handlers.items():
-        filename = handler_cfg.get("filename")
-        if filename:
-            handler_cfg["filename"] = os.path.join(new_log_root, filename)
-    return config
-
-
 def to_abs_path(yaml_path, file_path):
     """Converts a relative file path to an absolute path based on the directory of the given YAML file.
 
@@ -127,9 +116,6 @@ def _build_cc_image(self, cc_config_yaml: str):
             raise FileNotFoundError(f"Build image command '{build_image_cmd}' not found or is not executable.")
         command = [build_image_cmd, cc_config_yaml]
         output = run_command(command)
-        # TODO: get rid of this buffer time in between each build
-        #   we need it now otherwise the second call to image builder will fail
-        time.sleep(300.0)
         tar_file_path = _extract_cvm_tar_path(output)
         return tar_file_path
 
@@ -145,25 +131,13 @@ def _add_startup_kit_to_cc_config(self, cc_config_path: str, startup_kit_path: s
         with open(cc_config_path, "w") as f:
             yaml.safe_dump(data, f, default_flow_style=False)
 
-    def _change_log_dir(self, log_config_path: str):
-        with open(log_config_path, "r") as f:
-            config = json.load(f)
-
-        updated_config = update_log_filenames(config)
-
-        with open(log_config_path, "w") as f:
-            json.dump(updated_config, f, indent=4)
-
     def _package_for_participant(self, participant: Participant, ctx: ProvisionContext):
         """Package the startup kit for the participant."""
         if not participant.get_prop(self.cc_config_key):
             return
 
         dest_dir = Path(ctx.get_result_location())
 
-        log_config_path = dest_dir / participant.name / "local" / ProvFileName.LOG_CONFIG_DEFAULT
-        self._change_log_dir(log_config_path)
-
         cc_config_yaml = os.path.abspath(participant.get_prop(self.cc_config_key))
         if not os.path.exists(cc_config_yaml):
             raise RuntimeError(f"{cc_config_yaml=} does not exist")
@@ -191,5 +165,11 @@ def _package_for_participant(self, participant: Participant, ctx: ProvisionConte
         shutil.copy(tar_file_path, site_dir / f"{participant.name}.tgz")
 
     def package(self, project: Project, ctx: ProvisionContext):
-        for p in project.get_all_participants():
-            self._package_for_participant(p, ctx)
+        start_all_script = os.path.join(ctx.get_result_location(), ProvFileName.START_ALL_SH)
+        if os.path.exists(start_all_script):
+            os.remove(start_all_script)
+
+        participants = project.get_all_participants()
+
+        for participant in participants:
+            self._package_for_participant(participant, ctx)
diff --git a/nvflare/lighter/constants.py b/nvflare/lighter/constants.py
@@ -208,6 +208,7 @@ class ProvFileName:
     AZURE_START_SH = "azure_start.sh"
     AWS_START_SH = "aws_start.sh"
     EDGE_RESOURCES_JSON = "edge__p_resources.json"
+    START_ALL_SH = "start_all.sh"
 
 
 class CertFileBasename:
diff --git a/nvflare/lighter/impl/static_file.py b/nvflare/lighter/impl/static_file.py
@@ -898,7 +898,7 @@ def _create_start_all(self, project: Project, ctx: ProvisionContext):
         for c in project.get_clients():
             content = self._append(content, c)
 
-        utils.write(os.path.join(ctx.get_wip_dir(), "start_all.sh"), content, "t", exe=True)
+        utils.write(os.path.join(ctx.get_wip_dir(), ProvFileName.START_ALL_SH), content, "t", exe=True)
 
 
 def _remove_undefined_port(section: str) -> str:
diff --git a/nvflare/tool/code_pre_installer/install.py b/nvflare/tool/code_pre_installer/install.py
@@ -42,7 +42,7 @@ def define_pre_install_parser(cmd_name: str, sub_cmd):
         default=DEFAULT_APPLICATION_INSTALL_DIR,
         help=f"Installation prefix (default: {DEFAULT_APPLICATION_INSTALL_DIR})",
     )
-    parser.add_argument("-s", "--site-name", required=False, default="", help="Target site name (e.g., site-1, server)")
+    parser.add_argument("-s", "--site-name", required=True, help="Target site name (e.g., site-1, server)")
     parser.add_argument(
         "-ts",
         "--target_shared_dir",
diff --git a/tests/unit_test/lighter/cc_provision/impl/test_onprem_cvm.py b/tests/unit_test/lighter/cc_provision/impl/test_onprem_cvm.py
@@ -66,8 +66,11 @@ class TestOnPremCVMBuilder:
     """Test suite for OnPremCVMBuilder."""
 
     @patch("nvflare.lighter.utils.write")
-    def test_build_resources(self, mock_write, builder, basic_project, ctx):
+    @patch("os.path.exists")
+    @patch("nvflare.lighter.cc_provision.impl.onprem_cvm._change_log_dir")
+    def test_build_resources(self, mock_change, mock_exists, mock_write, builder, basic_project, ctx):
         """Test building resources for an entity."""
+        mock_exists.return_value = True
         server = basic_project.get_server()
         ctx[CC_AUTHORIZERS_KEY] = [{"id": "test_authorizer", "path": "test.path.TestAuthorizer", "args": {}}]
 
diff --git a/tests/unit_test/tool/code_pre_installer/test_install.py b/tests/unit_test/tool/code_pre_installer/test_install.py
@@ -221,6 +221,9 @@ def test_parse_args():
         default=DEFAULT_APPLICATION_INSTALL_DIR,
         help="Installation prefix (default: /opt/nvflare/apps)",
     )
+    mock_parser.add_argument.assert_any_call(
+        "-s", "--site-name", required=True, help="Target site name (e.g., site-1, server)"
+    )
     mock_parser.add_argument.assert_any_call(
         "-ts",
         "--target_shared_dir",

Original file line number	Diff line number	Diff line change
`@@ -8,7 +8,7 @@`
`8`	`8`	`"executor": {`
`9`	`9`	`"path": "nvflare.app_opt.pt.in_process_client_api_executor.PTInProcessClientAPIExecutor",`
`10`	`10`	`"args": {`
`11`		`- "task_script_path": "/vault/packages/share/hello-pt_cifar10_fl.py"`
	`11`	`+ "task_script_path": "/local/custom/hello-pt_cifar10_fl.py"`
`12`	`12`	`}`
`13`	`13`	`}`
`14`	`14`	`}`