[Q&A] Best way to perform checkpointing in a production system

[virginiafdez]

Python version (python3 -V)

3.12

NVFlare version (python3 -m pip list | grep "nvflare")

2.5.2

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

Ubuntu 22.04

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

We would like to add the following features to our FL production system:

1. Provide checkpoints at the beginning of a training optionally.
2. Load from checkpoint if the training is stopped and resumed

Our system supports the actual normal running of jobs, and also the possibility to test the apps via the nvflare simulator locally.

I have found some relevant discussions like #1973, where 1. has been addressed. In this case, my question is not that related to how to load the weights within the app, which I guess is done by passing the path to the weights to the persistor, but what is the best practice in terms of where these weights should be stocked in the server container. I guess it's better not to put this inside of the custom app, as the clients would also get the weights and this isn't necessary. Is there a folder that would be more dedicated for this type of files in a standard nvflare setup?

2.1 In a production setup, is it possible to resume a training if it has been stopped before finishing? This means, making use of an app folder that has already been created? Is there any best practice to do so? This seems to be relevant: https://nvflare.readthedocs.io/en/2.2/apidocs/nvflare.apis.fl_snapshot.html but I couldn't find any tutorial or example that used this so I was unsure.
2.2 In the simulator, would this even be possible? As far as I'm aware, the simulate_job folders are reset if you restart a training, so I'm not sure if it's even possible to bypass it.

Thank you for any bit of knowledge on the matter!

[chesterxgchen]

@ZiyueXu77 @holgerroth @YuanTingHsieh

[holgerroth]

Hi @virginiafdez, let me try to answer.

1. Yes, the persistor can be configured to read checkpoints from the disk. NVFlare doesn't manage existing pretrained checkpoints. You could have a static mount on the server where these can be loaded from. Also, you can define different apps for the server and clients. So, only the server app can contain the ckpt if you want to upload it as part of the job. See a meta.json with a deployment map for that.

2. We don't have an auto-resume option at this moment. In a production environment, the approach would be to extract the job result from the job storage and use the global model as initialization for the next job. This wouldn't restore all the states of the server and clients, but it would allow you to resume the training from the last successful global model.

With the simulator, you need to define a new working directory to prevent overwriting the old output. However, you don't need to extract from job storage, as the simulator saves the raw training artifacts after the job is completed (unlike in a production environment).

A more general "resume job" feature could be an interesting enhancement in the future @chesterxgchen @yanchengnv.

[virginiafdez]

Thanks for the answer! I guess it'll be fine for a simple setup.
We will follow this approach for now then!
We were actually looking into FedOpt, for which the state dict of the server's optimizer / scheduler would be cool to keep somewhere as well, not sure if that's already doable.

[holgerroth]

yes, you can save the optimizer state dict as part of the global checkpoint. Might need some customization of the persistor.