[Q&A] Return weights, as well as another metric from client during training (scatter_and_gather)

[virginiafdez]

Python version (python3 -V)

3.10

NVFlare version (python3 -m pip list | grep "nvflare")

2.5.2

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

Ubuntu 22.04

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

Hello!

I am running a scatter_and_gather workflow with two sites. I would like to adjust the number of global rounds based on a performance metric, instead of it being a fixed number. For this, when the weights are sent to the server from the client, I want to return a metric value in addition to the model weights.

Before, the weights were packed in a DXO (type DataKind.WEIGHTS, and MetaKey.NUM_STEPS_CURRENT_ROUND: self._n_iterations), transformed into a Shareable, and processed by a function that is populating attribute result_received_cb of the Task object which gets called with broadcast_and_wait.

This shareable is then - I presume - the client_task.result in that function. This is, I assume, what gets sent to the aggregator.

Because I want to add a metric to the shareable that is output by the client_task, I am putting the weights and the metrics DXOs into a Collection DXO, transforming that into a Shareable.

            outgoing_dxo = DXO(
                data = {"weights": dxo_model, "metrics": dxo_metrics},
                data_kind=DataKind.COLLECTION,
                meta={MetaKey.NUM_STEPS_CURRENT_ROUND: iterations})

Back in the controller, I'm trying to unpack the metrics and the weights, and sending the weights DXO only to the aggregator.
I printed the weights, they seem to be fine, but the aggregator is spitting the following:

For site 1:
2025-04-11 14:17:57,513 - DXOAggregator - WARNING - [identity=simulator_server, run=simulate_job, wf=scatter_and_gather, peer=site1, peer_run=simulate_job, peer_rc=OK, task_name=train, task_id=1234: Aggregation_weight missing for ? and set to default value, 1.0 This kind of message will show 10 times at most.

For site 2:

2025-04-11 14:18:43,809 - DXOAggregator - WARNING - [identity=simulator_server, run=simulate_job, wf=scatter_and_gather, peer=site2, peer_run=simulate_job, peer_rc=OK, task_name=train, task_id=1234]: discarding DXO from ? at round: None as None accepted already
2025-04-11 14:18:43,809 - ScatterAndGatherLDM - INFO - [identity=simulator_server, run=simulate_job, wf=scatter_and_gather, peer=site2, peer_run=simulate_job, peer_rc=OK, task_name=train, task_id=1234]: Contribution from site2 REJECTED by the aggregator.

I am not sure what the issue is: is some meta information missing somewhere regarding the client result? Why is the site set to "?" for client 1 and the result set to None for client 2?

Thanks a lot for any input on the behaviour of this workflow!

Virginia

[chesterxgchen]

here is an example, you might be able to reference: fed average with earlier stopping.

https://github.com/NVIDIA/NVFlare/blob/main/examples/hello-world/hello-fedavg/hello-fedavg.ipynb
or
with discussion in sel-paced learning tutorials: how to customized server side logics

https://github.com/NVIDIA/NVFlare/blob/main/examples/tutorials/self-paced-training/part-1_federated_learning_introduction/chapter-1_running_federated_learning_applications/01.3_server_side_customization/customize_server_logics.ipynb

In these examples. you work with Data Structure is FLModel, instead of DXO ( DXO is used behind the scene).

in each round you return the weight and metrics, the FedAvg controller will determine the stop condition based on metrics

[virginiafdez]

Thanks a lot! This was very helpful!