[Q&A] Does it really work in a real world setting? from Mac server and two linux client?

[zillurbmb51]

Python version (python3 -V)

3.11

NVFlare version (python3 -m pip list | grep "nvflare")

2.6.2

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

MacOS

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

I want to run a simple federated linear regression from my Mac in two linux clients. Attached is client_trainer.py and project.yml. Here is my directory structure.
How can I modify the client_trainer.py or project.yml to start nvflare server-client connection, run the training, save the outputs to my Mac and then disconnect the clients

When I use (nvflare) zillurrahman@Zillurs-MacBook-Pro poc_workspace % bash server/startup/start.sh

I get

(nvflare) zillurrahman@Zillurs-MacBook-Pro poc_workspace % WORKSPACE set to /Users/zillurrahman/Desktop/Desktop/zillur/work/nvflare/test1/poc_workspace/server/startup/..
PYTHONPATH is /local/custom:
start fl because of no pid.fl
new pid 10722
Execute exception: TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
         ^^^^^^^^^^^^^^^^^^^
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/private/fed/app/server/server_train.py", line 87, in main
    configure_logging(workspace)
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/fuel/utils/log_utils.py", line 312, in configure_logging
    assert os.path.isfile(log_config_file_path), f"missing log config file {log_config_file_path}"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

start fl because process of 10722 does not exist
new pid 10741
Execute exception: TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
         ^^^^^^^^^^^^^^^^^^^
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/private/fed/app/server/server_train.py", line 87, in main
    configure_logging(workspace)
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/fuel/utils/log_utils.py", line 312, in configure_logging
    assert os.path.isfile(log_config_file_path), f"missing log config file {log_config_file_path}"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

start fl because process of 10741 does not exist
new pid 10767
Execute exception: TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
         ^^^^^^^^^^^^^^^^^^^
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/private/fed/app/server/server_train.py", line 87, in main
    configure_logging(workspace)
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/fuel/utils/log_utils.py", line 312, in configure_logging
    assert os.path.isfile(log_config_file_path), f"missing log config file {log_config_file_path}"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

start fl because process of 10767 does not exist
new pid 10786
Execute exception: TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
         ^^^^^^^^^^^^^^^^^^^
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/private/fed/app/server/server_train.py", line 87, in main
    configure_logging(workspace)
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/fuel/utils/log_utils.py", line 312, in configure_logging
    assert os.path.isfile(log_config_file_path), f"missing log config file {log_config_file_path}"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

start fl because process of 10786 does not exist
new pid 10812
Execute exception: TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
         ^^^^^^^^^^^^^^^^^^^
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/private/fed/app/server/server_train.py", line 87, in main
    configure_logging(workspace)
  File "/Users/zillurrahman/miniconda3/envs/nvflare/lib/python3.11/site-packages/nvflare/fuel/utils/log_utils.py", line 312, in configure_logging
    assert os.path.isfile(log_config_file_path), f"missing log config file {log_config_file_path}"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 

My directory structure:

(nvflare) zillurrahman@Zillurs-MacBook-Pro poc_workspace % tree
.
├── 1
├── [email protected]
│   ├── local
│   ├── startup
│   │   ├── client.crt
│   │   ├── client.key
│   │   ├── fed_admin.json
│   │   ├── fl_admin.sh
│   │   ├── readme.txt
│   │   └── rootCA.pem
│   └── transfer
├── jobs
│   └── linear_regression_app
│       ├── app
│       │   ├── config
│       │   │   ├── config_fed_client.conf
│       │   │   └── config_fed_server.conf
│       │   ├── config_fed_client.json
│       │   ├── config_fed_server.json
│       │   └── custom
│       ├── custom
│       │   └── client_trainer.py
│       └── meta.conf
├── project.yml
├── server
│   ├── audit.log
│   ├── local
│   │   ├── authorization.json.default
│   │   ├── log.config.default
│   │   ├── privacy.json.sample
│   │   └── resources.json.default
│   ├── log.txt
│   ├── readme.txt
│   ├── startup
│   │   ├── fed_server.json
│   │   ├── rootCA.pem
│   │   ├── server.crt
│   │   ├── server.key
│   │   ├── signature.json
│   │   ├── start.sh
│   │   ├── stop_fl.sh
│   │   └── sub_start.sh
│   └── transfer
├── site-1
│   ├── audit.log
│   ├── daemon_pid.fl
│   ├── local
│   │   ├── authorization.json.default
│   │   ├── log.config.default
│   │   ├── privacy.json.sample
│   │   └── resources.json.default
│   ├── log.txt
│   ├── pid.fl
│   ├── readme.txt
│   ├── startup
│   │   ├── client.crt
│   │   ├── client.key
│   │   ├── fed_client.json
│   │   ├── rootCA.pem
│   │   ├── signature.json
│   │   ├── start.sh
│   │   ├── stop_fl.sh
│   │   └── sub_start.sh
│   └── transfer
└── site-2
    ├── local
    │   ├── authorization.json.default
    │   ├── log.config.default
    │   ├── privacy.json.sample
    │   └── resources.json.default
    ├── readme.txt
    ├── startup
    │   ├── client.crt
    │   ├── client.key
    │   ├── fed_client.json
    │   ├── rootCA.pem
    │   ├── signature.json
    │   ├── start.sh
    │   ├── stop_fl.sh
    │   └── sub_start.sh
    └── transfer

In my linux client:

(rapidsai) [zrahman@int5 ~]$ cd nvflare_client1/s
server/ site-1/ 
(rapidsai) [zrahman@int5 ~]$ cd nvflare_client1/site-1/
(rapidsai) [zrahman@int5 site-1]$ ls
local  readme.txt  startup  transfer
(rapidsai) [zrahman@int5 site-1]$ bash startup/start.sh 
(rapidsai) [zrahman@int5 site-1]$ WORKSPACE set to /home/zrahman/nvflare_client1/site-1/startup/..
PYTHONPATH is /local/custom:
start fl because of no pid.fl
new pid 1218896
Execute exception: TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
         ^^^^^^^^^^^^^^^^^^^
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/private/fed/app/client/client_train.py", line 80, in main
    configure_logging(workspace)
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/fuel/utils/log_utils.py", line 312, in configure_logging
    assert os.path.isfile(log_config_file_path), f"missing log config file {log_config_file_path}"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

start fl because process of 1218896 does not exist
new pid 1219173
Execute exception: TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
         ^^^^^^^^^^^^^^^^^^^
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/private/fed/app/client/client_train.py", line 80, in main
    configure_logging(workspace)
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/fuel/utils/log_utils.py", line 312, in configure_logging
    assert os.path.isfile(log_config_file_path), f"missing log config file {log_config_file_path}"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

start fl because process of 1219173 does not exist
new pid 1219528
Execute exception: TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
         ^^^^^^^^^^^^^^^^^^^
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/private/fed/app/client/client_train.py", line 80, in main
    configure_logging(workspace)
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/fuel/utils/log_utils.py", line 312, in configure_logging
    assert os.path.isfile(log_config_file_path), f"missing log config file {log_config_file_path}"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

start fl because process of 1219528 does not exist
new pid 1219892
Execute exception: TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
         ^^^^^^^^^^^^^^^^^^^
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/private/fed/app/client/client_train.py", line 80, in main
    configure_logging(workspace)
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/fuel/utils/log_utils.py", line 312, in configure_logging
    assert os.path.isfile(log_config_file_path), f"missing log config file {log_config_file_path}"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

start fl because process of 1219892 does not exist
new pid 1220447
Execute exception: TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/fuel/f3/mpm.py", line 154, in run
    rc = main_func(**kwargs)
         ^^^^^^^^^^^^^^^^^^^
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/private/fed/app/client/client_train.py", line 80, in main
    configure_logging(workspace)
  File "/home/zrahman/tools/rapidsai/lib/python3.11/site-packages/nvflare/fuel/utils/log_utils.py", line 312, in configure_logging
    assert os.path.isfile(log_config_file_path), f"missing log config file {log_config_file_path}"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 30, in isfile
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

start fl because process of 1220447 does not exist
System is in trouble and unable to start the task!!!!!

Any help? Please. Does it work in a real world settings where my laptop act as a server and admin and two linux machine as clients?
project.yml
client_trainer.py

[YuanTingHsieh]

@zillurbmb51 thanks for your question!

I will suggest you do the following steps:

(1) Run just in simulator mode in your mac to ensure you job is running good
(2) Use the following project.yml and run nvflare provision -p project.yml:

api_version: 3
name: example_project
description: NVIDIA FLARE sample project yaml file

participants:
  # Change the name of the server (server1) to the Fully Qualified Domain Name
  # (FQDN) of the server, for example: server1.example.com.
  # Ensure that the FQDN is correctly mapped in the /etc/hosts file.
  - name: server1
    type: server
    org: ServerOrg
    fed_learn_port: 8002
    admin_port: 8003
  - name: site-1
    type: client
    org: Site1Org
  - name: site-2
    type: client
    org: Site2Org
  - name: [email protected]
    type: admin
    org: AdminOrg
    role: project_admin

# The same methods in all builders are called in their order defined in builders section
builders:
  - path: nvflare.lighter.impl.workspace.WorkspaceBuilder
    args:
      template_file:
        - master_template.yml
        - aws_template.yml
        - azure_template.yml
  - path: nvflare.lighter.impl.static_file.StaticFileBuilder
    args:
      # config_folder can be set to inform NVIDIA FLARE where to get configuration
      config_folder: config

      overseer_agent:
        path: nvflare.ha.dummy_overseer_agent.DummyOverseerAgent
        # if overseer_exists is true, args here are ignored.  Provisioning
        #   tool will fill role, name and other local parameters automatically.
        # if overseer_exists is false, args in this section will be used and the sp_end_point
        # must match the server defined above in the format of SERVER_NAME:FL_PORT:ADMIN_PORT
        # 
        overseer_exists: false
        args:
          sp_end_point: server1:8002:8003

  - path: nvflare.lighter.impl.cert.CertBuilder
  - path: nvflare.lighter.impl.signature.SignatureBuilder

(3) Then you need to add this FQDN so all your machine can find:
(3.1) In your mac add the following line (sudo vim /etc/hosts):

127.0.0.1 server1

(3.2) In your linux machines add:

[your mac ip address] server1

(4) Then you can distribute (for example copy) the startup folder into each sites and run "./startup/start.sh"

(5) Then inside your mac you can start the admin console and interact with the system.

For a more complete documentation please check: https://nvflare.readthedocs.io/en/2.6/real_world_fl.html

[zillurbmb51]

Hi @YuanTingHsieh Thank you for your valuable suggestions.

I tried to follow your instructions but got stuck in point 3.2. I am unable to write in /etc/hosts (I do not have the write access).

Is there any workaround?

[zillurbmb51]

I have tried to create hosts file in the home directory of the linux machine for 3.2 but no result. Also in Mac, server can not communicate with any site even if the Mac is a site.

In our HPC, we have multiple app nodes. I tried the above example in two different nodes (one is a server and client and the other is only client). Server and client of the same host/node can communicate but the other client can not.