I am training Fedavg to get the benchmark accuracy with the given parameters. But, the accuracy is stuck under 50.

Here is all my code:

!git clone https://github.com/FedML-AI/FedML

cd /content/FedML/fedml_experiments/standalone/fedavg

!python main_fedavg.py --model mobilenet --dataset cifar10 --data_dir ./../../../data/cifar10 --partition_method hetero --comm_round 100 --epochs 20 --batch_size 64 --lr 0.001

I suppose to get over 80% accuracy at least according to these benchmark results.

https://wandb.ai/automl/fedml/runs/390hdz0e

from the step3 in docs https://doc.fedml.ai/simulation/examples/sp_fedavg_mnist_lr_example.html, I am told to execute the following command to run the example code: python torch_fedavg_mnist_lr_one_line_example.py --cf fedml_config.yaml However, when I tried to modify the args in this YAML file (eg. set the using_gpu to true), I found that in the runtime the training is still based on the CPU. So I checked the code in fedml/lib/python3.7/site-packages/fedml/arguments.py, line 63, and I found a snippet of code as follows: `

    path_current_file = path.abspath(path.dirname(__file__))
    if training_type == "simulation" and comm_backend == "single_process":
        config_file = path.join(path_current_file, "config/simulation_sp/fedml_config.yaml")
        cmd_args.yaml_config_file = config_file
    elif training_type == "simulation" and comm_backend == "MPI":
        config_file = path.join(
            path_current_file, "config/simulaton_mpi/fedml_config.yaml"
        )
        cmd_args.yaml_config_file = config_file
    elif training_type == "cross_silo":
        pass
    elif training_type == "cross_device":
        pass
    else:
        pass

` It seems during the simulation, it does not matter how you set the YAML file, the default one would be loaded.

When running the following code

#!/usr/bin/env bash

WORKER_NUM=$1

PROCESS_NUM=`expr $WORKER_NUM + 1`
echo $PROCESS_NUM

hostname > mpi_host_file

$(which mpirun) -np $PROCESS_NUM \
-hostfile mpi_host_file \
python torch_fedavg_mnist_lr_one_line_example.py --cf config/fedml_config.yaml

Encounter an error

(base) PS C:\Users\doubl\Desktop\script\fedml> bash run_step_by_step_example.sh 4                                       
5                                                                                                                       
run_step_by_step_example.sh: line 10: -np: command not found   

The code is excuted as the instructions in https://doc.fedml.ai/simulation/examples/mpi_torch_fedavg_mnist_lr_example.html, and all the dependency is installed.

The first error encountered is :

Building wheels for collected packages: fedml
  Building wheel for fedml (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [1027 lines of output]

The second error :

error: could not create 'build\bdist.win-amd64\wheel\fedml-0.7.305.data\purelib\examples\cross_silo\mqtt_s3_fedavg_hierarchical_manual_mnist_lr_example\one_line\main_fedml_cross_silo_hi.py': No such file or directory
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for fedml
  Running setup.py clean for fedml
Failed to build fedml

The last message displayed is :

DEPRECATION: fedml was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above. Discussion can be found at https://github.com/pypa/pip/issues/8368

The installation steps are all installed in accordance with the FEDML INSTALLATION ON WINDOWS OS teaching, also tried using older versions of python (eg: 3.7, 3.9), but still got the same problem.

When running the example of TEXT CLASSIFICATION in fednlp. bash run_simulation.sh 1 Encounter an error Is this caused by fedml or what else? All dependency is installed. Link: https://github.com/FedML-AI/FedML/tree/master/python/app/fednlp

I can not get the same results when I run the command: sh run_fedavg_standalone_pytorch.sh 0 10 10 10 shakespeare ./../../../data/shakespeare rnn hetero 100 1 0.8 sgd 0 my results is: https://wandb.ai/pilgrim_cz/fedml/runs/5jqpw1i4?workspace=user-pilgrim_cz the benchmart res is https://wandb.ai/automl/fedml/runs/144ey9w6 same problem when I run minst dataset Is there anything wrong? I did not change any code.

Running the fedavg on the configure: 20 rounds, 10 epochs, 2 clients, cifar-10 dataset, resnet56, but the program always crashed! The errors as the following:

I don't know the reason for the errors.

How to run fedml /python/examples/simulation/MPI locally_ base_ framework_ The code in example, I use readme.md runs. The error in the figure appears. I have deployed the mqtt server locally. I have configured comm in the configuration file_ args.mqtt_ config_ Path, but it doesn't work

How to remove mlops? I set using_mlops: false has no effect

Hi, I tried to run the FedGKT algorithm but the code got stuck in this part. For the FedAvg, it worked well. For my setting case, I just ran the code on the CPU. So can anyone help me with this? Thank you! P/s: Here I ran with 8 clients.

I was using FedML platform for testing distributed training results for object detection between single server and multiple clients. Unfortunately, MQTT has some issue where server or client sometimes cannot receive the packet properly and showing only:

receive_message. msg_type = 0, sender_id = 0, receiver_id = 0
receive_message. msg_type = 0, sender_id = 0, receiver_id = 0

where msg_type = 0 means received message has been corrupted. Although, I am using one server with multiple GPU to simulate server and clients based distributed training CROSS_SILO_HORIZONTAL mode, I could not understand why is this happening. Especially, when i use bigger data, this problem is happening in very early communication rounds.

So, if anyone has solution with this problem, please share your experience.

Just in case, I am attaching output of server and clients screenshots.

SERVER OUTPUT

CLIENT-1 OUTPUT

CLIENT-2 OUTPUT

I have training Cross-Silo Horizontal distributed training mode with following configuration settings:

common_args:
  training_type: "cross_silo"
  random_seed: 0
  scenario: "horizontal"
  using_mlops: false
  config_version: release
  name: "exp" # yolo
  project: "runs/train" # yolo
  exist_ok: true # yolo

environment_args:
  bootstrap: /home/gpuadmin/Project/FedML/python/app/fedcv/object_detection/config/bootstrap.sh

data_args:
  dataset: "bdd"
  data_cache_dir: ~/fedcv_data
  partition_method: "homo"
  partition_alpha: 0.5
  data_conf: "/home/gpuadmin/Project/FedML/python/app/fedcv/object_detection/data/bdd.yaml" # yolo
  img_size: [640, 640] # [640, 640]

model_args:
  model: "yolov5" # "yolov5"
  class_num: 13
  yolo_cfg: "/home/gpuadmin/Project/FedML/python/app/fedcv/object_detection/model/yolov5/models/yolov5s.yaml" # "./model/yolov6/configs/yolov6s.py" # yolo
  yolo_hyp: "/home/gpuadmin/Project/FedML/python/app/fedcv/object_detection/config/hyps/hyp.scratch.yaml" # yolo
  weights: "none" # "best.pt" # yolo
  single_cls: false # yolo
  conf_thres: 0.001 # yolo
  iou_thres: 0.6 # for yolo NMS
  yolo_verbose: true # yolo

train_args:
  federated_optimizer: "FedAvg"
  client_id_list:
  client_num_in_total: 2
  client_num_per_round: 2
  comm_round: 10
  epochs: 4
  batch_size: 64
  client_optimizer: sgd
  lr: 0.01
  weight_decay: 0.001
  checkpoint_interval: 1
  server_checkpoint_interval: 1

validation_args:
  frequency_of_the_test: 2

device_args:
  worker_num: 2
  using_gpu: true
  gpu_mapping_file: /home/gpuadmin/Project/FedML/python/app/fedcv/object_detection/config/gpu_mapping.yaml
  gpu_mapping_key: mapping_config5_2
  gpu_ids: [0,1,2,3,4,5,6,7]

comm_args:
  backend: "MQTT_S3"
  mqtt_config_path: /home/gpuadmin/Project/FedML/python/app/fedcv/object_detection/config/mqtt_config.yaml
  s3_config_path: /home/gpuadmin/Project/FedML/python/app/fedcv/object_detection/config/s3_config.yaml

tracking_args:
  log_file_dir: /home/gpuadmin/Project/FedML/python/app/fedcv/object_detection/log
  enable_wandb: true
  wandb_key: ee0b5f53d949c84cee7decbe7a6
  wandb_project: fedml
  wandb_name: fedml_torch_object_detection

During the training I have got very high mAP@50 and mAP@50:95 almost for every epoch from beginning till the end. Normally, mAP should be small for early training epochs and it should grow slowly in the later epochs. But in my case it is just fluctuating in range of 0.985 ~ 0.9885 for both clients. I have checked metric calculation functions borrowed from original YOLOv5 PyTorch platform. They are working fine. IF ANYBODY CAN SHARE THEIR PRELIMINARY RESULTS FOR DISTRIBUTED OBJECT DETECTION FOR ANY DATASET (COCO or PASCAL VOC). I would like to verify my result with their result. For solo YOLOv5s model training, ,mAP is much smaller and it is growing epochs by epochs.

Any clue from the Authors would be very much appreciated.

P.S. For mAP calculation, I used default val(train_data, device, args) function inside the YOLOv5Trainer class in the yolov5_trainer.py .

Hello, recently I'm trying to reproduce the demo of fedmlsdk on android device,and I followed the tutorial at https://github.com/FedML-AI/FedML/tree/master/android/fedmlsdk)/MobileNN/, and I see the line below：

run ./test_arm_android_64.sh, which will push demo.out to your android device and execute it.

however，i can not find the test_arm_android_64.sh in the repo? Could you please check the issue?

I have already installed mpi4py3.1.4 and libopenmpi-dev,and the Ubuntu is 18。I searched a lot of posts on the Internet, but I couldn't figure it out。

https://doc.fedml.ai/cross-silo/examples/mqtt_s3_fedavg_hierarchical_mnist_lr_example.html

Hi,@chaoyanghe I refer to the URL above The server uses ubuntu 22.04LTS+RTX2080ti Both silo1 and server are executed on ubuntu

silo2 executes on Raspberry Pi 4 + 64bit OS But the raspberry pie has no GPU, and the mapping gpu has been changed to false What else needs to be modified?

Thank you for your reply

Allow use of custom criteria of client trainer for classification task; allow custom client trainer for single process simulation; better exception info and hint for these two condition.

Hello，I want to ask for a help： In the "main_fedml_image_segmentation. py" file of the fedcv module, "SegmentationTrainer. py" and "SegmentationAggregator. py" under the "image_segmentation/trainer" module are imported during training. Suppose I modify different algorithms in "fedml_config. yaml", such as FedAvg, FedOpt, FedProx, but still use the same SegmentationTrainer and SegmentationAggregator, without any difference? How should different algorithms be implemented? In fedcv/segmentation， how do the algorithms in fedml/simulation/mpi work? They are all the same?

Hello, I tried to change the common args from 'backend:MPI' to 'backend:sp' in ego_network_node_clf, and it failed. So I wonder if fedgraphnn only support MPI. If not, how can i adjust the config parameter, thanks!