Hi, I am running Federated Learning with differential privacy folder. It seems the cifar10 is only ok with 10 clients (number of client = number of class). When changing the number of client different with number of class ( example: number of client =20 while number of class in cifar10 = 10). The system get error. I think it is necessary to edit the code with flexible number of client. Do you have any suggestion? I changed the number of client using bellow code:

And It get this error

Thanks.

I'm a little confused.

I was hoping I could just git submodule this or pip install tff-research or pip install -e git+https://github.com/google-research/federated.

Could you share the intended installation process? (for example how to use just utils in this project?) (I could use with some ugly hacks, but I rather there was a direct way, preferably without having to build everything with bazel)

Hi,

I noticed that you should include "pip install scipy==1.1.0 " and pip install tensorflow_addons" into your library required document.

Thanks

Environment: This docker container tf-nightly, tff-nightly, GPU: Tesla V100-SXM2 32 GB

I ran bazel test shared:fed_avg_schedule_test and it did not pass the tests because the memory was overused: test.log

Was wondering what could be done to get this to work? Ideally I would want to follow the same testing practices in this repo for my own code and experiments.

I can run

bazel run main:federated_trainer -- --task=emnist_cr --total_rounds=100
--client_optimizer=sgd --client_learning_rate=0.1 --client_batch_size=20
--server_optimizer=sgd --server_learning_rate=1.0 --clients_per_round=10
--client_epochs_per_round=1 --experiment_name=emnist_fedavg_experiment

(the original example) just fine.

I used the following to get a centralized training benchmark and facing a GFile attribute error every time after 1st epoch.

bazel run main:centralized_trainer -- --task=emnist_cr --num_epochs=100 --experiment_name=emnist_central_experiment --centralized_optimizer=sgd --centralized_learning_rate=0.1 --batch_size=64

A tf2.4 and tff-nightly environment was used as mention in another issue NotImplementedError("b/162106885") for Optimization Folder #24.

The following pic shows the call stack of this error.

Similar to the federated version of CIFAR100, we follow the methods in this paper. Currently, we assume that there are total 10 clients, each of which has 5000 images.

Hi, there,

I am trying to reproduce the FedAdam of cifar100 in figure 1 by using the command below:

bazel run :trainer -- --task=cifar100_image --total_rounds=4000 --client_optimizer=sgd --client_learning_rate=0.031622 --client_batch_size=20 --server_optimizer=adam --server_learning_rate=1.0 --clients_per_round=10 --client_epochs_per_round=1 --experiment_name=fed_Adaptive_sgd_adam_opt -server_adam_epsilon=0.1 --client_datasets_random_seed=1

I got val accuracy around 45% but expected 53% in the paper.

Did I miss something there? Thanks in advance for your help!

My machine runs out of RAM when trying to run a centralized baseline on StackOverflow NWP. The following code reproduces the issue (the process gets killed after RAM is overflown on a machine with 64 GB of RAM):

>>> from utils.datasets import stackoverflow_word_prediction
>>> datasets = stackoverflow_word_prediction.get_centralized_datasets(vocab_size=10000, max_sequence_length=20)

I believe the high memory usage comes from this line, which calls create_tf_dataset_from_all_clients(), which in turn creates the centralized dataset from tensor slices, so the whole dataset is kept in memory.

Is it possible to somehow create centralized SO NWP dataset without keeping everything in RAM?

Hi, Thanks for your framework. I tried to run your code but get this error "TypeError: Values of type <label=int32,pixels=float32[28,28]>* cannot be cast to type <pixels=float32[28,28],label=int32>*" when loading emnist data. Do you know how I can fix it? Thanks.

Hi, thanks for your share, I tried to run your code as Example usage do:

bazel run main:federated_trainer -- --task=emnist_cr --total_rounds=100
 --client_optimizer=sgd --client_learning_rate=0.1 --client_batch_size=20
 --server_optimizer=sgd --server_learning_rate=1.0 --clients_per_round=10
 --client_epochs_per_round=1 --experiment_name=emnist_fedavg_experiment

But I got following errors. Could you help me fix them? Thanks.

Hi, I know that this is a work in progress but, I am curious about how we can compute epsilon for this implementation of DP for federated learning.

Is it possible to use the compute epsilon proposed here (https://arxiv.org/abs/1908.10530) for the federated model?

Greetings,

In the targeted_attack example, docs refer to an implementation of the defenses against the attack (aggregate_fn.py). This file does not exist in the directory, and I would like to take a look into it.

thanks!

When trying to run experiments/emnist:run_experiments it fails because the calls it makes to bazel run fail because these get executed out of the normal workspace inside the cache build directory, thus no WORKSPACE file can be found.

Not a major/urgent issue, but just FYI:

In define_optimizer_flags the optimizer flag is defined one way, while in create_optimizer_fn_from_flags the optimizer flag is defined another way.

For example, if your prefix is '' then define_optimizer_flags defines the flag '_optimizer' while create_optimizer_fn_from_flags looks for 'optimizer'. I fixed it in my own repo by using the method prefixed in both define_optimizer_flags and create_optimizer_fn_from_flags.

Hi,

I am trying to reproduce the experiments in "Differentially Private Learning with Adaptive Clipping" (2021), the source code for which is provided under federated/differential_privacy. The paper does not report the final server learning rates used for DP-FedAvgM with clipping enabled. It simply states the following in Section 3.1 -

Therefore, for all approaches with clipping—fixed or adaptive—we search over a small grid of five server learning rates, scaling the values in Table 1 by {1, 10^1/4, 10^1/2, 10^3/4, 10}. For all configurations, we report the best performing model whose server learning rate was chosen from this small grid on the validation set.

It is not computationally feasible for me to search for the optimal server lr in every possible configuration so I was hoping you could specify the learning rates that were used for training the best performing models. Thank you.

Hi All,

Thank you for publishing the code.

I installed the packages in the same way as recommended here: https://github.com/google-research/federated/issues/57#issuecomment-1062468566

(tff) ady@vws35:~/code/federated/distributed_dp$ pip list
Package                       Version
----------------------------- -------------------
absl-py                       1.0.0
astunparse                    1.6.3
attrs                         21.2.0
cachetools                    3.1.1
certifi                       2021.10.8
charset-normalizer            2.0.12
cloudpickle                   2.0.0
cycler                        0.11.0
decorator                     5.1.1
dill                          0.3.4
dm-tree                       0.1.6
farmhashpy                    0.4.0
flatbuffers                   2.0
fonttools                     4.30.0
gast                          0.5.3
google-auth                   2.6.0
google-auth-oauthlib          0.4.6
google-pasta                  0.2.0
googleapis-common-protos      1.55.0
grpcio                        1.34.1
h5py                          3.6.0
idna                          3.3
importlib-metadata            4.11.3
jax                           0.2.28
jaxlib                        0.1.76
keras                         2.8.0
Keras-Preprocessing           1.1.2
kiwisolver                    1.4.0
libclang                      13.0.0
Markdown                      3.3.6
matplotlib                    3.5.1
mpmath                        1.2.1
numpy                         1.21.5
oauthlib                      3.2.0
opt-einsum                    3.3.0
packaging                     21.3
pandas                        1.4.1
Pillow                        9.0.1
pip                           21.2.4
portpicker                    1.3.9
promise                       2.3
protobuf                      3.19.4
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pyparsing                     3.0.7
python-dateutil               2.8.2
pytz                          2021.3
requests                      2.27.1
requests-oauthlib             1.3.1
rsa                           4.8
scipy                         1.8.0
semantic-version              2.8.5
setuptools                    58.0.4
six                           1.16.0
tensorboard                   2.8.0
tensorboard-data-server       0.6.1
tensorboard-plugin-wit        1.8.1
tensorflow                    2.8.0
tensorflow-addons             0.16.1
tensorflow-datasets           4.5.2
tensorflow-estimator          2.8.0
tensorflow-federated          0.20.0
tensorflow-io-gcs-filesystem  0.24.0
tensorflow-metadata           1.7.0
tensorflow-model-optimization 0.7.1
tensorflow-privacy            0.7.3
tensorflow-probability        0.16.0
termcolor                     1.1.0
tf-estimator-nightly          2.8.0.dev2021122109
tqdm                          4.28.1
typeguard                     2.13.3
typing_extensions             4.1.1
urllib3                       1.26.9
Werkzeug                      2.0.3
wheel                         0.37.1
wrapt                         1.14.0
zipp                          3.7.0
(tff) ady@vws35:~/code/federated/distributed_dp$ python --version
Python 3.9.7

However, I run in the following problem:

(tff) ady@vws35:~/code/federated/distributed_dp$ bazel run :fl_run --     --task=emnist_character     --server_optimizer=sgd     --server_learning_rate=1     --server_sgd_momentum=0.9     --client_optimizer=sgd     --client_learning_rate=0.03     --client_batch_size=20     --experiment_name=my_emnist_test     --epsilon=10     --l2_norm_clip=0.03     --dp_mechanism=ddgauss     --logtostderr  --total_rounds 2
WARNING: Output base '/h/ady/.cache/bazel/_bazel_ady/39df1af3e8de7748262d01b9bcee607d' is on NFS. This may lead to surprising failures and undetermined behavior.
DEBUG: Rule 'rules_python' indicated that a canonical reproducible form can be obtained by modifying arguments commit = "a0fbf98d4e3a232144df4d0d80b577c7a693b570", shallow_since = "1586444447 +0200" and dropping ["tag"]
DEBUG: Repository rules_python instantiated at:
  /h/ady/code/federated/WORKSPACE:5:15: in <toplevel>
Repository rule git_repository defined at:
  /h/ady/.cache/bazel/_bazel_ady/39df1af3e8de7748262d01b9bcee607d/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
INFO: Analyzed target //distributed_dp:fl_run (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //distributed_dp:fl_run up-to-date:
  bazel-bin/distributed_dp/fl_run
INFO: Elapsed time: 0.396s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/distributed_dp/fl_run '--task=emnist_character' '--server_optimizer=sgd' '--server_learning_rate=1' '--server_sgd_momentum=0.9' '--client_optimizer=sgd' '--client_learning_rate=0.03' '--client_batch_size=20' '--experiment_name=my_emnist_test' '--epsilon=10' '--l2_norm_clip=0.03' '--dp_mechanism=ddgauss'INFO: Build completed successfully, 1 total action
2022-03-17 16:06:23.299187: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-03-17 16:06:23.299213: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-03-17 16:06:30.611416: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-03-17 16:06:30.612106: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory
2022-03-17 16:06:30.612540: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory
2022-03-17 16:06:30.613066: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
2022-03-17 16:06:30.613540: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory
2022-03-17 16:06:30.613981: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory
2022-03-17 16:06:30.614415: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory
2022-03-17 16:06:30.614841: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
2022-03-17 16:06:30.614853: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
I0317 16:06:30.714181 140367148568768 sql_client_data.py:127] Loaded 3400 client ids from SQL database.
2022-03-17 16:06:30.717065: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
I0317 16:06:30.906410 140367148568768 sql_client_data.py:127] Loaded 3400 client ids from SQL database.
I0317 16:06:31.114517 140367148568768 keras_utils.py:365] Adding default num_examples metric to model
I0317 16:06:31.114643 140367148568768 keras_utils.py:368] Adding default num_batches metric to model
I0317 16:06:31.190037 140367148568768 keras_utils.py:365] Adding default num_examples metric to model
I0317 16:06:31.190158 140367148568768 keras_utils.py:368] Adding default num_batches metric to model
I0317 16:06:31.195640 140367148568768 fl_utils.py:71] Shared DP Parameters:
I0317 16:06:31.195845 140367148568768 fl_utils.py:72] {'clip': 0.03,
 'delta': 0.0002941176470588235,
 'dim': 1018174,
 'epsilon': 10.0,
 'mechanism': 'ddgauss',
 'num_clients': 3400,
 'num_clients_per_round': 100,
 'num_rounds': 2,
 'sampling_rate': 0.029411764705882353}
I0317 16:09:09.068672 140367148568768 fl_utils.py:151] ddgauss parameters:
I0317 16:09:09.068935 140367148568768 fl_utils.py:152] {'beta': 0.6065306597126334,
 'bits': 16,
 'dim': 1018174,
 'gamma': 0.0001248800740568264,
 'inflated_l2': 0.0707097967941405,
 'k_stddevs': 4,
 'local_stddev': 0.002430822051759469,
 'mechanism': 'ddgauss',
 'noise_mult_clip': 0.8102740172531564,
 'noise_mult_inflated': 0.3437744360709157,
 'padded_dim': 1048576.0,
 'scale': 8007.682631137392}
I0317 16:09:09.069010 140367148568768 ddpquery_utils.py:44] Conditional rounding set to True (beta = 0.606531)
I0317 16:09:09.157962 140367148568768 keras_utils.py:365] Adding default num_examples metric to model
I0317 16:09:09.158077 140367148568768 keras_utils.py:368] Adding default num_batches metric to model
I0317 16:09:10.512192 140367148568768 keras_utils.py:365] Adding default num_examples metric to model
I0317 16:09:10.512315 140367148568768 keras_utils.py:368] Adding default num_batches metric to model
I0317 16:09:12.278361 140367148568768 keras_utils.py:365] Adding default num_examples metric to model
I0317 16:09:12.278485 140367148568768 keras_utils.py:368] Adding default num_batches metric to model
I0317 16:09:14.021716 140367148568768 keras_utils.py:365] Adding default num_examples metric to model
I0317 16:09:14.021839 140367148568768 keras_utils.py:368] Adding default num_batches metric to model
I0317 16:09:14.286071 140367148568768 keras_utils.py:365] Adding default num_examples metric to model
I0317 16:09:14.286196 140367148568768 keras_utils.py:368] Adding default num_batches metric to model
Traceback (most recent call last):
  File "/h/ady/.cache/bazel/_bazel_ady/39df1af3e8de7748262d01b9bcee607d/execroot/org_federated_research/bazel-out/k8-opt/bin/distributed_dp/fl_run.runfiles/org_federated_research/distributed_dp/fl_run.py", line 290, in <module>
    app.run(main)
  File "/h/ady/.conda/envs/tff/lib/python3.9/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/h/ady/.conda/envs/tff/lib/python3.9/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "/h/ady/.cache/bazel/_bazel_ady/39df1af3e8de7748262d01b9bcee607d/execroot/org_federated_research/bazel-out/k8-opt/bin/distributed_dp/fl_run.runfiles/org_federated_research/distributed_dp/fl_run.py", line 272, in main
    program_state_manager, metrics_managers = training_utils.create_managers(
  File "/h/ady/.cache/bazel/_bazel_ady/39df1af3e8de7748262d01b9bcee607d/execroot/org_federated_research/bazel-out/k8-opt/bin/distributed_dp/fl_run.runfiles/org_federated_research/utils/training_utils.py", line 65, in create_managers
    tensorboard_release_manager = tff.program.TensorBoardReleaseManager(
AttributeError: module 'tensorflow_federated.python.program' has no attribute 'TensorBoardReleaseManager'
(tff) ady@vws35:~/code/federated/distributed_dp$