This repository is an official Tensorflow 2 implementation of Federated Continual Learning with Weighted Inter-client Transfer (ICML 2021)
Currently working on PyTorch version
There has been a surge of interest in continual learning and federated learning, both of which are important in deep neural networks in real-world scenarios. Yet little research has been done regarding the scenario where each client learns on a sequence of tasks from a private local data stream. This problem of federated continual learning poses new challenges to continual learning, such as utilizing knowledge from other clients, while preventing interference from irrelevant knowledge. To resolve these issues, we propose a novel federated continual learning framework, Federated Weighted Inter-client Transfer (FedWeIT), which decomposes the network weights into global federated parameters and sparse task-specific parameters, and each client receives selective knowledge from other clients by taking a weighted combination of their task-specific parameters. FedWeIT minimizes interference between incompatible tasks, and also allows positive knowledge transfer across clients during learning. We validate our FedWeIT against existing federated learning and continual learning methods under varying degrees of task similarity across clients, and our model significantly outperforms them with a large reduction in the communication cost.
The main contributions of this work are as follows:
-
We introduce a new problem of Federated Continual Learning (FCL), where multiple models continuously learn on distributed clients, which poses new challenges such as prevention of inter-client interference and inter-client knowledge transfer.
-
We propose a novel and communication-efficient framework for federated continual learning, which allows each client to adaptively update the federated parameter and selectively utilize the past knowledge from other clients, by communicating sparse parameters.
Please install packages from requirements.txt
after creating your own environment with python 3.8.x
.
$ pip install --upgrade pip
$ pip install -r requirements.txt
Please see config.py
to set your custom path for both datasets
and output files
.
args.task_path = '/path/to/task/' # for dataset
args.output_path = '/path/to/outputs/' # for logs, weights, etc.
Run below script to generate datasets
$ cd scripts
$ sh gen-data.sh
or you may run the following comamnd line directly:
python3 ../main.py --work-type gen_data --task non_iid_50 --seed 777
It automatically downloads 8 heterogeneous datasets
, including CIFAR-10
, CIFAR-100
, MNIST
, Fashion-MNIST
, Not-MNIST
, TrafficSigns
, Facescrub
, and SVHN
, and finally processes to generate non_iid_50
dataset.
To reproduce experiments, please execute train-non-iid-50.sh
file in the scripts
folder, or you may run the following comamnd line directly:
python3 ../main.py --gpu 0,1,2,3,4 \
--work-type train \
--model fedweit \
--task non_iid_50 \
--gpu-mem-multiplier 9 \
--num-rounds 20 \
--num-epochs 1 \
--batch-size 100 \
--seed 777
Please replace arguments as you wish, and for the other options (i.e. hyper-parameters, etc.), please refer to config.py
file at the project root folder.
Note: while training, all participating clients are logically swiched across the physical gpus given by --gpu
options (5 gpus in the above example).
All clients and server create their own log files in \path\to\output\logs\
, which include evaluation results, such as local & global performance and communication costs, and the experimental setups, such as learning rate, batch-size, etc. The log files will be updated for every comunication rounds.
@inproceedings{
yoon2021federated,
title={Federated Continual Learning with Weighted Inter-client Transfer},
author={Jaehong Yoon and Wonyong Jeong and Giwoong Lee and Eunho Yang and Sung Ju Hwang},
booktitle={International Conference on Machine Learning},
year={2021},
url={https://arxiv.org/abs/2003.03196}
}
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.