Git Product home page Git Product logo

hgo's Introduction

Democratizing Machine Learning

Resilient Distributed Learning with Heterogeneous Participants

Important ℹ️

Check the full version of the paper including additional details on the experiments as well as all the proofs of our theoretical results.

Paper: Democratizing_Machine_Learning___Extended_version.pdf

Experiments preparations

Install requirements

pip install -r requirements.txt

Download datasets

Description

This report contains the experiments of HgO on multiple models and datasets. Each model is trained on a given dataset under different configurations. We use the following combination:

  • Linear Regression: Boston
  • Logistic regression: Binary MNIST/ Fashion-MNIST/ CIFAR10
  • SVM: Phishing
  • Multinomial Logistic Regression: MNIST/ Fashion-MNIST/ CIFAR10
  • DNN: MNIST/ Fashion-MNIST/ CIFAR10

The different configuration are based on:

  • $\tau$ The unites of time the parameter server allows in each iteration.
  • $\lambda_i$ Thee computation rate associated with each worker.
  • $b_i$ The size of the gradient block each worker chooses to perform the computation.
  • $s_i$ The mini-batch size of the worker.

All experiments are conducted on either 50 or 100 workers with $q=80%$ of them active at each round unless stated otherwise.

We show only the main experiments however, you can run any experiment you want following the configuration provided below.

Learning rate

We first use cross-validation to select the best hyperparameters (learning rate) for each configuration according to SGD. HgO can have better learning rates (or selected dynamically) but we perform the experiments on the same values of the learning rate. We select the following learning rate for each model:

  • Linear Regression:
    • Boston: lr = 0.1
  • Logistic regression:
    • MNIST: lr = 0.01
    • Fashion-MNIST: lr = 0.001
  • SVM:
    • Phishing: lr = 0.0001
  • Multinomial Logistic Regression:
    • MNIST: lr = 0.01
    • Fashion-MNIST: lr = 0.001
    • CIFAR10: lr = 0.01
  • DNN:
    • MNIST: lr = 3
    • Fashion-MNIST: lr = 3
    • CIFAR10: lr = 3

Comparing HgO and SGD

A – HgO vs. SGD in terms of accuracy, convergence rate and gradient exchange.

  • Configuration
# >> Command
python main.py --model=MLR --dataset=mnist --workers=100 --batch_size=32 --tau=32 --rounds=150

# >> Configurations 
METRIC = "acc"
EVAL_ROUND = 5
BARS_ROUND = 25
PLOT_GRADS = True
# the options bellow attributes \lambda values to each worker following a normal distribution (mean, std)
WEAK_DEVICE = [500, 250] # mean / std
AVERAGE_DEVICE = [10000, 2500]
POWERFUL_DEVICE = [50000, 5000]

# >> Experiments
args.iid = 0 # Non-IID
args.iid_degree = 1 # Degree of non-iid.ness: 100% Non-IID.
config = [
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "1,0,0", 'legend': "SGD, $C_{weak}$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "SGD, $C_{1}$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0.7,0.3,0", 'legend': "SGD, $C_{2}$"},

    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "1,0,0", 'legend': "HgO, $C_{weak}$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "HgO, $C_{1}$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0.7,0.3,0", 'legend': "HgO, $C_{2}$"},
]
  • Output: Non-IID data (left); IID data (right)

  • Options

    You can configure the algorithm to use different configurations including models, aggregation rules, attacks, computation profiles, Byzantine works, etc.

B – HgO .vs SGD under different proportions of weak, average and powerful devices.

  • Configuration
# >> Command
python main.py --model=MLR --dataset=mnist --workers=100 --batch_size=32 --tau=32 --rounds=150

# >> Configurations 
METRIC = "acc"
EVAL_ROUND = 5
WEAK_DEVICE = [500, 250]
AVERAGE_DEVICE = [10000, 2500]
POWERFUL_DEVICE = [50000, 5000]

# >> Experiments
args.iid = 0 # Non-IID
args.iid_degree = 1 # Degree of non-iid.ness: 100% Non-IID.
config = [
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "1,0,0", 'legend': "SGD, $C_{weak}$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "SGD, $C_{1}$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0.7,0.3,0", 'legend': "SGD, $C_{2}$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0,0,1", 'legend': "SGD, $C_{powerful}$"},

    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "1,0,0", 'legend': "HgO, $C_{weak}$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "HgO, $C_{1}$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0.7,0.3,0", 'legend': "HgO, $C_{2}$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0,0,1", 'legend': "HgO, $C_{powerful}$"},
]
  • Output: Test Loss (left); Test Accuracy (right)

Options

You can configure the algorithm to use different configurations including models, aggregation rules, attacks, computation profiles, Byzantine works, etc.

C - Evaluating runtime and accuracy of HgO and SGD.

  • Configuration
# >> Command
python main.py --model=DNN --dataset=mnist --workers=100 --batch_size=32 --tau=32 --rounds=2001

# >> Configurations 
METRIC = "acc"
EVAL_ROUND = 50
# for DNN devices have more computation compared to MLR otherwis they are enable to participate.
WEAK_DEVICE = [5000, 500]
AVERAGE_DEVICE = [25000, 2500]
POWERFUL_DEVICE = [50000, 5000]

# >> Experiments
args.iid = 0 # Non-IID
args.iid_degree = 1 # Degree of non-iid.ness: 100% Non-IID.
config = [
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 3, 'tau': 32 * 100, 'c': "0.9,0.1,0",
     'legend': r"$SGD, C_{1}, \tau=\infty$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 3, 'tau': 32, 'c': "0.9,0.1,0",
     'legend': r"$SGD, C_{1}, \tau=32$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 3, 'tau': 32, 'c': "0.9,0.1,0",
     'legend': r"$HgO, C_{1}, \tau=32$"},
]
  • Output: Runtime under Non-IID data (left); Runtime under IID data (right)

  • Options

You can configure the algorithm to use different configurations including models, aggregation rules, attacks, computation profiles, Byzantine works, etc.

D - Behavior of HgO in Byzantine scenarios.

  • Configuration
# >> Command
python main.py --model=DNN --dataset=mnist --workers=100 --batch_size=32 --tau=32 --rounds=2001 --attack=FOE
# FOE: Fall of Empire. You can use the Little is enough attack as follow --attack=LIE

# >> Configurations 
METRIC = "acc"
EVAL_ROUND = 50
# for DNN devices have more computation compared to MLR otherwis they are enable to participate.
WEAK_DEVICE = [5000, 500]
AVERAGE_DEVICE = [25000, 2500]
POWERFUL_DEVICE = [50000, 5000]

# >> Experiments
args.iid = 1 #IID
config = [
    {'algo': "SGD", 'f': 30, 'gar': "median", 'lr': 3, 'c': "1,0,0", 'legend': "SGD, $C_{weak}$"},
    {'algo': "SGD", 'f': 30, 'gar': "median", 'lr': 3, 'c': "0.9,0.1,0", 'legend': "SGD, $C_{1}$"},
    {'algo': "SGD", 'f': 30, 'gar': "median", 'lr': 3, 'c': "0,0,1", 'legend': "SGD, $C_{powerful}$"},

    {'algo': "HgO", 'f': 30, 'gar': "median", 'lr': 3, 'c': "1,0,0", 'legend': "HgO, $C_{weak}$"},
    {'algo': "HgO", 'f': 30, 'gar': "median", 'lr': 3, 'c': "0.9,0.1,0", 'legend': "HgO, $C_{1}$"},
    {'algo': "HgO", 'f': 30, 'gar': "median", 'lr': 3, 'c': "0,0,1", 'legend': "HgO, $C_{powerful}$"}
]
  • Output: 30 Byzantine workers (left); 10 Byzantine workers (right)

  • Options

Other aggregation rules are available: tmean, krum or aksel.

You can use a combination of datasets and models to run more experiments on HgO

Reproducing the results of the main paper

Figure 1,4

# >> Command
python main.py --model=MLR --dataset=mnist --workers=100 --batch_size=32 --tau=32 --rounds=150

# >> Configurations 
METRIC = "acc"
EVAL_ROUND = 5
WEAK_DEVICE = [500, 250]
AVERAGE_DEVICE = [10000, 2500]
POWERFUL_DEVICE = [50000, 5000]

# >> Experiments
# ------------ Figure 4(a)
args.iid = 1
args.iid_degree = 0
# OR --------- Figure 1 and 4(b)
args.iid = 0
args.iid_degree = 1

config = [
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "1,0,0", 'legend': "SGD, $C_{weak}$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "SGD, $C_{1}$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0.7,0.3,0", 'legend': "SGD, $C_{2}$"},

    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "1,0,0", 'legend': "HgO, $C_{weak}$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "HgO, $C_{1}$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "0.7,0.3,0", 'legend': "HgO, $C_{2}$"},
]

Figure 3

# >> Command
python main.py --model=MLR --dataset=mnist --workers=100 --batch_size=32 --tau=32 --rounds=150

# >> Configurations 
METRIC = "acc"
EVAL_ROUND = 5
USE_DIFFERENT_HARDWARE = True
WEAK_DEVICE = [500, 250]
AVERAGE_DEVICE = [10000, 2500]
POWERFUL_DEVICE = [50000, 5000]

# >> Experiments
args.iid = 0
args.iid_degree = 1
config = [
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 0.01, 'tau': 32 * 100, 'c': "0.9,0.1,0",
     'legend': r"$SGD, C_{1}, \tau=\infty$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 0.01, 'tau': 32, 'c': "0.9,0.1,0",
     'legend': r"$SGD, C_{1}, \tau=32$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'tau': 32, 'c': "0.9,0.1,0",
     'legend': r"$HgO, C_{1}, \tau=32$"},
]

Figure 5

# >> Command
python main.py --model=MLR --dataset=mnist --workers=100 --batch_size=32 --tau=32 --rounds=150

# >> Configurations 
METRIC = "acc"
EVAL_ROUND = 5
# Weaker devices than preevious experiments.
WEAK_DEVICE = [250, 50]
AVERAGE_DEVICE = [5000, 500]
POWERFUL_DEVICE = [50000, 1000]

# >> Experiments
args.iid = 0
args.iid_degree = 1
config = [
        {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'dynamic': "1,0,0", 'legend': "HgO, $C_{weak}$"},
        {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'dynamic': "0.9,0.1,0", 'legend': "HgO, $C_{1}$"},
        {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'dynamic': "0.7,0.3,0", 'legend': "HgO, $C_{2}$"},
        {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'dynamic': "0,0,1", 'legend': "HgO, $C_{powerful}$"},
    ]

Figure 6 (a)

# >> Command
python main.py --model=MLR --dataset=mnist --workers=100 --batch_size=32 --tau=32 --rounds=150 --attack=FOE

# >> Configurations 
METRIC = "acc"
EVAL_ROUND = 5
WEAK_DEVICE = [500, 250]
AVERAGE_DEVICE = [10000, 2500]
POWERFUL_DEVICE = [50000, 5000]

# >> Experiments
args.iid = 0
args.iid_degree = 1
config = [
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "1,0,0", 'legend': "HgO, $No Attack$"},
    {'algo': "HgO", 'f': 10, 'gar': "median", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "HgO, $f=10$"},
    {'algo': "HgO", 'f': 10, 'gar': "krum", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "HgO, $f=10$"},
    {'algo': "HgO", 'f': 10, 'gar': "aksel", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "HgO, $f=10$"},
]

Figure 6 (b)

# >> Command
python main.py --model=MLR --dataset=mnist --workers=100 --batch_size=32 --tau=32 --rounds=150 --attack=FOE
...
config = [
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "1,0,0", 'legend': "HgO, No Attack"},
    {'algo': "HgO", 'f': 10, 'gar': "median", 'lr': 0.01, 'attack': "FOE", 'c': "0.9,0.1,0", 'legend': "HgO, f=10, FOE"},
    {'algo': "HgO", 'f': 10, 'gar': "median", 'lr': 0.01, 'attack': "LIE", 'c': "0.9,0.1,0", 'legend': "HgO, f=10, LIE"},
]

Figure 6 (c)

# >> Command
python main.py --model=MLR --dataset=mnist --workers=100 --batch_size=32 --tau=32 --rounds=150
...
config = [
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 0.01, 'c': "1,0,0", 'legend': "HgO, $f=0, C_1$"},
    {'algo': "HgO", 'f': 5, 'gar': "median", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "HgO, $f=5, C_1$"},
    {'algo': "HgO", 'f': 10, 'gar': "median", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "HgO, $f=10, C_1$"},
    {'algo': "HgO", 'f': 20, 'gar': "median", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "HgO, $f=20, C_1$"},
    {'algo': "HgO", 'f': 30, 'gar': "median", 'lr': 0.01, 'c': "0.9,0.1,0", 'legend': "HgO, $f=30, C_1$"},
]

Figure 7 (a)

# >> Command
python main.py --model=DNN --dataset=mnist --workers=100 --batch_size=32 --tau=32 --rounds=2001

# >> Configurations 
METRIC = "acc"
EVAL_ROUND = 50
WEAK_DEVICE = [5000, 500]
AVERAGE_DEVICE = [25000, 2500]
POWERFUL_DEVICE = [50000, 5000]

# >> Experiments
args.iid = 0
args.iid_degree = 1
config = [
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 3, 'c': "1,0,0", 'legend': "SGD, $C_{weak}$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 3, 'c': "0.9,0.1,0", 'legend': "SGD, $C_{1}$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 3, 'c': "0.7,0.3,0", 'legend': "SGD, $C_{2}$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 3, 'c': "0,0,1", 'legend': "SGD, $C_{powerful}$"},

    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 3, 'c': "1,0,0", 'legend': "HgO, $C_{weak}$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 3, 'c': "0.9,0.1,0", 'legend': "HgO, $C_{1}$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 3, 'c': "0.7,0.3,0", 'legend': "HgO, $C_{2}$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 3, 'c': "0,0,1", 'legend': "HgO, $C_{powerful}$"},
]

Figure 7 (b)

# >> Command
python main.py --model=DNN --dataset=mnist --workers=100 --batch_size=32 --tau=32 --rounds=2001
...
USE_DIFFERENT_HARDWARE = True
...
config = [
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 3, 'tau': 32 * 100, 'c': "0.9,0.1,0",
     'legend': r"$SGD, C_{1}, \tau=\infty$"},
    {'algo': "SGD", 'f': 0, 'gar': "average", 'lr': 3, 'tau': 32, 'c': "0.9,0.1,0",
     'legend': r"$SGD, C_{1}, \tau=32$"},
    {'algo': "HgO", 'f': 0, 'gar': "average", 'lr': 3, 'tau': 32, 'c': "0.9,0.1,0",
     'legend': r"$HgO, C_{1}, \tau=32$"},
]

End.

hgo's People

Contributors

karimboubouh avatar

Watchers

James Cloos avatar avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.