Git Product home page Git Product logo

distbench's Introduction

Distbench README

Introduction

Distbench is a tool for synthesizing a variety of network traffic patterns used in distributed systems, and evaluating their performance across multiple networking stacks.

A Distbench experiment consists of a single controller process and a set of worker processes. The controller initiates a traffic pattern among the set of workers, monitors traffic and generates a performance report for the RPCs. The traffic pattern, specified spatially and temporally and the RPC stack are all configurable by the controller.

Why Use Distbench ?

Use Distbench to:

  • Identify system bottlenecks for various common distributed computing tasks,
  • Compare RPC stack performance for various common traffic patterns,
  • Evaluate the impact of system configurations, application threading models and kernel configurations on large, scale-out applications.

Included documentation

For contributors, make sure you also consult the following:

distbench's People

Contributors

abhaypatil avatar alexisbarraza3293 avatar ashishvnaik avatar danmanj avatar kanthicn1 avatar kwzhao avatar oserres avatar pedrogzz18 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

paulosjca melodylail andy-intel isabella232 ajunlonglive oserres physolia abhaypatil obedmr test-mass-forker-org-1 mrkatebzadeh monkey-sheng danmanj amanbm alexisbarraza3293 kwzhao teutades wangwei6688

distbench's Issues

simple_test fails on Ubuntu 18.04

I'm new to gRPC, so this may be a "user error" and not a real issue.

On a freshly installed Ubuntu 18.04.5 LTS, I followed the instructions in:
https://github.com/google/distbench/blob/main/docs/getting-started.md

I started a test_sequencer and a node_manager on localhost manually.

bazel run :distbench -- test_sequencer &
bazel run :distbench -- node_manager --test_sequencer=localhost:10000 --port=9999 &

I checked that they are running.

$ sudo netstat -ltnp | grep distbench
tcp6       0      0 :::9999                 :::*                    LISTEN      45149/distbench
tcp6       0      0 :::10000                :::*                    LISTEN      44538/distbench

However, I noticed the following error when I run ./simple_test.sh

Failed! ((grpc::status: failed to connect to all addresses)

Distribution config with CDF points does not work.

I attempted to run some experiments on distbench with a new config that modeled the w1 workload distribution found in the HomaModule (an array of CDFs) but I came across an error that stated "Invalid Argument Error: Add a CDF or PMF to homa_test_w1.config."

Trying to compile with gRPC 1.41.0

Hi,
Not sure what's the best method to reach to you folks, but let me go here. I have an specific requirement to build distbench with gRPC 1.41.0. When I try to build it with the bazel build :all command I'm getting the following error:

Extracting Bazel installation...
Starting local Bazel server and connecting to it...
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
Analyzing: 23 targets (1 packages loaded)
Analyzing: 23 targets (1 packages loaded, 0 targets configured)
Analyzing: 23 targets (34 packages loaded, 36 targets configured)
ERROR: Traceback (most recent call last):
        File "/root/.cache/bazel/_bazel_root/41bc40a3bd5fcb088ad6274fd8e222db/external/build_bazel_rules_apple/apple/internal/testing/ios_rules.bzl", line 62, column 61, in <toplevel>
                ios_ui_test_bundle = rule_factory.create_apple_bundling_rule(
        File "/root/.cache/bazel/_bazel_root/41bc40a3bd5fcb088ad6274fd8e222db/external/build_bazel_rules_apple/apple/internal/rule_factory.bzl", line 904, column 55, in _create_apple_bundling_rule
                rule_attrs.append(_common_binary_linking_attrs(rule_descriptor))
        File "/root/.cache/bazel/_bazel_root/41bc40a3bd5fcb088ad6274fd8e222db/external/build_bazel_rules_apple/apple/internal/rule_factory.bzl", line 218, column 21, in _common_binary_linking_attrs
                apple_common.objc_proto_aspect,
Error: 'apple_common' value has no field or method 'objc_proto_aspect'
INFO: Repository go_sdk instantiated at:
  /distbench/WORKSPACE.bazel:37:16: in <toplevel>
  /root/.cache/bazel/_bazel_root/41bc40a3bd5fcb088ad6274fd8e222db/external/com_github_grpc_grpc/bazel/grpc_extra_deps.bzl:50:27: in grpc_extra_deps
  /root/.cache/bazel/_bazel_root/41bc40a3bd5fcb088ad6274fd8e222db/external/io_bazel_rules_go/go/private/sdk.bzl:268:28: in go_register_toolchains
  /root/.cache/bazel/_bazel_root/41bc40a3bd5fcb088ad6274fd8e222db/external/io_bazel_rules_go/go/private/sdk.bzl:92:21: in go_download_sdk
Repository rule _go_download_sdk defined at:
  /root/.cache/bazel/_bazel_root/41bc40a3bd5fcb088ad6274fd8e222db/external/io_bazel_rules_go/go/private/sdk.bzl:79:35: in <toplevel>
ERROR: /distbench/BUILD:191:16: error loading package '@com_github_grpc_grpc//src/compiler': at /root/.cache/bazel/_bazel_root/41bc40a3bd5fcb088ad6274fd8e222db/external/com_github_grpc_grpc/bazel/grpc_build_system.bzl:29:6: at /root/.cache/bazel/_bazel_root/41bc40a3bd5fcb088ad6274fd8e222db/external/build_bazel_rules_apple/apple/ios.bzl:22:5: initialization of module 'apple/internal/testing/ios_rules.bzl' failed and referenced by '//:_distbench_cc_grpc_proto_grpc_codegen'
ERROR: Analysis of target '//:_distbench_cc_grpc_proto_grpc_codegen' failed; build aborted: Analysis failed
INFO: Elapsed time: 26.284s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (34 packages loaded, 36 targets configured)
FAILED: Build did NOT complete successfully (34 packages loaded, 36 targets configured)

I modified the WORKSPACE.bazel in order to point to my custom gRPC. Below my change:

diff --git a/WORKSPACE.bazel b/WORKSPACE.bazel
index d35ccd8..3d919e5 100644
--- a/WORKSPACE.bazel
+++ b/WORKSPACE.bazel
@@ -16,13 +16,20 @@ rules_proto_toolchains()

 ##### gRPC Rules for Bazel
 http_archive(
-    name = "com_github_grpc_grpc",
+    name = "com_github_grpc_grpc2",
     sha256 = "bba65891b8c8eaea9f10b0ccccb4b6173590aee36937f5b52e8665720bcaa4c1",
     urls = [
         "https://github.com/grpc/grpc/archive/b0f37a22bbae12a4b225a88be6d18d5e41dce881.tar.gz",  # Release 1.43
     ],
     strip_prefix = "grpc-b0f37a22bbae12a4b225a88be6d18d5e41dce881",
 )
+
+new_local_repository(
+    name = "com_github_grpc_grpc",
+    path = "/custom-grpc",
+    build_file = "BUILD",
+)
+
 load("@com_github_grpc_grpc//bazel:grpc_deps.bzl", "grpc_deps")
 grpc_deps()

It seems to me like an Apple specific build, since I'm only interested in a Linux-only build, is there a way to overcome this?

In case that you're interested in how I'm building it, below the Dockerfile I'm playing with:

FROM archlinux

RUN pacman -Syu --noconfirm bazel git gcc which python
RUN git clone https://github.com/google/distbench.git
WORKDIR distbench
# Build
COPY custom-grpc /custom-grpc
COPY workspace.diff /

# Apply patches
RUN git apply /workspace.diff

RUN bazel build :all

Thanks in advance for your help.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.