Git Product home page Git Product logo

skydive-project / skydive Goto Github PK

View Code? Open in Web Editor NEW
2.6K 101.0 395.0 74.8 MB

An open source real-time network topology and protocols analyzer

Home Page: https://skydive.network

License: Apache License 2.0

Go 64.21% Shell 2.65% HTML 0.18% Makefile 0.76% CSS 1.56% JavaScript 25.43% Python 3.04% C 1.05% Nix 0.02% Roff 0.17% TypeScript 0.72% Dockerfile 0.03% Mustache 0.15% Jinja 0.02%
networking network-topology flow monitoring traffic-analysis traffic-monitoring metrics packet-sniffer packet-analyzer pcap

skydive's Introduction

GitHub license Slack Invite Slack Channel Weekly minutes Go Report Card CII Best Practices StackShare PyPI

Skydive

Skydive is an open source real-time network topology and protocols analyzer. It aims to provide a comprehensive way of understanding what is happening in the network infrastructure.

Skydive agents collect topology information and flows and forward them to a central agent for further analysis. All the information is stored in an Elasticsearch database.

Skydive is SDN-agnostic but provides SDN drivers in order to enhance the topology and flows information.

Key features

  • Captures network topology and flows
  • Full history of network topology and flows
  • Distributed
  • Ability to follow a flow along a path in the topology
  • Supports VMs and Containers infrastructure
  • Unified query language for topology and flows (Gremlin)
  • Web and command line interfaces
  • REST API
  • Easy to deploy (standalone executable)
  • Connectors to OpenStack, Docker, OpenContrail, Kubernetes

Quick start

All-in-one

The easiest way to get started is to download the latest binary and to run it using the all-in-one mode :

curl -Lo - https://github.com/skydive-project/skydive-binaries/raw/jenkins-builds/skydive-latest.gz | gzip -d > skydive && chmod +x skydive && sudo mv skydive /usr/local/bin/

SKYDIVE_ETCD_DATA_DIR=/tmp SKYDIVE_ANALYZER_LISTEN=0.0.0.0:8082 sudo -E /usr/local/bin/skydive allinone

Open a browser to http://localhost:8082 to access the analyzer Web UI.

Helm

If you are using Kubernetes then you can deploy skydive using helm directly from Git:

helm plugin install https://github.com/aslafy-z/helm-git --version 0.10.0
helm repo add skydive git+https://github.com/skydive-project/skydive@contrib/charts
helm repo update
helm install skydive-analyzer skydive/skydive-analyzer
helm install skydive-agent skydive/skydive-agent
kubectl port-forward service/skydive-analyzer 8082:8082

Open a browser to http://localhost:8082 to access the analyzer Web UI.

Docker

docker run -d --privileged --pid=host --net=host -p 8082:8082 -p 8081:8081 \
    -e SKYDIVE_ANALYZER_LISTEN=0.0.0.0:8082 \
    -v /var/run/docker.sock:/var/run/docker.sock -v /run/netns:/var/run/netns \
    skydive/skydive allinone

Open a browser to http://localhost:8082 to access the analyzer Web UI.

Docker Compose

To quick set up a more complete working environment (with history support), Docker Compose can be used to automatically start an Elasticsearch container, a Skydive analyzer container and a Skydive agent container.

curl -o docker-compose.yml https://raw.githubusercontent.com/skydive-project/skydive/master/contrib/docker/docker-compose.yml
docker-compose up

You can also use the Skydive command line client with:

docker run --net=host -ti skydive/skydive client query "g.V()"

Open a browser to http://localhost:8082 to access the analyzer Web UI.

Documentation

Skydive documentation can be found here:

The Skydive REST API is described using swagger here.

Tutorials

Skydive tutorials can be found here:

Get involved

Contributing

Your contributions are more than welcome. Please check https://github.com/skydive-project/skydive/blob/master/CONTRIBUTING.md to know about the process.

License

This software is licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

skydive's People

Contributors

adrianlzt avatar amorenoz avatar andrekassis avatar dubek avatar dvandra avatar eonpatapon avatar eranra avatar esnible avatar fdebonneval avatar grooverdan avatar hunchback avatar kalmanmeth avatar kdorfman avatar lebauce avatar masco avatar matrohon avatar mestery avatar nlewo avatar nplanel avatar orozery avatar patsevanton avatar pierrecregut avatar roytman avatar russellb avatar safchain avatar sh118415 avatar tomerlange avatar trgill avatar vperelmax avatar yanivbi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

skydive's Issues

Add Flows().Has validator

Currently the "Has" expression is sent directly to the agents or the storage. It could improve the performances to check that the expression matches the Flow definition before sending the request. So that the expressions not matching the flow definition or using a wrong type for the value will not be executed on each flow.

Improve traversal step reduce

At the moment, we iterate through all the step and check if a step can be reduced with the next one.

Let's consider we have the steps : A -> B -> C
If B -> C can be reduced to C, we'll have : A -> C

With the current code, we don't check that A -> C can be reduced.

Flatten Flow structure

Currently flows are made of embed structure for statistics, keeping a statistics per layer. The goal here is to flatten the flow structure to keep only one level of statistics. The will help to improve the performance. Moreover one packet will possibly generate more than one flow depending on the encapsulation. The flows generated could be linked.

Timeout with 'G.V().Flows()' when no agent

The gremlin request 'G.V().Flows()' return an empty list after the timeout expiration without reporting any error. When there is no agent connected to the analyzer, for example due to bad etcd configuration.

Add netlink Statistics to nodes

Add Statistics returned by netlink to interface nodes so that we can get dropped packet number, etc...
This need to add an async mechanism to get metrics in order to have them updated.

etcd initialization error

I'm getting errors while starting th eanalyzer

$ skydive analyzer --conf /etc/skydive/skydive.yml
2016-03-03T15:54:43.768Z devstack:skydive analyzer.go:46 analyzer glob.func1 > NOTI 005 Skydive Analyzer starting...
2016-03-03 15:54:43.768837 I | etcdserver: name = skydive
2016-03-03 15:54:43.768900 I | etcdserver: data dir = /tmp/skydive-etcd
2016-03-03 15:54:43.768994 I | etcdserver: member dir = /tmp/skydive-etcd/member
2016-03-03 15:54:43.769056 I | etcdserver: heartbeat = 100ms
2016-03-03 15:54:43.769142 I | etcdserver: election = 1000ms
2016-03-03 15:54:43.769202 I | etcdserver: snapshot count = 0
2016-03-03 15:54:43.769263 I | etcdserver: advertise client URLs = http://192.168.122.17:2379
2016-03-03 15:54:43.786243 I | etcdserver: restarting member cc743f7e0a51972 in cluster 1ddb10353625184f at commit index 4
2016-03-03 15:54:43.786392 I | raft: cc743f7e0a51972 became follower at term 2
2016-03-03 15:54:43.786503 I | raft: newRaft cc743f7e0a51972 [peers: [], term: 2, commit: 4, applied: 0, lastindex: 4, lastterm: 2]
2016-03-03 15:54:43.786974 I | etcdserver: set snapshot count to default 10000
2016-03-03 15:54:43.787072 I | etcdserver: starting server... [version: 2.3.0-alpha.0+git, cluster version: to_be_decided]
2016-03-03 15:54:43.797405 N | etcdserver: added local member cc743f7e0a51972 [http://localhost:2379] to cluster 1ddb10353625184f
2016-03-03T15:54:44.799Z devstack:skydive analyzer.go:52 analyzer glob.func1 > CRIT 006 Can't start Analyzer : client: etcd cluster is unavailable or misconfigured

It seems a note on etcd configuration is missing on the "install guide", or am I doing something wrong?

Trying to lookup flow only on supported nodes

Currently when we do a lookup for flow we lookup for all the graph nodes but only some of them can have link to flows. For example host node or netns node can't have link to flows. It makes sense to not send UUID of these nodes when doing lookups.

Use analyzers address when no etcd parameter provided

Currently we have to set the etcd servers parameter is the config file even if we are using the embed etcd. It would be better to use the analyzers list as etcd servers list when this list is not specified is the config file.

Add a Gremlin 'Sort' step

Right now, the flows are sorted on the agent when they are queried. This is time consuming and should be optional.

Add a reference on node to capture UUID

Currently we only know when there is capture on a node thanks to the State.FlowCapture metadata entry. It could be useful to know the UUID of the capture attached.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.