Git Product home page Git Product logo

fleetspeak's Introduction

Fleetspeak

.github/workflows/build.yml Go Report Card

Fleetspeak is a framework for communicating with a fleet of machines, with a focus on security monitoring and basic administrative use cases. It is a subproject of GRR, and can be seen as an effort to modularizing and modernizing its communication mechanism.

Status

We have this code working internally as part of our GRR installation.

Getting Started

On linux, assuming a recent version of the go development environment (see the go.mod file for the exact requirement) and virtualenv, the following sequence of commands will build and test this pre-release:

go get -v -t github.com/google/fleetspeak/...

# Assuming default $GOPATH:
cd ~/go/src/github.com/google/fleetspeak

# Setup virtualenv - fleetspeak provides some python integration libraries,
# and this ensures they are set up in a known way.
virtualenv $HOME/FSENV
source $HOME/FSENV/bin/activate

# Pre-requisite to installing Fleetspeak (used to compile protos).
pip install grpcio-tools

pip install -e fleetspeak_python/

# Set mysql parameters. The mysql datastore test will run if the following environment
# variables are set. Otherwise it will be skipped.
export MYSQL_TEST_USER=<username>
export MYSQL_TEST_PASS=<password>   # will assume null password if unset.
export MYSQL_TEST_ADDR=<host:port>

# Build and test the release:
fleetspeak/build.sh
fleetspeak/test.sh

Once built, you can take a look at the files and instructions in our demo directory.

DISCLAIMER

While the code presented here is in some sense feature complete, much of it is barely tested or documented, and breaking changes are still possible. Therefore, please consider this a preview release while the dust settles. Suggestions and pull requests are very much appreciated.

fleetspeak's People

Contributors

alexandr-ts avatar alrs avatar bgalehouse avatar bjackman avatar bugos avatar coperni avatar daschwanden avatar gnoack avatar golint-fixer avatar kant avatar max-vogler avatar mbushkov avatar mlkm avatar mol123 avatar ogarod avatar panhania avatar smu-ggl avatar the80srobot avatar torsm avatar tsehori avatar vpasdf avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fleetspeak's Issues

Productionize mysql datastore.

The mysql datastore is still a relatively simple branch of the sqlite datastore, and therefore was not engineered with performance in mind.

We should make a pass implementing the obvious optimizations and then run a load test to find any further low hanging fruit performance wise.

Add CI tests to check that all proto files are generated with the same version

We want to deal with the possible inconsistency between versions of protocol buffer generation using protoc-gen-go, as developers that use different versions may hit issues when building and/or testing their Fleetspeak deployments.

To overcome this, we want to add tests to Travis CI that regenerate the proto files using a fixed version of protoc-gen-go and diff them with the generated files in submitted pull requests. This can be done by using git diff that should be accessible through Travis CI.

Fix admin/cli library interface.

When adding to the admin/cli library, it is necessary to update Usage(), and necessary to manually add the functionality to any derived command line tools (e.g. demo/admin_cli).

This issue is to clean this up.

Suggested resolution is to restructure the cli library to define a command struct or interface, so that each operation's usage line can be defined next to its implementation. Then it can export a collection of known commands for derived tools to use.

clientConfig.GetFilesystemHandler() may return nil

func (x *Config) GetFilesystemHandler() *FilesystemHandler {
if x, ok := x.GetPersistenceHandler().(*Config_FilesystemHandler); ok {
return x.FilesystemHandler
}
return nil
}

Therefore, these may cause nil pointer errors.

clientConfig.GetFilesystemHandler().ConfigurationDirectory = configDirForClient
clientConfig.GetFilesystemHandler().StateFile = newStateFilePath
clientConfig.Server = []string{httpsListenAddress}
_, err = os.Create(clientConfig.GetFilesystemHandler().StateFile)

[mysql] Fleetspeak log flooded with "Can't create more than max_prepared_stmt_count statements"

In a bundled fleetspeak installation, this generates 5GB of logs in a few days, causing the machine to constantly run out of disk space.

E0829 21:22:57.212232    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)
E0829 21:23:08.611544    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)
E0829 21:23:08.612796    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)
E0829 21:23:08.613613    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)
E0829 21:23:37.111681    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)
E0829 21:24:03.511595    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)
E0829 21:24:03.513018    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)
E0829 21:24:03.513594    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)
E0829 21:24:07.711622    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)
E0829 21:24:07.712933    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)
E0829 21:24:27.211502    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)
E0829 21:24:27.212749    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)
E0829 21:24:29.911775    1796 messagestore.go:827] Failed to read server messages for processing: Error 1461: Can't create more than max_prepared_stmt_count statements (current value: 16382)

Default Notification Support

In order for streaming connection to work correctly when running multiple Fleetspeak server, they need to be able to find each other using matched Listener/Notifier components.

We need to implement a Listener/Notifier pair for multi-server situations, using, e.g. http(s).

Clean up dbtesting.

The dbtesting.go library has a few flaws:
-It requires every implementation test to call several methods to fully test a database, creating the maintenance burden.
-This result is several overlarge unit tests for each datastore.

To clean this issue up, it is suggested for dbtesting to define a single 'test my datastore' method and use T.Run(...) to divide this up into much smaller subtests. Bonus points for using T.Parallel() to make the resulting test suite run faster overall.

Refactor fetchResourceUsageRecordsTest to not depend on FakeTime

fetchResourceUsageRecordsTest relies on FakeTime, assuming that FakeTime affects resource usage records' server timestamps.

Even though this might be true for existing MySQL and sqlite datastore implementations, it's very much possible to have a datastore implementation that relies on the database server to provide server timestamps. In this case, the test would fail, since real (and not fake) timestamps will be written to the datastore, breaking the test's assumptions.

The test should be refactored along the lines of:

  1. We write a number of resource usage records to the datastore.
  2. We read these resource usage records back and use their timestamps to test that the filtering works as expected.

Fix sending of StdOutput messages to the server

Currently, the Fleetspeak client collects output from a daemon process's stdout and stderr streams at regular intervals and attempts to send that to the server (see client/daemonservice/execution/execution.go::flushOut()). However, that doesn't quite work because the messages do not have a destination set (an error is returned, which gets ignored).

We need to fix this as follows:

  • Add an option to the service configuration that controls whether to send output from stdout, stderr, or both to the server
  • Add a destination to the messages, so they actually get sent to the server. The destination will be configured in the service config.

Fleetspeak packaging and versioning.

We should provide versioned packages for general purpose fleetspeak servers and clients, with the understanding that sophisticated users may want to add additional code specific to their installation.

Previous PRs moving in this direction:
#130 #129 #128

Implement `readonly` flag in MySQL data store.

In the MySQL datastore, the handling of the readonly flag should be implemented in runInTx.

At least, recent versions of the golang sql package provide a way to pass it in to BeginTx. We don't know how complete the actual mysql driver support is.

Implement TLS encryption on MySQL connections

Hello,

In order to secure the connections made via MySQL, I think it could be good to allow the possibility to do TLS on MySQL connections.
This connection can be made from the module already imported via RegisterTLSConfig.

I'll try to look into implementing it but in the meantime I'm creating an issue to keep the idea in mind.

Thank you,

Build is failing on Windows

Build is failing on Windows

https://github.com/google/fleetspeak/actions/runs/7888323141/job/21525568468?pr=457

Related to

Run # TODO: The dependency installation fails, but good enough to make the tests pass.
go: modules disabled by GO111MODULE=off; see 'go help modules'
Dependency installation failed, continuing anyway ¯\_(ツ)_/¯

This is probably because the build started using Go 1.22 and Fleetspeak is doing something nonstandard when building... We should migrate to proper Go modules

https://go.dev/doc/go1.22#go-command

go get is no longer supported outside of a module in the legacy GOPATH mode (that is, with GO111MODULE=off). Other build commands, such as go build and go test, will continue to work indefinitely for legacy GOPATH programs.

Client acks messages sent between services on the client.

On the client, when service A sends a message to service B (on the same client), the fleetspeak client will send an ACK to the server.

This results in the following logs:

W0901 16:26:11.600130   25931 manager.go:209] system: Temporary error processing message 97e7d6d3e241aaf07944a169d5601d2d39ea62e58b22dfb39576fe015feefe90, will retry: unable to retrieve messages to ack: sql: no rows in result set
W0901 16:26:11.600792   25931 manager.go:209] system: Temporary error processing message 72739dbc6cbfe90879fdc89bcd9f20b05b706e21471a83a402db7f32349636b7, will retry: unable to retrieve messages to ack: sql: no rows in result set
W0901 16:26:12.802231   25931 manager.go:209] system: Temporary error processing message 936cddf4fea77cc8525cfa9273d5f8e65a158abd9e73cd1ae9bc21d9a0e9ce10, will retry: unable to retrieve messages to ack: sql: no rows in result set
W0901 16:26:13.099519   25931 manager.go:209] system: Temporary error processing message fa734a308287b58bf618fa768b54108e54a69e0e81162adadefd7f3b8eeb3193, will retry: unable to retrieve messages to ack: sql: no rows in result set
W0901 16:26:14.601890   25931 manager.go:209] system: Temporary error processing message 129c170732eae60c1738ea92caa4893649b4238a279025e4af802f877bd6109b, will retry: unable to retrieve messages to ack: sql: no rows in result set
W0901 16:26:14.899605   25931 manager.go:209] system: Temporary error processing message e707f680871c375adea0dd13d20c2829b83eab1ae3c6ccd59b7c7d9bf7431f51, will retry: unable to retrieve messages to ack: sql: no rows in result set
W0901 16:26:15.499319   25931 manager.go:209] system: Temporary error processing message 47812d87eae92816856bd4101ffe5cdeb58f533eca7ddbaf49d0fbd38e278c25, will retry: unable to retrieve messages to ack: sql: no rows in result set
W0901 16:26:16.699828   25931 manager.go:209] system: Temporary error processing message 33a77e4c31b6ea3679c0fafc1c40c8615078e32d6eec73f72c8d0a5c6bf2f0a4, will retry: unable to retrieve messages to ack: sql: no rows in result set

The MessageAckData contains empty message IDs.

{       # (null) size=120B
  tag1_message: {       # (null) size=0B
  }     # tag1_message[0]
  tag1_message: {       # (null) size=0B
  }     # tag1_message[1]
  tag1_message: {       # (null) size=0B
  }     # tag1_message[2]
  tag1_message: {       # (null) size=0B
  }     # tag1_message[3]
  tag1_message: {       # (null) size=0B
  }     # tag1_message[4]
[...]

No Acks should be sent in this case, special handling should be implemented around here:

if m.MessageType == "Die" && m.Destination != nil && m.Destination.ServiceName == "system" {

fleetspeakd gets stuck on stopping

when turning off my system, fleetspeakd hangs for 1:30 until it gets killed:

journalctl --boot=-1 -u fleetspeakd
-- Logs begin at Thu 2020-06-04 08:02:03 CEST, end at Wed 2020-06-24 09:55:26 CEST. --
Jun 24 09:06:01 ensonic systemd[1]: Started fleetspeakd.
Jun 24 09:06:39 ensonic fleetspeakd[1211]: ToolLogProto <redacted: ask if this info is is needed>
Jun 24 09:07:35 ensonic fleetspeakd.nanny[1211]: E0624 09:07:35.416194    1211 channel.go:157] error reading magic number: EOF
Jun 24 09:07:35 ensonic fleetspeakd.nanny[1211]: E0624 09:07:35.423312    1211 execution.go:494] channel produced error: error reading magic number: EOF
Jun 24 09:08:06 ensonic fleetspeakd.nanny[1211]: E0624 09:08:06.015727    1211 netdetect.go:187] checkUberproxyURI: error with HEAD request: write tcp 192.168.86.32:39216->74.125.128.129:443: write: broken pipe
Jun 24 09:11:45 ensonic fleetspeakd.nanny[1211]: E0624 09:11:45.115879    1211 execution.go:494] channel produced error: read of length 4 timed out
Jun 24 09:11:48 ensonic fleetspeakd.nanny[1211]: E0624 09:11:48.715772    1211 channel.go:157] error reading magic number: timed out
Jun 24 09:13:06 ensonic systemd[1]: Stopping fleetspeakd...
Jun 24 09:13:06 ensonic fleetspeakd.nanny[1211]: E0624 09:13:06.818931    1211 channel.go:157] error reading magic number: EOF
Jun 24 09:13:06 ensonic fleetspeakd.nanny[1211]: E0624 09:13:06.819025    1211 execution.go:494] channel produced error: error reading magic number: EOF

Record network usage of a Fleetspeak client (Bytes sent/received)

We want to record the network usage of a single Fleetspeak client deployed, so that we can later on plot, inspect and analyze this data for measuring performance and be alerted if there are any unexpected "spikes" in network usage.
An example for a use case is GRR, that has a dashboard for Client Load Stats (google/grr#859) which fetches its data from Fleetspeak server's datastore.

The concept is similar to other metrics gathered from Fleetspeak clients, such as resident memory size.
As an example, check out #275 to learn more about how IO MiB read/write is implemented.

Implementation details:
Other metrics gathered for individual Fleetspeak clients are using the ResourceUsageFetcher struct, and specifically the method ResourceUsageForPID. Unlike those metrics, the kernel does not have information about the network usage of a given process PID, so this cannot be fetched using the library gopsutil.
Thus, an idea for implementing this: add a new struct (a singleton effectively) that will become a field in ResourceUsageFetcher. This new singleton will also be shared with the StreamingCommunicator struct (note that an implementation may needed for polling as well), so network usage data can be recorded when messages enter/exit the Fleetspeak client in the methods readLoop and writeLoop, where both are called from the connect method of StreamingCommunicator.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.