Git Product home page Git Product logo

rabbitmq-trust-store's Introduction

RabbitMQ Certificate Trust Store

This repository has been moved to the main unified RabbitMQ "monorepo", including all open issues. You can find the source under /deps/rabbitmq_trust_store. All issues have been transferred.

Overview

This plugin provides an alternative TLS (x.509) certificate verification strategy. Instead of the traditional PKI chain traversal mechanism, this strategy simply checks the leaf certificate presented by a client against an approved ("whitelisted") set of certificates.

Rationale

When RabbitMQ is configured to use TLS for client connections, by default it will use the standard PKI certificate chain traversal process. What certificates are trusted is ultimately controlled by adjusting the set of trusted CA certificates.

This configuration is standard for data services and it works well for many use cases. However, this configuration is largely static: a change in trusted CA certificates requires a cluster reconfiguration (redeployment). There is no convenient means with which to change the set in realtime, and in particular without affecting existing client connections.

This plugin offers an alternative. It maintains a set (list) of trusted .PEM formatted TLS (x509) certificates, refreshed at configurable intervals. Said certificates are then used to verify inbound TLS-enabled client connections across the entire RabbitMQ node (affects all plugins and protocols). The set is node-local.

Certificates can be loaded into the trusted list from different sources. Sources are loaded using "providers". Two providers ship with the plugin: the local filesystem and an HTTPS endpoint.

New providers can be added by implementing the rabbit_trust_store_certificate_provider behaviour.

The default provider is rabbit_trust_store_file_provider, which will load certificates from a configured local filesystem directory.

Installation

This plugin ships with modern RabbitMQ versions. Like all plugins, it has to be enabled before it can be used:

rabbitmq-plugins enable rabbitmq_trust_store

Usage

Filesystem provider

Configure the trust store with a directory of whitelisted certificates and a refresh interval:

## trusted certificate directory path
trust_store.directory        = $HOME/rabbit/whitelist
trust_store.refresh_interval = 30

Setting refresh_interval to 0 seconds will disable automatic refresh.

Certificates are identified and distinguished by their filenames, file modification time and a hash value of file contents.

Installing a Certificate

Write a PEM formatted certificate file to the configured directory to whitelist it. This contains all the necessary information to authorize a client which presents the very same certificate to the server.

Removing a Certificate

Delete the certificate file from the configured directory to remove it from the whitelist.

Note: TLS session caching bypasses the trust store certificate validation and can make it seem as if a removed certificate is still active. Disabling session caching in the broker by setting the reuse_sessions ssl option to false can be done if timely certificate removal is important.

HTTP provider

HTTP provider loads certificates via HTTP(S) from remote server.

The server should have following API:

  • GET <root> - list certificates in JSON format: {"certificates": [{"id": <id>, "path": <url>}, ...]}
  • GET <root>/<path> - download PEM encoded certificate.

Where <root> is a configured certificate path, <id> - unique certificate identifier, <path> - relative certificate path to load it from server.

Configuration of the HTTP provider:

trust_store.providers.1      = http
trust_store.url              = https://example.cert.url/path
trust_store.refresh_interval = 30

The example above uses an alias, http for rabbit_trust_store_http_provider. Available aliases are:

  • file - rabbit_trust_store_file_provider
  • http - rabbit_trust_store_http_provider

In the erlang terms format:

{rabbitmq_trust_store,
 [{providers, [rabbit_trust_store_http_provider]},
  {url, "https://example.cert.url/path"},
  {refresh_interval, {seconds, 30}}
 ]}.

You can specify TLS options if you use HTTPS:

trust_store.providers.1      = http
trust_store.url              = https://example.secure.cert.url/path
trust_store.refresh_interval = 30
trust_store.ssl_options.certfile   = /client/cert.pem
trust_store.ssl_options.keyfile    = /client/key.pem
trust_store.ssl_options.cacertfile = /ca/cert.pem

In the erlang terms format:

{rabbitmq_trust_store,
 [{providers, [rabbit_trust_store_http_provider]},
  {url, "https://example.secure.cert.url/path"},
  {refresh_interval, {seconds, 30}},
  {ssl_options, [{certfile, "/client/cert.pem"},
                 {keyfile, "/client/key.pem"},
                 {cacertfile, "/ca/cert.pem"}
                ]}
 ]}.

HTTP provider uses If-Modified-Since during list request header to avoid updating unchanged list of certificates.

You can additionally specify headers (e.g. authorization) using Erlang term format:

{rabbitmq_trust_store,
 [{providers, [rabbit_trust_store_http_provider]},
  {url, "https://example.cert.url/path"},
  {headers, [{"Authorization", "Bearer token"}]},
  {refresh_interval, {seconds, 30}}
 ]}.

Example

examples/rabbitmq_trust_store_django is an example Django application, which serves certificates from a directory.

Listing certificates

To list the currently loaded certificates use the rabbitmqctl utility as follows:

    rabbitmqctl eval 'io:format(rabbit_trust_store:list()).'

This will output a formatted list of certificates similar to:

    Name: cert.pem
    Serial: 1 | 0x1
    Subject: O=client,CN=snowman.local
    Issuer: L=87613,CN=MyTestRootCA
    Validity: "2016-05-24T15:28:25Z - 2026-05-22T15:28:25Z"

Note that this command reads each certificate from disk in order to extract all the relevant information. If there are a large number of certificates in the trust store use this command sparingly.

How it Works

When the trust-store starts it configures TLS listening sockets, whitelists the certificates in the given directory, then accepting sockets can query the trust-store with their client's certificate. It refreshes the whitelist to correspond with changes in the directory's contents, installing and removing certificate details, after a refresh interval or a manual refresh (by invoking a rabbitmqctl eval 'rabbit_trust_store:refresh().' from the commandline).

Building from Source

See Plugin Development guide.

TL;DR: running

make dist

will build the plugin and put build artifacts under the ./plugins directory.

Copyright and License

(c) 2007-2020 VMware, Inc. or its affiliates.

Released under the MPL, the same license as RabbitMQ.

rabbitmq-trust-store's People

Contributors

acogoluegnes avatar ayanda-d avatar dcorbacho avatar dumbbell avatar gerhard avatar hairyhum avatar kjnilsson avatar lukebakken avatar mgrafl avatar michaelklishin avatar pjk25 avatar spring-operator avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rabbitmq-trust-store's Issues

Missing carriage return when printing cert loaded message

For instance:

=INFO REPORT==== 3-Aug-2016::17:34:24 ===
trust store: loading certificate 'trusttest-cert.pem' <<<< no blank line right here
=INFO REPORT==== 3-Aug-2016::17:34:47 ===
accepting AMQP connection <0.2618.0> (10.10.66.21:64934 -> 10.101.39.99:5671)

Plugin runs into an exception on invalid/unparseable HTTP response body

The rabbitmq-trust-store plugin crashes if the JSON object of the HTTP response cannot be parsed.

Stack trace:

2019-11-15 13:30:17.172 [error] <0.599.0> ** Generic server trust_store terminating
** Last message in was refresh
** When Server state == {state,[{rabbit_trust_store_http_provider,{http_state,"https://url.to.truststore/path/to/trust_store",[],[{"If-Modified-Since","Fri, 15 Nov 2019 12:29:15 GMT"}]}}],30000}
** Reason for termination ==
** {function_clause,[{rabbit_trust_store,terminate,[{badarg,[{jsx_decoder,incomplete,6,[{file,"src/jsx_decoder.erl"},{line,141}]},{rabbit_trust_store_http_provider,decode_cert_list,1,[{file,"src/rabbit_trust_store_http_provider.erl"},{line,76}]},{rabbit_trust_store_http_provider,list_certs,2,[{file,"src/rabbit_trust_store_http_provider.erl"},{line,28}]},{rabbit_trust_store,refresh_provider_certs,3,[{file,"src/rabbit_trust_store.erl"},{line,228}]},{rabbit_trust_store,'-refresh_certs/2-fun-0-',4,[{file,"src/rabbit_trust_store.erl"},{line,217}]},{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},{rabbit_trust_store,handle_info,2,[{file,"src/rabbit_trust_store.erl"},{line,169}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,637}]}]},{state,[{rabbit_trust_store_http_provider,{http_state,"https://url.to.truststore/path/to/trust_store",[],[{"If-Modified-Since","Fri, 15 Nov 2019 12:29:15 GMT"}]}}],30000}],[{file,"src/rabbit_trust_store.erl"},{line,175}]},{gen_server,try_terminate,3,[{file,"gen_server.erl"},{line,673}]},{gen_server,terminate,10,[{file,"gen_server.erl"},{line,858}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-11-15 13:30:17.173 [error] <0.599.0> CRASH REPORT Process trust_store with 0 neighbours crashed with reason: no function clause matching rabbit_trust_store:terminate({badarg,[{jsx_decoder,incomplete,6,[{file,"src/jsx_decoder.erl"},{line,141}]},{rabbit_trust_store_http_provider,...},...]}, {state,[{rabbit_trust_store_http_provider,{http_state,"https://url.to.truststore/...",...}}],...}) line 175
2019-11-15 13:30:17.173 [error] <0.598.0> Supervisor rabbit_trust_store_sup had child trust_store started with rabbit_trust_store:start_link() at <0.599.0> exit with reason no function clause matching rabbit_trust_store:terminate({badarg,[{jsx_decoder,incomplete,6,[{file,"src/jsx_decoder.erl"},{line,141}]},{rabbit_trust_store_http_provider,...},...]}, {state,[{rabbit_trust_store_http_provider,{http_state,"https://url.to.truststore/...",...}}],...}) line 175 in context child_terminated
2019-11-15 13:30:17.255 [error] <0.30984.12> CRASH REPORT Process <0.30984.12> with 0 neighbours crashed with reason: bad argument in jsx_decoder:incomplete/6 line 141
2019-11-15 13:30:17.256 [error] <0.598.0> Supervisor rabbit_trust_store_sup had child trust_store started with rabbit_trust_store:start_link() at <0.599.0> exit with reason bad argument in jsx_decoder:incomplete/6 line 141 in context start_error
2019-11-15 13:30:17.256 [error] <0.598.0> Supervisor rabbit_trust_store_sup had child trust_store started with rabbit_trust_store:start_link() at {restarting,<0.599.0>} exit with reason reached_max_restart_intensity in context shutdown
2019-11-15 13:30:17.257 [info] <0.42.0> Application rabbitmq_trust_store exited with reason: shutdown

Reproduction:

  • Environment: RabbitMQ 3.7.16 on Erlang 21.0.9 running on CentOS 7.
  • Unfortunately, the offending invalid HTTP response body is not available.

Setting Application Environment with Options can Fail on Erlang =< R16B03

Passing options along with application:set_env/4 fails on R16B03 which is what dpkg built Rabbit with for the 3.6.1 Debian package on a Trusty (64).

E.g.:

$ erl
Erlang R16B03 (erts-5.10.4) [source] [64-bit] [async-threads:10] [kernel-poll:false]

Eshell V5.10.4  (abort with ^G)
1> application:set_env(foo, bar, [{baz, brain}]). %% This is okay BUT ...
ok
2> application:set_env(foo, bar, [{alice, wonderland}], [{persistent, true}]). %% ... passing a property-list of options will fail.
** exception error: no function clause matching application:set_env(foo,bar,[{alice,wonderland}],[{persistent,true}]) (application.erl, line 294)
3>

The Trust-Store does this here.

Plug-in will terminate (stop working) if it fails to load a cert

If my white-listing directory contains a bad cert that the plug-in cannot read for whatever reason, the plug-in will terminate, and not allow other certs to be used. This is a denial of service condition. It would be best if the plug-in logged and ignored/skipped bad certs rather than quitting.

Shovel client doesn't seems to use the trust store logic

Hi Team,

Context
I m doing a shovel to a another broker AB behind a load balancer where the TLS is terminated.
The LB is not sending all it's certificate chain. The last one is not-self signed.
So my shovel ,on the client side, cannot verify the AB using the standard erlang process.
From the doc:
No trusted CA was found in the trusted store. The trusted CA is normally a so called ROOT CA, which is a self-signed certificate. Trust can be claimed for an intermediate CA (trusted anchor does not have to be self-signed according to X-509) by using option partial_chain.

So I had a look at partial_chain option and i have decided to leverage on this plugin that is implementing this function.

issue

I have installed the plugin and put directly the peer certificate (AB) into my trust store directory.
I have check if they are correctly loaded using rabbitmqctl eval 'io:format(rabbit_trust_store:list()).'
I have set up the shovel through the ui with this uri: amqps://[masked]@[masked]?verify=verify_peer&fail_if_no_peer_cert=true&server_name_indication=disable

I was expected the shovel, to use the plugin to do the verification.
From the log i can see (URI: amqps://[URL]): {options,{cacertfile,[]}}
So i m wondering if it's used the plugin ?

Any idea ?

Log which certs and added or removed

Would it be possible for audit and general troubleshooting purposes to please log which certs are loaded and/or removed by the plug-in when it starts or refreshes?

File provider: Using directory stats to detect certificate changes is unreliable

Currently, the file provider uses stat(2) on the certificates directory to detect if certificates were modified in some way (added, updated, removed).

This is unreliable for 2 reasons:

  1. The directory modification time is updated when files are added, removed or renamed, but not when a file's content is updated.
  2. Even if the underlying filesystem has subsecond time resolution (which is not always the case), the Erlang API doesn't. If a file is added/removed/renamed within the same second after a refresh, the file provider will miss that change.

The latter reason might also be a problem when the file provider uses the file modification time as an ID.

Introduce a built-in HTTP(S)-based provider

Now that #1 is merged, we should introduce a built-in provider that uses HTTPS.

Some concerns we have after a team-wide discussion:

  • What format should the certificate listing provider function expect? A list of JSON objects of a specific format (eventually maybe even a JWS/JW* one)? Should a list of new line-separated certificate URLs be returned by an HTTPS endpoint?
  • Should we use server certificate/key pair for HTTPS connections or a separate pair?

Trust store crashes

We are seeing problems in which the trust store provider crashes. This prevents client from connecting (because their certs are no longer trusted).

We see this problem on rabbitmq 3.6.10 (running Erlang 19.3)

The stack trace is:
** Generic server trust_store terminating
** Last message in was refresh
** When Server state == {state,[{rabbit_trust_store_file_provider,nostate}],
15000}
** Reason for termination ==
** {function_clause,
[{rabbit_trust_store,terminate,
[{{badmatch,false},
[{rabbit_trust_store_file_provider,ensure_directory,1,
[{file,"src/rabbit_trust_store_file_provider.erl"},
{line,95}]},
{rabbit_trust_store_file_provider,directory_path,2,
[{file,"src/rabbit_trust_store_file_provider.erl"},
{line,79}]},
{rabbit_trust_store_file_provider,list_certs,1,
[{file,"src/rabbit_trust_store_file_provider

We are running rabbitmq in kubernetes and mapping the white list directory from a host path.

Trust-Store doesn't revert SSL socket settings upon plugin disable

Enabling and disabling the trust-store plugin from the command-line seems fine. But, if we start it again after stopping it, then it fails. When the trust-store is stopped, it doesn't restore the OTP environment variables it changed, then it expects a clean slate when we start it again. The consequences are undesirable, not just for the trust-store which can't be started again, but any SSL listening socket that is created after the trust-store is stopped.

  • Store initial SSL settings in OTP Application environment.
  • Revert settings on SSL sockets which are currently listening.
  • Revert settings for SSL sockets which may be created in the future.

Include ability to list trusted certs

Currently, it is possible to see which certs are loaded by the trust store plug-in by executing

rabbitmqctl eval 'ets:tab2list(trust_store_whitelist).'

Can this be improved to show:

  • filename (from where the cert was loaded)
  • certificate serial number (hex is preferred, but decimal may also be included)
  • certificate DN
  • certificate issuing CA
  • certificate timeframe (issued/expired)

Pluggable way to load certificates

Currently this plugin can load certificates from a local filesystem. However, that's not always possible (think of PaaS environments which discourage local FS use). We should make it possible to load certificates from more sources by implementing a behaviour and configuring the plugin to use it, much like password hashing functions are pluggable in RabbitMQ server as of 3.6.0.

Some examples of alternative stores would be S3-compatible blob stores and services such as KeyWhiz. Supporting those two specifically is not in scope for this issue, by the way.

Provide a prebuilt ez archive

Are you able to make a prebuilt ez archive for those of us who don't have an environment set up for building this plugin from source?

Refactor and improve test suite code style

I reviewed the plugin with someone as part of an interview yesterday and we found out a few things that we were both unhappy with:

  • The test suite uses single letter bindings. Those have some use cases but not in this test suite. It's much more difficult to follow some tests because of this.
  • The test suite uses underscored bindings for no good reason.
  • Can we reconsider the use of lists:keyfind/3 here and use a proplist getter function?
  • There is no test that demonstrates how the plugin behaves when a certificate is replaced (e.g. you have a bob.pem and then upload a newer version of it under the same name)
  • This line could use some parentheses to make it more obvious what's going on.

I have someone who'd be interested in taking a look at this.

HTTP proxy support

We also face the problem of having a forward proxy between RabbitMQ and our certificate provider. We tried to set the http_proxy, https_proxy and no_proxy settings in the operating system and/or setting this parameters in the rabbitmq-env.conf, but it does not seem to work. We always get the error:

Unable to load certificate list for provider rabbit_trust_store_http_provider, reason: {failed_connect,[{to_address,{"xxx",yyy}},{inet,[inet],etimedout}]}

Is it even possible to configure this for the plugin or RabbitMQ at all?

Obtain certificates from an HTTP endpoint

I got the wind today from the PCF folks that uploading trusted certs would be very difficult when Rabbit is deployed to PCF. Specifically, adding a trusted cert would require cluster redeployment, given how BOSH works, which is not desirable at all and very risky.

Therefore, I would like to ask you to support an option of retrieving trusted certs from an HTTP endpoint. You may use XKMS standard (https://www.w3.org/TR/xkms) as an example. It discusses the Locate Service, which given cert DN or serial number responds with a list of matching certificates. If you are not into SOAP, you may translate it to REST.

Thanks!

Superfluous error messages when refreshing trust store

Since the fix of #73, the rabbitmq-trust-store plugin spams the log with unnecessary error messages during normal operation:

2020-05-03 03:24:34.773 [error] <0.843.0> Trust store will attempt to refresh certificates...
2020-05-03 03:25:04.788 [error] <0.843.0> Trust store will attempt to refresh certificates...
2020-05-03 03:25:34.804 [error] <0.843.0> Trust store will attempt to refresh certificates...
2020-05-03 03:26:04.821 [error] <0.843.0> Trust store will attempt to refresh certificates...
2020-05-03 03:26:34.839 [error] <0.843.0> Trust store will attempt to refresh certificates...
2020-05-03 03:27:04.853 [error] <0.843.0> Trust store will attempt to refresh certificates...

These should be logged at debug level instead.

Test suite failures with Erlang 23 on macOS

I'm seeing both a test flake and failure that appears to arise from my Erlang and platform combination.

The flake appears to involve ssl:setops/2 hanging at https://github.com/rabbitmq/rabbitmq-erlang-client/blob/13ab0bcd1cd769f3982deb4690b7e65e7ca3e0ed/src/amqp_main_reader.erl#L58

The failure arises from a slightly different structure of the tls_alert error we assert in certain tests - it's now wrapped by a socket error:

system_SUITE > http_provider_tests > validate_longer_chain
313
    #1. ***error,
314
            ***case_clause,
315
                 ***error,
316
                     ***socket_error,
317
                          ***tls_alert,
318
                              ***handshake_failure,
319
                                  "TLS client: In state connection received SERVER ALERT: Fatal - Handshake Failure\n"***,
320
                      ***expecting,'connection.start'***,
321
             [***system_SUITE,validate_longer_chain1,1,
322
                  [***file,"test/system_SUITE.erl"***,***line,371***]***]***
323

Support binaries as whitelist directory paths

This line would produce a confusing case clause failure if a binary as used as path. We should handle this better. Some RabbitMQ settings use binaries and newcomers confuse the two. At the same time, converting a binary to a list is pretty trivial.

Certificate chain verification logic - peer only

Currently, the plug-in will not match the client cert if it is accompanied by the chain (intermediate and root certs). This is probably because PEM files in the white-list directory contain just the peer cert (they are frequently just the X509 certs), and do not contain the chain. If the client does not send the chain, then plug-in matches the cert fine.

The client, though, may not be able to control whether the CA certs are sent. I know that I can control it in Java via KeyManager) but I don't know about. NET and other languages. Also, if the client does not send the chain, then original trust mechanism (based on CA via cacertfile option) would not work.

Could you please change the plug-in to ignore the chain and match only the peer cert to make it all compatible and trouble-free to adopt? Thanks

HTTP provider: If-Modified-Since is not up-to-date between certificate changes

We tried to use RabbitMQ with the rabbitmq-trust-store plugin. We configured the plugin to access an external service which provides the required endpoints.

When the plugin retrieves the list of certificates, the Last-Modified response header is set and the plugin sends the next query with the Last-Modified timestamp in the If-Modified-Since request header. If now something changes on the server, the Last-Modified timestamp is updated in the response header but the plugin still sends requests with the old If-Modified-Since timestamp. Somehow it seems that it is not updated after the first time receiving the Last-Modified timestamp.

Note, we are returning Last-Modified timestamp with 200 OK, if something changed and 304 Not Changed if nothing changed (without Last-Modified timestamp).

Default directory for windows

HOME environment variable can be false in windows. There are also "/" used in default directory path.
It should be at least mentioned in README to always set directory in windows.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.