Git Product home page Git Product logo

mongodb_exporter's Introduction

MongoDB exporter

Release Build Status codecov.io Code Coverage Go Report Card CLA assistant

This is the new MongoDB exporter implementation that handles ALL metrics exposed by MongoDB monitoring commands. This new implementation loops over all the fields exposed in diagnostic commands and tries to get data from them.

Currently, these metric sources are implemented:

  • $collStats
  • $indexStats
  • getDiagnosticData
  • replSetGetStatus
  • serverStatus

Info on Percona MongoDB exporter versions

The old 0.1x.y version (ex master branch) has been moved to the release-0.1x branch.

A flag, --compatible-mode, which exposes metrics with 0.1x compatible metric names has been implemented which simplifies migration from the old version to the current version.

Build the exporter

The build process uses the dockerized version of goreleaser so you don't need to install Go. Just run make release and the new binaries will be generated under the build directory.

├── build
│ ├── config.yaml
│ ├── mongodb_exporter_7c73946_checksums.txt
│ ├── mongodb_exporter-7c73946.darwin-amd64.tar.gz
│ ├── mongodb_exporter-7c73946.linux-amd64.tar.gz
│ ├── mongodb_exporter_darwin_amd64
│ │ └── mongodb_exporter <--- MacOS binary
│ └── mongodb_exporter_linux_amd64
│ └── mongodb_exporter <--- Linux binary

Running the exporter

If you built the exporter using the method mentioned in the previous section, the generated binaries are in mongodb_exporter_linux_amd64/mongodb_exporter or mongodb_exporter_darwin_amd64/mongodb_exporter

Docker

A docker image is available on the official percona repository.

Examples
# with podman
podman run -d -p 9216:9216 -p 17001:17001 percona/mongodb_exporter:0.40 --mongodb.uri=mongodb://127.0.0.1:17001

# with docker
docker run -d -p 9216:9216 -p 17001:17001 percona/mongodb_exporter:0.40 --mongodb.uri=mongodb://127.0.0.1:17001

Permissions

Connecting user should have sufficient rights to query needed stats:

      {
         "role":"clusterMonitor",
         "db":"admin"
      },
      {
         "role":"read",
         "db":"local"
      }

More info about roles in MongoDB documentation.

Example

mongodb_exporter_linux_amd64/mongodb_exporter --mongodb.uri=mongodb://127.0.0.1:17001

MongoDB Authentication

You can supply the mongodb user/password direct in the --mongodb.uri= like --mongodb.uri=mongodb://user:[email protected]:17001, you can also supply the mongodb user/password with --mongodb.user=, --mongodb.password= but the user and password info will be leaked via ps or top command, for security issue, you can use MONGODB_USER and MONGODB_PASSWORD env variable to set user/password for given uri

MONGODB_USER=XXX MONGODB_PASSWORD=YYY mongodb_exporter_linux_amd64/mongodb_exporter --mongodb.uri=mongodb://127.0.0.1:17001 --mongodb.collstats-colls=db1.c1,db2.c2
# or
export MONGODB_USER=XXX
export MONGODB_PASSWORD=YYY
mongodb_exporter_linux_amd64/mongodb_exporter --mongodb.uri=mongodb://127.0.0.1:17001 --mongodb.collstats-colls=db1.c1,db2.c2

Multi-target support

You can run the exporter specifying multiple URIs, devided by a comma in --mongodb.uri option or MONGODB_URI environment variable in order to monitor multiple mongodb instances with the a single mongodb_exporter instance.

--mongodb.uri=mongodb://user:[email protected]:27017/admin,mongodb://user2:[email protected]:27018/admin

In this case you can use the /scrape endpoint with the target parameter to retreive the specified tartget's metrics. When querying the data you can use just mongodb://host:port in the target parameter without other parameters and, of course without host credentials

GET /scrape?target=mongodb://127.0.0.1:27018

If your URI is prefixed by mongodb:// or mongodb+srv:// schema, any host not prefixed by it after comma is being treated as part of a cluster rather then as a standalone host. Thus clusters and standalone hosts can be combined like this:

--mongodb.uri=mongodb+srv://user:pass@host1:27017,host2:27017,host3:27017/admin,mongodb://user2:pass2@host4:27018/admin

Enabling collstats metrics gathering

--mongodb.collstats-colls receives a list of databases and collections to monitor using collstats. Usage example: --mongodb.collstats-colls=database1.collection1,database2.collection2

mongodb_exporter_linux_amd64/mongodb_exporter --mongodb.uri=mongodb://127.0.0.1:17001 --mongodb.collstats-colls=db1.c1,db2.c2

Enabling compatibility mode.

When compatibility mode is enabled by the --compatible-mode, the exporter will expose all new metrics with the new naming and labeling schema and at the same time will expose metrics in the version 1 compatible way. For example, if compatibility mode is enabled, the metric mongodb_ss_wt_log_log_bytes_written (new format)

# HELP mongodb_ss_wt_log_log_bytes_written serverStatus.wiredTiger.log.
# TYPE mongodb_ss_wt_log_log_bytes_written untyped
mongodb_ss_wt_log_log_bytes_written 2.6208e+06

will be also exposed as mongodb_mongod_wiredtiger_log_bytes_total with the unwritten label.

HELP mongodb_mongod_wiredtiger_log_bytes_total mongodb_mongod_wiredtiger_log_bytes_total
# TYPE mongodb_mongod_wiredtiger_log_bytes_total untyped
mongodb_mongod_wiredtiger_log_bytes_total{type="unwritten"} 2.6208e+06

Enabling profile metrics gathering

--collector.profile To collect metrics, you need to enable the profiler in MongoDB: Usage example: db.setProfilingLevel(2)

Level Description
0 The profiler is off and does not collect any data. This is the default profiler level.
1 The profiler collects data for operations that take longer than the value of slowms or that match a filter.
When a filter is set:
  • The slowms and sampleRate options are not used for profiling.
  • The profiler only captures operations that match the filter.
2 The profiler collects data for all operations.

Enabling shards metrics gathering

When shard metrics collection is enabled by --collector.shards, the exporter will expose metrics related to sharded Mongo. Example, if shards collector is enabled:

# HELP mongodb_shards_collection_chunks_count sharded collection chunks.
# TYPE mongodb_shards_collection_chunks_count counter
mongodb_shards_collection_chunks_count{collection="system.sessions",database="config",shard="rs1"} 250
mongodb_shards_collection_chunks_count{collection="system.sessions",database="config",shard="rs2"} 250

You can see shard name, it's collection, database and count.

Cluster role labels

The exporter sets some topology labels in all metrics. The labels are:

  • cl_role: Cluster role according to this table:
Server type Label
mongos mongos
regular instance (primary or secondary) shardsvr
arbiter shardsvr
standalone (empty string)
  • cl_id: Cluster ID
  • rs_nm: Replicaset name
  • rs_state: Replicaset state is an integer from getDiagnosticData() -> replSetGetStatus.myState. Check the official documentation for details on replicaset status values.

Usage Reference

See the Reference Guide for details on using the exporter.

Bug Reports / Feature PR

Refer to the Contribution Guide.

mongodb_exporter's People

Contributors

ademidoff avatar adnull avatar aleksi avatar algestam avatar aravandi avatar artemgavrilov avatar bupychuk avatar denisok avatar dependabot-preview[bot] avatar dependabot[bot] avatar dliakhov avatar hiroshi avatar idoqo avatar igroene avatar jirictvrtka avatar kanakagrawal avatar kbudde avatar marctc avatar mas15 avatar masaruhoshi avatar naughtygitcat avatar percona-csalguero avatar rfratto avatar ritbl avatar rnovikovp avatar shashanksinha252 avatar tregubov-av avatar trvrnrth avatar weastur avatar yaroslavpodorvanov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mongodb_exporter's Issues

Support `compatible-mode` for `indexstats`

The compatible-mode switch is ignored in indexstats_collector.go. This produces unique metrics names, containing the db, collection, and index name, for which dashboards and alerts are hard to produce.

By supporting compatible-mode and using a single metric name, with the db, collection, and index name as labels on the metric (as in the pre-0.20 releases), it will make it much easier to include in dashboards.

e.g. instead of a metric mongodb_collstats_storageStats_indexSizes_some_index_name, a metric mongodb_collstats_storageStats_indexSizes with a label key_name=some_index_name should be used instead. This would match how mongodb_indexstats_accesses_ops is used.

Cannot build

github.com/percona/mongodb_exporter/collector/mongos

/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongos/asserts.go:26: assertsTotal.WithLabelValues("regular").Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongos/asserts.go:27: assertsTotal.WithLabelValues("warning").Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongos/asserts.go:28: assertsTotal.WithLabelValues("msg").Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongos/asserts.go:29: assertsTotal.WithLabelValues("user").Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongos/asserts.go:30: assertsTotal.WithLabelValues("rollovers").Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongos/connections.go:36: connectionsMetricsCreatedTotal.Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongos/metrics.go:60: metricsGetLastErrorWtimeTotalMilliseconds.Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongos/metrics.go:62: metricsGetLastErrorWtimeoutsTotal.Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongos/metrics.go:80: metricsCursorTimedOutTotal.Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongos/network.go:32: networkBytesTotal.WithLabelValues("in_bytes").Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongos/network.go:32: too many errors

github.com/percona/mongodb_exporter/collector/mongod

/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongod/asserts.go:26: assertsTotal.WithLabelValues("regular").Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongod/asserts.go:27: assertsTotal.WithLabelValues("warning").Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongod/asserts.go:28: assertsTotal.WithLabelValues("msg").Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongod/asserts.go:29: assertsTotal.WithLabelValues("user").Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongod/asserts.go:30: assertsTotal.WithLabelValues("rollovers").Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongod/background_flushing.go:53: backgroundFlushingflushesTotal.Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongod/background_flushing.go:54: backgroundFlushingtotalMilliseconds.Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongod/connections.go:36: connectionsMetricsCreatedTotal.Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongod/global_lock.go:79: globalLockTotal.Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongod/index_counters.go:35: indexCountersTotal.WithLabelValues("accesses").Set undefined (type prometheus.Counter has no field or method Set)
/usr/local/go/src/github.com/percona/mongodb_exporter/collector/mongod/index_counters.go:35: too many errors

Why many metrics don't have member label ?

Hi. I start one mongodb_exporter to scrape metrics of a replica set with 3 nodes. But I cannot extract member label from many metrics, such as:

mongodb_ss_wt_block_manager_blocks_read{cl_id="6098d5af91ec5fad8a3c96d7",cl_role="shardsvr",rs_nm="shard1",rs_state="1"} 3773

I thought the member of a replica set was included in metrics such that we can group it by different members.

Should I deploy one mongodb_exporter per mongod/mongos instead of only one mongodb_exporter instance ?
Maybe mongodb_exporter needs have a --discover option like the one in mongostat ?

v0.20.4 binary does not accept new flag --discovering-mode

When running the binary with the latest release v0.20.4 it fails when providing the --discovering-mode with the following message:

Usage: mongodb_exporter

MongoDB Prometheus exporter

mongodb_exporter: error: unknown flag --discovering-mode
Flags:
  -h, --help                      Show context-sensitive help.
      --mongodb.collstats-colls=db1.col1,db2.col2
                                  List of comma separared databases.collections
                                  to get $collStats
      --mongodb.indexstats-colls=db1.col1,db2.col2
                                  List of comma separared databases.collections
                                  to get $indexStats
      --mongodb.uri=mongodb://user:[email protected]:27017/admin?ssl=true
                                  MongoDB connection URI ($MONGODB_URI)
      --mongodb.global-conn-pool
                                  Use global connection pool instead of creating
                                  new pool for each http request.
      --mongodb.direct-connect    Whether or not a direct connect should be
                                  made. Direct connections are not valid if
                                  multiple hosts are specified or an SRV URI is
                                  used.
      --web.listen-address=":9216"
                                  Address to listen on for web interface and
                                  telemetry
      --web.telemetry-path="/metrics"
                                  Metrics expose path
      --log.level="error"         Only log messages with the given severuty or
                                  above. Valid levels: [debug, info, warn,
                                  error, fatal]
      --disable.diagnosticdata    Disable collecting metrics from
                                  getDiagnosticData
      --disable.replicasetstatus
                                  Disable collecting metrics from
                                  replSetGetStatus
      --compatible-mode           Enable old mongodb-exporter compatible metrics
      --version                   Show version and exit

Seems like the CLI don't even know the flag, although the code for it is on the main branch and the feature is mentioned on the release notes

mongos metric collection fails while chunk migrations are in progress

Hi, It looks like the mongodb_exporter is failing for mongos router nodes whenever chunk migrations are in progress. For example:

$ curl http://mongos-router.domain:9104/metrics
An error has occurred during metrics collection:

5 error(s) occurred:
* collected metric mongodb_mongos_sharding_changelog_10min_total label:<name:"event" value:"moveChunk.to" > counter:<value:1 >  was collected before with the same name and label values
* collected metric mongodb_mongos_sharding_changelog_10min_total label:<name:"event" value:"moveChunk.start" > counter:<value:2 >  was collected before with the same name and label values
* collected metric mongodb_mongos_sharding_changelog_10min_total label:<name:"event" value:"moveChunk.commit" > counter:<value:1 >  was collected before with the same name and label values
* collected metric mongodb_mongos_sharding_changelog_10min_total label:<name:"event" value:"moveChunk.from_failed" > counter:<value:1 >  was collected before with the same name and label values
* collected metric mongodb_mongos_sharding_changelog_10min_total label:<name:"event" value:"moveChunk.from" > counter:<value:1 >  was collected before with the same name and label values

The easiest way to reproduce is to create collection on a sharded mongo cluster without sharding enabled, write a lot of data (ideally a few 10s/100s of block's worth), and then enable sharding and balancing on the collection, you should see that the exporter remains in an error state until the balancer finishes its work.

Collector crashes after timeouts?

Hello,

I've got PMM agent monitoring linux:metrics and mongodb:metrics. From time to time mongodb exporter crashes. Here is what I see in logs:

### Warning: the exporter is in beta/experimental state and field names are very
### likely to change in the future and features may change or get removed!
### See: https://github.com/Percona-Lab/prometheus_mongodb_exporter for updates
mongodb_exporter version: 1.0.5, git commit hash: 4651d4d535ed6c807606daf6a864338c90e06e35
Listening on 10.81.120.79:42003
E0123 07:18:08.927387    2341 connection.go:27] Cannot connect to server using url mongodb://xxx$@localhost:27017: read tcp 127.0.0.1:34918->127.0.0.1:27017: i/o timeout
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5f0b9e]

goroutine 41467 [running]:
panic(0x74e180, 0xc420012050)
	/usr/local/go1.7.1/src/runtime/panic.go:500 +0x1a1
gopkg.in/mgo%2ev2.(*Session).Close(0x0)
	/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/gopkg.in/mgo.v2/session.go:1612 +0x2e
github.com/Percona-Lab/prometheus_mongodb_exporter/collector.(*MongodbCollector).Collect(0xc42015fea0, 0xc420750180)
	/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongodb_collector.go:76 +0xa1
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc420344100, 0xc420750180, 0x93db20, 0xc42015fea0)
	/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/github.com/prometheus/client_golang/prometheus/registry.go:433 +0x63
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
	/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/github.com/prometheus/client_golang/prometheus/registry.go:434 +0x326

MongoDB version : "3.2.9-2.1". This node is a master of a 3-node replica set, but I've seen same logs on other nodes - crash after the timeout.

Alex

Release Arch Binaries

With Go's vendoring being as awesome as it is, could you please release compiled arch versions of the exporter.

This is the main reason I am not using this exporter personally.

Thanks!

exporter increase logicalSessionRecordCache.activeSessionsCount

Hi,

The exporter increase indefinitely logicalSessionRecordCache.activeSessionsCount
I already add the following configuration:
setParameter: logicalSessionRefreshMillis: 120000 localLogicalSessionTimeoutMinutes: 15
but still increase connection. When I stop exporter the logicalSessionRecordCache stop to increase.
I use mongo 4.4.5
what's happens and how can I solve it ?

Thanks

mongodb_exporter crashes when mongo is not available

mongodb_exporter crashes when mongodb is not available. Here's the error:

E0228 11:10:46.481870    7005 connection.go:27] Cannot connect to server using url mongodb://localhost:27017: no reachable servers`
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x67890e]

goroutine 26 [running]:
panic(0x7828e0, 0xc42000c090)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
gopkg.in/mgo%2ev2.(*Session).Close(0x0)
	/home/rui/Projects/golang/src/gopkg.in/mgo.v2/session.go:1612 +0x2e
github.com/percona/mongodb_exporter/collector.(*MongodbCollector).Describe(0xc42016e0a0, 0xc420062b40)
	/home/rui/Projects/golang/src/github.com/percona/mongodb_exporter/collector/mongodb_collector.go:47 +0xef
github.com/prometheus/client_golang/prometheus.(*registry).Register.func1(0x984c80, 0xc42016e0a0, 0xc420062b40)
	/home/rui/Projects/golang/src/github.com/prometheus/client_golang/prometheus/registry.go:218 +0x3b
created by github.com/prometheus/client_golang/prometheus.(*registry).Register
	/home/rui/Projects/golang/src/github.com/prometheus/client_golang/prometheus/registry.go:220 +0xc7

PMM-8489 Failed to get topology labels when the target server is mongos

Steps to Reproduce

  1. Set up a mongos server
  2. Update the mongodb.uri to the mongos and run mongodb_exporter

Actual

Failed to get topology labels when the target server is mongos, and get the error from logs:

time="2021-06-14T08:39:16Z" level=error msg="Cannot get topology info: error getting cluster ID: (CommandNotFound) no such cmd: replSetGetConfig: cannot get topology labels"

Expected

Should work fine with mongos

Note

It seems the error is caused by: https://github.com/percona/percona-toolkit/blob/49b570ed0ccb49cecd3515d444f3fef44a56d115/src/go/mongolib/util/util.go#L460
because it's not a ReplicationNotEnabledError but a CommandNotFound error.

Example

  mongo-exporter:
    image: bitnami/mongodb-exporter:0.20.5
    command:
      - '--disable.replicasetstatus'
    depends_on:
      - mongodb-router
    environment:
      - MONGODB_URI=mongodb://mongodb-router:27017
    restart: always
    ports:
      - 9216:9216

  mongodb-router:
    image: "bitnami/mongodb-sharded:4.4.4"
    environment:
      - MONGODB_ADVERTISED_HOSTNAME=mongodb-router
      - MONGODB_SHARDING_MODE=mongos
      - MONGODB_CFG_PRIMARY_HOST=mongodb-cfg
      - MONGODB_CFG_REPLICA_SET_NAME=cfgreplicaset
      - MONGODB_REPLICA_SET_KEY=replicasetkey123
    ports:
      - "27017:27017"

Add metric with mongodb status

It would be nice to have a metric indicating the mongodb status mongodb_up (0|1). There are other exporters doing that and it's really handy for monitoring. (ex. haproxy exports haproxy_up; elasticsearch_exporter exports elasticsearch_up; etc...).

Crash using TokuMX mongo shell v2.0.2-mongodb-2.4.10

Creating this issue on behalf of the user https://www.percona.com/forums/questions-discussions/percona-monitoring-and-management/46287-mongodb-problem

TokuMX mongo shell v2.0.2-mongodb-2.4.10

goroutine 34 [running]:
panic(0x74e180, 0xc420012050)
/usr/local/go1.7.1/src/runtime/panic.go:500 +0x1a1
github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongod.(*PreloadStats).Export(0x0, 0xc4200512c0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongod/metrics.go:363 +0x38
github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongod.(*ReplStats).Export(0xc4201ce1c0, 0xc4200512c0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongod/metrics.go:352 +0x90
github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongod.(*MetricsStats).Export(0xc420195440, 0xc4200512c0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongod/metrics.go:435 +0x4c7
github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongod.(*ServerStatus).Export(0xc4200b4300, 0xc4200512c0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongod/server_status.go:116 +0x2a7
github.com/Percona-Lab/prometheus_mongodb_exporter/collector.(*MongodbCollector).collectMongod(0xc420 0d7d20, 0xc4201831e0, 0xc4200512c0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongodb_collector.go:96 +0xd4
github.com/Percona-Lab/prometheus_mongodb_exporter/collector.(*MongodbCollector).collectMongodReplSet (0xc4200d7d20, 0xc4201831e0, 0xc4200512c0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongodb_collector.go:101 +0x49
github.com/Percona-Lab/prometheus_mongodb_exporter/collector.(*MongodbCollector).Collect(0xc4200d7d20 , 0xc4200512c0)
/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongodb_collector.go:71 +0x370
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc42019a780, 0xc4200512c0, 0x93db20, 0xc4200d7d20)
/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/github.com/prometheus/client_golang/prometheus/registry.go:433 +0x63
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
/mnt/workspace/pmm-client-tarball/pmm-client-1.0.5/src/github.com/prometheus/client_golang/prometheus/registry.go:434 +0x326
### Warning: the exporter is in beta/experimental state and field names are very
### likely to change in the future and features may change or get removed!
### See: https://github.com/Percona-Lab/prome...ngodb_exporter for updates
mongodb_exporter version: 1.0.5, git commit hash: 4651d4d535ed6c807606daf6a864338c90e06e35
Listening on 172.17.0.1:42003
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5b9bd8]

Log rotate ??

Hi

I see it rotates logs as I can see a bunch of logs like below and thats good. But is there a setting not only to rotate but to delete old log files?

-rw-r--r-- 1 prometheus iptvgrp 193 Oct 26 18:45 mongodb_exporter.ip-10-113-198-204.unknownuser.log.ERROR.20161026-184537.3156
-rw-r--r-- 1 prometheus iptvgrp 193 Oct 26 18:45 mongodb_exporter.ip-10-113-198-204.unknownuser.log.INFO.20161026-184537.3207
-rw-r--r-- 1 prometheus iptvgrp 193 Oct 26 18:45 mongodb_exporter.ip-10-113-198-204.unknownuser.log.WARNING.20161026-184548.3207
-rw-r--r-- 1 prometheus iptvgrp 193 Oct 26 18:45 mongodb_exporter.ip-10-113-198-204.unknownuser.log.ERROR.20161026-184548.3207
lrwxrwxrwx 1 prometheus iptvgrp 79 Oct 26 18:45 mongodb_exporter.WARNING -> mongodb_exporter.ip-10-113-198-204.unknownuser.log.WARNING.20161026-184548.3207
lrwxrwxrwx 1 prometheus iptvgrp 77 Oct 26 18:45 mongodb_exporter.ERROR -> mongodb_exporter.ip-10-113-198-204.unknownuser.log.ERROR.20161026-184548.3207
lrwxrwxrwx 1 prometheus iptvgrp 76 Oct 26 18:45 mongodb_exporter.INFO -> mongodb_exporter.ip-10-113-198-204.unknownuser.log.INFO.20161026-184548.3238

How to calculate replication Lag.

The readme has replication lag, but I guess it is supposed to be calculated from two different metrics by joining them. What is the correct way to do that?

Missing (probably) labels from mongos

I've added my mongos hosts to Prometheus and here's the sample of the labels I see from there:

mongodb_mongos_connections{alias="fqdn",instance="fqdn:9104",job="mongodb",state="current"}

Isn't there supposed to be a cluster label for "MongoDB Cluster Summary" dashboard to work correctly? Or should I just add it during the scrape step?

mongodb_exporter crashes on startup under CentOS 6

Running mongodb_exporter from pmm-client-1.1.1-1.x86_64 RPM under CentOS 6. I added mongodb:metrics with pmm-admin first but it showed as NOT running in "pmm-admin list" output. Running mongodb_exporter manually with this command line:

/usr/local/percona/pmm-client/mongodb_exporter -v 100 -web.listen-address=10.9.52.12:42003 -web.auth-file=/usr/local/percona/pmm-client/pmm.yml -web.ssl-cert-file=/usr/local/percona/pmm-client/server.crt -web.ssl-key-file=/usr/local/percona/pmm-client/server.key

produces this error:

mongodb_exporter version: 1.1.1, git commit hash: c890996b9133a3288530da4a9fb6e6fad067b55f
HTTPS/TLS is enabled
Listening on 10.9.52.12:42003
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6410a8]

Full error log with stack below.
error.txt

"Failed to get oplog status." error under high oplog pressure

mongodb_exporter outputs the following error under high oplog pressure (very high insert/update tests).

"E0504 18:31:11.270167 3301 oplog_status.go:111] Failed to get oplog status."

Perhaps this should be expected temporarily or something needs optimisation.

Tim

Running the exporter

Is it required to have one exporter per replicaset member or is having one exporter for the whole replicaset enough?

Fail build docker image.

When assembling Docker image error occurred.

Logs

Executing busybox-1.24.2-r8.trigger
Executing ca-certificates-20160104-r4.trigger
OK: 348 MiB in 44 packages
# github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongos
collector/mongos/asserts.go:26: assertsTotal.WithLabelValues("regular").Set undefined (type prometheus.Counter has no field or method Set)
collector/mongos/asserts.go:27: assertsTotal.WithLabelValues("warning").Set undefined (type prometheus.Counter has no field or method Set)
collector/mongos/asserts.go:28: assertsTotal.WithLabelValues("msg").Set undefined (type prometheus.Counter has no field or method Set)
collector/mongos/asserts.go:29: assertsTotal.WithLabelValues("user").Set undefined (type prometheus.Counter has no field or method Set)
collector/mongos/asserts.go:30: assertsTotal.WithLabelValues("rollovers").Set undefined (type prometheus.Counter has no field or method Set)
collector/mongos/connections.go:36: connectionsMetricsCreatedTotal.Set undefined (type prometheus.Counter has no field or method Set)
collector/mongos/metrics.go:60: metricsGetLastErrorWtimeTotalMilliseconds.Set undefined (type prometheus.Counter has no field or method Set)
collector/mongos/metrics.go:62: metricsGetLastErrorWtimeoutsTotal.Set undefined (type prometheus.Counter has no field or method Set)
collector/mongos/metrics.go:80: metricsCursorTimedOutTotal.Set undefined (type prometheus.Counter has no field or method Set)
collector/mongos/network.go:32: networkBytesTotal.WithLabelValues("in_bytes").Set undefined (type prometheus.Counter has no field or method Set)
collector/mongos/network.go:32: too many errors
# github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongod
collector/mongod/asserts.go:26: assertsTotal.WithLabelValues("regular").Set undefined (type prometheus.Counter has no field or method Set)
collector/mongod/asserts.go:27: assertsTotal.WithLabelValues("warning").Set undefined (type prometheus.Counter has no field or method Set)
collector/mongod/asserts.go:28: assertsTotal.WithLabelValues("msg").Set undefined (type prometheus.Counter has no field or method Set)
collector/mongod/asserts.go:29: assertsTotal.WithLabelValues("user").Set undefined (type prometheus.Counter has no field or method Set)
collector/mongod/asserts.go:30: assertsTotal.WithLabelValues("rollovers").Set undefined (type prometheus.Counter has no field or method Set)
collector/mongod/background_flushing.go:53: backgroundFlushingflushesTotal.Set undefined (type prometheus.Counter has no field or method Set)
collector/mongod/background_flushing.go:54: backgroundFlushingtotalMilliseconds.Set undefined (type prometheus.Counter has no field or method Set)
collector/mongod/connections.go:36: connectionsMetricsCreatedTotal.Set undefined (type prometheus.Counter has no field or method Set)
collector/mongod/global_lock.go:79: globalLockTotal.Set undefined (type prometheus.Counter has no field or method Set)
collector/mongod/index_counters.go:35: indexCountersTotal.WithLabelValues("accesses").Set undefined (type prometheus.Counter has no field or method Set)
collector/mongod/index_counters.go:35: too many errors
The command '/bin/sh -c apk add --update -t build-deps go git mercurial libc-dev gcc libgcc     && cd $APPPATH && go get -d && go build -o /bin/mongodb_exporter     && apk del --purge build-deps && rm -rf $GOPATH' returned a non-zero code: 2
ERROR: Build failed: exit status 1

RocksDB functionality: counters and gauges sometimes do not increment/change

The getServerStatus output for RocksDB was designed to be human-friendly and when numbers in the output get large the precision of the number disappears by applying human-friendly units, example: 10,000,000 will become the string "10M" in the output with no decimal precision.

The problem with this is it cannot be realistically graphed. The number needs to change from 10 million to 11 million for a change to occur in the number, and thus the graphs/metrics we are visualising with this tool.

Example:

rs2:PRIMARY> db.serverStatus().rocksdb.stats[14]
Cumulative writes: 10M writes, 63M keys, 10M batches, 1.0 writes per batch, ingest: 7.50 GB, 0.01 MB/s

This issue is to track a need for a feature in RocksDB or MongoRocks for machine-friendly output values, or at very least numbers with full precision, to resolve this problem.

Exposing db metrics for replicasets

What am I doing ?

I am trying to add some DB metrics for replicasets much like it's been done for shards here inside v1_compatibiliity.go. I tried making use of the getDatabaseStatList function to get DB stats and use those in conjunction with the replica set metrics as in the replSetMetrics function inside the same file.
I am running the docker-compose file in the repo(as is) to spin up a replicaset. And I am able to scrape the metrics along with metrics for replicaset from /metrics endpoint of the exporter

Issue

Now although the getDatabaseStatList is able to fetch the DB names, it returns an empty document while running this command when called from the exporter

r := client.Database(db).RunCommand(context.TODO(), bson.D{{Key: "dbStats", Value: 1}, {Key: "scale", Value: 1}})

I tried --compatibility-mode as well. I also don't see any errors

Questions:

  1. Is it recommended to do something like this for replicasets given how expensive it can get on systems with a lot of databases or collections?
  2. Why am I not getting any documents after running the dbStats command from the exporter?

@percona-csalguero Can you please suggest something? Any pointers would be helpful

Exporter process uses more CPU than expected

This is to track a low priority investigation into what consumes CPU in our exporter. In one instance of mongodb_exporter running under Percona Monitoring and Management (same exporter code) I can see it use 5-6% user-level CPU at all times.

An example:

$ ps aux|grep mongodb_exporter|grep -v grep
root      7666  5.4  0.4  23756 18224 ?        Sl   Nov06 727:30 /usr/local/percona/pmm-client/mongodb_exporter ...

Maybe this is CPU usage is normal but it would be good to make sure we aren't doing anything silly to cause this.

Grafana Dashboard ?

Do we have a offical grafana dashboard is that work with the latest version ?

No metrics for mongos

I am using percona grafana dashboards with the test cluster that runs with the docker-compose file included in this repo. Every instance gets data collected, but mongos does not collect most of the data.

Seems like it's trying to collect using diagnosticDataCollector but the getDiagnosticData command doesn't return anything. Looks like the right method to collect data for mongos would be with serverStatusCollector, which returns all the data needed, such as opCounters and others, but this collector is not being called by the exporter.

Is this expected behaviour?

Prometheus exporter continues to export after database has been dropped

Hi,

I've noticed that if we remove a mongo database the exporter will still retain all its metrics. I've not tested to see what happens if we drop a collection - but it's probably the same.

We find our dev server ends up with lots of dead metrics as a result. Might it be possible to enhance the exporter so that it forgets about collections after they are removed?

adjusting to 20.x metrics

So I've noted a few of the changes that we'll need to make in our Grafana dashboards, but there's a few around collections that I can't seem to find. e.g. mongodb_mongod_db_collections_total{db="admin"}.
I've tried running with --compatible-mode and --discovering-mode but ctrl-f'ing admin shows no results. Is there a way to get these metrics still or have they been removed?

questions about metrics for several bases

Hi,

I have a mongodb instance on a server1 containing only one base and I have metrics with the label rs_nm the name of the base:
ex: mongodb_sys_netstat_Tcp_RtoAlgorithm {cl_id = "608921f272bce3bb425f7d16", cl_role = "shardsvr", rs_nm = "<base name>", rs_state = "1"} 1

I have another instance on a server2 containing several bases and there I do not have the base label:
mongodb_sys_netstat_Tcp_RtoAlgorithm {cl_id = "", cl_role = "mongod", rs_state = "0"} 1

How should I configure the export to get this kind of info?

For server2 I have the following variables:
MONGODB_URI: mongodb: // <user>: <psswd> @ <host>: 27017 / admin? AuthSource = admin

my command line: --web.telemetry-path = / metrics --mongodb.direct-connect = false

Second question: what metric can I use to see the state (up or down) of my base (s)?

List all the metrics without a MongoDB install

Is it possible to list all the metrics emitted by mongodb_exporter without actually running an instance/replicaset of mongo?

I am going over the code to see if it's feasible but can't find any.

For example if I need all the rs metrics, like mongodb_rs_*, what do I do?

Thanks for any inputs

Lock stats are broken on MongoDB 3.2

MongoDB changed the lock stats again (a 3rd time!) in 3.2 and lock stats in the exporter are broken on this version as a result, always returning zero for lock stats.

locks.go will need to support a new lock stats schema when using 3.2.

connection leak

Hi. I've started three mongod on the same host, three mongodb_exporter on the same host to scrape each of three mongods. However I find the three mongod's connections from mongodb_exporter differs a lot.

(the following port 7024, 7042, 7052 are mongod's port)

image

I try to tcpdump one of mongodb_export's port, and nothing outputs.
image

I suspect is there any connection leak on the mongodb_exporter ? Why does it need so many connections to send serverStatus to mongod ?

hello cannot use

go build -o mongodb_exporter github.com/Percona-Lab/prometheus_mongodb_exporter

It gives -- cannot find package "gopkg.in/mgo.v2" in any of:
/home/devops/go/src/github.com/Percona-Lab/prometheus_mongodb_exporter/vendor/gopkg.in/mgo.v2 (vendor tree)

we have to get package with all dependencies than I tired

go get github.com/Percona-Lab/prometheus_mongodb_exporter
than

Cloning into '/home/devops/go/src/gopkg.in/mgo.v2'...
error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.

How can we solve this issue.

Thanks

linter doesn't run for forked repos

https://github.com/percona/mongodb_exporter/runs/2724195305?check_suite_focus=true

Run # use GITHUB_TOKEN because only it has access to GitHub Checks API
  # use GITHUB_TOKEN because only it has access to GitHub Checks API
  bin/golangci-lint run -c=.golangci-required.yml --out-format=line-number | env REVIEWDOG_GITHUB_API_TOKEN=*** bin/reviewdog -f=golangci-lint -level=error -reporter=github-pr-check
  bin/golangci-lint run -c=.golangci.yml --out-format=line-number | env REVIEWDOG_GITHUB_API_TOKEN=*** bin/reviewdog -f=golangci-lint -level=error -reporter=github-pr-review
  shell: /usr/bin/bash -e {0}
  env:
    GOROOT: /opt/hostedtoolcache/go/1.15.12/x64
reviewdog: This is Pull-Request from forked repository.
GitHub token doesn't have write permission of Check API, so reviewdog will
report results via logging command [1].
[1]: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/development-tools-for-github-actions#logging-commands
reviewdog: Reporting results for "golangci-lint"
reviewdog: No results found for "golangci-lint". 0 results found outside diff.
reviewdog: This is Pull-Request from forked repository.
GitHub token doesn't have write permission of Review API, so reviewdog will
report results via logging command [1] and create annotations similar to
github-pr-check reporter as a fallback.
[1]: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/development-tools-for-github-actions#logging-commands
level=warning msg="[runner] The linter 'golint' is deprecated (since v1.41.0) due to: The repository of the linter has been archived by the owner.  Replaced by revive."
level=warning msg="[runner] The linter 'interfacer' is deprecated (since v1.38.0) due to: The repository of the linter has been archived by the owner. "
level=warning msg="[runner] The linter 'scopelint' is deprecated (since v1.39.0) due to: The repository of the linter has been deprecated by the owner.  Replaced by exportloopref."
level=warning msg="[runner] The linter 'maligned' is deprecated (since v1.38.0) due to: The repository of the linter has been archived by the owner.  Replaced by govet 'fieldalignment'."

roadmap of percona/mongodb_exporter and dcu/mongodb_exporter

I'm new to open source community, but looks too many branches makes user and contributor hard to choose.

Take mongodb_exporter for example, I know/use dcu/mongodb_exporter first and then I would like to contribute to it during use. but then I found percona/mongodb_exporter has more features I want already. like MongoDB WiredTiger storage-engine metrics. I'm thinking if I should turn to use percona/mongodb_exporter.

since percona/mongodb_exporter is forked from dcu/mongodb_exporter, So my greenhand question is why not percona/mongodb_exporter contribute back to dcu, if so, all the users will easy to choose. No offence, I just want to know more about open source community.

Add config option to restrict collection to specific databases

Right now you can configure the exporter to monitor specific collections, but this is verbose if you want to monitor all the collections but only in a certain database.

Would we add an option to specify the databases to monitor and the exporter could discover all the collections within that database?

server returned error on SASL authentication step: Authentication failed.

When i config one mongodb with authentication, mongodb_exporter run on port 9104 ok
But when I run mongodb replica set and authentication with keyfile, then I run command mongodb_exporter -mongodb.uri mongodb://mongodb_exporter:[email protected]:27017 on server, output is server returned error on SASL authentication step: Authentication failed.
Can you tell me why and how to solve it? Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.