tikv / rust-prometheus Goto Github PK
View Code? Open in Web Editor NEWPrometheus instrumentation library for Rust applications
License: Apache License 2.0
Prometheus instrumentation library for Rust applications
License: Apache License 2.0
cargo publish
seems not support hyper with git dependence. we should use the published one.
Now using metrics across multi threads has a low performance, so we have to use local metric and then flush to the global metric sometimes, it is not convenient, maybe we can use core-local metrics, But I don't know whether it is easy to do or not.
See http://rocksdb.org/blog/2017/05/14/core-local-stats.html
Maybe we can parse /proc/pid/net/sockstat
Like
use prom;
prom.register(Box::new(c))
prom.gather()
Like
register_counter!(HTTP_COUNTER, counter_opts);
// then we can use
HTTP_COUNTER.inc();
In the ideal use of Prometheus, libraries provide their own instrumentation:
The short answer is to instrument everything. Every library, subsystem and service should have at least a few metrics to give you a rough idea of how it is performing.
โhttps://prometheus.io/docs/practices/instrumentation/
Bringing in the push client by default makes that hard to do: any library instrumented with Prometheus suddenly brings in hyper and all its dependencies.
Here are a couple of ideas on how to fix this:
https://prometheus.io/docs/instrumenting/exposition_formats/
variable_labels
can be expressed by generic tuples (variables in tuple length).
This brings several benifits:
variable_labels
can be eliminated.Now we find that the lock contention of Vector metrics is very serious. To avoid this, we usually implement a local metric mechanism then flush to the global metrics, this works well and has a high performance, but we can't always use this way conveniently in anywhere.
A better way is to use Core-local Statistics, like RocksDB does http://rocksdb.org/blog/2017/05/14/core-local-stats.html
Mostly, we use the integer for counter and gauge, so it is more efficient to use atomic int64 directly than float64.
I'd really like to contribute since I actually started working on a Prometheus client library some time ago but never got to finish it because I don't have enough time to do this right now. I found this project but the issues here have little to no descriptions or what actually is needed.
I have my own very simple Prometheus implementation. Since yalls is already more baked i would like to switch to it.
I use crate spin 0.4.6 and find that the spin lock has a better performance than the standard lock.
In our case, we can't hold the lock for a long time, so using spin has a great benefit.
#[bench]
fn bench_lock_write_map(b: &mut Bencher) {
let m = HashMap::with_capacity(100);
let l = Arc::new(RwLock::new(m));
b.iter(||{
let mut a = l.write().unwrap();
a.insert(1, 1);
})
}
#[bench]
fn bench_spinlock_write_map(b: &mut Bencher) {
let m = HashMap::with_capacity(100);
let l = Arc::new(spin::RwLock::new(m));
b.iter(||{
let mut a = l.write();
a.insert(1, 1);
})
}
#[bench]
fn bench_lock_read_map(b: &mut Bencher) {
let mut m = HashMap::with_capacity(100);
m.insert(1, 1);
let l = Arc::new(RwLock::new(m));
b.iter(||{
let a = l.read().unwrap();
a.get(&1).unwrap();
})
}
#[bench]
fn bench_spinlock_read_map(b: &mut Bencher) {
let mut m = HashMap::with_capacity(100);
m.insert(1, 1);
let l = Arc::new(spin::RwLock::new(m));
b.iter(||{
let mut a = l.read();
a.get(&1).unwrap();
})
}
test tests::bench_lock_read_map ... bench: 47 ns/iter (+/- 4)
test tests::bench_lock_write_map ... bench: 45 ns/iter (+/- 4)
test tests::bench_spinlock_read_map ... bench: 17 ns/iter (+/- 3)
test tests::bench_spinlock_write_map ... bench: 15 ns/iter (+/- 2)
Hi,
I'm currently implementing an exporter using this crate, so I'm mining metrics then exposing them.
This works fine - except I can't set a counter to a value. This is an absolute requirement for an exporter which monitors an external component. At each exporter run I have no idea what the last value of the counter was - so I can't increment by x.
Let me know if a PR would be appropriate.
Like Prometheus java client, like
counter.inc();
counter.with_label_values(&["1", "2"]).inc();
Suggested by @brian-brazil
To reproduce, run example_hyper with RUSTFLAGS="-Z sanitizer=thread" cargo run
on a recent nightly. Then run curl localhost:9898/metrics
twice.
The sanitizer is a new cool feature. See [HowTo] Sanitize your Rust code!
listening addr "127.0.0.1:9898"
==================
WARNING: ThreadSanitizer: data race (pid=18085)
Read of size 8 at 0x55e64c39ac08 by thread T3:
#0 lazy_static::lazy::{{impl}}::get<prometheus::counter::Counter,fn() -> prometheus::counter::Counter> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-0.2.2/src/lazy.rs:19 (prometheus-race+0x00000010a744)
#1 prometheus_race::{{impl}}::deref::__stability /home/bbigras/dev/rust/bugs/prometheus-race/<lazy_static macros>:21 (prometheus-race+0x00000010a744)
#2 prometheus_race::{{impl}}::deref /home/bbigras/dev/rust/bugs/prometheus-race/<lazy_static macros>:22 (prometheus-race+0x00000010a744)
#3 prometheus_race::main::{{closure}} /home/bbigras/dev/rust/bugs/prometheus-race/src/main.rs:44 (prometheus-race+0x00000010a212)
#4 prometheus_race::main::{{closure}} /home/bbigras/dev/rust/bugs/prometheus-race/src/main.rs:44 (prometheus-race+0x00000010a212)
#5 prometheus_race::main::{{closure}} /home/bbigras/dev/rust/bugs/prometheus-race/src/main.rs:44 (prometheus-race+0x00000010a212)
#6 hyper::server::{{impl}}::handle<closure> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/mod.rs:434 (prometheus-race+0x0000000956a0)
#7 hyper::server::{{impl}}::keep_alive_loop<closure,std::io::buffered::BufWriter<&mut hyper::net::HttpStream>> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/mod.rs:341 (prometheus-race+0x00000007c3a5)
#8 hyper::server::{{impl}}::handle_connection<closure,hyper::net::HttpStream> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/mod.rs:287 (prometheus-race+0x00000007d25e)
#9 hyper::server::handle::{{closure}}<closure,hyper::net::HttpListener> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/mod.rs:246 (prometheus-race+0x0000000d10cf)
#10 hyper::server::listener::spawn_with::{{closure}}<hyper::net::HttpListener,closure> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/listener.rs:50 (prometheus-race+0x0000000d1814)
#11 std::panic::{{impl}}::call_once<(),closure> /checkout/src/libstd/panic.rs:296 (prometheus-race+0x000000103fb8)
#12 std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> /checkout/src/libstd/panicking.rs:450 (prometheus-race+0x00000008deb5)
#13 panic_unwind::__rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:98 (prometheus-race+0x000000584c4a)
#14 std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> /checkout/src/libstd/panic.rs:361 (prometheus-race+0x00000008b253)
#15 std::thread::{{impl}}::spawn::{{closure}}<closure,()> /checkout/src/libstd/thread/mod.rs:357 (prometheus-race+0x00000008d2ca)
#16 alloc::boxed::{{impl}}::call_box<(),closure> /checkout/src/liballoc/boxed.rs:638 (prometheus-race+0x0000000c16b6)
#17 alloc::boxed::{{impl}}::call_once<(),()> /checkout/src/liballoc/boxed.rs:648 (prometheus-race+0x00000057cc14)
#18 std::sys_common::thread::start_thread /checkout/src/libstd/sys_common/thread.rs:21 (prometheus-race+0x00000057cc14)
#19 std::sys::imp::thread::{{impl}}::new::thread_start /checkout/src/libstd/sys/unix/thread.rs:84 (prometheus-race+0x00000057cc14)
Previous write of size 8 at 0x55e64c39ac08 by thread T2:
[failed to restore the stack]
Location is global '<null>' at 0x000000000000 (prometheus-race+0x0000009b9c08)
Thread T3 (tid=18279, running) created by thread T1 at:
#0 pthread_create /checkout/src/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:902 (prometheus-race+0x0000004903c7)
#1 std::sys::imp::thread::{{impl}}::new /checkout/src/libstd/sys/unix/thread.rs:72 (prometheus-race+0x00000057c915)
#2 std::thread::spawn<closure,()> /checkout/src/libstd/thread/mod.rs:412 (prometheus-race+0x00000008b372)
#3 hyper::server::listener::spawn_with<hyper::net::HttpListener,closure> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/listener.rs:45 (prometheus-race+0x0000000d15a8)
#4 hyper::server::listener::{{impl}}::accept<hyper::net::HttpListener,closure> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/listener.rs:31 (prometheus-race+0x0000000c9258)
#5 hyper::server::handle::{{closure}}<closure,hyper::net::HttpListener> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/mod.rs:248 (prometheus-race+0x0000000d1026)
#6 std::panic::{{impl}}::call_once<(),closure> /checkout/src/libstd/panic.rs:296 (prometheus-race+0x0000001040ab)
#7 std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> /checkout/src/libstd/panicking.rs:450 (prometheus-race+0x00000008dccb)
#8 panic_unwind::__rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:98 (prometheus-race+0x000000584c4a)
#9 std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> /checkout/src/libstd/panic.rs:361 (prometheus-race+0x00000008b153)
#10 std::thread::{{impl}}::spawn::{{closure}}<closure,()> /checkout/src/libstd/thread/mod.rs:357 (prometheus-race+0x00000008cf4a)
#11 alloc::boxed::{{impl}}::call_box<(),closure> /checkout/src/liballoc/boxed.rs:638 (prometheus-race+0x0000000c176c)
#12 alloc::boxed::{{impl}}::call_once<(),()> /checkout/src/liballoc/boxed.rs:648 (prometheus-race+0x00000057cc14)
#13 std::sys_common::thread::start_thread /checkout/src/libstd/sys_common/thread.rs:21 (prometheus-race+0x00000057cc14)
#14 std::sys::imp::thread::{{impl}}::new::thread_start /checkout/src/libstd/sys/unix/thread.rs:84 (prometheus-race+0x00000057cc14)
Thread T2 (tid=18278, running) created by thread T1 at:
#0 pthread_create /checkout/src/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:902 (prometheus-race+0x0000004903c7)
#1 std::sys::imp::thread::{{impl}}::new /checkout/src/libstd/sys/unix/thread.rs:72 (prometheus-race+0x00000057c915)
#2 std::thread::spawn<closure,()> /checkout/src/libstd/thread/mod.rs:412 (prometheus-race+0x00000008b372)
#3 hyper::server::listener::spawn_with<hyper::net::HttpListener,closure> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/listener.rs:45 (prometheus-race+0x0000000d15a8)
#4 hyper::server::listener::{{impl}}::accept<hyper::net::HttpListener,closure> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/listener.rs:31 (prometheus-race+0x0000000c9258)
#5 hyper::server::handle::{{closure}}<closure,hyper::net::HttpListener> /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.4/src/server/mod.rs:248 (prometheus-race+0x0000000d1026)
#6 std::panic::{{impl}}::call_once<(),closure> /checkout/src/libstd/panic.rs:296 (prometheus-race+0x0000001040ab)
#7 std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> /checkout/src/libstd/panicking.rs:450 (prometheus-race+0x00000008dccb)
#8 panic_unwind::__rust_maybe_catch_panic /checkout/src/libpanic_unwind/lib.rs:98 (prometheus-race+0x000000584c4a)
#9 std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> /checkout/src/libstd/panic.rs:361 (prometheus-race+0x00000008b153)
#10 std::thread::{{impl}}::spawn::{{closure}}<closure,()> /checkout/src/libstd/thread/mod.rs:357 (prometheus-race+0x00000008cf4a)
#11 alloc::boxed::{{impl}}::call_box<(),closure> /checkout/src/liballoc/boxed.rs:638 (prometheus-race+0x0000000c176c)
#12 alloc::boxed::{{impl}}::call_once<(),()> /checkout/src/liballoc/boxed.rs:648 (prometheus-race+0x00000057cc14)
#13 std::sys_common::thread::start_thread /checkout/src/libstd/sys_common/thread.rs:21 (prometheus-race+0x00000057cc14)
#14 std::sys::imp::thread::{{impl}}::new::thread_start /checkout/src/libstd/sys/unix/thread.rs:84 (prometheus-race+0x00000057cc14)
SUMMARY: ThreadSanitizer: data race /home/bbigras/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-0.2.2/src/lazy.rs:19 in lazy_static::lazy::{{impl}}::get<prometheus::counter::Counter,fn() -> prometheus::counter::Counter>
==================
[...]
Examples:
Now we use RwLock<f64>
, but we can use Atomic U64 like golang client to improve the performance.
But be careful that now the stable rust only supports AtomicUsize, so we must use this in 64 bit system only.
And we should do benchmark to see whether this can improve the performance or not.
The MetricVec is at least 2 or 3 times slower than Metric, we should improve this.
If LocalHistogramTimer
is not used, it is dropped immediately so that the timer cannot collect useful info. We need to explicitly state that it is must_use
in case of programmer's mistakes.
Hi,
The last version published on crates.io is 0.2.8, which doesn't compile any more with current Cargo versions.
Could you publish an update to crates.io?
Thanks!
As previously discussed in #141
Open an issue here in case of forgetting.
Double check that this is the recommended approach in client_golang, but it is what they do for the ProcessCollector
: https://github.com/prometheus/client_golang/blob/master/prometheus/process_collector.go#L125-L130
If we want to inc some VecMetric many times, we may first cache in a map then inc in batch.
let m = HashMap::new();
m["a"] += 1
m["b"] += 1
// many times
for (key, value) in &m {
metric.with_label_values(&[key]).add(value);
}
to compare with
metric.with_label_values(&["a"]).inc(1);
metric.with_label_values(&["b"]).inc(1);
// many times
we may refer https://github.com/pingcap/tikv/blob/master/src/util/macros.rs#L66 directly and change a little.
Hi,
just saw that calling register() on a Registry could panic without any possibility for the user to let that error bubble up the chain to be handled gracefully outside the registry (see the call to unnwrap() in https://github.com/pingcap/rust-prometheus/blob/master/src/registry.rs#L203)
Is that intentional? Would it not be better to map over the result of write?
Jan
This is related to #84. Right now, there are 7 direct dependencies, and 36 transitive dependencies. Removing hyper will go a long way, dropping 21 of the dependencies. The other big improvement we can have is getting rid of regex, which will drop a further 10. With both removed, there would be just 5 dependencies, all of which are direct:
fnv (1.0.5)
lazy_static (0.2.1)
libc (0.2.17)
protobuf (1.0.24)
quick-error (0.2.2)
Like
let timer = histogram.start_timer();
// do something
timer.observe_duration();
Hi,
at first: thanks for putting this crate together!!
I am wondering about the use of LocalHistgram: it seems I actively have to call flush() from the individual threads, meaning that I have to decide when to actually flush().
Do you think it makes sense to have any local histogram flushed implicitly when metrics are collected? That way one would not have to worry about periodically calling flush().
Does that thinking make sense and would it work to add it to the create at some point in time (mostly asking this to gain understanding not su much as a feature request yet)
UPDATE: I realise this would require registry to know all local histograms which sort of goes against their very reason. I keep the question open in case it is of interest. Please just close.
Thanks
jan
So we can monitor current running process CPU, Memory, etc.
The push client fails to push metrics after rebooting pushgateways, and halts.
I'm looking to use Prometheus in a Rust program. There are a few libraries on crates. Is this the one that people are coalescing around? It would be good if the README could have some small section stating its status and comparing it to alternatives. A sentence or two would be fine.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.