famedly / barad-dur Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 0.0 180 KB

matrix phone-home stats collector

Dockerfile 4.48% Rust 95.52%

barad-dur's People

Contributors

Stargazers

Watchers

barad-dur's Issues

review me - [closed]

In GitLab by @Shekhinah on Mar 9, 2021, 19:19

Merges main -> shekhinah/code-review

Database migrations

In GitLab by @jcgruenhage on Feb 15, 2021, 13:41

To help with operations, this tool should automatically do migrations when upgrading to a newer version with a new database schema. sqlx has some built-in support for this, which https://gitlab.com/famedly/services/user-directory/-/tree/main/migrations uses, so that can be used as an inspiration.

Support data migration from panopticon

In GitLab by @jcgruenhage on Feb 15, 2021, 13:27

Workflow for this is probably reading a dump of panopticon into temporary tables and then moving the data over to the correct tables. Should happen in a separate SQL script.

feat: add cmd arguments - [merged]

In GitLab by @Shekhinah on Mar 9, 2021, 13:35

Merges shekhinah/cmd-config-argument -> main

migrations, stats aggregation Ref #4 #5 - [merged]

In GitLab by @Shekhinah on Mar 1, 2021, 14:26

Merges shekhinah/aggregate-stats-reports -> main

I've changed again the table format in the migrations file. As this project isn't in use I think that's still legal.

Make ready for release - [merged]

In GitLab by @jcgruenhage on Oct 26, 2021, 13:25

Merges jcgruenhage/axum -> main

Fixes #6 and #7, as well as supporting offline compilation (not requiring a database for compilation) and replacing actix with axum.

feat: add ci script - [merged]

In GitLab by @Shekhinah on Mar 9, 2021, 10:45

Merges shekhinah/ci -> main

chore: bump version to 0.1.2 and update changelog - [merged]

In GitLab by @lrsksr on Apr 19, 2023, 13:41

Merges lrsksr/chore/release-v0.1.2 -> main

fix: make config.sample.yaml more helpful - [merged]

In GitLab by @Shekhinah on Mar 9, 2021, 16:54

Merges shekhinah/more-helpful-config-sample -> main

More release-ready-making - [merged]

In GitLab by @jcgruenhage on Oct 28, 2021, 10:06

Merges jcgruenhage/import-fixes -> main

I've tried out the import scripts and looked at them again in more details, and found a few things that I wasn't quite happy with. After fixing those up, I checked whether the aggregation worked, and no, while it did aggregate data, the data was off in comparison to what I expected. After checking again and modifying the query, I was able to get that into the right shape, although not without some support from an experienced SQL magician. Last but not least, I thought to myself: If I'm already doing all that stuff, I might as well refactor the code a bit, update deps, update to the newly stabilized 2021 edition of rust and extend documentation.

Multi-Stage Dockerfile instead of copying in binary

In GitLab by @jcgruenhage on Sep 2, 2021, 10:49

There's multiple issues with the current Dockerfile:

after a clean clone of the repo, docker built . will not work, as the release binary is copied in instead of being built in the container
the resulting container image is gigantic, because it's based on the rust container image

Something like https://gitlab.com/famedly/company/backend/services/CRM/-/blob/main/Dockerfile should work well

Use CI-CD templates instead of own script

In GitLab by @jcgruenhage on Aug 23, 2021, 09:48

Optimise DB schema for queries against the non-aggregated table

We'll likely need some indices on the homeserver and local_timestamp columns to get those queries to be more performant.

http server and sql client - [merged]

In GitLab by @Shekhinah on Feb 22, 2021, 09:43

Merges develop -> main

Rebased the commits and implemented suggested changes.

feat: build barad-dur docker container - [closed]

In GitLab by @jdreichmann on Apr 15, 2021, 15:00

Merges transcaffeine/package-docker -> main

'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: IndeterminateOffset'

In GitLab by @dnelson2023 on Feb 23, 2023, 24:29

I've deployed Barad-dur in a kubernetes environment using image: registry.gitlab.com/famedly/infra/services/barad-dur:latest.

The pod appears to be running and in the ready state, but it hasn't initialized the db table(s). It looks like it failed in setup_logging(). I'm hoping you can provide some info on how to fix this.

The following is logged:

thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: IndeterminateOffset', src/main.rs:23:22
stack backtrace:
   0:     0x7f5716902560 - std::backtrace_rs::backtrace::libunwind::trace::h8c197fa9a679d134
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x7f5716902560 - std::backtrace_rs::backtrace::trace_unsynchronized::h9ee19d58b6d5934a
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f5716902560 - std::sys_common::backtrace::_print_fmt::h4badc450600fc417
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x7f5716902560 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::had334ddb529a2169
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7f571693ee1e - core::fmt::write::h1aa7694f03e44db2
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/fmt/mod.rs:1209:17
   5:     0x7f57168fcd15 - std::io::Write::write_fmt::h61b2bdc565be41b5
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/io/mod.rs:1682:15
   6:     0x7f5716902325 - std::sys_common::backtrace::_print::h4ec69798b72ff254
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:47:5
   7:     0x7f5716902325 - std::sys_common::backtrace::print::h0e6c02048dec3c77
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:34:9
   8:     0x7f5716903bdf - std::panicking::default_hook::{{closure}}::hcdb7e705dc37ea6e
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:267:22
   9:     0x7f571690391a - std::panicking::default_hook::he03a933a0f01790f
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:286:9
  10:     0x7f57169042d8 - std::panicking::rust_panic_with_hook::he26b680bfd953008
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:688:13
  11:     0x7f5716904077 - std::panicking::begin_panic_handler::{{closure}}::h559120d2dd1c6180
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:579:13
  12:     0x7f5716902a0c - std::sys_common::backtrace::__rust_end_short_backtrace::h36db621fc93b005a
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys_common/backtrace.rs:137:18
  13:     0x7f5716903d92 - rust_begin_unwind
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:575:5
  14:     0x7f57165d1cc3 - core::panicking::panic_fmt::he7679b415d25c5f4
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:65:14
  15:     0x7f57165d1f83 - core::result::unwrap_failed::hb71caff146724b6b
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/result.rs:1791:5
  16:     0x7f57166575e0 - barad_dur::setup_logging::{{closure}}::h0df4cdb1401c1e14
  17:     0x7f571669830d - <fern::log_impl::Dispatch as log::Log>::log::h5b45ca66e1c7bbb4
  18:     0x7f5716897ebf - log::__private_api_log::hdb9ae2420b64e492
  19:     0x7f571660212f - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h0b6d7ca9358ddcb2
  20:     0x7f57166151a8 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::he71c60bdd6da9319
  21:     0x7f57165ea241 - tokio::runtime::task::core::Core<T,S>::poll::h50d165a00360f808
  22:     0x7f5716638967 - tokio::runtime::task::harness::Harness<T,S>::poll::h09a29a767476b9d7
  23:     0x7f571686b9c6 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::hc0f0ebb7679c7246
  24:     0x7f571686b5b0 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h87323f8ea004a904
  25:     0x7f571686e2d9 - tokio::macros::scoped_tls::ScopedKey<T>::set::h13371b3ca5c3ad02
  26:     0x7f571686acc1 - tokio::runtime::scheduler::multi_thread::worker::run::h8b3956965431363a
  27:     0x7f5716885732 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h9c10fae310dfcaf6
  28:     0x7f57168819ac - tokio::runtime::task::core::Core<T,S>::poll::h54a37ace3227557a
  29:     0x7f571687fb2f - tokio::runtime::task::harness::Harness<T,S>::poll::ha43acb01d8765877
  30:     0x7f5716872869 - tokio::runtime::blocking::pool::Inner::run::h2ad2814d0694f87e
  31:     0x7f5716872ef2 - std::sys_common::backtrace::__rust_begin_short_backtrace::hf5dd45ba19f3f8fb
  32:     0x7f5716878c65 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h93cdc31f609e70dc
  33:     0x7f5716906af3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hed812350c5aef7a8
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/alloc/src/boxed.rs:1987:9
  34:     0x7f5716906af3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc7df8e435a658960
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/alloc/src/boxed.rs:1987:9
  35:     0x7f5716906af3 - std::sys::unix::thread::Thread::new::thread_start::h575491a8a17dbb33
                               at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/sys/unix/thread.rs:108:17

Configure CI jobs

chore: add codeowners - [merged]

In GitLab by @nikzen on Apr 3, 2023, 10:50

Merges larodar-main-patch-21630 -> main

feat: better tests - [merged]

In GitLab by @Shekhinah on Mar 9, 2021, 16:41

Merges shekhinah/better-testing -> main

Aggregation

In GitLab by @jcgruenhage on Feb 15, 2021, 13:38

https://github.com/matrix-org/panopticon has a separate script in python, that handles aggregation. We want to handle this in the same tool, so we should write a future that handles this instead. For this, we need a timer loop, which spawns this future once a day (or do we want to spawn it more often?) to do the daily compaction.

fix: Handle timezone being unavailable during logging setup - [merged]

In GitLab by @agraven on Feb 23, 2023, 06:52

Merges agraven/no-timezone-fix -> main

Should fix #9

Unit testing

In GitLab by @jcgruenhage on Feb 15, 2021, 13:29

This should support phone home stats for different versions of synapse, and as far as other servers implement it, phone home stats of those other homeservers too. This is a future feature though, so it can wait a bit. For the first release, just a single request from the most recent synapse release is sufficient.

Collection endpoint

In GitLab by @jcgruenhage on Feb 15, 2021, 13:33

This should implement an HTTP endpoint, which takes in a JSON endpoint like https://github.com/matrix-org/panopticon does, and stores it into the database.

For this http endpoint, we need a simple actix-web server. https://gitlab.com/famedly/services/organisation-directory/ uses actix-web for http, but it's important not to take over too much, as the organisation directory does not use sqlx, but diesel instead, so the database interaction will be different.

feat: add http server and sql client - [merged]

In GitLab by @Shekhinah on Feb 25, 2021, 17:07

Merges shekhinah/collect-stat-reports -> main

Add template sync action

The sync action template should be set up for this repository. See the parent issue for more information.

Build binaries in docker images with cargo-auditable

As part of the work for auditable containers (https://github.com/famedly/product-management/issues/1446), we should use cargo auditable in all Dockerfiles to build our rust binaries, as that will embed info about the dependency tree in the binary itself.

In practice, this just means going through the project's Dockerfiles and replacing cargo build --release with cargo auditable build --release.

Update dependencies to fix RUSTSEC Advisories

In GitLab by @dominikrimpf on Dec 18, 2022, 20:50

Following errors appeared with cargo audit:

    Fetching advisory database from `https://github.com/RustSec/advisory-db.git`
      Loaded 474 security advisories (from /Users/dominik/.cargo/advisory-db)
    Updating crates.io index
warning: couldn't update crates.io index: registry: SecureTransport error: (null); class=Net (12)
    Scanning Cargo.lock for vulnerabilities (177 crate dependencies)
Crate:     chrono
Version:   0.4.19
Title:     Potential segfault in `localtime_r` invocations
Date:      2020-11-10
ID:        RUSTSEC-2020-0159
URL:       https://rustsec.org/advisories/RUSTSEC-2020-0159
Solution:  Upgrade to >=0.4.20
Dependency tree:
chrono 0.4.19
├── sqlx-core 0.5.9
│   ├── sqlx-macros 0.5.9
│   │   └── sqlx 0.5.9
│   │       └── barad-dur 0.1.0
│   └── sqlx 0.5.9
└── barad-dur 0.1.0

Crate:     regex
Version:   1.5.4
Title:     Regexes with large repetitions on empty sub-expressions take a very long time to parse
Date:      2022-03-08
ID:        RUSTSEC-2022-0013
URL:       https://rustsec.org/advisories/RUSTSEC-2022-0013
Solution:  Upgrade to >=1.5.5
Dependency tree:
regex 1.5.4
├── serde-hjson 0.9.1
│   └── config 0.11.0
│       └── barad-dur 0.1.0
└── axum 0.2.8
    └── barad-dur 0.1.0

Crate:     time
Version:   0.1.44
Title:     Potential segfault in the time crate
Date:      2020-11-18
ID:        RUSTSEC-2020-0071
URL:       https://rustsec.org/advisories/RUSTSEC-2020-0071
Solution:  Upgrade to >=0.2.23
Dependency tree:
time 0.1.44
└── chrono 0.4.19
    ├── sqlx-core 0.5.9
    │   ├── sqlx-macros 0.5.9
    │   │   └── sqlx 0.5.9
    │   │       └── barad-dur 0.1.0
    │   └── sqlx 0.5.9
    └── barad-dur 0.1.0

Crate:     tokio
Version:   1.12.0
Title:     Data race when sending and receiving after closing a `oneshot` channel
Date:      2021-11-16
ID:        RUSTSEC-2021-0124
URL:       https://rustsec.org/advisories/RUSTSEC-2021-0124
Solution:  Upgrade to >=1.8.4, <1.9.0 OR >=1.13.1
Dependency tree:
tokio 1.12.0
├── tower 0.4.10
│   ├── barad-dur 0.1.0
│   └── axum 0.2.8
│       └── barad-dur 0.1.0
├── tokio-util 0.6.8
│   ├── tower 0.4.10
│   └── axum 0.2.8
├── tokio-stream 0.1.7
│   └── sqlx-core 0.5.9
│       ├── sqlx-macros 0.5.9
│       │   └── sqlx 0.5.9
│       │       └── barad-dur 0.1.0
│       └── sqlx 0.5.9
├── tokio-rustls 0.22.0
│   └── sqlx-rt 0.5.9
│       ├── sqlx-macros 0.5.9
│       └── sqlx-core 0.5.9
├── sqlx-rt 0.5.9
├── hyper 0.14.14
│   ├── barad-dur 0.1.0
│   └── axum 0.2.8
├── barad-dur 0.1.0
└── axum 0.2.8

Crate:     tower-http
Version:   0.1.1
Title:     Improper validation of Windows paths could lead to directory traversal attack
Date:      2022-01-21
ID:        RUSTSEC-2022-0043
URL:       https://rustsec.org/advisories/RUSTSEC-2022-0043
Solution:  Upgrade to >=0.2.1 OR >=0.1.3, <0.2.0
Dependency tree:
tower-http 0.1.1
└── axum 0.2.8
    └── barad-dur 0.1.0

Could you bump your dependency versions to fix these issues?

fix: keep datapoints with additional fields - [merged]

In GitLab by @jcgruenhage on Aug 23, 2021, 24:27

Merges jcgruenhage/additional-fields -> main

To avoid loosing data on synapse updates which add new metrics, we shouldn't have #[serde(deny_unknown_fields)] in there.

Reported by @reivilibre

famedly / barad-dur Goto Github PK

barad-dur's People

Contributors

Stargazers

Watchers

barad-dur's Issues

Recommend Projects

Recommend Topics

Recommend Org