We are now using https://github.com/elastic-rs/elastic.
The new target should be https://github.com/elastic/elasticsearch-rs

It would be really great to also have this for torque: job files and environment.

After log rotation, the process will be sent a SIGHUP, so this needs to be handled to allow logging to thee new file instead of to the rotated version.

Addressed by #11.

sarchive may run as a service under e.g., systemd. This means it needs to respond adequately to SIGTERM. Ideally, SIGINT is also handled properly.

It is not totally clear what the best course of action would be when receiving such a signal:

1. quit immediately (possible losing the paths that are in the queue)
2. stop accepting new entries into the queue but process those that already have been seen
3. keep accepting new entries, but bail once the queue is empty.

Each of these obviously have advantages and disadvantages.

We currently have a very flat format, i.e., job.<jobid>_script and job.<jobid>_environment. While this suffices for finding job scripts, it has several drawbacks.

• there can be many jobs in the archive, meaning the number of entries in the single archival directory will become quite large.
• users may not always recall the exact job ID (there might be a few) and looking for a time might help pin down the problematic job.

A better archive could be organised by

• user
• cluster
• timestamps (e.g., yearly, monthly, daily, ...)

This issue occurs in the torque branch, where there is often a single directory to watch and the scheduler may create a large number of file when a job array is submitted.

To reproduce:

• start sarchive in torque mode, watching a single directory
• create a large number of files in this directory, e.g. 10K

Increasing the queue size through echo 65535 > /proc/sys/fs/inotify/max_queued_events does not always help. After creating 50 K files with 64K event queue size, the execution stopped at

[2019-06-13T22:08:58.955897586+02:00][sarchive::lib][INFO] copied 11 bytes from "/tmp/torque/17354.SC" to "/tmp/archive/job.17354_SC"
[2019-06-13T22:08:58.956109575+02:00][sarchive::lib][INFO] copied 11 bytes from "/tmp/torque/17355.SC" to "/tmp/archive/job.17355_SC"

Issue occurs with

• kernel 3.10.0-957.12.2.el7.ug.x86_64. (custom build based on the version number)
• glibc-2.17-260.el7_6.5.x86_64
• notify crate main branch with commit #56aac12
• running on a kvm VM with storage in a Ceph backend

How to compile and install ob ubuntu 20.04?
root@server1:/opt/archive/sarchive-production/src# rustc main.rs
error[E0463]: can't find crate for chrono
--> monitor.rs:22:1
|
22 | extern crate chrono;
| ^^^^^^^^^^^^^^^^^^^^ can't find crate

error[E0463]: can't find crate for crossbeam_channel
--> monitor.rs:23:1
|
23 | extern crate crossbeam_channel;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate

error[E0463]: can't find crate for crossbeam_utils
--> monitor.rs:24:1
|
24 | extern crate crossbeam_utils;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate

error: environment variable CARGO_PKG_VERSION not defined
--> main.rs:48:23
|
48 | const VERSION: &str = env!("CARGO_PKG_VERSION");
| ^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: this error originates in the macro env (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0432]: unresolved import clap
--> main.rs:23:5

Watchers are registered with the kernel, but if the directory vanishes, the watcher stops getting notification for the FD it is subscribed for. The FD for the new directory is obviously different, meaning that we are now not watching that directory.

A fix could be to watch a removal event of the watched directory. For this, we probably need an extra watcher that will signal the threads to cease operations and restart them, while waiting for the new directories to become available.

If this happens frequently, it might induce a race condition, where we ask to watch locations that have since disappeared again. It should be OK to quit with an error in this case.

• The final character seems to be \0
• This is the case in the files written by Slurm (not sure if it is a feature to have a \0-terminated string when reading the data)

# hexdump script
<snip>
0000130 2070 203c 616c 6d6d 7370 692e 0a6e 0000
000013f

• We can pinch off that final char to have an ASCII file that is saved

Needed:

• A config file to describe the output choices and settings for ES

  • Should be extensible, i.e., support for other output channels baked in from the start
  • Should be in a simple format (TOML?)

• The script file can be sent as is

• The environment file should probably be parsed and sent as a JSON object with an entry per item in the file?