Extracted from #19

---- commands::tests::timespan_next_ stdout ----
thread 'commands::tests::timespan_next_' panicked at 'assertion failed: `(left == right)`
  left: `2020-01-01T00:00:00+00:00`,
 right: `2019-12-31T13:00:00Z`: year 2019-01-01T00:00:00+00:00', src/commands.rs:852:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I'm guessing that this is a timezone issue, despite trying to force the unittest to UTC. Maybe a mismatch between the UTC test and the Local code ? Not sure yet.

@flexibeast what is your locale and timezone, so I can try to reproduce ?

The emlop binary could help in generating possible names (UI would be to add another argument after emlop complete <shell> <partial string>), but we still need to hook that into the shell completion generated by clap.

On windows, atty dereferences a potentially unaligned pointer.

In practice however, the pointer won't be unaligned unless a custom global allocator is used.

In particular, the System allocator on windows uses HeapAlloc, which guarantees a large enough alignment.

atty is Unmaintained

A Pull Request with a fix has been provided over a year ago but the maintainer seems to be unreachable.

Last release of atty was almost 3 years ago.

Possible Alternative(s)

The below list has not been vetted in any way and may or may not contain alternatives;

• std::io::IsTerminal - Stable since Rust 1.70.0
• is-terminal - Standalone crate supporting Rust older than 1.70.0

Potential segfault in the time crate

Impact

Unix-like operating systems may segfault due to dereferencing a dangling pointer in specific circumstances. This requires an environment variable to be set in a different thread than the affected functions. This may occur without the user's knowledge, notably in a third-party library.

The affected functions from time 0.2.7 through 0.2.22 are:

• time::UtcOffset::local_offset_at
• time::UtcOffset::try_local_offset_at
• time::UtcOffset::current_local_offset
• time::UtcOffset::try_current_local_offset
• time::OffsetDateTime::now_local
• time::OffsetDateTime::try_now_local

The affected functions in time 0.1 (all versions) are:

• at
• at_utc

Non-Unix targets (including Windows and wasm) are unaffected.

Patches

Pending a proper fix, the internal method that determines the local offset has been modified to always return None on the affected operating systems. This has the effect of returning an Err on the try_* methods and UTC on the non-try_* methods.

Users and library authors with time in their dependency tree should perform cargo update, which will pull in the updated, unaffected code.

Users of time 0.1 do not have a patch and should upgrade to an unaffected version: time 0.2.23 or greater or the 0.3. series.

Workarounds

No workarounds are known.

References

time-rs/time#293

See advisory page for additional details.

Shouldn't be too hard for tables with a single header line, but would be trickier for stats. Multi-header tables can probably be ignored for now, as they don't make use of --last anyway.

One solution would be to only show the headers if different from the previous flush. Another would be to group stats by kind instead of by date.

Here's the relevant part of the command output:

   Compiling serde_json v1.0.39
   Compiling failure v0.1.5
   Compiling assert_cli v0.6.3
   Compiling emlop v0.3.1 (/home/tasso/Documents/prog/git/emlop)
error[E0061]: this function takes 9 parameters but 8 parameters were supplied
   --> src/parser.rs:314:20
    |
51  | / pub fn new_hist<R: Read>(reader: R,
52  | |                          filename: String,
53  | |                          min_ts: Option<i64>,
54  | |                          max_ts: Option<i64>,
...   |
110 | |     Ok(rx)
111 | | }
    | |_- defined here
...
314 |           let hist = new_hist(File::open(filename).unwrap(),
    |  ____________________^
315 | |                             filename.into(),
316 | |                             filter_mints,
317 | |                             filter_maxts,
...   |
320 | |                             filter_pkg,
321 | |                             exact).unwrap();
    | |__________________________________^ expected 9 parameters

error: aborting due to previous error

For more information about this error, try `rustc --explain E0061`.
error: could not compile `emlop`.

To learn more, run the command again with --verbose.
➜  emlop git:(master)

Steps to reproduce:

• emerge foo in first terminal
• wait 5 secs
• emerge bar in second terminal
• Wait until emlop p in third terminal looks ok
• Ctrl-C emerge bar process
• emlop p in third terminal shows both foo and bar as currently emerging (should only see foo)

This was a know limitation at implementaion time, but I'm not sure how to properly solve it (emerge.log doesn't clearly identify the failed emerge process). Qlop and genlop get it right though, so it's possible ;)

At the moment, predict only shows the package which is emerged right now, but when emerging several packages (e.g. emerge ... @world), I also would love to see the predicted time for all packages to complete.

Want: Shows all 17 items with time estimations, prediction-accuracy and completion status

Switch to a different version of libc to resolve this issue.

Split off from #19

---- proces::tests::start_time stdout ----
thread 'proces::tests::start_time' panicked at 'Couldn't parse PID STARTED', src/proces.rs:130:29

@flexibeast what's your version of ps ? Mine (sys-process/procps-3.3.17-r1) doesn't output headers with the given options, but looking at the manpage it looks like --no-headers would be more safer than -h.

Strings like 2024-03-19T14:58:31+00:00 or 2024-03-19T14:58:31Z should be valid.

Calculating dependencies \
... done!
[ebuild  r  U  ] sys-libs/readline-7.0_p3:0/7::gentoo [6.3_p8-r3:0/0::gentoo] USE="-static-libs -utils" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild  rR    ] dev-libs/libpcre-8.41-r1:3::gentoo  USE="bzip2 cxx jit pcre16 readline recursion-limit (unicode) zlib -libedit -pcre32 -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB
[ebuild     U  ] app-shells/bash-4.4_p12::gentoo [4.3_p48-r1::gentoo] USE="net nls (readline) -afs -bashlogger -examples -mem-scramble -plugins" 0 KiB
[ebuild  rR    ] dev-libs/libpcre2-10.30::gentoo  USE="bzip2 jit pcre16 readline recursion-limit unicode zlib -libedit -pcre32 -static-libs" ABI_X86="(64) -32 (-x32)" 0 KiB

Total: 36 packages (3 upgrades, 33 reinstalls), Size of downloads: 0 KiB

Readline and bash parse incorrectly in this example. Fix is probably easy, but should come with a unittest.

❯ emlop -V
emlop 0.6.1   # Installed from Gentoo repo

To reproduce:

1. Run # emerge --sync
2. Run $ emlop l -ss -n 10
3. Latest dates for each repo sync are correct
4. Run # eix-sync
5. Run $ emlop l -ss -n 10
6. Latest dates for each repo sync are correct as well
7. Run # emaint sync -a
8. Run $ emlop l -ss -n 10 again
9. Dates for each repo sync haven't changed, those from # eix-sync are still the latest

From Gentoo Wiki - Repository_synchronization:

The emerge --sync command is now only a compatibility command. Primary control of all sync operations has been moved from emerge to emaint, and emerge --sync now just calls the emaint sync module with the --auto option. This performs a sync on only those repositories with the auto-sync setting set to yes or true. If the auto-sync option is not set or is absent for the configured repositories, then emerge --sync may sync no repositories at all.

Edit: I'm actually getting the same behavior with qlop -s.

I wonder why none of the sync operations are parsed when using emaint.

Edit 2: Turns out # emaint sync -a doesn't write anything to /var/log/emerge.log, unlike # emerge --sync or # eix-sync.

Closing as this is irrelevant, there is nothing wrong with emlop.

Observed

Currently emlop just labels everything as Sync, with one entry per repository. So if you have many repos enabled (via eselect-repository or something else) you'll get that many sync events instead of one event per emerge --sync you might expect.

Expected

• Show the repo name
• Treat each repo sync as distinct entries for stats
• Enable filtering on repo name

Typically rated by log rotation tools.

At least gzip, but let's see what formats we can easily read. Needs format autodetect.

For long builds logs (for example firefox: 1.4M lines, with the last >>> near the start at line 700), read_buildlog() can take over 1s (and it used to be close to 2s with the privious release of the rev_lines crate). Timings are longer during a build (either due to cpu use or file fragmentation). Given that b3sum or ripgrep can process the same file in under 100ms, we should be able to speed emlop up here.

Perhaps parsing the file front to back is the right thing to do. Could hopefully use fast string search algos.

Because the max width is computed over the whole parsed rows, not just the displayed ones. Guess we need to compute widths just before flush, not during add.

The tests are valid, but because we use Instant::now() to create the expected output, this can fail. On my machine this only happens under load.

Not sure how to fix it, we could mock Instant but this seems a bit heavy-handed. Being more lenient on the date output (we can remain strict on the duration output) might be more pragmatic.

Want to showcase the colors, maybe even use cli-movie format. Bonus point if there's a script to regenerate the screenshots.

The fix is easy (add a tab when printing the unknown), but unit-testing it is the real TODO.

Fix is easy, but this should be taken as an opportunity to add a unit test.

Because of the way we handle matching on exact package name as well as exact package category/name.
While we're at it, docs should mention that filtering works on sync repos too, not just packages.

It's already in GURU, but it should be in the main portage tree and in the wiki.

Current code just thinks the build took negative time to complete. Fixed code should:

• show merge time as 0s
• ignore these merge time for future prediction
• display a warning (like we do for non-utf8 data)

Given this emerge -p input:

[ebuild     U  ] app-admin/syslog-ng-3.13.2 [3.7.3] USE="-http%" 
[uninstall     ] dev-libs/eventlog-0.2.12 
[blocks b      ] dev-libs/eventlog ("dev-libs/eventlog" is blocking app-admin/syslog-ng-3.13.2)
[ebuild  r  U  ] dev-lang/php-7.1.13 [7.1.11]

emlop p outputs:

app-admin/syslog-ng                                                          4:05
dev-libs/eventlog                                                              22
dev-libs/eventlog ("dev-libs/eventlog" is blocking app-admin/syslog-ng  
dev-lang/php                                                                20:41

Expected:

app-admin/syslog-ng                                                          4:05
dev-lang/php                                                                20:41

Eg emlop l gcc llvm.

Could just try each search term each time, or create a combined search regex or aho-corasic struct. Need to bench what's better.

Once we have that, it could probably be used to speedup emlop p.

ansi_term is Unmaintained

The maintainer has adviced this crate is deprecated and will not
receive any maintenance.

The crate does not seem to have much dependencies and may or may not be ok to use as-is.

Last release seems to have been three years ago.

Possible Alternative(s)

The below list has not been vetted in any way and may or may not contain alternatives;

• anstyle
• console
• nu-ansi-term
• yansi

See advisory page for additional details.

When a package is new, we can't predict its merge time or use the elapsed time to improve global prediction, but it would still be nice to show how long it's been merging.

Actual:

$ emlop p
Pid 7094: ... -v --backtrack=100 -av opencascade      22:25
sci-libs/vtk-7.1.0                                
Estimate for 1 ebuilds (1 unknown, 0 elapsed)             0

Desired:

$ emlop p
Pid 7094: ... -v --backtrack=100 -av opencascade      22:25
sci-libs/vtk-7.1.0                                          - 10:12
Estimate for 1 ebuilds (1 unknown, 10:12 elapsed)         0

See https://www.reddit.com/r/Gentoo/comments/1avq4dq/comment/krfxjy0/

Would be useful for "Total time running emerge, including dep resolution but correctly accounting for parallel merges" stats, and maybe other aspects.

difference is unmaintained

The author of the difference crate is unresponsive.

Maintained alternatives:

• dissimilar

• treediff

• diffus

See advisory page for additional details.

From what I can see it appears to be mostly related to date/tz conversion. predict_tty seems to fail due to the test checking for the wrong output. predict_tty checks for No pretended merge found, but the command that is run outputs No ongoing merge found as it is called without parsing logs through a pipe.

https://dpaste.com/8ZPP9N73K.txt

See https://bugs.gentoo.org/784008

In other words, this:

1225702647: Started emerge on: Nov 03, 2008 08:57:27
1225702647:  *** emerge --tree --deep --ask --update --verbose world
1225702692:  >>> emerge (1 of 10) sys-apps/portage-2.2_rc13 to /
1225702699:  === (1 of 10) Cleaning (sys-apps/portage-2.2_rc13::/usr/portage/sys-apps/portage/portage-2.2_rc13.ebuild)
1225702699:  === (1 of 10) Compiling/Merging (sys-apps/portage-2.2_rc13::/usr/portage/sys-apps/portage/portage-2.2_rc13.ebuild)
1225702705:  === (1 of 10) Merging (sys-apps/portage-2.2_rc13::/usr/portage/sys-apps/portage/portage-2.2_rc13.ebuild)
1225702719:  === (1 of 10) Post-Build Cleaning (sys-apps/portage-2.2_rc13::/usr/portage/sys-apps/portage/portage-2.2_rc13.ebuild)
1225702719:  ::: completed emerge (1 of 10) sys-apps/portage-2.2_rc13 to /
1225702719:  ::: completed emerge (1 of 10) sys-apps/portage-2.2_rc13 to /
1225702719:  *** RESTARTING emerge via exec() after change of portage version.
1225702719:  *** terminating.
1225702720: Started emerge on: Nov 03, 2008 08:58:40
1225702720:  *** emerge --tree --update --resume --verbose --ignore-default-opts --deep
1225702722:  *** exiting unsuccessfully with status '1'.
1225702725:  *** terminating.

causes this:

2008-11-03 08:58:39 +00:00        27 sys-apps/portage-2.2_rc13
2008-11-03 08:58:39 +00:00        -1 sys-apps/portage-2.2_rc13

I have one package currently failing to compile and install (games-util/gamemode-9999 because ebuild updates haven't been merged yet).

If running emlop p after the package failed, it remains in the prediction list with its projected build time increasing until emerge ultimately finished. At least it looks like this wrong compile time isn't recorded anywhere so it won't affect future predictions.

emlop l | head terminates on a panic when the pipe closes. Should terminate silently instead. Check emlop s | head too as it outputs vis tabwriter, fix may be different.

Currently defaults to /var/tmp/ and has a --tmpdir option, but it'd be nice to automate as this is a common config tweak.

Problem is that it's parsed by bash from /etc/portage/make.conf and various other locations (for example /etc/portage/env/* for per-package config), but I'd rather avoid invoking the full machinery.

Maybe one trick would be to accept multiple dirs internally, and try them in order in parse::current::get_buildlog(). We won't need fancy bash logic in that case, just a list of files to search with a regex.

It would be nice to have an option to show something like (42 lines skipped) when --first/--last skips some lines.

Eg emlop stats --a<tab> completes the --avg flag but emlop s --a<tab> doesn't. Checked for bash only. This is arguably an upstream clap bug, but I'd like to track it here.

Typically created by log rotation tools.

Potential segfault in localtime_r invocations

Impact

Unix-like operating systems may segfault due to dereferencing a dangling pointer in specific circumstances. This requires an environment variable to be set in a different thread than the affected functions. This may occur without the user's knowledge, notably in a third-party library.

Workarounds

No workarounds are known.

References

• time-rs/time#293

See advisory page for additional details.

Firstly, thanks for emlop!

This just a small thing: i find times like a bare "45" for "45 seconds" to be disorienting, as from other contexts i'm used to sub-minute times being represented as e.g. "0:45" or "0:0:45". (And in those contexts, a bare "45" is used to mean "45 minutes" or "45 hours".)

Might you be willing to change the output format accordingly, or make this an option? (Unless this option already available, and i've missed it ....)

emerge --keep-going might end up using the "backup" resume list from /var/cache/edb/mtimedb when restarting, so emlop should read that if the main resume list is empty.

Or maybe that's still a fallible heuristic, and we need something smarter ?

Current merges are shown in random order (changes from one invocation to the next). They should be in merge start order.