assert-rs / assert_cmd Goto Github PK
View Code? Open in Web Editor NEWAssert process::Command - Easy command initialization and assertions
Home Page: https://docs.rs/assert_cmd
License: Apache License 2.0
Assert process::Command - Easy command initialization and assertions
Home Page: https://docs.rs/assert_cmd
License: Apache License 2.0
As described mesalock-linux/mesabox#41. I don't know why the first tests are always blocking for over 60 seconds. However, after the blocking issue, all testcases can be run without any problem.
test gnu::arch::test_x86_64 ... test gnu::arch::test_x86_64 has been running for over 60 seconds
test gnu::base32::test_decode ... test gnu::base32::test_decode has been running for over 60 seconds
This only happens in Travis: https://travis-ci.org/mesalock-linux/mesabox/jobs/411140601#L1305
I cannot reproduce this issue in my machine. Don't know why.
(Not sure if this should be an escargot question...)
I'm writing integration tests for different feature flags.
This is how I'm setting up the test env:
impl TestEnv {
fn new() -> Self {
TestEnv {
cache_dir: TempDir::new(".tldr.test.cache").unwrap(),
config_dir: TempDir::new(".tldr.test.config").unwrap(),
input_dir: TempDir::new(".tldr.test.input").unwrap(),
default_features: true,
features: vec![],
}
}
/// Disable default features.
fn no_default_features(mut self) -> Self {
self.default_features = false;
self
}
/// Add the specified feature.
fn with_feature<S: Into<String>>(mut self, feature: S) -> Self {
self.features.push(feature.into());
self
}
/// Return a new `Command` with env vars set.
fn command(&self) -> Command {
let mut build = escargot::CargoBuild::new()
.bin("tldr")
.current_release()
.current_target();
if !self.default_features {
build = build.arg("--no-default-features");
}
if !self.features.is_empty() {
build = build.arg(&format!("--feature {}", self.features.join(",")));
}
let run = build.run().unwrap();
let mut cmd = run.command();
println!("Command: {:?}", cmd);
cmd.env("TEALDEER_CACHE_DIR", self.cache_dir.path().to_str().unwrap());
cmd.env("TEALDEER_CONFIG_DIR", self.config_dir.path().to_str().unwrap());
cmd
}
}
I can then invoke the binary with various feature configs like this:
TestEnv::new().with_feature("logging").command()...
TestEnv::new().with_feature("networking").command()...
TestEnv::new().no_default_features().command()...
The problem is that when tests run in parallel, some of them randomly fail. They pass if I run them sequentially.
The way I understand it, escargot runs cargo for the specified target. It's cargo's job to determine whether to rebuild or not.
If the arguments change, then of course cargo will do a rebuild. But since tests run in parallel, it's possible that the "wrong" binary is used while the rebuild is still in process...
Is there a way to use a separate build directory per feature flag configuration? Or maybe you could add some kind of opt-in mutex feature to assert_cmd
that enforces assert_cmd
based tests to run sequentially?
Use cases
Considerations
assert_cmd
is testing Rust binaries, so we should help those usersassert_cmd
and would be best to be stablized outside of assert_cmd
Proposal
cargo run
in escargot but via CargoBuild
CargoBuild
would have a .run()
method that would return a CargoRun
CargoRun
would have .path()
and .command()
methods.CommandCargoExt
but point people to escargot
for any custom behaviorescargot
for those use cases.
At this point, CommandCargoExt
is primarily serving a low-effort path for a "getting started" user while serving as an example of how to do more complex things (via code) and pointing to more complex APIs (via docs)
Open Issues
This also brings it to parity with assert_cli
You actually might want to consider refactoring the current code to write stdin on a separate thread and then read stdout/stderr in parallel to avoid possible deadlock. Right now if the spawned process produces enough output to fill up its stdout or stderr buffers it'll block trying to write those, and if you are still writing stdin data you'll be blocked as well. I've got a
wait_with_input_output
function that does this in sccache which you can copy if you'd like. (Linking to an older version because the current source uses tokio-io and futures.)
So far, we haven't run into this but
From @rcoh assert-rs/assert_cli#100
assert_cli
version: 0.5cargo run
takes at minimum about 300ms on my computer:➜ angle-grinder git:(master) ✗ time cargo run --bin agrind -- '* | json | count' --file test_files/empty
Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
Running `target/debug/agrind '* | json | count' --file test_files/empty`
No data
cargo run --bin agrind -- '* | json | count' --file test_files/empty 0.22s user 0.06s system 92% cpu 0.307 total
It makes it impractical to run hundreds of integration tests with assert_cli.
I'm not positive what a better way would be -- I guess running the binary in dist directly if it exists?
The with_stdin
will take the ownership of command. Therefore you cannot write chained functions like this:
Command::new("cat")
.arg("-A")
.with_stdin("42")
.assert()
.success();
Error:
error[E0507]: cannot move out of borrowed content
--> /xxx.rs:43:5
|
43 | / Command::new("cat")
44 | | .arg("-A")
| |__________________^ cannot move out of borrowed content
I suspect when running cargo test --target TRIPLET
that main_binary
will still use the default TRIPLE
.
Downside: when the user is calling cargo test
without a --target
, assert_cmd passing --target
will cause a one-time large compile time cost
For example, I want to assert a command which only have specific stdout but does not have any stderr message. Currently, I have to write .stdout("msg").stderr("")
or stdout("").stderr("error msg")
.
I'm trying to use this for a project instead of the published version of assert_cli. I'm super impressed by what you've accomplished here in the short time!
It seems that currently, there is no easy way to check that the output of a command contains some text. Or did I miss something?
Predicates only has a str::contains
but stdout
is a Vec<u8>
. It seems that it could work with predicate::function
, but it is quite cumbersome. I'm basically looking for good way of writing .stdout(&predicate::function(|x: &Vec<u8>| String::from_utf8_lossy(x).contains("Done!")))
-- but ideally without the need to use from_utf8_lossy
. This last requirement is also why I opened this issue here, and not on the predicates, I expect we may need to talk about how to best deal with the Vec<u8>
representation.
Since its in our public interface, we are blocked on it for going to 1.0
See also assert-rs/predicates-rs#30
#57 / #4 is about a performance vs correctness issue in assert_cmd
. We should document the workaround for people who want the performance at the cost of correctness (or rather, they can ensure the correctness).
In how we document this: I noticed #6's documented workaround is in the cargo
module. The challenge is if people look at the trait or functions directly. We should find a way to address that for that workaround and this one.
As food for thoughts, here's how I'm using assert_cmd
.
The guiding idea is that I build a struct describing programs to run/delay/kill, a set of input files, and a set of post-run checks that operate on created files (including the program's stdout/stderr). I can run()
on that struct once it is set up, and it takes care of setting up the TempDir, scheduling program start, and running the checks.
/// Main struct described above. It's got builder-pattern methods,
/// and a `run` method that will do all the work and `assert` upon failure.
pub struct Exec {
cmds: Vec<Cmd>,
timeout: Duration,
files: Vec<FileIn>,
checks: Vec<FilePredicate>,
}
pub struct Cmd {
/// Program to run.
bin: Command,
/// Command name for `after()`, std/err filname prefix, and logs.
name: String,
/// Expected exit status. Some(0)=success, Some(n)=failure, None=timeout.
exit: Option<i32>,
/// Fail if the cmd exits too early.
mintime: Duration,
/// Fail if the cmd run for too long.
maxtime: Duration,
/// Current state.
state: CmdState,
/// List of signals to send to the process after startup.
signals: Vec<(CmdCond, c_int)>,
}
enum CmdState {
Wait(CmdCond),
Started(Child, Instant),
Done,
}
pub enum CmdCond {
/// Immediately true
None,
/// Duration elapsed
Delay(Duration),
/// Other Cmd exited
Cmd(String),
/// File-based predicate
Predicate(FilePredicate),
}
pub struct FilePredicate {
/// Desciption for assert-logging purpose.
desc: String,
/// Which file to operate on.
file: String,
/// Closure that tests the content of the file.
pred: Box<dyn Fn(&str) -> bool>,
}
pub enum FileIn {
FromFs(&'static str, &'static str),
Bin(&'static str, Vec<u8>),
}
With that in place, I have a pretty powerful way to write a unittest, using my crate's main
binary and a few ancillary processes:
// As basic as it gets.
exec().cmd(Cmd::any("echo", "echo", "-n a")).check("echo a", "echo.out", |s| s == "a").run();
// File from hardcoded data.
exec().inbytes("a", "a")
.cmd(Cmd::any("cat", "cat", "a"))
.check("cat a", "cat.out", |s| s == "a")
.run();
// File from file in test/ directory.
exec().infile("j", "input.basic.json")
.cmd(Cmd::any("cat", "cat", "j"))
.check("cat j", "cat.out", |s| s.starts_with("{"))
.run();
// run sequentially
let start = Instant::now();
exec().cmd(Cmd::any("s1", "sleep", "0.3"))
.cmd(Cmd::any("s2", "sleep", "0.3").after("s1"))
.cmd(Cmd::any("s3", "sleep", "0.3").after("s2"))
.cmd(Cmd::any("cat", "cat", "s1.out s2.out s3.out").after("s3"))
.run();
assert!(Instant::now() - start > Duration::from_millis(900));
// delayed start
exec().cmd(Cmd::any("0", "cat", "1.out 2.out").after(20))
.cmd(Cmd::any("1", "echo", "a"))
.cmd(Cmd::any("2", "echo", "b"))
.check("cat", "0.out", |s| s == "a\nb\n")
.run();
// timeout
exec().cmd(Cmd::any("s", "sleep", "1").maxtime(100).exit(None)).run();
exec().cmd(Cmd::any("s", "sleep", "0.05").mintime(50).maxtime(70)).run();
// Multiple signals, ordered by Instant.
exec().cmd(Cmd::any("s", "sleep", "1").signal(SIGINT, 70) // Added first but triggers last
.signal(SIGCONT, 10) // Triggers first but doesn't terminate
.signal(SIGTERM, 30) // Actual terminator
.exit(SIGTERM))
.run();
// Signal after another cmd exit.
exec().cmd(Cmd::any("s1", "sleep", "1").signal(SIGINT, "s2").maxtime(50).exit(SIGINT))
.cmd(Cmd::any("s2", "sleep", "0.01"))
.run();
// Signal after file content matches
exec().cmd(Cmd::any("s1", "sleep", "1").signal(SIGINT, ("s2.out", "4"))
.maxtime(100)
.exit(SIGINT))
.cmd(Cmd::bash("s2", "for i in $(seq 10);do echo $i;sleep 0.01;done"))
.timeout(1000)
.run();
// Main binary
exec().cmd(Cmd::main("m", "-V")).check("progname", "0.out", |s| s.starts_with("mybin"));
// Set/unset env
exec().cmd(Cmd::any("env", "env", "").env("foo", "bar")
.env("PATH", &format!("/ut:{}", env::var("PATH").unwrap()))
.env_remove("HOME"))
.check("added_foo", "env.out", |s| s.lines().any(|l| l == "foo=bar"))
.check("changed_path", "env.out", |s| s.lines().any(|l| l.starts_with("PATH=/ut:")))
.check("removed_home", "env.out", |s| !s.lines().any(|l| l.starts_with("HOME=")))
.run();
If there's enough interest, I could try to get this cleared for upstreaming. Currently there are a few extensions and API decisions that are specific to my project, and the code is a bit ad-hoc in places.
Originally posted by @vincentdephily in #74 (comment)
Right now, str
gets turned into a Utf8<Difference>
set of predicates. Should we add trimming and line ending normalization to that?
assert_cli did trimming. This would allow people to work around it by directly passing in a predicate rather than relying on IntoOutputPredicate
.
As discussed in this PR mesalock-linux/mesabox#24. We should use predicate::str::contains(str).from_utf8()
to convert Predicate<str>
to Predicate<[u8]>
for stdout()
and stderr()
comparison.
It's difficult to find answer in the documentation. There are several reasons:
assert_cmd
and predicate
's documentation are separated, you should look it back and forth.from_utf8()
is in the PredicateStrExt
trait, and it's difficult to find.For example, my intuition is that stdout()
accept a str like this: stdout("Hello, World!")
. But what if I want to check if it contains this str? Then I found that stdout()
accept a IntoOutputPredicate<P>
actually. Then I switch to read the doc of IntoOutputPredicate
and understand str
implement this trait. But still, how to check contains? I read the doc of predicate
and find out str::contains
. The doc doesn't tell me how to use it in assert_cmd
, then I'm confused. contains()
gives me a ContainsPredicate
. However, ContainsPredicate
only implements Predicate<str>
. At this point, I begin to be impatient and spent a lot of time reading the doc. Finally I find the PredicateStrExt
.
This is my first experience of using assert_cmd
and predicate
. I don't know how to improve the APIs. I think it worth to improve the docs, like giving more examples combining assert_cmd
and predicate
.
Another issue about testing CLIs is to correctly calculate code coverage. If using std::process::Command to fork and execute a CLI, all existing code coverage tools (tarpaulin, gcov, kcov, etc) will failed to calculate the statistics. There is not a better way to properly calculate the line coverage including unit tests and integration tests.
From https://github.com/rust-lang-nursery/cli-wg/issues/9#issuecomment-385487756
In getting feedback, there came up the need for timeout support.
We'd have to run the command in a thread, sleep for the time (in small increments?), and then call kill if needed.
https://doc.rust-lang.org/std/process/struct.Child.html#method.kill
It will be very useful if I can use a fixture file as input
predicates
is going to split into predicates
and predicates-core
. Only predicates-core
can show up in our public API.
This work was done for our IntoOutputPredicate
predicates but the IntoCodePredicates
were missed.
See https://github.com/assert-rs/assert_cmd/blob/master/src/assert.rs#L370
code
: take IntoCodePredicate
From<i32>
From<Predicate<i32>>
stdin
/ stdout
: take IntoOutputPredicate
From<str>
From<Predicate<str>>
From<[u8]>
Examples
Things to cover
assert
or earlier and not at the end of the expressionsuccess
is not automatically asserted.cargo test
takes a long time with because the first run of main_binary
takes a long time while holding the cargo
locking, blocking other tests
Example
running 6 tests
test blackbox_backup ... test blackbox_backup has been running for over 60 seconds
test blackbox_help ... test blackbox_help has been running for over 60 seconds
test blackbox_help ... ok
test blackbox_no_args ... ok
test clean_error_on_non_archive ... ok
test empty_archive ... ok
test incomplete_version ... ok
test blackbox_backup ... ok
From https://ci.appveyor.com/project/sourcefrog/conserve/builds/20036005
cargo test
is run, cargo
uses target/debug
for the OUT_DIR
main_binary
passes --target <TRIPLET>
to cargo
which changes the OUT_DIR
to target/<TRIPLE>/debug
cargo test
s build artifacts cannot be reused and those parts need to be rebuiltIn Issue #4 we had to choose between performance and correctness (if you do specify --target
, main_binary
should run that target's bin
) and sided on correctness
Call into escargot without passing in current_target
Call cargo test --target <TRIPLE>
Hiya, after the switch for faster bin lookup (released in 0.11), writing code to assert on examples has become quite verbose, albeit workable (trade offs!).
Code comparison (0.10 vs 0.11 with escargot
)
Previously (0.10):
use std::process::Command;
use assert_cmd::{cargo::CargoError, prelude::*};
#[test]
fn example_01_draw_text() -> Result<(), CargoError> {
Command::cargo_example("01_draw_text")?
.env("APP_DIR", env!("CARGO_MANIFEST_DIR"))
.args(&["--timeout", "0"])
.output()
.unwrap()
.assert()
.success();
Ok(())
}
Currently (0.11):
use assert_cmd::{cmd::OutputError, prelude::*};
use escargot::CargoBuild;
#[test]
fn example_01_draw_text() -> Result<(), OutputError> {
CargoBuild::new()
.example("01_draw_text")
.run()
.expect("Failed to create `cargo` command")
.command()
.env("APP_DIR", env!("CARGO_MANIFEST_DIR"))
.args(&["--timeout", "0"])
.ok()?
.assert()
.success(); // chain is pretty long now
Ok(())
}
With the --message-format
parameter, we get get cargo build
to tell us the path to the example executable:
cargo build --examples -p application_ui --message-format json
// .. 323 lines of other artifacts
{"reason":"compiler-artifact","package_id":"application_ui 0.9.0 (path+file:///D:/work/gitlab/azriel91/autexousious/crate/application_ui)","target":{"kind":["example"],"crate_types":["bin"],"name":"01_draw_text","src_path":"D:\\work\\gitlab\\azriel91\\autexousious\\crate\\application_ui\\examples/01_draw_text/main.rs","edition":"2018"},"profile":{"opt_level":"0","debuginfo":2,"debug_assertions":true,"overflow_checks":true,"test":false},"features":[],"filenames":["D:\\work\\gitlab\\azriel91\\autexousious\\target\\debug\\examples\\01_draw_text.exe"],"executable":"D:\\work\\gitlab\\azriel91\\autexousious\\target\\debug\\examples\\01_draw_text.exe","fresh":true}
The relevant bit is:
"executable":"D:\\work\\gitlab\\azriel91\\autexousious\\target\\debug\\examples\\01_draw_text.exe"
I didn't figure out if cargo-metadata
allows you to get similar information, but if we could get that information quickly (possibly within assert_cmd
), and not require users to compile cargo
, it's likely to be a nice solution.
edit:
ah, only just figured out what current_target()
and current_release()
meant (removed it from the comparison). Took a while to figure out that escargot
is a library to construct a cargo command, and not a magic swap-in replacement for cargo.
Hi. I'm currently a user of assert_cli
in tealdeer and am thinking about moving to assert_cmd
. The main issue I'm facing there is that the binary (when calling Assert::main_binary()
) is always compiled with default features, even if I compile the tests with --no-default-features
.
Does assert_cmd
have a solution for this? Ideally I'd either be able to pass the features to the main_binary
or cargo_bin
methods, or would have a way to specify the features per-invocation by other means (e.g. set_features(&["networking"])
or set_features(&[])
). This way, I could test different feature flags from a single test binary.
Alternatively compiling the main binary with the same flags as the test binary would be fine too.
Right now you have to hunt through all of the traits and it makes it hard to see the big picture.
Sometimes we want to provide testing for different features bin.
Collecting feedback or assert_fs. Please post even if you think it might not be needed for 1.0 so we can try to get the big picture of uses, challenges, etc. We're also looking at including experience reports in the 1.0 announcement blog post.
Preferably, please update to the latest version first but don't let upgrading stop you from providing feedback
To help spur feedback, consider these questions
Also, if your programs deals with the file system, please provide feedback on assert_fs. Even if you don't use it, I want to know why.
Areas to improve
stdin
handling is an unfriendly APISuccesses
fitzgen
assert_cmd is just such a pleasure to use every single time, I fall in love all over again
bravo bravo WG-cli
Running commands and dealing with output can be complex in many many ways, so assert_cmd smoothing that is excellent, very much welcome, and improves ergonomics significantly.
I have used [assert_cmd] in other projects and I am extremely pleased with it
[assert_cmd] pretty much IS my testing strategy so far, though my app under test is pretty small.
This library has made it really easy to add some test coverage to my project, even when I am just learning how to write Rust!
From @rask assert-rs/assert_cli#101
workaround: manually call binaries under test, see assert-rs/assert_cli#101 (comment)
assert_cli
version: 0.5.4
1.25.0
I have a CLI application in the works and wanted to see if assert_cli
was of any use for me. The crate works fine when running cargo test
. Neat stuff. :)
After running cargo test
I use kcov to generate code coverage reports for my code. When assert_cli
is in use kcov makes the tests fail with a signal: 11, SIGSEGV: invalid memory reference
.
I'm not sure if this is related to my newbie Rust code or if kcov and assert_cli
just don't like each other for some reason.
My code for the test:
// tests/lib_test.rs
extern crate assert_cli;
use std::env;
fn get_cwd() -> String {
env::current_dir().unwrap().to_str().unwrap().to_string()
}
#[test]
fn test_app_config_works() {
let cfg_file: &str = &format!("{}/tests/data/libtestwppr.toml", get_cwd());
assert_cli::Assert::main_binary()
.with_args(&["help"])
.succeeds()
.stdout()
.contains("list")
.unwrap();
assert_cli::Assert::main_binary()
.with_args(&["--configuration", cfg_file, "list"])
.succeeds()
.stdout()
.contains("plugin.php")
.stdout()
.contains("0.1.2")
.unwrap();
}
And the error I get with RUST_BACKTRACE=full
:
failures:
---- test_app_config_works stdout ----
thread 'test_app_config_works' panicked at 'CLI assertion failed: (command `cargo run -- --configuration /home/ojrask/Projects/ojrask/wp-plugin-repofier/tests/data/libtestwppr.toml help` expected to succeed)
status=failed
stdout=``````
stderr=```error: process didn't exit successfully: `rustc -vV` (signal: 11, SIGSEGV: invalid memory reference)
```', /home/ojrask/.cargo/registry/src/github.com-1ecc6299db9ec823/assert_cli-0.5.4/src/assert.rs:368:13
stack backtrace:
0: 0x5555556cc6f3 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h09c1ee131a74b1c4
at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
1: 0x5555556c3924 - std::sys_common::backtrace::_print::h47a337b62b6d5e9e
at libstd/sys_common/backtrace.rs:71
2: 0x5555556c92fd - std::panicking::default_hook::{{closure}}::h945a649c9017832e
at libstd/sys_common/backtrace.rs:59
at libstd/panicking.rs:380
3: 0x5555556c8f9b - std::panicking::default_hook::hcc534c2d30fbcda3
at libstd/panicking.rs:390
4: 0x5555556c9770 - std::panicking::rust_panic_with_hook::h09a7a3a353dc2f38
at libstd/panicking.rs:576
5: 0x5555556c962e - std::panicking::begin_panic::h8327f16bde15df70
at libstd/panicking.rs:537
6: 0x5555556c9529 - std::panicking::begin_panic_fmt::h42ff1d37404632d6
at libstd/panicking.rs:521
7: 0x5555555b667d - assert_cli::assert::Assert::unwrap::h0b7b03819134f9eb
at /home/ojrask/.cargo/registry/src/github.com-1ecc6299db9ec823/assert_cli-0.5.4/src/assert.rs:368
8: 0x55555557256c - lib_test::test_app_config_works::h1f9913950e84dc3c
at tests/lib_test.rs:13
9: 0x55555557f7f1 - <F as alloc::boxed::FnBox<A>>::call_box::h8106479744bcbe3a
at libtest/lib.rs:1462
at /checkout/src/libcore/ops/function.rs:223
at /checkout/src/liballoc/boxed.rs:788
10: 0x5555556e1efe - __rust_maybe_catch_panic
at libpanic_unwind/lib.rs:102
11: 0x555555577063 - std::sys_common::backtrace::__rust_begin_short_backtrace::hbe6e8f3a8f0dc6f8
at /checkout/src/libstd/panicking.rs:458
at /checkout/src/libstd/panic.rs:358
at libtest/lib.rs:1414
at /checkout/src/libstd/sys_common/backtrace.rs:136
12: 0x555555578162 - std::panicking::try::do_call::he62ddbc80b0afa86
at /checkout/src/libstd/thread/mod.rs:406
at /checkout/src/libstd/panic.rs:293
at /checkout/src/libstd/panicking.rs:479
13: 0x5555556e1efe - __rust_maybe_catch_panic
at libpanic_unwind/lib.rs:102
14: 0x55555557f582 - <F as alloc::boxed::FnBox<A>>::call_box::h4290b0f48d980187
at /checkout/src/libstd/panicking.rs:458
at /checkout/src/libstd/panic.rs:358
at /checkout/src/libstd/thread/mod.rs:405
at /checkout/src/liballoc/boxed.rs:788
15: 0x5555556d962b - std::sys::unix::thread::Thread::new::thread_start::h711c51a13a158afa
at /checkout/src/liballoc/boxed.rs:798
at libstd/sys_common/thread.rs:24
at libstd/sys/unix/thread.rs:90
16: 0x7ffff71ae7fb - start_thread
17: 0x7ffff6cc4b5e - clone
18: 0x0 - <unknown>
failures:
test_app_config_works
As I stated: the tests pass just fine when I run a plain cargo test
instead of using kcov. kcov operates by taking the built test binaries and running them through again, while looking at what is being run. Would this pose a problem for kcov or assert_cli which is used to run "external" bins that are not directly built from the test case?
When I remove the lib_test.rs
file from kcov runs the tests pass for kcov again, but nothing relating to assert_cli is reported as covered.
The application I'm working on is over at https://github.com/rask/wppr which contains the logic right now, the above tests are the only thing I've added which are not in the repo. You can see the coverage.sh
for a script which I use to run coverage tests on my code. (The in-repo coverage.sh
is missing the kcov definition for running the above lib_test.rs file, but it is the same as the wordpress_test-
pattern inclusion in there.)
Please let me know if this issue should be moved somewhere else. :)
No idea if it doesn't actually work, or if I'm doing something wrong and API is simply allowing for it, but either way it should be fixed.
assert_cmd = "0.6"
my test case:
#[test]
fn broek_dep() {
Command::new("cat")
.assert()
.set_stdin(vec![0])
.stdout("\0")
.success();
}
Which results in:
thread 'broek_dep' panicked at 'Unexpected stdout
command=`"cat"`
stdin=``````
code=0
stdout=``````
stderr=``````
',
Also on a side note, output of assert_cmd
when there's an error is rather confusing and not helpful when you're trying to deal with input/output that consists of raw, often unprintable, bytes. Debug
output without additional ``` would work a lot better IMO.
The API was thrown together using failure::Error
but our library should provide a concrete error that implements Fail
instead.
For example, I need to match:
lib_crate::sum_array (src/lib.rs:6):
push rbp
mov rbp, rsp
test rsi, rsi
je LBB13_1
lea r9, [4*rsi, -, 4]
shr r9, 2
inc r9
cmp r9, 8
jae LBB13_4
xor eax, eax
mov rcx, rdi
jmp LBB13_13
LBB13_1:
xor eax, eax
pop rbp
ret
on macos, but on windows all the LBB13_... are LBB14_.. and they change on linux as well. Also on windows and linux, the push+mov at the beginning, and pop at the end parts of my pattern are not present.
In rustc, I would write something like this
lib_crate::sum_array (src/lib.rs:6):
{{.*}}
test rsi, rsi
je LBB{{.*}}_1
lea r9, [4*rsi, -, 4]
shr r9, 2
inc r9
cmp r9, 8
jae LBB{{.*}}_4
xor eax, eax
mov rcx, rdi
jmp LBB{{.*}}_13
LBB{{.*}}_1:
xor eax, eax
{{.*}}
ret
where the {{regexp}}
is used to match any pattern.
What's the easiest way of doing something like this to assert outputs that using assert_cmd
?
I want compare command stdout with my tested app's stdout like this:
let output = Command::new("uname").output().unwrap();
let stdout = output.stdout;
new_cmd!()
.assert()
.success()
.stdout(stdout) // this will give an error
.stderr("");
I can't find a simple API from predicates. Can you suggest a proper way to handle this?
For other docs-related things, see my comment on the assert_fs issue as this crate is structured very similarly and suffers from most of the same issues.
I'm assuming referring to
Foremost, while the rustdoc is good as a reference, the upfront documentation is severely lacking. The readme is practically useless both for figuring out what this is (beyond the title, which is only helpful if one knows what all the words mean), and figuring out what it can do (and hence, whether it can be used for one's usecase). Going to the docs.rs page in some hope of more is equally discouraging (and also non-obvious from github as it's only linked through the badge), as the crate top-level documentation is an extremely succinct "overview" that doesn't really explain anything.
Based on #71
The automatic reaction is to write something like
let mut cmd = Command::new("cat");
cmd
.with_stdin()
.buffer("42");
let assert = cmd.assert();
assert
.success()
.stdout("42");
When you really need to write
let mut cmd = Command::new("cat");
cmd
.arg("-et");
cmd
.with_stdin()
.buffer("42")
.assert()
.stdout("42");
The problem is an issue with &mut
builders (we can't chain off the original cmd
), assuming with_stdin
will act like &mut
builder function when in reality you should only be acting on it at that point, etc.
Note: The lib.rs
documentation still has the broken approach
I'm running into what I think is a bug with with_stdin
, but I also could be misunderstanding the use-case for it!
I have boiled down my usage, to something where I can reproduce with just cat
let mut cmd = Command::new("cat");
cmd
.with_stdin()
.buffer("42");
let assert = cmd.assert();
assert
.success()
.stdout("42");
I expected this to function kinda like echo 42 | cat
which outputs 42
However this assert is not passing for me. Am I misunderstanding something, or is this a bug? I am using version 0.11.0
right now
Thanks for the time, and for writing and maintaining this library!
Steps to reproduce:
...Command::cargo_bin("cli-name").unwrap()
Cargo.yaml
(thus renaming the binary)The tests pass! One easy solution is to use env!(CARGO_PKG_NAME)
, but I'm more concerned about it having used a stale cache without even noticing it was doing the "wrong thing".
I can see how this happens and that this crate is not doing anything wrong, but I just wanted to point this out.
Hiya! I've appeared here since assert_cli
is deprecated, and tried swapping over from assert_cli 0.6
to assert_cmd 0.9
My really short experience (so far) has been something like this:
assert_cli
to assert_cmd
in the importsCommand
s from std
are extended by traits, cool!std
, okay it's there.stdin
? Searched and found CommandStdInExt
..with_stdin().buffer("..").unwrap()
? Why can't I call .buffer("")
twice? Oh we're no longer the builder.unwrap()
the assertion? No wait it just gives me the StdinCommand
OutputAssertExt
.Output
again? Oh it's a std
type.StdinCommand
, or was it a Result<StdinCommand>
.There's a lot of jumping around trying to figure out "what type do I have, is it a Builder
, a Result
, a normal type by extended by some trait". Also, the Result
's errors aren't all the same type (and aren't convertible), so I can't easily go main_binary()?
and then later on .output()?.assert().success()
.
I think we need an example on the first page (and perhaps README.md
) that does a complete test. The piecewise explanations are good on their own, but it's hard to tie them together. That is, literally every method call gives me a different type:
extern crate assert_cmd;
use assert_cmd::{cargo::CargoError, prelude::*};
use std::process::Command;
#[test]
fn integration_test() -> Result<(), CargoError> {
Command::main_binary()?
.env("APP_VAR", "value") // https://doc.rust-lang.org/std/process/struct.Command.html#method.env
.with_stdin() // https://docs.rs/assert_cmd/0.9.1/assert_cmd/trait.CommandStdInExt.html
.buffer("exit\n") // https://docs.rs/assert_cmd/0.9.1/assert_cmd/struct.StdInCommandBuilder.html#method.buffer
.output() // https://docs.rs/assert_cmd/0.9.1/assert_cmd/struct.StdInCommand.html#method.output
.unwrap() // Gives you back the `Output`
.assert() // https://docs.rs/assert_cmd/0.9.1/assert_cmd/assert/trait.OutputAssertExt.html#tymethod.assert
.success(); // https://docs.rs/assert_cmd/0.9.1/assert_cmd/assert/struct.Assert.html#method.success
Ok(())
}
FWIW I was trying to port this test over:
let environment =
assert_cli::Environment::inherit().insert("APP_DIR", env!("CARGO_MANIFEST_DIR"));
assert_cli::Assert::main_binary()
.with_env(&environment)
.stdin("exit\n")
.unwrap();
Off topic, CONTRIBUTING.md looks like it might have been copied from something called stager
.
It isn't always clear what you can do.
Hi! Today, I've written a bunch of tests for https://github.com/ferrous-systems/cargo-review-deps/blob/3ba1523f3b2c2cfe807bc76f42bf972d0efdc113/tests/test_cli.rs.
Initially, I've started with assert_cmd
, then I've switched to assert_cli
and now I have "write my own minimal testing harness on top of std::process::Command" on the todo list. I'd like to document my reasoning behind these decisions. Note that these are just my personal preferences though!
The first (but very minor) issue with assert_cmd
was it's prelude/extension traits based design. It makes for an API which reads well, but it is definitely a speed bump for the new users who want to understand what API surface is available to them. I think that long term such design might be an overall win, but it is a slight disadvantage when you learn the library.
The issue which fundamentally made me think "I'll stick with assert_cli for now" was the predicate-based design of assertions for stdout. It has two things which don't fit my style of testing. First, the same issue with prelude. The first example for .stdout
starts with
extern crate assert_cmd;
extern crate predicates;
use assert_cmd::prelude::*;
use std::process::Command;
use predicates::prelude::*;
That's an uncomfortable for me amount of things I need to import to do "quick & dirty smoke testing".
The second thing is extremely personal: I just hate assertion libraries with a passion :) Especially fluent assertion ones :-) I've tried them many times (b/c they are extremely popular), but every time the conclusion was that they make me less productive.
When I write tests, I usually follow the following stages.
assert!(a == b)
, without custom message. This the lowest-friction thing you can write, and making adding tests easy is super important.assert!(false)
in the console, and go to the test and write an elaborate hand-crafted error message with all kind of useful information, to help me debug the test.fn do_test(input: Input, expected: Expected)
which turns arrange
and assert
phases of a test into Data (some JSON-like pods, a builder DSL or just a string with some regex-based DSL). Internally, do_test
does all kind of fancy validation of input and custom comparisons of expected and actual outcomes.I feel that assertion libraries sit somewhere between 1 and 2 here: they are significantly more difficult to use then plain assertions, but are not as good error-message quality wise as hand-crafted messages. And they don't really help with 3, where you need some kind of domain-specific diffing. EDIT: what works great as a midpoint between 1 & 3 is pytest / better_assertions style of thing, which generates reasonable diff from the plain assert.
So, TL;DR, the reason I've switched to assert_cli
is to be able to do .stdout().contains("substring")
.
The reason why I want to write my own harness instead of assert_cli
has to do with mostly technical issues:
current_dir
nicely, so I have to write custom code anyway.stderr
, which usually contains the clue as to why stdout is wrong :)I also want to import a couple of niceties from Cargo's harness. The API I'd love to have will look like this:
// split by space to get a list of args. Not general, but covers 99% of cases
cmd("review-deps diff rand:0.6.0 rand:0.6.1")
// *rutime* arguments can be handled with the usual builder pattern
.arg("-d").arg(&tmp_dir)
// a builder for expectation, not too fluent and easily extendable (b/c this lives in the same crate)
.stdout_contains("< version = \"0.6.1\"")
// An optional debug helper. If streaming is enabled, all output from subprocess goes to the terminal as well. Super-valuable for eprintln! debugging
// .stream()
// the call that actually runs the thing.
// on error, it uncomnditionally prints process status, stdout & stderr, and only then a specific assert message.
.status(101)
// that's it, but I can imagine this returns something you can use for fine-grained checking of the output, so
// .output() # to get std::process::Output
// .text_output() # to get something isomorphic to `Output`, but with Strings instead of Vec<u8>.
It's not obvious that to get the output of a command without doing asserts on it (e.g. if I want to use another assertion tool) I have to still go through the Assert trait. I ended up writing a helper to read stdout and stderr to string via the Command handle but later saw I could have just used
get_output().stdout.to_string()
...
I'm thinking we should have a note in the assert documentation mentioning the ability to access this for completely custom logic outside of the scope of assert_cmd
The implementations are only for 'static
. Can we do better?
See https://docs.rs/assert_cmd/0.9.1/assert_cmd/assert/trait.IntoOutputPredicate.html
Hi, maybe is a my mistake, but i cant use the String type with 'stdout', only the str type.
let mut main = Command::main_binary().unwrap();
main.arg("tags")
.arg("list")
.arg("--use-csv");
let stdout = String::from("\"Name\",\"#id\"");
// stdout.push_str ....
// stdout.push_str ....
// stdout.push_str ....
main.assert()
.success()
.stdout(stdout.as_str());
On run this, i receive error[E0597]: stdout does not live long enough
. I cant use .stdout(stdout.as_str());
, because i receive the trait assert_cmd::assert::IntoOutputPredicate<_> is not implemented for std::string::String
.
Where am I going wrong :( ?
Thank you in advance for your attention.
This provides something like the macro in assert_cli, so this could be helpful for people to transition
Looking at https://docs.rs/assert_cmd/0.9.1/assert_cmd/, what should be imported to top-level and what shouldn't?
Examples
assert
, cargo
contains details that aren't as relevant for most users and shouldn't be top-levelA declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.