Accept external test suite files to generate test cases about test-case HOT 9 OPEN

#78 seems related.

There are rough plans into turning test_case into full test suite. There are also few unresolved design decisions. test_case should stay relatively simple (to ones writing/reading test code). I'd also love to keep in mind how the design will be affected by custom_test_frameworks once they hit stable - it may be an overshoot but I prefer to avoid users having to do huge rewrites of their code.

Issues like this definitely help clarify requirements.

Some thoughts:

• we can try achieve multi-format parsing if we include serde
• name test_scenario("file") or test_scenarios looks clean
• there may be problems trying to couple TestInput if the type is not specified within attribute
• there are plans to include context within test_case - this may go in tandem

Definitely need to think about this more.

from test-case.

Before I saw this ticket, I began work on a separate crate to provide this specific feature: test_vectors

It is not yet in working condition, but it may be a good proving ground for a feature in test-case while keeping the two interfaces independent. If test_vectors got to the point of being reliably usable, then it may be a candidate to merge its functionality into test-case.

from test-case.

Hmmm....

We are talking here about loading and parsing a CSV file in proc macro. Which means it would be similar to comptime (In Rust: const fn.

That would definitely make the compilation time longer.

Also, I would not implement just CSV processing. The whole feature would have to be generic. I do not want to support N different file formats.

@luke-biel what do you think?

from test-case.

I agree that more file formats would be ideal.

About compilation time, that is an issue but could be document and the feature be kept behind a feature flag.

from test-case.

there may be problems trying to couple TestInput if the type is not specified within attribute

What problems do you foresee?

from test-case.

The procedural macro has no access to the TestInput structure. It would only have an identifier.

from test-case.

there may be problems trying to couple TestInput if the type is not specified within attribute

What problems do you foresee?

I see this as a design limitation. First as @frondeus mentioned, this requires "Deserialize" trait implemented on TestInput - I'm not sure if we want to use serde or homebrew. Second, if I'd like to add some context to test (eg. it's name, path to file it's generated from) I would either add it as separate parameter or use Context<TestInput> and this approach should scale to normal test cases.

Overall it's not a big deal, I put it there to remember later.

from test-case.

I'd be happy to take a look at test_vectors in the future and if possible merge that into test case :D I appreciate the interest.
I see that you rely on TryFrom<&[u8]> in your impl. It's nicer than dependency on serde I initially envisioned :)

from test-case.

I haven't revived that project, and I do recall one or more blockers. One I saw you commenting on in rust-internals about build caching behavior for including files (if a non .rs file changes, build doesn't notice that and won't rebuild).

So I can't necessarily recommend that implementation. Maybe the API. ;-)

I was starting with TryFrom<&[u8> to get an "MVP" functionality working, but I personally think I may end up wanting serde for many cases.

I should note, one motivation for raw &[u8] input is that I envisioned generating tests from external test files which come from external sources especially for standardized test vectors, say where a (non-rust-specific) protocol publishes a set of files that should be parsable and another set that are parse errors. So the use case is "can we auto-generate tests for a rust implementation against an external specification", perhaps using git submodules or subtrees or build.rs hacks.

from test-case.