Set up CI about differential-datalog HOT 23 CLOSED

tutorial, path, ovn, and ovn_ftl is a reasonable subset.

from differential-datalog.

one more detail before I head out - I ran LD_LIBRARY_PATH=test/datalog_tests/tutorial/target/release/ test/datalog_tests/tutorial/tutorial_test with the absolute path after stack test on travis; it had the same output as what I get locally, but still returned an exit code 2 which is surprising

from differential-datalog.

That would of course be fantastic. There are two problems I am aware of. First, I don't think Travis supports stack-based Haskell projects out of the box. Second, I think Travis test runs are limited to 20 or 30 minutes, and unfortunately DDlog tests run longer.

Both issues can be solved. There exist examples of setting up Travis for stack-based projects, and we can definitely construct a light version of the test suite that finishes in under 20 minutes.

If this is something you are willing to work on, it would be incredibly useful. And I can definitely help with speeding up tests.

from differential-datalog.

I'm happy to give it a try!

from differential-datalog.

Wonderful, thank you! Do you want to try it in your fork of the project or shall I add you as collaborator to differential-datalog?

from differential-datalog.

What would be the difference?

from differential-datalog.

Actually I should try it on my fork first, since the setup might be quite involving. Once I get it working I can replicate the setup here.

from differential-datalog.

Ok, sounds good. Let me know how I can help. For the moment a "light" test suite could be just running one of the tests:

stack test --ta '-p tutorial'

from differential-datalog.

Is there a good subset of tests to run? E.g. all parser tests & some compiler tests & ovsdb tests. Otherwise I can pick a few arbitrary compiler tests.

from differential-datalog.

The MacOS test passed on Travis, but for linux I'm getting tutorial_test failed with exit code ExitFailure 2. Could you check if the test stack test --ta '-p tutorial' runs on linux?

from differential-datalog.

Have you merged the latest master? Your macos fix broke the test script on Linux.

from differential-datalog.

I did merge it, and tried to revert to the commit before my fix which didn't work either. Probably something wrong with the travis end. Any idea what ExitFailure 2 might mean?

from differential-datalog.

Hmm, I've seen it before. I think it's "file not found" returned by one of the commands invoked from the test script. Do you have the full output? Is cargo installed in the test container?

If the error happens after the Rust project was created, you can try running cargo directly and observing its output:

cd test/datalog_tests/tutorial
cargo build --release
target/release/tutorial_cli  < ../tutorial.dat

from differential-datalog.

Oh wait, the error mentions tutorial_test. Then can you please try:

LD_LIBRARY_PATH=test/datalog_tests/tutorial/target/release/  test/datalog_tests/tutorial/tutorial_test

but using absolute paths instead of relative.

from differential-datalog.

I am checking out for today. I was not able to reproduce the bug so far. One way to proceed might be to migrate your travis setup to the main DDlog repo so I can take it from there. Whatever is happening on Linux, tests should not fail without a meaningful error message, so I should at least make sure that one is generated.

from differential-datalog.

Thanks, made a PR that has the .travis files

from differential-datalog.

I just ran the tutorial test on a fresh ubuntu vm and it passed, so must be something wrong with travis setup.

from differential-datalog.

There used to be a nasty issue in that particular part of the test script related to lazy evaluation that could potentially manifest in a non-deterministic way. It's possible that I did not fix it properly. Still working on something else, but will get to this shortly...

from differential-datalog.

Ah that might be it, I'll poke around a bit. Laziness usually messes up sys calls

from differential-datalog.

A-ha, this helped me nail it. There was a genuine bug in the Rust code that generates C code for testing the FFI interface to DDlog. It basically did not call the function to properly shut down differential dataflow processing. Since both differential and DDlog use multiple threads, it is not too surprising that the bug manifested non-deterministically.

I am going to create a new pull request with my changes that I made in a branch and merge that into master.

Continuous integration, yay!

from differential-datalog.

I am keeping the issue open, as I still have to speedup the test suite, so we can run it on travis without timing out.

from differential-datalog.

That makes sense, nice! There might be some caching I can do on travis that may speed up installation etc. And the Travis time limit is 50 min for each job where some thing should be printed every 10 min. But it's probably still a good idea to speed up the tests.

from differential-datalog.

As of #102 the complete test suite runs in Travis

from differential-datalog.