More unit tests for column profiling about deequ HOT 5 CLOSED

Is this still relevant? Are there any specific tests which I can start working on?

from deequ.

Dear Krzysztof,

We would be happy to have more tests for the column profiling functionality. We have initial tests here: https://github.com/awslabs/deequ/blob/master/src/test/scala/com/amazon/deequ/profiles/ColumnProfilerTest.scala

It would be great if we had some tests on small (but real) public data.

Best,
Sebastian

from deequ.

Hi Sebastian,

That's sounds interesting. I can look for a dataset like Titanic and create test cases based on them.
I guess I should add them to the repo, so the tests can be run without any internet access?
Can I added them to the /test-data folder? Or any other propositions?
Is uncompressed few MB csv file ok as a test data or it is too big?
I will of course only add datasets if it adds any value to the tests.

Thanks
Krzysztof

from deequ.

Yes, we should put the data into our repository to not require internet access. We need to make sure that the license of the data allows that though. Adding an uncompressed file with a couple of MB is totally fine.

from deequ.

Pull request #191 was merged, resolving.

from deequ.