Can we get rid of test_dup_data_files()? about dpchecker HOT 3 CLOSED

Sounds good to me.
Maybe I misunderstood the initial request and the idea it to test for files with different names but identical content?

from dpchecker.

My reading of the requirements is that it's enough to verify that there are no duplicates in the metadata and that the files listed match what's there. I don't think that duplicate content is likely to come up without being caught by other checks - someone would have to also unintentionally create the field descriptions, etc for that file since our current checks will handle the case where a duplicate file gets accidentally copy-pasted. That said, probably wouldn't hurt to add a check at some point that looks for identical columns and row counts and throws a warning.

from dpchecker.

Fair point! I was thinking more along the lines of mistakes at upload: accidentally including 2 versions of the same file or something like that. But you're right the check for file names matching the metadata should catch that. Seems like a good enough reason to shelf this one, at least until we hear some better reasons on why it's worth pushing forward.

from dpchecker.