Running with the default command on larger files (over 1GB) leads to inconsistency across multiple runs
$ duplicut '/media/sf_kalishare/Wordlists/hashmob.net_2023-05-14.medium.found' -o '/media/sf_kalishare/Wordlists/hashmob.net_2023-05-14.medium.DUPLICUT'
duplicut successfully removed 0 duplicates and 42 filtered lines in 05 seconds
$ duplicut '/media/sf_kalishare/Wordlists/hashmob.net_2023-05-14.medium.found' -o '/media/sf_kalishare/Wordlists/hashmob.net_2023-05-14.medium.DUPLICUT'
duplicut successfully removed 384 duplicates and 0 filtered lines in 02 seconds
$ duplicut '/media/sf_kalishare/Wordlists/hashmob.net_2023-05-14.medium.found' -o '/media/sf_kalishare/Wordlists/hashmob.net_2023-05-14.medium.DUPLICUT'
duplicut successfully removed 0 duplicates and 384 filtered lines in 02 seconds
$ duplicut '/media/sf_kalishare/Wordlists/hashmob.net_2023-05-14.medium.found' -o '/media/sf_kalishare/Wordlists/hashmob.net_2023-05-14.medium.DUPLICUT'
duplicut successfully removed 0 duplicates and 385 filtered lines in 02 seconds
$ duplicut '/media/sf_kalishare/Wordlists/hashmob.net_2023-05-14.medium.found' -o '/media/sf_kalishare/Wordlists/hashmob.net_2023-05-14.medium.DUPLICUT'
duplicut successfully removed 221 duplicates and 385 filtered lines in 02 seconds
$ duplicut '/media/sf_kalishare/Wordlists/hashmob.net_2023-05-14.medium.found' -o '/media/sf_kalishare/Wordlists/hashmob.net_2023-05-14.medium.DUPLICUT'
duplicut successfully removed 378 duplicates and 385 filtered lines in 02 seconds
Any idea why this is occurring? i was expecting the same results over and over again,
On further testing it seems the cleanup stats go wild when writing to an already existing file, resulting in various inconsistent file size and words.