living-with-machines / deduplify Goto Github PK

• Process won't be write-limited for large datasets
• Current implementation grows as the square of the number of files

What's the problem?

The prefix /Users/${USER} is prepended to the filepaths that are saved in the JSON file. This means if we're running the hash process with the --restart flag in a different user account, the code does not actually skip the files that have already been hashed.

What needs to happen?

• /Users/${USER} needs to be removed from the filepaths when they are saved. This can be saved to a variable in Python like so:

import os
exp_usr = os.path.expanduser("~")

• Check the code can still read/move files without the expanded user section of the filepath, i.e., the paths still properly resolve.

The following code blocks needs updating for cases where len(name_freq) > 1 && len(set(file_list)) == 1, i.e., there are multiple filepaths that are different, but by coincidence have the same length.

What's the problem?

Currently, the GitHub Action to publish the package to Test PyPI runs on every push to an open Pull Request against the main branch. This is problematic since, after the first push, a distribution exists on the Test PyPI server and subsequent pushes fail due to a naming clash.

What needs to happen?

The GitHub Action workflow needs to be tweaked so that multiple pushes to a PR does not break pushing distributions to Test PyPI. Maybe this could be manually triggered by a comment on the PR?

Could this block:

be better written as:

for k, v in counted_hashes.items(): 
    db.update({"duplicate": v > 1}, where("hash") == k) 

Is that valid Python?

Or maybe:

for k, v in counted_hashes.items(): 
    cond = v > 1
    db.update({"duplicate": cond}, where("hash") == k) 

This option would definitely work

Update Contributing Guide on how to make a version bump with incremental https://pypi.org/project/incremental/ and then make a release

Build across many platforms and Python versions using CIBuildWheel https://cibuildwheel.readthedocs.io/en/stable/setup/#github-actions

https://github.com/alan-turing-institute/Living-with-Machines-code/pull/68