gc: keep last `n` versions of data files, while ignoring commits with only code changes about dvc HOT 5 CLOSED

How would an option to keep the last n versions be an improvement in that automated scenario?

If we want to guarantee that we always have the last n versions of datafiles, no matter how old they are, then relying on date does not work.

Example: if the job runs once a day, and we would like to always have the last two versions, then using --date 2_days_ago might fail. Considering this history:

day | status
--------------------------------
  4 | successful, data files updated in commit a02
  3 | job failed, no commit/updates
  2 | job failed, no commit/updates
  1 | successful, data files updated in commit a01

in day 4, running the dvc gc --date 2 would only keep the last version of data files (day 4) and remove the version of day 1. @dberenbaum

from dvc.

This is especially important if we would like to do this in an automated script on a regular interval, say every week (and hence we don't know about the history of commits to tune the command arguments).

If you want to do this on a regular interval, could you use dvc gc --date?

from dvc.

If you want to do this on a regular interval, could you use dvc gc --date?

That's a good idea; however, suppose this is a regular update job which changes the data files and latest run fails (for whatever reason). Then, using --date option, in the next run of this job probably only the last version of data files are kept and the older ones will be removed. @dberenbaum

from dvc.

How would an option to keep the last n versions be an improvement in that automated scenario?

from dvc.

Thanks for the explanation! We don't have any mechanism now in DVC to look for which commits make DVC-tracked changes, so we would need to implement that first. I think it also raises a lot of questions. Do you want the last two commits, or the last two versions of every DVC-tracked file? Do changes to dependencies (like the code files used to generate DVC outputs) count, or should it only be DVC-tracked outputs?

Have you tried to implement something similar yourself, like using git log -- '*.dvc' to find commits where .dvc files changed and passing those revisions to dvc gc?

from dvc.