`dvc add *` for 21gb of files that are already checked in to dvc takes 7 minutes about dvc HOT 6 CLOSED

We are aware of this and are working to fix it. Multiple issues are open regarding this, so I will close this in favor of those.

from dvc.

This seems to be a long standing issue. Is there an ETA for a fix or improvement to the dvc add performance?

Unfortunately, we don't have any ETA to share at the moment.

from dvc.

Duplicate of #8008, #7607, and #3177.

from dvc.

I have a similar issue with a datasets of ~50GB which takes hours whenever I add few files. Hope this will be fixed soon

from dvc.

@RadouaneK, if you are only adding a few files, you can instead pass the filenames of the dataset that you are modifying.

Eg:

data
└── file1

Instead of doing dvc add data, you can dvc add data/file1. Only those files will be updated. Similarly, you can pass a subdir if you are modifying a subdirectory of the dataset (eg: dvc add data/dir/subdir_or_file).

See https://dvc.org/doc/user-guide/data-management/modifying-large-datasets.

from dvc.

This seems to be a long standing issue. Is there an ETA for a fix or improvement to the dvc add performance?

from dvc.