If a dtool copy command fails because the connection is broken one can end up with a b

Fixed defect in dtool-http <a class="commit-link" data-hovercard-type="commit" data-ho

Fixed defect in dtool-s3 <a class="commit-link" data-hovercard-type="commit" data-hove

Fixed defect in dtool-azure <a class="commit-link" data-hovercard-type="commit" data-h

Fixed defect in dtool-irods <a class="commit-link" data-hovercard-type="commit" data-h

Fixed in: dtool 3.6.1 dtool-http 0.2.0 dtool

Ensure corrupted files do not end up in the dtool cache about dtool-create HOT 6 CLOSED

jic-dtool commented on August 12, 2024

Ensure corrupted files do not end up in the dtool cache

from dtool-create.

Comments (6)

tjelvar-olsson commented on August 12, 2024

The fix actually needs to be implemented in the s3, azure, http and irods storage broker backends. More specifically the methods:

dtool_s3.storagebroker.S3StorageBroker.get_item_abspath()
dtool_azure.storagebroker.AzureStorageBroker.get_item_abspath()
dtool_http.storagebroker.HTTPStorageBroker.get_item_abspath()
dtool_irods.storagebroker.IrodsStorageBroker.get_item_abspath()

For example the s3 implementation needs to change from:

        local_item_abspath = os.path.join(
            dataset_cache_abspath,
            identifier + ext
        )
        if not os.path.isfile(local_item_abspath):

            self.s3resource.Bucket(self.bucket).download_file(
                bucket_fpath,
                local_item_abspath
            )

        return local_item_abspath

To:

        local_item_abspath = os.path.join(
            dataset_cache_abspath,
            identifier + ext
        )
        if not os.path.isfile(local_item_abspath):

            tmp_local_item_abspath = local_item_abspath + ".tmp"
            self.s3resource.Bucket(self.bucket).download_file(
                bucket_fpath,
                tmp_local_item_abspath
            )
            os.rename(tmp_local_item_abspath, local_item_abspath)

        return local_item_abspath

from dtool-create.

tjelvar-olsson commented on August 12, 2024

Fixed defect in dtool-http jic-dtool/dtool-http@69ccb02

from dtool-create.

tjelvar-olsson commented on August 12, 2024

Fixed defect in dtool-s3 jic-dtool/dtool-s3@2942364

from dtool-create.

tjelvar-olsson commented on August 12, 2024

Fixed defect in dtool-azure jic-dtool/dtool-azure@3cc5d60

from dtool-create.

tjelvar-olsson commented on August 12, 2024

Fixed defect in dtool-irods jic-dtool/dtool-irods@3213a71