[performance] attempt to cross repo mount about image HOT 14 CLOSED

@runcom I'm pretty interested in something like this but curious what kind of UX you had in mind? would the user have to supply a from

[edit: is this simply a dupe of #312?]

from image.

[edit: is this simply a dupe of #312?]

yup 😢

from image.

@runcom I guess my use case is slightly different - I'd like a client to avoid copying blobs from a non-docker-registry source to a docker-registry destination, if it knows they're already there in a different namespace... thoughts? (so far I'm using skopeo for all registry communication and it'd be nice to keep it that way instead of manually hitting the API, but I accept it might not map sensibly to the CLI)

from image.

I'd like a client to avoid copying blobs from a non-docker-registry source to a docker-registry destination, if it knows they're already there in a different namespace

the issue here is what the registry can say about a particular blob (e.g. if it's on the registry basically). I'm not sure the docker registry API exposes this info. Otherwise, as said in the other issue there should be some cache about already pushed layers from a client but that won't, of course, know about all blobs. I'm not sure, @mtrmac do you have other ideas here?

from image.

the issue here is what the registry can say about a particular blob (e.g. if it's on the registry basically). I'm not sure the docker registry API exposes this info.

(We do ask by a HEAD /v2/$repo/blobs/$digest; a “sufficiently smart” (i.e. probably backed by a stateful database with some locking and indexes, instead of the current IIRC ~stateless implementation) registry could already maintain an index of $digest independent from $repo and tell us that the blob is available; but there is no “mount by $digest without knowing $repo” API.)

Otherwise, as said in the other issue there should be some cache about already pushed layers from a client but that won't, of course, know about all blobs.

Yeah. There’s one awkward aspect about that: containers/image is a library which can be used by any user with any privilege, and it’s not completely obvious where the cache should live and how should concurrency be managed. But, this being a cache, we can probably use ~/.cache or something like that—in the worst case we would do a full copy and things would work. As for concurrency, I’d be tempted to farm that out to SQLite, except that there don’t seem to be bindings in the standard library 😞

It seems quite possible overall, just a bit of work.

from image.

surfacing this issue, after realizing this morning that an operation like skopeo copy ... was re-pushing every layer. 🌊

from image.

Tackling this now

from image.

@vbatts you in the ocean?

from image.

@vbatts Was it actually copying them, or “only” spending a few roundtrips to notice that they already exist?

from image.

@vbatts Was it actually copying them, or “only” spending a few roundtrips to notice that they already exist?

Never mind, the existence check does not work cross-repo. Sorry.

from image.

@jonboulle
🎱 Reply hazy try again
🎱 As I see it, yes

from image.

@mtrmac you arrived at the answer, but yes, it does a full push for every layer regardless

from image.

@runcom @vbatts @mtrmac @vrothberg I think we have fixes for this, can we close this issue?

from image.

#536 can take advantage of most (but not quite all) opportunities for mounts automatically.

from image.