[question] Uploading packages with non-standard options to a private registry about conan HOT 3 CLOSED

Happy to help! 🙂

from conan.

Hi @kevin-mcvey

Thanks for your question!

However, my organization has our own Conan registry. I'm wondering if that can save me some time and money. What I'm interested in knowing is whether I can take the versions of my requirements that Conan is building (e.g. Arrow above) and upload them to that private registry, thus removing the need for these packages to be built from source, provided the options stay the same.

Of course, this is totally recommended, this is one of the reasons to use Conan and a private server to host the packages.
The only necessary thing is to do a conan upload, there can be some different strategies for this:

• Do a conan upload *, which will upload everything that you have in the cache. Note that it makes sense to do a copy of packages from ConanCenter to your own private server, and even disconnect ConanCenter and use only your private server. The upload can be quite efficient, Conan will use the revisions to check if something is already in the server and completely avoid the transfer of files.
• Conan can even provide way more control. From your conan install or a conan create command you can capture the dependency graph in a json file with --format=json > graph.json. Then capture a "package list" with conan list --graph=graph.json ... --format=json > pkglist.json, and use conan upload --list=pkglist.json to upload specifically those packages

For the second one, I think this section from the docs can help: https://docs.conan.io/2/examples/commands/pkglists.html#building-and-uploading-packages

Please let me know if this is what you were looking for.

I would appreciate any tips or pointers to documentation y'all are able to provide. Thank you for all your work on Conan, it's a fabulous tool. :)

Very happy to hear that, thanks for the feedback!

Some extra hints:

• In general, building all the package binaries you need from recipes and uploading them to your own server is a recommended practice, it gives you full control, security and reproducibility over your dependencies, and avoid possible issues with ConanCenter changes.
• The build_policy = 'missing' is only recommended for some exceptional cases, the default behavior of missing if a binary doesn't exist prevents developers and CI to be unnecessarily be re-building all the time from source just because the binaries doesn't exist. It is fine for CI or developers to explicitly pass --build=missing, but that is an option, if the recipe is declaring it, it might be hiding pipeline errors.
• Regarding the definition of options values of dependencies in recipes, note that this might not be the best approach in all cases, and defining profiles in profile files could be sometimes better. You can also use the more compact default_options = {"arrow/*:option": "value1", ...}

from conan.

Wow! @memsharded this is super helpful, the detail you provided is above and beyond. This worked for me right out of the box. Thank you very much!

from conan.