Comments (7)
Do you have any pointers for continuous integration? What would be the best way to force Conan to install the very latest commit on the main/master branch of (one of) my dependencies?
When using a package manager, the natural flow is enabling CI to build a new package version (or revision) for the commit or tag you want to make available for others to use. For first-party code, the recommendation would be to put the conanfile.py
in the source repo of the code that it is packaging, for several reasons:
- It is very likely that such code will have dependencies. Then the natural dev flow becomes
git clone myrepo && cd myrepo
+conan install .
+cmake ....
(or the build system you are using). - When changes are pushed, then CI can do a
conan create .
on that code. If it is not merged, it can build a package and put that binary in a secondary server "build" binary repo, not yet available for all builds. When the code is merged, the binary might be copied to the main server binary repo, making it available to other developers and CI jobs. This flow is called a "package promotion" and is kind of best practice in package management and devops
The issue is that we haven't managed yet to write comprehensive CI docs, the plan is it that it will be added to https://docs.conan.io/2/devops.html section, but it is difficult to get the time. Conan 2 has some important features to be able to support CI at scale, like the conan graph build-order
command or the lockfiles
to be able to distribute builds for multiple binary configurations, without needing to replicate the dependency graph topology on the CI system. I know it is not as nice as docs, but the tests in https://github.com/conan-io/conan/blob/develop2/test/integration/lockfile/test_ci.py should be understandable, and would be a very good reference of the main CI ideas. Note that these CI tests are not for the specifics of building a specific package when it gets some code changes, but for the most challenging CI process that is "now that I have a new package version/revision, what other packages need to be rebuilt in which order to validate that my main products are not broken by this new package".
The plan is to add this documentation in the following months.
from conan.
Hi @tttapa
Thanks for your question.
To use a specific commit of the original repo, it is necessary to include or add that commit in the recipe that creates the package.
What ConanCenter usually does when a repo doesn't publish releases, is adding a new cci.20240519
version based on the date, so it can get an updated package with latest changes, built on a specific commit. Such commit will be defined in the conandata.yml
of the package recipe.
If you want to customize ConanCenter recipes to include later releases or other customizations that do not exist yet in ConanCenter (or might never be possible to be merged upstream), then you can easily have your own package, with the version you want, without conflicting with the existing ConanCenter ones. This is done by doing a fork of the conan-center-index
Gtihub repo that contains all the recipes for ConanCenter, and then:
- Go to
recipes/eigen/all
and add there the version you want inconandata.yml
- Then either do a
conan create . --version=myversion
or use the new "local-recipes-index" feature (see https://blog.conan.io/2024/04/23/Introducing-local-recipes-index-remote.html) to automatically be able to use the new version.
from conan.
I see, thanks for the quick response, the local recipes index feature looks promising!
If I understand correctly, this approach would require me to use the latest public Conan Center version in my project's conanfile, correct? I cannot set my custom version as the default requirement, otherwise users of my project will not be able to do the standard conan install . --build=missing
without first conan create
ing or conan export
ing the recipe from my custom conan-center-index fork, or without adding it as a remote.
Do you know of any plans to add direct Git support to Conan? E.g. by directly specifying a URL to the custom recipe, such as requires = "eigen/custom.20240519 @ git+https://github.com/my/conan-center-index-fork/.../eigen/conanfile.py"
(where Conan would automatically download and export the custom recipe specified, using the custom version)? Has this been discussed elsewhere perhaps?
from conan.
If I understand correctly, this approach would require me to use the latest public Conan Center version in my project's conanfile, correct? I cannot set my custom version as the default requirement, otherwise users of my project will not be able to do the standard conan install . --build=missing without first conan createing or conan exporting the recipe from my custom conan-center-index fork, or without adding it as a remote.
No, you can use any version of ConanCenter, including older ones. The conan-center-index
contains all versions information, and it can resolve to any of them, in the same way when using "local-recipes-index" and when using the actual ConanCenter. The only difference will be the existence of binaries. Note that if the fork of conan-center-index
is aligned with the upstream, it might be even possible to use recipe from the local fork and a binary from the ConanCenter (for this, the recipe revisions must be identical, that is, the forked recipe must not have local changes)
Do you know of any plans to add direct Git support to Conan? E.g. by directly specifying a URL to the custom recipe, such as requires = "eigen/custom.20240519 @ git+https://github.com/my/conan-center-index-fork/.../eigen/conanfile.py" (where Conan would automatically download and export the custom recipe specified, using the custom version)? Has this been discussed elsewhere perhaps?
Yes, this has been discussed many times, and it is not planned. There are many limitations, complexities and problems associated with this approach. Conan follows the model of Maven for Java, with a strong abstraction over the dependencies references and using dedicated package servers, not coupling them with hardcoded source locations in recipes. The "local-recipes-index" feature was introduced to allow modifications to recipes in a convenient way, but for the most part is intended as a package creation flow, not as a package consumption flow, that is, packages should be created from the fork and stored in some package server.
from conan.
No, you can use any version of ConanCenter, including older ones.
Thanks, I meant that putting a Conan Center version (e.g. the latest, but could be any version) would be the only way to keep my project easily installable by users. IIUC, there are two options:
- Specify a public Conan Center version in my project's conanfile, e.g.
requires = "eigen/3.4.0"
.
In this case, a user can use my project after a simplegit clone project;
cd project;
conan install . --build=missing
, but they will end up with a build where some features are missing or broken, because they're missing some bug fixes to Eigen. - Specify a version from my own Conan Center fork in my project's conanfile, e.g.
requires = "eigen/custom.20240519"
.
Now I can be sure that users have a working version of Eigen, but unfortunately, this significantly raises the barrier to entry for users, making it harder for them to install and use my project. Because unless they first go and download my Conan Center fork and add it as a local remote on their system, they will not be able to runconan install . --build=missing
successfully out of the box.
Is there any way to make the installation process simpler for users of my project (i.e. no need to manually download and set up other repositories first), while also selecting a custom version by default so they don't end up with a subtly broken build?
from conan.
Thanks, I meant that putting a Conan Center version (e.g. the latest, but could be any version) would be the only way to keep my project easily installable by users. IIUC, there are two options:
It depends on who your users are. There are wildly different use cases, if those are other colleague developers in the same organization or company, or if you are planning to release something as open source library.
Specify a public Conan Center version in my project's conanfile, e.g. requires = "eigen/3.4.0".
In this case, a user can use my project after a simple git clone project; cd project; conan install . --build=missing, but they will end up with a build where some features are missing or broken, because they're missing some bug fixes to Eigen.
Yes, and they can also have other bugs when using a non-released version of eigen, because the maintainers didn't put the typical pre-release effort to stabilize and do more comprehensive tests. This is not an issue of the package manage per-se, but of the library and its maintainers. My experience suggests that if Eigen had very critical updates that haven't been released and the users wanted to use, we would already have an updated eigen/cci.date
. This is something that can be discussed, we are not closed to have these versions in ConanCenter, if it makes sense and it is good for the wider community, it might be consider to create a cci.date
release, that is quite simple.
Specify a version from my own Conan Center fork in my project's conanfile, e.g. requires = "eigen/custom.20240519".
Now I can be sure that users have a working version of Eigen, but unfortunately, this significantly raises the barrier to entry for users, making it harder for them to install and use my project. Because unless they first go and download my Conan Center fork and add it as a local remote on their system, they will not be able to run conan install . --build=missing successfully out of the box.
That is the thing that depend on who your users are. For organizations, this is not an issue, because they basically get the configuration from a conan config install
command, getting all the organization custom profiles, settings, remotes definitions, plugins and many more, and packages like eigen/custom.20240519
would already be in your own package server, and in cases it is a library in pre-compiled format for the different binary configurations, saving lots of time of CI and developers.
Now I can be sure that users have a working version of Eigen, but unfortunately, this significantly raises the barrier to entry for users, making it harder for them to install and use my project. Because unless they first go and download my Conan Center fork and add it as a local remote on their system
And if your users are open-source users, is it really necessary that they use the very latest eigen from source? What actual crashes or issues happen if they don't? Maybe it is worth to consider a cci.date
version in ConanCenter?
In any case, "significantly raises the barrier" means:
$ git clone <url-to-myrepo>
$ conan remote add local myrepo
If it is that important to use customized versions that cannot exist in ConanCenter, then those are relatively straightforward setup instructions, specially for open-source users that are typically faced with much more complicated project setups.
from conan.
Thanks, I appreciate the advice.
One more question on this topic:
Do you have any pointers for continuous integration? What would be the best way to force Conan to install the very latest commit on the main/master
branch of (one of) my dependencies?
Would your recommendation be different for dependencies that I maintain myself (i.e. where I could do git clone my-dep;
conan export ./my-dep
), versus third-party dependencies that come from the CCI?
from conan.
Related Issues (20)
- [question] Please help how to use NuGet Libraries with Conan and CMake for Builds. HOT 1
- Error uploading file: conanmanifest.txt, '404 Client Error: Not Found for url: https://artifactory.com/artifactory/api/conan/conan-test/v2/conans/xxx/revisions/b9c4826ad9445ea5257f6432b5a019dd/files/conanmanifest.txt[bug] HOT 3
- [question] Confusion around python_requires using minor_mode HOT 6
- [bug] incorrect paths generated by pkgconfigdeps.py when using --deployer option HOT 1
- [question] Filtering options for package to get path to local cache HOT 4
- [bug] `conan list` filtering option asks users to use scope patterns when those don't work in that context HOT 2
- [question] Set cmake_find_mode for all deps in CMakeDeps [conan 2] HOT 2
- [feature] Option to suppress generation of script files HOT 2
- [bug] Initial setup for Conan test suite is not automated or documented HOT 3
- [question] disable package revisions HOT 3
- [bug] `conan source` does not actually get `global.conf`'s `core.` confs HOT 1
- [question] Install a binary package without matching the compiler version HOT 3
- [question] preset name != profile name HOT 4
- [question] Include jinja profile template into another HOT 7
- [question] How to disable compatilibity? HOT 5
- [question] CMAKE_INSTALL_PREFIX in toolchain - local/cache build difference [conan 2] HOT 3
- [feature] Introduce os.version or os=Ubuntu in profiles HOT 2
- [question] PUBLIC compile definition not propagated HOT 3
- [question] How to access a value defined in the `[conf]` section of a profile? HOT 1
- [feature] Add properties to MakeDeps generator HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from conan.