Comments (3)
Hi @midhun1998
Thanks for your question.
We'd need to know more about the conanfile.py
, can you please share it? Most likely it contains some exports =
or exports_sources = ...
attributes that are copying all the local files into the recipe, or your package()
method contains too broad patterns.
Note that the local flow (source + build + export-pkg) is more a development flow, but not that intended for CI. The reproducibility will be lower and the possibility of doing more efficient CI is lost too.
Also, the recommended approach for having recipes that can build for authorized users and avoid code access for non-authorized users is https://docs.conan.io/en/1.64/reference/conanfile/tools/scm/git.html#example-implementing-the-scm-feature. Using the "scm" approach, the recipe captures the "url+commit" of the source, so authorized users can actually build new binaries from source when they want, while not authorized ones will only be able to use binaries, without code access.
from conan.
Thanks for quick reply, @memsharded !
We'd need to know more about the conanfile.py, can you please share it? Most likely it contains some
exports =
orexports_sources = ...
attributes that are copying all the local files into the recipe, or your package() method contains too broad patterns.
I double-checked and I don't see export*
attributes. I have added recipe for reference.
Note that the local flow (source + build + export-pkg) is more a development flow, but not that intended for CI. The reproducibility will be lower and the possibility of doing more efficient CI is lost too.
So, Does this mean the right flow is to run conan create
? If not, Could you please guide me?
One of the many reasons why we chose the source, install, build, etc process is to run custom regression such as smoke test and full regression after conan build
and conan export-pkg
. Let me know if we can still achieve this with conan create
?
Also, the recommended approach for having recipes that can build for authorized users and avoid code access for non-authorized users is https://docs.conan.io/en/1.64/reference/conanfile/tools/scm/git.html#example-implementing-the-scm-feature. Using the "scm" approach, the recipe captures the "url+commit" of the source, so authorized users can actually build new binaries from source when they want, while not authorized ones will only be able to use binaries, without code access.
Thanks for pointing me in the right direction! 🙂
All our packages have sources and recipes in the same repository. Should we still stick the above approach?
Here is a redacted version of conanfile.py
. I have removed specifics of product and commands other than that its the same.
class Xyz(ConanFile):
name = "xyz"
build_dir = "src"
def set_version(self):
if get_env("CONAN_PACKAGE_VERSION") is not None:
self.version = get_env("CONAN_PACKAGE_VERSION")
else:
self.version = "2.2.1"
def build_requirements(self):
self.build_requires("curl/202203.0.0@ci/prod")
...
# ---- More build_requires statements here ----
def build(self):
self.run("CUSTOM BUILD COMMAND")
def package(self):
# ---- Many self.copy instances which are copying specific files. Generic ones are below ---- #
for file in [
"src/common/a.h",
"src/b.h",
...
]:
self.copy(
os.path.basename(file),
dst=f"include/{self.name}",
src=os.path.dirname(file),
keep_path=False,
)
self.copy("README.md", dst=".", src="src/", keep_path=False)
self.copy(
"*.a",
dst=f"lib/{self.settings.target_arch}",
src="src/object_root/lib-" + str(self.settings.target_arch),
keep_path=False,
)
self.copy(
"*.so",
dst=f"lib/{self.settings.target_arch}",
src="src/object_root/pscb/lib-" + str(self.settings.target_arch),
keep_path=False,
)
self.copy(
"*",
dst=f"bin/{self.settings.target_arch}",
src="src/lib/module-x/install/" + str(self.settings.target_arch),
keep_path=True,
)
self.copy(
"product.bom",
dst=f"bin/bom",
src="src/lib/module.y",
keep_path=False,
)
def package_id(self):
del self.info.settings.compiler.libcxx
del self.info.settings.compiler.version
del self.info.settings.compiler.cppstd
def package_info(self):
self.cpp_info.libs += [
file.replace(
f"{self.package_folder}/lib/{self.settings.target_arch}/lib", ""
)
.replace(".a", "")
.replace(".lib", "")
for file in glob.glob(
f"{self.package_folder}/lib/{self.settings.target_arch}/lib*.a"
)
]
self.cpp_info.libs += [
file.replace(
f"{self.package_folder}/lib/{self.settings.target_arch}/lib", ""
)
.replace(".a", "")
.replace(".lib", "")
for file in glob.glob(
f"{self.package_folder}/lib/{self.settings.target_arch}/lib*.lib"
)
]
self.cpp_info.libdirs = [f"lib/{self.settings.target_arch}"]
self.cpp_info.libs = [lib for lib in self.cpp_info.libs]
To give more clarity, Here is the log of conan export-pkg
:
$ conan export-pkg --force -bf _build -sf _build -if _build . xyz/2.2.1@midhun/dev
[HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'url'. It is recommended to add it as attribute
Exporting package recipe
[HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'license'. It is recommended to add it as attribute
[HOOK - attribute_checker.py] pre_export(): WARN: Conanfile doesn't have 'description'. It is recommended to add it as attribute
xyz/2.2.1@midhunn/dev: Calling export_sources()
xyz/2.2.1@midhunn/dev export_sources() method: Copied 13 '.depends' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 11 '.hier' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 120 '.make' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 45 '.c' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 441 '.h' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 39 files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 8 '.js' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 2 '.json' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 2 '.html' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 1 '.yml' file
xyz/2.2.1@midhunn/dev export_sources() method: Copied 1 '.md' file
xyz/2.2.1@midhunn/dev export_sources() method: Copied 28 '.a' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 47 '.so' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 1 '.zip' file
xyz/2.2.1@midhunn/dev export_sources() method: Copied 84 '.1' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 3 '.txt' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 12 '.map' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 21 '.py' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 137 '.cc' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 18 '.png' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 23 '.test' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 5 '.sh' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 197 '.script' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 6 '.cpp' files
xyz/2.2.1@midhunn/dev export_sources() method: Copied 8 '.m4' files
...
xyz/2.2.1@midhunn/dev: A new conanfile.py version was exported
xyz/2.2.1@midhunn/dev: Folder: /workspace/midhunn/_work/.conan/data/xyz/2.2.1/midhun/dev/export
xyz/2.2.1@midhunn/dev: Exported revision: 948921590483f70de2de5280d1f9360e
ERROR: No package matching 'xyz' pattern found.
Packaging to ff4bbb08e95d07da66d2fed7d63999710a61ad13
xyz/2.2.1@midhunn/dev: Generating the package
xyz/2.2.1@midhunn/dev: Package folder /workspace/midhunn/_work/.conan/data/xyz/2.2.1/midhunn/dev/package/ff4bbb08e95d07da66d2fed7d63999710a61ad13
xyz/2.2.1@midhunn/dev: Calling package()
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.common' file
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.depends' file
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.md' file
xyz/2.2.1@midhunn/dev package(): Packaged 9 '.h' files
xyz/2.2.1@midhunn/dev package(): Packaged 4 '.a' files
xyz/2.2.1@midhunn/dev package(): Packaged 47 '.so' files
xyz/2.2.1@midhunn/dev package(): Packaged 4 files
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.5' file
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.0' file
xyz/2.2.1@midhunn/dev package(): Packaged 3 '.rules' files
xyz/2.2.1@midhunn/dev package(): Packaged 3 '.conf' files
xyz/2.2.1@midhunn/dev package(): Packaged 84 '.1' files
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.html' file
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.txt' file
xyz/2.2.1@midhunn/dev package(): Packaged 1 '.sbom' file
xyz/2.2.1@midhunn/dev: Package 'ff4bbb08e95d07da66d2fed7d63999710a61ad13' created
xyz/2.2.1@midhunn/dev: Created package revision 3e03ff821e0e927edbb9f89c4ae024a9
from conan.
So, Does this mean the right flow is to run conan create ? If not, Could you please guide me?
Yes, conan create
would typically be easier and more robust and work more consistently across developer and CI machines.
One of the many reasons why we chose the source, install, build, etc process is to run custom regression such as smoke test and full regression after conan build and conan export-pkg. Let me know if we can still achieve this with conan create?
You can add the specific tests in the build()
method, and control if they execute or not via user
confs, for example. That way a simple conan create . -c user.myteam:tests_integration=True
for example would be doing that automatically.
All our packages have sources and recipes in the same repository. Should we still stick the above approach?
Yes, the "scm" approach is for packages that the recipe and the sources are in the same repository, still don't want to export
the sources into the recipe for confidentiality/privacy.
self.build_requires("curl/202203.0.0@ci/prod")
Just to make sure, this is only valid if you are using curl
executable only and only at build time, but not using the libcurl
library, not the headers, not linking with the library. If you are doing the second case you need to start using the 2 profiles approach with --profile:build=xxx
and it will break. But this is how Conan 2 works, and it is extremely recommended to prioritize the upgrade to Conan 2, it was released more than 1 year ago and it is the supported version. Same with using the legacy self.copy()
, you need to start using the recommended copy(self, ...)
alternative.
xyz/2.2.1@midhunn/dev export_sources() method: Copied 13 '.depends' files
There is some detail missing in your recipe. The logs clearly indicate that your recipe has an export_sources()
method. Can you please double check?
from conan.
Related Issues (20)
- [feature] More customization options for Platform and PlatformToolset in MSBuildToolchain HOT 2
- [question] conan search SYCA_Casting/0.4.7@master/dev -r=artifactory gets timeout HOT 1
- [question] Components requirements to transitive library HOT 3
- [question] Visual Studio solution missing headers, but `conan build` works just fine HOT 3
- CMAKE_SYSTEM_VERSION is not correctly set when autodetected
- [question] Packaging NuttX libs with circular dependencies HOT 2
- [bug] Inconsistency in `conan install` `require` or `requires` HOT 1
- [bug] conan-2 backup sources workflow does not attempt to fetch from multiple remotes HOT 2
- [feature] Improve version-range-parsing error message HOT 2
- [bug] conan list --graph with non-existing file HOT 2
- [question] how to deploy without dependencies HOT 3
- [suggestion] Label only highest tag as latest release HOT 5
- [bug] Conflict with built-in cmake findpackage scripts can break the build HOT 2
- [question] Is there a way to define information in a recipe that can be used in CMake (or other build systems potentially) HOT 2
- [question] How to build C++ project with Apple Hardened Runtime enabled? HOT 12
- [question] Can I pass --vs-args or --ninja-args arguments to the meson tool when executing build command HOT 1
- [bug] file DOWNLOAD cannot compute hash on failed download, it shows 'status: [35;"SSL connect error"]' HOT 8
- [bug] test_package_folder attribute doesn't work HOT 1
- [question] Limitations of `CMakeToolchain` wrt legacy generators HOT 4
- [bug] Github actions fail when set `tools.system.package_manager:mode` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from conan.