[question] Handling package that is platform and variant agnostic but not the tests in the package about conan HOT 30 CLOSED

Sounds good, keep me posted, thanks for the feedback.

The thing is the functional tests are not very straight forward(due to legacy reasons, nothing to do with conan offcourse )in our use-case.

The thing is that such complexity will still be there, no matter if you use the test_package or an external conanfile.py. But the former might have a little less friction to use, for example, using the self.tool_requires(self.tested_reference_str) doesn't require to update the version when new version of the package is changed, because it is automatically defined by Conan.

from conan.

Sorry I figured out with : ctests = self.conf.get("user.build:ctests", default="")

from conan.

Happy to know that it worked.

Thanks for the feedback!

from conan.

Hi! Thanks a lot for your question.

Depending on your current workflow, the easiest way might be to decouple the package creation from the package tests. (A question remains to wether having such complex tests is in the spirit of the Conan's test_packages)

When you run conan create, you can pass -tf="" to skip running the tests automatically if needed, and then you can manually call conan test with your reference and passing the necessary profiles for each of your configurations

Let me know if that helps :)

from conan.

Some extra info after re-reading your question (Thanks @memsharded for the headsup!)

Note that you can also call conan create for each configuration, like conan create . --build=missing -tf=test1 -pr=profile1, conan create . --build=missing -tf=test2 -pr=profile2 etc, and the --build=missing flag will ensure you don't build more than once, and then the test packages will be automatically built for your configuration

from conan.

We use conan build + conan export-pkg. I do not see -tf option for conan build.

from conan.

The conan export-pkg has learned to run the test_package automatically in Conan 2, and equal to conan create, it also contains the conan export-pkg ... -tf/--test-folder argument that can be used to specify a different folder or to disable the automatic test (with an empty string).

It is not the conan build that do the test_package, because conan build doesn't create a package, but conan export-pkg does, so only conan export-pkg can test the package.

from conan.

Not sure if I was clear and we understand each other. https://docs.conan.io/2/tutorial/creating_packages/test_conan_packages.html
The tools binary is created on a platform and variant independent configuration.
But the tests that use the generated tools binary depends on platform and variant information as they generate code and compile afterwards. So, is the most optimal way to move out the tests to another package as they depend on platform and variant and use the tools binary as dependency?

from conan.

I think the problem is that we might be talking about different things:

• https://docs.conan.io/2/tutorial/creating_packages/test_conan_packages.html talks about the test_package functionality which is intended to test that the package is created correctly. This is enough to do it in 1 configuration typically, if the package is agnostic of the configuration.
• It seems that you are talking about full functional/e2e tests (or maybe unittests). The problem with unittests is that if they require some internal, non-public headers, they must be run at the time of building, they cannot be run later as those headers are not part of the package.

What is exactly your case? testing functional/e2e that doesn't require internal private headers?

Then, the second important thing to understand is if you want to actually package your tests and test results. In most cases it is not necessary to package the tests and the test results are just output in CI for red/green, but not really packaged. If you don't need to package the tests and test results, the test_package functionality and feature can be used for the purpose, because the test_package/conanfile.py has its own definition of settings that can be different from the package one. If you want to package the results, then there might be 2 options:

• Using another package as you mentioned
• Using package metadata: https://blog.conan.io/2023/10/24/Conan-launches-metadata-files.html

from conan.

It is functional or e2e tests for the tools package. And the tools package generates a binary that shall be available to the functional /e2e tests. The functional tests depend on the platform and variant as they generate code using the binary from tools package and compile and run the tests. My intension is not to package the tests but to be able to execute the tests. My dilemma is should i move the functional/e2e tests to another package or live in the same package? As the tools binary is using default Windows configuration while generated code in the tests folder need platform and variant configuration in order to compile, run static code analysis on the generated code.

from conan.

My dilemma is should i move the functional/e2e tests to another package or live in the same package?

If you are not packaging the tests or tests results, then you don't need another package, you just need a simpler conanfile.py consumer recipe, that depends on your package one. This recipe will typically do not need the package() or package_info() methods, because they are not creating a package, just requiring some dependencies and building+testing something locally.

The test_package/conanfile.py is nothing but this, just a consumer project that depends on the created package. But it is not part of the package, it is not packaged, the artifacts are not copied into the package or anything like that. At all effects it is the same as if you have an independent conanfile.py that consumes/requires your package, but the advantage is that it is automated and integrated.

In both cases, both with the test_package/conanfile.py or with an independent conanfile.py you can have multiple binary configurations, even if the package you created only has one configuration. The consumers will have their own settings and can be built and tested with multiple different configurations. The fact that the package has only 1 configuration do not affect the test_package one.

The only thing is that you might want to do a self.tool_requires(self.tested_reference_str) in the test_package/conanfile.py, and when you do create and test it you might want to do:

conan create . --build=missing --build-require -pr=myprofile

In general it would be less work and less maintenance if the tests are in the same package repo in test_package because it is more integrated and automatic than being separated.

I am not sure if I am fully understanding the question, please let me know if the above makes sense to you.

from conan.

I will try based on your suggestions and get back to you, The thing is the functional tests are not very straight forward(due to legacy reasons, nothing to do with conan offcourse )in our use-case.

from conan.

We have a cross compilation use-case. If I provide --build-missing argument, then it tries to build the missing binaries(inspite of providing build and host profiles) with default configuration instead of picking pre-built binaries from the conan remote. The structure of my folders are:
tools_pkg:

• src
• test
  - conanfile.py
  - CMakeLists.txt
  - testfolder1/CMakeLists.txt
  - testfolder2/CMakeLists.txt
• conanfile.py
• doc
• data
  Do you have an example for me ?
  Also the self.tested_reference_str when I print it says None in my case.

from conan.

We have a cross compilation use-case. If I provide --build-missing argument, then it tries to build the missing binaries(inspite of providing build and host profiles) with default configuration instead of picking pre-built binaries from the conan remote. The structure of my folders are:

This shouldn't be the case. The --build=missing checks if there are existing binaries first before deciding to build them. If it is building them is because those binaries are not found in the cache or any available remote.

If you want to know why some binary is missing, you can try with the conan graph info --build=missing first, or even the conan graph explain. But there is something there that is not very clear.

The setup should be pretty straightforward:

• ``conan new cmake_lib -d name=mypkg -d version=0.1
• Remove the src, modify the include and the CMakeLists.txt to make it a header-only library
• Modify the conanfile.py: remove the settings, the options, the build(), change the package() to do a copy() of the headers.
• Now you can do conan create . --build=missing with different profiles, and you will see how it only tests the different configurations, but it will not re-create over and over the package itself.

from conan.

Ctest_package.zip
Have I done the modifications correctly?
Another thing that I noticed from the example is :
conan create . --build=missing creates the package and the test_package with the same configurations.
My use-case is that : The top level uses default profile while the tests use a build profile and a host profile.

from conan.

My use-case is that : The top level uses default profile while the tests use a build profile and a host profile.

This is not fully clear. If the package is configuration agnostic, it doesn't depend on a specific profile, it can be built with any profle. If that is not the case, then it is not configuration agnostic. The typical use case is a header-only library, it is configuration agnostic, it doesn't matter in which platform or with which profile you build the package, the package will always end with the exact same headers packaged (and the same exact package_id). If your package is not configuration agnostic, then it depends on the profile, and it will get different binaries and different package_id for different configurations?

In the recipe that you provided, you still have settings = "os", "compiler", "build_type", "arch", and you still have a build() method that is building a binary that does depend on the configuration. It is not enough specifying package_type = "header-library", you are building a compiled library, with its specific settings. Please check my bullet point above:

• Modify the conanfile.py: remove the settings, the options, the build(), change the package() to do a copy() of the headers.

from conan.

The package itself does not need any configurations but the tests that use the binary of the package do need the configuration(build profile and host profile). Yes the tests will get different package_id for different configurations. In that case, will the above method work meaning using conan create . --build=missing and having a conanfile.py at the top level and another one in test folder? Or the only way is to seperate the package and the tests itself. In my case, the package itself is not dependning on a specifi cprofile but the tests are.

from conan.

). Yes the tests will get different package_id for different configurations. In that case, will the above method work meaning using conan create . --build=missing and having a conanfile.py at the top level and another one in test folder?

Yes, I think this works, and in general will be more integrated and automatic than having a different separate package.

from conan.

Sorry but how do the tests get the profile information if I trigger with conan create . --build=missing? I changed the package but I still miss the information regarding how shall the test_package know against which configurations shall it test.
Ctest_package.zip

from conan.

See the code attached:

• You can remove the full settings, you must do it if it is header-only
• The test_package can have full settings
• There is a build.py that shows how it works
• The package() is executed exactly once, but tested 3 times with different configs

Ctest_package.zip

from conan.

Yes I understood the attachment . Thanks very much for it. If I use conan build then, conan export-pkg would trigger the tests with the command:

 conan export-pkg .  -pr:b=default -pr:h=ndp -tf test_package/

from conan.

Is it not allowed to use the build() method in the top level conanfile.py? In the package () method is it not allowed to use self.run(cmd) for compiling the binary for the package before running the tests with export-pkg?

from conan.

s it not allowed to use the build() method in the top level conanfile.py? In the package () method is it not allowed to use self.run(cmd) for compiling the binary for the package before running the tests with export-pkg?

But if you build something, then the binary will not be really platform independent? Or you are only building and running tests.
If you want lets say, build and run tests, but don't let settings affect the package_id, the use case will be the same as a header-only library, you can check: https://docs.conan.io/2/tutorial/creating_packages/other_types_of_packages/header_only_packages.html#header-only-library-with-tests

• Keep the settings and options you need
• Check that whatever is built is not part of the package final artifacts.
• Recall to do self.info.clear() in the package_id()

from conan.

Thanks for your reply. I have the build method to build a matlab toolbox package which is a binary , this is a matlab command, it is not depending on any settings. All your provided inputs and suggestions work. However I would have one last question:
How can I possibly pass some selected tests to execute:
Normally I would use : -c user.build:ctests=StaticCode where StaticCode is a cmake target that executes static code analysis.
I am using this with conan export-pkg:

conan export-pkg . -of tools_build --user autosar --channel pkgtest -pr:b=default -pr:h=kdp -tf=test -c user.build:ctests=StaticCode

And in the test/conanfile.py , the test method contains:

def test(self):
        self.run("ctest --output-on-failure -R""")

from conan.

ctests = self.conf.get("user.build:ctests", default="")

Yes, that looks good, the way to get conf values in your code.

from conan.

It works but if I would like to use layout method with cmake_layout(self), Do I need to additionally handle anything?
I seem to be stuck at the cmake.build() with the error: "Error: could not load cache".
My folder structure is:
tools_pkg

• _source
• _tests
  conanfile.py
  CMakeLists.txt
  • tf1
    CMakeLists.txt
    • tfsubfolder1
      CmakeLists.txt (conatins tests that can be executed with ctests)
      I tried to use in the layout method however, still cannot get get the cmake_layout to work.
      self.folders.source = "."
      self.folders.build = os.path.join("build", str(self.settings.build_type))
      self.folders.generators = os.path.join(self.folders.build, "generators")

from conan.

Sorry I am not sure if I understand the issue or the layout. If you want to update the project we used above, and clarify instructions to reproduce the issue, that would help.

from conan.

Sorry It was a corrupted cache issue, fixed now. Is it possible to have the path from self.tested_reference_str ? I need to copy the binaries to a special location to suit some legacy scripts.

from conan.

Yes, something like self.dependencies[self.tested_reference_str].package_folder in the test_package generate() or build() method should work.

from conan.

I used it like this: self.dependencies.build.get(self.tested_reference_str) , Also works.
But in general, thank you very much for your support in helping me find a solution.
Ticket can be closed. Many Thanks!

from conan.