[question] Problem deploying oneMKL prepackaged binaries in custom package about conan HOT 21 CLOSED

Hi @LarrxX

Thanks for your question.

Some quick feedback first:

self.cpp_info.includedirs = ['include']
self.cpp_info.libdirs = ['lib']

These are already the default, not necessary

self.env_info.PATH.append(os.path.join(self.package_folder, "bin"))

This is the legacy Conan 1.X old environment model. It has no effect in Conan 2 at all. In fact it should be giving some warnings, and it will start failing in some future Conan 2.X versions.
You might want to use the self.buildenv_info or self.runenv_info instead.

Then, if you are using the CMakeDeps generator, sharing the ```....data.cmakegenerated from it for theonemkl`` dependency would be useful. I'd also check that the folders containined in that file do really contain the necessary files.

Would it be possible to have something that we can reproduce from our end? That would really help.

In fact, it is not necessary to add those folders to the PATH env-var, as Conan knows how to do this automatically already for the bindirs and libdirs when corresponding, for example the VirtualBuildEnv and VirtualRunEnv generators.

self.cpp_info.bindirs = ['bin', 'lib']

This, for the shared case is most likely not necessary either. Conan generators know how to deal with libdirs and bindirs in the different platforms.

To understand what is happening for the onemkl I think we would need more understanding and information.

The first thing that I'd try is to remove the --deployer=full_deploy part, and try to use the package directly from the cache (which is the most recommended way for most cases). This is to rule out possible effects from the deployment, for example I am not sure if you are aware that env-scripts in Linux are not relocatable. But we also would need to understand more about the platform, and see details about the specific platform.

In Conan 2, the conan export-pkg can also run an automatic test_package, like conan create, that really helps to test packages created with this method. Maybe

from conan.

Thank you for your detailed response @memsharded . If I read you correctly, everything I put in my package_info() is unnecessary ':-)

As a first step, here are the generators deployed for oneMKL.
generators.zip

I am using the following generators:
generators = "CMakeToolchain", "CMakeDeps", "VirtualBuildEnv", "VirtualRunEnv"

I am running the conan recipe and deployment on a windows machine, then transferring the deployed folder to an air-gapped windows machine (air-gapping is why we can't use conan directly. I tried insisting on installing a local artifactory, unsuccessfully).

I will try using the cache version directly by copying it manually, and look into how test_package works (I never used it before).

I have a feeling the time difference is going to be a hassle to deal with (especially that I will be off until next monday :) )

I will let you know if I have any new info after a round of testing today.

from conan.

Thank you for your detailed response @memsharded . If I read you correctly, everything I put in my package_info() is unnecessary ':-)

The self.cpp_info.libs = collect_libs(self) is necessary 🙂

generators = "CMakeToolchain", "CMakeDeps", "VirtualBuildEnv", "VirtualRunEnv"

In Conan 2, the last 2 are not necessary, they are already automatically used by default.

I will try using the cache version directly by copying it manually, and look into how test_package works (I never used it before).

Sounds good. If you want to share that recipe + test_package in a git repo or something, that would be great, so we can try it. If don't want to disclose it publicly feel free to write with details to [email protected] and we will handle it privately.

I have a feeling the time difference is going to be a hassle to deal with (especially that I will be off until next monday :) )
I will let you know if I have any new info after a round of testing today.

Sounds good, looking forward your feedback!

from conan.

The "good" news is that I am having the exact same error in the test_package. Here is my full conan recipe + test folder. You however need to have oneMKL installed on your system and set the folder in an environment variable (or just hardcode it in the recipe where needed :) )

onemkl.zip

from conan.

Thanks for the repro code!

I have been trying it in my laptop in Windows, and it seems to work fine!

SET MKL_DIR=C:\Program Files (x86)\Intel\oneAPI\mkl\2024.1
conan create .
...
======== Testing the package: Executing test ========
onemkl/2024.1.0 (test package): Running test()
onemkl/2024.1.0 (test package): RUN: Release\TestOneMKL
You are using oneMKL 2024.1

The only fix I needed was in test_package/conanfile.py -> test() method, use cmd = os.path.join(self.cpp.build.bindir, "TestOneMKL") (not TestOneMKLPackage).

Are you also trying it in Windows? Other platform? What is exactly the error for a conan create . with your recipe?

from conan.

I was using export-pkg, but create gives the same error. Here is the full output from my command (this way you can also see my profiles, maybe something in there is mucking things up?)

output.log

from conan.

That helps, thanks.

I am able to reproduce now with conan create . -o *:shared=True. It seems the static linkage is working fine but the shared one is failing, I'll have a look.

from conan.

Oh that's a relief, at least I'm not in a singularity. And yes, -o *:shared=True is mandatory since one of the libraries is 2GB !!!

from conan.

Weird thing is, I tried doing the same with another binary project and it is working perfectly. Maybe the binaries for oneMKL are different? Maybe I shouldn't be renaming them when I copy them?
I am going to try no renaming, and see what happens.

from conan.

It seems the source is the discrepancy in naming inside the package:

lib
   \mkl_blacs_ilp64.lib
bin
   \mkl_blacs_ilp64.2.dll

That extra .2 is confusing the CMake detection that assumes that the import-lib and the shared lib are named the same.

from conan.

I am going to try no renaming, and see what happens.

That sounds reasonable. In general the policy we have in ConanCenter is that the original binary names shouldn't be changed unless strictly necessary. Please let me know how it goes.

from conan.

I think it is coming from the renaming. The dlls and the .lib files are not the same. For example I have mkl_blacs_msmpi_ilp64.2.dll and mkl_blacs_ilp64.lib I'll fix this up and let you know

Sorry to have bothered with this non conan issue... (if it turns out to be that)

from conan.

I think it is coming from the renaming. The dlls and the .lib files are not the same. For example I have mkl_blacs_msmpi_ilp64.2.dll and mkl_blacs_ilp64.lib I'll fix this up and let you know

yes, exactly, I just posted that above 1 min ago: #16210 (comment) 🙂 🙂

from conan.

OK so no renaming at all does not work, and matching the renaming is not working either... I guess oneMKL has its own internal way of dealing with things.

They do have a CMakefile of hundreds of lines, guess I'll have to delve into it and try to understand how they combine their libs and dlls.

from conan.

If I want to copy and use oneMKL's own MKLConfig.cmake and MKLConfigVersion.cmake instead of using the ones generated by conan, how would I do that?

from conan.

You can use the in-package cmake files, as described here: https://docs.conan.io/2/examples/tools/cmake/cmake_toolchain/use_package_config_cmake.html

You need to make sure the file is packaged, and disable the generation of the CMakeDeps file for it.
Be aware of the limitations described in that documentation page: https://docs.conan.io/2/examples/tools/cmake/cmake_toolchain/use_package_config_cmake.html#important-considerations

from conan.

OK thanks, I'll try that approach and see what happens

from conan.

OK seems to be goind better, but there are a lot more intricacies than just cpying the libraries. I'm adapting my recipe to at least allow the oneMKL cmake file to pass without errors.

I will keep you informed and will post the final recipe when I get it working (but I'm off until next monday so don't hold your breath :) )

Thanks again for all your help.

from conan.

OK seems to be goind better, but there are a lot more intricacies than just cpying the libraries. I'm adapting my recipe to at least allow the oneMKL cmake file to pass without errors.

I am curious about what are the specifics of this, at the end of the day, it is just some compiled libraries, looking forward to your learnings.

I will keep you informed and will post the final recipe when I get it working (but I'm off until next monday so don't hold your breath :) )

Sounds fantastic. Actually, we have recently created https://github.com/conan-io/conan-recipes-contrib, still empty, but the goal is that this would be a place where this kind of recipes that cannot go into ConanCenter (closed-source for example), but that can still be useful for the community.

Thanks again for all your help.

My pleasure, happy to help! Thanks to you for the feedback.

from conan.

So here is the final working recipe for me. It's not generic enough I guess to upload it as a contrib (because the file list is hardcoded). But since I do not need to do any renaming anymore, I might be able to create something more generic (if and when I have the time)

onemkl.zip

Thanks again @memsharded for your help

from conan.

That is great news.

I am happy to see that from the Conan side, the recipe is actually reduced to just a copy (a very elaborated copy, with careful definitions of things to copy, but a copy) and just:

def package_info(self):
        self.cpp_info.builddirs = [self.package_cmake_subfolder]
        self.cpp_info.set_property("cmake_find_mode", "none")

Seems to make a working package.

It's not generic enough I guess to upload it as a contrib (because the file list is hardcoded). But since I do not need to do any renaming anymore, I might be able to create something more generic (if and when I have the time)

No rush, it can wait a bit. In any case the idea of contrib is that it can get recipes half-ready, it will not have the same requirements as ConanCenter, but instead to serve as an "incubator" or place to share partial solutions that even if not complete can still be useful to others.

Don't worry too much about that, if you can in the future contribute with a slightly improved version of your recipe, that is great, if not, also great, it will eventually be added by us or other users.

I think this ticket is solved then and can be closed then. Don't hesitate to re-open or create new tickets as necessary for any further question.
Thanks very much for your feedback!

from conan.