Error running keytool on windows with YubiHSM about yubihsm-shell HOT 11 CLOSED

Is C:\Program Files\Yubico\YubiHSM Shell\bin in PATH?

from yubihsm-shell.

Yes my path system variable content is below.

PATH C:\Program Files\Amazon Corretto\jdk11.0.12_7\bin;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT%\System32\WindowsPowerShell\v1.0;%SYSTEMROOT%\System32\OpenSSH;C:\Program Files\Git\cmd;C:\Program Files\Microsoft VS Code\bin;C:\Program Files\Microsoft SQL Server\150\Tools\Binn;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64;C:\Program Files (x86)\WiX Toolset v3.11\bin;C:\Program Files\Yubico\YubiHSM Shell\bin;

from yubihsm-shell.

That should be sufficient.

2 suggestions:

• Are there yubihsm-shell files in C:\Windows\System32? If yes, remove them and try again
• If the above doesn't work, add debug lines to yubihsm_pkcs11.conf so the content would look like the following, then try again and copy the debug output here. It could be helpful to see if it finds the libraries in the first place.

connector = http://10.128.50.203:12345
debug
dinout
libdebug

from yubihsm-shell.

The only yubihsm related file I see in System32 is for the KSP CNG which we also need.

I already have the debug lines added in my yubihsm_pkcs11.conf. Is there somewhere else I need to look for the debug output other than what I get when running keytool?

from yubihsm-shell.

Yubico Support had me try version 2.3.2 of the YubiHSM-shell from that 2022.6 release on yubico website and that version does not have the issue. It's working successfully and I see all the debug output when running keytool. So it seems like some change introduced after that version caused the issue.

from yubihsm-shell.

No, that's the only place for the debug output. If no libyubihsm log is printed out then the module/library is not loaded at all.

No change in the new release accounts for this error. However, it looks a lot like the typical error where old libyubihsm files are loaded first, which leads to a mismatch between the older version of libyubihsm and the new version of yubihsm_pkcs11. Can you use a dependency walker type of software to see where the libyubihsm.dll library is actually loaded from? The fact that System32 does contain yubihsmksp.dll leads me to suspect that libyubihsm.dll is also somewhere other than C:\Program Files\Yubico\YubiHSM Shell\bin.

from yubihsm-shell.

It will take me a while to be able to do that. Now that we have this working with the older version we need to finish implementing code signing and then I can come back to this and work on trying to get latest versions going. Several days at least.

I can tell you that this was a fresh deployment of Windows Server 2022 that had never had any YubiHSM software installed previously and that we had only downloaded the latest package from Yubico website and installed the CNG and yubihsm-shell. So if there is an older version of libyubihsm on there it would have had to be one of those that put it there I think.

from yubihsm-shell.

I understand. Please let us know how it goes.

The yubihsm-sdk is basically a collection of different softwares, among which are the yubihsm-shell and the yubihsm-cngprovider installers. libyubihsm.dll is part of the yubihsm-shell software/installer and that same installer is included in the yubihsm-sdk software packge.

Can I ask, if this is a completely new installation, how come you installed an older version of yubihsm-shell but the newest yubihsm-sdk? The good news is that if you're using it for code signing then you're probably using the asymmetric key functions and these haven't changed in the latest releases. However, I have to say that I do find it strange that yubihsmksp.dll ended up in System32 while libyubihsm.dll somewhere else. I look forward to your investigation.

from yubihsm-shell.

Can I ask, if this is a completely new installation, how come you installed an older version of yubihsm-shell but the newest yubihsm-sdk?

We didn't. We downloaded 2023.11 package from https://developers.yubico.com/YubiHSM2/Releases/ and installed the CNG provider and ubihsm shell (sic) from the installers included in that package. We had no issues setting up the CNG. Then when attempting to setup Java code signing ran into the original issue submitted with keytool (and jarsign). I submitted an issue on here but then also was pointed to support request on yubico website so submitted to that as well. Support team was the one who suggested trying the older version of yubihsm-shell as they said they replicated the issue with all versions higher than that one. So I uninstalled version packaged in 2023.11 and then tried the one from 2022.6 release as they suggested and it worked.

from yubihsm-shell.

Apologies for taking this long to figure it out. I believe your correspondence with Yubico support explained that the CNG provider installs its own version of libyubihsm in system32 and that leads to a mismatch between the libyubihsm version that the CNG provider vs yubihsm_pkcs11 depends on.

There is now a new YubiHSM SDK release (available on https://developers.yubico.com/YubiHSM2/Releases/) where the CNG provider is rebuilt using the latest libyubihsm. We will also be making sure that this mismatch does not occur again in future releases.

from yubihsm-shell.

Yes we found files in System32 as you suggested but they were named a little differently and I guess windows search didn't find them when I looked before. Thanks for your help.

from yubihsm-shell.