RSAOpenSSL doesn't appear to support HSM-managed keys?

[andyhopp]

Description

I have an OpenSSL dynamic engine installed that manages keys on an HSM, but when I attempt to use a PFX to create an X509Certificate2 that uses that key, I am presented with an error:
Error occurred during a cryptographic operation.

HasPrivateKey on the X509Certificate2 returns True.

A sample stack trace is below:
Unhandled exception. System.Security.Cryptography.CryptographicException: Error occurred during a cryptographic operation.
at System.Security.Cryptography.RSAOpenSsl.TrySignHash(ReadOnlySpan1 hash, Span1 destination, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding, Boolean allocateSignature, Int32& bytesWritten, Byte[]& signature)
at System.Security.Cryptography.RSAOpenSsl.SignHash(Byte[] hash, HashAlgorithmName hashAlgorithm, RSASignaturePadding padding)
at System.Security.Cryptography.RSAPKCS1SignatureFormatter.CreateSignature(Byte[] rgbHash)
...

Reproducing this error will be tricky unless you have access to an HSM. I am happy to volunteer the use of mine.

Configuration

~/.dotnet/dotnet --list-sdks
3.1.300 [/home/ec2-user/.dotnet/sdk]
Amazon Linux 2, x64

Is there a way to specify the engine that will be used by System.Security.Cryptography.OpenSsl through an appSetting, environment variable, etc? From my naive review of the code, it appears that the native interop EnsureOpenSslInitialized() method only calls OPENSSL_add_all_algorithms_conf().

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq
Notify danmosemsft if you want to be subscribed.

[andyhopp]

Update: I have implemented a class that uses the ENGINE_* functions to load/initialize/set as the default provider for (RSA/DSA/Cipher/etc.) and would like to contribute this back. Is there someone who would be willing to coach me on this?

[vcsjones]

I assume you mean add new public APIs and classes for other developers to consume, right? If that's the case - the place to start is to go through the API Review Process with a proposal. Basically open up a new issue with un-implemented methods for your new public APIs, give a few examples of usage, etc. An example proposal is here.

[bartonjs]

Even with the API proposal from #37383, how would you expect the PFX load to understand to associate the private key to an HSM? Or are you just hoping it creates a new key, instead of accessing an existing persisted key?

[andyhopp]

Terribly sorry; I had responded to the thread via email, but GitHub didn't attach it
The HSMs I'm familiar with actually use the private key payload as a "pointer" to the HSM-stored key material, and the cryptographic operations are handled by the dynamic engine (or CNG on Windows). This is handled transparently by OpenSSL/CNG, so no code modifications need to be made by the developer (beyond adding the dynamic engine API described above). This works for both PKCS12 and X509Store certs.

[bartonjs]

Do you have an example of such a "pointer"? We don't use OpenSSL's PKCS12/PFX loader any more, but manually associate the X.509 and PKCS8 data.

[vcsjones]

Is there a strong use case for loading dynamic engines programmatically as opposed to adding it to openssl.cnf?

Something like:

openssl_conf = openssl_init

[ openssl_init ]
engines = engine_section

[ engine_section ]
myhsm = myhsm_section

[ myhsm_section ]
engine_id = myhsm
dynamic_path = /usr/lib/engines/myhsm.so
default_algorithms = ALL
init = 1

I think that would be enough to get an dynamic engine loaded and initialized.

[bartonjs]

Given that the author didn't respond to the request for more information 2 years ago, I'm going ahead and closing this.

[benlongo]

Could this be reopened? I think the issue still stands

[bartonjs]

@benlongo This issue never had enough information to investigate it, so reopening will do no good. If you have information to add you are welcome to add it here (which might lead to reopening it), or just create a new issue.