If you decrypt a random input using RSAES-PKCS-v1_5, then there is a
non-negligible chance that the result will look like a valid plaintext
(that is why RSAES-PKCS-v1_5 shouldn't be used anymore). This was the
cause of an intermittent failure in a test that did a cms-encrypt
operation targetting multiple recipients.
The failure happened during key-only decrypt. The recipient decrypts
every RSA ciphertext -- only one is supposed to decrypt successfully,
which would reveal the right content-key. Occassionally, more than
one decrypted successfully.
Update the test by specifying the recipient cert in the decrypt op
(this avoids looping over all RSA ciphertexts).
Add a new test to get coverage for key-only decrypt, but use RSA-OAEP
during the encrypt op.
Fixes https://github.com/openssl/project/issues/380
Testing:
$ make TESTS='test_cms' test
Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23055)
