openssl/test
Tomas Mraz 78539b250b EVP_DigestSign/VerifyFinal: Duplicate the pctx to allow multiple calls
The legacy implementation duplicates the pctx before creating/verifying
the signature unless EVP_MD_CTX_FLAG_FINALISE is set. We have to do the
same with provided implementations.

Fixes #16321

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16422)
2021-08-26 16:06:57 +02:00
..
certs test_pem_reading: Test loading a key from a file with multiple PEM data 2021-07-02 15:33:34 +02:00
ct Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
d2i-tests
helpers Fix test code to not assume NUL terminated strings 2021-08-24 14:22:06 +01:00
ocsp-tests
recipes Extend tests for SM2 decryption 2021-08-24 14:22:07 +01:00
smime-certs Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
ssl-tests Update copyright year 2021-06-17 13:24:59 +01:00
testutil test: add a shim function for the apps's opt_legacy_okay() function 2021-07-12 09:13:20 +10:00
aborttest.c
acvp_test.c test: fix use after scope problem in ACVP test 2021-07-19 13:08:17 +10:00
acvp_test.inc
aesgcmtest.c
afalgtest.c Update copyright year 2021-07-29 15:41:35 +01:00
algorithmid_test.c
asn1_decode_test.c
asn1_dsa_internal_test.c
asn1_encode_test.c Update copyright year 2021-07-29 15:41:35 +01:00
asn1_internal_test.c TEST: Check that i2d refuses to encode non-optional items with no content 2021-07-10 17:05:07 +02:00
asn1_string_table_test.c
asn1_time_test.c
asynciotest.c
asynctest.c
bad_dtls_test.c
bftest.c
bio_callback_test.c Update copyright year 2021-06-17 13:24:59 +01:00
bio_core_test.c BIO_new_from_core_bio: Fix heap-use-after-free after attach 2021-06-25 10:11:29 +10:00
bio_enc_test.c
bio_memleak_test.c Update copyright year 2021-07-29 15:41:35 +01:00
bio_prefix_text.c
bio_readbuffer_test.c
bioprinttest.c
bn_internal_test.c
bn_rand_range.h
bntest.c bn: procduce correct sign for result of BN_mod() 2021-07-07 19:12:48 +10:00
bntests.pl
build.info test: make build descriptions more consistent 2021-07-12 09:13:20 +10:00
ca-and-certs.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
casttest.c
CAtsa.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
chacha_internal_test.c
cipher_overhead_test.c
cipherbytes_test.c
cipherlist_test.c
ciphername_test.c
clienthellotest.c
cmactest.c
cmp_asn_test.c
cmp_client_test.c Ensure libctx/propq is propagated when handling X509_REQ 2021-06-05 17:39:27 +10:00
cmp_ctx_test.c test_cmp_ctx: Avoid using empty X509 with i2d 2021-07-10 17:05:07 +02:00
cmp_hdr_test.c Work around a 32-bit mingw failure 2021-07-02 17:44:02 +01:00
cmp_msg_test.c Ensure libctx/propq is propagated when handling X509_REQ 2021-06-05 17:39:27 +10:00
cmp_protect_test.c Use the new ASN.1 libctx aware capabilities in CMP 2021-06-05 17:39:10 +10:00
cmp_server_test.c CMP: Add missing getter functions to CRMF API and CMP API 2021-06-30 10:38:23 +02:00
cmp_status_test.c Fix test code to not assume NUL terminated strings 2021-08-24 14:22:06 +01:00
cmp_vfy_test.c Use the new ASN.1 libctx aware capabilities in CMP 2021-06-05 17:39:10 +10:00
cms-examples.pl
cmsapitest.c
conf_include_test.c test: avoid memory leaks on errors 2021-06-26 11:33:52 +10:00
confdump.c Update copyright year 2021-06-17 13:24:59 +01:00
constant_time_test.c
context_internal_test.c
crltest.c
ct_test.c
ctype_internal_test.c
curve448_internal_test.c
d2i_test.c
danetest.c
danetest.in
danetest.pem
data2.bin
data.bin
default-and-fips.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
default-and-legacy.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
default.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
defltfips_test.c Add some testing for the case where the FIPS provider fails to load 2021-07-28 10:35:06 +10:00
destest.c test: add unit tests for TDES key wrap 2021-08-25 17:31:29 +02:00
dhtest.c dh_test: fix coverity 1473239 Argument cannot be negative (NEGATIVE_RETURNS) 2021-06-30 13:55:09 +10:00
drbgtest.c
dsa_no_digest_size_test.c
dsatest.c dsatest: Properly detect failure in generate/sign/verify 2021-08-11 12:07:08 +02:00
dtls_mtu_test.c
dtlstest.c
dtlsv1listentest.c
ec_internal_test.c
ecdsatest.c
ecdsatest.h
ecstresstest.c APPS & TEST: Use ossl_[u]intmax_t rather than [u]intmax_t 2021-06-22 19:50:11 +10:00
ectest.c
endecode_test.c Decoding PKCS#8: separate decoding of encrypted and unencrypted PKCS#8 2021-06-09 17:00:10 +02:00
endecoder_legacy_test.c test: fix coverity 1469427 Improper use of negative value (NEGATIVE_RETURNS) 2021-06-30 13:55:09 +10:00
enginetest.c Update copyright year 2021-06-17 13:24:59 +01:00
errtest.c err: clear flags better when clearing errors. 2021-06-10 18:11:45 +10:00
evp_extra_test2.c
evp_extra_test.c EVP_DigestSign/VerifyFinal: Duplicate the pctx to allow multiple calls 2021-08-26 16:06:57 +02:00
evp_fetch_prov_test.c
evp_kdf_test.c Add HKDF negative tests 2021-07-06 10:56:19 +10:00
evp_libctx_test.c Add a gettable for provider ciphers to return the EVP_CIPH_RAND_KEY flag 2021-06-08 15:16:06 +10:00
evp_pkey_dparams_test.c
evp_pkey_provided_test.c Add tests for EVP_PKEY_get_utf8_string_param(), both positive and negative 2021-08-18 17:05:57 +02:00
evp_test.c evp_test: use correct size in memory clear 2021-06-27 16:12:46 +10:00
exdatatest.c
exptest.c
fatalerrtest.c
ffc_internal_test.c
filterprov.c
filterprov.h
fips-alt.cnf Add some testing for the case where the FIPS provider fails to load 2021-07-28 10:35:06 +10:00
fips-and-base.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
fips.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
generate_buildtest.pl
generate_ssl_tests.pl Update copyright year 2021-06-17 13:24:59 +01:00
gmdifftest.c
hexstr_test.c
hmactest.c test: check for NULL returns better 2021-06-26 11:33:52 +10:00
http_test.c
ideatest.c
igetest.c
insta_ca.cert.pem Remove executable mode attributes of non-executable files 2021-07-13 16:04:32 +10:00
insta.priv.pem Remove executable mode attributes of non-executable files 2021-07-13 16:04:32 +10:00
keymgmt_internal_test.c
legacy.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
lhash_test.c
mdc2_internal_test.c
mdc2test.c
memleaktest.c
modes_internal_test.c
moduleloadtest.c
namemap_internal_test.c
ocspapitest.c
ossl_store_test.c test/ossl_store_test.c: Adapt the use of datadir for VMS paths 2021-06-26 06:43:08 +02:00
p_test.c
packettest.c
param_build_test.c
params_api_test.c
params_conversion_test.c test: handle not a number (NaN) values in the param conversion test. 2021-07-28 10:30:45 +10:00
params_test.c test: check for NULL returns better 2021-06-26 11:33:52 +10:00
pbelutest.c
pbetest.c
pem_read_depr_test.c Update copyright year 2021-07-29 15:41:35 +01:00
pemtest.c Update copyright year 2021-07-29 15:41:35 +01:00
pkcs7_test.c
pkcs7-1.pem
pkcs7.pem
pkcs12_format_test.c
pkey_meth_kdf_test.c
pkey_meth_test.c
pkits-test.pl
poly1305_internal_test.c
property_test.c
provfetchtest.c Add a test to check that RAND_bytes_ex() works with a child lib ctx 2021-06-24 14:48:15 +01:00
provider_fallback_test.c Update copyright year 2021-06-17 13:24:59 +01:00
provider_internal_test.c Merge ossl_provider_activate() and ossl_provider_activate_child() 2021-06-24 14:48:14 +01:00
provider_internal_test.cnf.in Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
provider_status_test.c Add test for provider gettables 2021-07-06 10:55:19 +10:00
provider_test.c
proxy.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
rand_status_test.c
rc2test.c
rc4test.c
rc5test.c
rdrand_sanitytest.c
README-dev.md
README-external.md
README.md
README.ssltest.md Move AllowClientRenegotiation tests 2021-06-15 12:36:51 +02:00
recordlentest.c
rsa_complex.c
rsa_mp_test.c
rsa_sp800_56b_test.c Allow small RSA exponents in the default provider 2021-08-13 10:35:56 +02:00
rsa_test.c Fix use of uninitialized memory in test_rsa_oaep 2021-07-01 09:57:05 +10:00
run_tests.pl Fix exit code for VMS in util/wrap.pl and test/run_tests.pl 2021-06-17 08:24:13 +01:00
sanitytest.c Update copyright year 2021-07-29 15:41:35 +01:00
secmemtest.c
serverinfo2.pem
serverinfo.pem
servername_test.c
session.pem
sha_test.c test: add test cases for SHAxxx helper functions 2021-06-16 18:30:28 +10:00
shibboleth.pfx
shlibloadtest.c
simpledynamic.c Update copyright year 2021-07-29 15:41:35 +01:00
simpledynamic.h TEST: Modify simpledynamic.[ch] to allow use on VMS as well 2021-06-24 15:55:14 +10:00
siphash_internal_test.c
sm2_internal_test.c Correctly calculate the length of SM2 plaintext given the ciphertext 2021-08-24 14:22:07 +01:00
sm3_internal_test.c sm3: fix function names after the big ossl_ prefix addition. 2021-06-21 12:14:14 +10:00
sm4_internal_test.c
smcont_zero.txt
smcont.bin
smcont.txt
sparse_array_test.c Update copyright year 2021-06-17 13:24:59 +01:00
srptest.c
ssl_cert_table_internal_test.c
ssl_ctx_test.c
ssl_old_test.c
ssl_test_ctx_test.c
ssl_test_ctx_test.cnf
ssl_test.c
ssl_test.tmpl
sslapitest.c Fix state name abbreviation 2021-08-19 08:24:16 +10:00
sslbuffertest.c
sslcorrupttest.c
stack_test.c
sysdefault.cnf Add oid_section to sysdefault.cnf to test adding new oids 2021-08-05 09:21:00 +10:00
sysdefaulttest.c
test_test.c
test.cnf
testcrl.pem
testdsa.pem
testdsapub.pem
testec-p256.pem
testecpub-p256.pem
tested448.pem
tested448pub.pem
tested25519.pem
tested25519pub.pem
testp7.pem
testreq2.pem
testrsa2048.pem
testrsa_withattrs.der
testrsa_withattrs.pem
testrsa.pem
testrsapss.pem
testrsapssmandatory.pem
testrsapub.pem
testsid.pem
testutil.h Ensure libctx/propq is propagated when handling X509_REQ 2021-06-05 17:39:27 +10:00
testx509.pem
threadstest_fips.c
threadstest.c test: fix test ordering in threads test 2021-06-30 17:54:45 +10:00
threadstest.h
time_offset_test.c
tls13ccstest.c
tls13encryptiontest.c Update copyright year 2021-06-17 13:24:59 +01:00
tls13secretstest.c
tls-provider.c
uitest.c
upcallstest.c Add a test for the newly added OBJ upcalls 2021-06-16 15:04:50 +10:00
user_property_test.c
v3_ca_exts.cnf
v3-cert1.pem
v3-cert2.pem
v3ext.c
v3nametest.c
verify_extra_test.c
versions.c
wpackettest.c
x509_check_cert_pkey_test.c
x509_dup_cert_test.c
x509_internal_test.c Fix test case for a2i_IPADDRESS 2021-08-06 12:31:24 +02:00
x509_time_test.c Fix test code to not assume NUL terminated strings 2021-08-24 14:22:06 +01:00
x509aux.c

Using OpenSSL Tests

After a successful build, and before installing, the libraries should be tested. Run:

$ make test                                      # Unix
$ mms test                                       ! OpenVMS
$ nmake test                                     # Windows

Warning: you MUST run the tests from an unprivileged account (or disable your privileges temporarily if your platform allows it).

If some tests fail, take a look at the section Test Failures below.

Test Failures

If some tests fail, look at the output. There may be reasons for the failure that isn't a problem in OpenSSL itself (like an OS malfunction or a Perl issue). You may want increased verbosity, that can be accomplished like this:

Full verbosity, showing full output of all successful and failed test cases (make macro VERBOSE or V):

$ make V=1 test                                  # Unix
$ mms /macro=(V=1) test                          ! OpenVMS
$ nmake V=1 test                                 # Windows

Verbosity on failed (sub-)tests only (VERBOSE_FAILURE or VF or REPORT_FAILURES):

$ make test VF=1

Verbosity on failed (sub-)tests, in addition progress on succeeded (sub-)tests (VERBOSE_FAILURE_PROGRESS or VFP or REPORT_FAILURES_PROGRESS):

$ make test VFP=1

If you want to run just one or a few specific tests, you can use the make variable TESTS to specify them, like this:

$ make TESTS='test_rsa test_dsa' test            # Unix
$ mms/macro="TESTS=test_rsa test_dsa" test       ! OpenVMS
$ nmake TESTS='test_rsa test_dsa' test           # Windows

And of course, you can combine (Unix examples shown):

$ make test TESTS='test_rsa test_dsa' VF=1
$ make test TESTS="test_cmp_*" VFP=1

You can find the list of available tests like this:

$ make list-tests                                # Unix
$ mms list-tests                                 ! OpenVMS
$ nmake list-tests                               # Windows

Have a look at the manual for the perl module Test::Harness to see what other HARNESS_* variables there are.

To report a bug please open an issue on GitHub, at https://github.com/openssl/openssl/issues.

For more details on how the make variables TESTS can be used, see section Running Selected Tests below.

Running Selected Tests

The make variable TESTS supports a versatile set of space separated tokens with which you can specify a set of tests to be performed. With a "current set of tests" in mind, initially being empty, here are the possible tokens:

 alltests      The current set of tests becomes the whole set of available
               tests (as listed when you do 'make list-tests' or similar).

 xxx           Adds the test 'xxx' to the current set of tests.

-xxx           Removes 'xxx' from the current set of tests.  If this is the
               first token in the list, the current set of tests is first
               assigned the whole set of available tests, effectively making
               this token equivalent to TESTS="alltests -xxx".

 nn            Adds the test group 'nn' (which is a number) to the current
               set of tests.

-nn            Removes the test group 'nn' from the current set of tests.
               If this is the first token in the list, the current set of
               tests is first assigned the whole set of available tests,
               effectively making this token equivalent to
               TESTS="alltests -xxx".

Also, all tokens except for "alltests" may have wildcards, such as *. (on Unix and Windows, BSD style wildcards are supported, while on VMS, it's VMS style wildcards)

Examples

Run all tests except for the fuzz tests:

$ make TESTS='-test_fuzz*' test

or, if you want to be explicit:

$ make TESTS='alltests -test_fuzz*' test

Run all tests that have a name starting with "test_ssl" but not those starting with "test_ssl_":

$ make TESTS='test_ssl* -test_ssl_*' test

Run only test group 10:

$ make TESTS='10' test

Run all tests except the slow group (group 99):

$ make TESTS='-99' test

Run all tests in test groups 80 to 99 except for tests in group 90:

$ make TESTS='[89]? -90' test

To run specific fuzz tests you can use for instance:

$ make test TESTS='test_fuzz_cmp test_fuzz_cms'

To stochastically verify that the algorithm that produces uniformly distributed random numbers is operating correctly (with a false positive rate of 0.01%):

$ ./util/wrap.sh test/bntest -stochastic

Running Tests in Parallel

By default the test harness will execute the selected tests sequentially. Depending on the platform characteristics, running more than one test job in parallel may speed up test execution. This can be requested by setting the HARNESS_JOBS environment variable to a positive integer value. This specifies the maximum number of test jobs to run in parallel.

Depending on the Perl version different strategies could be adopted to select which test recipes can be run in parallel. In recent versions of Perl, unless specified otherwise, any task can be run in parallel. Consult the documentation for TAP::Harness to know more.

To run up to four tests in parallel at any given time:

$ make HARNESS_JOBS=4 test

Randomisation of Test Ordering

By default, the test harness will execute tests in the order they were added. By setting the OPENSSL_TEST_RAND_ORDER environment variable to zero, the test ordering will be randomised. If a randomly ordered test fails, the seed value used will be reported. Setting the OPENSSL_TEST_RAND_ORDER environment variable to this value will rerun the tests in the same order. This assures repeatability of randomly ordered test runs. This repeatability is independent of the operating system, processor or platform used.

To randomise the test ordering:

$ make OPENSSL_TEST_RAND_ORDER=0 test

To run the tests using the order defined by the random seed 42:

$ make OPENSSL_TEST_RAND_ORDER=42 test