openssl/test
Tomas Mraz e64a169fc6 Add testcase for OSSL_trace_set_callback()
Also test the OSSL_TRACE_CATEGORY_TRACE tracing - this fails
on address sanitizer runs without the fix for #19915

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/19917)
2022-12-22 11:33:48 +01:00
..
certs test: add test case for deadlock reported in #19643 2022-12-08 11:11:11 +01:00
ct Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
d2i-tests
helpers stack: Do not add error if pop/shift/value accesses outside of the stack 2022-10-21 18:02:35 +02:00
ocsp-tests
recipes Add a CMS test for a bad encryption algorithm 2022-12-22 11:02:07 +01:00
smime-certs signature: Clamp PSS salt len to MD len 2022-12-08 11:02:52 +01:00
smime-eml test: add test case for deadlock reported in #19643 2022-12-08 11:11:11 +01:00
ssl-tests Add support for compressed certificates (RFC8879) 2022-10-18 09:30:22 -04:00
testutil Cleanup : directly include of internal/nelem.h when required. 2022-11-23 18:08:25 +01:00
aborttest.c
acvp_test.c Fix the checks of EVP_PKEY_CTX_set/get_* functions 2022-06-02 11:06:41 +02:00
acvp_test.inc
aesgcmtest.c Remove FIPS condition on IV gen test. 2022-09-21 17:02:59 +10:00
afalgtest.c Fix a memory leak in the afalg engine 2021-10-06 15:09:51 +02:00
algorithmid_test.c Fix the return check of OBJ_obj2txt 2021-11-22 11:17:48 +01:00
asn1_decode_test.c
asn1_dsa_internal_test.c
asn1_encode_test.c Update copyright year 2021-07-29 15:41:35 +01:00
asn1_internal_test.c Allow OBJ_create() to create an OBJ and NID with a NULL OID 2022-12-13 15:40:16 +01:00
asn1_string_table_test.c
asn1_time_test.c Cast values to match printf format strings. 2022-11-14 07:47:53 +00:00
asynciotest.c
asynctest.c test: Fix memory leak of asynctest 2022-09-23 14:30:09 +01:00
bad_dtls_test.c Fix the checks of RAND_bytes 2022-06-02 10:36:56 -04:00
bftest.c fix some code with obvious wrong coding style 2021-10-28 13:10:46 +10:00
bio_callback_test.c
bio_comp_test.c Fix a compilation failure in bio_comp_test.c 2022-10-24 14:15:15 +01:00
bio_core_test.c
bio_dgram_test.c test: condition out code that relies on CHACHA. 2022-10-10 14:16:08 +01:00
bio_enc_test.c Update copyright year 2022-05-03 13:34:51 +01:00
bio_memleak_test.c Fix SMIME_crlf_copy() to properly report an error 2022-12-22 11:01:06 +01:00
bio_prefix_text.c BIO_set_indent: fix return check 2021-11-22 14:43:44 +01:00
bio_readbuffer_test.c
bio_tfo_test.c Fix asan finding in bio_tfo_test 2022-08-12 16:13:13 +01:00
bioprinttest.c
bn_internal_test.c
bn_rand_range.h
bntest.c Improve FIPS RSA keygen performance. 2022-11-21 11:17:59 +01:00
bntests.pl
build_wincrypt_test.c Move types.h #undefs for wincrypt.h compatibility 2022-05-30 07:19:14 +02:00
build.info Implement OSSL_PROVIDER_get0_default_search_path, add docs and tests. 2022-12-06 18:24:06 +01:00
ca_internals_test.c libcrypto and test: rename asn1_string_to_time_t to ossl_asn1_string_to_time_t 2022-07-19 08:44:19 +02:00
ca-and-certs.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
casttest.c
CAtsa.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
cert_comp_test.c Fix coverity 1516095 deadcode 2022-10-21 10:34:55 +11:00
chacha_internal_test.c
cipher_overhead_test.c
cipherbytes_test.c
cipherlist_test.c Allow cipher strings to be given using its standard name 2021-09-30 12:20:01 +02:00
ciphername_test.c
clienthellotest.c fix some code with obvious wrong coding style 2021-10-28 13:10:46 +10:00
cmactest.c
cmp_asn_test.c CMP+CRMF: fix formatting nits in crypto/, include/, and test/ 2022-11-24 13:45:06 +01:00
cmp_client_test.c cmp_client_test.c: add tests for OSSL_CMP_CTX_get_status 2022-11-30 15:58:15 +01:00
cmp_ctx_test.c CMP: add API functions OSSL_CMP_CTX_get0_libctx() and OSSL_CMP_CTX_get0_propq() 2022-11-23 10:57:52 +01:00
cmp_hdr_test.c CMP+CRMF: fix formatting nits in crypto/, include/, and test/ 2022-11-24 13:45:06 +01:00
cmp_msg_test.c CMP+CRMF: fix formatting nits in crypto/, include/, and test/ 2022-11-24 13:45:06 +01:00
cmp_protect_test.c CMP+CRMF: fix formatting nits in crypto/, include/, and test/ 2022-11-24 13:45:06 +01:00
cmp_server_test.c CMP: Add missing getter functions to CRMF API and CMP API 2021-06-30 10:38:23 +02:00
cmp_status_test.c CMP+CRMF: fix formatting nits in crypto/, include/, and test/ 2022-11-24 13:45:06 +01:00
cmp_vfy_test.c test: cmp_vfy_test: fix defined but unused 2022-12-14 09:21:19 +00:00
cms-examples.pl
cmsapitest.c tests: clear error queue before executing a testcase 2022-10-27 18:39:29 +02:00
conf_include_test.c
confdump.c
constant_time_test.c fix some code with obvious wrong coding style 2021-10-28 13:10:46 +10:00
context_internal_test.c Update copyright year 2022-05-03 13:34:51 +01:00
crltest.c Update copyright year 2022-05-03 13:34:51 +01:00
ct_test.c Update copyright year 2022-05-03 13:34:51 +01:00
ctype_internal_test.c
curve448_internal_test.c
d2i_test.c
dane-cross.in Apply the correct Apache v2 license 2022-02-14 10:08:21 +01:00
danetest.c add OSSL_STACK_OF_X509_free() for commonly used pattern 2021-12-21 12:11:49 +01:00
danetest.in
danetest.pem
data2.bin
data.bin
default-and-fips.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
default-and-legacy.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
default.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
defltfips_test.c Fix copyrights 2022-02-03 13:56:38 +01:00
destest.c Add unit tests for weak key and key parity checks 2021-11-05 09:25:28 +10:00
dhkem_test.inc Add HPKE DHKEM provider support for EC, X25519 and X448. 2022-09-23 09:24:47 +01:00
dhtest.c Update copyright year 2022-05-03 13:34:51 +01:00
drbgtest.c Workaround egd rand source deficiencies 2022-10-24 12:02:16 +02:00
dsa_no_digest_size_test.c
dsatest.c dsatest: Properly detect failure in generate/sign/verify 2021-08-11 12:07:08 +02:00
dtls_mtu_test.c SSL object refactoring using SSL_CONNECTION object 2022-07-28 10:04:28 +01:00
dtlstest.c Add additional messages to the DTLS dropped records test 2022-09-22 12:22:10 +01:00
dtlsv1listentest.c
ec_internal_test.c Add test for EC_KEY_set_private_key() 2022-08-04 12:17:08 +03:00
ecdsatest.c Fix the checks of RAND_bytes 2022-06-02 10:36:56 -04:00
ecdsatest.h
ecstresstest.c
ectest.c Potential null pointer reference 2022-11-02 10:58:20 +01:00
endecode_test.c With fips provider 3.0.0 skip tests related to explicit curves handling 2022-09-16 08:34:53 +10:00
endecoder_legacy_test.c test: fix coverity 1469427 Improper use of negative value (NEGATIVE_RETURNS) 2021-06-30 13:55:09 +10:00
enginetest.c Update copyright year 2022-05-03 13:34:51 +01:00
errtest.c
event_queue_test.c Make OSSL_TIME a structure 2022-08-12 15:44:01 +01:00
evp_extra_test2.c Add test for EVP_PKEY_Q_keygen 2022-11-25 18:38:52 +01:00
evp_extra_test.c Fix the check of EVP_PKEY_decrypt_init 2022-12-05 13:04:18 +01:00
evp_fetch_prov_test.c Update copyright year 2022-05-03 13:34:51 +01:00
evp_kdf_test.c Add KMAC support to KBKDF. 2022-12-16 16:36:49 +01:00
evp_libctx_test.c Revert "Remove conditional FIPS dependence for 3DES" 2022-11-10 12:25:04 +01:00
evp_pkey_ctx_new_from_name.c Testing the EVP_PKEY_CTX_new_from_name without preliminary init 2022-04-29 14:13:02 +02:00
evp_pkey_dhkem_test.c Add HPKE DHKEM provider support for EC, X25519 and X448. 2022-09-23 09:24:47 +01:00
evp_pkey_dparams_test.c Update copyright year 2022-05-03 13:34:51 +01:00
evp_pkey_provided_test.c Honor OSSL_PKEY_PARAM_EC_POINT_CONVERSION_FORMAT as set and default to UNCOMPRESSED 2022-11-29 16:03:04 +01:00
evp_test.c Drop a spurious printf in evp_test.c 2022-12-05 10:58:20 +00:00
exdatatest.c
exptest.c Fix a logic flaw in test_mod_exp_zero 2022-12-21 16:31:15 +01:00
ext_internal_test.c Add support for compressed certificates (RFC8879) 2022-10-18 09:30:22 -04:00
fake_rsaprov.c Add test for EVP_PKEY_eq 2022-11-15 12:04:12 +01:00
fake_rsaprov.h Add test for EVP_PKEY_eq 2022-11-15 12:04:12 +01:00
fatalerrtest.c
ffc_internal_test.c ossl_ffc_params_copy: Copy the keylength too 2022-07-18 08:06:17 +01:00
filterprov.c
filterprov.h
fips_version_test.c With fips provider 3.0.0 skip tests related to explicit curves handling 2022-09-16 08:34:53 +10:00
fips-alt.cnf Add some testing for the case where the FIPS provider fails to load 2021-07-28 10:35:06 +10:00
fips-and-base.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
fips.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
generate_buildtest.pl
generate_ssl_tests.pl
gmdifftest.c
hexstr_test.c
hmactest.c Add test case to verify that the use after free issue is fixed. 2021-12-17 14:39:20 +11:00
hpke_test.c prevent HPKE sender setting seq unwisely 2022-12-08 10:59:03 +01:00
http_test.c http_test.c: Simplify constant init of 'server_args' struct for gcc-4.8.x 2021-12-21 12:18:04 +01:00
ideatest.c
igetest.c
insta_ca.cert.pem Remove executable mode attributes of non-executable files 2021-07-13 16:04:32 +10:00
insta.priv.pem Remove executable mode attributes of non-executable files 2021-07-13 16:04:32 +10:00
keymgmt_internal_test.c Replace "a RSA" with "an RSA" 2022-12-07 09:37:25 +11:00
legacy.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
lhash_test.c Add deprecation macro for 3.1 and deprecate OPENSSL_LH_stats 2022-06-22 09:36:14 +02:00
list_test.c list: add debug sanity checks 2022-11-16 18:02:02 +11:00
localetest.c Include the e_os.h before string.h 2022-05-13 08:30:41 +02:00
mdc2_internal_test.c
mdc2test.c Cleanup : directly include of internal/nelem.h when required. 2022-11-23 18:08:25 +01:00
membio_test.c Add a test for BIO_s_mem() when using datagrams 2022-07-28 08:06:52 +01:00
memleaktest.c
modes_internal_test.c fix some code with obvious wrong coding style 2021-10-28 13:10:46 +10:00
moduleloadtest.c
namemap_internal_test.c
ocspapitest.c ocspapitest: use TEST_true to report the exact failure 2022-07-14 14:10:30 +02:00
ossl_store_test.c fix some code with obvious wrong coding style 2021-10-28 13:10:46 +10:00
p_test.c Update copyright year 2022-05-03 13:34:51 +01:00
packettest.c Fix PACKET_equal test with BUF_LEN+1 on -Wstringop-overread 2022-11-07 19:21:56 +01:00
param_build_test.c Update copyright year 2022-05-03 13:34:51 +01:00
params_api_test.c Update copyright year 2022-05-03 13:34:51 +01:00
params_conversion_test.c Update copyright year 2022-05-03 13:34:51 +01:00
params_test.c Update copyright year 2022-05-03 13:34:51 +01:00
pbelutest.c
pbetest.c Fix copyrights 2022-02-03 13:56:38 +01:00
pem_read_depr_test.c Update copyright year 2021-07-29 15:41:35 +01:00
pemtest.c Update copyright year 2021-07-29 15:41:35 +01:00
pkcs7_test.c
pkcs7-1.pem
pkcs7.pem
pkcs12_api_test.c Allow PKCS12 export to set arbitrary bag attributes 2022-09-23 17:40:02 +01:00
pkcs12_format_test.c Cleanup : directly include of internal/nelem.h when required. 2022-11-23 18:08:25 +01:00
pkey_meth_kdf_test.c
pkey_meth_test.c
pkits-test.pl
poly1305_internal_test.c
priority_queue_test.c test: add priority queue unit test 2022-06-22 13:05:40 +10:00
property_test.c Fix occasional assertion failure when storing properties 2022-11-29 08:21:34 +01:00
prov_config_test.c Add a test for running the config twice 2021-08-27 09:52:19 +10:00
provfetchtest.c Update copyright year 2022-05-03 13:34:51 +01:00
provider_default_search_path_test.c Implement OSSL_PROVIDER_get0_default_search_path, add docs and tests. 2022-12-06 18:24:06 +01:00
provider_fallback_test.c
provider_internal_test.c Don't attempt to deactive child providers if we don't need to 2021-11-12 17:16:14 +00:00
provider_internal_test.cnf.in Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
provider_pkey_test.c Add test for EVP_PKEY_eq 2022-11-15 12:04:12 +01:00
provider_status_test.c Add test for provider gettables 2021-07-06 10:55:19 +10:00
provider_test.c Update copyright year 2022-05-03 13:34:51 +01:00
proxy.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
punycode_test.c punycode: update to use WPACKET instead of using custom range checking 2022-11-11 08:14:47 +11:00
quic_ackm_test.c Updates for OSSL_TIME changes 2022-08-24 14:05:46 +01:00
quic_cfq_test.c QUIC CFQ Fixes 2022-11-07 18:18:34 +00:00
quic_fc_test.c QUIC Flow Control 2022-09-26 08:01:55 +01:00
quic_fifd_test.c QUIC TX Packetiser and Streams Mapper 2022-11-24 08:15:20 +00:00
quic_record_test_util.h QUIC TX Packetiser and Streams Mapper 2022-11-24 08:15:20 +00:00
quic_record_test.c QUIC TX Packetiser and Streams Mapper 2022-11-24 08:15:20 +00:00
quic_stream_test.c QUIC TX Packetiser and Streams Mapper 2022-11-24 08:15:20 +00:00
quic_txp_test.c QUIC TX Packetiser and Streams Mapper 2022-11-24 08:15:20 +00:00
quic_txpim_test.c QUIC TXPIM 2022-11-07 18:18:04 +00:00
quic_wire_test.c QUIC TX Packetiser and Streams Mapper 2022-11-24 08:15:20 +00:00
quicapitest.c Add a DTLSv1_listen() test 2022-08-18 16:38:12 +01:00
rand_status_test.c
rand_test.c Remove redundant RAND_get0_private() call 2021-10-28 18:05:09 +10:00
rc2test.c
rc4test.c
rc5test.c
rdcpu_sanitytest.c Update copyright year 2022-05-03 13:34:51 +01:00
README-dev.md always use the same perl in $PATH 2021-09-02 12:55:39 +10:00
README-external.md Add external testing with oqsprovider 2022-03-09 17:57:37 +01:00
README.md
README.ssltest.md Make running individual ssl-test easier 2022-05-27 14:17:29 -04:00
recordlentest.c tests: clear error queue before executing a testcase 2022-10-27 18:39:29 +02:00
rsa_complex.c djgpp: Skip test/rsa_complex.c 2022-09-29 12:48:23 +02:00
rsa_mp_test.c
rsa_sp800_56b_test.c Update copyright year 2021-09-07 13:35:43 +02:00
rsa_test.c Fix use of uninitialized memory in test_rsa_oaep 2021-07-01 09:57:05 +10:00
run_tests.pl Update copyright year 2022-05-03 13:34:51 +01:00
safe_math_test.c Update copyright year 2022-05-03 13:34:51 +01:00
sanitytest.c Update copyright year 2022-05-03 13:34:51 +01:00
secmemtest.c Update copyright year 2022-05-03 13:34:51 +01:00
serverinfo2.pem
serverinfo.pem
servername_test.c
session.pem
sha_test.c
shibboleth.pfx
shlibloadtest.c
simpledynamic.c Update copyright year 2021-07-29 15:41:35 +01:00
simpledynamic.h
siphash_internal_test.c Update copyright year 2022-05-03 13:34:51 +01:00
sm2_internal_test.c feat: Add sm2 signature test case from GM/T 0003.5-2012 2021-10-12 18:09:14 +02:00
sm3_internal_test.c Apply the correct Apache v2 license 2022-02-14 10:08:21 +01:00
sm4_internal_test.c
smcont_zero.txt
smcont.bin
smcont.txt
sparse_array_test.c
srptest.c
ssl_cert_table_internal_test.c Cast values to match printf format strings. 2022-11-14 07:47:53 +00:00
ssl_ctx_test.c
ssl_old_test.c Update COMP_METHOD 2022-10-18 09:30:22 -04:00
ssl_test_ctx_test.c
ssl_test_ctx_test.cnf
ssl_test.c Add a test_ssl_new testcase 2022-06-03 12:07:18 +10:00
ssl_test.tmpl
sslapitest.c Fix no-ec enable-ktls build 2022-12-08 16:25:04 +01:00
sslbuffertest.c Check whether buffers have actually been allocated/freed 2022-10-27 10:52:52 +01:00
sslcorrupttest.c tests: clear error queue before executing a testcase 2022-10-27 18:39:29 +02:00
stack_test.c fix some code with obvious wrong coding style 2021-10-28 13:10:46 +10:00
sysdefault.cnf Add oid_section to sysdefault.cnf to test adding new oids 2021-08-05 09:21:00 +10:00
sysdefaulttest.c
test_test.c test: placate Clang's --Wbitwise-instead-of-logical 2022-06-27 11:43:47 +10:00
test.cnf
testcrl.pem
testdsa.pem
testdsapub.pem
testec-p112r1.pem TEST: Check property query support of apps/pkey 2022-08-17 09:20:41 +02:00
testec-p256.pem
testecpub-p256.pem
tested448.pem
tested448pub.pem
tested25519.pem
tested25519pub.pem
testp7.pem
testreq2.pem
testrsa2048.pem
testrsa2048pub.pem Test that signatures using hash name commands work properly 2022-11-07 14:40:09 +01:00
testrsa_withattrs.der
testrsa_withattrs.pem
testrsa.pem
testrsapss.pem
testrsapssmandatory.pem
testrsapub.pem
testsid.pem
testutil.h test: add two comparision options to fips version test utility code 2022-11-15 12:10:39 +01:00
testx509.pem
threadpool_test.c Split out thread pool tests into threadpool_test 2022-11-22 17:08:23 +01:00
threadstest_fips.c
threadstest.c Split out thread pool tests into threadpool_test 2022-11-22 17:08:23 +01:00
threadstest.h
time_offset_test.c
timing_load_creds.c timing_load_creds requires POSIX1.2001 due to rusage 2022-12-15 12:04:01 +01:00
tls13ccstest.c fix some code with obvious wrong coding style 2021-10-28 13:10:46 +10:00
tls13encryptiontest.c Rename SSL3_RECORD to TLS_RL_RECORD 2022-11-14 07:51:26 +00:00
tls13secretstest.c Remove some redundant code 2022-10-20 14:39:33 +01:00
tls-provider.c Update copyright year 2022-05-03 13:34:51 +01:00
trace_api_test.c Add testcase for OSSL_trace_set_callback() 2022-12-22 11:33:48 +01:00
uitest.c Fix the checks of UI_add_input_string 2022-06-02 10:36:56 -04:00
upcallstest.c Update copyright year 2022-05-03 13:34:51 +01:00
user_property_test.c
v3_ca_exts.cnf
v3-cert1.pem
v3-cert2.pem
v3ext.c Fix coverity issues in X509v3_addr 2022-11-21 12:41:25 +01:00
v3nametest.c test/v3nametest.c: Add check for OPENSSL_malloc 2022-06-22 17:05:48 +10:00
verify_extra_test.c Update copyright year 2022-05-03 13:34:51 +01:00
versions.c
wpackettest.c Fix the checks of RAND_bytes 2022-06-02 10:36:56 -04:00
x509_check_cert_pkey_test.c
x509_dup_cert_test.c Remove unused variable 'sctx' 2021-10-27 11:05:35 +02:00
x509_internal_test.c Update copyright year 2021-09-07 13:35:43 +02:00
x509_test.c Add test for X509 sign TBS cache regression. 2022-11-02 11:14:32 +01:00
x509_time_test.c apps & al : Fix various typos, repeated words, align some spelling to LDP. 2022-10-12 16:55:28 +11:00
x509aux.c Cleanup : directly include of internal/nelem.h when required. 2022-11-23 18:08:25 +01:00

Using OpenSSL Tests

After a successful build, and before installing, the libraries should be tested. Run:

$ make test                                      # Unix
$ mms test                                       ! OpenVMS
$ nmake test                                     # Windows

Warning: you MUST run the tests from an unprivileged account (or disable your privileges temporarily if your platform allows it).

If some tests fail, take a look at the section Test Failures below.

Test Failures

If some tests fail, look at the output. There may be reasons for the failure that isn't a problem in OpenSSL itself (like an OS malfunction or a Perl issue). You may want increased verbosity, that can be accomplished like this:

Full verbosity, showing full output of all successful and failed test cases (make macro VERBOSE or V):

$ make V=1 test                                  # Unix
$ mms /macro=(V=1) test                          ! OpenVMS
$ nmake V=1 test                                 # Windows

Verbosity on failed (sub-)tests only (VERBOSE_FAILURE or VF or REPORT_FAILURES):

$ make test VF=1

Verbosity on failed (sub-)tests, in addition progress on succeeded (sub-)tests (VERBOSE_FAILURE_PROGRESS or VFP or REPORT_FAILURES_PROGRESS):

$ make test VFP=1

If you want to run just one or a few specific tests, you can use the make variable TESTS to specify them, like this:

$ make TESTS='test_rsa test_dsa' test            # Unix
$ mms/macro="TESTS=test_rsa test_dsa" test       ! OpenVMS
$ nmake TESTS='test_rsa test_dsa' test           # Windows

And of course, you can combine (Unix examples shown):

$ make test TESTS='test_rsa test_dsa' VF=1
$ make test TESTS="test_cmp_*" VFP=1

You can find the list of available tests like this:

$ make list-tests                                # Unix
$ mms list-tests                                 ! OpenVMS
$ nmake list-tests                               # Windows

Have a look at the manual for the perl module Test::Harness to see what other HARNESS_* variables there are.

To report a bug please open an issue on GitHub, at https://github.com/openssl/openssl/issues.

For more details on how the make variables TESTS can be used, see section Running Selected Tests below.

Running Selected Tests

The make variable TESTS supports a versatile set of space separated tokens with which you can specify a set of tests to be performed. With a "current set of tests" in mind, initially being empty, here are the possible tokens:

 alltests      The current set of tests becomes the whole set of available
               tests (as listed when you do 'make list-tests' or similar).

 xxx           Adds the test 'xxx' to the current set of tests.

-xxx           Removes 'xxx' from the current set of tests.  If this is the
               first token in the list, the current set of tests is first
               assigned the whole set of available tests, effectively making
               this token equivalent to TESTS="alltests -xxx".

 nn            Adds the test group 'nn' (which is a number) to the current
               set of tests.

-nn            Removes the test group 'nn' from the current set of tests.
               If this is the first token in the list, the current set of
               tests is first assigned the whole set of available tests,
               effectively making this token equivalent to
               TESTS="alltests -xxx".

Also, all tokens except for "alltests" may have wildcards, such as *. (on Unix and Windows, BSD style wildcards are supported, while on VMS, it's VMS style wildcards)

Examples

Run all tests except for the fuzz tests:

$ make TESTS='-test_fuzz*' test

or, if you want to be explicit:

$ make TESTS='alltests -test_fuzz*' test

Run all tests that have a name starting with "test_ssl" but not those starting with "test_ssl_":

$ make TESTS='test_ssl* -test_ssl_*' test

Run only test group 10:

$ make TESTS='10' test

Run all tests except the slow group (group 99):

$ make TESTS='-99' test

Run all tests in test groups 80 to 99 except for tests in group 90:

$ make TESTS='[89]? -90' test

To run specific fuzz tests you can use for instance:

$ make test TESTS='test_fuzz_cmp test_fuzz_cms'

To stochastically verify that the algorithm that produces uniformly distributed random numbers is operating correctly (with a false positive rate of 0.01%):

$ ./util/wrap.sh test/bntest -stochastic

Running Tests in Parallel

By default the test harness will execute the selected tests sequentially. Depending on the platform characteristics, running more than one test job in parallel may speed up test execution. This can be requested by setting the HARNESS_JOBS environment variable to a positive integer value. This specifies the maximum number of test jobs to run in parallel.

Depending on the Perl version different strategies could be adopted to select which test recipes can be run in parallel. In recent versions of Perl, unless specified otherwise, any task can be run in parallel. Consult the documentation for TAP::Harness to know more.

To run up to four tests in parallel at any given time:

$ make HARNESS_JOBS=4 test

Randomisation of Test Ordering

By default, the test harness will execute tests in the order they were added. By setting the OPENSSL_TEST_RAND_ORDER environment variable to zero, the test ordering will be randomised. If a randomly ordered test fails, the seed value used will be reported. Setting the OPENSSL_TEST_RAND_ORDER environment variable to this value will rerun the tests in the same order. This assures repeatability of randomly ordered test runs. This repeatability is independent of the operating system, processor or platform used.

To randomise the test ordering:

$ make OPENSSL_TEST_RAND_ORDER=0 test

To run the tests using the order defined by the random seed 42:

$ make OPENSSL_TEST_RAND_ORDER=42 test