openssl/test
Matt Caswell d36a5dd05e Fix a copy&paste error in evp_extra_test
test_EC_priv_pub fails to test the case where both a private and public
key have been supplied.

Fixes #14349

Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14351)
2021-03-02 14:58:40 +00:00
..
certs check_sig_alg_match(): weaken sig nid comparison to allow RSA{,PSS} key verify RSA-PSS 2021-01-28 15:05:04 +01:00
ct Use .cnf for config files, not .conf 2020-03-06 18:25:13 +01:00
d2i-tests
helpers Add internal X509_add_certs_new(), which simplifies matters 2021-02-18 16:50:12 +01:00
ocsp-tests
ossl_shim test: updates for the new additional MAC_init arguments 2021-02-28 17:25:49 +10:00
recipes evp_test: updates for the new additional MAC_init arguments 2021-02-28 17:25:49 +10:00
smime-certs Update copyright year 2021-01-28 13:54:57 +01:00
ssl-tests Update copyright year 2021-02-18 15:05:17 +00:00
testutil test: add utility function to set the fake random callback on both the public and private instances 2021-03-02 13:23:17 +10:00
aborttest.c
acvp_test.c test: update tests to allow for params argument for the instantiate call on EVP_RAND_CTXs 2021-02-28 17:25:49 +10:00
acvp_test.inc Drop OPENSSL_NO_RSA everywhere 2020-12-20 12:19:42 +01:00
aesgcmtest.c Update copyright year 2021-01-28 13:54:57 +01:00
afalgtest.c Update copyright year 2020-07-16 14:47:04 +02:00
algorithmid_test.c TEST: Add an algorithm ID tester for libcrypto vs provider 2021-02-05 15:44:39 +01:00
asn1_decode_test.c Add a test for encoding/decoding using an invalid ASN.1 Template 2020-12-08 10:17:03 +00:00
asn1_dsa_internal_test.c Update copyright year 2020-10-15 14:10:06 +01:00
asn1_encode_test.c Add a test for encoding/decoding using an invalid ASN.1 Template 2020-12-08 10:17:03 +00:00
asn1_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
asn1_string_table_test.c
asn1_time_test.c Update copyright year 2020-04-23 13:55:52 +01:00
asynciotest.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
asynctest.c Rename OPENSSL_CTX prefix to OSSL_LIB_CTX 2020-10-15 11:59:53 +01:00
bad_dtls_test.c test: updates for the new additional MAC_init arguments 2021-02-28 17:25:49 +10:00
bftest.c Update copyright year 2020-04-23 13:55:52 +01:00
bio_callback_test.c
bio_enc_test.c
bio_memleak_test.c Update copyright year 2020-05-15 14:09:49 +01:00
bio_prefix_text.c TEST: Add test recipe and help program to test BIO_f_prefix() 2019-12-18 19:42:44 +01:00
bioprinttest.c TEST: Adjust test/bioprinttest.c to behave like the testutil routines 2020-06-06 19:18:30 +02:00
bn_internal_test.c Fix external symbols for bn 2021-02-26 10:32:22 +10:00
bn_rand_range.h Test of uniformity of BN_rand_range output. 2019-05-29 09:54:29 +10:00
bntest.c Update copyright year 2020-10-15 14:10:06 +01:00
bntests.pl
build.info Test errors from a provider can still be accessed after unload 2021-02-24 12:13:38 +00:00
ca-and-certs.cnf Cleanup cert config files for tests 2020-06-03 09:56:56 +02:00
casttest.c Update copyright year 2020-04-23 13:55:52 +01:00
CAtsa.cnf Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
chacha_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
cipher_overhead_test.c Update copyright year 2021-02-18 15:05:17 +00:00
cipherbytes_test.c Fix safestack issues in ssl.h 2020-09-13 11:09:45 +01:00
cipherlist_test.c Fix safestack issues in ssl.h 2020-09-13 11:09:45 +01:00
ciphername_test.c Fix safestack issues in ssl.h 2020-09-13 11:09:45 +01:00
clienthellotest.c Update copyright year 2020-04-23 13:55:52 +01:00
cmactest.c Add a CMAC test 2020-06-10 12:58:26 +01:00
cmp_asn_test.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
cmp_client_test.c Update copyright year 2021-01-28 13:54:57 +01:00
cmp_ctx_test.c Handle NULL result of ERR_reason_error_string() in some apps 2021-02-22 08:49:52 +01:00
cmp_hdr_test.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
cmp_msg_test.c Update copyright year 2021-01-28 13:54:57 +01:00
cmp_protect_test.c Update copyright year 2021-01-28 13:54:57 +01:00
cmp_server_test.c Use adapted test_get_libctx() for simpler test setup and better error reporting 2020-12-10 11:01:26 +01:00
cmp_status_test.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
cmp_vfy_test.c Update copyright year 2021-01-28 13:54:57 +01:00
cms-examples.pl
cmsapitest.c Update copyright year 2021-01-28 13:54:57 +01:00
conf_include_test.c Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
confdump.c Fix safestack issues in conf.h 2020-09-13 11:11:20 +01:00
constant_time_test.c Reorganize local header files 2019-09-28 20:26:35 +02:00
context_internal_test.c Update copyright year 2020-10-15 14:10:06 +01:00
crltest.c Fix safestack issues in x509.h 2020-09-13 11:09:45 +01:00
ct_test.c Fix safestack issues in ct.h 2020-09-13 11:10:41 +01:00
ctype_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
curve448_internal_test.c Fix ecx so that is uses a settable propertyquery 2020-09-23 17:31:40 +10:00
d2i_test.c Update copyright year 2020-04-23 13:55:52 +01:00
danetest.c Update copyright year 2021-02-18 15:05:17 +00:00
danetest.in
danetest.pem
data2.bin TEST: Make our test data binary 2020-11-24 11:01:58 +01:00
data.bin TEST: Make our test data binary 2020-11-24 11:01:58 +01:00
default-and-fips.cnf Update some nits around the FIPS module 2020-04-24 13:19:16 +02:00
default-and-legacy.cnf test/recipes/30-test_evp.t: Modify to test with different providers 2019-07-26 18:14:41 +02:00
default.cnf test/recipes/30-test_evp.t: Modify to test with different providers 2019-07-26 18:14:41 +02:00
defltfips_test.c Test that EVP_default_properties_is_fips_enabled() works early 2020-08-17 11:27:51 +01:00
destest.c Update copyright year 2020-04-23 13:55:52 +01:00
dhtest.c Update copyright year 2020-04-23 13:55:52 +01:00
drbgtest.c test: update tests to allow for params argument for the instantiate call on EVP_RAND_CTXs 2021-02-28 17:25:49 +10:00
dsa_no_digest_size_test.c Update copyright year 2020-04-23 13:55:52 +01:00
dsatest.c Swap to FIPS186-2 DSA generation outside of the FIPS module 2020-11-18 14:54:18 +00:00
dtls_mtu_test.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
dtlstest.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
dtlsv1listentest.c
ec_internal_test.c Fix external symbols related to ec & sm2 keys 2021-02-26 10:53:01 +10:00
ecdsatest.c test: update ECDSA and SM2 internal tests in line with the fake_random change 2021-03-02 13:23:17 +10:00
ecdsatest.h Fix header file include guard names 2019-09-28 20:26:36 +02:00
ecstresstest.c Remove tab characters from C source files. 2019-07-16 20:24:10 +10:00
ectest.c Simplify the EVP_PKEY_XXX_fromdata_XX methods. 2021-02-08 16:33:43 +10:00
endecode_test.c Update copyright year 2021-02-18 15:05:17 +00:00
endecoder_legacy_test.c Update copyright year 2021-02-18 15:05:17 +00:00
enginetest.c EVP: deprecate the EVP_X_meth_ functions. 2020-07-22 20:19:01 +10:00
errtest.c TEST: Adapt test/errtest for the 'no-err' configuration 2020-11-27 12:02:56 +01:00
evp_extra_test2.c Fix d2i_AutoPrivateKey_ex so that is uses the new decoder (and produces 2021-02-19 19:19:28 +10:00
evp_extra_test.c Fix a copy&paste error in evp_extra_test 2021-03-02 14:58:40 +00:00
evp_fetch_prov_test.c Rename OPENSSL_CTX prefix to OSSL_LIB_CTX 2020-10-15 11:59:53 +01:00
evp_kdf_test.c test: adjust tests to include extra argument to KDF derive call 2021-02-28 17:25:49 +10:00
evp_libctx_test.c Fix external symbols for bn 2021-02-26 10:32:22 +10:00
evp_pkey_dparams_test.c Add necessary checks of OPENSSL_NO_DH, OPENSSL_NO_DSA and OPENSSL_NO_EC 2020-12-16 11:56:38 +01:00
evp_pkey_provided_test.c evp_pkey_provided_test: Improve diagnostic output 2021-02-26 18:43:34 +01:00
evp_test.c test: update tests to allow for params argument for the instantiate call on EVP_RAND_CTXs 2021-02-28 17:25:49 +10:00
exdatatest.c Update copyright year 2020-06-04 14:33:57 +01:00
exptest.c
fatalerrtest.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
ffc_internal_test.c Fix external symbols related to dh keys 2021-02-26 10:50:30 +10:00
filterprov.c provider: add an unquery function to allow providers to clean up. 2021-02-24 21:24:36 +10:00
filterprov.h provider: add an unquery function to allow providers to clean up. 2021-02-24 21:24:36 +10:00
fips-and-base.cnf Add CLI tests in FIPS configuration 2020-08-24 14:49:03 +03:00
fips.cnf Test that EVP_default_properties_is_fips_enabled() works early 2020-08-17 11:27:51 +01:00
generate_buildtest.pl
generate_ssl_tests.pl Update copyright year 2020-04-23 13:55:52 +01:00
gmdifftest.c
gosttest.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
hexstr_test.c Change OPENSSL_hexstr2buf_ex() & OPENSSL_buf2hexstr_ex() to pass the separator 2020-12-07 17:14:58 +10:00
hmactest.c Update copyright year 2020-04-23 13:55:52 +01:00
http_test.c OSSL_HTTP_parse_url(): Handle any userinfo, query, and fragment components 2021-03-01 10:30:43 +01:00
ideatest.c Update copyright year 2020-04-23 13:55:52 +01:00
igetest.c Deprecate the AES_ige_*() functions 2019-12-04 17:46:38 +00:00
insta_ca.cert.pem Chunk 11 of CMP contribution to OpenSSL: CMP command-line interface 2020-05-13 19:42:00 +02:00
insta.priv.pem Chunk 11 of CMP contribution to OpenSSL: CMP command-line interface 2020-05-13 19:42:00 +02:00
keymgmt_internal_test.c Deprecate RSA harder 2020-11-18 23:38:34 +01:00
legacy.cnf test/recipes/30-test_evp.t: Modify to test with different providers 2019-07-26 18:14:41 +02:00
lhash_test.c Update copyright year 2020-10-15 14:10:06 +01:00
mdc2_internal_test.c Update copyright year 2020-04-23 13:55:52 +01:00
mdc2test.c Update copyright year 2020-04-23 13:55:52 +01:00
memleaktest.c test/memleaktest.c: Modify for use with address/leak sanitizer 2019-12-10 14:16:12 +01:00
modes_internal_test.c Update copyright year 2020-04-23 13:55:52 +01:00
moduleloadtest.c TEST: Add a simple module loader, and test the FIPS module with it 2020-12-01 11:06:03 +01:00
namemap_internal_test.c namemap: change ossl_namemap_empty() to do what the documentation says. 2020-06-21 16:49:51 +10:00
ocspapitest.c Fix safestack issues in x509.h 2020-09-13 11:09:45 +01:00
ossl_store_test.c STORE: clear err after ossl_store_get0_loader_int 2020-12-08 18:23:29 +01:00
p_test.c Fix compiling error on arm 2021-03-02 21:25:00 +10:00
packettest.c Make the PACKET/WPACKET code available to both libcrypto and libssl 2019-07-12 06:26:46 +10:00
param_build_test.c params: add OSSL_PARAM helpers for time_t. 2020-06-24 20:05:41 +10:00
params_api_test.c OSSL_PARAM: Correct the assumptions on the UTF8 string length 2021-02-18 16:58:17 +01:00
params_conversion_test.c Update copyright year 2020-04-23 13:55:52 +01:00
params_test.c OSSL_PARAM: Correct the assumptions on the UTF8 string length 2021-02-18 16:58:17 +01:00
pbelutest.c
pem_read_depr_test.c Test various deprecated PEM_read_bio_* APIs 2020-11-25 10:02:59 +00:00
pemtest.c
pkcs7-1.pem
pkcs7.pem
pkcs12_format_test.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
pkey_meth_kdf_test.c Update copyright year 2020-10-15 14:10:06 +01:00
pkey_meth_test.c Update copyright year 2020-08-06 13:22:30 +01:00
pkits-test.pl
poly1305_internal_test.c Add ChaCha related ciphers to default provider 2019-10-16 16:18:42 +10:00
property_test.c Rename OPENSSL_CTX prefix to OSSL_LIB_CTX 2020-10-15 11:59:53 +01:00
provider_fallback_test.c Rename OPENSSL_CTX prefix to OSSL_LIB_CTX 2020-10-15 11:59:53 +01:00
provider_internal_test.c provider: add option to load a provider without disabling the fallbacks. 2021-02-23 23:24:41 +10:00
provider_internal_test.cnf.in Use .cnf for config files, not .conf 2020-03-06 18:25:13 +01:00
provider_status_test.c Rename OPENSSL_CTX prefix to OSSL_LIB_CTX 2020-10-15 11:59:53 +01:00
provider_test.c Test errors from a provider can still be accessed after unload 2021-02-24 12:13:38 +00:00
proxy.cnf Cleanup cert config files for tests 2020-06-03 09:56:56 +02:00
rand_status_test.c rand: instantiate the DRBGs upon first use. 2020-08-28 10:19:56 +10:00
rc2test.c Update copyright year 2020-04-23 13:55:52 +01:00
rc4test.c Update copyright year 2020-04-23 13:55:52 +01:00
rc5test.c Update copyright year 2020-04-23 13:55:52 +01:00
rdrand_sanitytest.c Remove extern declarations of OPENSSL_ia32cap_P 2019-09-01 15:41:58 +02:00
README-dev.md Rename NOTES*, README*, VERSION, HACKING, LICENSE to .md or .txt 2020-07-05 11:29:43 +02:00
README-external.md Fix many MarkDown issues in {NOTES*,README*,HACKING,LICENSE}.md files 2020-07-05 11:29:43 +02:00
README.md test: document the random test ordering env variable 2020-12-15 20:03:07 +10:00
README.ssltest.md Fix issues reported by markdownlint 2020-05-08 16:22:02 +02:00
recordlentest.c Update copyright year 2021-02-18 15:05:17 +00:00
rsa_complex.c
rsa_mp_test.c Drop OPENSSL_NO_RSA everywhere 2020-12-20 12:19:42 +01:00
rsa_sp800_56b_test.c Drop OPENSSL_NO_RSA everywhere 2020-12-20 12:19:42 +01:00
rsa_test.c Remove RSA SSLv23 padding mode 2021-03-01 10:56:12 +01:00
run_tests.pl Update copyright year 2021-02-18 15:05:17 +00:00
sanitytest.c
secmemtest.c Update copyright year 2020-04-23 13:55:52 +01:00
serverinfo2.pem
serverinfo.pem
servername_test.c Update copyright year 2021-02-18 15:05:17 +00:00
session.pem
shibboleth.pfx
shlibloadtest.c TEST: Break out the local dynamic loading code from shlibloadtest.c 2020-12-01 11:06:03 +01:00
simpledynamic.c Fix simpledynamic.c - a typo and missed a header 2020-12-02 00:25:01 +11:00
simpledynamic.h Update copyright year 2021-01-28 13:54:57 +01:00
siphash_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
sm2_internal_test.c test: use the new set public and private together call 2021-03-02 13:23:17 +10:00
sm4_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
smcont_zero.txt Tests for processing zero-length content in SMIME format 2020-10-08 18:17:44 +03:00
smcont.txt
sparse_array_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
srptest.c Update copyright year 2021-02-18 15:05:17 +00:00
ssl_cert_table_internal_test.c Reorganize local header files 2019-09-28 20:26:35 +02:00
ssl_ctx_test.c Update copyright year 2020-05-15 14:09:49 +01:00
ssl_old_test.c Stop disabling TLSv1.3 if ec and dh are disabled 2021-02-05 15:22:40 +00:00
ssl_test_ctx_test.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
ssl_test_ctx_test.cnf Use .cnf for config files, not .conf 2020-03-06 18:25:13 +01:00
ssl_test.c Update copyright year 2021-02-18 15:05:17 +00:00
ssl_test.tmpl
sslapitest.c test: updates for the new additional MAC_init arguments 2021-02-28 17:25:49 +10:00
sslbuffertest.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
sslcorrupttest.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
stack_test.c Update copyright year 2021-02-18 15:05:17 +00:00
sysdefault.cnf
sysdefaulttest.c
test_test.c Fix --strict-warnings build 2019-08-18 21:45:16 +02:00
test.cnf Add -section option to 'req' command 2020-03-07 12:58:02 +03:00
testcrl.pem
testdsa.pem
testdsapub.pem
testec-p256.pem
testecpub-p256.pem
tested448.pem More testing for CLI usage of Ed25519 and Ed448 keys 2019-12-11 18:37:53 +01:00
tested448pub.pem More testing for CLI usage of Ed25519 and Ed448 keys 2019-12-11 18:37:53 +01:00
tested25519.pem More testing for CLI usage of Ed25519 and Ed448 keys 2019-12-11 18:37:53 +01:00
tested25519pub.pem More testing for CLI usage of Ed25519 and Ed448 keys 2019-12-11 18:37:53 +01:00
testp7.pem
testreq2.pem
testrsa2048.pem Add fips checks for rsa encryption 2020-09-18 14:20:38 +01:00
testrsa.pem Support writing RSA keys using the traditional format again 2020-09-09 18:32:10 +02:00
testrsapss.pem rsa_kmgmt: Return OSSL_PKEY_PARAM_DEFAULT_DIGEST for unrestricted PSS keys 2021-01-29 10:47:02 +01:00
testrsapssmandatory.pem RSA: properly generate algorithm identifier for RSA-PSS signatures 2021-02-05 14:04:59 +01:00
testrsapub.pem
testsid.pem
testutil.h test: add utility function to set the fake random callback on both the public and private instances 2021-03-02 13:23:17 +10:00
testx509.pem Extend X509 cert checks and error reporting in v3_{purp,crld}.c and x509_{set,vfy}.c 2020-09-11 07:42:22 +02:00
threadstest.c Add a multi-thread test for shared EVP_PKEYs 2021-02-02 12:21:21 +00:00
time_offset_test.c
tls13ccstest.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
tls13encryptiontest.c Move MAC removal responsibility to the various protocol "enc" functions 2020-07-06 09:26:00 +01:00
tls13secretstest.c Update copyright year 2021-01-28 13:54:57 +01:00
tls-provider.c Update copyright year 2021-02-18 15:05:17 +00:00
uitest.c Update copyright year 2020-10-15 14:10:06 +01:00
v3_ca_exts.cnf make various test CA certs RFC 5280 compliant w.r.t. X509 extensions 2021-01-20 15:53:47 +01:00
v3-cert1.pem
v3-cert2.pem
v3ext.c Update copyright year 2020-04-23 13:55:52 +01:00
v3nametest.c v3nametest: Make the gennames structure static 2020-12-09 13:41:30 +01:00
verify_extra_test.c Update copyright year 2021-01-28 13:54:57 +01:00
versions.c
wpackettest.c WPACKET: don't write DER length when we don't want to 2020-05-04 05:50:06 +02:00
x509_check_cert_pkey_test.c Update copyright year 2020-04-23 13:55:52 +01:00
x509_dup_cert_test.c Update copyright year 2020-04-23 13:55:52 +01:00
x509_internal_test.c Join the x509 and x509v3 directories 2019-05-29 09:32:50 +02:00
x509_time_test.c Update copyright year 2020-04-23 13:55:52 +01:00
x509aux.c Update copyright year 2021-01-28 13:54:57 +01:00

Using OpenSSL Tests

After a successful build, and before installing, the libraries should be tested. Run:

$ make test                                      # Unix
$ mms test                                       ! OpenVMS
$ nmake test                                     # Windows

Warning: you MUST run the tests from an unprivileged account (or disable your privileges temporarily if your platform allows it).

If some tests fail, take a look at the section Test Failures below.

Test Failures

If some tests fail, look at the output. There may be reasons for the failure that isn't a problem in OpenSSL itself (like an OS malfunction or a Perl issue). You may want increased verbosity, that can be accomplished like this:

Full verbosity, showing full output of all successful and failed test cases (make macro VERBOSE or V):

$ make V=1 test                                  # Unix
$ mms /macro=(V=1) test                          ! OpenVMS
$ nmake V=1 test                                 # Windows

Verbosity on failed (sub-)tests only (VERBOSE_FAILURE or VF or REPORT_FAILURES):

$ make test VF=1

Verbosity on failed (sub-)tests, in addition progress on succeeded (sub-)tests (VERBOSE_FAILURE_PROGRESS or VFP or REPORT_FAILURES_PROGRESS):

$ make test VFP=1

If you want to run just one or a few specific tests, you can use the make variable TESTS to specify them, like this:

$ make TESTS='test_rsa test_dsa' test            # Unix
$ mms/macro="TESTS=test_rsa test_dsa" test       ! OpenVMS
$ nmake TESTS='test_rsa test_dsa' test           # Windows

And of course, you can combine (Unix examples shown):

$ make test TESTS='test_rsa test_dsa' VF=1
$ make test TESTS="test_cmp_*" VFP=1

You can find the list of available tests like this:

$ make list-tests                                # Unix
$ mms list-tests                                 ! OpenVMS
$ nmake list-tests                               # Windows

Have a look at the manual for the perl module Test::Harness to see what other HARNESS_* variables there are.

To report a bug please open an issue on GitHub, at https://github.com/openssl/openssl/issues.

For more details on how the make variables TESTS can be used, see section Running Selected Tests below.

Running Selected Tests

The make variable TESTS supports a versatile set of space separated tokens with which you can specify a set of tests to be performed. With a "current set of tests" in mind, initially being empty, here are the possible tokens:

 alltests      The current set of tests becomes the whole set of available
               tests (as listed when you do 'make list-tests' or similar).

 xxx           Adds the test 'xxx' to the current set of tests.

-xxx           Removes 'xxx' from the current set of tests.  If this is the
               first token in the list, the current set of tests is first
               assigned the whole set of available tests, effectively making
               this token equivalent to TESTS="alltests -xxx".

 nn            Adds the test group 'nn' (which is a number) to the current
               set of tests.

-nn            Removes the test group 'nn' from the current set of tests.
               If this is the first token in the list, the current set of
               tests is first assigned the whole set of available tests,
               effectively making this token equivalent to
               TESTS="alltests -xxx".

Also, all tokens except for "alltests" may have wildcards, such as *. (on Unix and Windows, BSD style wildcards are supported, while on VMS, it's VMS style wildcards)

Examples

Run all tests except for the fuzz tests:

$ make TESTS='-test_fuzz*' test

or, if you want to be explicit:

$ make TESTS='alltests -test_fuzz*' test

Run all tests that have a name starting with "test_ssl" but not those starting with "test_ssl_":

$ make TESTS='test_ssl* -test_ssl_*' test

Run only test group 10:

$ make TESTS='10' test

Run all tests except the slow group (group 99):

$ make TESTS='-99' test

Run all tests in test groups 80 to 99 except for tests in group 90:

$ make TESTS='[89]? -90' test

To run specific fuzz tests you can use for instance:

$ make test TESTS='test_fuzz_cmp test_fuzz_cms'

To stochastically verify that the algorithm that produces uniformly distributed random numbers is operating correctly (with a false positive rate of 0.01%):

$ ./util/wrap.sh test/bntest -stochastic

Running Tests in Parallel

By default the test harness will execute the selected tests sequentially. Depending on the platform characteristics, running more than one test job in parallel may speed up test execution. This can be requested by setting the HARNESS_JOBS environment variable to a positive integer value. This specifies the maximum number of test jobs to run in parallel.

Depending on the Perl version different strategies could be adopted to select which test recipes can be run in parallel. In recent versions of Perl, unless specified otherwise, any task can be run in parallel. Consult the documentation for TAP::Harness to know more.

To run up to four tests in parallel at any given time:

$ make HARNESS_JOBS=4 test

Randomisation of Test Ordering

By default, the test harness will execute tests in the order they were added. By setting the OPENSSL_TEST_RAND_ORDER environment variable to zero, the test ordering will be randomised. If a randomly ordered test fails, the seed value used will be reported. Setting the OPENSSL_TEST_RAND_ORDER environment variable to this value will rerun the tests in the same order. This assures repeatability of randomly ordered test runs. This repeatability is independent of the operating system, processor or platform used.

To randomise the test ordering:

$ make OPENSSL_TEST_RAND_ORDER=0 test

To run the tests using the order defined by the random seed 42:

$ make OPENSSL_TEST_RAND_ORDER=42 test