openssl/test
Tomas Mraz 80b9eca279 Add test for handling NEW_CONNECTION_ID frame
Reviewed-by: Hugo Landau <hlandau@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/20892)
2023-05-17 14:04:18 +01:00
..
certs Generate some certificates with the certificatePolicies extension 2023-03-28 13:31:17 +02:00
ct
d2i-tests
helpers QUIC CHANNEL: Do not copy terminate cause as it is not modified after termination 2023-05-12 14:47:12 +01:00
ocsp-tests
recipes Add test for handling NEW_CONNECTION_ID frame 2023-05-17 14:04:18 +01:00
smime-certs apps/x509 etc.: allow private key input when public key is expected 2023-03-14 17:26:49 +01:00
smime-eml
ssl-tests test: update ssl_new tests in line with pedantic FIPS policy 2023-04-21 17:01:38 +01:00
testutil Replaced '{ 0, NULL }' with OSSL_DISPATCH_END in OSSL_DISPATCH arrays 2023-04-21 16:19:11 +02:00
aborttest.c
acvp_test.c
acvp_test.inc
aesgcmtest.c
afalgtest.c
algorithmid_test.c
asn1_decode_test.c
asn1_dsa_internal_test.c
asn1_encode_test.c
asn1_internal_test.c OBJ_nid2obj(): Return UNDEF object instead of NULL for NID_undef 2023-03-23 15:33:47 +01:00
asn1_string_table_test.c
asn1_time_test.c
asynciotest.c
asynctest.c
bad_dtls_test.c
bftest.c
bio_callback_test.c
bio_comp_test.c
bio_core_test.c Replaced '{ 0, NULL }' with OSSL_DISPATCH_END in OSSL_DISPATCH arrays 2023-04-21 16:19:11 +02:00
bio_dgram_test.c test//bio_dgram_test.c: Skip test when BIO_bind() fails 2023-03-07 15:24:54 +01:00
bio_enc_test.c
bio_memleak_test.c
bio_prefix_text.c
bio_readbuffer_test.c
bio_tfo_test.c
bioprinttest.c
bn_internal_test.c
bn_rand_range.h
bntest.c bn2bin(): Don't accept len < 0 2023-01-20 07:38:40 +00:00
bntests.pl
build_wincrypt_test.c build_wincrypt_test.c: Fix compilation with MSVC 2023-04-28 20:09:08 +02:00
build.info Add test for handling NEW_CONNECTION_ID frame 2023-05-17 14:04:18 +01:00
ca_internals_test.c
ca-and-certs.cnf APPS: generated certs bear X.509 V3, unless -x509v1 option of req app is given 2023-01-24 15:16:47 +01:00
casttest.c
CAtsa.cnf
cc_dummy.c QUIC CC: Use OSSL_PARAM 2023-05-01 11:03:54 +01:00
cert_comp_test.c
chacha_internal_test.c
cipher_overhead_test.c
cipherbytes_test.c
cipherlist_test.c
ciphername_test.c
clienthellotest.c
cmactest.c
cmp_asn_test.c coverity 1520506: error handling 2023-01-30 08:34:16 +11:00
cmp_client_test.c CMP client: fix checking new cert enrolled with oldcert and without private key 2023-05-12 10:46:27 +02:00
cmp_ctx_test.c
cmp_hdr_test.c coverity 1520505: error handling 2023-01-30 08:34:16 +11:00
cmp_msg_test.c CMP client: fix checking new cert enrolled with oldcert and without private key 2023-05-12 10:46:27 +02:00
cmp_protect_test.c adding provider_unload functions for cmp_ tests 2023-04-16 22:23:24 +10:00
cmp_server_test.c
cmp_status_test.c
cmp_vfy_test.c adding provider_unload functions for cmp_ tests 2023-04-16 22:23:24 +10:00
cms-examples.pl
cmsapitest.c CMS_add0_cert: if cert already present, do not throw error but ignore it 2023-02-24 08:49:26 +01:00
conf_include_test.c
confdump.c
constant_time_test.c
context_internal_test.c
crltest.c
ct_test.c
ctype_internal_test.c
curve448_internal_test.c
d2i_test.c
dane-cross.in
danetest.c
danetest.in
danetest.pem
data2.bin
data.bin
default-and-fips.cnf
default-and-legacy.cnf
default.cnf
defltfips_test.c
destest.c des: prevent error when using two key triple DES with a random key 2023-02-08 21:54:24 +11:00
dhkem_test.inc
dhtest.c
drbgtest.c
dsa_no_digest_size_test.c
dsatest.c Make DSA_sign() test for negative p,q,g values. 2023-03-31 15:17:27 -04:00
dtls_mtu_test.c
dtlstest.c DLTS → DTLS 2023-05-10 18:26:03 +01:00
dtlsv1listentest.c
ec_internal_test.c
ecdsatest.c Fix mem leak in ECDSA_sign(). 2023-03-31 14:57:47 -04:00
ecdsatest.h
ecstresstest.c
ectest.c New function EC_GROUP_to_params to convert an EC_GROUP to an array of OSSL_PARAM. 2023-02-08 10:27:07 -05:00
endecode_test.c
endecoder_legacy_test.c
enginetest.c
errtest.c
event_queue_test.c
evp_extra_test2.c Fix stack corruption in ui_read 2023-05-17 12:07:02 +02:00
evp_extra_test.c provider: return error if buf too small when getting ec pubkey param 2023-05-12 10:32:06 +02:00
evp_fetch_prov_test.c
evp_kdf_test.c
evp_libctx_test.c
evp_pkey_ctx_new_from_name.c
evp_pkey_dhkem_test.c
evp_pkey_dparams_test.c
evp_pkey_provided_test.c
evp_test.c providers: add Argon2 KDF 2023-03-17 11:12:47 +11:00
exdatatest.c
exptest.c Limit size of modulus for bn_mul_mont and BN_mod_exp_mont_consttime 2023-01-14 11:37:18 +01:00
ext_internal_test.c RFC7250 (RPK) support 2023-03-28 13:49:54 -04:00
fake_rsaprov.c Replaced '{ 0, NULL }' with OSSL_DISPATCH_END in OSSL_DISPATCH arrays 2023-04-21 16:19:11 +02:00
fake_rsaprov.h
fatalerrtest.c
ffc_internal_test.c Fix NULL deference when validating FFC public key. 2023-02-07 17:05:10 +01:00
filterprov.c Replaced '{ 0, NULL }' with OSSL_DISPATCH_END in OSSL_DISPATCH arrays 2023-04-21 16:19:11 +02:00
filterprov.h
fips_version_test.c
fips-alt.cnf
fips-and-base.cnf test: note that a default property query must be included for FIPS validity 2023-01-24 12:35:37 +00:00
fips.cnf
generate_buildtest.pl
generate_ssl_tests.pl
gmdifftest.c
hexstr_test.c
hmactest.c
hpke_test.c Fix a HPKE API to put libctx, propq as last (optional parameters). 2023-04-14 13:11:24 +10:00
http_test.c
ideatest.c
igetest.c
insta_ca.cert.pem
insta.priv.pem
keymgmt_internal_test.c
legacy.cnf
lhash_test.c
list_test.c
localetest.c
mdc2_internal_test.c
mdc2test.c
membio_test.c Implement BIO_s_dgram_mem() reusing the BIO_s_dgram_pair() code 2023-01-27 16:11:38 +01:00
memleaktest.c
modes_internal_test.c
moduleloadtest.c
namemap_internal_test.c
nodefltctxtest.c Add a test for no initialisation of the default config file 2023-02-22 10:03:14 +11:00
null.cnf Add a test for no initialisation of the default config file 2023-02-22 10:03:14 +11:00
ocspapitest.c
ossl_store_test.c
p_test.c Replaced '{ 0, NULL }' with OSSL_DISPATCH_END in OSSL_DISPATCH arrays 2023-04-21 16:19:11 +02:00
packettest.c Rationalize FIPS sources 2023-02-08 16:20:55 +01:00
pairwise_fail_test.c Add tests for FIPS keygen self test failures. 2023-02-08 17:00:55 +01:00
param_build_test.c
params_api_test.c
params_conversion_test.c
params_test.c
pbelutest.c
pbetest.c
pem_read_depr_test.c
pemtest.c Add a test for CVE-2022-4450 2023-02-07 17:05:10 +01:00
pkcs7_test.c
pkcs7-1.pem
pkcs7.pem
pkcs12_api_test.c Fix PKCS12_newpass() to work with PBES2. 2023-03-15 08:49:03 +11:00
pkcs12_format_test.c
pkey_meth_kdf_test.c
pkey_meth_test.c
pkits-test.pl
poly1305_internal_test.c
priority_queue_test.c
property_test.c Add negative test for unquoted property string 2023-01-20 10:15:53 +11:00
prov_config_test.c
provfetchtest.c Replaced '{ 0, NULL }' with OSSL_DISPATCH_END in OSSL_DISPATCH arrays 2023-04-21 16:19:11 +02:00
provider_default_search_path_test.c
provider_fallback_test.c
provider_internal_test.c
provider_internal_test.cnf.in
provider_pkey_test.c
provider_status_test.c
provider_test.c
proxy.cnf
punycode_test.c Add more punycode tests and remove ossl_a2ucompare() 2023-02-08 16:17:37 +01:00
quic_ackm_test.c QUIC CC: Major revisions to CC abstract interface 2023-05-01 11:03:54 +01:00
quic_cc_test.c Minor fixups 2023-05-01 11:03:54 +01:00
quic_cfq_test.c
quic_client_test.c Add simple interoperability test with Cloudflare quiche 2023-03-22 10:13:30 +11:00
quic_fc_test.c
quic_fifd_test.c QUIC FIFD: Add support for callback on frame ACK 2023-05-12 14:47:13 +01:00
quic_multistream_test.c QUIC APL: De-publicise SSL_attach_stream/SSL_detach_stream 2023-05-12 14:47:15 +01:00
quic_newcid_test.c Add test for handling NEW_CONNECTION_ID frame 2023-05-17 14:04:18 +01:00
quic_record_test_util.h
quic_record_test.c QUIC DEMUX: Allow MTU to vary over time and autodetect MTU 2023-01-13 13:20:17 +00:00
quic_stream_test.c test_rstream_random(): Test adding final empty frame 2023-03-06 12:26:16 +01:00
quic_tserver_test.c QUIC TSERVER: Add support for multiple streams 2023-05-12 14:47:11 +01:00
quic_txp_test.c Fix test cases using NEW_CONNECTION_ID frame 2023-05-17 14:04:18 +01:00
quic_txpim_test.c
quic_wire_test.c Fix test cases using NEW_CONNECTION_ID frame 2023-05-17 14:04:18 +01:00
quicapitest.c QUIC APL: Fix locking in XSO code and fix tests 2023-05-12 14:47:12 +01:00
quicfaultstest.c QUIC: Update faults test to use streams correctly 2023-05-12 14:47:12 +01:00
rand_status_test.c
rand_test.c
rc2test.c
rc4test.c
rc5test.c
rdcpu_sanitytest.c
README-dev.md
README-external.md
README.md
README.ssltest.md
recordlentest.c
rpktest.c RFC7250 (RPK) support 2023-03-28 13:49:54 -04:00
rsa_complex.c
rsa_mp_test.c
rsa_sp800_56b_test.c Add coverage test for ossl_rsa_sp800_56b_derive_params_from_pq 2023-02-08 10:31:01 -05:00
rsa_test.c Add Tests for RSA_sign_ASN1_OCTET_STRING & RSA_verify_ASN1_OCTET_STRING 2023-02-08 16:19:02 +01:00
rsa_x931_test.c Add libctx to x931 keygen. 2023-05-05 17:11:16 +01:00
run_tests.pl
safe_math_test.c
sanitytest.c Add sanity test for OSSL_sleep() 2023-03-18 20:00:57 +01:00
secmemtest.c
serverinfo2.pem
serverinfo.pem
servername_test.c
session.pem
sha_test.c
shibboleth.pfx
shlibloadtest.c
simpledynamic.c
simpledynamic.h
siphash_internal_test.c
sm2_internal_test.c
sm3_internal_test.c
sm4_internal_test.c
smcont_zero.txt
smcont.bin
smcont.txt
sparse_array_test.c
srptest.c
ssl_cert_table_internal_test.c
ssl_ctx_test.c Extend the min/max protocol testing 2023-05-05 15:25:37 +01:00
ssl_old_test.c
ssl_test_ctx_test.c
ssl_test_ctx_test.cnf
ssl_test.c test: update ssl_new tests in line with pedantic FIPS policy 2023-04-21 17:01:38 +01:00
ssl_test.tmpl
sslapitest.c Clear ownership when duplicating sessions 2023-05-12 10:23:29 +02:00
sslbuffertest.c Move recordmethod.h to be an "internal" header 2023-01-24 17:16:29 +00:00
sslcorrupttest.c
stack_test.c
sysdefault.cnf
sysdefaulttest.c
test_test.c
test.cnf
testcrl.pem
testdsa.pem
testdsapub.pem
testec-p112r1.pem
testec-p256.pem
testecpub-p256.pem
tested448.pem
tested448pub.pem
tested25519.pem
tested25519pub.pem
testp7.pem
testreq2.pem
testrsa2048.pem
testrsa2048pub.pem
testrsa_withattrs.der
testrsa_withattrs.pem
testrsa.pem
testrsapss.pem
testrsapssmandatory.pem
testrsapub.pem
testsid.pem
testutil.h
testx509.pem
threadpool_test.c
threadstest_fips.c
threadstest.c
threadstest.h
time_offset_test.c
timing_load_creds.c
tls13ccstest.c
tls13encryptiontest.c Extend the new_record_layer function 2023-01-24 17:16:29 +00:00
tls13secretstest.c Extend the new_record_layer function 2023-01-24 17:16:29 +00:00
tls-provider.c Replaced '{ 0, NULL }' with OSSL_DISPATCH_END in OSSL_DISPATCH arrays 2023-04-21 16:19:11 +02:00
trace_api_test.c add OSSL_TRACE_STRING(), OSSL_TRACE_STRING_MAX, and OSSL_trace_string() 2023-01-26 09:16:51 +01:00
uitest.c
upcallstest.c Replaced '{ 0, NULL }' with OSSL_DISPATCH_END in OSSL_DISPATCH arrays 2023-04-21 16:19:11 +02:00
user_property_test.c Replaced '{ 0, NULL }' with OSSL_DISPATCH_END in OSSL_DISPATCH arrays 2023-04-21 16:19:11 +02:00
v3_ca_exts.cnf
v3-cert1.pem
v3-cert2.pem
v3ext.c
v3nametest.c Fix GENERAL_NAME_cmp for x400Address (master) 2023-02-07 17:05:10 +01:00
verify_extra_test.c
versions.c
wpackettest.c Rationalize FIPS sources 2023-02-08 16:20:55 +01:00
x509_check_cert_pkey_test.c
x509_dup_cert_test.c
x509_internal_test.c
x509_test.c
x509_time_test.c
x509aux.c

Using OpenSSL Tests

After a successful build, and before installing, the libraries should be tested. Run:

$ make test                                      # Unix
$ mms test                                       ! OpenVMS
$ nmake test                                     # Windows

Warning: you MUST run the tests from an unprivileged account (or disable your privileges temporarily if your platform allows it).

If some tests fail, take a look at the section Test Failures below.

Test Failures

If some tests fail, look at the output. There may be reasons for the failure that isn't a problem in OpenSSL itself (like an OS malfunction or a Perl issue). You may want increased verbosity, that can be accomplished like this:

Full verbosity, showing full output of all successful and failed test cases (make macro VERBOSE or V):

$ make V=1 test                                  # Unix
$ mms /macro=(V=1) test                          ! OpenVMS
$ nmake V=1 test                                 # Windows

Verbosity on failed (sub-)tests only (VERBOSE_FAILURE or VF or REPORT_FAILURES):

$ make test VF=1

Verbosity on failed (sub-)tests, in addition progress on succeeded (sub-)tests (VERBOSE_FAILURE_PROGRESS or VFP or REPORT_FAILURES_PROGRESS):

$ make test VFP=1

If you want to run just one or a few specific tests, you can use the make variable TESTS to specify them, like this:

$ make TESTS='test_rsa test_dsa' test            # Unix
$ mms/macro="TESTS=test_rsa test_dsa" test       ! OpenVMS
$ nmake TESTS='test_rsa test_dsa' test           # Windows

And of course, you can combine (Unix examples shown):

$ make test TESTS='test_rsa test_dsa' VF=1
$ make test TESTS="test_cmp_*" VFP=1

You can find the list of available tests like this:

$ make list-tests                                # Unix
$ mms list-tests                                 ! OpenVMS
$ nmake list-tests                               # Windows

Have a look at the manual for the perl module Test::Harness to see what other HARNESS_* variables there are.

To report a bug please open an issue on GitHub, at https://github.com/openssl/openssl/issues.

For more details on how the make variables TESTS can be used, see section Running Selected Tests below.

Running Selected Tests

The make variable TESTS supports a versatile set of space separated tokens with which you can specify a set of tests to be performed. With a "current set of tests" in mind, initially being empty, here are the possible tokens:

 alltests      The current set of tests becomes the whole set of available
               tests (as listed when you do 'make list-tests' or similar).

 xxx           Adds the test 'xxx' to the current set of tests.

-xxx           Removes 'xxx' from the current set of tests.  If this is the
               first token in the list, the current set of tests is first
               assigned the whole set of available tests, effectively making
               this token equivalent to TESTS="alltests -xxx".

 nn            Adds the test group 'nn' (which is a number) to the current
               set of tests.

-nn            Removes the test group 'nn' from the current set of tests.
               If this is the first token in the list, the current set of
               tests is first assigned the whole set of available tests,
               effectively making this token equivalent to
               TESTS="alltests -xxx".

Also, all tokens except for "alltests" may have wildcards, such as *. (on Unix and Windows, BSD style wildcards are supported, while on VMS, it's VMS style wildcards)

Examples

Run all tests except for the fuzz tests:

$ make TESTS='-test_fuzz*' test

or, if you want to be explicit:

$ make TESTS='alltests -test_fuzz*' test

Run all tests that have a name starting with "test_ssl" but not those starting with "test_ssl_":

$ make TESTS='test_ssl* -test_ssl_*' test

Run only test group 10:

$ make TESTS='10' test

Run all tests except the slow group (group 99):

$ make TESTS='-99' test

Run all tests in test groups 80 to 99 except for tests in group 90:

$ make TESTS='[89]? -90' test

To run specific fuzz tests you can use for instance:

$ make test TESTS='test_fuzz_cmp test_fuzz_cms'

To stochastically verify that the algorithm that produces uniformly distributed random numbers is operating correctly (with a false positive rate of 0.01%):

$ ./util/wrap.sh test/bntest -stochastic

Running Tests in Parallel

By default the test harness will execute the selected tests sequentially. Depending on the platform characteristics, running more than one test job in parallel may speed up test execution. This can be requested by setting the HARNESS_JOBS environment variable to a positive integer value. This specifies the maximum number of test jobs to run in parallel.

Depending on the Perl version different strategies could be adopted to select which test recipes can be run in parallel. In recent versions of Perl, unless specified otherwise, any task can be run in parallel. Consult the documentation for TAP::Harness to know more.

To run up to four tests in parallel at any given time:

$ make HARNESS_JOBS=4 test

Randomisation of Test Ordering

By default, the test harness will execute tests in the order they were added. By setting the OPENSSL_TEST_RAND_ORDER environment variable to zero, the test ordering will be randomised. If a randomly ordered test fails, the seed value used will be reported. Setting the OPENSSL_TEST_RAND_ORDER environment variable to this value will rerun the tests in the same order. This assures repeatability of randomly ordered test runs. This repeatability is independent of the operating system, processor or platform used.

To randomise the test ordering:

$ make OPENSSL_TEST_RAND_ORDER=0 test

To run the tests using the order defined by the random seed 42:

$ make OPENSSL_TEST_RAND_ORDER=42 test