openssl/test
Dimitri John Ledkov 60f69680b2 test: fix 20-test_dgst.t to use hexkey
Currently 20-test_dgst.t calls a quite bogus command:

    $ openssl dgst -sha256 -hmac -macopt hexkey:FFFF test/data.bin test/data.bin
    hexkey:FFFF: No such file or directory
    HMAC-SHA2-256(test/data.bin)= b6727b7bb251dfa65846e0a8223bdd57d244aa6d7e312cb906d8e21f2dee3a57
    HMAC-SHA2-256(test/data.bin)= b6727b7bb251dfa65846e0a8223bdd57d244aa6d7e312cb906d8e21f2dee3a57
    805B632D4A730000:error:80000002:system library:file_ctrl:No such file or directory:crypto/bio/bss_file.c:297:calling fopen(hexkey:FFF, r)
    805B632D4A730000:error:10080002:BIO routines:file_ctrl:system lib:crypto/bio/bss_file.c:300:

Does not check status code, discards stderr, and verifies the
checksums as per above. Note that the checksum is for the HMAC key
"-macopt", and `hexkey:FFFF` is attempted to be opened as a file.

See HMAC values for key `-macopt` and `hexkey:FFFF` using `openssl-mac`:

    $ openssl mac -digest SHA256 -macopt hexkey:$(printf '%s' '-macopt' | xxd -p -u) -in ./test/data.bin HMAC
    B6727B7BB251DFA65846E0A8223BDD57D244AA6D7E312CB906D8E21F2DEE3A57

    $ openssl mac -digest SHA256 -macopt hexkey:FFFF -in ./test/data.bin HMAC
    7C02D4A17D2560A5BB6763EDBF33F3A34F415398F8F2E07F04B83FFD7C087DAE

Fix this test case to actually use HMAC with hexkey:FFFF as intended.

Signed-off-by: Dimitri John Ledkov <dimitri.ledkov@surgut.co.uk>

Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Paul Dale <ppzgs1@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/24068)
2024-04-10 09:33:56 +02:00
..
certs
ct
d2i-tests
helpers
ocsp-tests
recipes test: fix 20-test_dgst.t to use hexkey 2024-04-10 09:33:56 +02:00
smime-certs
smime-eml
ssl-tests
testutil
aborttest.c
acvp_test.c Fix the checks of EVP_PKEY_CTX_set/get_* functions 2022-06-02 11:06:41 +02:00
acvp_test.inc Drop OPENSSL_NO_RSA everywhere 2020-12-20 12:19:42 +01:00
aesgcmtest.c Remove FIPS condition on IV gen test. 2022-09-21 17:02:59 +10:00
afalgtest.c Copyright year updates 2023-09-28 14:23:29 +01:00
algorithmid_test.c
asn1_decode_test.c
asn1_dsa_internal_test.c Update copyright year 2020-10-15 14:10:06 +01:00
asn1_encode_test.c Update copyright year 2021-07-29 15:41:35 +01:00
asn1_internal_test.c
asn1_stable_parse_test.c
asn1_string_table_test.c
asn1_time_test.c Copyright year updates 2024-04-09 13:43:26 +02:00
asynciotest.c
asynctest.c
bad_dtls_test.c
bftest.c
bio_addr_test.c
bio_callback_test.c
bio_comp_test.c
bio_core_test.c
bio_dgram_test.c
bio_enc_test.c Update copyright year 2022-05-03 13:34:51 +01:00
bio_memleak_test.c
bio_meth_test.c Fix BIO_get_new_index() to return an error when it is exhausted. 2024-03-11 11:34:25 +00:00
bio_prefix_text.c
bio_readbuffer_test.c Fix DER reading from stdin for BIO_f_readbuffer 2021-03-22 09:07:36 +10:00
bio_tfo_test.c Fix asan finding in bio_tfo_test 2022-08-12 16:13:13 +01:00
bioprinttest.c
bn_internal_test.c
bn_rand_range.h
bntest.c
bntests.pl Following the license change, modify the boilerplates in test/ 2018-12-06 14:19:22 +01:00
build_wincrypt_test.c
build.info
ca_internals_test.c
ca-and-certs.cnf
casttest.c
CAtsa.cnf Changed the default value of the "ess_cert_id_alg" option 2023-08-25 15:05:51 +02:00
cc_dummy.c
cert_comp_test.c
chacha_internal_test.c
cipher_overhead_test.c Update copyright year 2021-02-18 15:05:17 +00:00
cipherbytes_test.c
cipherlist_test.c
ciphername_test.c
clienthellotest.c fix some code with obvious wrong coding style 2021-10-28 13:10:46 +10:00
cmactest.c
cmp_asn_test.c
cmp_client_test.c
cmp_ctx_test.c
cmp_hdr_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
cmp_msg_test.c
cmp_protect_test.c
cmp_server_test.c crypto/cmp/,apps/lib/cmp_mock_srv.c: various improvements on delayed delivery 2023-12-21 23:06:42 +01:00
cmp_status_test.c
cmp_vfy_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
cms-examples.pl
cmsapitest.c
conf_include_test.c Add config tests for including provider config files 2023-11-09 13:27:56 +01:00
confdump.c
constant_time_test.c
context_internal_test.c
crltest.c Update copyright year 2022-05-03 13:34:51 +01:00
ct_test.c Update copyright year 2022-05-03 13:34:51 +01:00
ctype_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
curve448_internal_test.c
d2i_test.c
dane-cross.in Apply the correct Apache v2 license 2022-02-14 10:08:21 +01:00
danetest.c add OSSL_STACK_OF_X509_free() for commonly used pattern 2021-12-21 12:11:49 +01:00
danetest.in Add last missing TLSA usage/selector/mtype test case 2023-11-29 10:31:42 +01:00
danetest.pem
data2.bin Remove the external BoringSSL test 2021-03-26 14:24:06 +01:00
data.bin
decoder_propq_test.c Fix decoders so that they use the passed in propq. 2023-08-16 18:02:51 +02:00
default-and-fips.cnf Make the activate setting more intuitive 2023-12-21 09:22:40 -05:00
default-and-legacy.cnf
default.cnf
defltfips_test.c
destest.c
dhkem_test.inc
dhtest.c
drbgtest.c
dsa_no_digest_size_test.c
dsatest.c
dtls_mtu_test.c SSL object refactoring using SSL_CONNECTION object 2022-07-28 10:04:28 +01:00
dtlstest.c Copyright year updates 2023-09-07 09:59:15 +01:00
dtlsv1listentest.c
ec_internal_test.c
ecdsatest.c
ecdsatest.h
ecstresstest.c APPS & TEST: Use ossl_[u]intmax_t rather than [u]intmax_t 2021-06-22 19:50:11 +10:00
ectest.c "foo * bar" should be "foo *bar" 2023-09-11 10:15:30 +02:00
endecode_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
endecoder_legacy_test.c test: fix coverity 1469427 Improper use of negative value (NEGATIVE_RETURNS) 2021-06-30 13:55:09 +10:00
enginetest.c Update copyright year 2022-05-03 13:34:51 +01:00
errtest.c
event_queue_test.c
evp_extra_test2.c Copyright year updates 2024-04-09 13:43:26 +02:00
evp_extra_test.c
evp_fetch_prov_test.c Make the activate setting more intuitive 2023-12-21 09:22:40 -05:00
evp_kdf_test.c Copyright year updates 2024-04-09 13:43:26 +02:00
evp_libctx_test.c
evp_pkey_ctx_new_from_name.c
evp_pkey_dhkem_test.c
evp_pkey_dparams_test.c Update copyright year 2022-05-03 13:34:51 +01:00
evp_pkey_provided_test.c Copyright year updates 2024-04-09 13:43:26 +02:00
evp_test.c
evp_xof_test.c
exdatatest.c Update copyright year 2021-04-08 13:04:41 +01:00
exptest.c Copyright year updates 2023-09-07 09:59:15 +01:00
ext_internal_test.c
fake_rsaprov.c Add Test to verify open_ex password checking works 2023-09-19 12:24:07 +02:00
fake_rsaprov.h
fatalerrtest.c test cleanup: move helper .c and .h files to test/helpers/ 2020-12-01 07:49:38 +01:00
ffc_internal_test.c DH_check_pub_key() should not fail when setting result code 2023-10-11 16:22:27 +02:00
filterprov.c
filterprov.h provider: add an unquery function to allow providers to clean up. 2021-02-24 21:24:36 +10:00
fips_version_test.c
fips-alt.cnf
fips-and-base.cnf test: note that a default property query must be included for FIPS validity 2023-01-24 12:35:37 +00:00
fips.cnf
generate_buildtest.pl
generate_ssl_tests.pl
gmdifftest.c Reduce the runtime/output from the gmdiff test 2021-05-11 18:15:06 +10:00
hexstr_test.c
hmactest.c
hpke_test.c Add additional internal HPKE hardening checks resulting from code audit. 2023-11-03 09:10:19 +01:00
http_test.c
ideatest.c
igetest.c
insta_ca.cert.pem
insta.priv.pem Remove executable mode attributes of non-executable files 2021-07-13 16:04:32 +10:00
invalid-x509.cnf
json_test.c
keymgmt_internal_test.c
legacy.cnf
lhash_test.c
list_test.c
localetest.c Include the e_os.h before string.h 2022-05-13 08:30:41 +02:00
mdc2_internal_test.c
mdc2test.c
membio_test.c
memleaktest.c Update copyright year 2021-05-06 13:03:23 +01:00
modes_internal_test.c
moduleloadtest.c
namemap_internal_test.c
nodefltctxtest.c
null.cnf Add a test for no initialisation of the default config file 2023-02-22 10:03:14 +11:00
ocspapitest.c
ossl_store_test.c fix some code with obvious wrong coding style 2021-10-28 13:10:46 +10:00
p_minimal.c
p_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
packettest.c
pairwise_fail_test.c Copyright year updates 2024-04-09 13:43:26 +02:00
param_build_test.c
params_api_test.c
params_conversion_test.c
params_test.c Copyright year updates 2024-04-09 13:43:26 +02:00
pbelutest.c Following the license change, modify the boilerplates in test/ 2018-12-06 14:19:22 +01:00
pbetest.c
pem_read_depr_test.c
pemtest.c
pkcs7_test.c
pkcs7-1.pem
pkcs7.pem
pkcs12_api_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
pkcs12_format_test.c
pkey_meth_kdf_test.c
pkey_meth_test.c
pkits-test.pl
poly1305_internal_test.c Add ChaCha related ciphers to default provider 2019-10-16 16:18:42 +10:00
priority_queue_test.c Add unit test for #22644 2023-11-08 11:09:12 +00:00
property_test.c
prov_config_test.c
provfetchtest.c Copyright year updates 2023-09-07 09:59:15 +01:00
provider_default_search_path_test.c Implement OSSL_PROVIDER_get0_default_search_path, add docs and tests. 2022-12-06 18:24:06 +01:00
provider_fallback_test.c
provider_internal_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
provider_internal_test.cnf.in Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
provider_pkey_test.c Copyright year updates 2023-09-28 14:23:29 +01:00
provider_status_test.c
provider_test.c test_provider_ex(): Add missing call failure checks 2023-10-09 10:21:19 +11:00
proxy.cnf Add config_diagnostics to our configuration files. 2021-08-04 08:15:14 +10:00
punycode_test.c
quic_ackm_test.c
quic_cc_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
quic_cfq_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
quic_client_test.c
quic_fc_test.c
quic_fifd_test.c Copyright year updates 2024-04-09 13:43:26 +02:00
quic_lcidm_test.c QUIC LCIDM: Minor fixes 2023-12-06 10:40:11 +00:00
quic_multistream_test.c
quic_newcid_test.c
quic_qlog_test.c QUIC QLOG: Fix ANSI 2024-03-06 20:44:48 +00:00
quic_rcidm_test.c
quic_record_test_util.h
quic_record_test.c QUIC QRL TEST: Remove dependency on legacy DEMUX-QRX routing 2023-12-21 08:12:00 +00:00
quic_srt_gen_test.c QUIC SRT GEN: Minor updates 2023-11-25 09:14:05 +00:00
quic_srtm_test.c
quic_stream_test.c QUIC SSTREAM: Fix bug in ossl_quic_sstream_is_totally_acked 2023-11-02 08:49:01 +00:00
quic_tserver_test.c quic_tserver_test: Raise the initial timeout limit 2023-10-25 09:35:31 +01:00
quic_txp_test.c
quic_txpim_test.c
quic_wire_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
quicapitest.c
quicfaultstest.c Copyright year updates 2024-04-09 13:43:26 +02:00
rand_status_test.c
rand_test.c
rc2test.c
rc4test.c
rc5test.c
rdcpu_sanitytest.c
README-dev.md
README-external.md
README.md Fix instructions for running tests on Windows 2023-12-18 10:47:31 +01:00
README.ssltest.md Make running individual ssl-test easier 2022-05-27 14:17:29 -04:00
recordlentest.c
recursive.cnf
rpktest.c
rsa_complex.c
rsa_mp_test.c
rsa_sp800_56b_test.c
rsa_test.c
rsa_x931_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
run_tests.pl
safe_math_test.c Update copyright year 2022-05-03 13:34:51 +01:00
sanitytest.c
secmemtest.c
serverinfo2.pem
serverinfo.pem
servername_test.c test: fix coverity 1451534: improper use of negative value 2021-03-24 09:12:43 +10:00
session.pem
sha_test.c
shibboleth.pfx
shlibloadtest.c
simpledynamic.c
simpledynamic.h
siphash_internal_test.c Copyright year updates 2024-04-09 13:43:26 +02:00
sm2_internal_test.c Add check for public key presence on sm2 signing 2024-04-09 20:38:58 +02:00
sm3_internal_test.c Apply the correct Apache v2 license 2022-02-14 10:08:21 +01:00
sm4_internal_test.c Update copyright year 2021-04-08 13:04:41 +01:00
smcont_zero.txt
smcont.bin apps/cms.c: Correct -sign output and -verify input with -binary 2021-05-22 14:28:51 +02:00
smcont.txt
sparse_array_test.c
srptest.c
ssl_cert_table_internal_test.c
ssl_ctx_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
ssl_handshake_rtt_test.c [feat] SSL RTT in both client and server statem. SSL_get_handshake_rtt makes it available 2023-06-02 05:46:46 +02:00
ssl_old_test.c Copyright year updates 2024-04-09 13:43:26 +02:00
ssl_test_ctx_test.c
ssl_test_ctx_test.cnf Use .cnf for config files, not .conf 2020-03-06 18:25:13 +01:00
ssl_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
ssl_test.tmpl
sslapitest.c
sslbuffertest.c Copyright year updates 2023-09-07 09:59:15 +01:00
sslcorrupttest.c tests: clear error queue before executing a testcase 2022-10-27 18:39:29 +02:00
stack_test.c
sysdefault.cnf
sysdefaulttest.c
test_asn1_parse.cnf Harden asn1 oid loader to invalid inputs 2023-12-13 11:10:36 -05:00
test_test.c
test.cnf Fix openssl req with -addext subjectAltName=dirName 2024-04-02 17:35:25 +02:00
testcrl.pem Import of old SSLeay release: SSLeay 0.8.1b 1998-12-21 10:52:47 +00:00
testdsa.pem
testdsapub.pem
testec-p112r1.pem TEST: Check property query support of apps/pkey 2022-08-17 09:20:41 +02:00
testec-p256.pem
testecpub-p256.pem Add private/public key conversion tests 2015-03-29 03:26:12 +01:00
tested448.pem
tested448pub.pem
tested25519.pem
tested25519pub.pem
testp7.pem Change PKCS#7 test data to take account of removal of 2000-08-25 01:29:41 +00:00
testreq2.pem
testrsa2048.pem Add fips checks for rsa encryption 2020-09-18 14:20:38 +01:00
testrsa2048pub.pem
testrsa_withattrs.der
testrsa_withattrs.pem Tests for creating req from PKCS8 keys with extra attrs 2021-05-11 12:12:32 +02:00
testrsa.pem
testrsapss.pem rsa_kmgmt: Return OSSL_PKEY_PARAM_DEFAULT_DIGEST for unrestricted PSS keys 2021-01-29 10:47:02 +01:00
testrsapssmandatory.pem
testrsapub.pem
testsid.pem Regenerate testsid.pem 2021-05-15 15:09:07 -07:00
testutil.h
testx509.pem
threadpool_test.c
threadstest_fips.c
threadstest.c
threadstest.h
time_offset_test.c Following the license change, modify the boilerplates in test/ 2018-12-06 14:19:22 +01:00
time_test.c
timing_load_creds.c timing_load_creds requires POSIX1.2001 due to rusage 2022-12-15 12:04:01 +01:00
tls13ccstest.c
tls13encryptiontest.c Add tests for return codes for EVP_CIPHER_CTX_get[block_size|iv_length] 2024-01-25 08:36:17 -05:00
tls13secretstest.c
tls-provider.c Copyright year updates 2024-04-09 13:43:26 +02:00
trace_api_test.c trace_api_test.c: Separate tracing statements 2023-10-13 15:04:42 +02:00
uitest.c
upcallstest.c
user_property_test.c
v3_ca_exts.cnf
v3-cert1.pem
v3-cert2.pem
v3ext.c
v3nametest.c
verify_extra_test.c
versions.c
wpackettest.c Copyright year updates 2023-09-07 09:59:15 +01:00
x509_check_cert_pkey_test.c Copyright year updates 2023-09-07 09:59:15 +01:00
x509_dup_cert_test.c
x509_internal_test.c
x509_load_cert_file_test.c Fix a memory leak on successful load of CRL 2024-03-07 11:03:31 +01:00
x509_test.c Add test for X509 sign TBS cache regression. 2022-11-02 11:14:32 +01:00
x509_time_test.c Copyright year updates 2024-04-09 13:43:26 +02:00
x509aux.c Cleanup : directly include of internal/nelem.h when required. 2022-11-23 18:08:25 +01:00

Using OpenSSL Tests

After a successful build, and before installing, the libraries should be tested. Run:

$ make test                                      # Unix
$ mms test                                       ! OpenVMS
$ nmake test                                     # Windows

Warning: you MUST run the tests from an unprivileged account (or disable your privileges temporarily if your platform allows it).

If some tests fail, take a look at the section Test Failures below.

Test Failures

If some tests fail, look at the output. There may be reasons for the failure that isn't a problem in OpenSSL itself (like an OS malfunction or a Perl issue). You may want increased verbosity, that can be accomplished like this:

Full verbosity, showing full output of all successful and failed test cases (make macro VERBOSE or V):

$ make V=1 test                                  # Unix
$ mms /macro=(V=1) test                          ! OpenVMS
$ nmake V=1 test                                 # Windows

Verbosity on failed (sub-)tests only (VERBOSE_FAILURE or VF or REPORT_FAILURES):

$ make test VF=1

Verbosity on failed (sub-)tests, in addition progress on succeeded (sub-)tests (VERBOSE_FAILURE_PROGRESS or VFP or REPORT_FAILURES_PROGRESS):

$ make test VFP=1

If you want to run just one or a few specific tests, you can use the make variable TESTS to specify them, like this:

$ make TESTS='test_rsa test_dsa' test            # Unix
$ mms/macro="TESTS=test_rsa test_dsa" test       ! OpenVMS
$ nmake TESTS="test_rsa test_dsa" test           # Windows

And of course, you can combine (Unix examples shown):

$ make test TESTS='test_rsa test_dsa' VF=1
$ make test TESTS="test_cmp_*" VFP=1

You can find the list of available tests like this:

$ make list-tests                                # Unix
$ mms list-tests                                 ! OpenVMS
$ nmake list-tests                               # Windows

Have a look at the manual for the perl module Test::Harness to see what other HARNESS_* variables there are.

To report a bug please open an issue on GitHub, at https://github.com/openssl/openssl/issues.

For more details on how the make variables TESTS can be used, see section Running Selected Tests below.

Running Selected Tests

The make variable TESTS supports a versatile set of space separated tokens with which you can specify a set of tests to be performed. With a "current set of tests" in mind, initially being empty, here are the possible tokens:

 alltests      The current set of tests becomes the whole set of available
               tests (as listed when you do 'make list-tests' or similar).

 xxx           Adds the test 'xxx' to the current set of tests.

-xxx           Removes 'xxx' from the current set of tests.  If this is the
               first token in the list, the current set of tests is first
               assigned the whole set of available tests, effectively making
               this token equivalent to TESTS="alltests -xxx".

 nn            Adds the test group 'nn' (which is a number) to the current
               set of tests.

-nn            Removes the test group 'nn' from the current set of tests.
               If this is the first token in the list, the current set of
               tests is first assigned the whole set of available tests,
               effectively making this token equivalent to
               TESTS="alltests -xxx".

Also, all tokens except for "alltests" may have wildcards, such as *. (on Unix and Windows, BSD style wildcards are supported, while on VMS, it's VMS style wildcards)

Examples

Run all tests except for the fuzz tests:

$ make TESTS='-test_fuzz*' test

or, if you want to be explicit:

$ make TESTS='alltests -test_fuzz*' test

Run all tests that have a name starting with "test_ssl" but not those starting with "test_ssl_":

$ make TESTS='test_ssl* -test_ssl_*' test

Run only test group 10:

$ make TESTS='10' test

Run all tests except the slow group (group 99):

$ make TESTS='-99' test

Run all tests in test groups 80 to 99 except for tests in group 90:

$ make TESTS='[89]? -90' test

To run specific fuzz tests you can use for instance:

$ make test TESTS='test_fuzz_cmp test_fuzz_cms'

To stochastically verify that the algorithm that produces uniformly distributed random numbers is operating correctly (with a false positive rate of 0.01%):

$ ./util/wrap.sh test/bntest -stochastic

Running Tests in Parallel

By default the test harness will execute the selected tests sequentially. Depending on the platform characteristics, running more than one test job in parallel may speed up test execution. This can be requested by setting the HARNESS_JOBS environment variable to a positive integer value. This specifies the maximum number of test jobs to run in parallel.

Depending on the Perl version different strategies could be adopted to select which test recipes can be run in parallel. In recent versions of Perl, unless specified otherwise, any task can be run in parallel. Consult the documentation for TAP::Harness to know more.

To run up to four tests in parallel at any given time:

$ make HARNESS_JOBS=4 test

Random numbers in tests

Some tests use random numbers as part of the test. In some cases a test failure may occur for some random numbers, but not for others. The seed used for the rand number generator can be set via the OPENSSL_TEST_RAND_SEED environment variable. It can also be set via the OPENSSL_TEST_RAND_ORDER environment variable which additionally randomises the order tests are run in (see below).

When a test fails the test harness will display the seed used during the test (displaying either the OPENSSL_TEST_RAND_SEED or OPENSSL_TEST_RAND_ORDER environment variable value that must be used to recreate the results), e.g.

$ make OPENSSL_TEST_RAND_SEED=42 test

Randomisation of Test Ordering

By default, the test harness will execute tests in the order they were added. By setting the OPENSSL_TEST_RAND_ORDER environment variable to zero, the test ordering will be randomised. This additionally seeds the random number generator used within the tests as described in the section above. If a randomly ordered test fails, the seed value used will be reported. Setting the OPENSSL_TEST_RAND_ORDER environment variable to this value will rerun the tests in the same order and will also seed the test random number generator. This assures repeatability of randomly ordered test runs. This repeatability is independent of the operating system, processor or platform used.

To randomise the test ordering:

$ make OPENSSL_TEST_RAND_ORDER=0 test

To run the tests using the order defined by the random seed 42:

$ make OPENSSL_TEST_RAND_ORDER=42 test