openssl/test
Matt Caswell 1b39bc9bcf Fix the drbgtest with randomized ordering
In drbgtest, test_set_defaults changes the default DRBGs. This works fine
when tests are run in the normal order. However if
OPENSSL_TEST_RAND_ORDER is defined then it may fail (dependent on the
ordering). This environment variable is defined for one of the Travis
tests, so this issue was causing intermittent travis test failures.

[extended tests]

Reviewed-by: Matthias St. Pierre <Matthias.St.Pierre@ncp-e.com>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/7338)
2018-10-03 16:41:33 +01:00
..
certs Limit scope of CN name constraints 2018-05-23 11:12:13 -04:00
ct
d2i-tests add test for CVE-2016-7053 2016-11-10 13:04:11 +00:00
ocsp-tests Fix OCSP_basic_verify() cert chain construction in case bs->certs is NULL 2017-08-16 14:32:38 -04:00
ossl_shim Fix BoringSSL external test failures 2018-08-22 15:15:19 +01:00
recipes FIPS 140-2 IG A.9 XTS key check. 2018-09-12 08:40:47 +10:00
smime-certs Add alternative CMS P-256 cert 2017-08-10 16:48:18 +01:00
ssl-tests Add a test for RSA key exchange with both RSA and RSA-PSS certs 2018-09-04 11:28:01 +01:00
testutil Update copyright year 2018-09-11 13:45:17 +01:00
aborttest.c
afalgtest.c Revert "Modify test/afalgtest to fail if the afalg engine couldn't be loaded" 2018-02-07 22:18:44 +01:00
asn1_decode_test.c TESTS: add test of decoding of invalid zero length ASN.1 INTEGER zero 2018-09-09 03:35:26 +02:00
asn1_encode_test.c Update copyright year 2018-02-27 13:59:42 +00:00
asn1_internal_test.c Update copyright year 2018-09-11 13:45:17 +01:00
asn1_string_table_test.c [Win] Fix some test method signatures ... 2017-08-16 10:36:34 -04:00
asn1_time_test.c test/asn1_time_test.c: make it work on 64-bit HP-UX. 2018-04-08 11:17:44 +02:00
asynciotest.c Update code for the final RFC version of TLSv1.3 (RFC8446) 2018-08-15 12:33:30 +01:00
asynctest.c Update copyright year 2018-05-29 13:16:04 +01:00
bad_dtls_test.c Remove unicode characters from source 2017-12-08 11:56:37 +01:00
bftest.c Consistent formatting for sizeof(foo) 2017-12-07 19:11:49 -05:00
bio_callback_test.c Fix bio callback backward compatibility 2018-03-19 14:20:53 +01:00
bio_enc_test.c Fix no-chacha and no-poly1305 2017-08-25 11:34:08 +01:00
bioprinttest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
bntest.c Fixed range of random produced in BN_is_prime_fasttest_ex() to be 1 < rand < w-1. It was using 1<= rand < w (which is wrong by 1 on both ends) 2018-06-22 07:07:20 +10:00
bntests.pl Make bntest be (mostly) file-based. 2016-11-28 12:26:05 -05:00
build.info Added DRBG_HMAC & DRBG_HASH + Added defaults for setting DRBG for master/public/private + renamed generate_counter back to reseed_counter + generated new cavs data tests 2018-09-28 07:22:13 +10:00
CAss.cnf
CAssdh.cnf
CAssdsa.cnf
CAssrsa.cnf
casttest.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
CAtsa.cnf Added support for ESSCertIDv2 2017-05-03 09:04:23 +02:00
chacha_internal_test.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
cipher_overhead_test.c VMS: turn on name mangling for all our programs 2018-09-14 11:00:38 +02:00
cipherbytes_test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
cipherlist_test.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
ciphername_test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
clienthellotest.c Enhance ssltestlib's create_ssl_ctx_pair to take min and max proto version 2018-03-19 18:24:30 +01:00
cms-examples.pl
cmsapitest.c Add a CMS API test 2018-05-08 08:43:39 +01:00
conf_include_test.c NCONF_get_number refix. 2018-07-11 09:03:22 +10:00
constant_time_test.c Update copyright year 2018-02-27 13:59:42 +00:00
crltest.c Factorise duplicated code. 2017-11-13 07:52:35 -05:00
ct_test.c Add an explicit cast to time_t 2018-09-13 09:00:29 +01:00
ctype_internal_test.c Update copyright year 2018-03-20 13:08:46 +00:00
curve448_internal_test.c VMS: turn on name mangling for all our programs 2018-09-14 11:00:38 +02:00
d2i_test.c Consistent formatting for sizeof(foo) 2017-12-07 19:11:49 -05:00
danetest.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
danetest.in
danetest.pem
destest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
dhtest.c test/dhtest.c: fix resource leak 2018-09-10 10:32:33 +01:00
drbg_cavs_data_ctr.c Added DRBG_HMAC & DRBG_HASH + Added defaults for setting DRBG for master/public/private + renamed generate_counter back to reseed_counter + generated new cavs data tests 2018-09-28 07:22:13 +10:00
drbg_cavs_data_hash.c Added DRBG_HMAC & DRBG_HASH + Added defaults for setting DRBG for master/public/private + renamed generate_counter back to reseed_counter + generated new cavs data tests 2018-09-28 07:22:13 +10:00
drbg_cavs_data_hmac.c Added DRBG_HMAC & DRBG_HASH + Added defaults for setting DRBG for master/public/private + renamed generate_counter back to reseed_counter + generated new cavs data tests 2018-09-28 07:22:13 +10:00
drbg_cavs_data.h Added DRBG_HMAC & DRBG_HASH + Added defaults for setting DRBG for master/public/private + renamed generate_counter back to reseed_counter + generated new cavs data tests 2018-09-28 07:22:13 +10:00
drbg_cavs_test.c Added DRBG_HMAC & DRBG_HASH + Added defaults for setting DRBG for master/public/private + renamed generate_counter back to reseed_counter + generated new cavs data tests 2018-09-28 07:22:13 +10:00
drbgtest.c Fix the drbgtest with randomized ordering 2018-10-03 16:41:33 +01:00
drbgtest.h Added DRBG_HMAC & DRBG_HASH + Added defaults for setting DRBG for master/public/private + renamed generate_counter back to reseed_counter + generated new cavs data tests 2018-09-28 07:22:13 +10:00
dsa_no_digest_size_test.c Add test for DSA signatures of raw digests of various sizes 2018-07-29 21:27:36 +02:00
dsatest.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
dtls_mtu_test.c Update copyright year 2018-05-29 13:16:04 +01:00
dtlstest.c Fix no-ec in combination with no-dh 2018-05-22 13:21:24 +01:00
dtlsv1listentest.c Update copyright year 2018-05-29 13:16:04 +01:00
ecdsatest.c Update copyright year 2018-03-20 13:08:46 +00:00
ecstresstest.c Update copyright year 2018-09-11 13:45:17 +01:00
ectest.c Use the new non-curve type specific EC functions internally 2018-07-31 09:08:38 +01:00
enginetest.c Add EVP_PKEY_METHOD redirection test 2017-10-12 00:03:32 +01:00
errtest.c Save and restore the Windows error around TlsGetValue. 2018-05-23 17:34:54 -04:00
evp_extra_test.c test/evp_extra_test.c: fix null pointer dereference 2018-09-10 11:11:43 +01:00
evp_test.c Trivial test improvements 2018-09-18 09:22:10 +02:00
evp_test.h Add support for multiple update calls in evp_test 2017-05-19 21:02:24 +01:00
exdatatest.c Update copyright year 2018-05-29 13:16:04 +01:00
exptest.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
fatalerrtest.c Update copyright year 2018-03-20 13:08:46 +00:00
generate_buildtest.pl Update copyright year 2018-05-29 13:16:04 +01:00
generate_ssl_tests.pl Consolidate the locations where we have our internal perl modules 2017-08-15 11:30:47 +02:00
gmdifftest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
gosttest.c Add a GOST test 2018-07-13 18:14:43 +01:00
handshake_helper.c Change Post Handshake auth so that it is opt-in 2018-08-20 15:14:01 +01:00
handshake_helper.h Update copyright year 2018-03-20 13:08:46 +00:00
hmactest.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
ideatest.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
igetest.c Consistent formatting for sizeof(foo) 2017-12-07 19:11:49 -05:00
lhash_test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
md2test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
mdc2_internal_test.c Update copyright year 2018-05-29 13:16:04 +01:00
mdc2test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
memleaktest.c Update secmemtest and memeleaktest to use the test infrastructure. 2017-04-12 10:59:53 +01:00
modes_internal_test.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
ocspapitest.c Wrap more of ocspapitest.c in OPENSSL_NO_OCSP 2017-12-08 09:16:36 -06:00
P1ss.cnf
P2ss.cnf
packettest.c Update copyright year 2018-03-20 13:08:46 +00:00
pbelutest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
pemtest.c Update copyright year 2018-09-11 13:45:17 +01:00
pkcs7-1.pem
pkcs7.pem
pkey_meth_kdf_test.c Update copyright year 2018-05-29 13:16:04 +01:00
pkey_meth_test.c Update copyright year 2018-05-29 13:16:04 +01:00
pkits-test.pl Many spelling fixes/typo's corrected. 2017-11-11 19:03:10 -05:00
poly1305_internal_test.c Update copyright year 2018-02-13 13:59:25 +00:00
rc2test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
rc4test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
rc5test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
rdrand_sanitytest.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
README Fix test documentation. 2017-09-08 13:58:59 -05:00
README.external Many spelling fixes/typo's corrected. 2017-11-11 19:03:10 -05:00
README.ssltest.md Session resume broken switching contexts 2017-10-04 10:21:08 +10:00
recordlentest.c Update copyright year 2018-03-20 13:08:46 +00:00
rsa_complex.c Add a compile time test to verify that openssl/rsa.h and complex.h can 2018-09-17 09:14:03 +10:00
rsa_mp_test.c rsa/rsa_gen.c: harmonize keygen's ability with RSA_security_bits. 2017-11-28 20:05:48 +01:00
rsa_test.c Use "" not <> for internal/ includes 2017-08-22 09:54:20 -04:00
run_tests.pl Update copyright year 2018-05-01 13:34:30 +01:00
sanitytest.c Update copyright year 2018-09-11 13:45:17 +01:00
secmemtest.c Update copyright year 2018-09-11 13:45:17 +01:00
serverinfo2.pem Add a SERVERINFOV2 format test file 2017-05-03 14:37:42 +01:00
serverinfo.pem
servername_test.c Update copyright year 2018-09-11 13:45:17 +01:00
session.pem Don't store the ticket nonce in the session 2018-06-07 10:58:35 +01:00
shibboleth.pfx
shlibloadtest.c Extend dladdr() for AIX, consequence from changes for openssl#6368. 2018-08-22 21:50:33 +02:00
siphash_internal_test.c SipHash: add separate setter for the hash size 2018-09-09 01:47:56 +02:00
sm2_internal_test.c Make SM2 ID stick to specification 2018-09-07 18:12:26 +08:00
sm4_internal_test.c SM4: Add SM4 block cipher to EVP 2017-10-31 15:19:14 +10:00
smcont.txt
srptest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
ssl_cert_table_internal_test.c VMS: turn on name mangling for all our programs 2018-09-14 11:00:38 +02:00
ssl_test_ctx_test.c Update copyright year 2018-03-20 13:08:46 +00:00
ssl_test_ctx_test.conf Implement Maximum Fragment Length TLS extension. 2017-11-05 17:46:48 +01:00
ssl_test_ctx.c Change Post Handshake auth so that it is opt-in 2018-08-20 15:14:01 +01:00
ssl_test_ctx.h Change Post Handshake auth so that it is opt-in 2018-08-20 15:14:01 +01:00
ssl_test.c Add a config option to disable automatic config loading 2018-04-17 16:33:15 +02:00
ssl_test.tmpl
sslapitest.c Fix no-tls1_2 2018-09-25 16:08:28 +01:00
sslbuffertest.c Update copyright year 2018-03-20 13:08:46 +00:00
sslcorrupttest.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
ssltest_old.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
ssltestlib.c Add a test for the certificate callback 2018-09-21 17:39:14 +01:00
ssltestlib.h Add a bi-directional shutdown test 2018-06-27 10:03:20 +01:00
Sssdsa.cnf
Sssrsa.cnf
stack_test.c Add a reserve call to the stack data structure. 2017-09-28 06:53:40 +10:00
sysdefault.cnf Apply system_default configuration on SSL_CTX_new(). 2018-03-19 10:22:49 -04:00
sysdefaulttest.c Update copyright year 2018-03-20 13:08:46 +00:00
test_test.c Relocate memcmp test. 2018-08-07 10:51:01 +10:00
test.cnf
testcrl.pem
testdsa.pem
testdsapub.pem
testec-p256.pem
testecpub-p256.pem
testp7.pem
testreq2.pem
testrsa.pem
testrsapub.pem
testsid.pem
testutil.h Test support for time_t comparisons. 2017-11-28 08:56:45 +10:00
testx509.pem
threadstest.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
time_offset_test.c Update copyright year 2018-05-29 13:16:04 +01:00
tls13ccstest.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
tls13encryptiontest.c VMS: turn on name mangling for all our programs 2018-09-14 11:00:38 +02:00
tls13secretstest.c VMS: turn on name mangling for all our programs 2018-09-14 11:00:38 +02:00
uitest.c [Win] Fix some test method signatures ... 2017-08-16 10:36:34 -04:00
Uss.cnf
v3-cert1.pem
v3-cert2.pem
v3ext.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
v3nametest.c Use void in all function definitions that do not take any arguments 2018-05-11 14:37:48 +02:00
verify_extra_test.c Update copyright year 2018-05-01 13:34:30 +01:00
versions.c Refuse to run test_cipherlist unless shared library matches build 2018-03-31 16:40:07 +02:00
wpackettest.c VMS: turn on name mangling for all our programs 2018-09-14 11:00:38 +02:00
x509_check_cert_pkey_test.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
x509_dup_cert_test.c Update the test framework so that the need for test_main is removed. Everything 2017-07-27 07:53:08 +10:00
x509_internal_test.c VMS: turn on name mangling for all our programs 2018-09-14 11:00:38 +02:00
x509_time_test.c Update copyright year 2018-05-29 13:16:04 +01:00
x509aux.c Update copyright year 2018-06-20 15:29:23 +01:00

How to add recipes
==================

For any test that you want to perform, you write a script located in
test/recipes/, named {nn}-test_{name}.t, where {nn} is a two digit number and
{name} is a unique name of your choice.

Please note that if a test involves a new testing executable, you will need to
do some additions in test/Makefile.  More on this later.


Naming conventions
=================

A test executable is named test/{name}test.c

A test recipe is named test/recipes/{nn}-test_{name}.t, where {nn} is a two
digit number and {name} is a unique name of your choice.

The number {nn} is (somewhat loosely) grouped as follows:

00-04  sanity, internal and essential API tests
05-09  individual symmetric cipher algorithms
10-14  math (bignum)
15-19  individual asymmetric cipher algorithms
20-24  openssl commands (some otherwise not tested)
25-29  certificate forms, generation and verification
30-35  engine and evp
60-79  APIs
   70  PACKET layer
80-89  "larger" protocols (CA, CMS, OCSP, SSL, TSA)
90-98  misc
99     most time consuming tests [such as test_fuzz]


A recipe that just runs a test executable
=========================================

A script that just runs a program looks like this:

    #! /usr/bin/perl

    use OpenSSL::Test::Simple;

    simple_test("test_{name}", "{name}test", "{name}");

{name} is the unique name you have chosen for your test.

The second argument to `simple_test' is the test executable, and `simple_test'
expects it to be located in test/

For documentation on OpenSSL::Test::Simple, do
`perldoc util/perl/OpenSSL/Test/Simple.pm'.


A recipe that runs a more complex test
======================================

For more complex tests, you will need to read up on Test::More and
OpenSSL::Test.  Test::More is normally preinstalled, do `man Test::More' for
documentation.  For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm'.

A script to start from could be this:

    #! /usr/bin/perl

    use strict;
    use warnings;
    use OpenSSL::Test;

    setup("test_{name}");

    plan tests => 2;                # The number of tests being performed

    ok(test1, "test1");
    ok(test2, "test1");

    sub test1
    {
        # test feature 1
    }

    sub test2
    {
        # test feature 2
    }


Changes to test/build.info
==========================

Whenever a new test involves a new test executable you need to do the
following (at all times, replace {NAME} and {name} with the name of your
test):

* add {name} to the list of programs under PROGRAMS_NO_INST

* create a three line description of how to build the test, you will have
to modify the include paths and source files if you don't want to use the
basic test framework:

    SOURCE[{name}]={name}.c
    INCLUDE[{name}]=.. ../include
    DEPEND[{name}]=../libcrypto libtestutil.a

Generic form of C test executables
==================================

    #include "testutil.h"

    static int my_test(void)
    {
        int testresult = 0;                 /* Assume the test will fail    */
        int observed;

        observed = function();              /* Call the code under test     */
        if (!TEST_int_equal(observed, 2))   /* Check the result is correct  */
            goto end;                       /* Exit on failure - optional   */

        testresult = 1;                     /* Mark the test case a success */
    end:
        cleanup();                          /* Any cleanup you require      */
        return testresult;
    }

    int setup_tests(void)
    {
        ADD_TEST(my_test);                  /* Add each test separately     */
        return 1;                           /* Indicate success             */
    }

You should use the TEST_xxx macros provided by testutil.h to test all failure
conditions.  These macros produce an error message in a standard format if the
condition is not met (and nothing if the condition is met).  Additional
information can be presented with the TEST_info macro that takes a printf
format string and arguments.  TEST_error is useful for complicated conditions,
it also takes a printf format string and argument.  In all cases the TEST_xxx
macros are guaranteed to evaluate their arguments exactly once.  This means
that expressions with side effects are allowed as parameters.  Thus,

    if (!TEST_ptr(ptr = OPENSSL_malloc(..)))

works fine and can be used in place of:

    ptr = OPENSSL_malloc(..);
    if (!TEST_ptr(ptr))

The former produces a more meaningful message on failure than the latter.