openssl/test
Richard Levitte 51a7c4b5f2 TEST: Add test recipe and help program to test BIO_f_prefix()
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/10531)
2019-12-18 19:42:44 +01:00
..
certs Add Restricted PSS certificate and key 2019-08-09 13:19:16 +01:00
ct
d2i-tests
ocsp-tests
ossl_shim Replace BUF_ string function calls with OPENSSL_ ones 2019-10-17 23:19:59 +02:00
recipes TEST: Add test recipe and help program to test BIO_f_prefix() 2019-12-18 19:42:44 +01:00
smime-certs Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
ssl-tests Add TLS tests for RSA-PSS Restricted certificates 2019-08-09 13:19:16 +01:00
testutil Deprecate most of debug-memory 2019-12-14 20:57:35 +01:00
aborttest.c
aesgcmtest.c Add EVP_CIPHER_CTX_tag_length() 2019-09-11 17:52:30 +10:00
afalgtest.c
asn1_decode_test.c Update source files for deprecation at 3.0 2019-11-07 11:37:25 +01:00
asn1_dsa_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
asn1_encode_test.c Update source files for deprecation at 3.0 2019-11-07 11:37:25 +01:00
asn1_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
asn1_string_table_test.c
asn1_time_test.c
asynciotest.c
asynctest.c Deprecate most of debug-memory 2019-12-14 20:57:35 +01:00
bad_dtls_test.c
bftest.c
bio_callback_test.c
bio_enc_test.c
bio_memleak_test.c Deprecate most of debug-memory 2019-12-14 20:57:35 +01:00
bio_prefix_text.c TEST: Add test recipe and help program to test BIO_f_prefix() 2019-12-18 19:42:44 +01:00
bioprinttest.c
bn_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
bn_rand_range.h
bntest.c Fix some typos 2019-12-11 19:04:01 +01:00
bntests.pl
build.info TEST: Add test recipe and help program to test BIO_f_prefix() 2019-12-18 19:42:44 +01:00
CAss.cnf Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
CAssdh.cnf Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
CAssdsa.cnf Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
CAssrsa.cnf Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
casttest.c
CAtsa.cnf Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
chacha_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
cipher_overhead_test.c Reorganize local header files 2019-09-28 20:26:35 +02:00
cipherbytes_test.c
cipherlist_test.c
ciphername_test.c
clienthellotest.c Test SSL_set_ciphersuites 2019-08-15 14:33:06 +01:00
cmp_asn_test.c chunk 6 of CMP contribution to OpenSSL 2019-12-12 10:57:25 +00:00
cmp_ctx_test.c chunk 6 of CMP contribution to OpenSSL 2019-12-12 10:57:25 +00:00
cmp_hdr_test.c chunk 6 of CMP contribution to OpenSSL 2019-12-12 10:57:25 +00:00
cmp_msg_test.c chunk 6 of CMP contribution to OpenSSL 2019-12-12 10:57:25 +00:00
cmp_protect_test.c chunk 6 of CMP contribution to OpenSSL 2019-12-12 10:57:25 +00:00
cmp_status_test.c chunk 5 of CMP contribution to OpenSSL 2019-10-29 14:17:39 +00:00
cmp_testlib.c Certificate Management Protocol (CMP, RFC 4210) extension to OpenSSL 2019-09-27 10:53:11 +01:00
cmp_testlib.h chunk 6 of CMP contribution to OpenSSL 2019-12-12 10:57:25 +00:00
cms-examples.pl
cmsapitest.c
conf_include_test.c Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
confdump.c Make sure we free the CONF structure allocated by confdump 2019-11-15 11:08:18 +00:00
constant_time_test.c Reorganize local header files 2019-09-28 20:26:35 +02:00
context_internal_test.c
crltest.c
ct_test.c Extend tests of SSL_check_chain() 2019-08-09 17:29:39 +01:00
ctype_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
curve448_internal_test.c Reorganize local header files 2019-09-28 20:26:35 +02:00
d2i_test.c
danetest.c Adapt two test programs that were using now deprecated functions 2019-11-03 18:38:23 +01:00
danetest.in
danetest.pem
default-and-fips.cnf Add fips module integrity check 2019-09-15 19:55:10 +10:00
default-and-legacy.cnf test/recipes/30-test_evp.t: Modify to test with different providers 2019-07-26 18:14:41 +02:00
default.cnf test/recipes/30-test_evp.t: Modify to test with different providers 2019-07-26 18:14:41 +02:00
destest.c
dhtest.c Check the DH modulus bit length 2019-09-09 14:43:57 +02:00
drbg_cavs_data_ctr.c
drbg_cavs_data_hash.c
drbg_cavs_data_hmac.c
drbg_cavs_data.h Fix header file include guard names 2019-09-28 20:26:36 +02:00
drbg_cavs_test.c Reorganize local header files 2019-09-28 20:26:35 +02:00
drbgtest.c Document "get/set-app-data" macros. 2019-10-23 08:31:21 -04:00
drbgtest.h
dsa_no_digest_size_test.c
dsatest.c
dtls_mtu_test.c Reorganize local header files 2019-09-28 20:26:35 +02:00
dtlstest.c
dtlsv1listentest.c
ec_internal_test.c Reorganize local header files 2019-09-28 20:26:35 +02:00
ecdsatest.c
ecdsatest.h Fix header file include guard names 2019-09-28 20:26:36 +02:00
ecstresstest.c
ectest.c Add self-test for EC_POINT_hex2point 2019-11-13 18:02:51 +02:00
enginetest.c
errtest.c Modernise the ERR functionality further (new functions and deprecations) 2019-09-12 17:59:52 +02:00
evp_extra_test.c Test that EVP_PKEY_set1_DH() correctly identifies the DH type 2019-12-16 14:27:26 +00:00
evp_fetch_prov_test.c Add fips module integrity check 2019-09-15 19:55:10 +10:00
evp_kdf_test.c Fix no-cmac and no-camellia 2019-11-14 09:44:18 +00:00
evp_pkey_dparams_test.c
evp_pkey_provided_test.c TEST: add tests of text and PEM printout of a provider made key 2019-11-29 20:55:16 +01:00
evp_test.c Fix Use after free when copying cipher ctx 2019-11-18 13:13:05 +10:00
evp_test.h
exdatatest.c
exptest.c
fatalerrtest.c
fips.cnf Add fips module integrity check 2019-09-15 19:55:10 +10:00
generate_buildtest.pl
generate_ssl_tests.pl Rework the perl fallback functionality 2019-09-12 12:49:31 +02:00
gmdifftest.c
gosttest.c
handshake_helper.c Explicitly test against NULL; do not use !p or similar 2019-10-09 21:32:15 +02:00
handshake_helper.h Fix header file include guard names 2019-09-28 20:26:36 +02:00
hmactest.c
ideatest.c
igetest.c Deprecate the AES_ige_*() functions 2019-12-04 17:46:38 +00:00
keymgmt_internal_test.c CORE & PROV: make export of key data leaner through callback 2019-11-14 10:53:14 +01:00
legacy.cnf test/recipes/30-test_evp.t: Modify to test with different providers 2019-07-26 18:14:41 +02:00
lhash_test.c
md2test.c
mdc2_internal_test.c
mdc2test.c Reorganize public header files (part 1) 2019-09-28 20:26:36 +02:00
memleaktest.c test/memleaktest.c: Modify for use with address/leak sanitizer 2019-12-10 14:16:12 +01:00
modes_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
namemap_internal_test.c test/namemap_internal_test.c: use "cookie" instead of "foo"... 2019-12-13 10:03:57 +01:00
ocspapitest.c
ossl_test_endian.h Fix header file include guard names 2019-09-28 20:26:36 +02:00
P1ss.cnf Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
P2ss.cnf Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
p_test.c Rename provider and core get_param_types functions 2019-08-15 11:58:25 +02:00
packettest.c
param_build_test.c
params_api_test.c test/params_api_test.c: Correct the checks of OSSL_PARAM_set_BN() 2019-11-03 11:19:04 +01:00
params_conversion_test.c
params_test.c
pbelutest.c
pemtest.c
pkcs7-1.pem
pkcs7.pem
pkey_meth_kdf_test.c
pkey_meth_test.c
pkits-test.pl
poly1305_internal_test.c Add ChaCha related ciphers to default provider 2019-10-16 16:18:42 +10:00
property_test.c Properties: make query cache reference count aware. 2019-11-18 18:51:26 +10:00
provider_internal_test.c Load the config file by default 2019-08-01 09:59:20 +01:00
provider_internal_test.conf.in
provider_test.c
rc2test.c
rc4test.c
rc5test.c
rdrand_sanitytest.c Remove extern declarations of OPENSSL_ia32cap_P 2019-09-01 15:41:58 +02:00
README TEST: Add test recipe and help program to test BIO_f_prefix() 2019-12-18 19:42:44 +01:00
README.external
README.ssltest.md
recordlentest.c
rsa_complex.c
rsa_mp_test.c Added internal functions for easy getting and setting all RSA parameters. 2019-10-17 18:07:45 +02:00
rsa_sp800_56b_test.c RSA generation: Use more bits of 1/sqrt(2) 2019-11-09 16:01:54 +01:00
rsa_test.c
run_tests.pl Rework test/run_tests.pl to support selective verbosity and TAP copy 2019-09-12 14:38:00 +02:00
sanitytest.c
secmemtest.c
serverinfo2.pem
serverinfo.pem
servername_test.c
session.pem
shibboleth.pfx
shlibloadtest.c Reorganize public header files (part 1) 2019-09-28 20:26:36 +02:00
siphash_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
sm2_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
sm4_internal_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
smcont.txt
sparse_array_test.c Reorganize private crypto header files 2019-09-28 20:26:34 +02:00
srptest.c
ssl_cert_table_internal_test.c Reorganize local header files 2019-09-28 20:26:35 +02:00
ssl_ctx_test.c Add ssl_ctx_test to test suite. 2019-11-08 08:23:15 +01:00
ssl_test_ctx_test.c
ssl_test_ctx_test.conf
ssl_test_ctx.c
ssl_test_ctx.h Fix header file include guard names 2019-09-28 20:26:36 +02:00
ssl_test.c
ssl_test.tmpl
sslapitest.c Deprecate most of debug-memory 2019-12-14 20:57:35 +01:00
sslbuffertest.c Deprecate most of debug-memory 2019-12-14 20:57:35 +01:00
sslcorrupttest.c Replace BUF_ string function calls with OPENSSL_ ones 2019-10-17 23:19:59 +02:00
ssltest_old.c Deprecate most of debug-memory 2019-12-14 20:57:35 +01:00
ssltestlib.c
ssltestlib.h Fix header file include guard names 2019-09-28 20:26:36 +02:00
Sssdsa.cnf Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
Sssrsa.cnf Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
stack_test.c
sysdefault.cnf
sysdefaulttest.c
test_test.c Fix --strict-warnings build 2019-08-18 21:45:16 +02:00
test.cnf Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
testcrl.pem
testdsa.pem
testdsapub.pem
testec-p256.pem
testecpub-p256.pem
tested448.pem More testing for CLI usage of Ed25519 and Ed448 keys 2019-12-11 18:37:53 +01:00
tested448pub.pem More testing for CLI usage of Ed25519 and Ed448 keys 2019-12-11 18:37:53 +01:00
tested25519.pem More testing for CLI usage of Ed25519 and Ed448 keys 2019-12-11 18:37:53 +01:00
tested25519pub.pem More testing for CLI usage of Ed25519 and Ed448 keys 2019-12-11 18:37:53 +01:00
testp7.pem
testreq2.pem
testrsa.pem
testrsapub.pem
testsid.pem
testutil.h Fix header file include guard names 2019-09-28 20:26:36 +02:00
testx509.pem
threadstest.c
time_offset_test.c
tls13ccstest.c
tls13encryptiontest.c Reorganize local header files 2019-09-28 20:26:35 +02:00
tls13secretstest.c Reorganize local header files 2019-09-28 20:26:35 +02:00
uitest.c
Uss.cnf Remove RANDFILE settings from configuration files 2019-11-24 08:35:14 +01:00
v3-cert1.pem
v3-cert2.pem
v3ext.c
v3nametest.c
verify_extra_test.c
versions.c
wpackettest.c
x509_check_cert_pkey_test.c
x509_dup_cert_test.c
x509_internal_test.c
x509_time_test.c add X509_cmp_timeframe() including its documentation 2019-12-04 15:17:55 +00:00
x509aux.c

How to add recipes
==================

For any test that you want to perform, you write a script located in
test/recipes/, named {nn}-test_{name}.t, where {nn} is a two digit number and
{name} is a unique name of your choice.

Please note that if a test involves a new testing executable, you will need to
do some additions in test/build.info. Please refer to the section "Changes to 
test/build.info" below.


Naming conventions
=================

A test executable is named test/{name}test.c

A test recipe is named test/recipes/{nn}-test_{name}.t, where {nn} is a two
digit number and {name} is a unique name of your choice.

The number {nn} is (somewhat loosely) grouped as follows:

00-04  sanity, internal and essential API tests
05-09  individual symmetric cipher algorithms
10-14  math (bignum)
15-19  individual asymmetric cipher algorithms
20-24  openssl commands (some otherwise not tested)
25-29  certificate forms, generation and verification
30-35  engine and evp
60-79  APIs:
   60  X509 subsystem
   61  BIO subsystem
   65  CMP subsystem
   70  PACKET layer
80-89  "larger" protocols (CA, CMS, OCSP, SSL, TSA)
90-98  misc
99     most time consuming tests [such as test_fuzz]


A recipe that just runs a test executable
=========================================

A script that just runs a program looks like this:

    #! /usr/bin/perl

    use OpenSSL::Test::Simple;

    simple_test("test_{name}", "{name}test", "{name}");

{name} is the unique name you have chosen for your test.

The second argument to `simple_test' is the test executable, and `simple_test'
expects it to be located in test/

For documentation on OpenSSL::Test::Simple, do
`perldoc util/perl/OpenSSL/Test/Simple.pm'.


A recipe that runs a more complex test
======================================

For more complex tests, you will need to read up on Test::More and
OpenSSL::Test.  Test::More is normally preinstalled, do `man Test::More' for
documentation.  For OpenSSL::Test, do `perldoc util/perl/OpenSSL/Test.pm'.

A script to start from could be this:

    #! /usr/bin/perl

    use strict;
    use warnings;
    use OpenSSL::Test;

    setup("test_{name}");

    plan tests => 2;                # The number of tests being performed

    ok(test1, "test1");
    ok(test2, "test1");

    sub test1
    {
        # test feature 1
    }

    sub test2
    {
        # test feature 2
    }


Changes to test/build.info
==========================

Whenever a new test involves a new test executable you need to do the
following (at all times, replace {NAME} and {name} with the name of your
test):

* add {name} to the list of programs under PROGRAMS_NO_INST

* create a three line description of how to build the test, you will have
to modify the include paths and source files if you don't want to use the
basic test framework:

    SOURCE[{name}]={name}.c
    INCLUDE[{name}]=.. ../include ../apps/include
    DEPEND[{name}]=../libcrypto libtestutil.a

Generic form of C test executables
==================================

    #include "testutil.h"

    static int my_test(void)
    {
        int testresult = 0;                 /* Assume the test will fail    */
        int observed;

        observed = function();              /* Call the code under test     */
        if (!TEST_int_eq(observed, 2))      /* Check the result is correct  */
            goto end;                       /* Exit on failure - optional   */

        testresult = 1;                     /* Mark the test case a success */
    end:
        cleanup();                          /* Any cleanup you require      */
        return testresult;
    }

    int setup_tests(void)
    {
        ADD_TEST(my_test);                  /* Add each test separately     */
        return 1;                           /* Indicate success             */
    }

You should use the TEST_xxx macros provided by testutil.h to test all failure
conditions.  These macros produce an error message in a standard format if the
condition is not met (and nothing if the condition is met).  Additional
information can be presented with the TEST_info macro that takes a printf
format string and arguments.  TEST_error is useful for complicated conditions,
it also takes a printf format string and argument.  In all cases the TEST_xxx
macros are guaranteed to evaluate their arguments exactly once.  This means
that expressions with side effects are allowed as parameters.  Thus,

    if (!TEST_ptr(ptr = OPENSSL_malloc(..)))

works fine and can be used in place of:

    ptr = OPENSSL_malloc(..);
    if (!TEST_ptr(ptr))

The former produces a more meaningful message on failure than the latter.