Fixes an issue where, when the test suite was being run with the older
Test::Harness package, the test suite would not complete correctly due
to evaluation of $harness->runtests()->has_errors, which is only
available for the newer TAP::Parser::Aggregator code path.
Fixes#17818.
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/17823)
Set the environment variable MALLOC_PERTURB_ during tests to perturb the
output from OPENSSL_malloc() calls (see the mallopt man page for details
about this environment variable). This could be a low cost way of spotting
uninit reads in "make test" runs in some situations.
In tests I have found it to be a little unreliable (sometimes it seemed to
not perturb the output for inexplicable reasons) - but since this is easy
to implement I think it is worthwhile.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/17768)
The exit code for VMS is a bit tricky, and while perl translates the
VMS status code from a typical C program to posix terms, it doesn't
automatically translate its exit code into the typical C program VMS
status code. Perl scripts are recommended to do so explicitly.
Therefore, we make util/wrap.pl and test/run_tests.pl simulate the
typical C program VMS status code for all non-zero exit codes, except
we give them all the error severity (according to the VMS C library
reference manual, exit codes 2 and above are treated as success...).
Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15787)
We add the concept of preparation recipes, which are performed
unconditionally. They are all expected to match the pattern
test/recipes/00-prep_*.t.
We add one such preparation recipe, test/recipes/00-prep_fipsmodule_cnf.t,
which helps us generate a test specific fipsmodule.cnf, to be used by
all other tests.
Fixes#15166
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15436)
The existing code prints a warning saying that verbose mode is ignored with
parallel jobs. This seems backward, more useful is disabling parallel jobs
when verbose is enabled.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/14137)
In OpenSSL 1.1.1 the script run_tests.pl has an effectiver
workaround to fall back to Test::Harness, if TAP::Harness
is not available. That code has substantially changed,
but it seems it should still fall back but doesn't.
Observed on SuSE Linux Enterprise Server 11 (SLES11).
Error messages:
Can't locate TAP/Parser.pm in @inc (@inc contains: /path/to/bld/openssl300/test/../util/perl /path/to/local/perl/lib/perl5 /usr/lib/perl5/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/5.10.0 /usr/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.10.0 /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.10.0 /usr/lib/perl5/vendor_perl .) at /path/to/local/perl/lib/perl5/parent.pm line 20.
BEGIN failed--compilation aborted at /path/to/bld/openssl300/test/run_tests.pl line 131.
and
Can't locate TAP/Harness.pm in @inc (@inc contains: /path/to/bld/openssl300/test/../util/perl /path/to/local/perl/lib/perl5 /usr/lib/perl5/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/5.10.0 /usr/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.10.0 /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.10.0 /usr/lib/perl5/vendor_perl .) at /path/to/local/perl/lib/perl5/parent.pm line 20.
BEGIN failed--compilation aborted at /path/to/bld/openssl300/test/run_tests.pl line 215.
Concerning the fix: the docs for parent.pm show, that without
the "-norequire" it puts the require statement in a BEGIN block
which probably runs before the eval, to the loading is no
longer encapsulated by the eval. Without the additional require
line, the loading doesn't happen at all, so the availability
testing fails. Combining the "-norequire" and an explicit
"require" worked for me.
Tested on the original problem platform SLES 11, but also on
SLES 12 and 15, RHEL 6, 7 and 8 plus Solaris 10 Sparc.
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/12500)
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/12762)
Make the improved semantics of VFO replace the previous VF and remove VFO
Add warnings about overriding use of HARNESS_VERBOSE* variables
Reviewed-by: Paul Dale <paul.dale@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/12279)
The environment variable `HARNESS_JOBS` can be used to control how many
jobs to run in parallel. The default is still to run jobs sequentially.
This commit does not define custom `rules`, and different versions of
`TAP::Harness` come with different strategies regarding the default
`rules` that define which test recipes can be run in parallel.
In recent versions of Perl, unless specified otherwise any task can be
run in parallel.
Reviewed-by: David von Oheimb <david.von.oheimb@siemens.com>
(Merged from https://github.com/openssl/openssl/pull/12326)
VERBOSE_FAILURES_ONLY (VFO): verbose output only of failed (sub-)tests
VERBOSE_FAILURES_PROGRESS (VFP): in addition summary for passed tests
This adds a workaroud for TAP::Parser not coping well with indentation.
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/12088)
The test_includes test was failing if OPENSSL_CONF_INCLUDE happened to
be set in the user's environment. To ensure that no tests accidentally
use this or other enviroment variables from the user's environment we
automatically set them centrally for all tests.
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/11691)
This includes a complete rework of how we use TAP::Harness, by adding
a TAP::Parser subclass that allows additional callbacks to be passed
to perform what we need. The TAP::Parser callbacks we add are:
ALL to print all the TAP output to a file (conditionally)
to collect all the TAP output to an array (conditionally)
EOF to print all the collected TAP output (if there is any)
if any subtest failed
To get TAP output to file, the environment variable HARNESS_TAP_COPY
must be defined, with a file name as value. That file will be
overwritten unconditionally.
To get TAP output displayed on failure, the make variable VERBOSE_FAILURE
or VF must be defined with a non-emoty value.
Additionally, the output of test recipe names has been changed to only
display its basename.
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9862)
Ironically enough not all installations get Module::Load::Conditional
installed by default... [It's a bit half-hearted, because such
installations are likely to lack more stuffi that is needed, but
nevertheless, it proved to be helpful.]
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4323)
Instead of having perl modules under test/testlib, util and util/perl,
consolidate them all to be inside util/perl.
Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4069)
File::Glob::glob is deprecated, it's use generates this kind of
message:
File::Glob::glob() will disappear in perl 5.30. Use File::Glob::bsd_glob() instead. at ../master/Configure line 277.
So instead, use a construction that makes the caller glob() use
File::Glob::bsd_glob().
Note that we're still excluding VMS, as it's directory specs use '['
and ']', which have a different meaning with bsd_glob and would need
some extra quoting. This might change, but later.
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/4040)
TAP::Parser::Aggregator::has_errors may return any number, not just 0
and 1. With Perl on VMS, any number from 2 and on is interpreted as a
VMS status, the 3 lower bits are the encoded severity (1 = SUCCESS,
for example), so depending on what has_errors returns, a test failure
might be interpreted as a success. Therefore, it's better to make
sure the exit code is 0 or 1, nothing else (they are special on VMS,
and mean SUCCESS or FAILURE, to match Unix conventions).
Reviewed-by: Tim Hudson <tjh@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3880)
Switch to TAP::Harness inadvertently masked test failures.
Test::Harness::runtests was terminating with non-zero exit code in case
of failure[s], while TAP::Harness apparently holds caller responsible
for doing so.
Reviewed-by: Rich Salz <rsalz@openssl.org>
TAP:Harness came along in perl 5.10.1, and since we claim to support
perl 5.10.0 in configuration and testing, we can only load it
conditionally.
The main reason to use TAP::Harness rather than Test::Harness is its
capability to merge stdout and stderr output from the test recipes,
which Test::Harness can't. The merge gives much more comprehensible
output when testing verbosely.
Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/3424)
Examples of possible expressions (adapt to your platform):
make test TESTS=-99
make test TESTS=10
make test TESTS=-9?
make test TESTS=-[89]0
make test TESTS=[89]0
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/2887)
So far, the test runner (test/run_tests.pl) could get a list of tests
to run, and if non were given, it assumes all available tests should
be performed.
However, that makes skipping just one or two tests a bit of a pain.
This change makes the possibilities more versatile, run_checker.pl
takes these arguments and will process them in the given order,
starting with an empty set of tests to perform:
alltests The current set becomes the whole set of
available tests.
test_xxx Adds 'test_xxx' to the current set.
-test_xxx Removes 'test_xxx' from the current set. If
nothing has been added to the set before this
argument, the current set is first initialised
to the whole set of available tests, then
'test_xxx' is removed from the current set.
list Display all available tests, then stop.
If no arguments are given, 'alltests' is assumed.
Reviewed-by: Emilia Käsper <emilia@openssl.org>
Trying to use normal perl conditions to conditionally 'use' a perl
module didn't quite work. Using the 'if' module to do so does work.
Reviewed-by: Andy Polyakov <appro@openssl.org>
As it turns out default glob's behaviour for quoted argument varies
from version to version, making it impossible to Configure or run
tests in some cases. The reason for quoting globs was to accommodate
source path with spaces in its name, which was treated by default glob
as multiple paths. File::Glob::glob on the other hand doesn't consider
spaces as delimiters and therefore works with unquoted patterns.
[Unfortunaltely File::Glob::glob, being too csh-ly, doesn't work
on VMS, hence the "pinning" is conditional.]
Reviewed-by: Richard Levitte <levitte@openssl.org>
The old cipherlist test in ssltest.c only tests the internal order of
the cipher table, which is pretty useless.
Replace this test with a test that catches inadvertent changes to the
default cipherlist.
Fix run_tests.pl to correctly filter tests that have "list" in their name.
(Also includes a small drive-by fix in .gitignore.)
Reviewed-by: Rich Salz <rsalz@openssl.org>
Add copyright to most .pl files
This does NOT cover any .pl file that has other copyright in it.
Most of those are Andy's but some are public domain.
Fix typo's in some existing files.
Reviewed-by: Richard Levitte <levitte@openssl.org>
To be able to run tests when we've built in a directory other than
the source tree, the testing framework needs a few adjustments.
test/testlib/OpenSSL/Test.pm needs to know where it can find
shlib_wrap.sh, and a number of other tests need to be told a different
place to find engines than what they may be able to figure out on
their own. Relying to $TOP is not enough, $SRCTOP and $BLDTOP can be
used as an alternative.
As part of this change, top_file and top_dir are removed and
srctop_file, bldtop_file, srctop_dir and bldtop_dir take their place.
Reviewed-by: Ben Laurie <ben@openssl.org>
The idea with this perl based testing framework is to make use of
what's delivered with perl and exists on all sorts of platforms.
The choice came to using Test::More and Test::Harness, as that seems
to be the most widely spread foundation, even if perl is aged.
The main runner of the show is run_tests.pl. As it currently stands,
it's designed to run from inside Makefile, but it's absolutely
possible to run it from the command line as well, like so:
cd test
OPENSSL_SRCDIR=.. perl run_tests.pl
The tester scripts themselves are stored in the subdirectory recipes/,
and initially, we have two such scripts, recipes/00-check_testalltests.t
and recipes/00-check_testexes.t. recipes/00-check_testalltests.t will
pick out the dependencies of "alltests" in test/Makefile, and check if
it can find recipes with corresponding names. recipes/00-check_testexes.t
does something similar, but bases it on existing compiled test binaries.
They make it easy to figure out what's to be added, and will be
removed when this effort is finished.
Individual recipes can be run as well, of course, as they are perl
scripts in themselves. For example, you can run only
recipes/00-check_testexes.t like so:
cd test
OPENSSL_SRCDIR=.. perl recipes/00-check_testexes.t
To make coding easier, there's a routine library OpenSSL::Test, which
is reachable in a perl script like so:
use lib 'testlib';
use OpenSSL::Test;
Reviewed-by: Rich Salz <rsalz@openssl.org>
