* lib/autotest/general.m4 (AT_INIT) <at_fn_group_postprocess>:
Exit after an unexpected passing test if $at_errexit.
* tests/autotest.at (errexit): Also try tests that xpass, skip,
xfail, or fail hard.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* lib/autotest/general.m4 (AT_INIT) <at_fn_group_postprocess>:
Don't cleanup the group directory when a test unexpectedly passes.
* tests/autotest.at (Cleanup): Check that an unexpected pass leaves
the test group directory intact.
Signed-off-by: Peter Rosin <peda@lysator.liu.se>
* tests/autotest.at (colored test results): Use exit not
Exit. Fixes test failure with dash 0.5.4.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* tests/autotest.at (parallel args but non-working mkfifo):
New test, to expose the failure v2.66-23-g991183c avoided.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* doc/autoconf.texi (Writing Testsuites) <AT_BANNER>: Update
description.
* lib/autotest/general.m4 (AT_INIT) <at_fn_banner>: Set banner
to single space, not empty line, once printed. For empty
banners, print a single empty line to separate them from a
previous test group category.
* tests/autotest.at (Banners): Insert another test group; adjust
tests accordingly. Extend test to cover semantic change.
* NEWS: Update.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* lib/autotest/general.m4 (_AT_ARG_OPTION): Fix translation of
hyphens to underscores when turning option names to variables.
Avoid macro name concatenation garbage with trailing `dnl'.
(AT_ARG_OPTION, AT_ARG_OPTION_ARG): Overhaul macro description.
The OPTIONS are space-separated, not comma-separated. The
negative form of AT_ARG_OPTION is prefixed with `--no-'.
* tests/autotest.at (AT@&t@_ARG_OPTION, AT@&t@_ARG_OPTION_ARG):
New tests.
* NEWS: Update.
* doc/autoconf.texi (Writing Testsuites): Document AT_ARG_OPTION
and AT_ARG_OPTION_ARG.
(testsuite Invocation): Call the thingies passed to the
testsuite options, not arguments. Note that the testsuite
author may add further package-specific options.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* lib/autotest/general.m4 (HELP_TUNING_BEGIN): New diversion.
(HELP_TUNING, HELP_OTHER, HELP_END): Bump diversion numbers.
(AT_INIT): Accept
--color and --color=never|auto|always. If desired, colorize
test results and testsuite summary on standard output.
[HELP_TUNING]: Divert content instead to ...
[HELP_TUNING_BEGIN]: ... this diversion, m4_wrapped until the
end, when we know whether AT_COLOR_TESTS has been specified.
(AT_COLOR_TESTS): New macro, set the default for color to auto.
* doc/autoconf.texi (Writing Testsuites): Document it.
(testsuite Invocation): Document --color* options.
* tests/local.at: Call AT_COLOR_TESTS for Autoconf's testsuite.
* tests/autotest.at (color test results): New test, mirroring
color.test from Automake.
* NEWS: Update.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* lib/autotest/general.m4 (AT_INIT): New variable $at_recheck.
Escape hyphen in $at_dir early. Accept command line switch
--recheck. Set $at_suite_log early, based on --directory
switch; with --recheck, include the list of FAILed and XPASSed
tests from old testsuite.log file in $at_groups. Document
--recheck in --help output.
* tests/autotest.at (recheck): New test.
* doc/autoconf.texi (testsuite Invocation): Document --recheck.
* NEWS: Update.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* tests/autotest.at (AT_CHECK_AT_PREP): Create a default
atconfig file in the directory of the testsuite.
(AT_CHECK_AT_TITLE): Also check that `./micro-suite -v' output
does not contain empty $at_srcdir expansion.
(srcdir propagation): Remove the atconfig file generated by
AT_CHECK_AT_PREP. Check each suite invocation for $at_srcdir
expansion.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* lib/autotest/general.m4 (AT_CLEANUP): Actually print test
title in verbose output. Fixes AUTOCONF-2.57-101-gc102ed8
regression.
* tests/autotest.at (AT_CHECK_AT_TITLE): Amend macro to check
for test title in -v output.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* tests/autotest.at (Erlang Eunit unit tests): Use "no" as
value-if-not-found for Erlang tools.
* tests/erlang.at: Likewise. Also, use AS_EXIT instead of plain
exit.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* tests/autotest.at (Hard fail): Check more restrictively for
passed tests, so that hostnames recorded in the log file do not
wrongly match.
Report by Ludovic Courtès.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
These are lightweight versions of AT_CHECK that automatically
add the equivalent of ! in front of the command and change a
failure exit status to 77 resp. 99. They expand to just
two lines of shell code at the expense of not supporting
tracing (but then so does AT_XFAIL_IF).
2009-07-12 Paolo Bonzini <bonzini@gnu.org>
* NEWS: Mention AT_SKIP_IF and AT_FAIL_IF.
* doc/autoconf.texi (Autotest): Document them.
* lib/autotest/general.m4 (_AT_LINE_ESCAPED, AT_SKIP_IF,
AT_FAIL_IF, _AT_CHECK_EXIT): New.
(AT_CHECK): Use _AT_LINE_ESCAPED.
* tests/autotest.st: Add tests for AT_SKIP_IF and AT_FAIL_IF.
Use AT_SKIP_IF.
* tests/local.st: Use AT_SKIP_IF.
* tests/autotest.at (AT_SKIP_PARALLEL_TESTS): Skip test if set -m
is not supported.
Reported by Ralf Wildenhues.
Signed-off-by: Eric Blake <ebb9@byu.net>
* lib/autotest/general.m4 (AT_INIT) <AT_JOB_FIFO_FD>: Avoid <>;
instead open write descriptor in each group and read descriptor in
main driver.
* tests/autotest.at (AT_SKIP_PARALLEL_TESTS): Relax condition.
Signed-off-by: Eric Blake <ebb9@byu.net>
* lib/autotest/general.m4 (AT_INIT) <driver loop>: Avoid syntax
errors on shells that don't recognize <>.
* tests/autotest.at (AT_SKIP_PARALLEL_TESTS): Also skip parallel
tests for this reason. Skip based on the shell to be tested,
not the shell driving the testsuite.
(parallel syntax error): Rearrange similar to previous patch.
(parallel test execution): Defer skip until after serial tests.
Signed-off-by: Eric Blake <ebb9@byu.net>
* tests/autotest.at (Syntax error): Skip test if shell aborts on
syntax error (AIX ksh88) or doesn't detect it (zsh).
* doc/autoconf.texi (Limitations of Builtins) <.>: Mention these
limitations.
Reported by Ralf Wildenhues.
Signed-off-by: Eric Blake <ebb9@byu.net>
* lib/autotest/general.m4 (AT_CHECK): Expand prior to adding
escapes, to avoid shell syntax errors caused by late macro
expansion.
* NEWS: Document this change.
* tests/autotest.at (Metacharacters in command from M4 expansion):
New test.
Signed-off-by: Eric Blake <ebb9@byu.net>
* lib/autotest/general.m4 (AT_CHECK_NOESCAPE): Keep older,
undocumented semantics, where unbalanced " cannot be used in the
stdout/stderr argument.
(AT_CHECK_UNQUOTED): Treat " in stdout/stderr as a literal, since
the text is used in double-quoted context.
* tests/autotest.at (unquoted output): New test.
* doc/autoconf.texi (Writing Testsuites) <AT_CHECK>: Mention which
shell expansions are handled.
Signed-off-by: Eric Blake <ebb9@byu.net>
* lib/autotest/general.m4 (_AT_DECIDE_TRACEABLE): Handle # in
test correctly. Latent bug in handling shell comment was first
fixed 2008-11-20, but regressed two patches later.
* tests/autotest.at (Shell comment in command): New test.
* NEWS: Document the fix.
Signed-off-by: Eric Blake <ebb9@byu.net>
* lib/autotest/general.m4 (AT_INIT) <at_fn_group_postprocess>:
Guarantee test failure on syntax error, rather than inheriting
status from previous test.
* tests/autotest.at (Syntax error): New test.
Signed-off-by: Eric Blake <ebb9@byu.net>
* tests/autotest.at (AT_CHECK_AT): Add pre-test argument.
(Tested programs, Startup error messages, AT_CHECK_AT_TITLE)
(Fallacy, Skip, errexit, Long test source lines)
(Debugging a successful test, Debugging script and environment)
(Debugging a failed test, Using atlocal)
(Choosing where testsuite is run): Adjust callers.
(AT_SKIP_PARALLEL_TESTS): New macro, to skip parallel tests except
under zsh, bash, or when TEST_PARALLEL_AUTOTEST is defined. Makes
it easier to avoid testsuite hangs for users with dash or other
less-tested shell.
(parallel test execution, parallel truth, parallel fallacy)
(parallel skip, parallel errexit)
(parallel autotest and signal handling): Use it.
* BUGS: Mention this.
Signed-off-by: Eric Blake <ebb9@byu.net>
* lib/autotest/general.m4 (Driver loop): Rewrite signal handler.
Start parallel jobs in their own process group, enabling job
control in the shell if possible, for better signal handling.
Deal with INT, TERM, and HUP in the testsuite driver. In the
parallel driver, propagate TSTP to jobs either as TSTP or as
STOP (to avoid fork bombs with ksh).
Inside the job processes, add PIPE handler to write back the
job token, so the master process does not hang.
Disable the parallel driver if job control is not provided or if
trap does not understand signal names.
* tests/autotest.at (parallel autotest and signals): New test,
exercises INT, TERM, and PIPE, serial and parallel, with and
without `make' in the loop.
Kudos to Richard Stevens for writing APUE.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* lib/autotest/general.m4 (AT_INIT) <at_fn_check_prepare_dynamic>:
Fix typo.
* NEWS: Clarify the potential impact to users.
* tests/autotest.at (AT_DATA_AUTOTEST): New macro, patterned after
AT_DATA_M4SUGAR.
(AT_CHECK_AT_PREP, AT_CHECK_AT_TITLE): Use it.
(unusual file names): Test that the recent echo fix does not
regress. Fix quoting bug that made the test a no-op.
Reported by Paolo Bonzini and Ralf Wildenhues.
Signed-off-by: Eric Blake <ebb9@byu.net>
* lib/autotest/general.m4 (AS_MESSAGE_LOG_FD, AT_JOB_FIFO_FD):
Define fds only when initializing the log fd so early error
messages do not try to write to it.
* tests/autotest.at (Startup error messages): New test.
* NEWS: Document this 2.63 regression.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
* lib/autotest/general.m4: Use the serial code path if no test
is to be run.
* tests/autotest.at (parallel test execution): Test -j and -jN
with `-k notmatched'.
Signed-off-by: Ralf Wildenhues <Ralf.Wildenhues@gmx.de>
