Makefile feature
Description:
'make check-clean' now cleans *.h5 files created by tests as well as
.log and .chkexe files.
Solution:
check-clean is now a separate target in its own right, which cleans less
than mostlyclean (check-clean < mostlyclean < clean < distclean).
Platforms tested:
mir, heping, modi4 (serial and parallel)
Feature: check-clean target
Description:
'make check-clean' cleans up output files from tests.
Solution:
Tests create foo.chkexe and foo.log files. Scripts create foo.chksh and
foo.logsh files. 'make check-clean' will clean these files up so that
the tests can be re-run.
Also suppressed some not-very-useful output of Makefiles when it would
echo commands.
Platforms tested:
mir, sleipnir, modi4
Misc. update:
Configuration feature
Description:
Serial test output is now stored in log files and printed when all tests
in a directory complete, or when a test fails. This should make test output
more readable and useful.
Also made changes to clean up ii_files directories that are created by some
C++ compilers/linkers.
Also fixed a few minor Makefile bugs.
Solution:
When serial tests run, their output is saved in *.log or *.logsh. While
running, tests only print when they begin and when they complete; their
more specific output (from the log file) is printed if the test fails or
when all tests have completed.
Comments welcome.
Platforms tested:
mir, modi4 (parallel and serial), copper, shanti
Bug fix
Description:
The testh5diff.sh script fails on mcr because of random srun messages
thrown into the output
Solution:
Added some code to filter out the srun messages before the output is compared.
Platforms tested:
LLNL mcr
Misc. update:
Bug fix
Description:
The ".chksh" file for a test script was being created in the "source"
location rather than the build location. This can cause problems when
multiple builds are running because "slower" machines will see the ".cshsh"
file from faster machines and will not run the test script as they should.
Solution:
Use 'basename' command to strip off the path of the script and create
the ".chksh" file in the build location.
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committest
bug fix.
Description:
-nt is not a universal option for the test command in all platforms.
The use of it in conclude.am cause some platforms to bark at the
Makefile generated.
Solution:
Created a command script bin/newer which tests if file1 is newer
than file2. Replace the "test file1 -nt file2" by
"newer file1 file2".
Platforms tested:
Tested in sol.
Bug fix
Description:
Fixed bugs that caused tests to be unable to find testhdf5.sh during
daily tests, and another that broke sol and shanti.
Solution:
When tests are run, their path is ./$testname .
When scripts are run, their path has no leading "./". Fixed.
Sol and shanti don't recognize the -e option for test, but they do know
-f. Fixed.
Also modified release notes.
Platforms tested:
mir, shanti, heping
Feature: tests now use parallel make and only run once
Description:
When make is invoked in parallel (using the -j option), several tests will
now be run simultaneously. This should speed up testing on a number of
systems.
When a test passes, it creates a foo.chkexe or foo.chksh file, which prevents
the test from running again unless the test or library changes.
Solution:
Most of the changes live in config/conclude.am. Added *.chkexe and *.chksh
files to the list of files to be cleaned at "make mostlyclean" time.
Parallel tests still run one at a time, but use the same make instructions
as serial tests.
Platforms tested:
mir, eirene, sleipnir, pommier, copper, modi4 (some errors occurred, probably
due to the test being run before updating hdf5. Will re-run tests after
checkin.)
Bug fix for parallel case for new "Some objects were not comparable" patch to h5diff.
Description:
The parallel h5diff wouldn't print out "Some objects were not comparable" because the worker
tasks were not communicating the not_cmp flag of the diff_opt_t struct back to the manager, who ultimately
prints everything.
Also, some miscellaneous fixes for error printing. Some errors were printed out with printf instead of fprintf(stderr,...).
In parallel environments, this can result in output getting lost.
Solution:
Had the worker tasks pass along the not_cmp flag to the manager when they sent along
the number of differences they found.
Platforms tested:
heping (pp), sol (pp)
Misc. update:
modified the script test so that when the test files do not exist , they are created
Description:
Solution:
Platforms tested:
linux
solaris
Misc. update:
bug fix
Description:
when 2 objects were not comparable, the final print information for the non verbose mode printed "0 differences found"
Solution:
replaced instead with a Summary message that says
"Some objects were not comparable"
Platforms tested:
linux
solaris
Misc. update:
Improvement
Description:
The actual stderr output was modified by the FILTER and was not
available for display if errors detected later.
Solution:
Copy the actual stderr to a temporary file and do filtering on that.
Platforms tested:
LANL Flash.
Misc. update:
Improvement.
Description:
assign testph5diff.sh to the Parallel test scripts so that
it will be tested for the check-p target.
Platforms tested:
heping pp.
Misc. update:
bug fix
Description:
check-s and check-p were set as pre-requistics of test. They
would get executed in parallel if parallel make is used. This
could cause problems since serial tests are used to be executed
before parallel tests. It is not known if it is always okay to
run serial and parallel tests in simultantously.
Solution:
Change check-s and check-p as actions of the target test so
that they get executed sequnentially.
Platforms tested:
heping PP.
Added check-s and check-p targets.
Description:
Added check-s and check-p targets to 1.7 branch to match changes to 1.6
branch. Now parallel and serial tests can be run separately.
Platforms tested:
mir, modi4, copper
Bug fix
Description:
Various system or software (e.g. MPE) will print some diagnosis
messages to stderr that cannot be suppressed. They messed up
the output matching. Installed the STDERR_FILTER() to remove
all these messages.
Platforms tested:
LANL flash.
Misc. update:
Description: Removed support for SRB driver
Solution: Removed or modified appropriate files; ran reconfigure
to regenerate Makefile.in and configure files.
Platforms tested: heping and shanti
Misc. update: ran bin/chkmanifest on heping
Configuration feature
Description:
Different Fortran compilers mangle function names in different ways
(upper case, lower case, adding underscores). To link between
Fortran and C functions, we need to know what a given function's
name is under a given compiler.
Solution:
Use autoconf's FC_WRAPPERS check to determine the Fortran
naming scheme and define the FC_FUNC_ macro to name our
functions (in H5f90proto.h). Removed references to
our old FNAME macro, as well as flags that indicated whether
function names were upper or lower case.
Platforms tested:
mir, pommier, modi4, copper, more
Bug fix
Description:
On some machines, $RUNSERIAL variable needs to be used to run
tests. Set $RUNTESTS (which is used for non-parallel tests) to
be $RUNSERIAL in configure.am.
Also, since I was updating all Makefiles.in anyway, I updated
commence.am to point to autotools installs in AFS instead of
those on heping.
Platforms tested:
mir, sleipnir, modi4, copper
Purpose:
"Bug fix"
Description:
Hardcoded Makefiles to use /bin/sh instead of letting configure
detect shell automatically. This is what v1.6 does, and avoids
problems on janus.
Platforms tested:
sleipnir, copper, modi4, mir
Feature - conditional compilation
Description:
SRB file driver and tests are now compiled only when SRB is enabled
(using --with-srb during configure).
Solution:
Added an automake conditional in configure.in, altered Makefiles.am in
src and test directories to depend on that conditional.
This should make a nice example for posterity to add conditionally
compiled sources.
Platforms tested:
heping (only configure change)
Purpose:
Added C++ wrapper for Packet Table API.
Description:
Added macro for high-level C++ library (LIBH5CPP_HL), which changes every
Makefile.in.
Added directories for high-level C++ library (though currently only Packet
Table API is supported).
Added both C++ source and tests.
Platforms tested:
sleipnir, mir, modi4
Misc. update:
Purpose:
Bug fix.
Description:
ph5diff fails on modi4 due to the way snprintf works on IRIX.
Solution:
The C99 standard says that, if there isn't enough room in the string,
snprintf should return the number of characters that
would have been written to the output string if there were enough room.
The snprintf on modi4 would return the number of characters that is was able to write
succesfully to the string if space ran out. The ph5diff logic that checks if
the output buffer was full did not handle this sort of return value correctly.
Used VSNPRINTF_WORKS from configure test to check how snprintf works and do
the logic accordingly.
Platforms tested:
modi4
Misc. update:
Bug Fixes
Description:
Fixes for several bugs, including dumping of excess output to a temporary file, fix for printing
hsize_t datatype, and the long awaited fix for intermixed output.
Solution:
Fix 1: Overflow file
Previously, any output that a worker task made was buffered locally in memory, up to a point. Any
output beyond the size of the buffer (used to be 10k) was discarded. Now, the memory buffer size has been
changed to 1k and any output beyond this amount is sent a temporary file. This way, no output is lost
and memory usage is kept under control. The temporary file is deleted as soon as a worker task finishes
sending its contents to the manager.
Fix 2: hsize_t printing
Printing of the hsize_t datatype used to be handled by %Hu passed to HDfprintf. However, there is no corresponding HDvsnprintf that
is able to print hsize_t types. These are now printed with the aid of H5_PRINTF_LL_WIDTH.
Fix 3: Intermixed output fix
Intermixed output would occur on some machines (although I haven't seen it happen for a while) due to the unpredictability of the underlying network
and the speed at which various message would travel. This has been fixed by having all output send to the manager
for printing. The worker tasks no longer print the output themselves upon receipt of a token, but instead
send that data to the manager.
Platforms tested:
heping, eirene, tg-login (the only place that seems to still experience intermixed output every now and then)
Misc. update:
Automake version upgrade
Description:
Upgraded automake version from 1.6.2 to 1.9.5.
Changed bin/reconfigure script to use automake 1.9.5.
Changed configure.in and Makefiles to use new FCFLAGS and FC variables
instead of FFLAGS and F9X.
Automake and configure should now do the lion's share of the work
supporting Fortran 9X; macros in acsite.m4 are now mostly unused (will
be cleaned later).
Altered how configure handles pmake; now root-level Makefile.in is
processed by bin/reconfigure to have a .MAKEFLAGS target, since
automake no longer allows us to define unused variables.
Configure now always checks for C++ compiler even if it is not
used, since automake thinks this is the Right Thing To Do and
will break otherwise.
Platforms tested:
Sol, copper, heping, mir, sleipnir, eirene, pommier, kelgia, modi4.
Bug fix
Description:
The GASS VFL driver header file was bringing in the <string.h> header file,
which several other source code modules needed also, but weren't including
explicitly themselves.
Solution:
Add includes for <string.h> to files which actually need them.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/C++ as CC
Configuration not tested by h5committest...
Bug fix.
Description:
MPI_LONG_LONG, which is not standard yet, was used to pass the
nubmer of differences found. This was needed because number of
differences is defined as type hsize_t which can be arbitary large
such that there is no MPI type that matches it. The value is passed
between processes as an array of bytes in order to be portable. But this
may not work in non-homogeneous MPI environments.
This fix was actually Leon's idea.
Platforms tested:
Tested in QSC in which this failed.
Removed GASS configuration and software packages associated with
it. The following software are no longer configurable.
checking for main in -lcrypto... yes
checking for SSL_get_version in -lssl... yes
checking globus_common.h usability... yes
checking globus_common.h presence... yes
checking for globus_common.h... yes
checking for globus_module_activate in -lglobus_common... yes
checking for main in -lglobus_gass_cache... yes
checking for main in -lglobus_gaa... yes
checking for main in -lglobus_gss... yes
checking for main in -lglobus_gss_assist... yes
checking for main in -lglobus_io... yes
checking for main in -lglobus_gass_transfer_assist... yes
checking for main in -lglobus_gass_transfer... yes
checking for globus_gass_open in -lglobus_gass_file... yes
Platforms tested:
h5committested (but copper was down.) tested in modi4 too.
bug fix.
Description:
When MPE library is used, it prints two extra message lines that
interfere with the expected output.
Solution:
Filter out those two lines of text from stderr.
Also added a provision to print the whole generated output when failure
is detected.
Platforms tested:
Tested in copper, serial, parallel and parallel with MPE.
Misc. update:
bug fix/new feature.
Description:
(committing changes made by Leon.)
ph5diff now is a real program by itself. It has its
own main (ph5diff_main.c).
Codes common to h5diff and ph5diff are in h5diff_common.c.
Removed the hack that hide failures from testh5diff.sh.
Platforms tested:
heping(serial, pp), sol (pp), copper(pp).
Misc. update:
Purpose:
Bug fix
Description:
modi4 dies during build with strange errors.
The root cause of these is a two-year-old hack in HDF5's libtool
script that only takes effect on IRIX.
Solution:
Edited the libtool hack (by editing ltmain.sh) to correct a bug in
the hack.
Also made sure that compiler-specific DEFAULT_LIBS are used
when linking.
Platforms tested:
sleipnir, copper, modi4, sol
Bug fix.
Description:
ph5diff is still being fixed.
Changed this to report all failures but will not flag them as
errors so that daily tests can continue. Will be fixed soon.
Platforms tested:
Tested in sol (pp) where it has most failures.
Misc. update:
Purpose:
Bug fix
Description:
Fixed the error on copper where output would match, except the worker tasks are forcefuly terminated at the end.
Solution:
If an exit code other than 0 was returned, the
parallel environment would assume that there
was an error and would terminate all other
tasks. Because h5repack relies on the return
value of h5diff, it was necessary to return
values other than 0. Fixed by having parallel
runs of h5diff always return 0, whereas the
return value of serial runs is unchanged.
Platforms tested:
copper
Misc. update:
Bug fix
Description:
Arabica exhibited strange errors when linker found wrong versions of
header files. This happened because include directories were
given to linker in the wrong order.
Solution:
Move include directories from AM_CFLAGS variable to INCLUDES
variable to put them before CPPFLAGS variable. Trust me, it works.
This bug may also have contributed to strange errors on other platforms
(kelgia?).
Platforms tested:
copper, sleipnir, arabica.
(h5dump broke while building on arabica, but this happened in
a clean checkout, too).
Bug fix
Description:
pmake (on modi4, for instance) complains about undefined variables
if it is run without the -V flag, which turns those errors to warnings.
Solution:
Added test to configure.in to see if $MAKE will allow Makefiles
with undefined variables. If not, sets -V flag in AM_MAKEFLAGS.
Ensured that all custom make targets use AM_MAKEFLAGS.
Also defined all variables that caused errors in top-level Makefile.am.
This means that pmake can be used to build hdf5 *from the top level
only*. To run make from a subdirectory, still need to use -V flag
(or use make or gmake).
Platforms tested:
modi4, heping, copper, sleipnir
bug fix.
Description:
Need eval before the RUNCMD command because some machines like
AIX, has RUNPARALLEL in the style as
MP_PROCS=3 MP_TASKS_PER_NODE=3 poe ./a.out
that throws the shell script off.
Platforms tested:
Tested in copper (pp) where it failed before.
Tested in heping pp too.
Misc. update:
Bug fixed.
Description:
Heping used to have a network limit that if too many rsh connections
occurred in a second, it stopped accepting connections for 30 seconds.
That caused testph5diff to fail when too many mpi jobs are executed
repidly. A sleep 2 was in to go around this problem. Heping's limit
has been raised, thus no more need for this hack. Sleep 2 is removed.
Platforms tested:
Tested in heping (pp).
Misc. update:
Correction to previous print_manager_output bugfix
Description:
Although the previous bugfix did fix all the errors, it was not the correct way
to fix the problem. Since the manager will only buffer output when run in
parallel mode, in serial mode this function should not even be called.
Solution:
Wrapped print_manager_output in #define's to prevent it from being included in
serial builds.
Platforms tested:
heping (serial + parallel)
Misc. update:
Purpose:
bug fix
Description:
a string with 2 characters was declared as having size 2. changed to size 3 (+ null character)
this was causing compiler warnings with g++
Solution:
Platforms tested:
linux
free bsd (g++)
Misc. update:
Fixed numerous ph5diff bugs.
Description:
Fixed manager output printing
Fixed out of order output printing
Fixed test script execution problem
Temporary fix for large amounts of output overflowing buffer.
Solution:
The manager task buffers its output. However, since the manager task
never gets a print token, this output was lost. Solution: new function called
print_manager_output that prints buffered output is called in places where the
manager buffers its output.
printf was apparently buffering output. This means that a task would sometimes
print even after it had given up its print token. Added fflush() call after
printf() calls, which seems to have fixed the problem.
calling rsh multiple times in succession seems to overwhelm something in Linux,
as it begins to refuse new connections until the old ones reset. Since each
call to mpirun in the test script starts up 4 rsh sessions, the test script
eventually is unable to run any further tests. Solution: Added a short delay in the
testscript between successive calls to mpirun to allow old connections to reset.
The 10k output buffer was of insufficient size to hold the large amounts of
output generated by some of the tests. Since code to buffer to a file has not
been implemented yet, a temporary fix was to increase the size of the output
buffer to 50k.
Platforms tested:
heping
Misc. update:
Bug fix
Description:
h5diff sometimes will 'exit(status)' but if it is in parallel mode,
it needs to close up the workers and the MPI environment before exit(..).
Solution:
Created h5diff_exit() that will just exit in serial mode but does all
the shutdown properly if in parallel mode.
Platforms tested:
tested in heping, serial and parallel.
Misc. update:
Bug fix
Description:
h5diff_main.c contained some debug printf statment that produced
unexpected output. Commented them out.
testh5diff.sh: implemented code to test ph5diff tests but
ph5diff has some errors that generated unexpected output.
Temporary hacked to skip tests if run by daily test account.
Platforms tested:
Tested in heping, serial and parallel.
new features
Description:
Added a parallel option to testh5diff.sh to do parallel h5diff tests.
Added testph5diff.sh to invoke testh5diff.sh with parallel mode.
Platforms tested:
Tested in heping (serial and pp).
Features
Description:
Added the option to build ph5diff and run testphdf5.sh when
parallel mode is enabled.
Platforms tested:
tested in heping, both serial and parallel.
Misc. update:
New feature
Description:
Add basic code for new B-tree implementation. They don't do much yet,
aren't hooked up to anything yet and the format may change, but I'd like to
start getting them into the daily tests.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Too minor to require h5committest
