Since these examples need to follow filesystem paths, the Makefiles need
to create directories in the examples directory; added this to the
Makefile.am.
Tested on Windows, mir, juniper
To activite this test,
add the command option -i.
For example, at IBM AIX, type "poe testphdf5 -i" will test the library with independent IO with file setview. It simply replaces all the collective IO tests with independent IO with file setview.
"make check-vfd" will now run all tests in the test directory with different
file drivers (at least, all of those tests that use the testing framework's
FAPL). Tests that fail will be skipped.
This is not a perfect fix, but is better than nothing.
Along with this change, check-vfd should be added to the Daily Tests.
1) Added trace file support to the metadata cache. This allows capture
of all metadata cache calls in trace files for purposes of optimization
and debuging.
2) Added an expunge entry function. This allows an entry to be deleteded
from the cache without writing it to disk even if it is dirty.
3) Added a function call to resize pinned entries.
4) Added code to deal with entries that are dirty on load. This is
needed in support of a bug fix which can alter object headers on
load to repair files.
5) Added progress reporting code to the "MDC API smoke check" test in
cache_api.c. To enable the progress reporting, set report_progress
to TRUE in mdc_api_call_smoke_check().
Tested with h5committest, and a parallel test on phoenix (dual athelon
linux box).
Code cleanup
Description:
Trim trailing whitespace in Makefile.am and C/C++ source files to make
diffing changes easier.
Platforms tested:
None necessary, whitespace only change
Configuration feature
Description:
'make install' now tests both static and shared libraries if both are installed.
Solution:
Previously, shared libraries were only tested when static libraries were not installed.
Also cleaned up line in commence.am that was including HL library in all Makefiles.
Platforms tested:
mir (Makefile change only)
Bug fix - bugzilla #552
Description:
On Cray X1, trying to use : as an argument confused the system.
Solution:
Added a test in configure to see if : as an argument is bad.
If so, skipped the test.
Platforms tested:
mir, Cray X1 (change to configure only)
Bug fix.
Description:
It failed when only 1 is used to test.
Solution:
Cleanup the code a little so that it works for any number
of processes to invoke it.
Platforms tested:
h5committested,
Some collective chunk IO macro names are confusing, change them to more meaningful
names.
Description:
H5Pset_dxpl_mpio_chunk_opt will set a flag so that the library can do one linked IO or mul
ti-chunk IO with collective in chunking storage directly. That is, the library won't do an
alyses to determine this.
The flags for the enum type we used before are:
H5FD_MPIO_OPT_ONE_IO
H5FD_MPIO_OPT_MULTI_IO
They are not good names because of the following two reasons:
1. It doesn't reflect chunking storage
2. OPT is kind of redundant and misleading,
Solution:
We change the names to
H5FD_MPIO_CHUNK_ONE_IO
H5FD_MPIO_CHUNK_MULTI_IO
Platforms tested:
Since only macro names are changed, no need to test with h5committest.
Misc. update:
Bug fix.
Description:
Sometimes the parallel prefix is given in the form of nfs:/mnt/pfs which
if given to a non-MPIO VFD like the default H5Fcreate, it would fail.
Called h5_rmprefix which would return the non-prefix component
part of the file name which would be okay for the default H5Fcreate and
such.
Platforms tested:
Tested in heping parallel.
Feature
(Code and tests are done by Christian. I just committed the code.)
Description:
Rewrote the purpose of this test. Now it tests these three cases,
/*
* Test following possible scenarios,
* Case 1:
* Sequential create a file and dataset with H5D_ALLOC_TIME_EARLY and large
* size, no write, close, reopen in parallel, read to verify all return
* the fill value.
* Case 2:
* Sequential create a file and dataset with H5D_ALLOC_TIME_EARLY but small
* size, no write, close, reopen in parallel, extend to large size, then close,
* then reopen in parallel and read to verify all return the fill value.
* Case 3:
* Sequential create a file and dataset with H5D_ALLOC_TIME_EARLY and large
* size, write just a small part of the dataset (second to the last), close,
* then reopen in parallel, read to verify all return the fill value except
* those small portion that has been written. Without closing it, writes
* all parts of the dataset in a interleave pattern, close it, and reopen
* it, read to verify all data are as written.
*/
Platforms tested:
Tested in copper, tg-ncsa and heping, all in parallel mode.
1) Check in potential fix to unreproduceable bug in t_cache observed on
Cobalt
2) Check in code supporting added pinned entry features in the metadata
cache.
Description:
1) Elena encountered a bug in t_cache when running on Cobalt. On
investigation I found a potential race condition in t_cache that
could explain her observation.
2) Quincey requested additions to the metadata cache allowing renaming
of pinned entryies, and marking of pinned or protected entries as
dirty.
Solution:
1) Modified t_cache.c to prevent the potential race condition. Elena
was unable to reproduce the bug on Cobalt, so we don't know if my
fix actually addressed the problem.
2) Added code supporting the requested functions. The changes were
relatively trivial, but required substantial new test code.
Platforms tested:
h5committest, serial test on heping, parallel test on phoenix.
Misc. update:
Bug fix.
Description:
The ${TR}, though avoid the error, is inconvenient. It is needed
because configure.in use the char range in the style of 'a-z'.
The other style of '[a-z]' is more commonly accepted by all tr
but autoconf tends to strip away [], making the syntax rather
clumsy.
Solution:
Learned from autoconf that it avoids the use of character range
by just spell all the letters out. Changed our tr commands to
use those variables defined by autoconf. Also removed the
definition of ${TR} since it will cause inconsistancy when
autoconf also use plain 'tr' in its generated code.
The Makefile.in are changed because the elimination of ${TR} from
configure triggered its removal from all Makefil.in. That is okay
because ${TR} is not used at all in Makefile.
Platforms tested:
h5committested. (sol failed to connected). also tested in shanti
using both /usr/ucb/tr and /bin/tr (the bad one before.)
Misc. update:
Updated both INSTALL and RELEASE files.
Bug fix
Description:
Fix two test bugs:
1) Unitialized variable bug in test of H5C_get_entry_status()
(in test/cache.c)
2) Incorrect initialization of constant in t_cache when running
with MPE. (in testpar/t_cache.c)
Solution:
1) Don't test uninitialize variable.
2) Correct initialzation of constant in MPE case.
Platforms tested:
Tested on Heping and Copper.
Misc. update:
Code cleanup/feature twist
Description:
Adjust recent H5AC routines to take H5F_t pointers instead of H5C_t
pointers, to match the rest of the H5AC routines.
This change propagated into a few of the tests, which also had some
compiler warnings cleaned up...
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Linux 2.4/64 (mir) w/C++ & FORTRAN
Add pinned entry capability to cache.
Description:
For frequently accessed cache entries, the protect/unprotect overhead
is sometimes a bottleneck.
Solution:
Allow entries to be pinned in the cache. Pinned entries can't be
evicted, but can be flushed or modified.
Platforms tested:
h5committested -- minus one small typo in test/cache.c whose fix was
tested on copper and heping only.
Misc. update:
Several bug fixes
Description:
Added config file for Cray X1 (previous file was misnamed)
Simplified some code in hyperslab.c test that seemed to have been
confusing compiler on Cray X1.
Fixed typo in config/commence.am
Cleaned up hl/src/Makefile.am
Solution:
All four fixes should be straightforward. The failure on Cray was
very difficult to debug, but involved arithmetic errors.
This change seems to fix it.
Platforms tested:
heping, copper, sol, some Cray X1 (more testing when system comes back up)
new feature
Description:
1) separated the HL library into "public" and "private" header files, with the same caracteristics as the basic library
2) added the public headers to hdf5.h (with a conditional include macro, defined in configure.in)
3) added the path to HL in all Makefile.am 's , because of the inclusion in hdf5.h
Solution:
Platforms tested:
linux 32, 64
AIX
solaris
with fortran and c++
(one packet table example fails)
Misc. update:
bug fix.
Description:
The MPE_Stop_log did not work in copper. It spewed out MPE errors
and ended in infinite looping.
Solution:
Changed to a smaller test size to avoid generating huge MPE log files
when MPE is configured in.
Platforms tested:
Copper (mpe)
Misc. update:
Bug fix.
Description:
When MPE is used, the test generates huge Clog files in /tmp that
could fill up the disk (like in Copper.)
Solution:
Turned off MPE logging.
Platforms tested:
heping (pp) and copper(pp)
Misc. update:
Adding parallel tests for optional collective chunk APIs
Description:
Three new APIs
"H5Pset_dxpl_mpio_chunk_opt_ratio
H5Pset_dxpl_mpio_chunk_opt_num
H5Pset_dxpl_mpio_chunk_opt"
for optional optimization choices from users
have been added to the libraries.
This check-in adds six tests to verify the funcationality and correctedness
of these APIs.
These tests need to be verified with 3 or more processors and with MPI-IO driver only.
Solution:
Using H5Pinsert, H5Pget, H5Pset to verify that the library indeed goes into the branch we hope for.
Using H5_HAVE_INSTRUMENT macro to isolate these changes so that it won't affect or be misused by the application.
Platforms tested:
h5committest(shanti still refused to be connected)
Parallel tests on heping somehow are skipped. Manually testing at heping. Have checked
1,2,3,4,5 processes.
Misc. update:
Bug fix
Description:
Previous checkin did a bad thing; 'make clean' failed in example directories.
Solution:
Fixed commence.am so that examples no longer break, and fixed a mistake
in conclude.am.
Platforms tested:
heping (minor makefile change)
Misc. update:
Add tests for optional APIs to support collective chunk IO
Description:
In order to test whether library picks up the user's options,
The number of chunks need to be varied for different processes,
Selection of the number of processes selected in one chunk also
need to be varied.
Solution:
Create two cases,
1. Each chunk only selected by one unique process, this case
library should use independent for collective call.
2. One-third of the processes occupies the top half of the whole domain,
The rest of the processes occupies the lower half of the domain.
The total number of chunk is a fixed number 8.
Platforms tested:
Linux 2.4 with mpich 1.2.6(only)
Since I only checked in the code that handles the selection, haven't added any new tests yet. So it won't affect any platforms.
Misc. update:
Bug fix
Description:
make check-clean didn't clean results of example tests
Solution:
Fixed Makefiles so that check-clean recurses into example directories.
Also a little Makefile cleanup.
Platforms tested:
mir, modi4, heping, copper
change the array size of collective chunking features of parallel tests.
Description:
Previously array size for collective optimization tests
including
cchunk1,
cchunk2,
cchunk3,
cchunk4,
ccontw,
ccontr,
cschunkw,
cschunkr,
ccchunkw,
ccchunkr
are fixed,
They are only valid for some good number of processors(1,2,3,4,6,8,12,16,24,32,48 etc).
Recently there are more requests for parallel tests to be valid on some odd number of processes such as 5,7,11,13 etc.
Solution:
I change the array size to be dynamic rather than static. Now the fastest change array size is a function of mpi_size. dim2 = constant *mpi_size. After some tunings, theoretically the above tests should be valid for any number of processors. However, other parallel tests still need to be tuned.
To verify the correctness of these tests, using mpirun -np 5 ./testphdf5 -b cchunk1 at heping.
Platforms tested:
h5committest(shanti is refused to be connected)
at heping, 5 and 7 processes are used to verify the correctness.
Misc. update:
Add new tests
Description:
Collective IO doesn't work for some platforms/mpio packages when more than
one process has no contributions to IO.
Solution:
1. Add a collective IO test to verify the correctness of the library when
more than one process has no contributions to IO.
2. Add the similar MPI-IO test in t_mpi to help us maintain in more platforms.
Platforms tested:
heping, mir, copper
Misc. update:
Improvement.
Description:
The t_cache takes a long long time to run and it tests HDF5 calls.
Move it to the back and let more basic tests to run first so that
basic features are tested first.
Platforms tested:
Tested in heping with pp.
code cleanup
Description:
remove two printf lines accidently added for debugging at NCSA cobalt.
Solution:
Platforms tested:
No need to test.
Misc. update:
Enhance collective chunk IO supports
Description:
Add a new test to check the correctness of the HDF5 library behavior for collective IO mode when one process doesn't have any contribution for IO.
Solution:
Platforms tested:
IBM AIX 5.2(copper)
Linux (heping) mpich-1.2.6
Misc. update:
Attempt to ensure that the parallel cache test runs at a reasonable
speed with large numbers of processors.
Description:
In some cases, the number of random locks and unlocks was a multiple of
the MPI rank.
Solution:
Use rank % 4 instead of simply rank.
Platforms tested:
copper
Misc. update:
Add a file that I forgot in my last checkin.
Description:
t_cache.c is the source file for the new parallel metadata cache test.
Solution:
See above.
Platforms tested:
h5committested before the last checkin. Will run another
h5committest shortly, but it shouldn't be necessary as there
are no new changes.
Misc. update:
1) Add parallel test for metadata cache
2) Split serial test for metadata cache into two parts
3) Fix bug in which cache was flushed needlessly when the
cache wasn't full.
4) Performance improvements
5) Update API for parallel cache coherency bug fix.
Description:
See above.
Solution:
See above.
Platforms tested:
h5committest
Misc. update:
bug fix 504
Description:
testpar/t_mpi would hang if $HDF5_NOCLEANUP is set. E.g.,
% env HDF5_NOCLEANUP=yes mpirun -np 3 ./t_mpi
This happened because the environment variables are not exported to all
mpi processes by the mpirun command. So, some attempted to do cleanup
while others don't and some hang waiting for others to act.
Solution:
Instead individual program checking getenv, they all just called h5_cleanup
no matter. h5_cleanup now uses getenv_all to check the $HDF5_NOCLEANUP if
it is in parallel mode.
Platforms tested:
h5committested, tested pp in heping too.
Feature.
Description:
Modified it so that it can be compiled outside of HDF5 library as a standalone
program. e.g., mpicc -DSTANDALONE prog.c.
Platforms tested:
Tested in Red storm and heping.
Patch.
Description:
Copper would fail with a message of
0032-113 Out of memory in routine unknown, task 0
when run with 3 processes and size 1MB in MPI-IO tests.
It seems to be a copper MPIO error.
Solution:
Reduced the upper bound of default write size to 1/2MB (but
tests only go to 1/4MB) for now, pending permenant fix from
Copper.
Platforms tested:
Copper.
Minor bug fixes.
Description:
1. Changed free() calls to HDfree()
2. Corrected behavior of -m command-line parameter
3. Changed return value to always return 0.
Solution:
2. The -m flag tells the test to run only the MPI IO tests. However, it would incorrectly make
the test run both the MPI and POSIX tests (same as the default behavior).
3. This test is known to fail on many platforms, and, even on those platforms where it usually passes,
it is known to suffer transient failures (especially with small test file sizes). It's outcome is
also very dependent on the filesystem on which the testfile is created. Corrected
the program to always return success, so that it doesn't interfere with the daily tests.
The motivation for this is that
this test has nothing to do with the HDF library and is an auxiliary test. Failures in t_posix_complaint
do not necessarily mean that parallel HDF will fail, but simply indicate something to look into,
especially on new platforms. This is now an "output only" test, and any errors will be only be visible
in the output.
Platforms tested:
copper (all of these were minor changes)
Misc. update:
bug fix.
Description:
Fixed the segmentation fault errors in modi4, copper and tg-login.
It was due to the misuse of trying to realloc a pointer returned by
getenv_all. (not supposed to.)
Also rearranged the code so that option is checked first, then check
with environment variable, then use default setup. This saves the
need to do realloc at all.
Platforms tested:
Heping, modi4, shanti, copper (copper showed a different error now.)
Bug fix.
Description:
For some strange reason, getopt() does not appear to be defined in unistd.h
on colonelk when the source is compiled with -D_POSIX_SOURCE.
Solution:
Inserted some extern's for the missing variables to make the compiler happy.
Platforms tested:
colonelk, copper
Misc. update:
Feature
Description:
Added blurb about future todo's for this test.
Added support for HDF5_PARAPREFIX to determine the directory where the test file
is stored.
Solution:
Used getenv_all to get the value of HDF5_PARAPREFIX. Note that, if a command-line
parameter is passed to the program to specify a path, it will override the value of
HDF5_PARAPREFIX.
Platforms tested:
copper, colonelk
Misc. update:
Added t_posix_compliant to the rest of the build and patched up minor
compile bugs/warnings encountered on other platforms.
Description:
It seems that <getopt.h> needs to be included to get the file to build, even though the man page
seems to indicate that <unistd.h> should be sufficient.
Solution:
Platforms tested:
copper, colonelk, sol
Feature
Description:
Added posix compliance tests.
Solution:
These tests do increasingly complicated sets of writes followed by reads.
POSIX standards say that any read that can be proven to occur after a write
must include the data in that write. These tests attempt to verify whether the
underlying filesystem and i/o layer provide such guarantees.
Platforms tested:
copper, colonelk, red storm
Misc. update:
Bug fix
Description:
Fortran type generation was broken in two ways. Fixed both.
Solution:
Firstly, there were a couple of path problems. Fixed a typo and
specified the full path of a file.
Secondly, the dependencies weren't right when building with HDF5-specific
commands (make lib, make check-s, etc.). Tweaked dependencies
to fix the problem.
Platforms tested:
mir, modi4, sleipnir
