Bug fixes
Description:
This checkin fixes an occasional error on kelgia on sol during distclean.
It also causes test scripts to depend properly on the programs they're
supposed to be testing.
Solution:
The kelgia bug was due to some files being cleaned by automake and manually.
Removed the manual cleaning in src/Makefile.am.
Test script dependencies now need to be specified manually, since the
makefile can't guess what they test from their name. Currently all test
scripts in a given directory have a single list of dependencies--this was
easy and seems to be sufficient.
These dependencies are listed in the SCRIPT_DEPEND variable in the Makefile.am.
Platforms tested:
heping, mir, modi4, sol
Misc. update:
minor tidy up.
Description:
Changed both test programs and test scripts to use the same suffixes
(.chkexe and .chklog). Changed from .log to .chklog to avoid running
into potential conflicts by other "things" (e.g., config.log).
Tested:
Heping.
New feature
Description:
All datasets were 1D only before. Added -g to use 2D datasets such
that it allows more options in testing various access patterns.
Platforms tested:
heping pp.
Misc. update:
Code cleanup
Description:
Trim trailing whitespace, which is making 'diff'ing the two branches
difficult.
Solution:
Ran this script in each directory:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Platforms tested:
FreeBSD 4.11 (sleipnir)
Too minor to require h5committest
New Feature.
Description:
Added code and changes so that h5perf (pio) can be built by
standalone mode (that is built by just "h5pcc" without all these
internal files.) With the standalone mode, h5perf can be built
against different versions of hdf5 library and to contrast the
performance among them.
(Note that pio_standalone.c and pio_standalone.h are used in
standalone build only.)
Platforms tested:
Tested in copper and heping.
Bug fix and improvement.
Description:
The test output were not displayed for parallel tests.
The serial tests output were always displayed whether the test was
actually ran this time or not.
Solution:
Moved the display of test output inside the target where the actual
tests are run.
Platforms tested:
Tested in heping using pp and fortran.
Bug fix and code minor cleanup.
Description:
The attempt to redirect stderr together to the log files were
done in the wrong order such that stderr output goes to where
stdout WAS. Fixed that.
Code minor cleanup--use shell variables to hold the name of the
log file and the test instead of generating them repeatedly. This
makes easier code reading, less crowded and avoid typos.
Only conclude.in is changed. The Makefile.in changes are all
derived from automake.
Platforms tested:
h5committested.
Makefile feature
Description:
'make check-clean' now cleans *.h5 files created by tests as well as
.log and .chkexe files.
Solution:
check-clean is now a separate target in its own right, which cleans less
than mostlyclean (check-clean < mostlyclean < clean < distclean).
Platforms tested:
mir, heping, modi4 (serial and parallel)
Feature: check-clean target
Description:
'make check-clean' cleans up output files from tests.
Solution:
Tests create foo.chkexe and foo.log files. Scripts create foo.chksh and
foo.logsh files. 'make check-clean' will clean these files up so that
the tests can be re-run.
Also suppressed some not-very-useful output of Makefiles when it would
echo commands.
Platforms tested:
mir, sleipnir, modi4
Misc. update:
Configuration feature
Description:
Serial test output is now stored in log files and printed when all tests
in a directory complete, or when a test fails. This should make test output
more readable and useful.
Also made changes to clean up ii_files directories that are created by some
C++ compilers/linkers.
Also fixed a few minor Makefile bugs.
Solution:
When serial tests run, their output is saved in *.log or *.logsh. While
running, tests only print when they begin and when they complete; their
more specific output (from the log file) is printed if the test fails or
when all tests have completed.
Comments welcome.
Platforms tested:
mir, modi4 (parallel and serial), copper, shanti
Bug fix
Description:
An earlier bug fix to plotting when the minimum number of processors was not 1
caused the -procs argument to be ignored.
Solution:
Added a check to see what the -procs argument was set to and plot accordingly.
Platforms tested:
colonelk
Misc. update:
Bug fix
Description:
The ".chksh" file for a test script was being created in the "source"
location rather than the build location. This can cause problems when
multiple builds are running because "slower" machines will see the ".cshsh"
file from faster machines and will not run the test script as they should.
Solution:
Use 'basename' command to strip off the path of the script and create
the ".chksh" file in the build location.
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committest
bug fix.
Description:
-nt is not a universal option for the test command in all platforms.
The use of it in conclude.am cause some platforms to bark at the
Makefile generated.
Solution:
Created a command script bin/newer which tests if file1 is newer
than file2. Replace the "test file1 -nt file2" by
"newer file1 file2".
Platforms tested:
Tested in sol.
Feature
Description:
gen_report.pl throws away the plotting commands used to make the plots
in gnuplot.
Solution:
Deciding that these commands may be useful if the someone wants to reproduce
the plots (or, more importantly, manipulate them further in gnuplot), the
commands are now saved in a file called gnuplot.script
Platforms tested:
colonelk
Misc. update:
Bug fix
Description:
When plotting, if the minimum number of processors was not 1, gen_report.pl would fail.
Solution:
The minimum number of processors in the datafile was inadvertantly hard-coded to be 1.
Fixed it to dynamically determine the number of processors from the data.
Platforms tested:
colonelk.
Misc. update:
Bug fix
Description:
Fixed bugs that caused tests to be unable to find testhdf5.sh during
daily tests, and another that broke sol and shanti.
Solution:
When tests are run, their path is ./$testname .
When scripts are run, their path has no leading "./". Fixed.
Sol and shanti don't recognize the -e option for test, but they do know
-f. Fixed.
Also modified release notes.
Platforms tested:
mir, shanti, heping
Feature: tests now use parallel make and only run once
Description:
When make is invoked in parallel (using the -j option), several tests will
now be run simultaneously. This should speed up testing on a number of
systems.
When a test passes, it creates a foo.chkexe or foo.chksh file, which prevents
the test from running again unless the test or library changes.
Solution:
Most of the changes live in config/conclude.am. Added *.chkexe and *.chksh
files to the list of files to be cleaned at "make mostlyclean" time.
Parallel tests still run one at a time, but use the same make instructions
as serial tests.
Platforms tested:
mir, eirene, sleipnir, pommier, copper, modi4 (some errors occurred, probably
due to the test being run before updating hdf5. Will re-run tests after
checkin.)
bug fix
Description:
check-s and check-p were set as pre-requistics of test. They
would get executed in parallel if parallel make is used. This
could cause problems since serial tests are used to be executed
before parallel tests. It is not known if it is always okay to
run serial and parallel tests in simultantously.
Solution:
Change check-s and check-p as actions of the target test so
that they get executed sequnentially.
Platforms tested:
heping PP.
Added check-s and check-p targets.
Description:
Added check-s and check-p targets to 1.7 branch to match changes to 1.6
branch. Now parallel and serial tests can be run separately.
Platforms tested:
mir, modi4, copper
Description: Removed support for SRB driver
Solution: Removed or modified appropriate files; ran reconfigure
to regenerate Makefile.in and configure files.
Platforms tested: heping and shanti
Misc. update: ran bin/chkmanifest on heping
Feature: make install now installs h5perf
Description:
h5perf is now installed in hdf5/bin during 'make install.'
Same change as in the 1.6 branch.
Platforms tested:
mir
Configuration feature
Description:
Different Fortran compilers mangle function names in different ways
(upper case, lower case, adding underscores). To link between
Fortran and C functions, we need to know what a given function's
name is under a given compiler.
Solution:
Use autoconf's FC_WRAPPERS check to determine the Fortran
naming scheme and define the FC_FUNC_ macro to name our
functions (in H5f90proto.h). Removed references to
our old FNAME macro, as well as flags that indicated whether
function names were upper or lower case.
Platforms tested:
mir, pommier, modi4, copper, more
Bug fix
Description:
On some machines, $RUNSERIAL variable needs to be used to run
tests. Set $RUNTESTS (which is used for non-parallel tests) to
be $RUNSERIAL in configure.am.
Also, since I was updating all Makefiles.in anyway, I updated
commence.am to point to autotools installs in AFS instead of
those on heping.
Platforms tested:
mir, sleipnir, modi4, copper
Purpose:
"Bug fix"
Description:
Hardcoded Makefiles to use /bin/sh instead of letting configure
detect shell automatically. This is what v1.6 does, and avoids
problems on janus.
Platforms tested:
sleipnir, copper, modi4, mir
Feature - conditional compilation
Description:
SRB file driver and tests are now compiled only when SRB is enabled
(using --with-srb during configure).
Solution:
Added an automake conditional in configure.in, altered Makefiles.am in
src and test directories to depend on that conditional.
This should make a nice example for posterity to add conditionally
compiled sources.
Platforms tested:
heping (only configure change)
Purpose:
Added C++ wrapper for Packet Table API.
Description:
Added macro for high-level C++ library (LIBH5CPP_HL), which changes every
Makefile.in.
Added directories for high-level C++ library (though currently only Packet
Table API is supported).
Added both C++ source and tests.
Platforms tested:
sleipnir, mir, modi4
Misc. update:
Automake version upgrade
Description:
Upgraded automake version from 1.6.2 to 1.9.5.
Changed bin/reconfigure script to use automake 1.9.5.
Changed configure.in and Makefiles to use new FCFLAGS and FC variables
instead of FFLAGS and F9X.
Automake and configure should now do the lion's share of the work
supporting Fortran 9X; macros in acsite.m4 are now mostly unused (will
be cleaned later).
Altered how configure handles pmake; now root-level Makefile.in is
processed by bin/reconfigure to have a .MAKEFLAGS target, since
automake no longer allows us to define unused variables.
Configure now always checks for C++ compiler even if it is not
used, since automake thinks this is the Right Thing To Do and
will break otherwise.
Platforms tested:
Sol, copper, heping, mir, sleipnir, eirene, pommier, kelgia, modi4.
Removed GASS configuration and software packages associated with
it. The following software are no longer configurable.
checking for main in -lcrypto... yes
checking for SSL_get_version in -lssl... yes
checking globus_common.h usability... yes
checking globus_common.h presence... yes
checking for globus_common.h... yes
checking for globus_module_activate in -lglobus_common... yes
checking for main in -lglobus_gass_cache... yes
checking for main in -lglobus_gaa... yes
checking for main in -lglobus_gss... yes
checking for main in -lglobus_gss_assist... yes
checking for main in -lglobus_io... yes
checking for main in -lglobus_gass_transfer_assist... yes
checking for main in -lglobus_gass_transfer... yes
checking for globus_gass_open in -lglobus_gass_file... yes
Platforms tested:
h5committested (but copper was down.) tested in modi4 too.
Purpose:
Bug fix
Description:
modi4 dies during build with strange errors.
The root cause of these is a two-year-old hack in HDF5's libtool
script that only takes effect on IRIX.
Solution:
Edited the libtool hack (by editing ltmain.sh) to correct a bug in
the hack.
Also made sure that compiler-specific DEFAULT_LIBS are used
when linking.
Platforms tested:
sleipnir, copper, modi4, sol
Bug fix
Description:
Arabica exhibited strange errors when linker found wrong versions of
header files. This happened because include directories were
given to linker in the wrong order.
Solution:
Move include directories from AM_CFLAGS variable to INCLUDES
variable to put them before CPPFLAGS variable. Trust me, it works.
This bug may also have contributed to strange errors on other platforms
(kelgia?).
Platforms tested:
copper, sleipnir, arabica.
(h5dump broke while building on arabica, but this happened in
a clean checkout, too).
Bug fix
Description:
pmake (on modi4, for instance) complains about undefined variables
if it is run without the -V flag, which turns those errors to warnings.
Solution:
Added test to configure.in to see if $MAKE will allow Makefiles
with undefined variables. If not, sets -V flag in AM_MAKEFLAGS.
Ensured that all custom make targets use AM_MAKEFLAGS.
Also defined all variables that caused errors in top-level Makefile.am.
This means that pmake can be used to build hdf5 *from the top level
only*. To run make from a subdirectory, still need to use -V flag
(or use make or gmake).
Platforms tested:
modi4, heping, copper, sleipnir
Purpose:
Bug fix
Description:
Parallel builds were breaking in tools/lib
Solution:
talign didn't realize it depended on libh5tools.la because its
dependencies listed the full path (../../tools/lib/libh5tools.la).
Changed this, and made similar changes in a couple of other
directories.
This checkin should also add the --foreign flag to every Makefile.in
Platforms tested:
sleipnir (minor change)
New feature
Description:
Add basic code for new B-tree implementation. They don't do much yet,
aren't hooked up to anything yet and the format may change, but I'd like to
start getting them into the daily tests.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Too minor to require h5committest
Bug fix
Description:
Found the permanant fix to automake/CVS dependency problem
Solution:
Added AM_MAINTAINER_MODE macro to configure.in.
Now automake will never try to regenerate Makefiles, Makefiles.in,
configure, H5config.h, etc. when they are out of date, nor will it
print any warnings.
Developers should be very very careful to use reconfigure script,
and can add --enable-maintainer-mode flag to configure on heping
to regenerate these files correctly.
Platforms tested:
heping sleipnir copper
Bug fix
Description:
Configuration files' timestamps were incorrect.
Solution:
Update Makefiles.in. Also added correct paths to autotools on heping,
so heping build should be able to re-generate configuration files even
if they are still broken.
Platforms tested:
sleipnir
Bug fix
Description:
Dependencies between configure files (aclocal, configure.in, configure,
Makefiles.am and .in) are still causing Makefiles to try to run autotools
during build.
Solution:
Committed all Makefiles.in to update their timestamps.
As a temporary measure, forcibly prevented automake from running
autotools during build by redefining the variables it uses.
Platforms tested:
sleipnir (No changes to Makefile content)
Bug fix
Description:
Different compilers use different flags to include Fortran module files
Solution:
Changed boilerplate to use configure variable rather than hardcoded -M flag.
Platforms tested:
sleipnir, sol, copper
Configuration feature
Description:
HDF5 now uses automake to generate Makefiles
Solution:
Makefile.in files are now generated from Makefile.am files.
To reconfigure (after chaning a Makefile.am or configure.in):
/bin/sh bin/reconfigure.sh
Platforms tested:
Many
Added ability to select which io performance types to graph
Description:
Originally, POSIX, MPIO, and PHDF5 were all graphed. Now, any subset of these can be graphed individually via the io_type command-line param.
Platforms tested:
heping
Misc. update:
Feature additions:
script has support for all sorts of interesting plots.
fix axis scaling
Description:
Added support for 2-d and 3-d plots of the data. Command-line options are also added to customize these plots.
Axis tick labels are now much improved and match the data and are logically spaced.
Platforms tested:
heping (gnuplot must be installed to do the plots)
Misc. update:
Bug fixes (provided by Leon Arber).
Description:
gen_report spewed many messages and could not report speeds that are
less than 100MB/s.
Solution:
Removed -w which prints warning messages.
Fixed code to recognize speeds under 100MB/s.
Platforms tested:
Eirene.
Bug fix.
Description:
Function _flushall() is not available on Cygwin. So a Cgywin macro is added so the compiler will not call this function when building HDF5 on Cygwin.
Solution:
Change the following codes:
#ifdef H5_HAVE_SYSTEM
#ifdefined WIN32
_flushall();
#else
HDsystem ("sync");
HDsystem ("df >/dev/null");
#endif
to:
#ifdef H5_HAVE_SYSTEM
#if defined(WIN32) && ! defined(__CYGWIN__)
_flushall();
#else
HDsystem ("sync");
HDsystem ("df >/dev/null");
#endif
Platforms tested:
Cygwin 1.5.11, VC 6.0 on XP.
Linux 2.4 (heping)
Solaris 2.7 (arabica)
Misc. update:
Bug Fix/Code Cleanup/Doc Cleanup/Optimization/Branch Sync :-)
Description:
Generally speaking, this is the "signed->unsigned" change to selections.
However, in the process of merging code back, things got stickier and stickier
until I ended up doing a big "sync the two branches up" operation. So... I
brought back all the "infrastructure" fixes from the development branch to the
release branch (which I think were actually making some improvement in
performance) as well as fixed several bugs which had been fixed in one branch,
but not the other.
I've also tagged the repository before making this checkin with the label
"before_signed_unsigned_changes".
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel & fphdf5
FreeBSD 4.10 (sleipnir) w/threadsafe
FreeBSD 4.10 (sleipnir) w/backward compatibility
Solaris 2.7 (arabica) w/"purify options"
Solaris 2.8 (sol) w/FORTRAN & C++
AIX 5.x (copper) w/parallel & FORTRAN
IRIX64 6.5 (modi4) w/FORTRAN
Linux 2.4 (heping) w/FORTRAN & C++
Misc. update:
Description: Restore 6 old error API functions back to the library to be backward
compatible with v1.6. They are H5Epush, H5Eprint, H5Ewalk, H5Eclear, H5Eset_auto,
H5Eget_auto. These functions do not have error stack as parameter.
Solution: Internally, these functions use default error stack.
Platforms tested: h5committest and fuss.
Misc. update: RELEASE.txt
Purpose:
HDF5 now supports SZIP with no encoder.
Description:
SZIP can be configured to have both encoder and decoder or just to have the decoder. HDF5 can now query the configuration of any filter, and will throw errors if users try to write using a filter with encoding disabled.
Solution:
Added H5Zget_filter_info function, changed API for H5Pget_filter and H5P_get_filter_by_id. See SZIP RFC.
Platforms tested:
Copper (fortran, C++, parallel), Sleipnir (C++), Arabica (fortran, C++), Verbena (fortran, C++)
Misc. update:
Code cleanup
Description:
Reduce compiler warnings on SGI IRIX
Platforms tested:
FreeBSD 4.10 (sleipnir)
SGI IRIX6 (Cheryl's machine)
Too minor to require full h5committest
Bug fixes
Description:
Updated dependencies
Fixed error with C++ compiler builds of main library
Added H5Pset_data_transform to MPE info
Platforms tested:
FreeBSD 4.9 (sleipnir)
Description: This program test performance of create or open datasets and create
attributes for datasets.
Platforms tested: h5committest
Misc. update: MANIFEST
Code cleanup
Description:
Clean up a few loose ends and warnings for the 1.6 compatibility changes
to the error API.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Code cleanup
Description:
Clear up a couple of minor issues with compiling the library with a C++
compiler.
Platforms tested:
FreeBSD 4.9 (sleipnir)
Too minor to require h5committest
Updated the copyright notice--mostly by rearranging
some text to make them consistent.
Solution:
Platforms tested:
"h5committested"--sol is down, so, no SUn test.
Misc. update:
feature change, new feature.
Description:
Had the HDF5_PARAPREFIX default set to /tmp.
Changed it to null so that data files are created in current directory
by default.
Also added a code to print data filename used when debug level > 0.
Platforms tested:
Tested in modi4 parallel.
Misc. update:
Code cleanup
Description:
Remove the fill-value option entirely, since the 1.5.x library's default
behavior is now the same as the 1.4.x behavior and also there is no
corresponding code for implementing fill-values with the POSIX or MPI-I/O tests.
Platforms tested:
FreeBSD 4.8 (sleipnir) w/parallel
too small to need h5committest
Code cleanup
Description:
Change some variables that are keywords in C++ to non-keywords.
Platforms tested:
FreeBSD 4.8 (sleipnir)
too minor to require full h5committest
minor improvment
Description:
corrected an omission in the usage help output.
Added more parameters checks.
Solution:
Platforms tested:
Only in modi4.
Misc. update:
Bug Fix
Description:
On HP-UX platforms, the "open()" function needs to specify that the
file being created has permissions S_IRWXU, which means that it gives
read/write/execute permissions to the user who created the file (how
nice of it!).
Solution:
Added the S_IRWXU flag to the open() function call.
Platforms tested:
HP-UX 11
Arabica,
Modi4,
Verbena
Misc. update:
Compatibility fix
Description:
The H5P[set|get]_fapl_mpiposix calls changed between v1.4.x and v1.5.x.
Solution:
Wrap them in the v1.4 backward compatibility #ifdefs and update tests, etc.
Platforms tested:
FreeBSD 4.8 (sleipnir) w/paralle & v1.4 compatibility
h5committest pointless
"Bug fix"
Description:
This version is showing very slow HDF5 I/O write speeds (about 40%
of MPIO). It was because the library is writing fill values by
default. There is a no-fill option which set to FALSE. Changed
it to TRUE since MPIO and POSIX are not doing fill values.
This is just a stop gap fix since the better fix would be to
change the library to not write fill-values in this case.
Platforms tested:
modi4 parallel and LANL Q paralllel.
No h5committest test because it is very trivia.
Misc. update:
Purpose:
Added sanity check that min number of processes be not greater
than max number of processes.
Platforms tested:
copper only since this is a very simple change.
Misc. update:
Bug fix.
Description:
The default setting of max transfer size is larger than
the default setting of number of bytes per process, result
in complaints that Transfer size is too big.
Solution:
Cap the min and max transfer size with number of bytes per process.
Always cap min transfer size with the max xfer size.
Platforms tested:
Copper.
Misc. update:
Code Improvements/Bug Fixes
Description:
Comparison of equality of a double/float variable to 0.0 is not
guaranteed to work and is bad practice.
Solution:
Test that the absolute value of the variable is < a very small
positive number.
Platforms tested:
Modi4 (Parallel & Fortran)
Verbena (C++ & Fortran)
Arabica (Fortran)
Misc. update:
Bug Fix
Description:
The "mkstemp()" function isn't as portable as I hoped it would be (it
fails on Crays and other beasts).
Solution:
Removed and just use a fixed filename instead.
Platforms tested:
Linux
Solaris w/ zlib
Irix w/ zlib
Misc. update:
Bug Fix
Description:
When I changed the H5P{set,get}_fapl_mpiposix functions to accept
another parameter, I didn't change it in these files.
Solution:
Added the new "use_gpfs" parameter.
Platforms tested:
Modi4 (only affects the parallel stuff, so no need for full testing).
Misc. update:
Update
Description:
version 1.4 has renamed the tool from pio_perf to h5perf.
Folded the chagne into this branch too.
Also updated the Copyright notes.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}? Tested in burrwhite with PP only.
[If no, why not?] It is a pretty straight forward Makefile change.
Code cleanup
Description:
Various code cleanups to allow the development branch to be compiled with
a C++ compiler (i.e. CC=g++ )
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) C++
Add benchmark
Description:
Add multi-D hyperslab benchmark to repo. It's not hooked up to any makefiles
yet, so it's not being compiled or tested, etc. I'm just stashing it here
so I don't forget about it until I have time to integrate it properly.
Code cleanup
Description:
Cleaned up more compiler warnings...
Platforms tested:
FreeBSD 4.6 (sleipnir) w and w/o parallel
Linux 2.2.x (eirene) w/FORTRAN & C++
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/FORTRAN & parallel
Update, Bug Fix, and Feature Add
Description:
- Updated how AC_{ENABLE,WITH} help messages were being generated.
Autoconf now gives you an AC_HELP_STRING macro to use to create
them.
- Fixed the problem with Linux LFS on RedHat 7.3 machines. It wasn't
finding getdents64(), so we can't rely on that being present
anymore...
- Added GPFS detection and setting.
- Updated how compression is specified. It's no longer necessary to
test for HAVE_ZLIB_H, HAVE_LIBZ, and HAVE_COMPRESS2. The one macro
{H5_}HAVE_COMPRESSION takes care of all of these.
Solution:
Changed the check for Linux LFS from looking for getdents() to
looking at the version number of the kernel (using the uname -r
command). You can still override with the --enable-linux-lfs switch
if you really believe that your <2.4 kernel has LFS support.
Platforms tested:
Linux (2.2 and 2.4)
Bring in sync with v1.4 versions that have copyright statements
added.
Platforms tested:
eirene(pp). Only one machine tested because all changes are merely
comments.
New Feature
Description:
Added interleaved block pattern to MPI and HDF5 access methods. Also,
changed the blocks to just use the same size as the regular blocks for I/O,
instead of a separate block size.
Platforms tested:
IRIX64 6.5 (modi4) w/parallel
Feature Add
Description:
Allow you to specify the percentage you'd like a random buffer to be
compressible. It can go from 0% (non-compressible) to 100%
(completely compressible).
Platforms tested:
FreeBSD
Feature Add
Description:
Use the /dev/urandom device for random data if it's available. (That
is stat() doesn't return a -1 when ran on it). Use random() otherwise.
Platforms tested:
Linux, FreeBSD
Bug Fix
Description:
The test for compression needed to be reversed.
Solution:
Instead of "!(defined...) used just "(defined..."
Platforms tested:
Modi4
Bug Fix
Description:
Not all platforms have zlib. This shouldn't be compiled then.
Solution:
ifdef-ed the code out if the zlib wasn't specified.
Platforms tested:
Linux
Feature Update
Description:
It now writes an uncompressed version of the file out. The first
tests didn't look all that wonderful. Here is a typical output:
Buffer size == 1MB
Uncompressed Write Time: 0.55s
Uncompressed Write Throughput: 116.00MB/s
Compressed Write Time: 8.76s
Compressed Write Throughput: 7.31MB/s
Platforms tested:
Linux
Compress I/O Test Add
Description:
This is the initial check-in of the compress I/O test. It will write
out compressed buffers to a file.
I need to implement a write of uncompressed data first for
comparison...
Platforms tested:
Linux
Feature Add
Description:
Added support for testing only the "write" operations. Use the "-w"
or "--write-only" flags for this.
Platforms tested:
Linux (parallel) & Modi4 (parallel)
Bug Fix
Description:
There was an error in the way it was determining if there were any
reads to report.
Solution:
Wrote a subroutine which would determine if the reads are defined in
the structure before trying to print them out.
Platforms tested:
Linux
Add Report Generator
Description:
This is a generator for ASCII and Excel(tm)(c) reports generated by
the pio_perf benchmark we run. There are instructions at the top of
the file on how to run the program and the input format it expects,
etc.
Platforms tested:
Linux
Code cleanup and new feature.
Description:
Cleaned out some old debug print statments.
Added option -B for interleaved I/O block size.
(Just parsing and sanity check is done. Real I/O implementation
is not done yet.)
Platforms tested:
eirene(pp)
Removal of bad code
Description:
Albert came up with a better way of determining which performance
tests to run. I removed the broken way I did them...
Platforms tested:
Linux-pp
bug fix
Description:
pio_info_g has been moved to the test/libh5test.a but the
pio_* source files still have the old definition and declaration.
Some compilers hated the duplcation.
Solution:
Replaced the duplication with #include h5test.h and added test/ to
the include search path too.
Platforms tested:
eirene, modi4.
Update
Description:
Only print out the parameters for a run if we're in debug mode one
and the 0'th process or if we're in debug mode > 0.
Platforms tested:
Linux
Feature Addition
Description:
Added feature which prints out the parameters and the MPI_Info object
for a run for each process.
Added some functions for the GPFS system. They are ifdef'ed out right
now (well, they need to ahve the non-existant H5_HAVE_GPFS macro set)
and aren't in use just yet. But the stub functions are there.
Platforms tested:
Linux
Folded the changes made in v1.4 into v1.5.
print library version information in help page too.
Specify the nofill feature supported in v1.5 only.
Print the values of the KB, MB and GB in case a user wants to see them.
Setup a macro, H5_HAVE_NOFILL, to indicate if Dataset no fill feature
is supported. If not, --no-fill is an invalid option.
Platforms tested:
modi4
Update
Description:
Updating the source for the PIO performance tests with the current
code that's in the V1.4 branch. It includes the newest command-line
parameters.
Platforms tested:
Linux-pp
Feature
Description:
moved the routines of setting up and dumping MPI-info object to
test library so that it is avaiable for all tests too.
Platforms tested:
modi4(pp), eirene (serial)
Bug fix
Description:
Was not able to handle data size (file size) larger than 32bits.
Was using long, which is only 4 bytes big in SP, thus overflowing
into negative when trying to address 2GB or larger.
Solution:
Changed those variables involved in file size/offset calculation
to type off_t. (If a certain system/compiler has off_t defined
as 4 bytes, it can't write to file size larger than 2GB anyway.)
Note that the lseek of SP with -D_LARGE_FILE still fails for
offset larger than 2GB (works for 2GB). That has to be fixed
soon.
Platforms tested:
burrwhite (linux 2.4) and modi4 parallel.
Name Change
Description:
Changed the macro's name from "MILLISECOND" to "MICROSECOND" since,
as Quincey pointed out, that's what it is ;).
Platforms tested:
Linux
Bug fix, feature
Description:
Added code to time pure MPI_File_write_xxx and read_xxx routines.
Moved the setting of the timer_g to when timer is created (new)
and destroyed.
Platforms tested:
modi4pp
Modification
Description:
Put some of the variables into the global space so that random
functions can access them. (The output and timer variables which
point to the output file and timer resp.)
Placed the debug level 4 prints into the timer module so that it's
easier to automatically put timer start/stop printfs for new timers
which are created.
Added timers for MPI Time calls (called HDF5_MPI_WRITE and
HDF5_MPI_READ).
Platforms tested:
Linux
Bug Fix
Description:
There was a duplicate declaration of print_indent() in the header and
in the source files.
Solution:
Removed the public declaration and made the function static again.
Platforms tested:
Arabica
Update
Description:
Changed the description of the help message so that the --hdf5,
--mpio, --raw flags better say what they do.
Platforms tested:
Linux
Bug fix
Description:
The symbol "RAW" is already defined in some ioctl.h files.
Changed it to RAWIO.
Platforms tested:
Tflops.
(I am commiting this from TFLOPS. Will check it out immediate
to test on local machines.)
Code cleanup
Description:
Platform dependent code related to the struct stat and fstat
calls polluted source codes. Hard to maintain.
Solution:
Platform dependent code are moved to H5private.h and then internal
code can #include H5private.h. Repeat those macro definition for
the stdio and multi drivers since they area examples for writing
a virtual file driver. They must not use any internal code.
Platforms tested:
eirene (parallel), modi4 (serial including gass driver.)
Bug Fix
Description:
So, for Raw I/O in parallel, if you open a file with truncation by
multiple processes, it looks as if one process could open the file
and start writing to it while another process also opens the file
with truncation, thus wiping out all of the stuff the first process
wrote to the file.
This is bad.
Also added some garbage collection to the pio_perf routine to reclaim
the space taken by some of the tables.
Solution:
Placed an MPI_Barrier() statement after the Raw open()/create() call
so that all processes are synced up before they start writing to the
file.
Added free() calls to the tables which weren't being free'd.
Platforms tested:
Linux-pp (eirene)
Purpose:
Bone-headed Bug Fix
Description:
There were blanks being put into the output. The cause: the
"print_indent()" routine was printing indents for all of the
processes, but only process 0 should have been printing them out at
all (since process 0 is the one which prints out the reports).
Solution:
Check to make sure that we're process 0 before printing the indents.
Platforms tested:
Linux
Remove perf and mpi-perf from the parallel test targets since their
functions are replaced by pio_perf.
Platforms tested:
modi4 and eirene, both parallel modes.
Change default actions.
Description:
Change the default maximum number of processes (-P) to use all processes
instead of just 1 (old default). Someone most likely wants to test
the I/O performance with all processes involved.
Also starts performance measurement with maximum number of processes
and decrement it with each loop. If the performance measurement
needs to restart, it can run with fewer processes if those loops
have completed.
Platforms tested:
modi4 and eirene.
Bug Fix
Description:
Throughput wasn't being calculated correctly.
Solution:
We were using a value other than the actual time. Changed so that
we're using the correct structure to grab the time out of it.
Platforms tested:
Linux(pp)
Feature add and algorithm reworking.
Description:
Added a "--debug" flag so we can print out various extra debugging
information.
Reworked the algorithm so that it's printing the correct throughput.
Here's how it's supposed to work:
process
T_0 T_1 T_2 T_3 ... T_n
iteration 1
2
.
.
.
m
Retrieve the maximum time from each iteration over the number of
processes. (So, if T_i had the maximum time in iteration j, then use
that time). Calculate the "Throughput" of iteration j:
S_j = (raw_size / T_i)
Collect that information over all of the iterations. Then output the
Max, Min, and Ave of all of the S_k's.
Platforms tested:
Linux (pp)
Purpose:
Small Fix
Description:
Fixed the Min/Max/Average accumlation stuff...
Solution:
Actually thought about the code and made it accumulate the
information in the correct way.
Platforms tested:
Linux
Purpose:
Feature Fix
Description:
Added timer from open to close for write. Changed reporting of time
for write and read to only measure the actual read/write instead of
the open/close as well...
Platforms tested:
Linux(pp)
Purpose:
Feature Fix
Description:
Instead of outputting the # of kilobytes in a transfer size, output
the total bytes. This helps when you're interested in just doing a
copy-paste type of thing for the transfer buffer size.
Solution:
Changed the output report
Platforms tested:
Linux(pp)
Purpose:
Feature Add
Description:
Added output which tells how large the file is (that is, the number
of dsets * number of elements in a dset * sizeof(int)).
Platforms tested:
Linux(pp)
The default minimum xfer size of 1K was way too slow for
parallel file system like the PFS of Tflops. Set it to
128K to complete soon for default settings.
Platforms tested:
Tflops, modi4.
Extension
Description:
int64_t type is not available everywhere.
#include H5private.h which has platform dependent hooks
to define int64_t to something available.
Platforms tested:
Tflops
Removing the DPSS (gridstorage) driver source code.
Description:
The DPSS (using Grid-Storage) driver is retired.
Removed the configure option with-gridstorage from configure.in.
Cvs remove the following files
./src/H5FDdpss.c
./src/H5FDdpss.h
./test/dpss_read.c
./test/dpss_write.c
Regenerated Dependencies files (some had to be hand-edited since
'make depend' did not cover them.)
Removed reference to DPSS Virtual file driver from H5F.c.
Platforms tested:
modi4 (Parallel; -with-gass=...), eirene, arabica (fortran, cxx).
Purpose:
Feature Fix
Description:
Added the minimum, maximum, and average time and MB/s for the write
and read operations. It now prints the report out in a pretty clear
format. It also includes how many iterations were done for the
write/read operation.
Platforms tested:
Linux
Bug fix
Description:
All processes, including those that are not part of the PIO test
sub-communicator, all attempted to run the PIO test. It resulted
in failures for those processes that are not supposed to get involved.
Solution:
The function that creates the sub-communicator also returns a
parameter indicating if the process is included in the PIO test
sub-communicator. Then only those processes will really do the
PIO test.
Platforms tested:
eirene (pp) and Modi4 (pp)
Purpose:
Feature Fix
Description:
Changed so the "pio_perf" module handles creating and destroying the
MPI Comm. Worked it so we get the minimum, maximum, and average times
over a set of iterations.
Solution:
Lots. Had to pull the MPI Comm code from the "pio_engine" module and
place it in the "pio_perf" module. Then worked on a way to have all
processes send their time output to process 0, who collects it and
gives back the min, max, and avg times for the iterations.
Platforms tested:
Linux. Doesn't work if you use more than 1 processor...*hrmph*
Purpose:
Bug Fix
Description:
The --raw, --hdf5, --mpiio options weren't being done correctly.
Solution:
I had to change some of the tests for the io_type to & instead of |
so that they'd work how I wanted them to work.
Platforms tested:
Linux
Purpose:
Feature Changes
Description:
Okay, I needed to add in more parameters so that the user can modify
how things are supposed to work with the PIO programs. Also needed
to change the algorithm a bit to make these work. And needed to add
in timing for the READ option.
Solution:
Added the above things. The parameters took a major rewrite of the
command-line parsing stuff. Here's the usage statement:
usage: pio_perf [OPTIONS]
OPTIONS
-h, --help Print a usage message and exit
-d N, --num-dsets=N Number of datasets per file [default:1]
-f S, --file-size=S Size of a single file [default: 64M]
-F N, --num-files=N Number of files [default: 1]
-H, --hdf5 Run HDF5 performance test
-i, --num-iterations Number of iterations to perform [default: 1]
-m, --mpiio Run MPI/IO performance test
-o F, --output=F Output raw data into file F [default: none]
-P N, --max-num-processes=N Maximum number of processes to use [default: 1]
-p N, --min-num-processes=N Minimum number of processes to use [default: 1]
-r, --raw Run raw (UNIX) performance test
-X S, --max-xfer-size=S Maximum transfer buffer size [default: 1M]
-x S, --min-xfer-size=S Minimum transfer buffer size [default: 1K]
F - is a filename.
N - is an integer >=0.
S - is a size specifier, an integer >=0 followed by a size indicator:
K - Kilobyte
M - Megabyte
G - Gigabyte
Example: 37M = 37 Megabytes
Platforms tested:
Linux, but not fully finished...
Added nfiles and ndsets parameters checking.
Removed iteration variable since iterations would be done
in Control routine.
Platforms tested:
Eirene(pp) and modi4(pp)
Bug fix (or more like feature)
Description:
MPI_File_open does not truncate the filesize if file already exists.
This created confusion during debugging as what the real file size
is. It also interfere the real write bandwidth since the times
required to allocate new disk-space vanishes for subsequent writes
that are for offset shorter than previous file sizes.
Added a MPI_File_set_size to reset the file size to 0 for every new
file.
Another bug is that the 'remove()' call may not work for MPIO/PHDF5
files. (e.g., filename may have some MPI prefix like "pfs:filename").
Replaced "remove" with MPI_File_delete for those cases.
Platforms tested:
modi4(pp) and eirene (pp)
bug fix, new feature
Description:
Added HDF5 write in do_write.
Added a complete do_read.
Still need timing code for the read part.
Platforms tested:
eirene (pp), modi4(pp).
Did not test serial since all changes were done in parallel area.
Purpose:
DOH!
Description:
Ugh! I forgot to add the -m flag to the list of "short" parameter
codes. It wasn't even looking for them. *sigh*
Solution:
Added it
Platforms tested:
Linux
Bug fixes, new features
Description:
There was a coding error in handling file open flags. Changed it
to use &.
Added do_cleanup to cleanup temporary test files but only if
$HDF5_NOCLEANUP is not set. This is consistent with other test programs.
Added logic so that each process is writing its own slabs of data only.
Moved the number of process, rank of process and the communicator used
for PIO run to be global variables. Makes the coding easier. (but this
is not thread-safe.)
Platforms tested:
modi4(pp) and eirene(pp).
Code cleanup
Description:
Tweaked internal error handling macros to reduce the size of the library's
object code by about 10-20%.
Also cleaned up some compiler warnings...
Platforms tested:
FreeBSD 4.4 (sleipnir)
Purpose:
Feature Fix
Description:
Changed default size of file to 512MB. The "-m" flag is now in
megabytes as well. This makes running things a bit faster.
Platforms tested:
Linux
Purpose:
Bug Fix
Description:
Fixed so that it will display the correct timing data. It will also
write to the correct file (which it wasn't before).
Solution:
Put the code in for displaying the time. Had to change the way I was
passing an object to the pio_fopen() function from just being a
structure to being a pointer so that the changes could be propagated
back.
Platforms tested:
Linux
Purpose:
Bug Fix
Description:
I wasn't calculating the total time correctly.
Solution:
I had to subtract the previous time from the current time. This
wasn't being done...DOH
Platforms tested:
Linux
Purpose:
Small Fixes
Description:
After conversation with Albert, here are some small fixes for the
performance stuff. Not too significant. Though, we did add the
"buffer size" as a parameter I pass to the engine.
Purpose:
Feature Fix
Description:
Added code so that it measures the time it takes to do I/O and return
that to the calling function.
This code doesn't quite work yet. There is something wrong with the
MPI code in the "pio_engine.c" file...I don't know what's up with
it...
Platforms tested:
Linux
Bug fix
Description:
Cleaned up obvious syntax errors to make it to compile.
Changed unnecessarily unsigned variables to signed to avoid
messy compiler warnings.
Platforms tested:
eirene (parallel)
Backward Compatibility Fix
Description:
One of H5P[gs]et_cache's parameters changed between v1.4 and the development
branch.
Solution:
Added v1.4 compat stuff around H5P[gs]et_cache implementation and testing
to allow v1.4.x users to continue to use their source code without
modification.
These changes are for everything except the FORTRAN wrappers - I spoke with
Elena and she will make the FORTRAN wrapper changes.
Platforms tested:
FreeBSD 4.4 (hawkwind)
Purpose:
Code Cleanup and Feature Add
Description:
Finally checking in the changes I made to the performance code. It
just modularizes it a bit more and performs some more checks, etc. I
also renamed the timer functions to be more inline with how other
things are named here...
Platforms tested:
Linux
Purpose:
bug Fix
Description:
Changed the code so that if parallel stuff isn't enabled, then we
don't compile the parallel code.
Solution:
Cleaned up the code and put #ifdef's around it checking for parallel
flags.
Platforms tested:
Linux
Bug patch
Description:
pio_xxx.c will fail compiling in serial mode.
I temporary disable the compile of the pio-perform code from
the Makefile. Will fix it after sunday.
Platforms tested:
eirene (serial).
New addition
Description:
Initial version of the Parallel I/O performance measurement program.
Not fully implemented yet but checking them in before I may destroy
them by accident.
Don't run this in small file system (like AFS or eirene) since it
generates gigabytes test files.
Platforms tested:
modi4 64bits. It compiled and ran but took a long time because
the current test parametes are too "wild".
Kludge
Description:
Since we're only about halfway through converting the internal use of
property lists from the "old way" to the generic property lists, we turned
off snapshots to avoid exposing lots of API changes to users, until the
APIs settled down.
Getting the snapshots rolling again seems to have become a priority, so
some changes are going to have to be made now that were going to be
postponed until we were completely finished with the conversion. This
requires that the old API functions be able to deal with both the old
and new property lists smoothly.
Solution:
Kludge together the property list code so that they can transparently handle
dealing with both the old and new property lists
Platforms tested:
FreeBSD 4.4 (hawkwind)
Code cleanup for better compatibility with C++ compilers
Description:
C++ compilers are choking on our C code, for various reasons:
we used our UNUSED macro incorrectly when referring to pointer types
we used various C++ keywords as variables, etc.
we incremented enum's with the ++ operator.
Solution:
Changed variables, etc.to avoid C++ keywords (new, class, typename, typeid,
template)
Fixed usage of UNUSED macro from this:
char UNUSED *c
to this:
char * UNUSED c
Switched the enums from x++ to x=x+1
Platforms tested:
FreeBSD 4.4 (hawkwind)
Bug fix
Description:
Added the condition that Parallel programs are dependent on
the hdf5 library too.
Platforms tested:
eirene (parallel), modi4(serial and parallel).
New feature.
Description:
Test programs were assumed to be serial programs only.
There was no provision to test parallel programs automatically.
Solution:
Added $(TEST_PARA_PROGS) to hold parallel test programs and
added appropriate action entry to test them if defined.
Platforms tested:
Eirene (parallel).
Bug fix.
Description:
The default file name used is /foo/test.out that usually
is not legal. Changed it to /tmp/test.out.
Platforms tested:
eirene (parallel).
bug fix
Description:
Could not get to compile as parallel program in parallel mode.
That was because I did not include hdf5.h before testing for the
H5_HAVE_PARALLEL macro. (DUMB!)
Solution:
#include hdf5.h first.
Platforms tested:
eirene (parallel)
Bug fix
Description:
Included all those generated temporary files in the CLEAN target
so that they would be cleaned in "make clean".
Platforms tested:
eirene.
Bug fix/code cleanup
Description:
Was using old-style property list for dataset transfer.
Solution:
Switched to use generic property list code for dataset transfer proplist.
Platforms tested:
FreeBSD 4.4 (hawkwind)
New feature
Description:
Started this directory for performance measurement programs.
The programs here got compiled but not automatically run (just
like the examples direcotry.)
The programs have existed but now gathered to this one directory.
iopipe.c, chunk.c and overhead.c were from test. perf.c and mpi-perf.c
were from testpar.
Platforms tested:
eirene (serial and parallel).
overhead failed during run due to some property error. This is probably
due to the recent change of properties code.
perf and mpi-perf do not compile correctly for Parallel mode.
Checking them to make them available to others.
