Purpose:
Bug fix
Description:
modi4 dies during build with strange errors.
The root cause of these is a two-year-old hack in HDF5's libtool
script that only takes effect on IRIX.
Solution:
Edited the libtool hack (by editing ltmain.sh) to correct a bug in
the hack.
Also made sure that compiler-specific DEFAULT_LIBS are used
when linking.
Platforms tested:
sleipnir, copper, modi4, sol
Bug fix
Description:
Arabica exhibited strange errors when linker found wrong versions of
header files. This happened because include directories were
given to linker in the wrong order.
Solution:
Move include directories from AM_CFLAGS variable to INCLUDES
variable to put them before CPPFLAGS variable. Trust me, it works.
This bug may also have contributed to strange errors on other platforms
(kelgia?).
Platforms tested:
copper, sleipnir, arabica.
(h5dump broke while building on arabica, but this happened in
a clean checkout, too).
Bug fix
Description:
pmake (on modi4, for instance) complains about undefined variables
if it is run without the -V flag, which turns those errors to warnings.
Solution:
Added test to configure.in to see if $MAKE will allow Makefiles
with undefined variables. If not, sets -V flag in AM_MAKEFLAGS.
Ensured that all custom make targets use AM_MAKEFLAGS.
Also defined all variables that caused errors in top-level Makefile.am.
This means that pmake can be used to build hdf5 *from the top level
only*. To run make from a subdirectory, still need to use -V flag
(or use make or gmake).
Platforms tested:
modi4, heping, copper, sleipnir
Bug fix (sorta)
Description:
Disable btree2 testing until Monday, when I can look into the failures
without a toddler on my lap. :-)
Platforms tested:
FreeBSD 4.11 (sleipnir)
Purpose:
Bug fix
Description:
Parallel builds were breaking in tools/lib
Solution:
talign didn't realize it depended on libh5tools.la because its
dependencies listed the full path (../../tools/lib/libh5tools.la).
Changed this, and made similar changes in a couple of other
directories.
This checkin should also add the --foreign flag to every Makefile.in
Platforms tested:
sleipnir (minor change)
New feature
Description:
Add basic code for new B-tree implementation. They don't do much yet,
aren't hooked up to anything yet and the format may change, but I'd like to
start getting them into the daily tests.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Too minor to require h5committest
Bug fix
Description:
Found the permanant fix to automake/CVS dependency problem
Solution:
Added AM_MAINTAINER_MODE macro to configure.in.
Now automake will never try to regenerate Makefiles, Makefiles.in,
configure, H5config.h, etc. when they are out of date, nor will it
print any warnings.
Developers should be very very careful to use reconfigure script,
and can add --enable-maintainer-mode flag to configure on heping
to regenerate these files correctly.
Platforms tested:
heping sleipnir copper
Bug fix
Description:
Configuration files' timestamps were incorrect.
Solution:
Update Makefiles.in. Also added correct paths to autotools on heping,
so heping build should be able to re-generate configuration files even
if they are still broken.
Platforms tested:
sleipnir
Bug fix
Description:
Dependencies between configure files (aclocal, configure.in, configure,
Makefiles.am and .in) are still causing Makefiles to try to run autotools
during build.
Solution:
Committed all Makefiles.in to update their timestamps.
As a temporary measure, forcibly prevented automake from running
autotools during build by redefining the variables it uses.
Platforms tested:
sleipnir (No changes to Makefile content)
Bug fix
Description:
Different compilers use different flags to include Fortran module files
Solution:
Changed boilerplate to use configure variable rather than hardcoded -M flag.
Platforms tested:
sleipnir, sol, copper
Configuration feature
Description:
HDF5 now uses automake to generate Makefiles
Solution:
Makefile.in files are now generated from Makefile.am files.
To reconfigure (after chaning a Makefile.am or configure.in):
/bin/sh bin/reconfigure.sh
Platforms tested:
Many
Remove feature
Description:
Retire threaded, balanced binary tree code from HDF5 use. Requiescat in
pace...
Also, regenerate dependencies files.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Too minor to require full h5committesting (the code is already
disconnected from everything except its tests)
Add new internal data structure
Description:
Add an implementation of skip lists to the library (see comment in
src/H5SL.c for references to the papers describing them) as a potential
replacement for our current threaded, balanced binary tree container.
Skip lists are much simpler to implement and should be faster to use.
Also, added new error codes to release branch, so bump the minor version
number to indicate that the library is no longer perfectly compatible with
the 1.6.3 release.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Solaris 2.7 (arabica)
Too minor to require further testing (the skip lists aren't actually
used by any library code yet)
Cleanup & add test
Description:
Clean up test makefile a bit and add metadata cache test to 1.6 branch.
Platforms tested:
FreeBSD 4.10 (sleipnir)
Too minor to require h5committest
Bug fix
Description:
Fix situation where deleting a chunked datasets with B-tree nodes that
weren't in the metadata cache would die with a core dump.
Platforms tested:
FreeBSD 4.10 (sleipnir)
Linux 2.4 (heping)
Solaris 2.7 (arabica)
Code cleanup
Description:
Various minor tweaks to clean code up and bring it into closer
syncronization with the release branch.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
h5committested
IRIX64 6.5 (modi4)
Purpose:
Bug Fix
Description:
If an HDF5 file grows larger than its address space, it dies and is unable to
write any data. This is more likely to happen since users are able to change
the number of bytes used to store addresses in the file.
Solution:
HDF5 now throws an error instead of dying. In addition, it "reserves" address
space for the local heap and for object headers (which do not allocate space
immediately). This ensures that after the error occurs, there is enough address
space left to flush the entire file to disk, so no data is lost.
A more complete explanation is at /doc/html/TechNotes/ReservedFileSpace.html
Platforms tested:
sleipnir, copper (parallel), verbena, arabica, Windows (Visual Studio 7)
Solution:
Platforms tested:
Misc. update:
Description:
Replaced the old metadata cache with a cache with a modified LRU
replacement policy. This should improve the hit rate.
Solution:
Since we want to flush cache entries in increasing address order, I
used the threaded binary B-tree code to store the cache entries.
There is a fair bit of overhead here, so we may want to consider
other options.
While the code is designed to allow the support of other replacement
algorithms, at present, only a modified version of LRU is supported.
The modified LRU algorithm requires that a user selectable portion
of the cache entries be clean. The clean entries are evicted first
when writes are not permitted. If the pool of clean entries is used
up, the cache grows beyond its user specified maximum size. The
cache can also exceed its maximum size if the combined size of the
protected (or locked) entries exceeds the maximum size of the cache.
Platforms tested:
eirene (serial, parallel, fp), h5committested
Misc. update:
Description: Added new API H5Fget_name and new test program called filename.c. This function
returns the name of the file by object ID(file, group, dataset, named datatype, and attribute)
which belongs to the file.
Platforms tested: h5committest and fuss.
Misc. update: MANIFEST and RELEASE.txt
Code cleanup
Description:
Add more files that should be removed to the clean & distclean targets.
Platforms tested:
FreeBSD 4.10 (sleipnir)
Too minor to require h5committest
New Feature
Description:
Add the data transform function, H5Pset_transform().
Platforms tested:
"h5committested".
Copper was down. Ran parallel tests in sol instead.
Misc. update:
Code cleanup.
Description:
The routines residing in testframe.c was defined in testhdf5.h due to
historical reason. It really belongs to h5test.h because those routines
reside in libh55test.a.
Solution:
Moved them to the right place. Also removed the duplicated occurance
of testframe.c in the TEST_SRC.
Platforms tested:
Tested in o2 (SGI) parallel.
Bug fix
Description:
Correct two problems with variable-length datatypes in datasets:
- When overwriting an entire dataset, writing the fill value to the
file would be skipped, causing problems for VL datatypes when
objects in the file had been unlinked (and thus the space in the
file was not all zeros)
- When an application has set a fill-value for a dataset and the
dataset's datatype contained a VL datatype, the library was filling
space on disk with the memory form of the VL information, instead
of the disk form.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Code cleanup & reorganization
Description:
Move further in the testing framework cleanup, eliminating all the
global variables (moving them into testframe.c as static variables) from the
testing framework code and moving it into the libh5test.a.
Platforms tested:
FreeBSD 4.9 (sleipnir) w & w/o thread-safety, c++ & parallel
h5committested
Code cleanup
Description:
Refactor library testing framework (used for the testhdf5 & ttsafe tests)
to remove almost all of the duplicated code, moving the common code into a
new 'testframe.c' source file.
Platforms tested:
FreeBSD 4.9 (sleipnir) w & w/o thread-safety
h5committest
Code cleanup & add a feature
Description:
Added support for querying the file ID of named datatypes to new
H5Iget_file_id function.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Description: The error test script compares the error messages to the standard
output. The file names and line numbers in the messages can be different
for users.
Solution: Temporarily disable the test until find a good solution.
Platforms tested: None. Disabling a test.
Bug fix
Description:
When datasets are deleted from a file, they are removed from the sieve
buffer, but instead of invalidating only the part of the sieve buffer affected,
the sieve buffer code would throw away the entire sieve buffer, potentially
including other raw data in the buffer that hadn't been written to disk yet.
Solution:
Improve the sieve buffer clearing code to handle partial invalidations.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest
Bug fix
Description:
Allow a user block to be "inserted" in front of a file (probably by
writing a validly-sized userblock to a new file and then appending another
HDF5 file to the new file).
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committested
New test
Description:
Added test to check for dangling objects being handled correctly by the
library.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committested
New test feature
Description:
Added the VFD_LIST definition to support the "make check-vfd" tests.
Platforms tested:
h5committested.
Since check-vfd is not invoked by default, I went to each machine
to invoke them by hand. The target worked fine but there were
errors reported from those driver tests. (Those errors were beyond
the scope of this make target test.)
Misc. update:
New internal feature
Description:
Add internal API for building and working with heaps (H5HP). This will be
used for the LRU algorithm in the new metadata cache code.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir)
Bug Fix
Description:
When calling H5Fopen with the core VFL driver, but without the
H5F_ACC_CREAT flag goes ahead and creates a memory file.
Solution:
Check for the H5F_ACC_CREAT flag before allowing the memory file to be
created.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
Code cleanup
Description:
Added more test datafiles to clean directive.
Platforms tested:
FreeBSD 4.6 (sleipnir) w and w/o parallel
Linux 2.2.x (eirene) w/FORTRAN & C++
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/FORTRAN & parallel
