Description: a new dataspace, null dataspace.
Solution: it's a dataspace without any element. This is a staged check-in.
After daily test succeeds, will add a test.
Platforms tested: h5committest
Misc. update: RELEASE.txt
Description: integer-float software conversion. There are some
print commands left in the source code. Will clean them up once
no problem is found in test.
Platforms tested: h5committest.
Misc. update: RELEASE.txt
bug fix
Description:
the fletcher filter used a temporary 2 byte word buffer to compute the checksum.
this is non portable between big-endian/little endian.
Solution:
replaced with a buffer of 1 byte type
Platforms tested:
linux
solaris
solaris 64 bit
AIX
windows
Misc. update:
Code cleanup/bug fix
Description:
Fix configuration files to allow 64-bit compilation of the library with
a 64-bit version of zlib.
Solution:
Moved important compiler flags settings from CC macro to CFLAGS macro.
Instead of setting CC (F9X, CXX) environment variable for compiling for 64-bit
code, users should just set the CFLAGS (FFLAGS, CXXFLAGS) to -q64.
Updated INSTALL file to reflect this change.
Platforms tested:
IBM p690 (copper) w/parallel, FORTRAN, C++, srcdir.
Platforms specific - h5committest does not apply.
Description: software conversion from float to integer(integer to float hasn't
be done yet).
Solution: This is the software conversion.
Platforms tested: h5committest
Misc. update: RELEASE.txt
Purpose:
replaced name of delete filter with remove filter for the new function H5Premove_filter
Description:
Solution:
Platforms tested:
linux
Misc. update:
New Feature. (Contributed by Robb)
Description:
Allow h5ls to specify a VFL with the '--vfl=' command line flag.
Add MPI-I/O & MPI-POSIX VFL drivers to list of drivers available for
h5ls and h5dump.
Platforms tested:
FreeBSD 4.9 (sleipnir) w/parallel
h5committested
Bug fix.
Description:
Allow H5Tget_native_type() to handle opaque fields in compound datatypes.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Bug fix/optimization
Description:
Address slowdown in MPI-I/O file metadata operations that was introduced
mid-stream. We now _require_ a POSIX compliant parallel file system for the
MPI-I/O file driver (as well as for the MPI-POSIX file driver).
Also optimized file open operation when the file is being created by
reducing the number of collective & syncronizing calls.
Additionally, refactor the MPI routines into a common place, eliminating
duplicated code.
Platforms tested:
FreeBSD 4.9 (sleipnir) w/parallel
h5committest
Code cleanup
Description:
Add C++ and FORTRAN wrappers for new H5Pdelete_filter routine, along with
documentation and a note in the release notes.
Platforms tested:
FreeBSD 4.9 (sleipnir)
Linux 2.4 (verbena) w/ C++ and FORTRAN
Too minor for full h5committest
Add tests & docs
Description:
Added tests for the recent B-tree bug fixes & document their fix in the
release notes.
Platforms tested:
FreeBSD 4.9 (sleipnir)
Too minor to require h5committest
Bug fix
Description:
Correct two problems with variable-length datatypes in datasets:
- When overwriting an entire dataset, writing the fill value to the
file would be skipped, causing problems for VL datatypes when
objects in the file had been unlinked (and thus the space in the
file was not all zeros)
- When an application has set a fill-value for a dataset and the
dataset's datatype contained a VL datatype, the library was filling
space on disk with the memory form of the VL information, instead
of the disk form.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Bug fix
Description:
When two property lists are compared, the H5Pequal routine was just
comparing the raw information for the property values. This causes problems
when the raw information contains pointers to other information.
Solution:
Allow a 'compare' callback to be registered for properties, so that a user
application get perform the comparison itself, allowing for "deep" compares of
the property value.
This was exported to the H5Pregister & H5Pinsert routines in the development
branch, but not the release branch.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Purpose: Maintenance
Description: Fortran APIs MAC OS X port for IBM XL Fortran compiler
Solution: Brought back changes from 1.6 branch
Platforms tested: pommier, h5comittested; this time h5committest
complained about copperpp directory and didn't run;
tests on verbena and sol passed.
Misc. update:
Bug fix
Description:
Fix small memory leak of fill-value information inside the library.
Solution:
"reset" the fill-value object header message, so that the internal buffer
gets freed.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Bug fix
Description:
Range check the szip 'pixels per block' against the chunk size of the
dataset when attempting to create a new dataset, since the szip library
requires the PPB to be at least the size of the fastest changing dimension
in the chunk.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor for h5committest
Bug fix
Description:
Fixed bug in hyperslab iteration where certain combinations of flattened
and non-flattened dimensions would cause incorrect locations to be iterated
over.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Code cleanup, bug fixes
Description:
Wrap up rest of changes necessary for fixing the "short" MPI-I/O read
problem that Robb reported.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Bug fix
Description:
Attributes which were created with scalar dataspaces were reporting their
dataspace as a simple dataspace when queried later.
Solution:
Fix the dataspace handling code when reading in the attribute message from
the file to set the extent type correctly.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Bug fix.
Description:
Using a selection offset with hyperslab selections in chunked datasets
was getting into an infinite loop and hanging the application.
Solution:
Apply the selection offset to the hyperslab selection properly.
Platforms tested:
FreeBSD 4.9 (sleipnir) w & w/o parallel
h5committest
Bug fix
Description:
Make a copy of the index value for H5Giterate and use that instead of
dereferencing the index pointer.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor to need h5committest
Bug fix
Description:
Variable length strings and sequences with NULL pointers were not handled
by library, causing problems access the data. This also affected fill values
for variable-length datatypes.
Solution:
Address the issues in the library by detecting NULL sequences/strings
and avoid trying to convert them.
Patched up dumper to display NULL sequences/strings.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Code optimization
Description:
Improved integer & floating-point datatype conversions by removing some
corner cases. Got rid of algorithm which walked the buffer of elements to
convert backwards, switching to algorithm which computes the non-overlapping
space in the buffer and repeatedly converts it in the forward direction.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Cray SV1 & T3E (T90 is not accessible)
Improvement
Description:
If the library is built in parallel mode, then name the "h5cc" and
"h5fc" programs "h5pcc" and "h5pfc" resp. This saves the user from
worrying about mixing serial and parallel builds of the library and
having to specify the pathnames and so-forth.
Platforms tested:
Linux (w/ parallel & Fortran)
Modi4 (w/ parallel & Fortran)
Sol (Serial & Fortran & C++)
Misc. update:
Updated release/docs/RELEASE
Bug fix & code cleanups
Description:
Change our use of MPI derived datatypes to not create datatypes with
"0-sized" lengths, which causes the LANL Q machine to hang.
Also, get rid of "prefer MPI derived datatypes" environment variable since
it has no advantage.
Platforms tested:
FreeBSD 4.9 (sleipnir) w & w/o parallel
h5committest
Description: new function H5Iget_file_id. It returns file ID given an obeject
(group, dataset, or attribute) ID.
Platforms tested: h5committest
Misc. update: RELEASE.txt
Bug fix
Description:
Single hyperslab selections (which were set with only one call to
H5Sselect_hyperslab) that had dimensions that could be "flattened" but were
interspersed with dimensions that could not be flattened were not correctly
handled, causing core dumps.
Solution:
Re-work "flattening" code to handle this case properly.
Platforms tested:
FreeBSD 4.9 (sleipnir) w/parallel
h5committest
bug fix, update documentation
Description:
version 8 of Code Warrior has a bug on the open file function
that causes one of our tests to fail
documented this in the install instructions
removed a CW specific header file include , that was left from version 6
from the file testhdf5.c
Platforms tested:
Code Warrior
linux
Misc. update:
Purpose: Bug fix
Description: Parameter rdcc_nelmts of the h5pget_cache_f subroutine
had wrong INTEGER(SIZE_T) instead of INTEGER type.
Solution: Fixed the type
Platforms tested: arabica in 64-bit mode (where INTEGER(SIZE_T) is not
the same as INTEGER)
Misc. update:
Bug fix
Description:
The 'char *' type is one of the "strongly" aligned types on Crays, but
a 'void *' is "weakly" aligned. So, assigning a 'void *' (pointing to a
location to place a 'char *') to a 'char **' can change the pointer value
during the assignment.
Solution:
Don't alias the 'void *' where the variable-length information ('char *'
or 'hvl_t') will go. Use a temporary variable on the stack to build up the
information about the VL string or sequence and then memcpy() the temporary
variable directly to the location pointed to with the 'void *'
Platforms tested:
FreeBSD 4.9 (sleipnir)
Cray SV1 (wind)
specific to Cray problems, h5committest not necessary.
Bug fix
Description:
When too many messages were inserted into an object header, the library
had an internal pointer to the "new message" that was pointing to the incorrect
location when the array of messages was re-allocated.
In the worst case, this could cause a file to be corrupted.
Solution:
Update the internal pointer when the array is re-allocated.
Platforms tested:
FreeBSD 4.9 (sleipnir)
Too small to require h5committest
Feature add
Description:
Add a few new fields to the H5G_stat_t structure, to allow more information
about the object header to be retrieved.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Add feature
Description:
Add H5Fget_freespace() routine, to check the amount of free space in a
file. This information is only valid until the file is closed currently,
however (until we start recording the free space information in the file
itself).
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Improved algorithm (bug fix, sorta)
Description:
The internal algorithm for adding new IDs in the ID manager code (H5I) was
adding new IDs to the front of the linked list and never adjusting the order
of the items on the list (unless an ID was deleted). If many new objects were
created, they would push earlier ones _way_ down the list (especially if the
objects were being leaked in the application, as they appear to be in the
current HDF-EOS5 library) and would cause O(n) search time for items on the
list.
The ID caching code in the ID manager was avoiding this behavior sometimes,
but it was adding IDs that were looked up to the very tail of the cache and
they would frequently leave the cache before helping.
Solution:
Implemented a "move to front" scheme for the linked list of IDs, which
improves the lookup situation for frequently accessed objects.
Removed ID caching code now, as the "move to front" algorithm actually
works better.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Bug/feature fix.
Description:
Relax restriction on parallel writing to compact datasets to allow partial
I/O.
Updates to reference manual mentioning the issues involved are delayed until
reference manual 'lock' is removed later this week.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Bug fixes and code cleanup
Description:
Lots of changes here:
- Fixed bug #691 - when shared datatypes are used in attributes they
are incorrectly copied into the attribute instead of referring
the the named datatype in the file. This required bumping the
version of the attribute message. The new version of the attribute
message is only written out when a shared datatype is used in
the attribute. [Also, this format change made the size of the
attribute smaller.]
- Added information to attribute debugging routine so that shared
datatypes are displayed correctly with the h5debug tool.
- Refactored the H5O* routines to extract code that was common to
several routines into subroutines to call.
- Added 'link' method for H5O message sub-classes, which increments
the link count on shared objects when a message is created which
shares them.
- Corrected [unreported] bug where the link count was not being
decremented on the shared object when a object header message
with a reference to that object was deleted from the file.
- Reduced size of shared message from 49 bytes (which was incorrect
anyway and should have been 48 bytes) to 10 bytes, which required
bumping the version of "shared" messages.
- Refactored some of the shared datatype routines to allow for easier
queries of "committedness" internally to the library and also
added routine to easily increment/decrement the reference count of
a shared datatype.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Bug fix
Description:
The MPI_File_set_size() routine on ASCI Red is not able to extend files
so that they are larger than 2GB.
Solution:
Add an extra macro which controls whether MPI_File_set_size() can handle
>2GB offsets or if our "older" way of reading a byte, then writing a byte at
the appropriate offset should be used.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Code cleanup, etc.
Description:
Generalize Ray's datatype fixes to handle packing compound datatypes which
are the base type of an array or variable-length type, etc.
Also track "packedness" of a compound datatype from it's creation, instead
of only setting the 'packed' flag after the datatype was explicitly packed.
Updated docs to reflect that a compound datatype is allowed to grow (but
not shrink).
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Bug fix.
Description:
Correct bug where a file opened twice, once with read-write permission
and once with read-only permission would cause closing the file with the
read-only file ID to fail because it was trying to flush information out
of the file.
Solution:
Check the permissions on file IDs that are being closed and only flush
when the particular file ID was opened with write permission.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too small to need h5committest
Misc. update:
Bug fix
Description:
The VFL driver ID in a file's access proprty list wasn't being reference
counted correctly, causing the VFL driver to get prematurely closed after
several calls to "H5Pget_access_plist->H5Pclose".
Solution:
Increment VFL driver ID reference count when copy of file's access property
list is made in H5Pget_access_plist()
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Bug Fix and code cleanup
Description:
Correct error in H5T_detect_class that was causing nested compound datatypes
with to not detect the datatype class of fields correctly, which caused errors
with fill-values, variable-length datatypes and chunks later on.
Return the rank of the array datatype from H5Tget_array_dims(), like
H5Sget_dims().
Lots of cleanups to datatype code, to make the handling of arrays, compound
types, variable-length strings and sequences and enumerated types more
consistent and robust.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Bug fix
Description:
When datasets are deleted from a file, they are removed from the sieve
buffer, but instead of invalidating only the part of the sieve buffer affected,
the sieve buffer code would throw away the entire sieve buffer, potentially
including other raw data in the buffer that hadn't been written to disk yet.
Solution:
Improve the sieve buffer clearing code to handle partial invalidations.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest
Bug fix
Description:
H5Gget_objname_by_idx should allow NULL for the 'name' parameter, to allow
for querying for the name's length.
Solution:
Allow NULL for the 'name' parameter, the internal functions were already
ready to handle that case.
Clean up RM information about H5Gget_objname_by_idx and
H5Gget_objtype_by_idx also.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest
Bug fix
Description:
H5Gget_num_objs, H5Gget_objname_by_idx and H5Gget_objtype_by_idx were
only accepting a group ID, instead of a location ID, as our documentation for
them stated.
Solution:
Allow them to accept a location ID.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest
Bug fix
Description:
I/O on chunked datasets with a scalar dataspace for the memory dataspace
was not working correctly.
Solution:
Translate the scalar dataspace into a n-dimensional (where n is the number
of dimensions of the file's dataspace) dataspace of dimensions 1x1x1...
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest (modulo modi4 which is not working correctly)
Bug fix
Description:
Linking and unlinking objects with multiple internal and/or trailing '/'s
was not handled well in the library.
Solution:
"Normalize" strings by removing trailing '/'s and collapsing multiple
internal '/'s down into just one '/' before operating on the name.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committested
Bug fix
Description:
Allow a user block to be "inserted" in front of a file (probably by
writing a validly-sized userblock to a new file and then appending another
HDF5 file to the new file).
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committested
Update readme
Description:
Noted that hobj_ref_t's and the objno field in the H5G_stat_t structure
can now be directly compared for equality by applications.
Add more contents related to szip
Fix other bugs
Description:
Describe where to find szip, how to turn off szip compression.
Solution:
Platforms tested:
Misc. update:
Bug fix
Description:
I/O on chunked datasets with point selections was not working correctly.
Solution:
Re-wrote some parts of raw data I/O routines that build the selections for
each chunk to correctly handle point selections.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest
Bug fix
Description:
When a non-default indexed storage B-tree internal 'K' value is set by the
user, the chunked datasets created in that file (until it is closed) use the
user's 'K' value and the data can be accessed correctly, but the 'K' value is
not stored in the file.
However, once the file is closed and re-opened, the non-default 'K' value
is lost and the data in the chunked datasets will not be able to be accessed
correctly.
Solution:
Store the indexed storage B-tree internal 'K' value in the superblock.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest
Update install_windows.txt.
Description:
the description of using the allexample.dsw needs to be modified with
the addition of szip to HDF5 library.
Some typos are corrected.
Solution:
Platforms tested:
Misc. update:
Bug fix
Description:
H5T_BKG_TEMP was accidentally removed from library code, but is used by
application's datatype conversion routines.
Solution:
Revert removal of H5T_BKG_TEMP.
Platforms tested:
h5committest
Purpose: Maintenance after 1.6.0 release
Description: I cleaned the file. Please note that information about known
problems, config features, and tested platforms is there.
Please do not forget to modify those sections too!
Solution:
Platforms tested:
Misc. update:
Update configuration
Description:
Now that we've split the tree (Yea!), change the main trunk back to debug
on and production off.
Also, change the version # to 1.7.0
Description: When make install prefix=NEW_DIR is used to specify
new location for the installed HDF5 library, h5redeploy
should be used to fix h5cc, etc. scripts. INSTALL file
suggested to fix scripts by hand.
Solution: Modified instructions to use h5redeploy.
Platforms tested: verbena
Misc. update:
Description: After talking to Barbara we decided to use
ftp://ftp.ncsa.uiuc.edu/HDF/HDF5/current/ location for
Intel and Cray Fortran source code patches.
Solution: Edited the file to point to new location.
Platforms tested:
Misc. update:
update for release
Description:
Set the version information to 1.6.0-pre1 to get ready for
v1.6 release.
Platforms tested:
h5committested.
Misc. update:
Purpose: Maintenance for the 1.6.0 release.
Description: I updated compilers information for Crays and added
h5fc and h5c++ to the new tools sections in REELASE.txt.
I added instructions for how to build HDF5 using Intel compilers
on Windows.
Solution:
Platforms tested:
Misc. update:
Purpose: Maintenance
Description: Bumped the version number to 1.5.62 after creating
a tar ball for the fourth round of testing.
Solution: run bin/h5vers -s scipt on arabica
Platforms tested: N/A
Misc. update:
Purpose: Maintenance for the third round of testing
Description: Increased the version number to 1.5.59 after creating
a tar ball for testing.
Solution:
Platforms tested:
Misc. update:
Bug fix
Description:
The dataset's modification time was getting set whenever raw data was
written with H5Dwrite. Unfortunately, this is a metadata change (which are
required to be performed collectively) and H5Dwrite may be called independently
from a parallel program, resulting in metadata cache corruption and/or program
hangs.
Solution:
Don't update the modification time when raw data it written. :-(
Platforms tested:
FreeBSD 4.8 (sleipnir) w/parallel
h5committest
Misc. update:
Noted in release notes and also sent to Frank for updating the docs.
Code cleanup/new features.
Description:
Clean up whitespace & formatting.
Make the -S (simple) output display a dataset's modification time in UTC
rather than local timezone.
Display variable-length datatype information.
Simplified & enhanced bitfield datatype information displayed.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest
API tweak.
Description:
The H5Sget_select_bounds() API call was using hsize_t arrays for retrieving
the 'start' and 'end' coordinates, which is counter to the rest of the dataspace
API.
Solution:
Change the arrays to be hssize_t instead.
Platforms tested:
FreeBSD 4.8 (sleipnir) w/C++
FreeBSD 4.8 (sleipnir) w/parallel
h5committested
Misc. update:
Updated all docs for this change.
Added 1.4 compatibility #ifdef's
Purpose: Maintenance for 1.6 release
Description: I did more cleanup and restructuring of the file.
Please take a look at the file when you have a chance.
Thank you!
Solution:
Platforms tested:
Misc. update:
Purpose: Maintenance for 1.6 release
Description: HISTORY.txt was not updated in 1.5 branch since 1.2.0 release.
RELEASE.txt had a lot of redundant info that is now in HISTORY.txt
INSTALL_Windows_withF90.txt was lost in 1.5 branch.
Solution: I updated HISTORY.txt to include release history up to 1.4.5 release.
After that I removed all information related to the 1.4.* releases
from RELEASE.txt file. "Platforms Tested" section was brought up
to date based on the information provided by developers for 1.6.0 first
round of testing. RELEASE.txt still needs a lot of work.
INSTALL_Windows_withF90.txt is also a first draft, more work is needed.
Platforms tested: N/A
Misc. update:
Purpose: Maintenance for the second round of testing
Description: Used bin/h5vers to change version number to 1.5.55
Solution:
Platforms tested: arabica
Misc. update:
Add decriptions to use libtestD.dll.
Description:
libtestD.dll should be placed to a place that the user can find in order to
test HDF5 DLL within HDF5 workspace.
Solution:
Explained in document.
Platforms tested:
No need
Misc. update:
code warrior update
Description:
changed the project file to include the new H5Dio.c and remove the old H5Farray.c files
updated install instructions
Solution:
Platforms tested:
code warrior
Misc. update:
Purpose: Maintenance
Description: I changed release version to 1.5.54.
Email about tetsing of 1.5.53 will be send after this checkins.
Solution: used bin/h5vers -s to change the version number
Platforms tested: Done on arabica
Misc. update:
New feature/enhancement
Description:
Chunked datasets are handled poorly in several circumstances involving
certain selections and chunks that are too large for the chunk cache and/or
chunks with filters, causing the chunk to be read from disk multiple times.
Solution:
Rearrange raw data I/O infrastructure to handle chunked datasets in a much
more friendly way by creating a selection in memory and on disk for each chunk
in a chunked dataset and performing all of the I/O on that chunk at one time.
There are still some scalability (the current code attempts to
create a selection for all the chunks in the dataset, instead of just the
chunks that are accessed, requiring portions of the istore.c and fillval.c
tests to be commented out) and performance issues, but checking this in will
allow the changes to be tested by a much wider audience while I address the
remaining issues.
Platforms tested:
h5committested, FreeBSD 4.8 (sleipnir) serial & parallel, Linux 2.4 (eirene)
Feature Add
Description:
Added knob so that the programmer can enable or disable GPFS
hints during runtime instead of having it only enabled at
configure/compile time. Some of the public APIs were changed
to add an extra parameter for this option...
Platforms tested:
Blue (LLNL). It only affects the MPI/POSIX driver, so no need
to test it on non-GPFS platforms.
Misc. update:
Add a little more comments on where to put DLL.
Description:
User can put HDF5DLL in the place where the application can find.
It doesn't have to be at system directory.
Solution:
Add comments to "only suggest the user to use system directory".
Platforms tested:
text file, no need to be tested.
Misc. update:
Update stdio driver supported on windows
Description:
To tell users how to set and use stdio driver.
Solution:
Platforms tested:
Text file doesn't have to be tested.
Misc. update:
Update
Description:
The H5FC was already in this doc...but under "bug fixes". Removed
that since it's really a new feature.
Platforms tested:
IBM/PC Jr.
Misc. update:
Bug fix
Description:
This fixes a bug in the low-level metadata caching code in the library
which could possibly lose metadata during file I/O when a lot of objects are
inserted into a group.
This also fixes a couple of (similar) fencepost bugs in the B-tree
deletion code.
Solution:
For the metadata bug - call the low-level driver's 'write' routine instead
of H5FD_write.
For the B-tree bug - include the correct number of keys.
Platforms tested:
FreeBSD 4.8 (sleipnir) w/C++
Linux 2.4 (burrwhite) w/FORTRAN
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/FORTRAN & parallel
(h5committest is still not working for me on burrwhite)
Misc. update:
Purpose:
Maintenance
Description:
* Added support for generic properties.
* Added support for time allocation properties.
* Added support for variable length datatypes (only datatypes based
on INTEGER, REAL and CHARACTER Fortran types are supported).
* added some missing functions
Solution:
I am checking in new Fortran APIs and their man pages to support
1.5 features listed above.
Not all APIs have tests yet. APIs were written in Fall 2002, and I am afraid
that I will loose the code or totally forget what I did if I wait longer. ;-)
Platforms tested:
arabica (fortran), eirene (fortran), modi4 (parallel, fortran)
Purpose:
1.update windows release doc related to the addition of semi-automatic generated H5Tinit.c project and workspace files.
2. Make the doc more clear.
Description:
1. H5Tinit.c was manually maintained before and it might affect the users who wants to build and test from HDF5 snapshot release.
2. Make sure that we are supporting .Net only on XP and also emphasize to DLL users to the DLL critical section of the doc.
Solution:
1. Add a section for those users to semi-auto generate H5Tinit.c.
Platforms tested:
Doc files, no need to test.
Misc. update:
Bug Fix
Description:
Metadata cache in parallel I/O can cause hangs in applications which
perform independent I/O on chunked datasets, because the metadata cache
can attempt to flush out dirty metadata from only a single process, instead
of collectively from all processes.
Solution:
Pass a dataset transfer property list down from every API function which
could possibly trigger metadata I/O.
Then, split the metadata cache into two sets of entries to allow dirty
metadata to be set aside when a hash table collision occurs during
independent I/O.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel
Misc. update:
Updated release_docs/RELEASE
New feature for developers.
Description:
Added "function stack" tracing to library. This allows developers (there
is no public API) to call H5FS_print within the library and get a listing
of the functions traversed to reach that point in the library. Eventually,
I may add support for reporting the parameters to each function also...
Mainly for debugging parallel I/O programs, but I think it will come in
handy in other cases also.
The function stack tracking is controlled with a configure switch:
--enable-funcstack, which defaults to enabled currently. When we branch
for 1.6, we should change the default setting on the branch to be disabled.
Also, added a destructor to the thread-specific keys when thread-safety is
turned on in the library. Otherwise, they were leaking memory and causing
difficult to debug errors in threaded programs (like the test/ttsafe test).
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) w/thread-safety enabled.
Misc. update:
Updated MANIFEST with new files added (src/H5FS.c & src/H5FDprivate.h)
Update release_docs/RELEASE with thread-safety bug fix.
Bug Fix
Description:
When calling H5Fopen with the core VFL driver, but without the
H5F_ACC_CREAT flag goes ahead and creates a memory file.
Solution:
Check for the H5F_ACC_CREAT flag before allowing the memory file to be
created.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Update
Description:
Added description of the configure bugfix for people specifying
--with-*=/usr/include,/usr/lib in their scripts during configure...
Bug fix
Description:
Currently, when the library encounters an object header message that isn't
know, it fails to open that object in the file.
Solution:
Allow the library to skip over the unknown object header message and
continue to process the remaining messages, in the hope that the skipped
message isn't important later. If it is important, it will be caught at
a higher level of the library.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
Purpose:
add "shuffle" filter note
Description:
Solution:
Platforms tested:
Misc. update:
Update MANIFEST if you add or remove any file.
Update release_docs/RELEASE for bug fixes, new features, etc.
Update applicable document files too.
Purpose:
New function.
Description:
H5Dget_offset returns the offset of a dataset's data relative to the
beginning of a file.
Platforms tested:
arabica(simple function, one test should be enough.)
Misc. update:
RELEASE.txt
New feature
Description:
Add support for scalar dataspaces in parallel I/O.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Also, FreeBSD 4.7 (sleipnir) serial & parallel
Misc. update:
Update release_docs/RELEASE for bug fixes, new features, etc.
Purpose:
Change release note for new functions(H5Gget_comment, H5Arename,
H5Aget_storage_size).
Misc. update:
Update release_docs/RELEASE for bug fixes, new features, etc.
Purpose:
Added missing fortran functions h5set(get)_buffer_f.
Also added docs and tests for them.
Solution:
Currently functions do not accept conversion and background buffers.
This corresponds to H5set(get)_buffer call with buffer pointers
set to NULL. If there is a demand, I can overload the functions
to have new parameters and go through all trouble creating functions
for all supported datatypes.
Platforms tested:
Solaris 2.7, Linux 2.2., IRIX64-6.5
Purpose:
Added new F90 APIs
Description:
I added new F90 APIs, tests, and documentation for the following
functions:
h5fget_obj_count_f h5pequal_f h5tget_member_index_f
h5fget_obj_ids_f h5pget_fclose_degree_f
h5pset_fclose_degree_f
Documentation for exisiting functions was missing:
h5freopen_f, h5fflush_f, h5fmount_f, h5unmount_f, h5fget_create_plist_f,
h5fget_access_plist_f.
Platforms tested:
Solaris 2.7, Linux 2.2 and IRIX64-6.5
Purpose:
New API functions
Description:
Added API functions to return pointer to low-level file handle
(H5Fget_vfd_handle and H5FDget_vfd_handle) and related property list
setting functions(H5Pset_family_offset and H5Pset_multi_type).
Platforms tested:
Linux 2.2(eirene), Solaris 2.7(arabica), IRIX64 6.5(modi4)
Purpose:
Added new fortran functions
Description:
Source code, tests and documentation for h5glink2_f and h5gmove2_f
functions were added. I also added missing man page for h5glink_f
Platforms tested:
Solaris 2.7, IRIX64-6.5 and Liniux 2.2
Purpose:
Added missing fortran functions.
Description:
Four Library Fortran API functions have been added:
h5get_libversion_f, h5_check_version_f, h5garbage_collect_f and
h5dont_atexit_f. Only first two functions were tested.
Documentation file and RELEASE.txt were updated.
Platforms tested:
Solaris 2.7, IRIX64-6.5 and Linux 2.2
API name change
Description:
Change all "space time" references to "alloc time", including API functions
and macro definitions, etc.
Platforms tested:
FreeBSD 4.6 (sleipnir) w/C++
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/parallel & FORTRAN
Code cleanup/New Feature
Description:
Improve the space allocation in the file by re-using freed space more
effectively.
Platforms tested:
FreeBSD 4.6 (sleipnir) w/serial & parallel
Purpose:
Merged from 1.4 branch
Description:
API tracing improvements
Platforms tested:
Linux (--disable-hsizet didn't work before my changes and still doesn't
work, but --enable-hsizet is fine).
Note: This checkin includes temporary code in H5FDmpiposix.c to turn
off GPFS byte range token prefetches on ASCI/Blue. Once the HDF5 API
supports the necessary prerequisite functionality this temporary code
can be migraged up above HDF5.
Purpose:
update windows support.
Description:
1. Since we don't support w98 anymore, delete the description about w98.
Actually DLL may work on w98.
2. Release dll work for new HDF5 release.
Solution:
Platforms tested:
Bug Fix
Description:
It was possible to create corrupted metadata information (either in memory
or in the file or both) with a parallel I/O program because of the way
metadata writes were being handled for writes out of the metadata cache.
Solution:
Added a dataset transfer property called "block before metadata write"
which is used by the MPI-I/O and MPI-posix drivers to sync up all the
processes before attempting a metadata write. This property is currently
only for metadata writes from the metadata cache.
Platforms tested:
IRIX64 6.5 (modi4) w/parallel
Bug Fix
Description:
When parallel I/O is used, the MPI-I/O VFL driver uses a "lazy" model to
call MPI_File_set_view() in order to reduce the number of calls to this
function. However, this is unsafe, because if a collective I/O which uses
MPI derived types (and thus uses MPI_File_set_view()) is immediately
followed by an independent I/O, the code will attempt to call
MPI_File_set_view() in order to switch back to the default view of the
file. MPI_File_set_view() is a collective call however, and this causes
the application to hang.
Solution:
Removed "lazy" MPI_File_set_view() code, instead set the file view when it
is needed (with MPI derived types) and immediately set the file view back to
the default view before leaving the I/O routine.
Platforms tested:
IRIX64 6.5 (modi4) w/parallel. Also, tested with the latest development
and release code for the SAF library, which now works correctly with this
change. (Although the release branch of the SAF library seems to have a
bug, this 1.4.4 release candidate code gets as far as the version the SAF
library is released on top of (1.4.2-patch1, I believe)).
New feature.
Description:
There is some discussion among the SAF team as to whether it is better
to use MPI derived types for raw data transfers (thus needing a
MPI_File_set_view() call), or whether it is better to use a sequence of
low-level MPI types (i.e. MPI_BYTE) for the raw data transfer.
Solution:
Added an in internal flag to determine whether derived types are preferred
(the default), or whether they should be avoided. An environment variable
("HDF5_MPI_PREFER_DERIVED_TYPES") can be set by users to control whether MPI
types should be used or not. Set the environment variable to "0" (i.e.:
'setenv HDF5_MPI_PREFER_DERIVED_TYPES 0' to avoid using MPI derived types.
Platforms tested:
IRIX64 6.5 (modi4) w/parallel
Bug fix.
Description:
The chunking code was using internal allocation routines to put blocks on
a free list for reuse, instead of using the system allocation routines (ie.
malloc, free, etc.). This causes problems when user filters attempt to
allocate/free chunks for their algorithm's use.
Solution:
Switched the chunking code back to using the system allocation routines,
we can address performance issues with them if it becomes a real problem.
Platforms tested:
Linux 2.2.x (eirene) && IRIX64 6.5 (modi4)
Code optimization
Description:
Avoid creating MPI types (and thus requiring a MPI_File_set_view() call)
when contiguous selections are used for dataset I/O. This should be a
performance improvement for those sorts of selections.
Platforms tested:
Linux 2.2.x (eirene) w/parallel && IRIX64 6.5 (modi4) w/parallel & FORTRAN
Bug fix
Description:
I/O on "Regular" hyperslab selections could fail to transfer correctly
if the number of elements in the selection's row did now fit "evenly"
into the buffer being used for the transfer.
Solution:
Correct the calculation of the block & count offsets within the optimized
"regular" hyperslab routines.
Platforms tested:
FreeBSD 4.5 (sleipnir)
Update
Description:
Explained that, if the user uses the "make install prefix=NEW_DIR"
option, they'll need to modify the installed h5cc file to reflect the
change.
Bug Fix
Description:
H5Dcreate and H5Tcommit allow "empty" compound and enumerated types (i.e.
ones with no members) to be stored in the file, but this causes an assertion
failure and is somewhat vapid.
Solution:
Check the datatype "makes sense" before using it for H5Dcreate and
H5Tcommit.
Platforms tested:
FreeBSD 4.5 (sleipnir)
Bug Fix (#709)/Code improvement.
Description:
Allow chunks for chunked datasets to be cached when file is open for
read-only access.
Platforms tested:
IRIX64 6.5 (modi4) w/parallel
Bug fix (bug #777)
Description:
Current code allows a compound datatype to be inserted into itself.
Solution:
Check if the ID for the member is the same as the ID for the compound
datatype and reject it if so.
Platforms tested:
FreeBSD 4.5 (sleipnir)
Bug Fix for bug #789
Description:
Creating a 1-D dataset region reference caused the library to hang (go into
an infinite loop).
Solution:
Corrected algorithm for serializing hyperslab regions.
Platforms tested:
FreeBSD 4.5 (sleipnir)
New feature.
Description:
Added a "small data" block allocation mechanism to the library, similar to
the mechanism used for allocating metadata currently.
See the RFC for more details:
http://hdf.ncsa.uiuc.edu/RFC/SmallData/SmallData.html
This reduces the number of I/O operations which hit the disk for my test
program from 19 to 15 (i.e. from 393 to 15, overall).
Platforms tested:
Solaris 2.7 (arabica) w/FORTRAN and FreeBSD 4.5 (sleipnir) w/C++
Purpose:
Bug fix (#699), fix provided by a user, approved by Quincey
Description:
When a scalar dataspace was written to the file and then
subsequently queried with the H5Sget_simple_extent_type function,
type was reported H5S_SIMPLE instead of H5S_SCALAR.
Solution:
Applied a fix (see bug report 699)
Platforms tested:
Solaris 2.7 and Linux 2.2.18
Code improvement
Description:
The metadata aggregation code in the library was not terribly smart about
extending contiguous regions of metadata in the file and would not extend
them as far as possible. This causes space in the file to be wasted, also.
Solution:
Be smarter about extending the space used in the file for metadata by
checking whether new metadata blocks allocated in the file are at the end
of the current metadata aggregation region and append them to the metadata
region if so. This has the nice side benefit of reducing the number of
bytes we waste in the file and reducing the size of the file by a small
amount in some cases.
This reduces the number of I/O operations which hit the disk for my test
program from 53 to 19 (i.e. from 393 to 19, overall).
Platforms tested:
Solaris 2.7 (arabica) w/FORTRAN and FreeBSD 4.5 (sleipnir) w/C++
Bug Fix
Description:
The "dirty" flag for symbol table entries and symbol table nodes was not
being cleared when they were flushed to the file, causing lots of extra
metadata I/O.
Solution:
Reset the symbol table entry & nodes' flags when thy are flushed to disk.
This reduces the number of I/O operations which hit the disk for my test
program from 83 to 53 (i.e. from 393 to 53, overall).
Platforms tested:
Solaris 2.7 (arabica) w/FORTRAN & FreeBSD 4.5 (sleipnir) w/C++
Code cleanup/bug fix
Description:
The "metadata accumulator" cache in the library (which is designed to catch
small metadata writes/reads and bundle them together into larger I/O
buffers) was incorrectly detecting the important case of metadata pieces
being written sequentially to the file, adjoining but not overlapping.
Additionally, the metadata accumulator was not being used to cache data
read in from disk, only caching writes.
Solution:
Fix accumulator to correctly cache adjoining metadata writes and also to
cache metadata read from disk.
Between these two fixes, the number of I/O requests which resulted in actual
reads/writes to the filesystem dropped from 393 requests to 82 for the
particular test I was using. :-)
Platforms tested:
Solaris 2.7 (arabica) w/FORTRAN & FreeBSD 4.5 (sleipnir) w/C++
Document Bug Fix
Description:
Under certain [obscure] circumstances, an object header would get paged out
of the metadata cache, and when it was accessed again and brought back into
the cache, and immediately had additional metadata added to it (an
attribute, usually, or perhaps adding an object to a group), and needed to
be extended with a continuation message, but there was no room in any
existing object header chunks for the continuation message and an existing
object header message needed to be moved to the new object header chunk (I
told you it was obscure :-), the object header message moved to the new
chunk (not the new metadata being added) would get corrupted. *whew* :-)
Solution:
Actually copy the "raw" object header message information of the object
header message being moved to the new chunk, instead of relying on the
"native" object header message information being re-encoded when the object
header is flushed. This is because when an object header is paged out of
the metadata cache and subsequently brought back in, the "native"
information pointer in memory is reset to NULL and only the "raw"
information exists.
Platforms tested:
Solaris 2.7 (arabica) & FreeBSD 4.5 (sleipnir)
Purpose:
update, remove hdf4-related stuff.
Description:
hdf4 related tools have been moved out of HDF5 CVS tree, The install doc should reflect this.
Solution:
Platforms tested:
