Code optimization
Description:
Reduce the number of times the number of elements in a selection is
computed.
Platforms tested:
Solaris 2.7 (arabica)
FreeBSD 4.9 (sleipnir) w/parallel
too minor to require h5committest
Code optimization
Description:
Move the element size for the selection into the selection iterator instead
of always passing it as a parameter.
Also, eleminate another 64-bit multiply for "all" selections.
Platforms tested:
Solaris 2.7 (arabica)
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Code optimization
Description:
Stop straddling the line and always use elements instead of bytes to
determine how how I/O to perform on a selection.
Platforms tested:
Solaris 2.7 (arabica)
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Code optimization
Description:
Instead of dynamicly allocating various arrays for various pieces of
information about a selection or selection iterator, just use fixed size
array of size H5S_MAX_RANK (as the rest of the library does).
Platforms tested:
Solaris 2.7 (arabica)
FreeBSD 4.9 (sleipnir)
h5committest
Code cleanup
Description:
Update null dataspace changes to try to write older version of dataspace
information whenever possible.
Refactor common code to only one location.
Allow I/O operations to succeed on null dataspaces.
Platforms tested:
FreeBSD 4.9 (sleipnir)
h5committest
Code optimization
Description:
Eliminate memory allocations for I/O vectors when using the default
vector size.
Platforms tested:
Solaris 2.7 (arabica)
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Code optimization
Description:
Eliminate unnecessary allocation and point at existing data structure
instead.
Platforms tested:
Solaris 2.7 (arabica)
too minor to require h5committest
Code optimization
Description:
Fix H5S_select_copy so it doesn't call calloc() for allocating memory that
will be immediately overwritten.
Platforms tested:
Solaris 2.7 (arabica)
too small to require h5committest
Code optimization
Description:
Move handling for free list arrays that have no maximum size to separate
set of routines and optimize computations for free list arrays with maximum
size to avoid re-computing sizes all the time.
Platforms tested:
h5committest
Solaris 2.7 (arabica)
Code cleanup/optimization
Description:
Query property list values once, at the beginning of the I/O routines,
instead of querying the property list values multiple (lots!) of times in
lower level routines.
Solution:
Create "property list caches" for internal library queries of the property
list values.
Platforms tested:
IBM p690 (copper) w/parallel & fphdf5
h5committest
Bug fix
Description:
Fixed bug in hyperslab iteration where certain combinations of flattened
and non-flattened dimensions would cause incorrect locations to be iterated
over.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor to require h5committest
Bug fix
Description:
Clean up a few allocations of zero-sized blocks that were detected with
the new free-list assertions.
Platforms tested:
FreeBSD 4.9 (sleipnir) w & w/o parallel
too minor to require h5committest
Code cleanup
Description:
Clean up compiler warnings, especially the 'FUNC' variable not used which
comes out in production mode.
Solution:
Had to add a new FUNC_ENTER_NOAPI_NOINIT_NOFUNC macro for those non-API
functions which don't need the 'FUNC' variable defined. (This will be _so_
much easier when C99 is standard on all our supposed platforms, since it has a
__FUNC__ macro... )
Platforms tested:
FreeBSD 4.9 (sleipnir)
too minor for h5committest (although there were lots of files changed, the
change was minor in each one)
API tweak.
Description:
The H5Sget_select_bounds() API call was using hsize_t arrays for retrieving
the 'start' and 'end' coordinates, which is counter to the rest of the dataspace
API.
Solution:
Change the arrays to be hssize_t instead.
Platforms tested:
FreeBSD 4.8 (sleipnir) w/C++
FreeBSD 4.8 (sleipnir) w/parallel
h5committested
Misc. update:
Updated all docs for this change.
Added 1.4 compatibility #ifdef's
Performance improment
Description:
Speed up chunked dataset I/O. This breaks down into several areas:
- Compute chunk selections in the file by using hyperslab operations
instead of iterating over each element in the selection.
- If the file and memory selections are the same shape, use the file
chunk selections to compute the memory chunk selections.
This required several additional dataspace, dataspace selection and
hyperslab routines.
Platforms tested:
h5committestted (although Fortran tests failed for some reason)
New feature/enhancement
Description:
Chunked datasets are handled poorly in several circumstances involving
certain selections and chunks that are too large for the chunk cache and/or
chunks with filters, causing the chunk to be read from disk multiple times.
Solution:
Rearrange raw data I/O infrastructure to handle chunked datasets in a much
more friendly way by creating a selection in memory and on disk for each chunk
in a chunked dataset and performing all of the I/O on that chunk at one time.
There are still some scalability (the current code attempts to
create a selection for all the chunks in the dataset, instead of just the
chunks that are accessed, requiring portions of the istore.c and fillval.c
tests to be commented out) and performance issues, but checking this in will
allow the changes to be tested by a much wider audience while I address the
remaining issues.
Platforms tested:
h5committested, FreeBSD 4.8 (sleipnir) serial & parallel, Linux 2.4 (eirene)
Code cleanup/new feature.
Description:
Split FUNC_LEAVE into API and non-API specific versions. This allows a
solution to compiling this branch with C++, as well as reducing the size
of the binaries produced.
Platforms tested:
FreeBSD 4.7 (sleipnir) w/serial, parallel (including MPE) & thread-safe
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
Bug fix & code cleanup
Description:
Hyperslab code for collapsing dimensions was incorrectly collapsing
selections inappropriately when the fastest changing dimension couldn't
be collapsed.
Also add some more assertions which will make similar bugs easier to find.
Solution:
Break out of loop earlier.
Platforms tested:
modi4 (parallel), too small to triple check.
Bug fix (ID 841)
Description:
h5dumptst (generating h5dump test files) failed with an assertion error
in H5S_select_iterate. H5S_select_iterate did not account for scalar
type that has rank=0, thus no dimension sizes.
Solution:
Put the assertion of valid sizes only after verify rank is larger than
0. (fix is coached by Quincey.)
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}? YES.
Misc. update:
Update release_docs/RELEASE for bug fixes, new features, etc. YES.
Code cleanup
Description:
Switch order of test on array bounds to avoid reading data outside of array
Located by purify.
Platforms tested:
FreeBSD 4.6 (sleipnir) w/C++
Solaris 2.7 (arabica) w/C++
IRIX64 6.5 (modi4) w/parallel & fortran
Bug fix/Code cleanup/New Feature
Description:
Correct problems with writing fill-values to external storage and allocate
the data storage at the correct times.
Also, mostly straighten out the strange code which allocates and fills
raw data storage for datasets. Things are still a bit odd in that the
fill-values for chunked datasets are written when the space is allocated,
instead of in a separate routine, but there are two reasons for this:
it's inefficient (especially in parallel) to iterate through all the chunks
twice, and (more importantly) the space needed to store compressed chunks
isn't known until we've got a buffer of compressed fill-values ready to
write to the chunk.
Additionally, add in the H5D_SPACE_ALLOC_INCR and H5D_SPACE_ALLOC_DEFAULT
setting for the "space time", which incorporate the previous behavior of
the space allocation for chunked datasets.
The default settings for the different types of dataset storage are now
as follows:
Contiguous - Late
Chunked - Incremental
Compact - Early
This checkin also incorporates a change to the behavior of external data
storage in two ways - fill-values are _never_ written to external storage
(under the assumption that writing fill-values is triggered by allocating
space in an HDF5 file, and since space is not allocated in the file, the
fill-values should not be written) and external data files are now created
if they don't exist when data is written to them. The fill-value will
probably need to be revisited at some time in the future, this just seemed
like the safer course currently.
I think I cleaned up some compiler errors also, before getting bogged down
in the fixes for the space allocation and fill-values.
Platforms tested:
FreeBSD 4.6 (sleipnir) w/serial & parallel. Will be testing on IRIX64
6.5 (modi4) in serial & parallel shortly.
Code cleanup
Description:
Combined H5P_isa_class and H5I_object functionality into a new internal
H5P API function: H5P_object_verify, which checks that a property list is
the appropriate class and then returns the property list object associated
with the property list ID.
This reduces the source code by about 200 LOC and trims the library binary
some more.
Platforms tested:
FreeBSD 4.6 (sleipnir)
Code cleanup
Description:
Changed the last HRETURN* statements in the FUNC_ENTER macros into HGOTO*
macros, which reduces the size of the library binary in certain
configurations by another 10%
Platforms tested:
FreeBSD 4.6 (sleipnir) serial & parallel, IRIX64 6.5 (modi4) serial &
parallel
Code cleanup
Description:
Clean up a few warnings which were showing up with --enable-production
turned on.
Platforms tested:
FreeBSD 4.6 (sleipnir) serial & parallel
Code cleanup
Description:
Change most (all?) HRETURN_ERROR macros to HGOTO_ERROR macros, along with
HRETURN macros to HGOTO_DONE macros. This unifies the error return path
from functions and reduces the size of the library by up to 10% on some
platforms.
Additionally, I improved a lot of the error cleanup code in many routines.
Platforms tested:
FreeBSD 4.6 (sleipnir) serial & parallel and IRIX64 6.5 (modi4) serial &
parallel.
Code cleanup
Description:
Created a new H5I function which combined the some of the functionality of
H5I_get_type and H5I_object: H5I_object_verify.
Using this new function in the library trims another ~200 lines of code off
the library and makes the resulting binaries smaller and faster also.
Platforms tested:
FreeBSD 4.6 (sleipnir)
Code cleanup
Description:
Move operations on each type of selection into the source code file for
each selection type (all->H5Sall.c, hyperslab->H5Shyper.c, etc.)
Remove central H5S_select_<foo> operations, instead calling the operations
through function pointers in each selection (a much more object-oriented
approach).
Platforms tested:
FreeBSD 4.6 (sleipnir)
Large code cleanup/re-write
Description:
This is phase 1 of the data I/O re-architecture, with the following changes:
- Changed the selection drivers to not actually do any I/O, they
only generate the sequences of offset/length pairs needed for
the I/O (or memory access, in the case of iterating or filling
a selection in a memory buffer)
- Wrote more abstract I/O routines which get the sequence of offset/
length pairs for each selection and access perform the I/O or
memory access.
Benefits of this change include:
- Removed ~3400 lines of quite redundant code, with corresponding
reduction in the size of library binary.
- Any selection can now directly access memory when performing I/O,
if no type conversions are required, instead of just "regular"
hyperslab and 'all' selections, which speeds up I/O.
- Sped up I/O for hyperslab selections which have contiguous lower
dimensions by "flattening" them out into lesser dimensional objects
for the I/O.
No file format or API changes were necessary for this change.
The next phase will be to create a "selection driver" for each type of
selection, allowing each type of selection to directly call certain
methods that only apply to that type of selection, instead of passing
through dozens of functions which have switch statements to call the
appropriate method for each selection type. This will also reduce
the amount of code in the library and speed things up a bit more.
Phase 3 will involve generating an MPI datatype for all types of selections,
instead of only "regular" hyperslab and 'all' selections. This will
allow collective parallel I/O for all I/O operations which don't
require type conversions. It will also open up the door for allowing
collective I/O on datasets which require type conversion.
Phase 4 will involve changing the access pattern to deal with chunked
datasets in a more optimal way (in serial).
Phase 5 will deal with accessing chunked datasets more optimally for
collective parallel I/O operations.
Platforms tested:
FreeBSD 4.6 (sleipnir) w/ parallel & C++ and IRIX64 6.5 (modi4) w/parallel
Code cleanup
Description:
Broke the FUNC_ENTER macro into several macros, with more specialized
uses (which followup mail will describe). This was designed to move
most/all of the checks which could be done at compile time to that point,
instead of needlessly performing them (over & over :-) at run-time.
This reduces the library's size (and thus staticly linked binaries) and
has a minor speedup effect also.
Platforms tested:
IRIX64 6.5 (modi4) with parallel & FORTRAN enabled, and additional testing
on FreeBSD and Solaris immediately after the checkin.
Code cleanup.
Description:
Took Robb's recent ideas for improving the FUNC_ENTER/FUNC_LEAVE macros
equivalents in the SAF library and adapted them to our library. I added
an additional macro which is equivalent to FUNC_ENTER:
FUNC_ENTER_NOINIT - Has the API tracing code, etc. from FUNC_ENTER but
none of the library or interface initialization code. This is to
be used _only_ for static functions and those which explicitly
cannot have the library or interface initialization code enabled
(like the API termination routines, etc.).
This allowed many more of the functions in the library [but not all yet :-(]
to be wrapped with FUNC_ENTER[_NOINIT]/FUNC_LEAVE pairs.
It also reduced the size of the library and executables (by cutting out a
bunch of code which was never executed), I'll e-mail the exact results when
I've finished editing it.
Platforms tested:
IRIX64 6.5 (modi4)
Code cleanup
Description:
Previously, the I/O pipeline (pline), external file list (efl) and fill-
value (fill) structs were passed down the raw data function call chain,
even into and/or through functions which didn't use them. Since all three
of these pieces of information are available from the dataset creation
property list, just pass the dataset creation property list down the
function call chain and query for the information needed in a particular
function.
Platforms tested:
FreeBSD 4.5 (sleipnir)
Code tweak.
Description:
Allow the 'fill' parameter of H5S_select_fill to be NULL and allocate a
temporary buffer for it, if so.
Platforms tested:
FreeBSD 4.5 (sleipnir)
Purpose:
New feature
Description:
Fill-value's behaviors for contiguous dataset have been redefined.
Basicly, dataset won't allocate space until it's necessary. Full details
are available at http://hdf.ncsa.uiuc.edu/RFC/Fill_Value, at this moment.
Platforms tested:
Linux 2.2.
New Feature
Description:
Added new H5Dfill() routine to fill the elements in a selection for a
memory buffer with a fill value. This is a user API wrapper around some
internal routines which were needed for the fill-value modifications
from Raymond as well as Pedro's code for reducing the size of a chunked
dataset.
Platforms tested:
FreeBSD 4.5 (sleipnir) [and IRIX64 6.5 (modi4) in parallel, in a few
minutes]
Bug Fix & Code Cleanup
Description:
The MPI-IO optimized transfer routines
(H5S_mpio_spaces_read/H5S_mpio_space_write) are not being invoked in all
the cases where they could be used.
Additionally, the code for determining if an optimized transfer is wrapped
into the actual I/O transfer routine in a very confusing way.
Solution:
Re-enabled MPI-IO optimized transfer routines in all the cases where they
should work.
Extracted all the pre-conditions for optimized transfers into separate
routines from the transfer routines.
Platforms tested:
FreeBSD 4.5 (sleipnir) & IRIX64 6.5 (modi4)
Bug Fix & Feature
Description:
The selection offset was being ignored for optimized hyperslab selection
I/O operations.
Additionally, I've found that the restrictions on optimized selection
I/O operations were too strict and found a way to allow more hyperslabs
to use the optimized I/O routines.
Solution:
Incorporate the selection offset into the selection location when performing
optimized I/O operations.
Allow optimized I/O on any single hyperslab selection and also allow
hyperslab operations on chunked datasets.
Platforms tested:
FreeBSD 4.5 (sleipnir)
Bug fix & feature add
Description:
Added new API function H5Sget_select_type to determine type of selection in
a dataspace. Return values are defined by the H5S_sel_type enumerated type
in H5Spublic.h
Also, hyperslab operations involving a "all" or "none" selection are not
generating the correct resulting selections.
Solution:
Added more code to make hyperslab operations against an "all" or "none"
selection generate the correct results.
Platforms tested:
FreeBSD 4.5 (sleipnir)
Code speedups, etc.
Description:
Bring in new algorithms and data structures for dealing with hyperslabs.
This speeds up the hyperslab I/O for non-regular hyperslabs by a huge
amount.
Currently, the new API functions are ifdef'ed out, pending discussion
and consensus approval.
Platforms tested:
FreeBSD 4.4 (hawkwind)
Code cleanup (sorta)
Description:
When the first versions of the HDF5 library were designed, I remembered
vividly the difficulties of porting code from a 32-bit platform to a 16-bit
platform and asked that people use intn & uintn instead of int & unsigned
int, respectively. However, in hindsight, this was overkill and
unnecessary since we weren't going to be porting the HDF5 library to
16-bit architectures.
Currently, the extra uintn & intn typedefs are causing problems for users
who'd like to include both the HDF5 and HDF4 header files in one source
module (like Kent's h4toh5 library).
Solution:
Changed the uintn & intn's to unsigned and int's respectively.
Platforms tested:
FreeBSD 4.4 (hawkwind)
Code clean/bug fix
Description:
H5FL (free-list manager) code currently is taking an hsize_t as the size
of a memory block to allocate. On many machines, the size of an hsize_t
is greater than the size of a size_t, potentially leading to incorrect
memory allocations in rare circumstances.
Solution:
Changed hsize_t parameters and variables to size_t.
Platforms tested:
FreeBSD 4.3 (hawkwind)
