Code cleanup
Description:
Remove debugging information and match bug-fix changes to szip filter
code.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest not needed.
Description: when a datatype element size is bigger than I/O datatype
conversion buffer(1MB), library reports failure.
Solution: adjust conversion buffer to be at least equal to one element size
Platforms tested: h5committest
New feature/enhancement
Description:
Chunked datasets are handled poorly in several circumstances involving
certain selections and chunks that are too large for the chunk cache and/or
chunks with filters, causing the chunk to be read from disk multiple times.
Solution:
Rearrange raw data I/O infrastructure to handle chunked datasets in a much
more friendly way by creating a selection in memory and on disk for each chunk
in a chunked dataset and performing all of the I/O on that chunk at one time.
There are still some scalability (the current code attempts to
create a selection for all the chunks in the dataset, instead of just the
chunks that are accessed, requiring portions of the istore.c and fillval.c
tests to be commented out) and performance issues, but checking this in will
allow the changes to be tested by a much wider audience while I address the
remaining issues.
Platforms tested:
h5committested, FreeBSD 4.8 (sleipnir) serial & parallel, Linux 2.4 (eirene)
Code restructure.
Description:
Restructured tests so they shouldn't depend on shared open()'s and can be
tested with CodeWarrior
Platforms tested:
FreeBSD 4.8 (sleipnir)
Triple check not needed.
code warrior port
Description:
added a flag H5_NO_SHARED_WRITING, it is defined only for CW in the H5pubconf.h
this avoids doing some tests for CW that fail on shared writing
Solution:
Platforms tested:
windows (cw and ms)
Linux 2.4 (rockaway)
SunOS 5.7 (arabica)
IRIX 6.5 (modi4)
Misc. update:
Purpose: Windows maintenance and code cleanup
Description: There were a lot of places in the source code where varibale
was declared but never used.
Solution: Cleaned the source code to avoid warnings on Windows and Unix
platforms.
Platforms tested: arabica, modi4, Windows
Probably my environment is wrong on Linux, cannot
compile at all when fortran is enabled
If daily tests fails tomorrow, this will be my fault
but I want to go home now :-)
Misc. update:
Bug Fix
Description:
When using the STDIO driver, the dsets test would fail.
Solution:
Need to flush the data to disk before doing an HDopen call.
Platforms tested:
Modi4 (Parallel & Fortran)
Arabica (Fortran)
Verbena (Fortran & C++)
Misc. update:
Purpose: Windows port
Description: Windows cannot convert from _uint64_t to double. One of
the tests (test_set_local) used conversion to fill a data
buffer. Compilation failed on Windows.
Solution: Fixed the code not to use conversion
Platforms tested: CRAY T90IEEE, arabica, modi4 with parallel, Windows 2000
Only static tests (both debug and release) passed on Windows.
DLLs have multiple problems ;-)
Misc. update:
Code cleanup
Description:
Switched some system/library calls to use wrappers.
Platforms tested:
FreeBSD 4.8 (sleipnir) w/C++
Linux 2.4 (burrwhite) w/FORTRAN
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/parallel & FORTRAN
(h5committest not run due to my ongoing difficulties with C++ on burrwhite).
Code cleanup & new tests
Description:
Updated szip and shuffle testing to reflect changes to API functions.
Added testing for new API functions: H5Pmodify_filter, H5Pget_filter_by_id
and H5Pall_filters_avail.
Added testing for "can apply" and "set local callbacks.
Cleaned up various things, also.
Solution:
Platforms tested:
FreeBSD 4.8 (sleipnir) w/szip
Linux 2.4 (sleipnir) w/szip
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/szip, FORTRAN & parallel
Misc. update:
Code cleanup
Description:
Updated copyright.
Removed szip-specific testing in favor of standard testing, now that bugs
in the szip filter are fixed.
Updated to latest version of H5Pset_szip API
Solution:
Platforms tested:
FreeBSD 4.8 (sleipnir)
IRIX64 6.5 (modi4) w/parallel
Misc. update:
To add tests for szip compression in HDF5
Description:
Three tests were added:
1) szip filter itself
3) szip + shuffle + fletch32
2) fletch32 + shuffle + szip
Solution:
Since H5Dget_storagesize returns 0(Quincey can fix it or it may be gone),
I create another function called test_internal_filter_szip, which is the almost the same
as test_internal_filter but comments out H5Dget_storage_size function.
Platforms tested:
Since there are changes of configure.in and configure,I didn't use h5committest.
I tested with four platforms.
1) Linux 2.4 (eirene) 2) Solaris 2.7(arabic
3) windows 2000(VS 6.0) 4) SGI IRIX6.5-64(modi4)
For test 1)-3), only basic C tests were done For modi4 test, I tested 64-bit C,parallel and fortran.
All tests passed, except a warning message from szip library when checksum is used in some order, which doesn't cause any real problems.
Misc. update:
Update
Description:
Updated the Copyright statement
Platforms tested:
Linux (This change is only in the comments, so I just check that the
modules still compile)
Misc. update:
Code cleanup
Description:
Clean up miscellaneous warnings which have crept into the code.
Fix "_POSIX_C_SOURCE not defined" warning on FreeBSD.
Adjust gcc compiler flags to be more concise for production mode.
Refactor the H5O code so that there is a stronger boundary between code
in the H5O package and code in the library which just calls H5O routines.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel and gcc 2.95.4 & gcc 3.2.2
Misc. update:
Update MANIFEST if you add or remove any file.
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
Purpose:
bug fixed for windows
Description:
windows fopen functions by default opens a text mode file. It will fail for HDF5 file.
So to use HDopen HDread HDclose for test.
Solution:
Change from fopen, fread, fclose to HDopen, HDread, HDclose.
Platforms tested:
linux2.2.18smp, windows 2000
Misc. update:
Update MANIFEST if you add or remove any file.
Update release_docs/RELEASE for bug fixes, new features, etc.
Update applicable document files too.
Purpose:
Bug fix(kinda)
Description:
Windows handle "fopen, fseek, fread, fclose" C functions slightly different.
Solution:
change to HDopen, HDlseek, HDread, HDclose to cover both Unix and Windows.
Platforms tested:
arabica
Purpose:
New feature to H5Dget_offset
Description:
If user block is set, H5Dget_offset should be able to return the absolute
offset from the beginning of file.
Platforms tested:
eirene, arabica
Code Cleanup & New Tests
Description:
tmisc.c:
Switched from using H5_HAVE_COMPRESSION flag in favor of
H5_HAVE_FILTER_DEFLATE.
dsets.c:
Switched from using H5_HAVE_COMPRESSION flag in favor of
H5_HAVE_FILTER_DEFLATE.
Refactored I/O filter tests to allow new filters to be [more] easily
added.
Added tests for shuffle & deflate+shuffle I/O filters (if the filter(s)
are enabled).
Added test for creating a new dataset with a filter that is not
available.
Added test for attempting to read a dataset created with a filter that
is not available.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir)
Purpose:
Add a testing case for shuffling algorithm.
Description:
A control test:
Since shuffling 8-bit array should not change the original result; so we use a
8-bit array to test whether after shuffling the array keeps the same values.
Solution:
Platforms tested:
eirene,arabica,modi4
Misc. update:
Update MANIFEST if you add or remove any file.
Update release_docs/RELEASE for bug fixes, new features, etc.
Update applicable document files too.
Purpose:
New function.
Description:
H5Dget_offset returns the offset of a dataset's data relative to the
beginning of a file.
Platforms tested:
arabica(simple function, one test should be enough.)
Misc. update:
RELEASE.txt
Code cleanup
Description:
Clean up some compile warnings and have these tests create their data files
with the correct file drivers and directories, etc.
Platforms tested:
FreeBSD 4.6 (sleipnir) w and w/o parallel
Linux 2.2.x (eirene) w/FORTRAN & C++
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/FORTRAN & parallel
API name change
Description:
Change all "space time" references to "alloc time", including API functions
and macro definitions, etc.
Platforms tested:
FreeBSD 4.6 (sleipnir) w/C++
Solaris 2.7 (arabica) w/FORTRAN
IRIX64 6.5 (modi4) w/parallel & FORTRAN
Code cleanup/More tests
Description:
Cleaned up some compiler warnings and wrote additional tests for space
allocation and storage size routines.
Platforms tested:
FreeBSD 4.6 (sleipnir) w/serial & parallel. Will be testing on IRIX64
6.5 (modi4) in serial & parallel shortly.
Purpose:
Design for compact dataset
Description:
Compact dataset is stored in the header message for dataset layout.
Platforms tested:
arabica, eirene.
Update, Bug Fix, and Feature Add
Description:
- Updated how AC_{ENABLE,WITH} help messages were being generated.
Autoconf now gives you an AC_HELP_STRING macro to use to create
them.
- Fixed the problem with Linux LFS on RedHat 7.3 machines. It wasn't
finding getdents64(), so we can't rely on that being present
anymore...
- Added GPFS detection and setting.
- Updated how compression is specified. It's no longer necessary to
test for HAVE_ZLIB_H, HAVE_LIBZ, and HAVE_COMPRESS2. The one macro
{H5_}HAVE_COMPRESSION takes care of all of these.
Solution:
Changed the check for Linux LFS from looking for getdents() to
looking at the version number of the kernel (using the uname -r
command). You can still override with the --enable-linux-lfs switch
if you really believe that your <2.4 kernel has LFS support.
Platforms tested:
Linux (2.2 and 2.4)
Purpose:
Modify H5Fclose behavior
Description:
The HDF5 actual file close behaves in several ways in terms of if there
are still objects(dataset, group, datatype) opened in file.
Solution:
Added a new file access property, file close degree. It has four values,
H5F_CLOSE_DEFAULT
H5F_CLOSE_WEAK
H5F_CLOSE_SEMI
H5F_CLOSE_STRONG
The way a file is closed is decided by these values.
Platforms tested:
IRIX64 6.5, SunOS 5.6, FreeBSD 4.4
Code cleanup
Description:
Windows is generating hundreds of warnings from some of the practices in
the library. Mostly, they are because size_t is 32-bit and hsize_t is
64-bit on Windows and we were carelessly casting the larger values down to
the smaller ones without checking for overflow.
Also, some other small code cleanups,etc.
Solution:
Re-worked some algorithms to eliminate the casts and also added more
overflow checking for assignments and function parameters which needed
casts.
Kent did most of the work, I just went over his changes and fit them into
the the library code a bit better.
Platforms tested:
FreeBSD 4.4 (hawkwind)
Kludge
Description:
Since we're only about halfway through converting the internal use of
property lists from the "old way" to the generic property lists, we turned
off snapshots to avoid exposing lots of API changes to users, until the
APIs settled down.
Getting the snapshots rolling again seems to have become a priority, so
some changes are going to have to be made now that were going to be
postponed until we were completely finished with the conversion. This
requires that the old API functions be able to deal with both the old
and new property lists smoothly.
Solution:
Kludge together the property list code so that they can transparently handle
dealing with both the old and new property lists
Platforms tested:
FreeBSD 4.4 (hawkwind)
Code cleanup for better compatibility with C++ compilers
Description:
C++ compilers are choking on our C code, for various reasons:
we used our UNUSED macro incorrectly when referring to pointer types
we used various C++ keywords as variables, etc.
we incremented enum's with the ++ operator.
Solution:
Changed variables, etc.to avoid C++ keywords (new, class, typename, typeid,
template)
Fixed usage of UNUSED macro from this:
char UNUSED *c
to this:
char * UNUSED c
Switched the enums from x++ to x=x+1
Platforms tested:
FreeBSD 4.4 (hawkwind)
New Features!
Description:
Start migrating the internal use of property lists in the library from the
older implementation to the new generic property lists.
Currently, only the dataset transfer property lists are migrated to the
new architecture, all the rest of the property list types are still using
the older architecture.
Also, the backward compatibility features are not implemented yet, so
applications which use dataset transfer properties may need to make the
following changes:
H5Pcreate(H5P_DATASET_XFER) -> H5Pcreate_list(H5P_DATASET_XFER_NEW)
and
H5Pclose(<a dataset transfer property list>) -> H5Pclose_list(id)
This still may have some bugs in it, especially with Fortran, but I should
be wrapping up those later today.
Platforms tested:
FreeBSD 4.4 (hawkwind)
Bug Fix, Code Cleanup, Code Optimization, etc.
Description:
Fold in the hyperslab speedups, clean up compile warnings and change a
few things from using 'unsigned' or 'hsize_t' to use 'size_t' instead.
Platforms tested:
FreeBSD 4.3 (hawkwind), Solaris 2.7 (arabica), Irix64 6.5 (modi4)
Update
Description:
Changed includes of the form:
#include <hdf5_file.h>
to
#include "hdf5_file.h"
so that gcc can pick them up easier without including the system
header files since we don't care about them.
Platforms tested:
Linux
Code tweak
Description:
Large data arrays were being created on the stack.
Solution:
Hoisted the data variables out of the function(s) and make into static
variables.
Platforms tested:
FreeBSD 4.3 (hawkwind)
Clean up warnings
Description:
The "FAILED" macro is defined by Windows and is causing warnings and
potential errors when compiled on that platform.
Solution:
Change our macro from FAILED to H5_FAILED.
Platforms tested:
FreeBSD 4.2 (hawkwind)
Bug fix
Description:
Datasets were allowed to be created with chunks larger than the maximum
dimension for each dimension.
Solution:
Wrote test to verify new error checking is working correctly.
Platforms tested:
FreeBSD 4.2 (hawkwind)
Code cleanup.
Description:
Fixed _lots_ (I mean _tons_) of warnings spit out by the gcc with the
extra warnings. Including a few show-stoppers for compression on IRIX
machines.
Solution:
Changed lots of variables' types to more sensible and consistent types,
more range-checking, more variable typecasts, etc.
Platforms tested:
FreeBSD 4.2 (hawkwind), IRIX64-64 (modi4)
Bug fix.
Description:
Properly detect whether the library has compression support and skip
tests (with messages) if there is no compression.
Platforms tested:
FreeBSD 4.2 (hawkwind)
Backward compatibility code
Description:
Add in code to allow the library to emulate the v1.2 API and behavior.
Platforms tested:
FreeBSD 4.2 (hawkwind)
The "FILENAME" declared extern in h5test.h is not always used.
It was used in h5_cleanup to remove temporary files created
during tests. Not all tests codes have used this routine.
Indeed, quite a few of test programs do "#define FILENAME ".
Also, h5_cleanup needs to work in tandem with h5_fixname.
h5_fixname accepts an explicite base_name argument instead
of using the global variable FILENAME. That is cleaner.
Solution:
Added char *base_name[] as a new argument to h5_cleanup, in
the same style as h5_fixname. Removed "extern char *FILENAME..."
from use. Also, undo some unnecessary declaration of "char *FILENAME"
from some tests which don't use it at all (yet).
Platforms tested:
modi4-64(irix64), arabica(solari2.7), eirene(linux)
(arabica could not launch tests automatically. I had to hack
in LD_LIBRARY_PATH to make them run.)
