Code cleanup & reorganization
Description:
Move further in the testing framework cleanup, eliminating all the
global variables (moving them into testframe.c as static variables) from the
testing framework code and moving it into the libh5test.a.
Platforms tested:
FreeBSD 4.9 (sleipnir) w & w/o thread-safety, c++ & parallel
h5committested
Code cleanup
Description:
Refactor library testing framework (used for the testhdf5 & ttsafe tests)
to remove almost all of the duplicated code, moving the common code into a
new 'testframe.c' source file.
Platforms tested:
FreeBSD 4.9 (sleipnir) w & w/o thread-safety
h5committest
Fixed a dumb mistake.
Description:
Other tests barked at the changes.
Solution:
Move the declaration of Index and Test[] into h5test.c.
Platforms tested:
Tested in eirene.
Misc. update:
Code reorg.
Description:
Move the InitTest() from individual tests (testhdf5 and ttsafe) to
libh5test (h5test.c) so that it can be shared among all tests.
Platforms tested:
Only tested in Eirene via serial with thread-safe enabled.
No other platforms test since this is pretty trivial.
bug fix, update documentation
Description:
version 8 of Code Warrior has a bug on the open file function
that causes one of our tests to fail
documented this in the install instructions
removed a CW specific header file include , that was left from version 6
from the file testhdf5.c
Platforms tested:
Code Warrior
linux
Misc. update:
Update
Description:
Updated the Copyright statement
Platforms tested:
Linux (This change is only in the comments, so I just check that the
modules still compile)
Misc. update:
New internal feature
Description:
Add internal API for building and working with heaps (H5HP). This will be
used for the LRU algorithm in the new metadata cache code.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir)
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
Bug Fix
Description:
When file space was returned to the file space free-list for reuse,
occasionally raw data allocations which used space from the free-list
would overlap with the metadata accumulator and get over-written with
the cached information in the accumulator, corrupting the data.
Solution:
Check if the space about to be recycled on the free-list is going to be
used for raw data and also overlaps with the metadata accumulator cache,
avoiding using space that fits those criteria.
This fixes bug #701
Platforms tested:
FreeBSD 4.5 (sleipnir)
New feature.
Description:
Added a test to verify the correctness of information provided by
configure in H5config.h. Some information, such as SIZEOF some
types can be hardcoded by config/<machine>. This test verified
the information is indeed correct.
Currenly, only size of C language types are verified.
Platforms tested:
Eirene, regular, arabica.
Code cleanup (sorta)
Description:
When the first versions of the HDF5 library were designed, I remembered
vividly the difficulties of porting code from a 32-bit platform to a 16-bit
platform and asked that people use intn & uintn instead of int & unsigned
int, respectively. However, in hindsight, this was overkill and
unnecessary since we weren't going to be porting the HDF5 library to
16-bit architectures.
Currently, the extra uintn & intn typedefs are causing problems for users
who'd like to include both the HDF5 and HDF4 header files in one source
module (like Kent's h4toh5 library).
Solution:
Changed the uintn & intn's to unsigned and int's respectively.
Platforms tested:
FreeBSD 4.4 (hawkwind)
Update
Description:
Changed includes of the form:
#include <hdf5_file.h>
to
#include "hdf5_file.h"
so that gcc can pick them up easier without including the system
header files since we don't care about them.
Platforms tested:
Linux
Bug fix (sort of)
Description:
The RCSID string in H5public.h was causing the C++ code problem as it
was included multiple times and C++ did not like multiple definitions
of the same static variable.
Solution:
Since we don't really make use of the RCSID strings as we have not
installed it in all source files, we decided to remove it.
Platforms tested:
eirene (linux), modi4 (IRIX64-64) both serial and parallel modes.
New Feature
Description:
Added array datatype tests to the regression tests. These datatype
combinations are tested currently:
1-D array of atomic datatypes
3-D array of atomic datatypes
array of array of atomic datatypes
array of compound of atomic datatypes
array of compound of array datatypes
array of VL of atomic datatypes
array of VL of array datatypes
Also added a test to verify that the older style compound datatype with
array fields works correctly.
Platforms tested:
FreeBSD 4.1.1 (hawkwind)
Bug fix
Description:
"Time" datatypes (H5T_UNIX_D*) were not being stored and retrieved in
the datatype object header message correctly.
Solution:
Store endian-ness and precision in the datatype object header message and
added test to continue to track them working correctly.
This fixes bug #512.
Platforms tested:
FreeBSD 4.1.1 (hawkwind)
The "FILENAME" declared extern in h5test.h is not always used.
It was used in h5_cleanup to remove temporary files created
during tests. Not all tests codes have used this routine.
Indeed, quite a few of test programs do "#define FILENAME ".
Also, h5_cleanup needs to work in tandem with h5_fixname.
h5_fixname accepts an explicite base_name argument instead
of using the global variable FILENAME. That is cleaner.
Solution:
Added char *base_name[] as a new argument to h5_cleanup, in
the same style as h5_fixname. Removed "extern char *FILENAME..."
from use. Also, undo some unnecessary declaration of "char *FILENAME"
from some tests which don't use it at all (yet).
Platforms tested:
modi4-64(irix64), arabica(solari2.7), eirene(linux)
(arabica could not launch tests automatically. I had to hack
in LD_LIBRARY_PATH to make them run.)
----------------------
./src/H5T.c
Fixed a typo in the registration of the `unsigned char' to
`unsigned long long' type conversion that caused it to not be
registered, falling back to software whenever that conversion
path was taken.
./MANIFEST
./test/Makefile.in
./test/testhdf5.c
./test/testhdf5.h
./test/theap.c [REMOVED]
./test/lheap.c [NEW]
./test/tohdr.c [REMOVED]
./test/ohdr.c [NEW]
./test/tstab.c [REMOVED]
./test/stab.c [NEW]
Removed the `t' from the front of these names and made each
test a stand-alone program following the format of most of the
other tests.
./test/big.c
Uses libh5test.a but always sets the low-level driver to 1GB
file family.
The `#if' near the top to set the data space to 8GB has been
simplified now that `long_long' is always defined and the
error message is improved when `long_long' isn't wide enough.
Cleanup code was added to the error handling.
./test/gheap.c
./test/istore.c
Uses libh5test.a. Added error cleanup code.
./test/dtypes.c
./test/h5test.c
Added 68 new tests that check hardware and software
conversions between `long long' and `unsigned long long' and
the other integer types. The tests only run on machines where
sizeof(long_long)!=sizeof(long). We test a total of 180
different integer conversions, half in hardware and half in
software.
Cut down the number of times each test is run from 5 to 1 so
it doesn't take so long. If you want to run more times
there's a constant that can be changed at the top of the file.
./test/extend.c
Removed unused variable.
./test/h5test.c
./test/h5test.h
./test/external.c
./test/fillval.c
The h5_cleanup() returns true/false so it can be used in an `if'
statement to clean up additional files.
./doc/html/Environment.html
Indented. Added HDF5_PREFIX and HDF5_DRIVER descriptions.
./src/H5P.c
Changed the trace type for the second argument from `Iu' to
`x' since it's an output parameter.
./INSTALL
Added a warning that the GNU zlib that comes with the latest
version of HDF4 is too old to use with HDF5 and must be
renamed so configure doesn't see it when `--enable-hdf4' is
used.
----------------------
./doc/html/H5.format.html
./src/H5Oattr.c
./src/H5Odtype.c
./src/H5Oefl.c
./src/H5Olayout.c
./src/H5Osdspace.c
./src/H5Oshared.c
Added version numbers to some object header messages so we can
update them easier in the future. The library currently just
gives up if the version numbers don't match, but in the future
the library could handle multiple versions of a message.
./test/testhdf5.c
Removed an argument from the H5version() call that I missed
last time.
----------------------
./src/H5Flow.c
./src/H5Fprivate.h
./src/H5Fsplit.c
Changed the allocation size request from `size_t' to `hsize_t'
because it was overflowing for the `big' test.
./src/H5detect.c
If `long double' and `double' are the same size then we define
H5T_NATIVE_LDOUBLE to be the same as H5T_NATIVE_DOUBLE.
Similarly for `long' vs. `long long' and `unsigned long' vs.
`unsigned long long'.
./test/Makefile.in
Added `big' to the list of tests to normally run.
./test/big.c
Added a check to see if the file system supports holes and if
it doesn't then the test is skipped.
./RELEASE
Added a couple minor details details about API tracing and
symbolic links.
./src/H5public.h
Added comments about the use of hbool_t. Fixed a comment
spelling error.
./test/testhdf5.h
Changed the way the version number is printed. The old method
was `hdf5-1.2.3d' and the new method is `hdf5 version 1.2
release 3'
./tools/h5ls.c
Only prints the max dimension if it differs from the current
dimension or if verbose mode is enabled. Added switches `-?',
`-h', and `--help' to print a usage message. Added switches
`-v' and `--verbose' to generate more verbose output. Added
switches `-V' and `--version' to print the version number and
exit. The version number is printed like: This is h5ls
version 1.0 release 24'
./bin/h5vers [NEW]
This script prints, sets, and/or increments the hdf5 version
number. It can be run from the top directory or any of the
child directories like src, tools, test, etc. Some examples:
$ h5vers # Display current version
1.0.24
$ h5vers -v
version 1.0 release 24 # Display current version.
$ h5vers -s 5.2.8 # Set version and display
5.2.8
$ h5vers -s 2.1
2.1.0
$ h5vers -s hdf5-1.0.24a.tar.bz2
1.0.24
$ h5vers -s 'version 2.0 release 8'
2.0.8
$ h5vers -s 'junk 22 junk 33 more junk 66 and 99 junk'
33.66.99
$ h5vers -i major # Increment from 1.0.24
2.0.0
$ h5vers -i minor # Increment from 1.0.24
1.1.0
$ h5vers -i release # Increment from 1.0.24
1.0.25
$ h5vers ~/hdf5/src/H5public.h # Use an alternate file
1.0.24
./bin/checkapi [NEW]
Run from the src directory with arguments H5[A-Z]*.c and it
will print the locations of each place where an API function
was called from within the library. Use it as the compile or
grep command under Emacs and you can C-x ` through the list.
./bin/debug-ohdr [NEW]
Keeps track of H5O_open() and H5O_close() debugging messages
and lists the file addresses of the object headers that are
opened but never closed. You must enable the `o' debugging at
configuration time and pipe stderr into this script.
./bin/errors
Added a note to indicate that this script no longer works
because of changes in the HRETURN_ERROR() and HGOTO_ERROR()
macros.
./bin/iostats [NEW]
Watches output from the Linux strace program and accumulates
statistics about low-level access to an hdf5 file. The output
is a list of 2d data points which can be plotted by gnuplot to
show file seeking behavior.
./MANIFEST
Added new files.
to cleanup temporary files used by that particular module.
Added a cleanup() routine to testhdf5 that calls all the other
cleanup_xxx() routines to do cleanup.
Cleanup action can be disabled either by -c (no cleanup) option
or HDF5_NOCLEANUP environment variable.
----------------------
./html/Dataspaces.html
./html/Errors.html
./html/Files.html
./html/H5.api.html
./html/review1.html
./src/H5private.h
./src/H5public.h
./test/dsets.c
./test/dtypes.c
Removed all the types like `int32' and `intn' into private
headers since they violate the naming scheme and pollute
application name space. Besides, our test files only use them
in a handful of places and it's probably useless to export
them to the app. The app is always written in terms of
standard numeric types or its own numeric types and probably
never in terms of HDF5 numeric types. If it were, then the
user would have to copy from their type to hdf5 type for
almost every hdf5 API function call! Same goes for return
values. I also removed SUCCEED/FAIL from the API since apps
should be checking against zero anyway.
if (FAIL==(space=H5Screate_simple(...))) /*wrong*/
if ((space=H5Fcreate_simple(...)<0)) /*right*/
./src/H5.c
Changed arguments of H5version() from `uintn' to `unsigned'.
./src/H5Tpublic.h
./src/H5T.c
Changed return type of H5Tget_nmembers() from `intn' to `int'
./src/H5A.c
./src/H5Aprivate.h
./src/H5Apublic.h
Changed `H5Asearch_func_t' to `H5A_search_func_t' and moved
its definition from the public to the private header file.
./html/H5.format.html
Documented changes made to the external file list (H5O_EFL)
message.
./src/H5D.c
./src/H5Dprivate.h
./src/H5E.c
./src/H5Epublic.h
./src/H5O.c
./src/H5Oefl.c
./src/H5Oprivate.h
./src/H5P.c
./src/H5Ppublic.h
Added partial support for external raw data files. HDF5 can
now describe external raw data files by listing the file
names, offsets, and size for a dataset. However, we will
restrict a dataset to be stored "contiguously" when the
external file list is viewed as a single address space. The
current implementation is unable to read/write to external
files--that will come later this week as will documentation.
For now, take a look at ./test/external.c, particularly the
calls to H5Pset_external().
./test/Makefile.in
./test/external.c [NEW]
./MANIFEST
Added tests for external storage. Note: the read test is
supposed to fail at this point since reading external datasets
is not implemented yet. There is no write test.
./src/H5S.c
./src/H5Sprivate.h
./src/H5Ssimp.c
Added H5S_get_npoints_max() to return the maximum possible
number of data points in a data space.
Added an extra argument to H5S_get_dims() which returns the
maximum dims.
./src/H5F.c
./src/H5Fprivate.h
./src/H5Fpublic.h
./src/H5M.c [DEPRICATED]
./src/H5Mpublic.h [DEPRICATED]
Changed `template' to `property list' in lots of places.
./src/H5Osdspace.c
Removed an extra `\n' from a print statement.
./src/H5S_public.h
Changed H5S_UNLIMITED to the maximum size_t value.
./test/extend.c
"Extendable" is spelled "extendible".
./src/H5Farray.c
./src/H5V.c
./src/H5Vprivate.h
./test/hyperslab.c
Strides are now type ssize_t instead of int. These have
nothing to do with the sample granularity arguments for
hyperslabs, which are also called "strides" in the code.
./test/tstab.c
Changed assumptions about default address and length sizes.
----------------------
./html/Errors.html [NEW]
./html/H5.user.html
./MANIFEST
Documents the new error handling interface and gives examples.
./src/H5.c
./src/H5private.h
./src/H5Apublic.h
./src/H5E.c
./src/H5Eprivate.h
./src/H5Epublic.h
Rewrote error handling. Got rid of `push' overloading and
added a few API functions. The error stack is statically
allocated and not entered into H5A, simplifying error handling
within the error handler. Rudimentary support for threads.
Changed the names of some errors.
./src/H5G.c
./src/H5Gnode.c
./src/H5H.c
./src/H5O.c
./src/H5T.c
Changed H5ECLEAR to H5E_clear().
./src/Makefile.in
Alphabetized source list.
./test/dsets.c
Turned off error reporting around functions that are expected
to fail. Error messages are sent to stdout.
./test/testhdf5.c
./test/testhdf5.h
Turned off automatic error reporting since this file mostly
calls internal functions and does its own error reporting.
while he is visiting LLNL. I changed the default creation template
offset and length to 4. Will fix the problem later.
Changes since 19980205
----------------------
./src/H5H.c
./src/H5Hprivate.h
./src/H5O.c
./src/H5Ocont.c
./src/H5Odtype.c
./src/H5Oefl.c
./src/H5Olayout.c
./src/H5Oname.c
./src/H5Onull.c
./src/H5Oprivate.h
./src/H5Odspace.c
./src/H5Ostab.c
./src/debug.c
./html/H5.format.html
Added an extra 4-byte field after the heap magic number for
alignment on the DEC alpha. Changed object header message
alignment to 8-bytes.
./src/H5F.c
./src/H5Farray.c
./src/H5Ffamily.c
./src/H5Fistore.c
./src/H5Flow.c
./src/H5Fprivate.h
./src/H5Fsec2.c
./src/H5Fstdio.c
./src/H5Gnode.c
./src/H5O.c
./src/H5Odtype.c
./src/H5P.c
./src/H5Pprivate.h
./src/H5T.c
./src/H5Tconv.c
./src/H5Tpkg.h
./src/H5Tpublic.h
./src/H5V.c
./src/H5detect.c
./test/cmpd_dset.c
./test/dsets.c
./test/dtypes.c
./test/extend.c
./test/hyperslab.c
./test/istore.c
./test/th5p.c
./test/theap.c
Fixed a few irix64 warnings regarding size_t vs. int,
variables set but not used, printf formats
./config/irix64
Added `-woff 1196' to get rid of errors about __vfork() being
implicitly defined in a system header file.
./src/H5B.c
Fixed a stack alignment problem.
----------------------
./src/H5private.h
Changed HDF5_MAJOR_VERSION to 5
./src/H5Aprivate.h
./src/H5Apublic.h
Added group atoms.
./src/H5D.c
./src/H5P.c
./test/istore.c
./test/tohdr.c
./test/tstab.c
Updated for symbol table interface changes.
./src/H5F.c
./src/H5Fprivate.h
Simpler handling of special case files: empty files and files
with a single object.
./src/H5G.c
./src/H5Gent.c
./src/H5Gnode.c
./src/H5Gpkg.h
./src/H5Gprivate.h
./src/H5Gpublic.h
./src/H5Gshad.c (DELETED)
./src/H5Gstab.c
./src/Makefile.in
Removed shadows, simplifying code. Symbol table entries are
allowed to cache only constant meta data. Fixed naming.
./src/H5O.c
./src/H5Oprivate.h
Access to object headers is always done through a symbol table
entry instead of a file address. Added stubs for opening and
closing object headers to be used when deletion is implemented.
Added new tests.
./test/dsets.c NEW
Tests datasets.
./test/dspace.c NEW
Tests data spaces.
./test/dtypes.c NEW
Tests data types.
./test/istore.c
./test/tfile.c
./test/th5p.c
./test/theap.c
./test/tohdr.c
Changed to work with new function names.
./test/testhdf5.c
Removed the data type and dataset tests since they're now in
their own files. They had to be rewritten because of extensive
changes to those interfaces.
