Code checkpoint
Description:
More progress on fractal heap, can now re-open an existing heap, although
the free space algorithm still needs work.
Also, use the new "pinned entry" metadata cache code.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (heping)
Solaris 9 (shanti)
Linux 2.4/64 (mir)
Code cleanup
Description:
Big clean up on high-level library code, including:
- Removing include of HDF5 private header file(!)
- Reduce number of compiler warnings
- Untangle header files, etc.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/C++
Linux 2.4/64 (mir) w/C++ & FORTRAN
Add pinned entry capability to cache.
Description:
For frequently accessed cache entries, the protect/unprotect overhead
is sometimes a bottleneck.
Solution:
Allow entries to be pinned in the cache. Pinned entries can't be
evicted, but can be flushed or modified.
Platforms tested:
h5committested -- minus one small typo in test/cache.c whose fix was
tested on copper and heping only.
Misc. update:
new feature
Description:
added support for the printout of dataset region references differences
added a new test for this
merged the h5diff generator of test files into a single file
Solution:
Platforms tested:
linux 32, 64
solaris
Misc. update:
Several bug fixes
Description:
Added config file for Cray X1 (previous file was misnamed)
Simplified some code in hyperslab.c test that seemed to have been
confusing compiler on Cray X1.
Fixed typo in config/commence.am
Cleaned up hl/src/Makefile.am
Solution:
All four fixes should be straightforward. The failure on Cray was
very difficult to debug, but involved arithmetic errors.
This change seems to fix it.
Platforms tested:
heping, copper, sol, some Cray X1 (more testing when system comes back up)
Bug fix.
Description:
./hl/src/H5LTparse.h was marked as DO_NOT_DISTRIBUTE by mistake.
It should not be installed but must be present for building.
Solution:
Fixed.
Platforms tested:
Should work.
Description: Here is what I did:
modified MANIFEST according Quincey's instructions to fix
the problem exposed by bin/release (compilation fails
because some header files were not included in the distribution)
fixed small typo and modified include_HEADERS line in
src/Makefile.am to address the problem exposed by bin/release
ran bin/h5vers to change version number to 1.8.0-alpha0
ran bin/release to release tar ball to the ftp server
Solution:
Platforms tested: mir MANY,MANY,MANY times
Misc. update:
Code checkpoint
Description:
Add in more fractal heap support for odd allocation patterns (entries that
are too large for various block sizes, etc.)
Broke out fractal heap code into more source code modules.
Refactored fractal heap test to make it more understandable and
maintainable for the long term.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (heping)
Linux 2.4/64 (mir)
Solaris 2.9 (shanti)
new feature
Description:
1) separated the HL library into "public" and "private" header files, with the same caracteristics as the basic library
2) added the public headers to hdf5.h (with a conditional include macro, defined in configure.in)
3) added the path to HL in all Makefile.am 's , because of the inclusion in hdf5.h
Solution:
Platforms tested:
linux 32, 64
AIX
solaris
with fortran and c++
(one packet table example fails)
Misc. update:
bug fixes
Description:
h5dump/h5ls were not displaying long doubles correctly
Solution:
1) the print datatype functions were incorrectly testing for the valid return value from H5Tequal,
(TRUE), causing the display of an incorrect name of a dataype in error cases from H5Tequal
2) h5tools_print_str did not have a case for native long double
3) added a file generator for a long double dataset
4) added one script test for the long double data (commented , some sytems don't have a native long double match, and the output differs)
5) added a vms file and h5dump script test
Platforms tested:
linux 32, 64
solaris
AIX
Misc. update:
Description: cross_read.c is a new program to test reading data created
on an OpenVMS, a little-endian, and a big-endian machine. The three
data files are vms_data.h5, be_data.h5, and le_data.h5. To generate
these data files, run gen_cross.c on these machines and change the file
names.
Platforms tested: fuss and h5committest
Misc. update: MANIFEST
bug fix
Description:
percent relative error was done using integer arythmetic; use floating point instead
added the case for unsigned long long integer to float conversion
Solution:
Platforms tested:
linux (32,64)
AIX
solaris
Misc. update:
bug fix
Description:
1) added a more explainative usage message
2) the percent relative error for the integer type (division) was being done using integer arythmetic; use floating point arythmetic instead
3) added a new test for integer percent
Solution:
Platforms tested:
linux (32,64)
AIX
solaris
Misc. update:
bug fix, new features
Description:
when comparing links , the output for the number of differences found was not being done
Solution:
print it
add 3 more tests that test the output of differences for 1) groups 2) datatypes 3) links
Platforms tested:
linux
Misc. update:
Code checkpoint
Description:
The fractal heap code is reasonably stable and is passing all its
current tests, so checkpoint everything with CVS.
Also, add "statistics" module for v2 B-trees code, which is only a
stub right now.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Mac OSX (amazon)
Linux 2.4 (chicago)
Description: I ran bin/chkmanifest BEFORE adding files to CVS. Therefore
script ignored all new files and reported success.
Solution: Added missing files and told myself to remember this next time.
Platforms tested: heping with bin/chkmanifest
Misc. update:
New feature & code cleanup
Description:
Update fractal heap to be able to insert objects into a direct block
hanging off the header.
Extract "octal dump" code into separate routine so that both the local
heaps & fractal heaps can use it.
Other code cleanups & support to get this far.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
Moved gif2h5 tool to hl directory
Description:
Added a tools directory under the hl directory and moved the gif2h5 tool
to that directory.
Solution:
The gif2h5 tool was originally built in the tools directory, but this
introduced dependency issues that required special checks in the
Makefiles.am and required the top-level build order to be changed
because it depended on the HL library.
For simplicity in the Makefiles now and in the future, the gif2h5 tool
was moved to be underneath the hl library.
Platforms tested:
mir, copper, modi4, shanti
Added HL examples
Description:
Examples existed for the HL lite, image, and table interfaces, but
were never included in the distro... until now!
Solution:
Added source files to CVS, updated Makefile and MANIFEST.
Platforms tested:
mir, heping, shanti
Incrementtal checkin
Description:
Revise & update v2 B-tree code to separate the internal package-specific
routines from the library-callable "private" routines.
Similar updates for the fractal heap code.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Mac OSX (amazon)
New feature
Description:
Check in initial "fractal heap" code, for supporting the group redesign.
Also, remove some remnants of the segmented heap/block tracker/B+tree code
which I didn't find during the last pass.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Mac OSX (amazon)
Linux 2.4
Maintenance
Description:
Remove half-built (half-baked? :-) implementation of B+tree, block tracker
and segmented heaps. (In preparation for starting the new heap coding)
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (heping)
1) Add parallel test for metadata cache
2) Split serial test for metadata cache into two parts
3) Fix bug in which cache was flushed needlessly when the
cache wasn't full.
4) Performance improvements
Description:
See above.
Solution:
See above.
Platforms tested:
h5committest
Misc. update:
Code cleanup
Description:
Check in some of the code cleanups from working on the external link
support. (This doesn't include any of the external link features)
Platforms tested:
FreeBSD 4.11 (sleipnir)
Mac OSX.4 (amazon)
Linux 2.4
Description: Renamed lex.yy.c to H5LTanalyze.c, y.tab.c to H5LTparse.c, and
y.tab.h to H5LTparse.h in hl/src. Changed definition of opaque type to ommit the tag.
Changed the test for opaque type in hl/test, too.
Platforms tested: h5committest, fuss, and sleipnir.
Misc. update: MANIFEST
Bug fix
Description:
Fogot to commit file config/examples.am in previous checkin.
Solution:
The examples.am file contains common Makefile rules for the example
directories.
Added high-level example directories
Description:
Refactored common code out of examples Makefiles.am, added high-level
example directories, added packet table examples.
Solution:
Examples now draw from a common config/examples.am file, which
contains rules for installing, uninstalling, and cleaning examples.
High-level example directories are mostly empty, except for the
C and C++ packet table tests.
Platforms tested:
mir, sleipnir, copper, shanti
New feature
Description:
Add in a combination of Peter's & my code to support copying
variable-length data from one file to another, although currently only
supported with contiguous data storage.
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committest
New feature
Description:
Check in baseline for compact group revisions, which radically revises the
source code for managing groups and object headers.
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
This initiates the "unstable" phase of the 1.7.x branch, leading up
to the 1.8.0 release. Please test this code, but do _NOT_ keep files created
with it - the format will change again before the release and you will not
be able to read your old files!!!
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!! WARNING!!!!
Solution:
There's too many changes to really describe them all, but some of them
include:
- Stop abusing the H5G_entry_t structure and split it into two separate
structures for non-symbol table node use within the library: H5O_loc_t
for object locations in a file and H5G_name_t to store the path to
an opened object. H5G_entry_t is now only used for storing symbol
table entries on disk.
- Retire H5G_namei() in favor of a more general mechanism for traversing
group paths and issuing callbacks on objects located. This gets us out
of the business of hacking H5G_namei() for new features, generally.
- Revised H5O* routines to take a H5O_loc_t instead of H5G_entry_t
- Lots more...
Platforms tested:
h5committested and maybe another dozen configurations.... :-)
New feature
Description:
Add in baseline "object copy" code from Peter [in the form of a new API
routine: H5Gcopy()]. There's still some work to do (like handling variable-
length datatypes and possibly support for references) and it hasn't been tested
on mounted files yet, but the core functionality is there and working
correctly.
I've also got a set of patches to update the 1.6 branch with tweaks to
keep the branches mostly in sync, but Elena will kill me if I import them
before the 1.6.5 release is out... :-)
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committested
Bug fix.
Description:
The previous way of detecting the MPICH version fails if the suspected
$CC does not give the expected MPICH version information. Though no
consequence, it generates error messages that alarm users.
Also, mpich2 gives the version information as if it were mpich v1.0.2
which was incorrectly included as an earlier version of the origianl
mpich.
Solution:
Implemented a more robust code to detect the version and narrow the
recognition down to v1.2.0-v1.2.5. Also, the code were duplicated in
two files (linux-gnulibc1 and ia64-linux-gnu). Combined them into a
new file called mpich.
Platforms tested:
Tested in heping using mpicc and mpich2/mpicc and also in tg-ncsa.
Misc. update:
Updated MANIFEST.
new h5diff test
Description:
added a test to the test h5diff script that compares a file to itself.
this test is done to test some features of the library that
open the same file and the root group twice
Solution:
Platforms tested:
linux
solaris
Misc. update:
Bug fix
Description:
Changed the name of a file from the last fix.
createnoenc.c is not gen_noencoder.c
Solution:
This is just a more standard name for the file.
Platforms tested:
tested in previous check-in; forgot to commit to cvs.
Code cleanup/reorganization
Description:
Merge back some more changes extracted from the "compact group" set.
This bunch cleans up and prepares the H5G_* routines for eventual import of
new features.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
Mac OS X.4
Code cleanup/bug fix
Description:
Add missing files to manifest
Remove C++ style single-line comment from recent checkin and also re-indent
in a few places.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/thread-safe
Not configured w/h5committest
Bug fix & code cleanup
Description:
Fix another bug in the file mounting code and refactor the unmount
code that it is simpler.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
Too minor to require h5committest
Bug fix
Description:
Rewrite code for mounting files to clean up layers of kludges and implement
a much cleaner and more maintainable design.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4
new test hor h5repack, to syncronize tests between unix and windows
it requires a new file added
Description:
Solution:
Platforms tested:
linux
Misc. update:
bug fix.
Description:
-nt is not a universal option for the test command in all platforms.
The use of it in conclude.am cause some platforms to bark at the
Makefile generated.
Solution:
Created a command script bin/newer which tests if file1 is newer
than file2. Replace the "test file1 -nt file2" by
"newer file1 file2".
Platforms tested:
Tested in sol.
Description:
1.7 daily tests failed in HL library when Fortran was not
enabled; also long long instead of long_long declarations
were used and caused failures on Windows
Solution:
Removed Fortran related functions (except H5IM_find_palette) from
H5IM.c file and put it into fortran/src/H5IMcc.c
Modified appropriate Makefile.am files, regenerated Makefile.in files, updated header files, MANIFEST, etc.
Note: H5IM_find_palette should be also in H5IMcc.c, but
for some unknown to me reason, linking fails if the function is
included in H5IMcc.c. I will really appreciate if some C guru
will look into the code and figure out what is wrong :-) Thanks!
Platforms tested: mir with g95, heping with and without Fortran enabled
Misc. update:
New feature
Description:
Add group creation & access property lists, dataset access property lists
and named datatype creation & access property lists. Currently have
<foo>_extend() API names, which will need to be changed for the final release.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (heping)
Add new html document files for windows users
Add new project file dt_arith.dsp and dt_arithdll.dsp
Remove dt_atomic.dsp and dt_atomicdll.dsp
Description:
Solution:
Platforms tested:
Misc. update:
Description: Renamed test/dt_atomic.c to test/dt_arith.c(an abbreviation
of "arithmetic data types") to better describe the program. This checkin
broke down into three sections when the floating-point numbers are the
source, normalized, denormalized, and special values. If there's any
difference of destination values, only normalized value test reports
failure. The other 2 report only as warning.
Platforms tested: h5commitest and fuss
Misc. update: MANIFEST
Several files added into windows directory
Description:
Added:
Batch files for intel and .net
Documents for windows
Removed:
Short instruction for windows
Solution:
Platforms tested:
Misc. update:
Two new project files added.
Description:
Two new test tools h5repart project files added into windows/tools/testfiles
Solution:
Platforms tested:
Misc. update:
Bug fix/improvement
Description:
Added another step to fortran type detection. This should help on some
bizarre platforms.
Solution:
Formerly, H5fortran_detect.f90 was included in the distribution. Now
it is generated by H5test_kind.f90 (written by Elena).
Platforms tested:
verbena, modi4, copper
Description:See details from Bug #213. Family member file size wasn't saved
anywhere in file. When family file is opened, the first member size determine
the member size.
Misc. update: Added a family file as the source file for backward
compatibility test.
Description: See details from Bug #213. Family member file size wasn't saved
anywhere in file. When family file is opened, the first member size determine
the member size.
Solution: This is the fourth step of checkin. A test suit is added for h5repart,
including a program to generate the test files, a script file to run h5repart,
and a program to verify repartitioned files can be opened by the library.
There's a change from the first step of checkin. Family name template is no
longer saved in the superblock because different pathname can make the name
different.
In the third step of checkin, h5repart has been modified. If h5repart is used
to change the size of family member file, the new size(actual member size) is saved
in the superblock.
In the second step of checkin, multi driver is checked against the driver
name saved in superblock. Wrong driver will result in a failure with an error message indicating
multi driver should be used. This change includes split driver because it's a special case for multi
driver.
In the first step of checkin. Family member size and name template(unused at this stage) are saved
in file superblock. When file is reopened,the size passed in thrin superblock. A different size
will trigger a failure with an error message indicating the right size. Wrong driver to open family
file will cause a failure, too.
Platforms tested: h5committest and fuss.
Misc. update: MANIFEST
Added documentation
Description:
Added "Automake Use Cases" document to Tech Notes.
This is intended as a reference or an introductory document; more
in-depth documentation is in progress.
Platforms tested:
Change only to documentation. Documentation is viewable on Windows.
new fortran parallel test
Description:
added a test that writes and reads several datasets
Solution:
Platforms tested:
linux (intel compiler, tungsten)
AIX (copper)
Misc. update:
Feature
Description:
Added "support" for UTF-8 character encoding.
Solution:
Wrote tests to check that UTF-8 can be used in a number of places in
HDF5 (object names, data, etc.). These tests live in test/tunicode.c.
Added a new UTF-8 character encoding for datatypes.
Platforms tested:
mir, modi4, heping
Misc. update:
Three project files and three workspaces added into hdf5/windows/misc
Description:
Add the following files
/windows/misc/H5fortran_detect_gen/H5fortran_detect_gen.dsp
./windows/misc/H5fortran_detect_gen/H5fortran_detect_gen.dsw
./windows/misc/fortrantypegen/H5fortran_detect.dsp
./windows/misc/fortrantypegen/fortrantypegen.dsw
./windows/misc/matchtypegen/H5match_types.dsp
./windows/misc/matchtypegen/matchtypegen.dsw
Solution:
Platforms tested:
Misc. update:
New feature.
Description:
Add "memory pool" internal data structure. This set of routines is
designed to add a way to allocate small pieces of information for a particular
purpose and then free all the pieces at once (i.e. without having to free each
piece individually). Memory pools are also good for localizing lots of small
allocations that logically belong together.
For example, if you were constructing a temporary linked list, you could
create a new memory pool, allocate all the nodes for the list from the memory
pool and when you were done with the list, just destroy the pool instead of
tracking through the list and freeing each block independently.
Platforms tested:
FreeBSD 4.11 (sleipnir)
h5committest
Cray X1 Port
Description:
Porting 1.7 branch to Cray X1. With these changes, HDF5 builds, but
there are some errors in the tests. Working on the errors.
Solution:
Added nv1-cray file to config directory.
Cleaned up some code in hl/c++ that was causing compiler to complain.
Platforms tested:
Cray X1, mir, sleipnir
Misc. update:
Configuration feature
Description:
Fortran integer types (not reals) are now automatically detected at
build-time.
Solution:
Two helper programs are used, one to detect what types the Fortran compiler
has access to, and one to generate header files for C and Fortran matching
up types.
Platforms tested:
mir, copper, modi4, pommier (last week)
Misc. update:
MANIFEST updated, H5f90fortran_types.f90 removed from configure.in, since
it is not longer generated by configure.
Bug fix/code cleanup
Description:
Add tests to determine that very long (64K+) object names are working.
Fixed a couple of bugs in h5dump where they weren't...
Platforms tested:
FreeBSD 4.11 (sleipnir)
Solaris 2.9 (shanti)
Feature - libtool shared library versioning
Description:
Libtool provides a mechanism for different versions of the same shared
library to be distinguished. Now this is applied to HDF5 when it is
built as a shared library.
Solution:
The version number is stored in config/lt_vers.am, and included in
src/Makefile.am.
This number will be automatically updated by bin/h5vers; developers
only need to update it when they change the API.
*** IMPORTANT ***
Any time the API changes, the version number in config/lt_vers.am must
be updated!
Platforms tested:
mir, eirene, verbena, modi4
Misc. update:
Description: Removed support for SRB driver
Solution: Removed or modified appropriate files; ran reconfigure
to regenerate Makefile.in and configure files.
Platforms tested: heping and shanti
Misc. update: ran bin/chkmanifest on heping
Description: Split test/dtypes.c because it's so big. Moved all integer-integer,
floating-floating, integer-floating, floating-integer, derived integer, derived
floating conversion tests to a new test program, dt_atomic.c. The test remains
the same basically.
Platforms tested: fuss and h5committest
Misc. update: MANIFEST
add a new feature for the test image API, read a palette from an ASCII file and attach several
palettes to an image
Description:
Solution:
Platforms tested:
linux
solaris
AIX
IRIX
Misc. update:
New feature
Description:
Add first iteration of "segmented heap" code, which will be used to store
links in groups in a more flexible way than the previous "local heap" mechanism.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Solaris 2.9 (shanti)
add 1 test that reads realistic dimension scales data from an ASCII file and generates an hdf5 file with DSs
Description:
Solution:
Platforms tested:
linux
solaris
IRIX
Misc. update:
Update MANIFEST for windows project files.
Description:
Many windows project files are added into CVS. MANIFEST needs to be updated.
Solution:
Platforms tested:
Misc. update:
Purpose:
Added C++ wrapper for Packet Table API.
Description:
Added macro for high-level C++ library (LIBH5CPP_HL), which changes every
Makefile.in.
Added directories for high-level C++ library (though currently only Packet
Table API is supported).
Added both C++ source and tests.
Platforms tested:
sleipnir, mir, modi4
Misc. update:
Added Packet Table to high-level APIs
Description:
The Packet Table is an API that allows the user to append records
("packets") to a table, and read the back again. It supports
fixed-length records with a defined datatype and variable-length
records. It also supports a "current record" index to track the
user's position in the table.
Solution:
The Packet Table code lives in hl/src, and its tests in hl/test.
Some code is shared between the H5TB table and the H5PT Packet Table
in the form of functions in H5HL_private.c.
Some documentation exists for a previous version of the API. Updated
documentation and C++ wrapper API coming soon.
Platforms tested:
sleipnir, eirene, copper, modi4
Description: Somehow, the hardware conversions between "long double" and other native floating-point
types were left out.
Solution: Added the hardware conversion functions in H5Tconv.c and test cases in dtypes.c.
Platforms tested: h5committest and fuss.
Misc. update: updated MANIFEST to replace bin/reconfigure.sh with bin/reconfigure
feature
Description:
h5repack support for scaleoffset compression
Checking in early to help debug the filter.
Solution:
Added messages and command line to handle new scale offset filter.
Note: TESTS ARE DISABLED FOR NOW. The filter is not
complete, repack tests may fail due to know problems.
PLEASE DO NOT MESS WITH THE SCALEOFFSET TESTS AT THIS TIME.
They will be enabled when the filter is ready.
Platforms tested:
verbena,copper,shanti
Misc. update:
MANIFEST
New feature
Description:
Expand v2 B-tree code to support splitting the root node when enough
records are inserted and move metadata cache callbacks into their own source
file.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Too minor for h5committest
New feature
Description:
Add basic code for new B-tree implementation. They don't do much yet,
aren't hooked up to anything yet and the format may change, but I'd like to
start getting them into the daily tests.
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Too minor to require h5committest
Bug fix
Description:
Two files in the bin directory didn't make it into CVS.
Solution:
Added depcomp and compile to CVS repository.
Platforms tested:
Already testsed with these files.
Configuration feature
Description:
HDF5 now uses automake to generate Makefiles
Solution:
Makefile.in files are now generated from Makefile.am files.
To reconfigure (after chaning a Makefile.am or configure.in):
/bin/sh bin/reconfigure.sh
Platforms tested:
Many
added a first batch of dimension scales
fix some small bubgs in lite (a close function was not being called )
Description:
this batch contains the basic API functions described in the RFC and a minimal test file
Solution:
Platforms tested:
linux
solaris 64
AIX
windows
Misc. update:
changed the name of the hl table files back to H5TB.c and H5TB.h
this can be done now because the H5TB.c (threaded binary trees) file was removed from the hdf5 library
Description:
Solution:
Platforms tested:
linux
Misc. update:
Description: Removed PABLO from the source
Solution:
Platforms tested: arabica with 64-bit, copper with parallel,
heping with GNU C and C++ and PGI fortran (but
I disabled hl, there is some weird problem only
on heping: F9XMODFLAG is not
propagated to the Makefile files
Misc. update:
A new configuration file hdf5/config/i686-pc-cygwin is add for HDF5 on Cygwin. So MANIFEST should be updated to include this file.
Description:
Solution:
Platforms tested:
Misc. update:
Remove feature
Description:
Retire threaded, balanced binary tree code from HDF5 use. Requiescat in
pace...
Also, regenerate dependencies files.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Too minor to require full h5committesting (the code is already
disconnected from everything except its tests)
Bug Fix/Code Cleanup/Doc Cleanup/Optimization/Branch Sync :-)
Description:
Generally speaking, this is the "signed->unsigned" change to selections.
However, in the process of merging code back, things got stickier and stickier
until I ended up doing a big "sync the two branches up" operation. So... I
brought back all the "infrastructure" fixes from the development branch to the
release branch (which I think were actually making some improvement in
performance) as well as fixed several bugs which had been fixed in one branch,
but not the other.
I've also tagged the repository before making this checkin with the label
"before_signed_unsigned_changes".
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel & fphdf5
FreeBSD 4.10 (sleipnir) w/threadsafe
FreeBSD 4.10 (sleipnir) w/backward compatibility
Solaris 2.7 (arabica) w/"purify options"
Solaris 2.8 (sol) w/FORTRAN & C++
AIX 5.x (copper) w/parallel & FORTRAN
IRIX64 6.5 (modi4) w/FORTRAN
Linux 2.4 (heping) w/FORTRAN & C++
Misc. update:
Modify the cache code (H5C) to support automatic cache resizing to
adapt to the work load at run time.
Description:
Different applications require different sized caches to maintain
an acceptable hit rate. This set of changes attempts to provide the
ability to adjust to circumstances automatically.
Solution:
Added highly configurable code to allow the user to either set a
fixed cache size, or allow the cache to grow and shrink according to
conditions.
If enabled, cache size increases are triggered when the hit rate
drops below a user specified threshold in a user specified interval.
Cache size reductions (if enabled) are triggered when either the
hit rate exceeds some user specified threshold over a user specified
interval, when the cache contains "enough" entries that haven't been
accessed for a user specified interval, or some mix of the above.
See the header comments on the H5C_auto_size_ctl_t structure in
H5Cprivate.h for further details.
At present, the cache resize configuration options are not
accessible via the user API. Must add this.
Platforms tested:
h5committested, hepingi (serial), and copper (parallel)
Misc. update:
Port
Description:
Initial work for supporting GNU FORTRAN/F95 on FreeBSD. I think I've
got things mostly set up correctly, but I'm getting an internal compiler
error on one of the FORTRAN sources, so I'm not going to add this configuration
to the daily tests yet.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/GNU FORTRAN
Not tested in h5committest
add hl fortran
Description:
Solution:
Platforms tested:
linux (absfot and pgf90)
solaris (32 and 64 bit)
AIX
note : HP gives a compiling error , to be fixed in the future
Misc. update:
Add new internal data structure
Description:
Add an implementation of skip lists to the library (see comment in
src/H5SL.c for references to the papers describing them) as a potential
replacement for our current threaded, balanced binary tree container.
Skip lists are much simpler to implement and should be faster to use.
Also, added new error codes to release branch, so bump the minor version
number to indicate that the library is no longer perfectly compatible with
the 1.6.3 release.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Solaris 2.7 (arabica)
Too minor to require further testing (the skip lists aren't actually
used by any library code yet)
bug fix
Description:
the Makefile.in of the hl tests was not including a "srcdir" include path
the test table .c file was missing also the srcdir path
Solution:
add it
Platforms tested:
Misc. update:
insert high level library into main library
Description:
configure.in has a new entry (--enable-hl, disable-hl) it is on by default
the configure message prints info regarding if hl is enabled or not
the tree is
/hl
/hl/src
/hl/test
Solution:
Platforms tested:
linux
solaris
AIX
Misc. update:
Fix SZIP filter to dynmically detect encoder.
Description:
Solution:
See:
http://hdf.ncsa.uiuc.edu/RFC/SZIP/Szip_dynamic_12_Oct.pdf
Platforms tested:
verbena, arabica, cobalt.
Note RE testing:
All automatic tests work as before.
The feature is a configuration thing, so the tests are manual.
Tested with the following configs:
pre-release version of SZIP, with dynamic library,
1. no SZIP
2. SZIP, decoder only
3. SZIP, encoder+decoder
released (static lib) SZIP
4. no SZIP
5. SZIP, decoder only
6. SZIP, encoder+decoder
after build, switch szip in LD_LIBRARY_PATH at run time
7. SZIP decoder
8. SZIP encoder
Misc. update:
Manifest updated
bug fix
Description:
Description:
one case was not handled in the combination of input options (layout and filters)
Solution:
redo the algorythm that handles all cases
Solution:
Platforms tested:
linux
Misc. update:
Refactor and update fortran configuration
Description:
Changed configure.in to include the production, debugging and profiling
flags for FORTRAN in the command line.
Added '-flags' files for PGI and Intel C compilers
Added '-fflags' files for PGI and Intel FORTRAN compilers
Updated 'gnu-flags' file for GNU C compiler to not modify compiler flags
when gcc is not being used.
Make detection of the correct FORTRAN compiler a bit smarter for Linux
platforms. (And carried over some of the minor changes to other affected
machines)
Platforms tested:
Linux 2.4 (tungsten) w/PGI and Intel C & FORTRAN compilers.
Changes not tested by h5committest
Bug fix & code cleanup
Description:
More dataset cleanups to get to a point where we can fix the chunked I/O
bug.
Also fix a couple of errors in the recent file object resurrection changes
which should hopefully address the recent daily test failres (H5T.c)
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Solaris 2.7 (arabica)
h5committest
new test
Description:
added a test that generates and copies a file with a dataset with fill value
(this is to test the property list function H5Pequal)
Solution:
Platforms tested:
linux
solaris
aix
Misc. update:
Code cleanup
Description:
Fix another batch of minor differences between the development and release
branches.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
Too minor to require h5committest
Code cleanup
Description:
Clean up collective chunking code a bit.
Also, add '--enable-instrument' configure flag to have a mechanism for
determining that optimized operations happened correctly in the library (instead
of just the "normal" way) by allowing 'flag' properties to be set outside the
library and set when the "right" thing happens. This is mainly for debugging
and regression checks, so we make certain we don't break optimized I/O by
accident. It's enabled by default when --enable-debug is on (which is on by
default in the development branch and off by default in the release branch),
but can also be independently controlled with its own configure flag.
Platforms tested:
FreeBSD 4.10 (sleipnir) w/parallel
IBM p690 (copper) w/parallel
h5diff and h5repack changes
Description:
h5diff
introduced the following four modes of output:
Normal mode: print the number of differences found and where they occured
Report mode: print the above plus the differences
Verbose mode: print the above plus a list of objects and warnings
Quiet mode: do not print output (h5diff always returns an exit code of 1 when differences are found)
h5repack
added an extra parameter for SZIP filter (coding method)
the new syntax is
-f SZIP=<pixels per block,coding>
(pixels per block is a even number in 2-32 and coding method is 'EC' or 'NN')
Example of use:
./h5repack -i file1 -o file2 -f SZIP=8,NN -v
updated usage messages, test scripts and files accordingly
Solution:
Platforms tested:
linux
AIX
solaris
Misc. update:
Purpose:
Bug Fix
Description:
If an HDF5 file grows larger than its address space, it dies and is unable to
write any data. This is more likely to happen since users are able to change
the number of bytes used to store addresses in the file.
Solution:
HDF5 now throws an error instead of dying. In addition, it "reserves" address
space for the local heap and for object headers (which do not allocate space
immediately). This ensures that after the error occurs, there is enough address
space left to flush the entire file to disk, so no data is lost.
A more complete explanation is at /doc/html/TechNotes/ReservedFileSpace.html
Platforms tested:
sleipnir, copper (parallel), verbena, arabica, Windows (Visual Studio 7)
Solution:
Platforms tested:
Misc. update:
h5repack changes
Description:
there were some requests to change some minor h5repack features
h5repack only made a warning about a non available filter in verbose mode ( -v )
without -v it kept silent, and users sometimes missed this warning
the request was that it should print this warning always. so, the new format, is e.g
./h5repack -i test_szip.h5 -o out.h5
Warning: dataset </dset_szip> cannot be read, SZIP filter is not available
due to this, and to avoid a lot of these messages in the shell test script, I modified
the script h5repack.sh so that it detects the presence of all filters in the environment
(previously it only detected SZIP)
the test files were also divided in more files , to make the script code easier to
follow
Solution:
Platforms tested:
linux
AIX (no szip)
solaris (no szip, no gzip )
Misc. update:
h5dump new tests
Description:
added new tests for the print of array indices (nested objects, several ranks)
Solution:
Platforms tested:
linux
AIX
solaris
Misc. update:
Bug fix
Description:
The "shared" raw B-tree node can get freed before all the B-tree nodes
had been flushed out to disk and released by the cache.
Solution:
Implement a simple reference counting wrapper for objects in the library
and use it to hold the shared raw B-tree nodes so they aren't freed before all
references to them in memory are released.
Platforms tested:
Solaris 2.7 (arabica)
FreeBSD 4.10 (sleipnir)
IRIX64 6.5 (modei4)
h5dump new tests
Description:
added more tests for the escape/not escape feature for string data (with vlen, with
compound, with char data)
Solution:
Platforms tested:
linux
solaris
AIX
Misc. update:
h5dump new tests
Description:
added new tests for the -p option, superblock, file contents, fill values, array indices.
Solution:
Platforms tested:
linux
AIX
solaris
Misc. update:
Description:
Replaced the old metadata cache with a cache with a modified LRU
replacement policy. This should improve the hit rate.
Solution:
Since we want to flush cache entries in increasing address order, I
used the threaded binary B-tree code to store the cache entries.
There is a fair bit of overhead here, so we may want to consider
other options.
While the code is designed to allow the support of other replacement
algorithms, at present, only a modified version of LRU is supported.
The modified LRU algorithm requires that a user selectable portion
of the cache entries be clean. The clean entries are evicted first
when writes are not permitted. If the pool of clean entries is used
up, the cache grows beyond its user specified maximum size. The
cache can also exceed its maximum size if the combined size of the
protected (or locked) entries exceeds the maximum size of the cache.
Platforms tested:
eirene (serial, parallel, fp), h5committested
Misc. update:
Purpose:
HDF5 now supports SZIP with no encoder.
Description:
SZIP can be configured to have both encoder and decoder or just to have the decoder. HDF5 can now query the configuration of any filter, and will throw errors if users try to write using a filter with encoding disabled.
Solution:
Added H5Zget_filter_info function, changed API for H5Pget_filter and H5P_get_filter_by_id. See SZIP RFC.
Platforms tested:
Copper (fortran, C++, parallel), Sleipnir (C++), Arabica (fortran, C++), Verbena (fortran, C++)
Misc. update:
Description: Added new API H5Fget_name and new test program called filename.c. This function
returns the name of the file by object ID(file, group, dataset, named datatype, and attribute)
which belongs to the file.
Platforms tested: h5committest and fuss.
Misc. update: MANIFEST and RELEASE.txt
h5dump output change, new tests
Description:
the storage layout output format the storage layout output format had some changes
same for the user defined filter
add an option (-y) for not printing the array indices (default is print indices )
the option for escaping non printable characters covers all characters (default is not escape)
(this might be not very portable, the test files are tstring.ddl and tstringe.ddl )
add tests for the new options
Solution:
Platforms tested:
linux
solaris
AIX
Misc. update:
Description: This is the second effort to correct XML dumper after adding null
dataspace test for attribute and dataset. Since XML schema hasn't been updated
for null space, took out null space test from tdset.h5 and tattr.h5 and put it
into a seperate file, tnullspace.h5. Only h5dump tests this null space file;
XML dumper doesn't do it at this moment. We'll wait until XML schema is updated
first.
Platforms tested: h5committest and RH 8(fuss)
Misc. update: MANIFEST(added two new files in tools/testfiles, tnullspace.h5
and tnullspace.ddl)
Description:
added the code for print strings with new line and display the path of references (new source files h5tools_ref.c and .h )
added a test suite in testh5dump.sh.in for
( note : to create testh5dump.sh , one must redo ./configure; this detects the availability of filters
and generates testh5dump.sh accordingly)
1) storage layout
2) fill value
3) print reference with path
4) print strings with new lines
5) filters
Solution:
Platforms tested:
linux
solaris
AIX
Misc. update:
Update shell scripts
Description:
Switch to generating the testh5dump.sh script at configure time, so we can
determine which filters are available to test.
Platforms tested:
FreeBSD 4.9 (sleipnir)
too small to require h5committest
h5dump new version
Description:
added the changes already made for 1.6
support for dumping of
1) filters
2) storage layout
3) fill value
4) comments
5) superblock
6) file contents
7) array indices
Solution:
Platforms tested:
linux
solaris
AIX
Misc. update:
Code optimization & bug fix
Description:
When dimension information is being stored in the storage layout message
on disk, it is stored as 32-bit quantities, possibly truncating the dimension
information, if a dimension is greater than 32-bits in size.
Solution:
Fix the storage layout message problem by revising file format to not store
dimension information, since it is already available in the dataspace.
Also revise the storage layout data structures to be more compartmentalized
for the information for contiguous, chunked and compact storage.
Platforms tested:
FreeBSD 4.9 (sleipnir) w/parallel
Solaris 2.7 (arabica)
h5committest
New Feature
Description:
Add the data transform function, H5Pset_transform().
Platforms tested:
"h5committested".
Copper was down. Ran parallel tests in sol instead.
Misc. update:
new tests for h5repack
Description:
added more tests both to the test program and shell script that test
a variation of different filter converssions
Solution:
Platforms tested:
linux
Misc. update:
new tests for h5repack
Description:
added tests that do layout type to layout type conversion in a matrix of 9 between compact, contiguous and chunking
Solution:
Platforms tested:
linux
afs has problems; I could not telnet to sol and copper, arabica is really slow (meaning
waiting 1 minute for a typed character) and the writing of a file gave an error
arabica 181% afs: failed to store file (145)
afs: failed to store file (145)
Misc. update:
1) new function for tools library
2) new test script for h5repack
Description:
1) currently all the tools (h5dump, h5diff, etc) do not check if a filter is available
for reading some dataset that might have a filter not available on the current configuration (the behaviour
of the tools until now was to trigger a library error, saying that the dataset cannot be read
due to the lack of the filter)
Solution:
1) added a new function h5tools_canreadf that checks if a dataset can be read
depending on the availability of filters.
this function was added in calls for h5diff and h5repack.
instead of triggering the library error, a message is printed, saying that the dataset
cannot be read (the print is optional, it is on on verbose mode)
2) added a shell script that tests the commannd line tool behaviour of h5repack
the script does a series of runs of h5repack with several options on the same file (this file test4.h5
was added to the testfiles dir).
then, it runs the h5diff tool, with the input and output files , in each run.
the goal of the test is also to check item 1) . the binary file was saved with filters
that might not be available on other configurations
Platforms tested:
linux (all filters enabled)
linux (some filters disabled)
solaris (some filters disabled)
AIX (some filters disabled)
windows (all filters on and off )
Misc. update:
Bug fix/optimization
Description:
Address slowdown in MPI-I/O file metadata operations that was introduced
mid-stream. We now _require_ a POSIX compliant parallel file system for the
MPI-I/O file driver (as well as for the MPI-POSIX file driver).
Also optimized file open operation when the file is being created by
reducing the number of collective & syncronizing calls.
Additionally, refactor the MPI routines into a common place, eliminating
duplicated code.
Platforms tested:
FreeBSD 4.9 (sleipnir) w/parallel
h5committest
Purpose: Maintenance
Description: Fortran APIs MAC OS X port for IBM XL Fortran compiler
Solution: Brought back changes from 1.6 branch
Platforms tested: pommier, h5comittested; this time h5committest
complained about copperpp directory and didn't run;
tests on verbena and sol passed.
Misc. update:
Code cleanup
Description:
Refactor library testing framework (used for the testhdf5 & ttsafe tests)
to remove almost all of the duplicated code, moving the common code into a
new 'testframe.c' source file.
Platforms tested:
FreeBSD 4.9 (sleipnir) w & w/o thread-safety
h5committest
h5diff new feature
Description:
added compare for attributes
a new options flag (-a) was added to the options structure. it is 0 by default (no compare )
the output of the compare is the same that for datasets, and all the other flags also apply for attributes
(the memory compare is done in the same function diff_array)
all the other requirements for compare of datasets (type, space) are identical too
Platforms tested:
linux
solaris 2.7
IRIX
Misc. update:
Code cleanup
Description:
Removed "H5Git" routines, now that there are library routines which perform
the same functionality.
Platforms tested:
FreeBSD 4.9 (sleipnir)
Linux 2.4 (verbena) w/FORTRAN
too minor for h5committest
Description: The standard output from Error API test has some non-standard
message like path name or line number.
Solution: use sed to remove any non-standard information in testerror.sh
Platforms tested: h5committest
to avoid printing error messages.
Description: If enable-hdf5v1_6 is configured in, make some functions
compatible with v1.6. Error test program print out some error messages as
it succeeds.
Solution: Use #ifdef H5_WANT_H5_V1_6_COMPAT statements. Use shell script
to compare error test output with the standard one.
Platforms tested: h5committest
Purpose:
Bug fix
Description:
pablo build failed.
Solution:
Fixed the makefile so that the needed header files are again created on the
fly.
Platforms tested:
modi4, copper, verbena, arabica
Misc. update:
Purpose: Bug fix.
Description: Parallel make fails because Pablo IDs are not assigned.
Solution: Extracted the Pablo IDs into header files and added them to the
pablo subdirectory instead of creating them on the fly.
Platforms tested: h5committested by Albert
Misc. update: Have updated MANIFEXT
This is an omibus update to the h5dump program.
1. Fixes bug 888
2. updates XML output to 1.6
Description:
Solution:
Platforms tested:
verbena, arabica, copper (64)
Misc. update:
MANIFEST updated
Need to do release notes.
Update
Description:
Revamped the configuration system. The configurations for the Fortran
and C++ libraries are no longer separate from the "main"
configuration system. This involved removing the "configure*" and
"aclocal.m4" files from the fortran/ and c++/ subdirectories. Also
merging settings in the config/ subdirectories into the main config/
subdirectory.
Fortran header files had to be modified a little for Linux. It was
checking if it was a Linux machine by some #defines, however with the
-std=c99 switch, these defines weren't there. I added a check for
some other ones which should be there whether the -std=c99 switch is
used or not.
Platforms tested:
Verbena (Fortran & C++)
Sol (Fortran & C++)
Copper (Fortran & C++)
Modi4 (Parallel, Fortran, & C++)
Misc. update:
New tool.
Description:
A new tool to build the HDF5 library. This was historically the
makeh5 command. Added here so that it can be used in machines outside
of the HDF Groups. Having it in the source allows version specific
customization.
Platforms tested:
Did run h5committest which failed due to other source code problems.
Tested it by hand by ../hdf5/bin/buildhdf5 and passed as far as its
own functionality.
Misc. update:
Purpose: Maintenance
Description: I added three new configuration files in c++/config
directory for Cray SV1, T3E and T90IEEE
Solution: Modified MANIFEST to add new files.
Platforms tested:
Misc. update:
Purpose: Maintenance for 1.6 release
Description: I added INSTALL_Windows_withf90.txt file that was lost
in 1.5 branch.
Solution: Modified MANIFEST to include new file.
Platforms tested:
Misc. update:
Code cleanup
Description:
Limit the scope on more function prototypes/macros/typedefs.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committest not necessary.
New test
Description:
Added test to check for dangling objects being handled correctly by the
library.
Platforms tested:
FreeBSD 4.8 (sleipnir)
h5committested
New feature/enhancement
Description:
Chunked datasets are handled poorly in several circumstances involving
certain selections and chunks that are too large for the chunk cache and/or
chunks with filters, causing the chunk to be read from disk multiple times.
Solution:
Rearrange raw data I/O infrastructure to handle chunked datasets in a much
more friendly way by creating a selection in memory and on disk for each chunk
in a chunked dataset and performing all of the I/O on that chunk at one time.
There are still some scalability (the current code attempts to
create a selection for all the chunks in the dataset, instead of just the
chunks that are accessed, requiring portions of the istore.c and fillval.c
tests to be commented out) and performance issues, but checking this in will
allow the changes to be tested by a much wider audience while I address the
remaining issues.
Platforms tested:
h5committested, FreeBSD 4.8 (sleipnir) serial & parallel, Linux 2.4 (eirene)
Purpose: Maintenance for T3E
Description: Cray updated T3E to run unicosmk2.0.6.X
Configuration failed.
Solution: Created new unicosmk2.0.6.X files in the config and fortran/config
directories and added them to MANIFEST
Platforms tested: Cray T3E
Misc. update:
Feature Add
Description:
It's now possible to print out characters as actual characters
instead of their decimal equivalent numbers. It's the same thing that
h5ls does. The flag to do this is "--string" or "-r".
Platforms tested:
Verbena (Fortran & C++)
Arabica (Fortran)
Modi4 (Fortran & Parallel)
Misc. update:
Purpose: Catching up with 1.4 branch
Description: 1.5 didn't have unicos10.0.X configuration files
for both C and fortran.
Solution: Added the files and updated MANIFEST
Platforms tested: bin.chmanifest on arabica
Misc. update:
Addition
Description:
Added "h5fc" utility. This is the same as the "h5cc" utility but for
Fortran. It only handles static libraries since that's the only
library we build for Fortran.
Note! I've tested the code on the normal platforms but I haven't
tested the utility itself.
Platforms tested:
Arabica (Fortran)
Modi4 (Fortran & Parallel)
Burrwhite (Fortran & C++
Misc. update:
Purpose: New tool
Description: Pankaj created h5import tool long time ago. I am finally checking
the source code in for the 1.6 release.
Known problems: h5import test will crash if srcdir option
is used; will be fixed before the release.
Solution:
Platforms tested: vebena, arabica, modi4 were tested. h5import tests passed
I also tested on arabica with srcdir option to make sure that
there is a problem when srcdir is used. After that I disabled
h5import tests. Daily tests will build h5import, but will not run
h5import tests.
Misc. update:
added test files for h5diff
Description:
2 binary hdf5 files, generated by h5difftst.c located in /tools/h5diff
1 of several text files to be added , that have a predifined h5diff output , to be compared with
the output of the program rum by testh5diff.sh
Solution:
Platforms tested:
Misc. update:
Purpose: Bug fix.
Description: When I added support for filters, I forgot to add
fortan man pages file to the MANIFEST file.
When I run chkmanifest, I discovered that there was also
a typo ( ./dac/Makefile.in instead of ./doc/Makefile.in )
Solution: added ./doc/html/fortran/h5z_FORTRAN.html and chnages "a" to "o"
Platforms tested: arabica with bin/chkmanifest
Misc. update:
Purpose: Catching up with C library
Description: Added new files for PGI compilers support on Linux 2.4
config/linux-gnuaout
fortran/config/linux-gnuaout
Added new source and test files for filters
fortran/src/H5Zf.c
fortran/src/H5Zff.f90
fortran/test/tH5Z.f90
Solution:
Platforms tested: arabica(C and F90), burrwhite (pgcc and pgf90), modi4 (F90 and parallel)
Misc. update:
Bug fix
Description:
"make distclean" is failing currently in the doc/html/ed_libs and
doc/html/ed_styles directories for lack of a 'Dependencies' file.
Solution:
Add it.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Misc. update:
Purpose:
Maintenance
Description:
Added support for variable length datatypes.
Solution:
I added support for Fortran VL types to 1.4 branch before the
hdf5-1.4.5 release. Now I am bringing changes back to the 1.5 branch
Platforms tested:
arabica (fortran), eirene (fortran), modi4 (parallel, fortran)
New internal feature
Description:
Add internal API for building and working with heaps (H5HP). This will be
used for the LRU algorithm in the new metadata cache code.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir)
Code cleanup
Description:
Clean up miscellaneous warnings which have crept into the code.
Fix "_POSIX_C_SOURCE not defined" warning on FreeBSD.
Adjust gcc compiler flags to be more concise for production mode.
Refactor the H5O code so that there is a stronger boundary between code
in the H5O package and code in the library which just calls H5O routines.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) serial & parallel and gcc 2.95.4 & gcc 3.2.2
Misc. update:
Update MANIFEST if you add or remove any file.
Code cleanup.
Description:
Break up the ~9350 line H5T.c module into smaller pieces, which contain
code for a particular feature or support for a datatype class.
This should make the "main" H5T code (still in H5T.c) easier to support,
as well as removing some of the "minor" routines from the user applications
which don't use them (my rough estimates show about 4% reduction (~30K on
a FreeBSD machine) in optimized, staticly-linked binaries for very simple
programs)
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir)
Misc. update:
Update MANIFEST
New feature for developers.
Description:
Added "function stack" tracing to library. This allows developers (there
is no public API) to call H5FS_print within the library and get a listing
of the functions traversed to reach that point in the library. Eventually,
I may add support for reporting the parameters to each function also...
Mainly for debugging parallel I/O programs, but I think it will come in
handy in other cases also.
The function stack tracking is controlled with a configure switch:
--enable-funcstack, which defaults to enabled currently. When we branch
for 1.6, we should change the default setting on the branch to be disabled.
Also, added a destructor to the thread-specific keys when thread-safety is
turned on in the library. Otherwise, they were leaking memory and causing
difficult to debug errors in threaded programs (like the test/ttsafe test).
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
FreeBSD 4.7 (sleipnir) w/thread-safety enabled.
Misc. update:
Updated MANIFEST with new files added (src/H5FS.c & src/H5FDprivate.h)
Update release_docs/RELEASE with thread-safety bug fix.
Update
Description:
MANIFEST:
- Added .autom4te.cfg and H5FDfphdf5.[ch] files.
.autom4te.cfg
- When the new versions of libtool/autoconf come out, this file
will help get rid of the annoying autom4te.cache directory.
Platforms tested:
Linux
Bug fix
Description:
Currently, when the library encounters an object header message that isn't
know, it fails to open that object in the file.
Solution:
Allow the library to skip over the unknown object header message and
continue to process the remaining messages, in the hope that the skipped
message isn't important later. If it is important, it will be caught at
a higher level of the library.
Platforms tested:
FreeBSD 4.7 (sleipnir)
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
fortran/config/powerpc-ibm-aix5.x
To cover AIX 5.x.
Removed Files:
fortran/config/powerpc-ibm-aix4.2.1.0
fortran/config/powerpc-ibm-aix4.3.2.0
These files are outdated and have been replaced by powerpc-ibm-aix4.x.
Modified Files:
MANIFEST
Update it.
Platforms tested:
Only on LLNL blue. LLNL SP are the only machines that use
these files and that I have access.
Update
Description:
Added new files to the MANIFEST and added a new configure switch for
enabling building of the Flexible parallel HDF5 module.
Platforms tested:
Modi4 Eirene Arabica
Purpose:
Bug fix, code improvement
Description:
Fortran tests didn't cleanup created files. Also HDF5_PREFIX and
HDF5_PARAPREFIX were not used to specify location of the files.
There was a redundant file in the testpar directory that contained
an error reporting function used by both serial and parallel tests.
Solution:
Created library h5test_fortran.a that contains functions used by the
serial and parallel tests. It includes Fortran and C functions that
may be called from Fortran programs to report errors, to modify
file names and to cleanup files after run. Modified test code to
use new functions.
Platforms tested:
Linux 2.2 (eirene) serial
IRIX64 (modi4) parallel with HDF5_PREFIX and HDF5_PARAPREFIX set to $SCR
Solris 2.7 with mpich 1.2.4 with HDF5_PARAPREFIX set to /tmp/epourmal
Purpose:
Design for compact dataset
Description:
Compact dataset is stored in the header message for dataset layout.
Platforms tested:
arabica, eirene.
Code cleanup
Description:
Move get/set routines for each type of property list (file creation,
dataset creation, file access and dataset transfer) into their own source
code module.
Platforms tested:
FreeBSD 4.6 (sleipnir)
Compress I/O Test Add
Description:
This is the initial check-in of the compress I/O test. It will write
out compressed buffers to a file.
I need to implement a write of uncompressed data first for
comparison...
Platforms tested:
Linux
Update
Description:
Updated the way we used the AC_INIT macros so that it's current to
the new standard way and not deprecated anymore.
Also, added a test for some functions during parallel Fortran
configure. This required the creation of an H5config_fortran.h file
so that we can test for these...
Platforms tested:
Elena is going to test the changes to the fortran after she checks in
her other code. I will test on SDSC machines after getting her code.
The other changes were tested on Linux.
Autotools Update
Description:
I've updated autoconf, automake, and libtool to the latest/greatest
versions; 2.53, 1.6, and 1.4.2 resp.
Many changes come with the new versions:
- ltconfig is no longer used
- acconfig.h is no longer used (#define values are declared with
the macro)
- regeneration of all of the aclocal.m4, configure, and
H5config.h.in files.
- new config.{guess,sub} files
- new ltmain.sh file
Platforms tested:
AIX (blue), and Linux
