Push support for shared attributes into more dark corners of the "dense"
attribute storage support.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Add [quite] limited ability to update (ie. write) data for objects in
fractal heap. Limited to just updating objects in managed heap blocks (i.e.
not 'tiny' or 'huge' objects) and must be updated with data of the same length
as the object in the heap. Updating objects in compressed heaps does work
though [as long as the data isn't 'tiny' or 'huge'].
Needed for changing the data value or the name of an attribute that is
stored in dense or shared storage.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
messages to shared messages and vice versa).
Extended objcopy test to test all "permutations" of shared/unshared and
new/old group format. If this proves to take a long time to test, I'll
go back and have it run fewer tests when HDF5_TEST_EXPRESS is enabled.
Tested on smirom, kagiso, copper, and Windows.
Bug fix.
Description:
Test failed when core and multi VFD is used.
SOlution:
Failure was because the test file was created with the specified VFD access
property but later reopened by default access. That won't work for VFDs such
as multi that produced files that are incompatible with the default sec2
file driver.
Fixed it by using the same VFD access property when it reopens the same test
file.
However, it still fails for the core VFD which does not create any real file.
It is meaningless to try to reopen its file. Fixed it by adding core to
the incompatible list of VFDs so that it would skip the test.
Split is already coded as incompatible VFD but I don't understand why
it is incompatible in this case. Removed it from the incompatible list.
Tested Platforms:
Copper in which the daily test failed.
Set up tests for next phase of testing dense attribute operations, with
shared attributes.
Other minor code cleanups...
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Add "attribute exists" internal routine to make verifying that an attribute
with the same name doesn't already exist easier.
Tweak "trace" script to produce more whitespace in H5TRACE macros, in order
to make them easier to read.
Minor other whitespace cleanups
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Switch from using H5L_index_t/H5L_INDEX_<foo> to H5_index_t/H5_INDEX_<foo>
in order to accommodate indices on aspects of attributes as well as links.
Add basic support for deleting attributes in dense storage (needs more
support/tests for shared attributes in dense storage still).
Misc. cleanups, etc.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Migrate "internalish" attribute operations into new source code file.
Add test & basic support for opening attributes in dense storage (shared
attributes not tested or supported yet).
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Take out separate memory type in the file for SOHM objects and create
aliases for existing memory types for SOHM use.
Tested on:
FreeBSD/32 4.11 (sleipnir)
Add first pass of "dense" attribute storage to objects. Lots of parts of
this are stubbed out, but all the tests are passing and I'll work on the corner
cases soon.
Eliminated several unused parameters from object header message callback
routines.
Other, miscellaneous code cleanups, etc. (and probably some things I've
forgotten about... :-)
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
AIX/32 5.? (copper)
Finish moving object header message routines into their own source code
module, along with renaming them to have "H5O_msg_" prefix...
Tested on:
Mac OS X/32 10.4.8 (amazon)
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
AIX/32 5.? (copper)
Migrate more object header routines to use the H5O_msg_ prefix and put
them into the src/H5Omessage.c code module.
Tested on:
Mac OS X/32 10.4.8 (amazon)
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
AIX/32 5.? (copper)
Refactor object header code to separate process of creating an object
header message from the process of writing to an existing one.
Start renaming operations that deal with object header messages to have
"H5O_msg_" prefix...
Tested on:
Mac OS X/32 10.4.8 (amazon)
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
AIX/32 5.? (copper)
to make the test pass.
These changes involve changes to the file format of SOHMs, but that's okay
because nobody should have been using SOHMs yet anyway.
Tested on Windows, kagiso, copper, and heping
Fix problem with copying objects when the dataset to copy has an "older"
version of the layout message and needs to compute the size of the dataset's
storage after the dataspace information is available.
Minor random cleanups also...
Tested on:
FreeBSD/32 4.11 (sleipnir)
Too minor/weird to test w/h5committest...
Make the memory size of the max. compact & min. dense attribute storage
variables align with the property list size, instead of the size on disk, to
better handle big-endian machines.
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
AIX/32 5.? (copper)
Propagate object creation properties up into group, dataset and named
datatype property lists, when those property lists are retrieved for
existing objects in a file.
Also, add H5Tget_create_plist() API routine, to allow named datatype
property lists to be retrieved for named datatypes.
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
AIX/32 5.? (copper)
Add fields for tracking attributes on objects to object header prefix,
when using the latest version of the format.
Reduce size used to store type of header message from 2 bytes to 1. (If we
have >256 message types, we probably have a different problem... :-)
Also, add "birth" time for object (i.e. creation time) [named to align
with similar fields in file systems]
Correct bug in H5Gget_objinfo() - retrieve the ctime field from object info,
instead of the mtime field.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Reduce the size of some of the group information to more reasonable bounds.
(16-bit values instead of 32-bit values).
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
When using the latest version of the file format, move the "modification
time" information into the object header prefix, which is more efficient.
Also add "access time" and "change time" (for metadata) fields, all of which
take about the same space as the previous modification time header message.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
General cleanups, in preparation for adding some attribute-tracking fields
to the object header prefix.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Add # of attributes to the object info struct and deprecate
H5Aget_num_attrs().
Tested on:
FreeBSd/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
AIX/32 5.? (copper)
Finish removing library's internal code that uses H5G_get_objinfo() and
retarget it at either getting the link information or the object information,
as appropriate. (Still need to add user-level tests for H5Oget_info(), but
since several internal components of the library depend on the internal version,
it appears to be working correctly).
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/322.4 (heping)
Linux/64 2.4 (mir)
AIX/32 5.? (copper)
Mac OS X/32 10.4.8 (amazon)
Add tests for H5Oopen_by_idx() (in a rare fit of superlative coding I
apparently got the implementation complete correct in my earlier checkin :-)
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Add tests for H5Literate(), in all combinations.
Sweep up a few minor issues with H5Literate() that the tests exposed.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Finish implementation of H5Literate() [still needs real tests]
Clean up datatype macro warnings a bit more.
Unify iterator callback macros and put up in public header file, with the
iterator directions.
Tested on:
Mac OS X/32 10.4.8 (amazon)
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
AIX/32 5.? (copper)
Clean up warnings assigned to me from today's meeting - still need
to tackle the datatype conversion ones...
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
Add src/H5Oalloc.c file forgotten in previous object header checkin
Break out more object header routines into separate file (debugging routines
this time).
Fix "//" style comment in recent ISOHM source changes.
Fix 'size_t' vs. 'unsigned' problem (visible in 64-bit testing) in recent
ISOHM test changes.
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
version and size information into the superblock to eliminate a read when
loading it.
This is a file format change, and hopefully the last one (knock on wood).
Tested on kagiso and Windows (mostly just a SOHM change).
Basic support for H5Literate() routine. Still needs to be fleshed out and
refactored to simplify. Also, needs tests. :-)
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
AIX/32 5.? (copper)
Mac OS X/32 10.4.8 (amazon)
Added new refreg_name.c file to test H5Rget_name and H5Iget_name for region references.
H5Rget_name tests are skipped for now since they do not work (at least for me :-)
Platforms tested:
heping and copper
The version of libtool used by HDF5 isn't directly affected by the reconfigure
script; instead, libtoolize --force must be used by hand. Libtool was the
source of the problem, so rolling its version back to 1.5.14 should solve the
issue (at least temporarily).
Reconfigure should still work on both heping and kagiso.
Tested on heping, kagiso, and tg-login3.
Add support for "delete by index" to "old-style" groups, finishing
implmentation of H5Ldelete_by_idx() routine.
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
Aix/32 5.? (copper)
More progress toward getting H5Ldelete_by_idx() working - removals on
densely stored links should work now (still needs some more tests). Still
need to write code for removals on "old-style" groups.
Tested on:
FreeBSD/32 4.11 (sleipnir)
More progress toward getting H5Ldelete_by_idx to work fully - can now
delete by index in compact group (but not dense groups or "old-style" groups
yet). Need to go write a v2 B-tree routine to delete from the B-tree by index
before the dense storage will work properly...
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Straighten out some convoluted code when links were being deleted, which
could cause the "delete" callback for user-defined links to not get called when
the group they were in was deleted.
Had to compromise on the "delete" callback though - only calls the callback
with the ID for the file the link is in, instead of the group, since the group
is being held open upstream in the calling sequence during a group deletion and
this prevents a group and its ID from being created. (This could possibly be
worked around, but would cause a fair bit of havoc in the code and I'm not
entirely certain it's worth it...)
Tested on:
Linux/32 2.6 (chicago)
This feature is still in progress; Shared Object Header Messages are not
complete as a feature and are not thoroughly tested. There are still
"TODO" comments in the code (comments with the word "JAMES" in them,
so as not to be confused with other TODO comments).
Hopefully this checkin will reduce the liklihood of conflicts as I finish
implementing this feature.
All current tests pass on juniper, copper (parallel), heping, kagiso, and mir.
Add new H5Lget_val_by_idx() routine & tests.
Also includes most of changes for H5Ldelete_by_idx() routine.
Tested on:
Mac OS X/32 10.4.8 (amazon)
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
AIX/32 5.? (copper)
Finished implementation of H5Lget_info_by_idx for all cases: old vs. new
group formats, compact vs. dense new link storage, increasing vs. decreasing
vs. native iteration order.
Also, refactor symbol table "foo by index" routines to be more generic
and share more code by using a single B-tree iteration callback which makes
callbacks to a specific "get <foo>" callback.
Tested on:
Mac OS X/32 10.4.8 (amazon)
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
AIX/32 5.? (copper)
Should disable linking against shared libraries in Fortran for compilers that
don't support shared libraries.
Should also fix problem when the wrong Fortran file extension was specified.
If these changes don't solve the Daily Test issues, I'll look at backing out
the autotool version change until I have time to fix them.
Tested on heping, kagiso, juniper.
Fix retrieving name for references to root group.
Also, move 'ref' test earlier in testing, right after 'getname' test
Tested on:
Linux/64 2.6 (chicago2)
Implement basic framework for H5Lget_info_by_idx and get it working for
creation order indices on compact groups.
Clean up code a bit.
Close resource link in user-defined link traversal.
Tested on:
Linux/32 2.6 (chicago)
Update H5Iget_name() tests to reflect that the name of the object for an
ID can be found in many more situations now.
Tested on:
FreeBSD/32 4.11 (sleipnir)
Refactor generic property list initialization code to put property list
specific routines in property list modules, instead of scattered to the four
winds. Also, introduce property list class initialization objects, to make
adding new property list classes in the library easier.
Fix daily test failure by using H5Pget_elink_prefix() API routine instead
of looking at the "raw" generic property list information.
Tested on:
Mac OS X/32 10.4.8 (amazon)
FreeBSD/32 4.11 (sleipnir) w/threadsafe
Linux/32 2.4 (heping) w/C++ & FORTRAN
Linux/64 2.4 (mir) w/build-all & 1.6 compat
the linkval buffer, per Elena and Frank's suggestions while revising
the documentation. Added error checking using this size, as well as a
couple of tests.
Tested on juniper, kagiso, and sol.
Finish internal work necessary to track creation order in v2 B-tree when
group is in "dense" storage form.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Add support for compress heap blocks in "managed" heaps.
Also, fix bug when the first direct block moves between having a parent
and not having one (and vice versa), which occurs when the heap moves between
having a direct root block and having an indirect root block.
Tested on:
FreeBSD 4.11 (sleipnir) w/threadsafe
Linux/32 2.4 (heping) w/C++ & FORTRAN
Linux/64 2.4 (mir) w/build-all & 1.6 compat
Reduce memory usage for common cases of I/O pipeline filter memory usage.
Also, clean up some more code...
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Finish new version of the I/O pipeline message, which is much smaller than
the previous version. This version is used with the "use the latest version
of the format" flag.
Closed several memory leaks/overruns (found with valgrind).
Also, lots of compiler & formatting cleanups.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Fix several errors in the "latest version" of the object header format
changes and enable the new version when requested now.
Clean up several confusing or duplicated sections of code.
Tested on:
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
FreeBSD/32 4.11 (sleipnir)
through driver info functions in H5FD_direct_open, H5FD_direct_read, and H5FD_direct_write. The
external change is to add a new API function H5Pget_fapl_direct to query 3 control values, the
memory boundary, file system block size, and the maximal buffer size for copying data.
handle flush callbacks which can dirty other entries, and
resize and/or rename the target entry.
This feature is needed by the fractal heap code.
Also added associated test code.
H5Commit tested. Test failed on heping, but the
error appears to be a syntax error in an un-related
file.
Tests on copper & sol passed, along with tests on
phoenix.
Add 'initval' parameter to "lookup3" checksum routine (and implicitly to
the metadata checksum routine), to allow chaining several checksums together
easily (which isn't used by these modules, but will be used in my next checkin)
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
memory boundary, file block size, and maximal copy buffer size; second in H5FD_direct_write and
H5FD_direct_read, the library checks whether data buffer is aligned. If it is, then write and
read the data directly instead of making a copy buffer.
Bug fix (related to 544)
Description:
h5_get_file_size() was coded to return 0 if failed but
file size can be 0. Changed the failure return value to -1
which is allowed by the returned type of off_t which is a signed
type.
Also changed the checking code of the stat call to just == 0
since that is how it is defined.
Test:
Could test it in heping only. Both Sol and Copper failed to
compiled due to error in the Direct IO VFD code.
So new flush2 tests generates false failures for DLL version on windows.
Two cases of flush2 tests were skipped temporarily on windows.
The inserted windows macros should be removed when a better solution is found.
Tested on vs6.0 and vs 7.1 and Linux.
Update datasets and the layout, attributes and fill-value
object header messages to use the latest version
of the file format flag.
Tested on:
FreeBSD 4.11 (sleipnir)
Linux/64 2.4 (mir)
Linux/32 2.4 (heping)
option --enable-direct-vfd/--disable-direct-vfd to enable/disable Direct I/O support. The default
is enabled. There's a small test in test/vfd.c. Another way to test it is to set environment
variable HDF5_DRIVER to "direct" and run "make check" in the test/ directory. There'll be some
further improvement in the following checkin including allowing user to provide memory boundary
value, file block size, and copying buffer size.
Add "use the latest format" support for dataspace object header encode/
decode routines and clean up format a bit for the latest format (new to 1.8.x
releases)
Remove storing 'perm' parameter for array datatypes in memory and the file,
and add test to make certain that if any user applications are attempting to
store them, we get some reports back. (Should be unlikely, since the RefMan
says that the parameter is not implemented and is unsupported).
Carry those changes into the tests, etc.
Clean up a bunch more compiler warnings.
Tested on:
FreeBSD/32 4.11 (sleipnir) w/threadsafe
Linux/32 2.4 (heping) w/FORTRAN & C++
Linux/64 2.4 (mir) w/enable-1.6-compat
Description: Add a new part to the flush test that checks to see what happens in case a file is flushed, and
then a new dataset is created and the program exits without flushing this subsequent dataset.
The test verifies that, at the very least, the data written out before the H5Fflush call is correct.
and quote its arguments. Also checks for the 'socket' library on
Solaris.
If this patch passes the Daily Tests and makes the user happy, I'll
port it back to the 1.6 branch.
Tested on mir and sol.
Finish 'use the latest version of the format' changes to the datatype
object header message. Compound and enumerated types will now be encoded
more efficiently by packing the field names & member offsets better.
Tested on:
Linux 2.6/32 (chicago)
Linux 2.6/64 (chicago2)
Add the "use the latest version of the format" support to datatype
messages. And a regression test to check that it's working.
Also, found that we have been over-allocating space for compound datatype
messages (when they have array datatypes or can use the latest version of the
format) and trimmed the size back.
Clean up datatype & dataspace encode/decode routines by having them
allocate & release "fake" file structures, which gets them out of needing to
"know" about the internals of file structures.
Other minor whitespace/formatting cleanups, etc.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Further minor modifications to the file format for tracking links in groups.
This is tentatively the "final" file format for groups.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Fix file handle destroy routine to not attempt to flush out partially
initialized file handles (when opening a file fails).
Tested on: (until they finished testing, this time... :-)
FreeBSD/32 4.11 (sleipnir) w/threadsafe
Linux/64 2.4 (mir) w/1.6 compat
Linux/32 2.4 (heping) w/FORTRAN & C++
Mac OSX/32 10.4.8 (amazon)
Add test to fractal heaps to exercise issues with opening the same heap
through two different file handles.
Fix issues with file handle contexts in metadata cache callbacks for heap
components.
Fix bug in file close handling where cached information was being
invalidated even when another file handle was open to the file.
Tested on:
FreeBSD/32 4.11 (sleipnir) w/threadsafe
Linux/64 2.4 (mir) w/1.6 compat
Linux/32 2.4 (heping) w/FORTRAN & C++
Mac OSX/32 10.4.8 (amazon)
File format is not stable, don't keep files produced!
Description:
First stage of checkins modifying the format of groups to support creation
order. Implement "dense" storage for links in groups.
Try to clarify some of the symbols for the H5L API.
Add the H5Pset_latest_format() flag for FAPLs, to choose to use the newest
file format options (including "dense" link storage in groups)
Add the H5Pset_track_creation_order() flag for GCPLs, to enable creation
order tracking in groups (although no index on creation order yet).
Remove --enable-group-revision configure flag, as file format issues are
now handled in a backwardly/forwardly compatible way.
Clean up lots of compiler warnings and other minor formatting issues.
Tested on:
FreeBSD/32 4.11 (sleipnir) w/threadsafe
Linux/32 2.4 (heping) w/FORTRAN & C++
Linux/64 2.4 (mir) w/enable-v1.6 compa
Mac OSX/32 10.4.8 (amazon)
AIX 5.3 (copper) w/parallel & FORTRAN
Description:
Add a second test case to the serial flush tests. This tests the case when the file is not flushed out to disk and verifies that it fails as expected.
Platforms:
Linux (heping)
Review, revise & checkin in Peter's latest round of object copy changes,
which add basic support for datasets & attributes with reference datatypes.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Add 'loookup3' checksum routine and switch to using it for metadata
checksums - it's just as "strong" as the CRC32 and about 40% faster in general
(with some compiler optimizations, it's nearly as fast as the fletcher-32
algorithm).
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Some of the tests cannot be run on VMS since they try to open
the same file twice.
Solution:
Bypass the tests according to the H5_CANNOT_OPEN_TWICE variable setting.
Platforms tested:
VMS server and heping.
Add "op" routine to perform operation on heap object "in situ", to allow
for faster operations on dense links during B-tree traversal & lookup.
Refactor the "read" routine to use the internal version of the "op" routine,
to keep the code duplication as low as possible.
Tested on:
Mac OS X.4/PPC (amazon)
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Add a CRC algorithm to the library, initially for "small" (<256 byte)
metadata blocks.
Update checksum tests to verify it's working correctly.
Tested:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
(Will be testing on more platforms after checkin)
Improve density of the B-tree further. For greater depths of B-trees,
the gains are over 100%...
Also, don't split internal nodes with 3->4 splits, use a 1->2 split
instead, so that the density of the nodes around a split is maximized.
Tested:
Mac OS X/PPC 10.4 (amazon)
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Split edge nodes in the tree with a 1->2 node split, instead of a 2->3 node
split, which creates a more dense tree when a pattern of record insertions
occurs (because it leaves behind full nodes instead of 2/3 full nodes).
Tested:
FreeBSD/32 4.11 (sleipnir)
Linux/64 2.4 (mir)
Linux/32 2.4 (heping)
Solaris/64 2.9 (shanti)
Thread safe error test fails due to the changes in the error stack.
Solution:
Updated the expected error stack.
Platforms tested:
heping (too minor, probably will fail on Tuesday anyway)
Refactor the file storage of "twig" nodes in the B-tree to allow them to
store more records, increasing the average density of the B-tree 30-40%.
Increase # of records in "insert lots" regression test to still create
B-tree of depth 4
Update h5debug to interpret difference of 'branch' and 'twig' internal
nodes in B-tree correctly.
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
Solaris/64 2.9 (shanti)
Re-order the fheap & btree2 tests so that the btree2 test runs first,
because the fractal heaps use v2 B-trees for tracking huge objects.
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
Solaris/64 2.9 (shanti)
These errors should be investigated more thoroughly later. The underlying
problem in links.c seems to be that files opened multiple times don't share
the same H5F_shared_t struct. Perhaps identifying when this is the case
would be helpful?
Tested on mir.
Add support for checksumming fractal heaps. This is always enabled for
the heap header and indirect blocks (as they are "pure" metadata) and is
optional for direct blocks, since they may be used for "raw" data.
Also, rearrange direct block routines in H5HFcache.c to be in a more
sensible location in the file. (probably should have been a separate checkin,
since the diffs are mostly useless for this checkin... *sigh*)
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Since these examples need to follow filesystem paths, the Makefiles need
to create directories in the examples directory; added this to the
Makefile.am.
Tested on Windows, mir, juniper
Tweak the library's new faster fletcher32 algorithm to always produce the
same checksum as the previous fletcher32 code in the fletcher32 I/O pipeline
filter and switch the filter to use the library's version of the algorithm.
Tested on:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Too minor to require h5committest
Break out a bunch of the misc. routines that were in src/H5.c into more
specific modules.
Add optimized fletcher32 checksum routine, for checksumming metadata as
well as raw data.
Tested On:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago2)
Will test further after checkin...
Add a bunch more regression tests:
- Verify opening a fractal heap more than once works correctly
- Check that 0 & 1-sized object insertions are handled correctly
- Check that bad heap IDs are detected when attempting to read an
object
- Check that doubling tables with different parameter values are
working correctly
Add some more checks to the code to detect 0-sized object insertions and
bad heap IDs for reading.
Tested On:
FreeBSD/32 4.11 (sleipnir)
Linux/32 2.4 (heping)
Linux/64 2.4 (mir)
Solaris/64 2.9 (shanti)
- Migrate "direct block location" routine from H5HFman.c to H5HFdblock.c,
which is a more appropriate location
- Optimize performance of heap code by taking advantage of pinned
indirect blocks and use them without putting a metadata cache
protect/unprotect pair around them.
- Other minor compiler warning cleanups and optimizations...
Tested On:
FreeBSD/32 4.11 (sleipnir)
Linux/64 2.4 (mir)
Solaris/64 2.9 (shanti)
Object header locations (H5O_loc_t's) can now "hold open" a file and
decrement its open object count when they close. This means that
locations (H5O_loc_t's and H5G_loc_t's) should always be freed.
Added more thorough tests to ensure that external files are closed.
flag, and pin an entry as it is inserted.
The objective is to avoid some function call overhead in fheap.
Also added matching test code in test/cache.c & test/cache_common.c
(also testpar/t_cache.c checked in separately by accident)
h5commit tested
Several changes, all mooshed together:
- Add support for "tiny" objects - which can be stored in the heap
ID itself, instead of in the heap data blocks.
- Flesh out support for compressed direct blocks, but comment it
out until John's got some metadata cache changes in place to
support it.
- Add support for applying I/O pipeline filters to 'huge' objects
- Refactor 'huge' object code to store information for 'huge' objects
directly in the heap ID, when there are I/O pipeline filters
applied to the heap (and the heap ID is large enough to hold the
information)
- Update h5debug tool to correctly handle 'huge' & 'tiny' objects.
- Misc. other code cleanups, etc.
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/64 2.4 (mir)
Solaris/64 2.9 (shanti)
Allow the heap ID length to be chosen at heap creation time, to allow
for making heap IDs long enough to directly embed the file offset & length
of 'huge' objects in the heap ID (which allows them to be retrieved directly
from disk, instead of requiring them to be looked up in the B-tree that tracks
'huge' objects)
Tested on:
FreeBSD/32 4.11 (sleipnir)
Linux/64 2.4 (mir)
Solaris/64 9 (shanti)
Clean up a couple of compiler warnings in the 'huge' object code in the
library.
Add a bunch of 'huge' object regression tests
Tested on:
FreeBSD 4.11 (sleipnir)
Linux/64 2.4 (mir)
Solaris/64 2.9 (shanti)
Move metadata cache address rename for fractal heap indirect block
immediately after reallocing the storage on disk - to prevent another
routine in the library from confusing the cache by allocating more storage
and attempting to insert it in metadata cache at the old block's location.
Gotta like those monte carlo/random insert & delete tests, which exposed
this... :-)
Tested:
Solaris 2.9 (shanti)
Not failing on other machines
Initial revision to add support for "huge" objects in the heap (which are
actually stored directly in the file, but are tracked with v2 B-tree that is
accessed through heap header).
Testing:
FreeBSD 4.11 (sleipnir)
Linux/64 2.4 (mir)
Linux/32 2.4 (heping)
Mac OS X.4 (amazon)
Update code formatting a little and refactor to add a better mechanism
for performing callbacks when removing records from the B-tree or deleting
entire B-tree.
Testing:
FreeBSD 4.11 (sleipnir)
Linux/64 2.4 (mir)
Linux/32 2.4 (heping)
Mac OS 10.4 (amazon)
Daily test for ttsafe failed on several machines due to the changed error stack.
Solution:
Updated the source code to use new error stack.
Platforms tested:
shanti
More tweaks on fractal heap statistics, to better separate information
about managed objects from information about "huge" objects.
Also, clean up some compiler warnings, etc. on 64-bit platforms & Windows.
Platforms tested:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago)
Refactor fields in heap header to clean out information from dropped
features and also group related fields together more nicely.
Add fields to heap header for storing information about "huge" objects.
Platforms tested:
Linux/32 2.6 (chicago)
Linux/64 2.6 (chicago)
Refactor fractal heap IDs to include "flag byte" as part of the ID. This
byte will be used for the heap ID format version as well as flags to indicate
whether the heap object is a "tiny"/normal/"huge" object (with storage
mechanisms optimized for each type of object).
Platforms tested:
Linux/32 2.4 (chicago)
Too minor to require h5committest
Clean up compiler warnings/failures in test/links.c, especially when
--disable-production flag used with --enable-group-revision
Modify binary dumping in h5dump to clean up files created [a band-aid
solution to not actually creating the files in the srcdir, but better than
just leaving the files around... :-/ ]
Tested:
FreeBSD 4.11 (sleipnir) (w/ configure flags above)
Too minor to require h5committest
Users can create external links using H5L_create_external(). These links
point to an object in another HDF5 file. Users can alter the behavior of
external links or create new kinds of links by registering callbacks
using the H5L interface.
Added tests, tools support, etc.
Also a number of other, minor changes have been made (some restructuring of
the H5L interface, for instance).
Additional documentation and examples are forthcoming.
ttsafe test failed on the tts_error call. The error was due to the outdated
expected error stack; fixed.
Test was also moved to use new error APIs.
Platforms tested: Linux 2.4 (heping) and Solaris 2.9 (shanti)
"make check-vfd" will now run all tests in the test directory with different
file drivers (at least, all of those tests that use the testing framework's
FAPL). Tests that fail will be skipped.
This is not a perfect fix, but is better than nothing.
Along with this change, check-vfd should be added to the Daily Tests.
Fix the last scattered bunch of problems with the object deletion code,
which appears to be completely working now (for objects that are stored
within heap blocks - standalone objects aren't implemented yet).
Also, re-work the regression test to speed up some of the existing tests
and add in 100-200 more combinations of tests - overall, its probably even
slower than it was... :-/
Tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (chicago)
Mac OS X (amazon)
Checkpoint further work on object deletion code for fractal heaps. I
believe everything is finally working for this sub-task. More regression tests
soon, but I don't foresee any problems.
Machines:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (chicago)
Refactored object deletion code for fractal heap to fix scaling problems.
Passing all previous tests again now.
Tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (chicago)
Mac OS X (amazon)
1) Added trace file support to the metadata cache. This allows capture
of all metadata cache calls in trace files for purposes of optimization
and debuging.
2) Added an expunge entry function. This allows an entry to be deleteded
from the cache without writing it to disk even if it is dirty.
3) Added a function call to resize pinned entries.
4) Added code to deal with entries that are dirty on load. This is
needed in support of a bug fix which can alter object headers on
load to repair files.
5) Added progress reporting code to the "MDC API smoke check" test in
cache_api.c. To enable the progress reporting, set report_progress
to TRUE in mdc_api_call_smoke_check().
Tested with h5committest, and a parallel test on phoenix (dual athelon
linux box).
adjustment
Description:
These tests (fheap, btree2) are under development and are not used by
the library yet. Move them to the end so that their failure do not block
other current library code tests.
Platforms tested:
heping.
Feature
Description:
Revised Link APIs.
Solution:
New link APIs use H5L*
H5*create_expand do not create links to the objects created; this must
be done manually with H5Llink.
Added APIs to link an object given its ID (H5Llink), to copy links (H5Lcopy),
and changed creation APIs (H5Lcreate_hard and H5Lcreate_soft) and query
API (H5Lget_linkinfo instead of H5Gget_objinfo).
All old APIs are still supported in H5Gdeprec.c .
Platforms tested:
sol, mir, copper
Misc. update:
Forgot to update MANIFEST and release docs. Will do after checkin.
Code checkpoint
Description:
Revise fractal heap code to support deletions in up to 2nd level deep
indirect blocks. Needs some revisions to support deeper indirect blocks...
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (chicago)
h5committest
Added/adjusted Test Express mode.
Description:
btree2:
Added TestExpress mode to skip some long tests (the 5th recursive and
the two random tests.)
fheap:
Changed the express mode messages to be more informative.
Un-excluded some tests from express mode.
Platforms tested:
Red Storm.
Code cleanup
Description:
Trim trailing whitespace in Makefile.am and C/C++ source files to make
diffing changes easier.
Platforms tested:
None necessary, whitespace only change
New feature
Description:
Check in Peter's code to add support for "shallow copy", "create
intermediate groups", "no attributes" and "expand soft links" support.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (chicago) w/ & w/o group-revision enabled
h5committest
Description: Mac OS 10.4 on PowerPC chip has some errors to convert (unsigned)
long long to long double. When the bit sequences are 0x003ff..., 0x007fff...,
0x00ffff..., 0x01ffff..., 0x7fffff..., the converted values are twice as big
as they should be.
Solution: Detect the error in configure and disable the compiler conversion
and test case.
Platforms tested: h5committest, Mac OS 10.4, and fuss.
Bug fix
Description:
Since the new default Express level is 1, some fheap tests were being skipped.
Solution:
At least temporarily, fheap now skips these tests if it is run at express level 2
or greater.
Platforms tested:
mir
Configuration feature
Description:
'make install' now tests both static and shared libraries if both are installed.
Solution:
Previously, shared libraries were only tested when static libraries were not installed.
Also cleaned up line in commence.am that was including HL library in all Makefiles.
Platforms tested:
mir (Makefile change only)
testing feature
Description:
HDF5 Test Express change; now accepts values 0 to 3.
Solution:
As explained in the RFC and documented in the code, there are now four
levels of "express-ness", 0 (no limit) to 3 (1 min limit).
It is now up to test writers to take advantage of the various levels
to tune their tests' run times.
Platforms tested:
mir (minor change)
Description: Data conversion from long double to (unsigned) long long
returns some incorrect values on Mac OS 10.4 and SGI IRIX64 6.5. The
conversions start to go wrong when the long double is
20041683600089727.779961 (0x4351ccf385ebc8a0bfcc2a...). If adjusting
the values higher by assigning 0x...c8a0cf... or 0x...c8a0df..., the
converted values go wild.
Solution: Detect this error in configure.in and disable compiler
conversions from long double to (unsigned) long long for Mac OS 10.4
and IRIX64 6.5.
Platforms tested: h5committest, Mac OS 10.4, and IRIX64 6.5.
Code checkpoint
Description:
Add in more new features for the fractal heap code, mostly bringing in
more ability for deleting objects (which isn't completely working yet).
Also, checkpoint free space manager code, which is essentially complete
(although it needs some more work after the metadata cache has some additional
features)
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (chicago)
h5committest
Bug fix
Description:
Fix bug which could lead to files with incorrect count of messages in the
header for an object. The exact sequence of operations is complicated and
is described in the release notes.
Solution:
Mark merged null header messages as dirty.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (chicago)
Mac OS/X (amazon)
Bug fix - bugzilla #552
Description:
On Cray X1, trying to use : as an argument confused the system.
Solution:
Added a test in configure to see if : as an argument is bad.
If so, skipped the test.
Platforms tested:
mir, Cray X1 (change to configure only)
Feature
Description:
Sometimes the parallel prefix is given in the form of nfs:/mnt/pfs which
if given to a non-MPIO VFD like the default H5Fcreate, it would fail.
Added a new routine h5_rmprefix which would return the non-prefix component
part of the file name which would be okay for the default H5Fcreate and
such.
Platforms tested:
Tested in heping parallel.
Added Exress Test mode.
Description:
The test takes a long time to run. It tempted developers to skip the
h5committest.
Solution:
Added Express Test mode to allow tester an option to run it in express
mode (shorter time.) There are already encoded QAK macro which skips
many tests. Follow that pattern to skip tests when HDF5TestExpress is
set.
Platforms tested:
Tested in heping, both serial and parallel.
1) Check in potential fix to unreproduceable bug in t_cache observed on
Cobalt
2) Check in code supporting added pinned entry features in the metadata
cache.
Description:
1) Elena encountered a bug in t_cache when running on Cobalt. On
investigation I found a potential race condition in t_cache that
could explain her observation.
2) Quincey requested additions to the metadata cache allowing renaming
of pinned entryies, and marking of pinned or protected entries as
dirty.
Solution:
1) Modified t_cache.c to prevent the potential race condition. Elena
was unable to reproduce the bug on Cobalt, so we don't know if my
fix actually addressed the problem.
2) Added code supporting the requested functions. The changes were
relatively trivial, but required substantial new test code.
Platforms tested:
h5committest, serial test on heping, parallel test on phoenix.
Misc. update:
Code checkpoint
Description:
Update fractal heap code to allow objects to be deleted
Fix bugs in fractal heap object insertion
Improve free space manager code to allow sections to be merged and
shrink the container
Another try at making the Windows compilers happy...
Platforms tested:
FreeBSD 4.11 (sleipnir)
Mac OS X.4/PPC (amazon)
Linux 2.4 (chicago & heping)
Solaris 2.9 (shanti)
AIX 5.? (copper) w/parallel
Bug fix.
Description:
The ${TR}, though avoid the error, is inconvenient. It is needed
because configure.in use the char range in the style of 'a-z'.
The other style of '[a-z]' is more commonly accepted by all tr
but autoconf tends to strip away [], making the syntax rather
clumsy.
Solution:
Learned from autoconf that it avoids the use of character range
by just spell all the letters out. Changed our tr commands to
use those variables defined by autoconf. Also removed the
definition of ${TR} since it will cause inconsistancy when
autoconf also use plain 'tr' in its generated code.
The Makefile.in are changed because the elimination of ${TR} from
configure triggered its removal from all Makefil.in. That is okay
because ${TR} is not used at all in Makefile.
Platforms tested:
h5committested. (sol failed to connected). also tested in shanti
using both /usr/ucb/tr and /bin/tr (the bad one before.)
Misc. update:
Updated both INSTALL and RELEASE files.
Code checkpoint
Description:
Checkpoint fractal heap improvements, as well as move the free space
manager code that it's using into a separate package.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4/64 (mir) w/C++ & FORTRAN
Linux 2.4/32 (heping)
Solaris 2.9 (shanti)
AIX 5.? (copper) w/FORTRAN & parallel
Description: Function prototype H5E_walk_t and structure H5E_error_t wasn't backward
compatible.
Solution: Make them compatible with v1.6 and provide new definitions of H5E_walk_stack_t
and H5E_error_stack_t.
Platforms tested: fuss and h5committest.
Purpose: Maintenance
Description: dtypes.c compilation failed with gcc 4.1 due to the
missing prototype of the opaque_funcs function.
Failure was reported by a user who tried to build
hdf5-1.8.0-alpha2.
I reproduced the failure on sleipnir with gcc41
compiler.
Solution: Added prototype definition.
Platforms tested: sleipnir with gcc41, heping and copper
Misc. update:
Bug fix
Description:
A datatype wasn't being closed in th5s.c
Solution:
This minor oversight caused an error with my Object Header code.
The datatype is now closed in the test.
Platforms tested:
windows, mir, copper
Bug fix
Description:
Fix two test bugs:
1) Unitialized variable bug in test of H5C_get_entry_status()
(in test/cache.c)
2) Incorrect initialization of constant in t_cache when running
with MPE. (in testpar/t_cache.c)
Solution:
1) Don't test uninitialize variable.
2) Correct initialzation of constant in MPE case.
Platforms tested:
Tested on Heping and Copper.
Misc. update:
Code checkpoint
Description:
More progress on fractal heap, can now re-open an existing heap, although
the free space algorithm still needs work.
Also, use the new "pinned entry" metadata cache code.
Platforms tested:
FreeBSD 4.11 (sleipnir)
Linux 2.4 (heping)
Solaris 9 (shanti)
Linux 2.4/64 (mir)
Code cleanup/feature twist
Description:
Adjust recent H5AC routines to take H5F_t pointers instead of H5C_t
pointers, to match the rest of the H5AC routines.
This change propagated into a few of the tests, which also had some
compiler warnings cleaned up...
Platforms tested:
FreeBSD 4.11 (sleipnir) w/parallel
Linux 2.4/64 (mir) w/C++ & FORTRAN
Add pinned entry capability to cache.
Description:
For frequently accessed cache entries, the protect/unprotect overhead
is sometimes a bottleneck.
Solution:
Allow entries to be pinned in the cache. Pinned entries can't be
evicted, but can be flushed or modified.
Platforms tested:
h5committested -- minus one small typo in test/cache.c whose fix was
tested on copper and heping only.
Misc. update:
Several bug fixes
Description:
Added config file for Cray X1 (previous file was misnamed)
Simplified some code in hyperslab.c test that seemed to have been
confusing compiler on Cray X1.
Fixed typo in config/commence.am
Cleaned up hl/src/Makefile.am
Solution:
All four fixes should be straightforward. The failure on Cray was
very difficult to debug, but involved arithmetic errors.
This change seems to fix it.
Platforms tested:
heping, copper, sol, some Cray X1 (more testing when system comes back up)
