Fixed CHICAGO: Bug 2121 - h5diff - incorrect and lack of output for the
different set of attributes (different number and names)
Description:
Previously h5diff compared attributes correctly only when two objects have
the same number of attributes and attribute names are identical.
This fix covers all other cases.
Also didn't display useful information about attribute difference.
This fixes both issues.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Improve the previous fix for Bug 2216 - GMQS: h5diff - memory leak when
compares vlen string in dataset or attributes
Description:
Improve the fix along with the previous checkin r20266.
Add a new function to tool lib, h5tools_detect_vlen_data() which return
TRUE if include any kind of vlen data all at once, either VLEN-data or
VLEN-string and so on.
Also updated h5ls and h5dump code accordingly.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Cmake - jam
Fixed Bug 1904 - h5ls : usage of '-a' option is not clear to display expected output
Description:
Fixed incorrect content and improved help page for -a option of h5ls.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), Cmake (jam)
Fix Bug 2120 - h5copy: improve to copy an object into same HDF file
Description:
Before the fix users can't copy an object (dataset or group) with
different name if input file (-i) and output file (-o) is same.
This feature would be useful to clone any object with different name and
reuse contents.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Cmake (jam)
Fixing Bug 2161 - GMQS: h5dump - only on Windows, skip displaying a data value every a certain lines in array type dataset
Description:
Fixed h5dump for skipping some values for long array type dataset on
Windows. This issue only occurred on Windows due to the different
return behavior from _vsnprintf() funtion.
Tested:
Windows, jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Fixing Bug 2092 - h5dump does not display index for a dataset
Description:
h5dump skip displaying array indices every certain number when the
array type dataset is relatively big. The certain number varies
according to the size of each array.
This checkin fix the skipping array indices problem. This fix also
correct indentation of the dataset data output.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Description:
When using the scale-offset filter with floating point data or fill values, big
endian machines would save some metadata in the wrong byte order. This caused
such datasets to yield incorrect data when read on little endian machines.
Fixed the scale-offset filter to always save this metadata in the right byte order (i.e. little endian).
Tested: jam, amani, heiwa (h5committedt); fedora, linew
Bug 2089 - GMQS: h5diff segfault on a compound dataset with fixed length +
vlen string type order
Description:
This is fix for the segfault when h5diff compares a compound dataset with
combination of fixed length string types and vlen string types in certain
orders. Optimized vlariable length string handling codes. The fix is
referred from h5dump handling vlen strings.
For testing, several compound datasets were added with various combinations.
Previous failed cases:
- Vlen string, Fixed length string, Vlen string, Fixed length string
- Fixed length string, Fixed length string, Vlen string, Vlen string
- Fixed length string, Vlen string, Fixed length string, Vlen string
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Add "--enable-unsupported" configure flag.
Description
The "--enable-unsupported" configure flag allows a user to
prevent configure from failing due to the use of incompatible
options, such as c++ with parallel.
Specifying --enable-unsupported will bypass all of configure's
checks for incompatible and unsupported combinations of flags.
There are no guarantees that the library will be configured
in any sort of working condition, but that's the risk of
using the --enable-unsupported flag.
I've changed all default error messages related to unsupported
option combinations to indicate that using --enable-unsupported
will allow configure to complete without error.
Tested:
by hand on jam, tested all unsupported configure option
combinations with and without the new flag, making sure
the flag allows configure to finish without error.
(h5committest wouldn't do any good here; it won't test
the new option, and since we're enabling unsupported
combinations, failures are likely to occur in build or
tests with --enable-unsupported turned on anyways.
That's why they're unsupported!)
Add additional error checking to catch erroneous user input.
Description:
Attempting to retrieve a links's name by index in the case
where the link is external and the file that the object is
located in doesn't exist was causing a segmentation fault
(in production) and an assertion failure (in debug).
The segfault wasn't occuring until the metadata accumulator
attempted a write, so I've added error checking higher in
the pipeline in H5O_protect (where there was previously just
an assert) to catch this. I've also added additional asserts
in the H5F layer where there were none.
Additionally, I added another case to the links.c test to
test that this fails gracefully instead of segfaulting or
asserting out.
Tested:
h5committest and gandalf (mac os x)
Improve h5diff performance.
Description:
The following changes for improving h5diff performance:
1) use HDmemcmp() before comparing each elements
(memcmp() is very fast at both linew and jam)
2) replace the expensive H5Tequals() calls
3) retrieve datatype information at dataset level not each element level
for compound datasets
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Fixed Bug# 1979 Output from h5ls -rdlS on nested compound datatypes is
difficult to parse.
Description:
Update to add curly brackets for the nested compound members, when
S (--simple) option is used with -l (--label), so user can tell
which members blong to which compound type.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32)
Fixed h5diff to handle variable-length strings in a compound dataset
correctly. Also variable-length string array in a compound dataset.
Bug #1989.
Description:
Garbage values were displayed when h5diff compared variable-length
strings (or string array) in a compound type dataset.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (Mac32)
Fix for bug# 2040 - h5copy should fail gracefully for expected failure copying to non-exist nested group without -p option.
Description:
Fixed h5copy to fail gracefully when copying object to non-exist group without -p option. This is expected to be failed.
Tested:
jam, amani, heiwa
Backout the previous changes (r19598):
(Fix for bug# 2040 - h5copy should fail gracefully for expected failure copying to non-exist nested group without -p option.)
Description:
Some failure occurred on talwit, so backout the changes and will put it back
after the fix.
(Fixed h5copy to fail gracefully when copying object to non-exist group without -p option. This is expected to be failed.
Merged from hdf5 trunk r19598.)
Fix for bug# 2040 - h5copy should fail gracefully for expected failure copying to non-exist nested group without -p option.
Description:
Fixed h5copy to fail gracefully when copying object to non-exist group without -p option. Expected to be failed.
Tested:
jam, amani, heiwa
unknown versions of the C Intel compilers.
Modified apple config file to display Intel compiler information.
Updated RELEASE.txt notes accordingly.
Platfroms tested: jam, amani and dubna (Mac 64-bit) with Intel C, Fortran and C++ compilers.
Bug 1707 is that H5Eset_auto causes a seg fault when an application uses -DH5_USE_16_API with
the 1.8 library to compile.
I created a branch off the trunk called set_auto to continue working on the problem.
Tested on jam - backing out, simple.
Description:
Library versions 1.6.3 and earlier contain a bug which causes them to be unable
to perform certain operations on a group if that group's symbol table
information is not cached in the parent group's symbol table. Versions 1.8.0
to 1.8.5 did not cache this information. Modified library to cache this
information.
Tested: jam, amani, heiwa (h5committest)
through H5Eset_auto. I changed the union in the structure H5E_auto_t. Another change is to
make H5Eget_auto fail if H5Eset_auto is called to set the printing function. I'll write a
document for it.
Tested on heiwa, jam, and amani.
The property change in configure.in, config, and Makefile.am came from the merge of the 1.8
library change.
Add --exclude-path option
Description:
Specified path to an object will be excluded from comparing the two files or two groups. If group is specified all the member objects will be excluded.
Related to "1890: h5diff excluding object for file comparison via command line"
Tested:
jam, amani and heiwa
Fix for Bug1975 h5diff - support recursive comparison on group when specified as an object
Description:
Compare member objects and groups recursively when two files or groups are specified to be compared. Support parallel diff and handling symbolic links accordingly.
Tested:
jam, amani, heiwa
Fix for Bug1896 h5repack - changing layout to COMPACT does not work
Description:
Make h5repack be able to convert a layout to COMPACT for small size dataset as default. Also add verifying layout changes in our test script.
Tested:
jam, amani, heiwa
Description:
The fractal heap header structure keeps a pointer to the file associated with
it. However, it is possible for that file pointer to be closed while the
header is still in cache (through the shared file pointer). Previously, the
header's file pointer was not updated and subsequently pointed to an invalid
file structure. Modified fractal heap code to update the file pointer every
time the header is accessed.
Tested: jam, linew, amani (h5committest)
Add windows threads support to HDF5.
Description:
Added calls to the windows threads library to the H5TS layer, and wrapped
most calls to either pthreads or windows threads library with portable
H5TS-style defines. Modified tests to use portable function definitions
as well.
This can be configured via CMake with the HDF5_ENABLE_THREADSAFE
option, and should work on windows vista and later operating systems.
Tested:
h5committest, plus threadsafe with pthreads on jam and amani,
and tested on a Windows Vista VM with threadsafe using windows threads.
H5T_ORDER_MIXED specifically for compound types and its derived types. They report mixed
orders if they have members of different orders.
There is no change to configure.in, config, and Makefile.am in the top directory. They have
some property changes when I merged the code from 1.8.
Tested on jam. I tested the same change for 1.8 with h5committest.
1. For enum type, members shouldn't be defined yet.
2. H5T_ORDER_NONE only works for reference and fixed-length
string.
3. For opaque type, the order will be ignored.
4. For compound type, all restrictions above apply to the
members.
I'll change H5Tget_order and do another commit.
There is no change to configure.in, config, and Makefile.am. There is some property change for
these files when I did a merge from 1.8.
Tested on jam. But I tested the 1.8 on heiwa, and amani.
herr_t. To minimize the change of the library's behavior, in the function
H5Z_prelude_callback of H5Z.c, if the return value of can_apply is FALSE and
the filter is MANDATE, this function returns a FAILURE. If the return value is FALSE
but the filter is OPTIONAL, this function returns a SUCCEED. During the IO, the filter
will fail and return a size of zero. But the pipeline will skip this filter.
Tested the same change for 1.8 on jam, linew, and amani. Tested on jam with szip.
a part of the h4h5tools distribution long time ago, but the INSTALL file
was not updated in the development branch and slipped into 1.8 releases and current trunk.
Description:
In certain circumstances, the direct I/O driver did not perform correctly when
data was unaligned. The driver has been patched to fix this. Also added some
potential performance improvements for the unaligned case, and strengthened the
test for whether the data needs to be aligned.
Tested: cobalt
Description:
A bug introduced in 1.8.5 causes local heap data blocks to be mis-aligned when
sizeof_offsets + 2*sizeof_lengths is not a multiple of 8. In this case, the
address of the data block as stored in the heap prefix is aligned but the actual
data block is not. This causes files created with these sizes to be corrupted,
and prevents uncorrupted files with these sizes to be unreadable. Modified
local heap code to account for alignment.
Tested: jam, amani, linew (h5committest)
Add --no-dangling-links option to h5ls.
Description:
Related to "Bug 1830 - Following an dangling external link in h5ls should set non-zero return code."
If --no-dangling-links option is specified and any dangling link is found, return exit code 1 (error).
Tested:
jam, amani and heiwa
Change h5ls not to manipulate special characters in object name or attribute name for smart display.
Description:
Related to "But1784 h5ls has input and output issues with links that have '\' characters in them."
Tested:
jam, amani and linew
Description:
Previously, the chunk cache would, when looking for a chunk, alwaylook for the
chunk in the b-tree before checking the cache. Reworked the functions that
look up chunks to always check the cache first and avoid the b-tree lookup if
the chunk is in cache.
Tested: jam, linew, amani (h5committest)
PHDF5 changed to use "mpiexec", instead of mpirun, as the default MPI
applications startup command as defined in the MPI-2 definition, section
4.1. Note that only mpich related (using mpicc command) definitions are
changed to mpiexec. Other favors like hcc, AIX, champion, are not changed
since I don't have a way to verify them yet.
Tested:
jam and amani, using both current and new mpich.
Did not h5committested since this affects PHDF5 configure only.
Support follow symbolic links.
Description:
Add '--follow-symlinks' option to follow symbolic links (soft and external).
Update help page according to RM.
Remove some warning messages from compiler.
Tested:
jam, amani and linew
Fixed a bug (1855) in H5DSattach_scale, H5DSis_attached and H5DSdetach_scale
caused by using H5Tget_native_type function to determine the native
type for reading REFERENCE_LIST attribute. The bug was exposed on Mac PPC.
Platforms tested: juniper (Mac PPC), jam, amani, linew
Rename '--follow-links' to '--follow-symlinks'
Description:
The '--follow-links' option is to follow symbolic links (soft and external).
Make the name more intuitive and specific to the feature.
Tested:
jam, amani and linew
Description:
Previously, it was possible for a chunk to be flushed due to chunk operations in
the callback from H5B_iterate in H5D_chunk_prune_by_extent. Because flushing
the chunk can force it to be reallocated if it is filtered, this can change the
contents of the chunk b-tree in the middle of H5B_iterate. Because H5B_iterate
uses a locally cached copy of the b-tree, this causes subsequent operations
to be passed incorrect data. Rewrote H5D_chunk_prune_by_extent to avoid
H5B_iterate entirely.
Also fixed a bug in the dataset expand algorithm that could cause extra chunks
to be created.
Tested: jam, linew, smirom (h5committest)
Fixed a bug in the H5DSdetach_scale function when 0 bytes
were allocated after the last reference to a dim. scale
was removed from the list of references in a VL element of the
DIMENSION_LIST attribute; modified the function to comply
with the Spec: DIMENSION_LIST attribute is deleted now when no
dimension scales left attached.
Platforms tested: jam with electric fence, amani amd linew
Add bug fixes for Tools.
Description:
Added list:
1726 - NPOESS: h5repack loses attributes for datasets of type H5T_REFERENCE
1814 - NPOESS: h5repack doesn't handle references to the groups as an
element of a dataset
1817 - h5copy fail to copy dangling link by specifying link path directly
1793 - h5ls on a non-existent file gives 0 return code
Tested:
no, just text update.
Fix memory leaks
Description
Added a routine to free memory which addresses a memory leak
when variable length strings are used as fill values.
Tested:
h5committest and valgrind (on jam/amani) to confirm freed memory.
Added notes about
+ PropList::PropList(id) pclass id/plist id
+ new DataSet::getInMemDataSize
+ CommonFG::getLinkval and CommonFG::getComment take default values
+ bugzilla 1061
+ read/write methods of DataSet and Attribute
Improve configure's large-file support control.
Description:
Modified configure to now attempt to add defines necessary for
supporting largefiles on all systems, instead of solely on linux. This
is in response to user requests to enable largefile support on Solaris
by default, as well as to give extra control on AIX (instead of just
jamming the necessary flag into the config files).
The old --enable-linux-lfs flag has been removed in favor of the
--enable-largefile flag (enabled by default), which can be used on all
platforms.
On systems where large files cannot be supported in this manner,
configure will report as such.
Tested:
h5committest
AIX (NCSA's blue_print machine)
duty, liberty, and linew.
Autotool Upgrade / Libtool Bug Fix
Description:
Updated autotools referenced in bin/reconfigure as follows:
Autoconf 2.64 --> Autoconf 2.65
Automake 1.11 --> Automake 1.11.1
Libtool 2.2.6a --> Libtool 2.2.6b-mcg
The referenced libtool version is a custom version of 2.2.6b. It
has been tweaked to fix a bug in libtool that occurs
when using PGI 10.0 compilers. A check incorrectly categorizes
the C++ compiler as version 1.0 instead of 10.0, and the link
line is subsequently set up incorrectly and fails to compile.
A patch has been made available and will be included in the next
release of libtool, but in the meantime I've applied the patch to a custom
installation as indicated above. This bin/reconfigure now references
the custom installation, and the resulting configure script will correctly
categorize the PGI 10.0 C++ compiler.
Ran bin/reconfigure to regenerate configure and makefiles.
Tested:
jam and amani with PGI 10.0 compilers.
h5committest
Description:
Attempting to copy an object with NULL references (all bytes zero) with the
H5O_COPY_EXPAND_REFERENCE_FLAG flag set would cause a failure or an assertion
(depending on whether it was in debug mode). Changed copy routine to detect
NULL references (object and region) and avoid attempting to expand the reference
in this case.
Tested: jam, linew, amani (h5committest)
Description:
Modified algorithm for extending a dataset with early allocation so it only
deals with the new chunks. Formerly, it would loop over all chunks, checking
to see if each existed in cache and on disk, causing major performance issues
with large numbers of chunks.
Tested: jam, linew, amani (h5committest)
Description:
The core file driver previously matched files only based on the name used to
open them, even when a backing store is used. This caused problems in the
failing test when it traversed an external link to itself, and did not recognize
that it was the same file. Similar tests in links.c were skipped when the core
file driver was used. The failure was previously masked by inappropriate file
flushes caused by external link traversal. Added code to the core file driver
to keep track of low level file information of the backings store to allow for
accurate recognition of duplicate files. Also re-enabled all links.c tests for
the core file driver.
Tested: Fedora (core and sec2 drivers; no time for full commit test, will
monitor daily tests)
Description:
Support for expanding external links was not implemented in H5Ocopy, even though
a flag existed for H5Pset_copy_object to enable this. This patch implements
this feature.
Tested: jam, amani, linew (h5committest); Fedora
Description:
Previously, the library would reopen the source file when traversing an external
link if unable to find the target file otherwise. This has been corrected.
Also moved the call to H5F_flush from H5F_try_close to H5F_dest, so the file is
only flushed when the last identifier for the file is closed. This prevernts
situations where the library could attempt to flush a file with protected
metadata.
Tested: jam, amani, linew (h5committest); Fedora
bugzilla 1754: h5diff: support comparing through links.
Description:
Fix incorrect (or hanging) behavior in parallel mode when specifying
invalid combination of command options.
(ex: -v and -q , --no-dangling-links without --follow-links)
Add relate test case
Update h5diffgentest.c due to add test files this and previous time
Note:
svn #18266 (prior to this)
svn #18164 (original check-in)
Tested:
h5committest (jam, amani and linew)
bugzilla 1754: h5diff: support comparing through links.
(original check-in svn revision #18164)
Description:
fix the hang issue in parallel mode when compare external-link.
add --no-dangling-links option.
add test cases (#450-#459) relate to the new option.
improve test script to check exit code.
update --help relate to the new options.
correct some indentations.
Tested:
h5committest (jam, amani and linew)
Description:
In certain cases, removal of an object in a v1 b-tree would cause the leftmost
key in the right neighbor to be overwritten. While this did not pose a problem
for group b-trees, with chunked datasets it would overwrite the offset value
of the neighbor's leftmost child, causing corruption. Reworked the code to
differentiate between b-trees whose children are fundamentally associated with
their left key and those who are associated with their right key.
Tested: jam, linew, amani (h5committest)
Fix Bug1672 - Display garbage value on LE machine for BE data.
Description:
Casuing by calling diff_dataset twice when -r or no option was given.
Change to call it once which fix the problem.
It also improves the performance. (diffing time in half)
According further test, it also occurred on BE machine as well and it
seems occruing only with the BE data attached to Bugzilla.
Don't know how the file was created. Anyway this fix will prevent
from diffing with garbage values in similiar potential case.
Tested on:
linux32 (jam) , linux64 (almani), solaris (linew)
Description:
H5HG_load made improper assumptions about the ordering of object indices, namely
that they are in order. Not only is this not guaranteed by the file format
spec, but this condition can be violated if id's "wrap around" which can happen
when overwriting VL data. H5HG_load has been fixed to handle any order of
indices. Also fixed some other bugs involving allocation of global heaps in
memory.
Tested: jam, linew, amani (h5committest)
Description:
The H5TB API makes some improper assumptions about the order of compound
datatype members. Namely, it assumes that members remain in the order in which
they were inserted. Unfortunately, this assumption is inherent in the design of
the interface. The library has been patched so that this assumption holds in
situations relevant to H5TB.
Tested: jam, linew, amani (h5committest)
CFLAGS overhaul
Description:
Modified the way configure handles CFLAGS.
(note: all changes also apply to FCFLAGS, CXXFLAGS, CPPFLAGS, and LDFLAGS).
1. The configure process will now always preserve a user's CFLAGS
environment variable setup. Any additional flags necessary for compilation
added at configure time will be passed into the Makefiles as AM_CFLAGS,
which is an automake construct to be used in addition to CFLAGS.
This will allow a user to have the final say, as CFLAGS will always appear
later in the compile line than AM_CFLAGS. Additionally, setting CFLAGS
during make will no longer completely erase all flags set by configure,
since they're maintained in AM_CFLAGS.
2. Additionally, where possible, flags previously being assigned directly
into CFLAGS (and thus propagating into h5cc) have now been redirected into
H5_CFLAGS, so they're used ONLY for compiling hdf5, and not embedded into
the h5cc wrapper script as well.
*Note that H5_CFLAGS ultimately is assigned into AM_CFLAGS for use in the
Makefiles. Complete description of changes and build process will be
included in a Configure Document that Elena and I are working on.
3. Removed unsupported config files. This includes:
config/dec-osf*
config/hpux11.00
config/irix5.x
config/powerpc-ibm-aix4.x
config/rs6000-ibm-aix5.x
config/unicos*
4. Modified configure summary to display additional values. Specifically,
appropriate AM_* variables are being shown, as well as H5_FCFLAGS and
H5_CXXFLAGS, which were for some reason not already present.
Tested:
- H5committest
- Tested on all THG / NCSA machines, using several combinations of the more
prominent configure options (c++, fortran, szip, threadsafe, parallel,
et cetera). (Thanks to Quincey for rysnc testing setup!)
- With regards to new automated testing, anything *necessary* for
compilation will be caught by the daily tests as it stands now. (i.e.,
if LDFLAGS is not properly set when szip is used, linking will fail).
Additionally, with regards to which flags get into h5cc, if any
*necessary* flags have been improperly removed, then daily tests should
fail during make installcheck. Additional machine-specific desired
behaviors and/or checks may have to be set up separately within the
daily tests, so this is something to work on.
Description:
When copying a dataset using a vlen inside a compound, the various dataset
copying callbacks would allocate a background buffer but would not use it when
converting from disk to memory, only memory to disk. This caused an assertion
failure as compounds always need a background buffer. These callbacks have
been modified to use the background buffer for both conversions.
Tested: jam, linew, smirom (h5committest)
Makefile Bug Fix
Description:
Added $(EXEEXT) extension to H5detect when it's executed to generate
H5Tinit.c so it works correctly on platforms that require the full
extension when running executables.
Tested:
h5committest, mingw, and user-verified (via submitted patch).
