- HDFFV-5928 - GMQS: h5diff problem and improvement on comparsing the same objects
Description:
Fixed:
1) adding h5tools_is_obj_same() function to check if two given IDs or paths point to the same object. This function can be very useful for other tools and applications.
2) using h5tools_is_obj_same() at h5diff() and diff() in h5diff.c. If two paths point to the same object, there is no need to check the details of the object since we know there is no difference. The fix will increase the performance by skipping the content comparison. It also fixed the problem of reporting difference for some cases of comparing the same file, e.g. empty files or files with incomparable objects the same file.
Test update:
Updat prvious test cases (171, 172, 530) affected by this fix, so they
still perfrom originally intended testing without bypassing.
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), cmake
a check for empty attribute in h5dump.c. I also added a check for empty dataset in tools/lib/h5tools.c.
Tested on jam. But I tested the same changes in 1.8 on jam, linew, and amani.
Bring Coverity changes back to trunk:
r20276:
Add recursive is vlen string function.
Cleanup resource leaks for issues: 200,202,329,688,811,812
r20277:
Check types and close by adding error section: issue 687
r20278:
Replaced implicit pointer conversion with (ocrt_info.new_obj != NULL).
r20280:
Addressed coverity issues 927-929 & 583. The real issue is failure to check
file name length -- at least at the H5FD interface level. This needs more work, but at least I have dealt with the issue in H5FDfamily.c
r20337:
H5O_type_t obj_type = H5O_TYPE_UNKNOWN;
r20338:
Added udata.name = NULL; to prevent potential uninitialized use after done: label.
r20339:
coverity issues: 686,828,1670-1673,1707-1711
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
FreeBSD/32 8.2 (loyalty) w/gcc4.6, in debug mode
FreeBSD/64 8.2 (freedom) w/gcc4.6, in debug mode
Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-amd64 2.6 (amani) w/Intel compilers, w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, w/threadsafe, in production mode
Linux/PPC 2.6 (heiwa) w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in debug mode
Clean up various warnings & code formatting issues.
Bring changes from Coverity branch to trunk:
r20085:
Purpose: Fix coverity issue 793
Description: Modified H5S_hyper_project_simple_higher() to free the entire span
list in new_space on failure.
r20091:
This is a fix for coverity bug #1683.
Changed the two printfs to use %lu (unsigned long) for printing "dset_size".
r20162:
Purpose: Fix coverity issue 785
Description: Modified H5T_enum_nameof() to free "name" on failure if it was
allocated. Also clarified some code in H5S_hyper_rebuild_helper().
r20189:
Addressed coverity defect 783.
H5SL_new_node() in H5SL.c was failing to free space allocated in its
first alloc if the second alloc failed. Added a call to H5FL_FREE
to address this issue.
This is purely to keep coverity happy -- if this code is ever triggered,
we have much larger problems.
Note that this fix will trigger an unused return value complaint
from coverity next week.
r20190:
Fixed Coverity issues 1561 1565 and 1678 (UNUSED_VALUES) by moving checks of return values to after the function call.
r20191:
Fixed coverity issues 643 644 and 1678 (CHECKED_RETURN).
r20232:
Addressed coverity issues 923-925. Replaced calls to sprintf with calls
to HDsnprintf.
r20233:
Fix coverity issue 662. Don't try to sort 0 attributes in H5Aint.c.
r20234:
Fix coverity issue 664. Check for NULL before dereferencing in H5Gdeprec.c.
r20271:
Purpose: Fix coverity issue 784
Description: Modified H5_debug_mask() to keep a list of files opened for use as
a debugging output stream, and modified H5_term_library to close these files on
exit.
r20272:
addressed coverity issues 838 & 955. Issue was use of strcpy() -- existing
code was safe, but modified to use strncpy() to keep coverity happy.
r20273:
Addresed coverity issues 1388 and 1389.
Initialized sel_iter->type to NULL in H5S_select_iter_init.
r20275:
Purpose: Fix valgrind issue in mf.c
Description: Fixed bug (incomplete if statement) in test_mf_fs_alloc_free() so
the retrieved node gets freed.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-amd64 2.6 (amani) w/Intel compilers, w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, w/threadsafe, in production mode
Linux/PPC 2.6 (heiwa) w/C++ & FORTRAN, w/threadsafe, in debug mode
General shared library improvements for CYGWIN / AIX
Description:
Shared libraries are disabled on both CYGWIN and AIX due
to inability to build them correctly. Part of the problem
in both of these situations is the lack of the libtool
flag -no-undefined, which tells libtool that all needed
symbols are defined at link time (a requirement on these
systems) and that it's okay to build shared libraries.
Another problem are lack of dependencies between wrapper
libraries and core C HDF5 library.
This patch addresses both of these by fixing configure to
add in -no-undefined flag for libtool during linking and
adds automake dependencies in the Makefile.am files.
After testing, both CYGWIN and AIX now generate shared
libraries, but there are still some test failures in each.
(cache_api, dt_arith, and testerror.sh on CYGWIN, and
fortran tests on AIX).
Even though the shared libraries are not quite perfect,
this is a general improvement to what we had before, so
I'm applying the patch anyways. Note that default behavior
of shared libraries on these systems being disabled has
NOT been changed and requires the use of the
--enable-unsupported to attempt to build them.
We will need to address the test failures in each
architecture prior to formally supporting shared
libraries on each.
Tested:
h5committested & CYGWIN tested (on bangan)
(AIX tested by Albert on bp-login2)
Add "silent make" mode configure option.
Description:
Automake 1.11 has a new option available that allows for a
silent make mode. This functionality needs to be explicitly
enabled in configure.in via the use of the automake macro
AM_SILENT_RULES, which is what this commit is adding.
This introduces a new configure option:
--{en|dis}able-silent-rules
This option is on by default, and simplies compile and link
line outputs when building the library. Disabling this option
will print full "verbose" output (i.e., full compile and
linking lines for each target).
Tested:
This was tested on jam & h5committested
Remove global variable for the path name in the group traversal code. Also
clean up a bunch of compiler warnings.
Tested on:
Mac OS X/32 10.6.7 (amazon) w/debug & production
FreeBSD/32 6.3 (duty) w/debug
FreeBSD/64 6.3 (liberty) w/debug
Solaris/32 2.7 (linew) w/debug
- Revise shared Fortran library disabling scenarios in configure
- Improve configure output summary
Description:
Shared Fortran libraries are not supported on Mac, but were being
disabled by configure in a way that also forced the C libraries
to be static-only. This has been fixed, so now only shared Fortran
is disabled while shared C can remain.
This prompted two additional changes:
1. While working on the check that addresses whether or not
shared Fortran libraries are allowed, removed old and no
longer needed check(s) that disable shared Fortran
libraries with HP, Intel 8, PGI, and Absoft compilers.
(Essentially, Mac is the only situation in which Fortran
shared are disabled by configure.)
2. Having two different states of libraries (i.e. shared C
library with static-only Fortran library) was not apparent
in the configure summary, which labeled all libraries as
either shared and/or static. I've added lines to both the
C++ and Fortran output sections to list shared/static-ness
of these libraries specifically.
Additionally, I've made sure that the new --enable-unsupported
configure option correctly overrides configure if it tries to
disable a shared library.
Tested:
jam, fred, & h5committest
Fixed Bug 2184 - GMQS: h5diff - incorrect calculation code for
--use-system-epsilon option
Description:
Merged from HDF5 1.8 branch r20369.
Fixed h5diff for --use-system-epsilon option: the calculation changed
from ( |a - b| / b ) to ( |a - b| ). This was decided for better
performance and was corrected only in HDF5 trunk, so 1.8 got updated.
Also comments for equal_XXX() function were updated correctly.
Also help page and RM got updated correctly.
Also add test cases for testing the differences w/wo the option.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE)
Fixed CHICAGO: Bug 2121 - h5diff - incorrect and lack of output for the
different set of attributes (different number and names)
Description:
Previously h5diff compared attributes correctly only when two objects have
the same number of attributes and attribute names are identical.
This fix covers all other cases.
Also didn't display useful information about attribute difference.
This fixes both issues.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Improve the previous fix for Bug 2216 - GMQS: h5diff - memory leak when
compares vlen string in dataset or attributes
Description:
Related to the previous checkin r20270 and r20266.
Improve h5tools_detect_vlen() code for better performance. H5Tdetect_class
already recusive on given type so don't need to be part of recusive call
again. Also improve error handlings in h5tools_detect_vlen and
h5tools_detect_vlen_str functions.
Also updated h5ls and h5dump code accordingly.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Cmake - jam
Improve the previous fix for Bug 2216 - GMQS: h5diff - memory leak when
compares vlen string in dataset or attributes
Description:
Improve the fix along with the previous checkin r20266.
Add a new function to tool lib, h5tools_detect_vlen_data() which return
TRUE if include any kind of vlen data all at once, either VLEN-data or
VLEN-string and so on.
Also updated h5ls and h5dump code accordingly.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Cmake - jam
Fixed Bug 2216 - GMQS: h5diff - memory leak when compares vlen string in
dataset or attributes
Description:
Test for dataset : valgrind --leak-check=full ./h5diff -v h5diff_dset1.h5 h5diff_dset2.h5 /g1/VLstring
Test for attr : valgrind --leak-check=full ./h5diff h5diff_attr1.h5 h5diff_attr2.h5
Both test cases are in testing script.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Cmake - jam
htri_t H5Tdetect_vlen_str(hid_t tid) to h5tools. This needs to be called before any H5Aread/H5Dread and if TRUE, then call vlen_reclaim function after the corresponding h5tools_dump_xxx().
Tested: local linux and valgrind
Fixing Bug 2161 - GMQS: h5dump - only on Windows, skip displaying a data value every a certain lines in array type dataset
Description:
Fixed h5dump for skipping some values for long array type dataset on
Windows. This issue only occurred on Windows due to the different
return behavior from _vsnprintf() funtion.
Tested:
Windows, jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Fixing Bug 2092 - h5dump does not display index for a dataset
Description:
h5dump skip displaying array indices every certain number when the
array type dataset is relatively big. The certain number varies
according to the size of each array.
This checkin fix the skipping array indices problem. This fix also
correct indentation of the dataset data output.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Bring changes from Coverity branch to trunk:
r19975:
Fixed potential mem leak at H5O_attr_open_by_name
r19980:
Fix coverity issue 792.
Free tmp_env_prefix in H5Lexternal.c line 365 if it is not NULL but its contents are 0 when it goes out of scope.
r20039:
Eliminate warnings about nested extern and implicit declarations of parallel_print and address Coverity defects 712-781 by #including h5tools_utils.h in h5diff_array.c, h5diff_attr.c, h5diff_dset.c and h5diff_util.c.
r20046:
Purpose: Address TOCTOU warnings in h5jam and h5unjam
Description: Coverity is afraid that the state of the input file could change
between the call to stat() and the call to open(). This is called a time-of-
check time-of-use (TOCTOU) vulnerability. Modified stat calls to fstat which
uses an open file pointer so it (hopefully) won't complain any more.
r20047:
Addressed coverity issues 135-137, 462-464. Local pointers that needed to be freed in case of error were moved out of a switch statement in src/H5Tnative.c, set to NULL, and checked before freeing.
Tested on:
Mac OS X/32 10.6.6 (amazon) w/debug & production
(h5committested on Coverity branch)
Bring changes from Coverity branch to trunk:
r19930:
Fix memory leaks involving VL attributes in h5repack and h5diff. The buffers in
copy_attr and diff_attr were not checked for the presence of a vlen before being
freed, and vlen storage was never reclaimed. Added checks and calls to
H5D_vlen_reclaim().
r19933:
Purpose: Fix memory leak in H5L_move_cb()
Description: H5L_move_cb copied the source link using H5O_msg_copy() but freed
it manually using H5MM_xfree(). Since H5O_link_copy allocates the link using
H5FL_MALLOC, this causes the link to be allocated from the free list but is
never put back on the free list when it is freed. This prevents the link free
list from shutting down properly. Modified H5L_move_cb() and H5L_move_dest_cb()
to free the link properly using H5O_msg_free().
r19973:
Fix resource leaks by freeing string created by HD5f2string
r19974:
Issue #345: Inialize buf variable to null
Tested on:
Mac OS X/32 10.6.6 (amazon) w/debug & production
(h5committested on Coverity branch)
Clean up Coverity warnings, and fix some style issues:
r19735:
Fix for memory leak in test/mf found by valgrind.
r19736:
Fix memory leak in h5repack. The buffer in copy_objects, when copying the
entire dataset at once, was not checked for the presence of a vlen, and vlen
storage was never reclaimed. Added check and call to H5D_vlen_reclaim().
r19772:
Change H5assert() to
if (H5T_VLEN != src->shared->type || H5T_VLEN != dst->shared->type)
HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, FAIL, "not a H5T_VLEN datatype")
r19774:
removed unused priv.
r19775:
removed unused variables
r19778:
Fix memory leak comparing for variable length data types.
r19834:
Fixed memory leaks found by valgrind. Memory errors remain for another day.
Tested on:
Mac OS X/32 10.6.6 (amazon) w/debug & production
(h5committested on branch)
Bug 2089 - GMQS: h5diff segfault on a compound dataset with fixed length +
vlen string type order
Description:
This is fix for the segfault when h5diff compares a compound dataset with
combination of fixed length string types and vlen string types in certain
orders. Optimized vlariable length string handling codes. The fix is
referred from h5dump handling vlen strings.
For testing, several compound datasets were added with various combinations.
Previous failed cases:
- Vlen string, Fixed length string, Vlen string, Fixed length string
- Fixed length string, Fixed length string, Vlen string, Vlen string
- Fixed length string, Vlen string, Fixed length string, Vlen string
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Fix compile error for Windows from previous checkin r19757.
Description:
Previous log:
Improve h5diff performance.
The following changes for improving h5diff performance:
1) use HDmemcmp() before comparing each elements
(memcmp() is very fast at both linew and jam)
2) replace the expensive H5Tequals() calls
3) retrieve datatype information at dataset level not each element level
for compound datasets
Tested:
jam (linux32-LE), Windows
Improve h5diff performance.
Description:
The following changes for improving h5diff performance:
1) use HDmemcmp() before comparing each elements
(memcmp() is very fast at both linew and jam)
2) replace the expensive H5Tequals() calls
3) retrieve datatype information at dataset level not each element level
for compound datasets
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Fixed h5diff to handle variable-length strings in a compound dataset
correctly. Also variable-length string array in a compound dataset.
Bug #1989.
Description:
Garbage values were displayed when h5diff compared variable-length
strings (or string array) in a compound type dataset.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (Mac32)
Bring Coverity revisions from branch back to trunk, and clean up some other
misc. compiler warnings also.
r19500:
Fix coverity items 1446 and 1447. Moved up calls to memset in test_cont in
ohdr.c so the test never tries to close uninitialized locations.
r19501:
Fix coverity items 1398-1445. Various uninitialized variable errors in fheap.c.
r19502:
Fixed coverity issue 579 and some additional warnings in the file as well.
r19503:
Bug fix: This fix addressed the "RESOURCE_LEAK" problems #789 and 790, run 26
r19504:
minor mods to try to keep coverity from flagging false positives.
r19505:
Fixed coverity issues 566 - 571. Declared variables that are passed to functions that use them as arrays to be arrays of size 1.
Tested on:
Mac OS X/32 10.6.4 (amazon) w/debug, production & parallel
(h5committested on trunk)
Fix for group comparison with exclude-path case. Use relative path.
Description:
Related to "1890: h5diff excluding object for file comparison via command line" checkins. (r19406)
Tested:
jam, amani and heiwa
Add --exclude-path option
Description:
Specified path to an object will be excluded from comparing the two files or two groups. If group is specified all the member objects will be excluded.
Related to "1890: h5diff excluding object for file comparison via command line"
Tested:
jam, amani and heiwa
Fix for Bug1975 h5diff - support recursive comparison on group when specified as an object
Description:
Compare member objects and groups recursively when two files or groups are specified to be compared. Support parallel diff and handling symbolic links accordingly.
Tested:
jam, amani, heiwa
Corrected use/name of source folder aliases.
Duplicated FindMPI.cmake so that non-c++ compiler is found first (recommemded commands did not work).
Tested: local linux with mpich
Merge r19290 & r19291 from 1.8 branch to trunk:
r19290:
Correct another error in metadata accumulator dirty region calculations
(this time with a corner case when freeing data in the file).
r19291:
Avoid getting object information for soft/external links if we aren't
going to traverse across the link itself.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-amd64 2.6 (amani) w/Intel compilers, w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, w/threadsafe, in production mode
Linux/PPC 2.6 (heiwa) w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in debug mode
Mac OS X/32 10.6.4 (amazon) in debug mode
Mac OS X/32 10.6.4 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
Mac OS X/32 10.6.4 (amazon) w/parallel, in debug mode
Bring Coverity changes from branch to trunk:
r19161:
Fixed the part for matching the subset info with dataset
r19189:
BZ1646: h5dump does not check number of dimensions for subsetting parameters against the dataset
Changed subset_t structure from holding hsize_t pointers to holding new subset_d pointers, which hold the original hsize_t pointer + len. this len is then checked against dataset ndims in the handle_dataset function of h5dump.
Changed all references to use new data structure.
Added tests for each subset parameter.
r19190:
Added new h5dump ddl files
Tested on:
Mac OS X/32 10.6.4 (amazon) w/debug & production
(h5committested on branch)
Clean up misc. compiler warnings.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
Mac OS X/32 10.6.4 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
Mac OS X/32 10.6.4 (amazon) w/parallel, in debug mode
Bring changes on Coverity branch back to trunk:
r19040:
Fixed coverity #440 - NULL check after dereference. We moved the NULL check up
into the IF block and changed it to assertion.
r19041:
Maintenance: Addressed Coverity issues 441 and 449 by initializing proper
variables
r19042:
In function H5O_chunk_protect (H5Ochunk.c):
- Initialize H5O_chunk_proxy_t pointers chk_proxy and ret_value.
- Free chk_proxy on error.
r19043:
Addressed coverity issues 442 - 448 by initializing pointers to NULL.
Tested on:
Mac OS X/32 10.6.4 (amazon) w/debug & production
(Too minor to require h5committest)
