Move error_stack setup to tools library
Add structure for h5dump properties
Eliminate duplicated h5diff functions
Use properties structure in place of argument lists.
* commit '54957d37f5aa73912763dbb6e308555e863c43f4':
Commit copyright header change for src/H5PLpkg.c which was added after running script to make changes.
Add new files in release_docs to MANIFEST. Cimmit changes to Makefile.in(s) and H5PL.c that resulted from running autogen.sh.
Merge pull request #407 in HDFFV/hdf5 from ~LRKNOX/hdf5_lrk:hdf5_1_10_1 to hdf5_1_10_1
Change copyright headers to replace url referring to file to be removed and replace it with new url for COPYING file.
gcc 4.9.2 was used to create the warning list
- implicit casts
- shadowed variables
- various enum issues
- other minor fixes (comments, unused macros, etc.)
Tested on: h5committest
Bring changes from hdf5_1_8_coverity branch back to trunk:
r20878:
Issue 76: Check if H5Tget_nmembers(type) fails and simply return(FALSE). Also move printf to after check.
r20880:
Issue 192: Create ret_val var set to -1. Add out label for failures to jump to, return ret_val at bottom.
r20882:
Fixes for coverity:
1) bug #1679: remove dead code in test/mf.c
2) bug #1680: remove dead code in tools/lib/h5diff_dset.c
r20883:
Fix coverity issue 585
Description:
Changed variable "c" in processStrData in h5import.c to an int, to match fgetc
return value, and removed call to feof, instead checking if c == EOF.
Tested on:
MacOSX/64 10.9.3 (amazon) w/C++, FORTRAN & parallel
(too minor to require h5committest)
HDFFV-5919 - GMQS: h5diff - The number of object difference is not consistent between dataset and group/type when attribute(s) have differences
Description:
Object differences are not consistent between dataset and group/datatype when their attribute(s) have differences. This is because attribute(s) differences is not accumulated to group or datatype object’s difference, but accumulated to dataset difference.
To fix, do not accumulate attribute difference to dataset difference. This is referred to h5diff’s default behavior and also past report from users that users were confused by the accumulated behavior. (also can’t figure out only for dataset difference , also hard to spot dataset difference when it has lots of attributes or differences)
This also lead to fix inconsistent format indicating difference between dataset and group/datatype object.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE cmake), cmake (jam)
Address HDFFV-7942 - h5diff: incorrect result for comparing attribute data with different type size in same class
Description:
When comparing attribute data values with same type class but different size, the result was incorrect. It was due to the size difference and got truncated.
Fixed to match up the smaller type size to big type size like what dataset does.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE cmake), cmake (jam)
HDFFV-7693 - h5diff produces different behavior between versions: 1.8.3 and 1.8.6
Description:
There were two issues on this.
One was not displaying all the comparable objects if non-comparable
object/attribute exist and compared before comparables ones. This issue
occurred after 1.8.4 release. This is the issue from user.
The other issue was not displaying all the comparable attributes,
if non-comparable object/attribute exist and compared ahead. This issue
were exist even before 1.8.4 release. So it's possible some attribute
comparison may have not displayed correctly in the past, if non-comparable
data were exist in common object.
Fixed h5diff to display all the comparable object and attribute regardless
of non-comparables.
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Cmake (jam, Windows)
- h5repack: h5repack failed to copy dataset if the layout is changed from chunked with
unlimited dims to contiguous. (PC -- 2011/07/15)
- h5diff: "--delta" option considers two NaN of the same type are different, which is wrong
based on http://www.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Diff. (PC -- 2011/07/15)
Improve the previous fix for Bug 2216 - GMQS: h5diff - memory leak when
compares vlen string in dataset or attributes
Description:
Related to the previous checkin r20270 and r20266.
Improve h5tools_detect_vlen() code for better performance. H5Tdetect_class
already recusive on given type so don't need to be part of recusive call
again. Also improve error handlings in h5tools_detect_vlen and
h5tools_detect_vlen_str functions.
Also updated h5ls and h5dump code accordingly.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Cmake - jam
Improve the previous fix for Bug 2216 - GMQS: h5diff - memory leak when
compares vlen string in dataset or attributes
Description:
Improve the fix along with the previous checkin r20266.
Add a new function to tool lib, h5tools_detect_vlen_data() which return
TRUE if include any kind of vlen data all at once, either VLEN-data or
VLEN-string and so on.
Also updated h5ls and h5dump code accordingly.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Cmake - jam
Fixed Bug 2216 - GMQS: h5diff - memory leak when compares vlen string in
dataset or attributes
Description:
Test for dataset : valgrind --leak-check=full ./h5diff -v h5diff_dset1.h5 h5diff_dset2.h5 /g1/VLstring
Test for attr : valgrind --leak-check=full ./h5diff h5diff_attr1.h5 h5diff_attr2.h5
Both test cases are in testing script.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Cmake - jam
Bring changes from Coverity branch to trunk:
r19975:
Fixed potential mem leak at H5O_attr_open_by_name
r19980:
Fix coverity issue 792.
Free tmp_env_prefix in H5Lexternal.c line 365 if it is not NULL but its contents are 0 when it goes out of scope.
r20039:
Eliminate warnings about nested extern and implicit declarations of parallel_print and address Coverity defects 712-781 by #including h5tools_utils.h in h5diff_array.c, h5diff_attr.c, h5diff_dset.c and h5diff_util.c.
r20046:
Purpose: Address TOCTOU warnings in h5jam and h5unjam
Description: Coverity is afraid that the state of the input file could change
between the call to stat() and the call to open(). This is called a time-of-
check time-of-use (TOCTOU) vulnerability. Modified stat calls to fstat which
uses an open file pointer so it (hopefully) won't complain any more.
r20047:
Addressed coverity issues 135-137, 462-464. Local pointers that needed to be freed in case of error were moved out of a switch statement in src/H5Tnative.c, set to NULL, and checked before freeing.
Tested on:
Mac OS X/32 10.6.6 (amazon) w/debug & production
(h5committested on Coverity branch)
Clean up Coverity warnings, and fix some style issues:
r19735:
Fix for memory leak in test/mf found by valgrind.
r19736:
Fix memory leak in h5repack. The buffer in copy_objects, when copying the
entire dataset at once, was not checked for the presence of a vlen, and vlen
storage was never reclaimed. Added check and call to H5D_vlen_reclaim().
r19772:
Change H5assert() to
if (H5T_VLEN != src->shared->type || H5T_VLEN != dst->shared->type)
HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, FAIL, "not a H5T_VLEN datatype")
r19774:
removed unused priv.
r19775:
removed unused variables
r19778:
Fix memory leak comparing for variable length data types.
r19834:
Fixed memory leaks found by valgrind. Memory errors remain for another day.
Tested on:
Mac OS X/32 10.6.6 (amazon) w/debug & production
(h5committested on branch)
Bring r18076-18096 from hdf5_1_8_coverity branch to trunk:
r18076:
Correct Coverity issue #1 by removing dead code
r18077:
Fix coverity item 142. When an error occurred while copying a linked list in
H5S_point_copy, the library would not free the partially allocated list. Added
code to free the list in this case.
r18078:
Correct Coverity issue #2 by removing impossible to reach code.
r18079:
Correct #3 by removing impossible to reach code.
r18080:
Correct Coverity issue #4 by removing impossible to reach code.
r18081:
fix coverity 26 , check (dblik->parent) before calls H5HF_man_iblock_detach().
r18082:
Fixed coverity issues 321 and 316.
321: freed sm_buf in error handling to remove resource leak.
Also set sm_buf to NULL after other instances in which
it is freed to prevent double free.
316: initialized nmembs to 0.
r18083:
Correct Coverity issue #6 by removing debugging knob from error reporting code.
r18084:
Fix coverity item 269 + others. When a error occurred in a function using the
h5tools error framework, the "past_catch" variable would not be set to true
because that statement was before the label that goto jumped to. This could
cause a failure in the cleanup section to go back to the start of the section,
freeing variables twice, etc.
Moved the label infront of past_catch=TRUE.
r18085:
fixed coverity #27, check if (heap) before use heap->obj....
r18086:
fixed coverity #28, check curr_span not null before use it at if(curr_span &&
(io_bytes_left==0 || curr_seq>=maxseq))
r18087:
Correct Coverity issue #7 by cleaning up correctly on error
r18088:
Correct Coverity #8 by removing unchanged variable checking code.
r18089:
Correct Coverity issue #9 - remove impossible to reach code.
r18090:
Correct Coverity issue #11 by removing impossible to reach code. Also clean
up some minor style issues.
r18091:
Fix coverity items 314 and 318. Changed the improper assertion of the return
value of a library function to a check, and a return(void) on failure.
r18092:
Fix coverity item 70. Changed the improper assertion of the return value of a
library function to a check, and a return(void) on failure.
r18093:
Correct Coverity issue #12 by removing dead code.
r18094:
Correct Coverity issue #16 by removing debugging code.
r18095:
Fixed coverity issue # 271.
Removed redundant checking and freeing of sm_buf1 and sm_buf2.
r18096:
Correct Coverity issue #17 by refactoring test to remove dead code.
Also, removed previous "coverity" statements in comments, we'll review
those issues again and see if we can figure them out, now that we have more
experience with Coverity.
Tested on:
Mac OS X/32 10.6.2 (amazon)
Bring Coverity changes into the trunk:
r17877:
Error 266: Uninitialized memspace set to -1. Changed malloc and free to HDmalloc and HDfree. Removed unused dtype var.
r17878:
Error 265: Uninitialized mem_space set to -1. Changed malloc and free to HDmalloc and HDfree.
Error 267: Uninitialized smspace set to -1. Changed malloc and free to HDmalloc and HDfree.
r17879:
Error 242: Uninitialized aid set to -1.
Error 243: Uninitialized sid set to -1.
Uninitialized tid set to -1 for consistency
r17880:
Error 242: reinitialized aid to -1 after close to be ready for reuse.
Error 243: reinitialized sid to -1 after close to be ready for reuse.
reinitialized tid to -1 for consistency after close to be ready for reuse.
r17881:
use valgrind to check there is a memory leak. The fix is to free ptrstr in line 5838 at xml_dump_group() of h5dump.c after it is used. run the valgrind after the fix, no memory leak for that part of the code.
r17882:
Fix Coverity items 256 and 269.
r17883:
Error 222-230: initialized hid_t vars to -1 after close to be ready for reuse.
Also added H5Tclose for tid in gent_bigdims
r17884:
Bug fix (Coverity run2, view 23, dead code)
(this is pair-program done by Albert, Elena and Larry).
Confirmed and fixed the deadcode in hyperslab read branch of function
diff_datasetid.
(Discovered other bad code that diff_datasetid() should be recoded.
Bug 1693 is entered for this.)
r17906:
Fix Coverity item 260.
r17907:
262: Initialized hid_t's dtype, dtype_tmp and file to -1. Initialized H5T_t * dt to NULL.
r17908:
Fix Coverity item 261.
r17909:
Fix Coverity item 248.
r17910:
Revise fix for Coverity item 248.
r17911:
Resolved coverity issues #s 263, 162, 163, 164. All issues in dsets.c. Initialized fid and did hid_t's. filter_corrupt function was returning in the middle of an if statement, bypassing free calls. Updated error handling to free buffers and hid_t's appropriately.
r17912:
(done by Larry and Albert)
Cleanup Coverity view warnings (#231-241) about
using uninitialized variables. Initialized all
of them.
r17913:
Resolved issue 251 and 264. Initialized tid2 to -1 and initialized buffers that were freed in case of an error.
r17914:
Resolved coverity issues 66, 220, and 221:
66: Negative Return assignment ignored
220,221: Initialized hid_t's.
r17915:
Fix Coverity item 247.
r17916:
Fix Coverity item 246.
r17917:
Fix Coverity item 245.
r17918:
Fix Coverity item 244.
r17919:
Coverity Issue #84: Moved asserts in H5T_cmp to the top of the function, and converted them to HDassert. Coverity complaining about using potentially NULL pointer without checking it. Want to see if Coverity will accept Assertions as acceptable checking before using the value.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (jam) w/PGI compilers, w/default API=1.8.x,
w/C++ & FORTRAN, w/threadsafe, in debug mode
Linux/64-amd64 2.6 (smirom) w/Intel compilers, w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in debug mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
Mac OS X/32 10.6.2 (amazon) in debug mode
Mac OS X/32 10.6.2 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
ISSUE : the tools use the following formula to read by hyperslabs: hyperslab_size[i] = MIN( dim_size[i], H5TOOLS_BUFSIZE / datum_size) where H5TOOLS_BUFSIZE is a constant defined of 1024K. This is OK as long as the datum_size does not exceed 1024K, otherwise we have a hyperslab size of 0 (since 1024K/(greater than 1024K) = 0). This affects h5dump. h5repack, h5diff
SOLUTION: add a check for a 0 size and define as 1 if so.
TEST FOR H5DUMP: Defined a case in the h5dump test generator program of such a type (an array type of doubles with a large array dimension, that was the case the user reported). Since the written file commited in svn would be around 1024K, opted for not writing the data (the part of the code where the hyperslab is defined is executed, since h5dump always reads the files). Defined a macro WRITE_ARRAY to enable such writing if needed. Added a run on the h5dump shell script. Added 2 new files to svn: tools/testfiles/tarray8.ddl, tools/testfiles/tarray8.h5. NOTE: while doing this I thought of adding this dataset case to an existing file, but that would add the large array output to those files (the ddls). The issue is that the file list is increasing.
TEST FOR H5DIFF: for h5diff the check for reading by hyperslabs is H5TOOLS_MALLOCSIZE (128 * H5TOOLS_BUFSIZE) or 128 Mb. This makes it not possible to add such a file to svn, so used the same method as h5dump (only write the dataset if WRITE_ARRAY is defined). As opposed to h5dump, the hyperslab code is NOT executed when the dataset is empty (dataset is not read). Added the new dataset to existing files and shell run (tools/h5diff/testfiles/h5diff_dset1.h5 and tools/h5diff/testfiles/h5diff_dset2.h5 and output in tools/h5diff/testfiles/h5diff_80.txt).
TEST FOR H5REPACK: similar issue as h5diff with the difference that the hyperslab code is run. Added a run to the shell script (with a filter, otherwise the code uses H5Ocopy).
tested: linux (h5commitest failed , apparently it did not detect the code changes in /tools/lib that fix the bug: the error in an assertion in the hyperslab of 0. I am sure that making h5ccomitest --distclean will detect the new code , but don't want to wait more 3 hours :-) )
Solution: for compound types, recursively apply that check
Two new cases are added
1) the compound type has a different number of members. Message printed is
<obj1> has X members <obj2> has Y members
Where X and Y are the number of members of each compound type being compared
2) the compound type has not comparable types (for example a double and an int at the same index)
In this case the message
Comparison not possible: object1 is of class1 and object2 is of class2
Is replaced with
Comparison not possible: object1 has a class1 and object2 has a class2
Modified the test generator program to have these 2 cases
Added a shell run for these 2 cases
Tested: windows, h5committest
