Merge 64-bit ID changes from branch to trunk. (Plus a few minor cleanups
that aren't on the branch)
Tested on:
Mac OSX/64 10.9.4 (amazon) w/C++ & FORTRAN
(h5committested on branch already for a week)
Prep work for adding a memory free function, part of HDFFV-8551.
Tested on:
32-bit LE linux (jam)
* with C++, Fortran
* with thread safety on
These are minor changes (almost entirely renaming foo functions
to HDfoo) that don't change functionality. This should not
require a full commit test.
HDFFV-7942 - h5diff: incorrect result for comparing attribute data with different type size in same class
Description:
This is additional fix due to daily build fail against previous r22684.
Refer to the svn -r22684 log for previous details
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), cmake (jam)
Address HDFFV-7942 - h5diff: incorrect result for comparing attribute data with different type size in same class
Description:
When comparing attribute data values with same type class but different size, the result was incorrect. It was due to the size difference and got truncated.
Fixed to match up the smaller type size to big type size like what dataset does.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE cmake), cmake (jam)
Fix for HDFFV-7998-h5diff: incorrect behavior and output for comparing symbolic dangling links without following-symlinks option
Description:
Fix not to check and display dangling link status without --follow-symlinks option.
Berfor the fix, when comparing symbolic link to symbolic link without follow-symlinks option, h5diff followed to check if those links are dangling or not. It caused to display output incorrectly (dangling link instead of symbolic link).
The fix also improved performance when comparing lots of symbolic links without the --follow-symlinks option.
Test cases were added and tagged with jira#.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE cmake), Cmake (jam)
Fix for HDFFV-7644 GMQS: h5diff - incorrect behaviors when comparing HDF5 files with different type of objects (dataset, group, type) as common objects
Description:
Fixed failure for comparing same named object with different object types in comparing groups. (common object comparison)
Prior to the fix, h5diff resulted in error. After the fix, h5diff detects such case as non-comparable and display messages accordingly.
Test cases were added and tagged with jira#.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE cmake), Cmake (jam)
Fix for HDFFV-7837 - h5diff: incorrect behavior with exclude-path option when unique object exist only in one file
Description:
If unique objects exists only in one file and try to exclude
the unique objects with --exclude-path option, h5diff missed
excluding some objects.
Fixed to exclude objects correctly in such case.
Test cases were added and tagged with jira#.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE cmake), Cmake (jam)
HDFFV-7770 - h5diff segmentation fault on JPSS files
Description:
Fixed segfault over non-comparable attribute with different dimention or
rank, along with '-c' option to display details.
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows, Cmake
Added a funtion to reset dataset & hyperslab buffer size for h5repack from an
environment variable.
This is performance debugging purpose for now.
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), Windows (32-LE), cmake
HDFFV-7693 - h5diff produces different behavior between versions: 1.8.3 and 1.8.6
Description:
There were two issues on this.
One was not displaying all the comparable objects if non-comparable
object/attribute exist and compared before comparables ones. This issue
occurred after 1.8.4 release. This is the issue from user.
The other issue was not displaying all the comparable attributes,
if non-comparable object/attribute exist and compared ahead. This issue
were exist even before 1.8.4 release. So it's possible some attribute
comparison may have not displayed correctly in the past, if non-comparable
data were exist in common object.
Fixed h5diff to display all the comparable object and attribute regardless
of non-comparables.
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Cmake (jam, Windows)
- h5repack: h5repack failed to copy dataset if the layout is changed from chunked with
unlimited dims to contiguous. (PC -- 2011/07/15)
- h5diff: "--delta" option considers two NaN of the same type are different, which is wrong
based on http://www.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Diff. (PC -- 2011/07/15)
H5Dread() produces different output for invalid enum type values
on systems of different endiannes. On BE systems, it always
emits -1. On LE systems, the invalid value is emitted. This
difference caused the h5diff test to fail since it tests to
see if different invalid enum values are reported as different
(on BE systems they are not since -1 == -1).
This fix removes the "invalid vs. different invalid" test
while we fix the library bug. HDFFV-7527 will be reopened
in JIRA until the full test passes.
Tested on: jam, heiwa, linew
not represent a valid enum value.
Description:
The h5diff code compares enum values by converting them to strings and
then comparing them. When the enum value is out of range and can't be
converted to an enum string representation, the comparison was skipped.
The code now flags differences when one of the two enum values is out
of range (two out of range values are compared in memory). A test has
been added to the tools test script.
This fixes JIRA HDFFV-7527
Tested on: jam, koala, heiwa (h5committest)
- HDFFV-5928 - GMQS: h5diff problem and improvement on comparsing the same objects
Description:
Fixed:
1) adding h5tools_is_obj_same() function to check if two given IDs or paths point to the same object. This function can be very useful for other tools and applications.
2) using h5tools_is_obj_same() at h5diff() and diff() in h5diff.c. If two paths point to the same object, there is no need to check the details of the object since we know there is no difference. The fix will increase the performance by skipping the content comparison. It also fixed the problem of reporting difference for some cases of comparing the same file, e.g. empty files or files with incomparable objects the same file.
Test update:
Updat prvious test cases (171, 172, 530) affected by this fix, so they
still perfrom originally intended testing without bypassing.
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), cmake
Fixed Bug 2184 - GMQS: h5diff - incorrect calculation code for
--use-system-epsilon option
Description:
Merged from HDF5 1.8 branch r20369.
Fixed h5diff for --use-system-epsilon option: the calculation changed
from ( |a - b| / b ) to ( |a - b| ). This was decided for better
performance and was corrected only in HDF5 trunk, so 1.8 got updated.
Also comments for equal_XXX() function were updated correctly.
Also help page and RM got updated correctly.
Also add test cases for testing the differences w/wo the option.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE)
Fixed CHICAGO: Bug 2121 - h5diff - incorrect and lack of output for the
different set of attributes (different number and names)
Description:
Previously h5diff compared attributes correctly only when two objects have
the same number of attributes and attribute names are identical.
This fix covers all other cases.
Also didn't display useful information about attribute difference.
This fixes both issues.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Bug 2089 - GMQS: h5diff segfault on a compound dataset with fixed length +
vlen string type order
Description:
This is fix for the segfault when h5diff compares a compound dataset with
combination of fixed length string types and vlen string types in certain
orders. Optimized vlariable length string handling codes. The fix is
referred from h5dump handling vlen strings.
For testing, several compound datasets were added with various combinations.
Previous failed cases:
- Vlen string, Fixed length string, Vlen string, Fixed length string
- Fixed length string, Fixed length string, Vlen string, Vlen string
- Fixed length string, Vlen string, Fixed length string, Vlen string
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Fixed h5diff to handle variable-length strings in a compound dataset
correctly. Also variable-length string array in a compound dataset.
Bug #1989.
Description:
Garbage values were displayed when h5diff compared variable-length
strings (or string array) in a compound type dataset.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (Mac32)
Add extra test cases for bug1975 h5diff - support recursive comparison on group when specified as an object.
Description:
Additional tests for combination of group recursive and --follow-symlinks
with multi-linked external links with several files (same name/strucure).
Tested:
jam, amani, heiwa
Add --exclude-path option
Description:
Specified path to an object will be excluded from comparing the two files or two groups. If group is specified all the member objects will be excluded.
Related to "1890: h5diff excluding object for file comparison via command line"
Tested:
jam, amani and heiwa
Fix for Bug1975 h5diff - support recursive comparison on group when specified as an object
Description:
Compare member objects and groups recursively when two files or groups are specified to be compared. Support parallel diff and handling symbolic links accordingly.
Tested:
jam, amani, heiwa
Enable tools lib to be built as a dll on windows. Added two get/set functions for progname and d_status.
Also add windows import/export declarations to functions.
Updated error_mesg() and warn_mesg() to remove progname argument and use get functions
Tested:
Windows, linux
bugzilla 1754: h5diff: support comparing through links.
Description:
Fix incorrect (or hanging) behavior in parallel mode when specifying
invalid combination of command options.
(ex: -v and -q , --no-dangling-links without --follow-links)
Add relate test case
Update h5diffgentest.c due to add test files this and previous time
Note:
svn #18266 (prior to this)
svn #18164 (original check-in)
Tested:
h5committest (jam, amani and linew)
this error
home/hdftest/snapshots-hdf5/current/tools/lib/h5tools_ref.c:126: undefined reference to `d_status'
happens because the original developer of h5tools_ref.c declared these 2 variables in that source file
extern char *progname;
extern int d_status;
this means that all programs that use this source file (typically all tools) will have to define these 2 variables, even if they are *not* used.
examples
const char *progname = "h5jam";
int d_status = EXIT_SUCCESS;
"progname" is usually used to print the program name in the usage, and "d_status" is just a return value of the tool, usually.
I was wondering about removing these 2 declarations from h5tools_ref.c
extern char *progname;
extern int d_status;
and define those variables on each tool if they are only needed (it seems to me that only h5ls and h5dump actually "need" them)
tested: smirom parallel
ISSUE : the tools use the following formula to read by hyperslabs: hyperslab_size[i] = MIN( dim_size[i], H5TOOLS_BUFSIZE / datum_size) where H5TOOLS_BUFSIZE is a constant defined of 1024K. This is OK as long as the datum_size does not exceed 1024K, otherwise we have a hyperslab size of 0 (since 1024K/(greater than 1024K) = 0). This affects h5dump. h5repack, h5diff
SOLUTION: add a check for a 0 size and define as 1 if so.
TEST FOR H5DUMP: Defined a case in the h5dump test generator program of such a type (an array type of doubles with a large array dimension, that was the case the user reported). Since the written file commited in svn would be around 1024K, opted for not writing the data (the part of the code where the hyperslab is defined is executed, since h5dump always reads the files). Defined a macro WRITE_ARRAY to enable such writing if needed. Added a run on the h5dump shell script. Added 2 new files to svn: tools/testfiles/tarray8.ddl, tools/testfiles/tarray8.h5. NOTE: while doing this I thought of adding this dataset case to an existing file, but that would add the large array output to those files (the ddls). The issue is that the file list is increasing.
TEST FOR H5DIFF: for h5diff the check for reading by hyperslabs is H5TOOLS_MALLOCSIZE (128 * H5TOOLS_BUFSIZE) or 128 Mb. This makes it not possible to add such a file to svn, so used the same method as h5dump (only write the dataset if WRITE_ARRAY is defined). As opposed to h5dump, the hyperslab code is NOT executed when the dataset is empty (dataset is not read). Added the new dataset to existing files and shell run (tools/h5diff/testfiles/h5diff_dset1.h5 and tools/h5diff/testfiles/h5diff_dset2.h5 and output in tools/h5diff/testfiles/h5diff_80.txt).
TEST FOR H5REPACK: similar issue as h5diff with the difference that the hyperslab code is run. Added a run to the shell script (with a filter, otherwise the code uses H5Ocopy).
tested: linux (h5commitest failed , apparently it did not detect the code changes in /tools/lib that fix the bug: the error in an assertion in the hyperslab of 0. I am sure that making h5ccomitest --distclean will detect the new code , but don't want to wait more 3 hours :-) )
Remove another call to H5E_clear_stack() from within the library.
Clean up lots of compiler warnings.
Tested on:
Mac OS X/32 10.5.6 (amazon)
(followup on other platforms forthcoming)
Solution: for compound types, recursively apply that check
Two new cases are added
1) the compound type has a different number of members. Message printed is
<obj1> has X members <obj2> has Y members
Where X and Y are the number of members of each compound type being compared
2) the compound type has not comparable types (for example a double and an int at the same index)
In this case the message
Comparison not possible: object1 is of class1 and object2 is of class2
Is replaced with
Comparison not possible: object1 has a class1 and object2 has a class2
Modified the test generator program to have these 2 cases
Added a shell run for these 2 cases
Tested: windows, h5committest
PG compiler complains about array out of bounds (a rank of zero was not checked)
Adding a scalar dataset to the test generator program. this case is run on a previous existing run, the case was added to 2 existing files
Tested: windows, linux
Remove trailing whitespace from C/C++ source files, with the following
script:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Tested on:
Mac OS X/32 10.5.5 (amazon)
No need for h5committest, just whitespace changes...
bug fixes.
Description:
Added code to create an empty hdf5 (named h5diff_empty.h5) in order to test
if h5diff compares correctly an empty hdf5 vs. a non-empty one.
Tested:
Tested in kagiso of h5diffgentest itself.
Verified by h5dump that h5diff_empty.h5 was indeed empty.
Then "h5diff h5diff_empty.h5 h5diff_basic1.h5" returned 0 (should have
returned non-zero).
