HDFFV-8214 - h5repack failed converting small chunked dataset (size < 1K) to contiguous layout.
Description:
h5repack failed converting small chunked dataset (size < 1K) to contiguous layout.
The first case was when chunk dim is bigger than the dataset dim (at leat one), h5repack failed with displaying error stacks.
The other case is when chunk dim is smaller than the dataset dim, h5repack failed to change layout.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), emu (solaris-BE),fred (mac64-LE), Windows (32-LE cmake), cmake (jam)
HDFFV-8012 - h5repack changes max dims and cause failure if only "-f none" is used without changing layout for chunked dataset when a chunk dim is bigger than a dataset dim
Description:
"h5repack -f <obj>:NONE <file.h5> out.h5" command failed if source file contains chunked dataset and a chunk dim is bigger than a dataset dim.
Another issue is that the command changed max dims if chunk dim is smaller than the dataset dim.
These issue occurred when dataset size is smaller than 64k (compact size limit)
Fixed them.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE cmake), cmake (jam)
Changes resulting from Klocwork static analysis tool, from Mark Miller
@ LLNL (miller86@llnl.gov).
Tested on:
Mac OS X/64 10.7.4 (amazon) w/debug, C++ & FORTRAN, using gcc 4.7.x
(too minor to require h5committest)
Fix for HDFFV-8107 testh5diff will fail if build/test in HDF5 source tree
Description:
This is sub-task for "HDFFV-8105 testh5diff.sh uses the wrong operator (-a) in an if statement."
From the HDFFV-8105's update, h5diff test failed if build&test is performed in HDF5 source tree because 'cp' try to copy test files to self dir.
It's addressed by skipping if cp's src dir and dest dir is same.
Also this applied for all other tools under src/tools dir.
No change to the CMakeLists.txt files because CMake cautions/demands that in-source builds be avoided.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), some manual tests as well
Fix h5repacktst (test file generating binary) to address HDFFV-8016 h5repacktst failed in AIX 32bit mode
Description:
Segmentation fault occurred when h5repacktst try to generate a dataset when
system memory is limited to 256MB.
Add error handling to display apporiate message to exit and adjust the test
dataset according to the original size which didn't causing out of memory.
Tested: test was done by limit memory in csh.
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE)
Fix for HDFFV-7993 - h5repack fails with error "chunk size must be <= maximum dimension size for fixed-sized dimensions"
Description:
Fixed a failure when change the chunk size of a specified chunked dataset with unlimited max dims.
Also took care of converting to contiguous and compact from the dataset.
Test cases were added and tagged with jira#.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE cmake), Cmake (jam)
Clean up a few warnings and use constant values for free-space
parameters for the metadata block size test.
Tested on:
Linux/64 2.6.18 (koala) w/production & szip
(h5committested also)
Add new "metadata block size" command line option ('-M <x>' or
'--metadata_block_size=<x>') for h5repack.
Tested on:
Mac OSX/64 10.7.3 (amazon) w/debug)
(h5committest upcoming)
Task for HDFFV-7862 - Select data by chunk direction to improve performance in h5repack
Description:
h5repack sometimes became very slow when handling big chunked datasets in
certain cases. (when chunk boundary doesn't match with a hyperslab boundary.)
The main issue was from figuring out a hypeslab without considering chunk
boundary to read from and write to such datasets.
The update was made to figure out a better hyperslab unit with considering
chunk boundary to improve performance for such cases prior to the update.
Tested:
jam (linux32-LE), koala (linux64-LE), ostrich (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows
HDFFV-7602 - GMQS: HDF5 command tools: Provide framework for reusable test files among tools
Stage2 working for h5repack cmake test.
Description:
Update cmake script to clean up correctly.
Revisit h5repack cmake test script updated from r21869.
Tested:
cmake test for Windows (32-LE) and jam (linux32-LE)
Fix for HDFFV-7840 h5repack: memory leak over one of the h5diff test file
Description:
Turned out that there were two causes of memory leaks.
1. for handling variable length string in attribute.
2. for handling compound type with non-reference members.
The first issue is fixed in copy_attr() which is updated to use h5tools_detect_vlen to take care of vlen string as well as vlen data.
The second is fixed in copy_refs_attr() of compound handling code.
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Windows (32-LE), Cmake (jam)
Added a funtion to reset dataset & hyperslab buffer size for h5repack from an
environment variable.
This is performance debugging purpose for now.
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), Windows (32-LE), cmake
h5dump was used in test script but was not invoked by RUNSERIAL.
THis does not work in batch machines like Blue Gene in LLNL.
Solution:
Added $RUNSERIAL to invoke $H5DUMP_BIN.
Tested:
LLNL BlueGene (udawn)
Ran bin/reconfigure to update the Makefile.in in directories not part of the fortran directory check=in. Updates Makefile.in due to changes made in configure.in for the Fortran 2003 additions.
Tested on all platforms run under daily tests.
Correct char arrays that were too small for strings defined. Also, change
cleanup to work correctly.
Tested on:
Mac OS X/32 10.6.8 (amazon) w/debug
FreeBSD/32 8.2 (loyalty) w/debug
FreeBSD/64 8.2 (freedom) w/debug
Linux/64 2.6 (heiwa) w/debug
Linux/64 2.6 (ember) w/parallel
Switch from H5P_DATASET_ACCESS_DEFAULT to H5P_DEFAULT for calls to
H5Rdereference2().
Tested on:
Mac OS X/32 10.6.8 (amazon) w/debug
(too minor to require h5committest)
- h5repack: h5repack failed to copy dataset if the layout is changed from chunked with
unlimited dims to contiguous. (PC -- 2011/07/15)
- h5diff: "--delta" option considers two NaN of the same type are different, which is wrong
based on http://www.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Diff. (PC -- 2011/07/15)
Purpose:
Work for HDFFV-7602 - HDF5 command tools: Provide framework for reusable
test files among tools
Description:
Provide framework to share test files among tools for tools test.
Tested:
jam (linux32-LE), koala (linux64-LE), heiwa (linuxppc64-BE)
General shared library improvements for CYGWIN / AIX
Description:
Shared libraries are disabled on both CYGWIN and AIX due
to inability to build them correctly. Part of the problem
in both of these situations is the lack of the libtool
flag -no-undefined, which tells libtool that all needed
symbols are defined at link time (a requirement on these
systems) and that it's okay to build shared libraries.
Another problem are lack of dependencies between wrapper
libraries and core C HDF5 library.
This patch addresses both of these by fixing configure to
add in -no-undefined flag for libtool during linking and
adds automake dependencies in the Makefile.am files.
After testing, both CYGWIN and AIX now generate shared
libraries, but there are still some test failures in each.
(cache_api, dt_arith, and testerror.sh on CYGWIN, and
fortran tests on AIX).
Even though the shared libraries are not quite perfect,
this is a general improvement to what we had before, so
I'm applying the patch anyways. Note that default behavior
of shared libraries on these systems being disabled has
NOT been changed and requires the use of the
--enable-unsupported to attempt to build them.
We will need to address the test failures in each
architecture prior to formally supporting shared
libraries on each.
Tested:
h5committested & CYGWIN tested (on bangan)
(AIX tested by Albert on bp-login2)
Add "silent make" mode configure option.
Description:
Automake 1.11 has a new option available that allows for a
silent make mode. This functionality needs to be explicitly
enabled in configure.in via the use of the automake macro
AM_SILENT_RULES, which is what this commit is adding.
This introduces a new configure option:
--{en|dis}able-silent-rules
This option is on by default, and simplies compile and link
line outputs when building the library. Disabling this option
will print full "verbose" output (i.e., full compile and
linking lines for each target).
Tested:
This was tested on jam & h5committested
- Revise shared Fortran library disabling scenarios in configure
- Improve configure output summary
Description:
Shared Fortran libraries are not supported on Mac, but were being
disabled by configure in a way that also forced the C libraries
to be static-only. This has been fixed, so now only shared Fortran
is disabled while shared C can remain.
This prompted two additional changes:
1. While working on the check that addresses whether or not
shared Fortran libraries are allowed, removed old and no
longer needed check(s) that disable shared Fortran
libraries with HP, Intel 8, PGI, and Absoft compilers.
(Essentially, Mac is the only situation in which Fortran
shared are disabled by configure.)
2. Having two different states of libraries (i.e. shared C
library with static-only Fortran library) was not apparent
in the configure summary, which labeled all libraries as
either shared and/or static. I've added lines to both the
C++ and Fortran output sections to list shared/static-ness
of these libraries specifically.
Additionally, I've made sure that the new --enable-unsupported
configure option correctly overrides configure if it tries to
disable a shared library.
Tested:
jam, fred, & h5committest
Change to use HDxxx macros.
Description:
Originally this started for fixing incorrect pointer usage. But that got
fixed through coverity merge. So this is mainly changing to use HDxxx
macros and clean up some related code.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE)
Bring changes from Coverity branch to trunk:
r19930:
Fix memory leaks involving VL attributes in h5repack and h5diff. The buffers in
copy_attr and diff_attr were not checked for the presence of a vlen before being
freed, and vlen storage was never reclaimed. Added checks and calls to
H5D_vlen_reclaim().
r19933:
Purpose: Fix memory leak in H5L_move_cb()
Description: H5L_move_cb copied the source link using H5O_msg_copy() but freed
it manually using H5MM_xfree(). Since H5O_link_copy allocates the link using
H5FL_MALLOC, this causes the link to be allocated from the free list but is
never put back on the free list when it is freed. This prevents the link free
list from shutting down properly. Modified H5L_move_cb() and H5L_move_dest_cb()
to free the link properly using H5O_msg_free().
r19973:
Fix resource leaks by freeing string created by HD5f2string
r19974:
Issue #345: Inialize buf variable to null
Tested on:
Mac OS X/32 10.6.6 (amazon) w/debug & production
(h5committested on Coverity branch)
Clean up Coverity warnings, and fix some style issues:
r19735:
Fix for memory leak in test/mf found by valgrind.
r19736:
Fix memory leak in h5repack. The buffer in copy_objects, when copying the
entire dataset at once, was not checked for the presence of a vlen, and vlen
storage was never reclaimed. Added check and call to H5D_vlen_reclaim().
r19772:
Change H5assert() to
if (H5T_VLEN != src->shared->type || H5T_VLEN != dst->shared->type)
HGOTO_ERROR(H5E_ARGS, H5E_BADTYPE, FAIL, "not a H5T_VLEN datatype")
r19774:
removed unused priv.
r19775:
removed unused variables
r19778:
Fix memory leak comparing for variable length data types.
r19834:
Fixed memory leaks found by valgrind. Memory errors remain for another day.
Tested on:
Mac OS X/32 10.6.6 (amazon) w/debug & production
(h5committested on branch)
Clean up tool code.
Remove "h5test.h" from tool code. Currently only in h5repack.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), linew (solaris-BE)
Additional fix for Bug 1821 - h5repack outputs compression information where it is not supposed to
Description:
Remove unnecessary lopping code, which only lowers performance.
Tested:
jam, amani, heiwa
Fix for Bug1896 h5repack - changing layout to COMPACT does not work
Description:
Make h5repack be able to convert a layout to COMPACT for small size dataset as default. Also add verifying layout changes in our test script.
Tested:
jam, amani, heiwa
Corrected use/name of source folder aliases.
Duplicated FindMPI.cmake so that non-c++ compiler is found first (recommemded commands did not work).
Tested: local linux with mpich
Clean up misc. compiler warnings.
Tested on:
FreeBSD/32 6.3 (duty) in debug mode
Mac OS X/32 10.6.4 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
Mac OS X/32 10.6.4 (amazon) w/parallel, in debug mode
Bring changes from Coverity branch back to trunk:
r19079 & 19080:
[BZ1942] h5dump -u to generate XML, it does not respect the -m option
xml version of dump_data function didn't check for use of fp_format variable.
Added new test expected file for committed bug 1942
r19103, 19104 & 19105:
[BZ1821] h5repack -v did not display correct output for a selected compression. Needed new test for comparing output of -v option.
Added new test file for solution to BZ1821
BZ1821 - Bring test changes from the shell script actually used.
Tested on:
Mac OS X/32 10.6.4 (amazon) debug & production
(h5committested on branch)
