Bring r17266 from 1.8 branch to trunk:
Clean up various compiler warnings from IRIX64 build
Tested on:
Mac OS X/32 (amazon) debug & production
(yes, I know it's not an IRIX64 system :-)
Too minor to require h5committest
Fix BZ #1583
Description:
The --enable-static-exec flag was broken. Configure was adding the
-all-static linker flag to an environment variable which was subsequently
never used. (looks like it was used in 1.6 Makefile.ins, but never
moved to 1.8's Makefile.ams when we added automake support).
To fix, I've added this environment varibale to the link lines of all the
tools. Now, when --enable-static-exec is invoked, it will correctly
generate statically linked executables in the installed bin directory.
Tested:
I've tested manually to ensure that the generated execs are static when
the flag is used and dynamic when not. I've also tested when szip is
used, as this was a use case mentioned in the bug report.
Also: ran full tests with make check on jam and smirom for good measure.
added a test to the C program test (not to the script), that verifies the layout and filters
tested: windows (development and tested manually) , linux, solaris
The file used for input is located in the common source tools for testfiles, in tools/testfiles
Modified the h5repack shell script to read files from this location (h5repack reads its input files from a dedicated testfiles location in h5repack/testfiles)
Changed the h5diff open file call to use h5tools_fopen, so that it can open all file drivers
Tested: windows, linux, solaris
Adding dependency to h5repack test script.
h5repack.sh needs h5repacktst to run first.
Tested:
Problem only appeared on Glory, but tested the dependence
manually on jam by trying to run only the script and checking
that it first ran the h5repacktst test program.
Description:
h5repack previously would not take named datatypes into consideration when copying
datasets and attributes. This would cause extra anonymous datatypes in the target file
at best, and cause errors halfway through the repacking at worst. h5repack should now
always handle named datatypes correctly. Named datatypes are also now converted to the
native type when -n is given.
Tested: jam, linew, smirom (h5committest)
- Updated bin/reconfigure to use latest version of automake (1.10.2).
Re-generated Makefile.in's by running bin/reconfigure.
- Added libtool version numbers to c++, fortran, hl, hl c++, and hl fortran
libraries.
Tested:
jam, liberty, smirom
ISSUE : h5repack does not handle group creation order flags.
ACTION: call H5P(g)(s)et_link_creation_order functions when handling groups, add new groups with these flags to the test generation program, and verify results in the test program.
TEST: in the test program, function that compares property lists, added code to verify groups
tested: windows, linux, solaris
ISSUE : the tools use the following formula to read by hyperslabs: hyperslab_size[i] = MIN( dim_size[i], H5TOOLS_BUFSIZE / datum_size) where H5TOOLS_BUFSIZE is a constant defined of 1024K. This is OK as long as the datum_size does not exceed 1024K, otherwise we have a hyperslab size of 0 (since 1024K/(greater than 1024K) = 0). This affects h5dump. h5repack, h5diff
SOLUTION: add a check for a 0 size and define as 1 if so.
TEST FOR H5DUMP: Defined a case in the h5dump test generator program of such a type (an array type of doubles with a large array dimension, that was the case the user reported). Since the written file commited in svn would be around 1024K, opted for not writing the data (the part of the code where the hyperslab is defined is executed, since h5dump always reads the files). Defined a macro WRITE_ARRAY to enable such writing if needed. Added a run on the h5dump shell script. Added 2 new files to svn: tools/testfiles/tarray8.ddl, tools/testfiles/tarray8.h5. NOTE: while doing this I thought of adding this dataset case to an existing file, but that would add the large array output to those files (the ddls). The issue is that the file list is increasing.
TEST FOR H5DIFF: for h5diff the check for reading by hyperslabs is H5TOOLS_MALLOCSIZE (128 * H5TOOLS_BUFSIZE) or 128 Mb. This makes it not possible to add such a file to svn, so used the same method as h5dump (only write the dataset if WRITE_ARRAY is defined). As opposed to h5dump, the hyperslab code is NOT executed when the dataset is empty (dataset is not read). Added the new dataset to existing files and shell run (tools/h5diff/testfiles/h5diff_dset1.h5 and tools/h5diff/testfiles/h5diff_dset2.h5 and output in tools/h5diff/testfiles/h5diff_80.txt).
TEST FOR H5REPACK: similar issue as h5diff with the difference that the hyperslab code is run. Added a run to the shell script (with a filter, otherwise the code uses H5Ocopy).
tested: linux (h5commitest failed , apparently it did not detect the code changes in /tools/lib that fix the bug: the error in an assertion in the hyperslab of 0. I am sure that making h5ccomitest --distclean will detect the new code , but don't want to wait more 3 hours :-) )
Bug Fix
Description:
Fixing BZ #1381. The --includedir=DIR configure option, which is used
to specify the installation location of C header files, did not work
correctly as the path was hard-coded in config/commence.am. I'm presuming
this is because an older version of automake didn't know where to put
c header files. In any case, removing this line now defaults the includedir
to the same directory that it is currently hard-coded to, and also fixes
the configure flag to allow customization of this value.
Tested:
jam, liberty
Remove another call to H5E_clear_stack() from within the library.
Clean up lots of compiler warnings.
Tested on:
Mac OS X/32 10.5.6 (amazon)
(followup on other platforms forthcoming)
the way h5ls prints types, it starts searching for NATIVE types first. One solution would be h5ls not to detect these native types, using for example the same print datatype function that h5dump does, that would make the output look the same on all platforms ("32-bit little-endian integer" would be printed instead). Drawback, this "native" information would not be available. Other solution is to have not one but 2 expected outputs and make the shell script detect the endianess and compare with one output or other
tested: h5committest
Correct error introduced in r16353 with layout version, and add test
so it gets caught earlier.
Tested on:
FreeBSD/32 6.3 (duty)
Too minor to require h5committest
Description:
- Remove need to set LD_LIBRARY_PATH when using shared szip library.
- Libtool 2.2.6a is now used to generate libraries.
- 'make check install' dependency bug is fixed, and should no longer
break the build.
- removed hard coding of shell in config/commence.am, as this causes
problems on Solaris with the new version of libtool.
- RELEASE.txt with appropriate changes.
Tested:
- kagiso, smirom, linew (merged from 1.8, pretty quick tests)
Remove trailing whitespace from C/C++ source files, with the following
script:
foreach f (*.[ch] *.cpp)
sed 's/[[:blank:]]*$//' $f > sed.out && mv sed.out $f
end
Tested on:
Mac OS X/32 10.5.5 (amazon)
No need for h5committest, just whitespace changes...
-t T, --threshold=T Threshold value for H5Pset_alignment
-a A, --alignment=A Alignment value for H5Pset_alignment
2) bug fix
the printing of the dataset name was not done for references (verbose mode)
tested: windows, linux
Add a userblock to an HDF5 file during the repack. The user gives
give a filename and userblock size as command line parameters to
h5repack and the contents of that file are stored in the
userblock for the HDF5 file created by h5repack.
New flags to handle this -u and -b
Tested : windows, linux
datatype versions are encountered.
Description: The library now recognizes some problems with datatype versions in
H5O_decode_helper(), and, if not performing strict format checks, automatically
corrects them. Framework added for other message decode routines to
automatically correct file errors. Datatype version information added to
h5debug.
Tested: kagiso, smirom, linew (h5committest)
Summary: when using h5diff to compare the results of h5repack (or other tools that copy one HDF5 file to another), a new option is needed to allow h5diff to make an "absolute" comparison of the 2 files. This is the "contents" mode explained in the usage below.
If this mode is present, objects in both files must match (must be exactly the same). If this does not happen, the tool returns an error code of 1 (instead of the success code of 0)
Changes to the h5repack test script: the call to h5diff was changed to include -c (maintaining the previous -q).
tested: windows, linux, solaris
the parsing of the command line strings for the shared object header message type was not being done correctly (strcmp)
fix: substituted by strncmp
tested: windows, linux
if these are detected this syntax is used, otherwise the one in usage is used
there was another -i option for
-i L2, --indexed=L2 Minimum number of links in the indexed format
That was changed to -d
-d L2, --indexed=L2 Minimum number of links in the indexed format
Tested: windows, linux
Description:
The fortran Makefile.am used HDF_FORTRAN to indicate it is part of the
Fortran API source so that conclude.am will give fortran api prefix in the
test output. The symbox HDF_FORTRAN is also used in configure for a different
purpose (indicated --enable-fortran). They conflicted.
Similar problem for the symbol HDF_CXX.
Solution:
Changed all the involved Makefile.am to use "FORTRAN_API" instead. It is
a more appropriate name. Same for CXX_API.
Along the way, discovered that the Makefile.am of hl/fortran/test and
hl/cxx/test did not have those symbols at all. Added them in.
Platform tested:
Kagiso only. It is a trivia change.
Improvement.
Description:
src/libhdf5.settings was the initial configure summary and is installed.
Then configure is changed to dump a summary of the configure settings to
the output and also append it to src/libhdf5.settings. That created
two different output formats and duplicated information. This is the
initial attempt to clean up this confusion and unify the output format.
It is decided to use the src/libhdf5.settings template as the unified means.
This requires more macros symbols be defined. The following symbols are
all related to generating the src/libhdf5.settings file.
AC_SUBST(EXTERNAL_FILTERS)
AC_SUBST(MPE) MPE=no
AC_SUBST(STATIC_EXEC) STATIC_EXEC=no
AC_SUBST(HDF_FORTRAN) HDF_FORTRAN=no
AC_SUBST(FC) HDF_FORTRAN=no
AC_SUBST(HDF_CXX) HDF_CXX=no
AC_SUBST(CXX) HDF_CXX=no
AC_SUBST(HDF5_HL) HDF5_HL=yes
AC_SUBST(GPFS) GPFS=no
AC_SUBST(LINUX_LFS) LINUX_LFS=no
AC_SUBST(INSTRUMENT) INSTRUMENT=no
AC_SUBST(CODESTACK) CODESTACK=no
AC_SUBST(HAVE_DMALLOC) HAVE_DMALLOC=no
AC_SUBST(DIRECT_VFD) DIRECT_VFD=no
AC_SUBST(THREADSAFE) THREADSAFE=no
AC_SUBST(STATIC_SHARED)
AC_SUBST(enable_shared)
AC_SUBST(enable_static)
AC_SUBST(UNAME_INFO) UNAME_INFO=`uname -a`
The src/libhdf5.settings.in has CONDITIONAL's added to it too. The
untrue conditions turned into a "#" and these lines are cleaned by the
post processing script.
Platform tested:
h5committest on kagiso, smirom and linew.
Description: Applying update to autotools that was applied to 1.8 a couple
of weeks ago to the trunk.
Updated bin/reconfigure script to reflect the new versions of
libtool and automake in the /home1/packages/ directory.
Rearranged configure.in script. When using libtool 2.2.2, the
libtool script doesn't generate until later in the configuration
process, so I had to move a test that parsed through the libtool
script to a point after where it was actually being generated.
Ran libtoolize on the project, and ran bin/reconfigure to
regenerate configure and Makefile.in's throughout.
Tested: kagiso, smirom, linew (h5committest)
Add work-around to allow reading files that were produced with a buggy
earlier version of the library, which could create objects with the wrong
object header message count. There is now a configure flag
"--enable-strict-format-checks" which triggers a failure on reading a file
with this sort of corruption (when enabled) and allows the object to be read
(when disabled). The default value for the "strict-format-checks" flag is
yes when the "debug" flag is enabled and no when the "debug" flag is disabled.
Note that if strict format checks are disabled (allowing objects with
this particular kind of corruption to be read) and the file is opened with
write access, the library will re-write the object header for the corrupt
object with the correct # of object header messages.
This closes bugzilla bug #1010.
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.4.10 (amazon) in debug mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
TO DO: szip, nbit and scale offset
NOTE: the symbol H5Z_SHUFFLE_TOTAL_NPARMS was made public
Tested: windows, teragrid with icc 8.1, linux (kagiso), solaris (linew)
Tweak the constants for the shared message flags to be equal to the
other flags useds for the underlying messages.
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.4.10 (amazon) in debug mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
changed the logic of the filter compare function to first compare filter type, then filter client data values (separate here in each individual case, since some filters return private filter values. Todo: handle the nbit and scale-offest cases)
tested: windows, linux
Following the new feature of h5repack to allow multiple filters for all datasets and the new function has_filters that checks if the repacked file has all the filters requested, I added a new function
has_filters_obj
that does the same for each dataset. The previous function that checked this only ckecked if the user input filters were in the output dataset. This new function does this but checks if the filters are exactly the same. Currently the behavior of h5repack is to delete all filters that are present in the input file (dataset) and replace them with the requested ones, so they must match exactly.
We might consider adding other logical operations, like keep the existing ones.
Additionally , the function also checks if the filter parameters match.
While doing this I noticed that for the shuffle filter , the values returned do not match and also the same for the N-bit and scale-offset
The new function that checks for the filter values fails then, and so I commented the h5repack tests that do this for the N-bit and scale-offset filter (previously for the same bug on the shuffle filter I added special code on the compare filter function but this is temporary until I find the issue)
tested: windows, linux, solaris
now the minus sign shows there was a DECREASE in compression
percentage is calculated from
per = (b-a)/a;
where a= size of dataset before
b = size after
tested: windows, linux
Change H5Oget_info -> H5Oget_info_by_name and re-add H5Oget_info in a
simpler form for querying a particular object, to align with other new API
routines.
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.4.10 (amazon) in debug mode
Changed H5Acreate2 -> H5Acreate_by_name, to be more consistent with
other new API routines.
Re-added simpler form of H5Acreate2, which creates attributes directly
on an object.
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.4.10 (amazon) in debug mode
Make H5Tarray_create() and H5Tget_array_dims() versioned, and drop the
"perm" parameter from the '2' versions.
Shift internal library usage to '2' versions.
Add simple regression tests for '1' versions.
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.4.10 (amazon) in debug mode
Remove --enable-hdfv1_6 configure flag, now that all dependencies on it
have been removed and new --with-default-api-version=[v16|v18] and
--disable-deprecated-symbols flags have taken its place.
Update types in bin/trace so that bin/reconfigure runs cleanly.
Run bin/reconfigure to regenerate configure script, along with
other associated files, etc.
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.4.10 (amazon) in debug mode
Make H5Pget_filter API versioned and switch internal usage to
H5Pget_filter2.
Add regression test for H5Pget_filter1.
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.4.10 (amazon) in debug mode
Make H5Dopen versioned and change all internal usage to use H5Dopen2
Add simple regression test for H5Dopen1
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.4.10 (amazon) in debug mode
