Purpose:
Remove H5_MPI_SPECIAL_COLLECTIVE_IO_WORKS and
H5_MPI_COMPLEX_DERIVED_DATATYPE_WORKS #defines from source.
Description:
Two advanced parallel functionalities, special collective IO and
complex derived datatypes, are not supported by older
implementations of mpi, and thus our code limits the use of these
features with #ifdefs and has checks in configure to set them (or
not). Unfortunately, configure can't actually run a parallel check
to see if these features are working (nor not) so it resorts to
looking in the config files where they are explicity enabled or
disabled based on versions of mpi, sytems being built on, or for
no documented reason at all (i.e. just set to on or off as some
'default'). Overriding these settings is easy if need be, provided
it is known that it needs to be done to get improved performance,
and oftentimes it is not.
Most new MPI implementations successfully handle the functionality
requested when these #defines are set, and many of the "turn these
features off" cases in the config files are for old (> 5 years)
versions of MPI and retired systems (such as NCSA's tungsten).
Therefore, the decision has been made to remove the support for
these old versions of MPI and systems that cannot handle these
behaviors. The #ifdefs and supporting setup in the config/ files
and configure script has been removed, and the code executed when
these options were not set removed from the source.
In passing, this commit also cleans up some whitespace issues in
both t_mpi.c and H5Dmpio.c. Furthermore, in t_mpi.c, the special
collective IO test was not getting regularly run due to it being
written to work only with four processes (we regularly test with
six, previously with three), and thus it failed when actually run
due to an out of bounds data buffer assignment. It has been
modified to run at any number of processes greater than four, and
the memory problem has been fixed so the test passes.
Tested:
jam, h5committest, ember
removed option defines and #ifdef/#endif
refactored all printf to HDfprintf in h5dump.c
formatted and indention improvements
synched with 1.8 branch
Tested: local linux
configure.in that tried to distinguish different versions of AIX. This is no
longer needed.
Tested:
NCSA Blue-print. No h5committest since this affected AIX system only.
Remove old, uneeded configuration customizations for FreeBSD systems.
(config/freebsd)
Also, convert a large stack variable into a dynamic memory allocation
and clean up a few compiler warnings (tools/h5dump/h5dumpgentest.c)
Tested on:
FreeBSD/32 8.2 (loyalty) w/debug & production
FreeBSD/64 8.2 (freedom) w/debug & production
General shared library improvements for CYGWIN / AIX
Description:
Shared libraries are disabled on both CYGWIN and AIX due
to inability to build them correctly. Part of the problem
in both of these situations is the lack of the libtool
flag -no-undefined, which tells libtool that all needed
symbols are defined at link time (a requirement on these
systems) and that it's okay to build shared libraries.
Another problem are lack of dependencies between wrapper
libraries and core C HDF5 library.
This patch addresses both of these by fixing configure to
add in -no-undefined flag for libtool during linking and
adds automake dependencies in the Makefile.am files.
After testing, both CYGWIN and AIX now generate shared
libraries, but there are still some test failures in each.
(cache_api, dt_arith, and testerror.sh on CYGWIN, and
fortran tests on AIX).
Even though the shared libraries are not quite perfect,
this is a general improvement to what we had before, so
I'm applying the patch anyways. Note that default behavior
of shared libraries on these systems being disabled has
NOT been changed and requires the use of the
--enable-unsupported to attempt to build them.
We will need to address the test failures in each
architecture prior to formally supporting shared
libraries on each.
Tested:
h5committested & CYGWIN tested (on bangan)
(AIX tested by Albert on bp-login2)
- Revise shared Fortran library disabling scenarios in configure
- Improve configure output summary
Description:
Shared Fortran libraries are not supported on Mac, but were being
disabled by configure in a way that also forced the C libraries
to be static-only. This has been fixed, so now only shared Fortran
is disabled while shared C can remain.
This prompted two additional changes:
1. While working on the check that addresses whether or not
shared Fortran libraries are allowed, removed old and no
longer needed check(s) that disable shared Fortran
libraries with HP, Intel 8, PGI, and Absoft compilers.
(Essentially, Mac is the only situation in which Fortran
shared are disabled by configure.)
2. Having two different states of libraries (i.e. shared C
library with static-only Fortran library) was not apparent
in the configure summary, which labeled all libraries as
either shared and/or static. I've added lines to both the
C++ and Fortran output sections to list shared/static-ness
of these libraries specifically.
Additionally, I've made sure that the new --enable-unsupported
configure option correctly overrides configure if it tries to
disable a shared library.
Tested:
jam, fred, & h5committest
Fix Bug 2120 - h5copy: improve to copy an object into same HDF file
Description:
Before the fix users can't copy an object (dataset or group) with
different name if input file (-i) and output file (-o) is same.
This feature would be useful to clone any object with different name and
reuse contents.
Tested:
jam (linux32-LE), amani (linux64-LE), heiwa (linuxppc64-BE), tejeda (mac32-LE), linew (solaris-BE), Cmake (jam)
mpif90 is already the default parallel Fortran if mpicc is the C compiler.
Tested: Jam (serial and parallel). No need for committest since this change is
limited to Linux system.
Also updated root CMakeLists.txt to output a message when unsopported options are configured with the PARALLEL option. CMake will still generate files.
Tested: windows and local linux
The parallel test ran out of memory because 32bit binary default to use
less memory. Changed RUNPARALLEL to use larger memory (LDR_CNTRL=MAXDATA=0x2000
0000@DSA).
Tested: BP parallel.
RUNSERIAL=${RUNSERIAL="env LDR_CNTRL=MAXDATA=0x20000000@DSA"}
kind of mess things up for the parallel HDF5 configure.
Changed it to
RUNSERIAL=${RUNSERIAL="env LDR_CNTRL=MAXDATA=0x20000000@DSA MP_PROCS=1 MP_TASKS_PER_NODE=1 poe"}
if --enable-parallel.
Also merge powerpc-ibm-aix5.x with ibm-aix by making powerpc-ibm-aix5.x to
source ibm-aix just like ibm-aix6.x. This is in preparation that all AIX
platforms will eventually just use ibm-aix directly.
Fixed a small problme in ibm-flags: some older xlc (e.g. v7.1) prints the
version information with a leading blank. So, accommodate this pattern by
removing the "^".
Tested: LLNL uP, 32/64bits, serial/parallel. (parallel has tests failed
but this change is mainily for configure issues.)
Also tested BP, 32/64bits but serial only.
(Forgot to commit this file when bin/config.guess was committed.)
Description:
In an AIX 6.1 system, configure --enable-shared could not build a shared
HDF5 library. The problem was because the version of config.guess was too
old to recongnize AIX 6.X and also configure.in had a local fix which did
not recognize AIX 6.X.
Solution:
1. Mike McGreevy updated bin/config.guess to handle AIX 6.X.
2 Albert fixed configure.in to recognize AIX 6.X.
Note that though HDF5 can build shared lib for AIX 6.X systems
but it still could not install the proper library as in AIX 5.X
systems. Also, bin/config.sub should be updated too.
Tested: BP which is the AIX 6.1 system that exposed this problem.
unknown versions of the C Intel compilers.
Modified apple config file to display Intel compiler information.
Updated RELEASE.txt notes accordingly.
Platfroms tested: jam, amani and dubna (Mac 64-bit) with Intel C, Fortran and C++ compilers.
Description:
The development (--disable-production) mode ran a long time, especially on the
testhdf5 (select) that exceeded the wall clock 10 minutes limits and was
terminated. It turned out the development had DEBUG_CFLAG set to "-g -xildoff".\
The -xildoff was added back when h5detect had errors when -g only was used. But
that bug does not exist any more since when only -g is used to build, h5detect
did not fail and all tests passed. Therefore -xildoff is removed.
Furthermore, when "-g -O" is used to build the library, debuggers such as
dbx can still use the executable, though some debugging ability like displaying
local variables is not available in optimized code. But overall execution time
shorten.
Therefore, the C, Fortran aand CXX DEBUG flags are all set to "-g -O" by
default.
Tested: in Linew using the default development mode because the changes affect
only the Sun platform.
Description:
testhdf5/select runs very slow when in --disable-production mode using
pgcc compiler. This was caused by the -Mchkfpstk compiler switch added
to the DEBUG_FLAGS in config/pgi-flags file.
Solution:
Removed -Mchkfpstk from both pgi-flags and pgi-fflags.
See Bug 2026 for dettails.
Tested: (no h5committest test since they don't use PGI compilers.)
Jam (linux32): using pgcc/pgf90/pgCC using default and --enable-production.
Amani (linux64):
using pgcc/pgf90/pgCC (32 bits mode by "-tp k8-32"), using default and --enable-production.
Corrected use/name of source folder aliases.
Duplicated FindMPI.cmake so that non-c++ compiler is found first (recommemded commands did not work).
Tested: local linux with mpich
Bump optimization down for Intel 10.* compilers, due to apparent bad cod
e generation.
Tested on:
Linux/64 2.6 (cobalt) w/debug & production & Intel 11.1 compilers
(h5committest not appropriate)
Description: aix -q32 default to a memory limit that some HDF5 tests would
fail in malloc.
Solution:
Temporary hardset
RUNSERIAL=${RUNSERIAL="env LDR_CNTRL=MAXDATA=0x20000000@DSA"}
to bypass this problem until the test programs are fixed. Though -q64 binary
has a higher default memory limit and does not need this patch, it is applied
the same for the sake of simplicity since it is a temporary fix.
Tested:
in uP, both -q32 and -q64 modes. No h5committest since this applies to the
AIX platform only.
Description: aix -q32 default to a memory limit that some HDF5 tests would
fail in malloc.
Solution:
Temporary hardset
RUNSERIAL=${RUNSERIAL="env LDR_CNTRL=MAXDATA=0x20000000@DSA"}
to bypass this problem until the test programs are fixed. Though -q64 binary
has a higher default memory limit and does not need this patch, it is applied
the same for the sake of simplicity since it is a temporary fix.
Tested:
in BP, both -q32 and -q64 modes. No h5committest since this applies to the
AIX platform only.
Corrected use of "SKIP" message in cmake.
Corrections to cmake configuration files (for cmake external projects).
Community suggested changes.
Tested: local linux
Bring changes from Coverity branch back to trunk:
r19079 & 19080:
[BZ1942] h5dump -u to generate XML, it does not respect the -m option
xml version of dump_data function didn't check for use of fp_format variable.
Added new test expected file for committed bug 1942
r19103, 19104 & 19105:
[BZ1821] h5repack -v did not display correct output for a selected compression. Needed new test for comparing output of -v option.
Added new test file for solution to BZ1821
BZ1821 - Bring test changes from the shell script actually used.
Tested on:
Mac OS X/32 10.6.4 (amazon) debug & production
(h5committested on branch)
Added ibm-flags to display the IBM XL compilers version information so that configure
can report them in the configure summary and the libhdf5.settings file.
For now, only the C compiler (xlc) versions are displayed.
Also moved the C compiler options setting from ibm-aix to ibm-flags.
Tested: NCSA Blue-print, both 32 and 64 bit modes.
Improve configure's large-file support control.
Description:
Modified configure to now attempt to add defines necessary for
supporting largefiles on all systems, instead of solely on linux. This
is in response to user requests to enable largefile support on Solaris
by default, as well as to give extra control on AIX (instead of just
jamming the necessary flag into the config files).
The old --enable-linux-lfs flag has been removed in favor of the
--enable-largefile flag (enabled by default), which can be used on all
platforms.
On systems where large files cannot be supported in this manner,
configure will report as such.
Tested:
h5committest
AIX (NCSA's blue_print machine)
duty, liberty, and linew.
When gcc is used in the AIX platform, it failed to compile tconfig.c
because it does not recognize the __int64 type. The ibm-aix has the
sizeof __int64=8 cached, as if all compilers supported this type.
Solution:
Removed the cached value for the sizeof __int64 so that during
configure, gcc can report it does not support it and not to use
it in the source code.
Tested:
NCSA Blue which is an AIX machine, using gcc or xlc compiler.
(The gcc still have failure but is being fixed as separated bugs.)
The mixed use of RUNTEST (original) and RUNTESTS (new) caused
confusion. E.g., the timings in test/ was still using the old
$RUNTEST. It made more sense to use $RUNTEST which is used
by the dejagnu feature of automake. So, I changed all
$RUNTEST or $RUNTESTS to $RUNEXEC.
config/commence.am & config/conclude.am are the two files
that got changes. Also fixed an error in test/Makefile.am.
The rest are changed by bin/reconfigure.
Tested: h5committested.
Added the xlc-* and mpcc_r-* BASENAME patterns to be recognized as IBM
compilers so that the ibm compiler options can be added properly.
This allows non-system-default compiler command names (e.g. xlc-m.n.k.l)
be recognized.
(need to figure out a better way to recognize it is an IBM compiler
and its version information.)
Tested: LLNL AIX machine.
-qmoddir=. because -qmoddir defaults to current directory already.
-static because it is not mentioned in "man xlf". Don't know it was a typo
or just an obsoleted option.
Removing both of them produced no ill or visible effects.
Tested: Up of LLNL
configure cleanup
Description
1. Removed check that disables shared libraries when pgcc is used. Apparently
they previously did not work with libtool 1.5.14 and on mir, neither of
which is in use anymore. Testing on jam and amani appear fine, so this
check is unnecessary.
2. Removed check that disables shared C++ libraries when -xarch is used to
enable 64 bit mode. We now use m64 flag to build 64 bit binaries, so
we shouldn't explicitly disable shared C++ libraries if -xarch is
discovered.
3. Removed old configure output summary section from configure.in, since
we now use the libhdf5.settings file.
4. Updated powerpc-ibm-aix5.x config file to once again put c99 flag into
H5_CFLAGS (and thus NOT get into h5cc), as decided in our dev meeting.
Tested:
h5committest
full build with pgcc on jam / amani
xlf90 and mpxlf90_r respectively. The fortran 90 compilers support
free format source files. This allows the removal of -k so that
the default compiler command is cleaner.
Tested: AIX Up at LLNL.
(Note: mpxlf90_r does not work but this could have been an undetected
old problem.)
for stream VFD. -lsocket is still used for linking, but it
is added by libtool and not by us.
Set up debug flags for Fortran.
Platforms tested: linew 32 and 64-bit.
Configure / Config File Fixes
Description:
1. Made AM_CPPFLAGS assignments in configure a bit safer. (i.e., check to
ensure include paths exist before adding them to AM_CPPFLAGS).
2. Change AIX config file to add -D_LARGE_FILES into AM_CXXFLAGS instead
of H5_CXXFLAGS (same with *_CFLAGS). This will propagate it into
h5c++, where it is needed to compile correctly.
3. Added missing line containing AM_CFLAGS to configure output summary.
Tested:
h5committest, and uP (AIX machine, via Albert).
Configure Fix
Description:
uP (AIX Machine) was failing because configure needs to see -k flag in
FCFLAGS for some checks to behave as intended. Since the config files
were switched to passing all flags into H5_FCCFLAGS, this broke this
behavior.
To allow configure to see this again, FCFLAGS and all other user flags
are saved before doing anything, and restored at the end of configure,
so if needed we can set these values in configure and/or the config/
files without (permanently) overriding or changing anything a user sets.
Also modified the powerpc-ibm-aix5.x config file to once again put -k
into FCFLAGS in addition to H5_FCFLAGS (the former of which will be
restored after configure completes the checks).
Tested:
h5committest (jam and amani finished, linew still running, though
I wanted to get this in for daily tests since it needs to be run on
uP for final verification).
CFLAGS overhaul
Description:
Modified the way configure handles CFLAGS.
(note: all changes also apply to FCFLAGS, CXXFLAGS, CPPFLAGS, and LDFLAGS).
1. The configure process will now always preserve a user's CFLAGS
environment variable setup. Any additional flags necessary for compilation
added at configure time will be passed into the Makefiles as AM_CFLAGS,
which is an automake construct to be used in addition to CFLAGS.
This will allow a user to have the final say, as CFLAGS will always appear
later in the compile line than AM_CFLAGS. Additionally, setting CFLAGS
during make will no longer completely erase all flags set by configure,
since they're maintained in AM_CFLAGS.
2. Additionally, where possible, flags previously being assigned directly
into CFLAGS (and thus propagating into h5cc) have now been redirected into
H5_CFLAGS, so they're used ONLY for compiling hdf5, and not embedded into
the h5cc wrapper script as well.
*Note that H5_CFLAGS ultimately is assigned into AM_CFLAGS for use in the
Makefiles. Complete description of changes and build process will be
included in a Configure Document that Elena and I are working on.
3. Removed unsupported config files. This includes:
config/dec-osf*
config/hpux11.00
config/irix5.x
config/powerpc-ibm-aix4.x
config/rs6000-ibm-aix5.x
config/unicos*
4. Modified configure summary to display additional values. Specifically,
appropriate AM_* variables are being shown, as well as H5_FCFLAGS and
H5_CXXFLAGS, which were for some reason not already present.
Tested:
- H5committest
- Tested on all THG / NCSA machines, using several combinations of the more
prominent configure options (c++, fortran, szip, threadsafe, parallel,
et cetera). (Thanks to Quincey for rysnc testing setup!)
- With regards to new automated testing, anything *necessary* for
compilation will be caught by the daily tests as it stands now. (i.e.,
if LDFLAGS is not properly set when szip is used, linking will fail).
Additionally, with regards to which flags get into h5cc, if any
*necessary* flags have been improperly removed, then daily tests should
fail during make installcheck. Additional machine-specific desired
behaviors and/or checks may have to be set up separately within the
daily tests, so this is something to work on.
PathScale was mistaken as the Gcc compiler such that gcc flags were added
to it but PathScale did not honor many of them. (e.g. -Wvariadic-macros,
-Wold-style-definition, -Winit-self, -Wvariadic-macros, -Wmissing-include-dirs,
-Winit-self, -Wc++-compat)
Solution:
Added a pre-condition to detect pathScale compiler before Gcc is
accepted.
Tested:
Sandia Glory where PathScale compiler is available.
gfortran is more likely installed with gcc now.
Tested:
jam and amani which covers both linux and linux64.
No h5committest because this is gnu compiler specific, usually
applicable to Linux.
For gcc v4.3 and v4.4, with production mode,
if -O3 is used, H5Tinit.c would fail to compile. Actually bad H5Tinit.c is
produced.
If -O (same as -O1) is used, H5Tinit.c compiled okay but test/dt_arith
would fail with:
Testing hard normalized long double -> signed char conversions time: comm
and terminated abnormally
When -O0 (no optimizatio) is used, H5Tinit.c compilete okay and all tests
passed.
Solution:
Changed the config/gnu-flags to use -O0 for gcc v4.3 and 4.4 for production
flags for now so that build and tests would work. Still need to investigate
if the error is due to Gcc optimization bug or our code errors.
Tested:
In liberty with gcc43 and gcc44. Also, tested in jam and smirom with
older versions of gcc.
There are no gcc 43 or 44 compilers installed in other THG machines that
I know.
Bug Fix
Description:
Fixing BZ #1381. The --includedir=DIR configure option, which is used
to specify the installation location of C header files, did not work
correctly as the path was hard-coded in config/commence.am. I'm presuming
this is because an older version of automake didn't know where to put
c header files. In any case, removing this line now defaults the includedir
to the same directory that it is currently hard-coded to, and also fixes
the configure flag to allow customization of this value.
Tested:
jam, liberty
Description:
- Remove need to set LD_LIBRARY_PATH when using shared szip library.
- Libtool 2.2.6a is now used to generate libraries.
- 'make check install' dependency bug is fixed, and should no longer
break the build.
- removed hard coding of shell in config/commence.am, as this causes
problems on Solaris with the new version of libtool.
- RELEASE.txt with appropriate changes.
Tested:
- kagiso, smirom, linew (merged from 1.8, pretty quick tests)
Add base support for extensible array "data blocks" to code, tests and
h5debug.
Tested on:
Mac OS X/32 10.5.4 (amazon) in debug mode
Mac OS X/32 10.5.4 (amazon) w/C++ & FORTRAN, w/threadsafe,
in production mode
FreeBSD/32 6.3 (duty) in debug mode
FreeBSD/64 6.3 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/Intel compilers w/default API=1.6.x,
w/C++ & FORTRAN, in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
Linux/64-amd64 2.6 (abe) w/parallel, w/FORTRAN, in production mode
rank 0 in job 77 honest1.ncsa.uiuc.edu_60945 caused collective abort of all ranks
exit status of rank 0: killed by signal 9
Up to this point, we decide to turn off the use of advanced MPI derived datatype features inside HDF5 so that complicated selection tests are ensured to be handled by HDF5. This check-in reflects this.
Update the gcc flags for version 4.3
Clean up warnings
Tested on:
FreeBSD/32 6.2 (duty) in debug mode
FreeBSD/64 6.2 (liberty) w/C++ & FORTRAN, in debug mode
Linux/32 2.6 (kagiso) w/PGI compilers, w/C++ & FORTRAN, w/threadsafe,
in debug mode
Linux/64-amd64 2.6 (smirom) w/default API=1.6.x, w/C++ & FORTRAN,
in production mode
Linux/64-ia64 2.6 (cobalt) w/Intel compilers, w/C++ & FORTRAN,
in production mode
Solaris/32 2.10 (linew) w/deprecated symbols disabled, w/C++ & FORTRAN,
w/szip filter, in production mode
Mac OS X/32 10.5.3 (amazon) in debug mode
Linux/64-ia64 2.4 (tg-login3) w/parallel, w/FORTRAN, in production mode
Description:
The fortran Makefile.am used HDF_FORTRAN to indicate it is part of the
Fortran API source so that conclude.am will give fortran api prefix in the
test output. The symbox HDF_FORTRAN is also used in configure for a different
purpose (indicated --enable-fortran). They conflicted.
Similar problem for the symbol HDF_CXX.
Solution:
Changed all the involved Makefile.am to use "FORTRAN_API" instead. It is
a more appropriate name. Same for CXX_API.
Along the way, discovered that the Makefile.am of hl/fortran/test and
hl/cxx/test did not have those symbols at all. Added them in.
Platform tested:
Kagiso only. It is a trivia change.
