Commit Graph

599 Commits

Author SHA1 Message Date
Ward Fisher
ce78cf2d73
Merge branch 'master' into MT-for-MSVC-release 2018-11-15 15:49:05 -07:00
Greg Sjaardema
d18457a2d0
RPATH handling is not needed for archive library
If not building shared libraries, then the RPATH handling should not be specified.  Although ignored on some compilers, the CRAY / Intel compiler will error out when installing the NetCDF executables if the RPATH settings are enabled.
2018-11-06 10:11:57 -07:00
Ward Fisher
6eec91bc5f Merge branch 'leak1.dmh' into v4.6.2-release-branch.wif 2018-11-01 12:09:53 -06:00
Dennis Heimbigner
245961de00 re: github issues
https://github.com/Unidata/netcdf-c/issues/1168
    https://github.com/Unidata/netcdf-c/issues/1163
    https://github.com/Unidata/netcdf-c/issues/1162

This PR partially fixes memory leaks in the netcdf-c library,
in the ncdump utility, and in some test cases.

The netcdf-c library now runs memory clean with the assumption
that the --disable-utilities option is used. The primary remaining
problem is ncgen. Once that is fixed, I believe the netcdf-c library
will run memory clean with no limitations.

Notes
-----------
1. Memory checking was performed using gcc -fsanitize=address.
   Valgrind-based testing has yet to be performed.
2. The pnetcdf, hdf4, and examples code has not been tested.

Misc. Non-leak changes
1. Make tst_diskless2 only run when netcdf4 is enabled (issue 1162)
2. Fix CmakeLists.txt to turn off logging if ENABLE_NETCDF_4 is OFF
3. Isolated all my debug scripts into a single top-level directory
   called debug
4. Fix some USE_NETCDF4 dependencies in nc_test and nc_test4 Makefile.am
2018-10-30 20:48:12 -06:00
Ward Fisher
f5482d5f37 Recreating 4.6.2 release branch. 2018-10-22 16:52:07 -06:00
Ben Boeckel
005a1bf0b4 cmake: fix detection of the mmap function 2018-10-16 09:44:20 -04:00
Ben Boeckel
918cdf5155 cmake: remove duplicate check for fsync 2018-10-16 09:44:12 -04:00
Dennis Heimbigner
553de966d6 Add some visual studio bug fixes 2018-10-11 12:09:42 -06:00
Dennis Heimbigner
4636584d5b Revert/Improve nc_create + NC_DISKLESS behavior
re: https://github.com/Unidata/netcdf-c/issues/1154

Inadvertently, the behavior of NC_DISKLESS with nc_create() was
changed in release 4.6.1. Previously, the NC_WRITE flag needed
to be explicitly used with NC_DISKLESS in order to cause the
created file to be persisted to disk.

Additional analyis indicated that the current NC_DISKLESS
implementation was seriously flawed.

This PR attempts to clean up and regularize the situation with
respect to NC_DISKLESS control. One important aspect of diskless
operation is that there are two different notions of write.

1. The file is read-write vs read-only when using the netcdf API.
2. The file is persisted or not to disk at nc_close().

Previously, these two were conflated. The rules now are
as follows.

1. NC_DISKLESS + NC_WRITE means that the file is read/write using the netcdf API
2. NC_DISKLESS + NC_PERSIST means that the file is persisted to a disk file at nc_close.
3. NC_DISKLESS + NC_PERSIST + NC_WRITE means both 1 and 2.

The NC_PERSIST flag is new and takes over the obsolete NC_MPIPOSIX flag.
NC_MPIPOSIX is still defined, but is now an alias for the NC_MPIIO flag.

It is also now the case that for netcdf-4, NC_DISKLESS is independent
of NC_INMEMORY and in fact it is an error to specify both flags
simultaneously.

Finally, the MMAP code was fixed to use NC_PERSIST as well.
Also marked MMAP as deprecated.

Also added a test case to test various combinations of NC_DISKLESS,
NC_PERSIST, and NC_WRITE.

This PR affects a number of files and especially test cases
that used NC_DISKLESS.

Misc. Unrelated fixes
1. fixed some warnings in ncdump/dumplib.c
2018-10-10 13:32:17 -06:00
Dennis Heimbigner
a5a34f6aba
Merge branch 'master' into nc_mpiio_nc_mpiposix 2018-10-06 13:33:55 -06:00
Dennis Heimbigner
8072d1f6bb Modify DAP2 and DAP4 to optionally allow Fillvalue/Variable mismatch
re: issue https://github.com/Unidata/netcdf-c/issues/1151

Modify DAP2 and DAP4 code to handle case when _FillValue type is not
same as the parent variable type.

Specifically:
1. Define a parameter [fillmismatch] to allow this mismatch;
   default is to disallow.
2. If allowed, forcibly change the type of the _FillValue to match
   the parent variable.
3. If allowed Convert the values to match new type
4. Generate a log message
5. if not allowed, then fail

Implementing this required some changes to ncdap_test/dapcvt.c
Also added test cases.

Minor Unrelated Changes:
1. There were a number of warnings about e.g.
   assigning a const char* to a char*. Fix these
2. In nccopy.1, replace .NP with .IP "n"
   (re PR https://github.com/Unidata/netcdf-c/pull/1144)
3. fix minor error in ncdump/ocprint
2018-10-01 15:51:43 -06:00
Wei-keng Liao
0ed70756cc Ignore flags NC_MPIIO and NC_MPIPOSIX. 2018-09-22 20:22:34 -05:00
Wei-keng Liao
bc63c08f41 add a check for MPI_Info_f2c 2018-09-21 17:32:36 -05:00
Wei-keng Liao
5ed6c239e1 More replacement of "parallel-netcdf" with "PnetCDF". 2018-09-20 11:45:25 -05:00
Wei-keng Liao
48da78e133 Use PnetCDF instead of parallel-netcdf to avoid confusion with
parallel netcdf4. Also, update PnetCDF web page.
2018-09-17 17:18:48 -05:00
Ward Fisher
784d777bff Merge branch 'master' into provenance.dmh 2018-09-06 15:13:09 -06:00
Ed Hartnett
efe8e3e8e6
Merge branch 'master' into ejh_loop_cleanup_2 2018-09-06 07:49:18 -06:00
Ward Fisher
746cece75d For some reason, the check for H5free() was never added to CMakeLists.txt. This becomes an issue with newer compilers, as observed in Ubuntu 18.04 and HDF5 1.10.2. The issue isn't seen in HDF 1.8.20. 2018-09-05 14:42:51 -06:00
Ward Fisher
fbe0a18b1c
Merge branch 'master' into ejh_loop_cleanup_2 2018-09-05 11:22:55 -06:00
Ward Fisher
d0bb5ddde8 Merge remote-tracking branch 'origin/inmemory10.dmh' into combined-pr.wif 2018-09-04 13:39:34 -06:00
Ward Fisher
3c3119bed2 Merge remote-tracking branch 'origin/dapcurlopt.dmh' into combined-pr.wif 2018-09-04 12:50:55 -06:00
Dennis Heimbigner
d62a9e623c Fix the NC_INMEMORY code to work in all cases with HDF5 1.10.
re: github issue https://github.com/Unidata/netcdf-c/issues/1111

One of the less common use cases for the in-memory feature is
apparently failing with HDF5-1.10.x.  The fix is complicated and
requires significant changes to libhdf5/nc4memcb.c. The current
setup is detailed in the file docs/inmeminternal.dox.

Additionally, it was discovered that the program
nc_test/tst_inmemory.c, which is invoked by
nc_test/run_inmemory.sh, actually was failing because of the
above problem. But the failure is not detected since the script
does not return non-zero value.

Other Changes:
1. Fix nc_test_tst_inmemory to return errors correctly.
2. Make ncdap_tests/findtestserver.c and dap4_tests/findtestserver4.c
   be generated from ncdap_test/findtestserver.c.in.
3. Make LOG() print output to stderr instead of stdout to
   avoid contaminating e.g. ncdump output.
4. Modify the handling of NC_INMEMORY and NC_DISKLESS flags
   to properly handle that NC_DISKLESS => NC_INMEMORY. This
   affects a number of code pieces, especially memio.c.
2018-09-04 11:27:47 -06:00
Ward Fisher
ede7c5da60
Merge branch 'master' into provenance.dmh 2018-09-04 11:22:36 -06:00
Ed Hartnett
27281dfe4b
Merge branch 'master' into ejh_loop_cleanup_2 2018-08-28 17:32:40 -06:00
Ed Hartnett
5eb0d19227 merged master 2018-08-28 10:03:49 -06:00
Wei-keng Liao
ca0d2bef72 resolve a merge conflict 2018-08-27 17:57:07 -05:00
Wei-keng Liao
f88a34600e
Merge branch 'master' into default_format 2018-08-27 17:43:12 -05:00
Dennis Heimbigner
79e38de840 Add the ability to set some additional curlopt values
Add the ability to set some additional curlopt values via .daprc (aka .dodsrc).
This effects both DAP2 and DAP4 protocols.

Related issues:
[1] re: esupport: KOZ-821332
[2] re: github issue https://github.com/Unidata/netcdf4-python/issues/836
[3] re: github issue https://github.com/Unidata/netcdf-c/issues/1074

1. CURLOPT_BUFFERSIZE: Relevant to [1]. Allow user to set the read/write
buffersizes used by curl.
This is done by adding the following to .daprc (aka .dodsrc):
	HTTP.READ.BUFFERSIZE=n
where n is the buffersize in bytes. There is a built-in (to curl)
limit of 512k for this value.

2. CURLOPT_TCP_KEEPALIVE (and CURLOPT_TCP_KEEPIDLE and CURLOPT_TCP_KEEPINTVL):
Relevant (maybe) to [2] and [3]. Allow the user to turn on KEEPALIVE
This is done by adding the following to .daprc (aka .dodsrc):
	HTTP.KEEPALIVE=on|n/m
If the value is "on", then simply enable default KEEPALIVE. If the value
is n/m, then enable KEEPALIVE and set KEEPIDLE to n and KEEPINTVL to m.
2018-08-26 17:04:46 -06:00
Dennis Heimbigner
2ea1cf5f1b There was a request to extend the provenance information
stored in the _NCProperties attribute to allow two things:
1. capture of additional library dependencies (over and above
   hdf5)
2. Recognition of non-netcdf libraries that create netcdf-4 format
   files.

To this end, the _NCProperties format has been extended to be
and arbitrary set of key=value pairs separated by commas.
This new format has version = 2, and uses commas as the pair separator.
Thus the general form is:
    _NCProperties = "version=2,key1=value,key2=value2..." ;

This new version is accompanied by a new ./configure option of the form
    --with-ncproperties="key1=value1,key2=value2..."
that specifies pairs to add to the _NCProperties attribute for all
files created with that netcdf library.

At this point, what is missing is some programmatic way to
specify either all the pairs or additional pairs
to the _NCProperties attribute. Not sure of the best way
to do this.

Builders using non-netcdf libraries can specify
whatever they want in the key value pairs (as long
as the version=2 is specified first).

By convention, the primary library is expected to be the
the first pair after the leading version=2 pair, but this
is convention only and is neither required nor enforced.

Related changes:
1. Fixed the tests that check _NCProperties to properly operate with version=2.
2. When reading a version 1 _NCProperties attribute, convert it to look
   like a version 2 attribute.
2. Added some version 2 tests to ncdump/tst_fileinfo.c and
   ncdump/tst_fileinfo.sh

Misc Changes:
1. Fix minor problem in ncdap_test/testurl.sh where a parameter to
   buildurl needed to be quoted.
2. Minor fix to ncgen to swap switches -H and -h to be consistent
   with other utilities.
3. Document the -M flag in nccopy usage() and the nccopy man page.
4. Modify a test case to use the nccopy -M flag.
2018-08-25 21:44:41 -06:00
Ed Hartnett
33a47710b8 got par tests working in cmake build 2018-08-17 08:52:34 -06:00
Ed Hartnett
59a25093d9 added setting of mpiexec command to CMake build 2018-08-16 07:33:04 -06:00
Wei-keng Liao
4341e4f6c9 spell check 2018-07-28 16:10:07 -05:00
Wei-keng Liao
9d4fa6f542
Merge branch 'master' into configure_error_out 2018-07-26 01:40:53 -05:00
Ed Hartnett
ef7b525ce4 merged master 2018-07-23 09:45:39 -06:00
Jin-Whu
5bd160086b
Merge branch 'master' into MT-for-MSVC-release 2018-07-23 10:38:11 +08:00
Ward Fisher
b570d10e01
Merge branch 'master' into enable-cdf5 2018-07-17 11:26:02 -06:00
Wei-keng Liao
3141c1b2cc error out when conflicts with PnetCDF are detected 2018-07-16 14:42:37 -05:00
Ed Hartnett
9ac15aac46 attempting to fix ENABLE_SET_LOG_LEVEL problem with cmake build 2018-07-16 11:45:26 -06:00
Ed Hartnett
966fec740d attempting to fix ENABLE_SET_LOG_LEVEL problem with cmake build 2018-07-16 08:39:23 -06:00
Ed Hartnett
cdfa6abad8 attempting to fix ENABLE_SET_LOG_LEVEL problem with cmake build 2018-07-16 08:34:00 -06:00
Ed Hartnett
82d0302fa9 attempting to fix ENABLE_SET_LOG_LEVEL problem with cmake build 2018-07-16 06:39:15 -06:00
Ed Hartnett
987ce47f5e removed duplicate handling of logging 2018-07-16 06:24:57 -06:00
Ward Fisher
8d150a56b3 Merge remote-tracking branch 'origin/interndocs.dmh' into pr-aggregation.wif 2018-07-13 11:01:43 -06:00
Greg Sjaardema
ddabcda2c2 Test should be off by default 2018-07-11 13:40:37 -04:00
Greg Sjaardema
c3f63b6ffe Enable metadata_perf test in CMake build
Currently defaults to enabled, can change top-level
CMakeLists.txt file to change default to disabled
2018-07-11 13:40:31 -04:00
Wei-keng Liao
26f1d9137c revise #1033 and emulate CMAKE OPTION to use 3rd option AUTO 2018-07-01 19:42:03 -05:00
Dennis Heimbigner
c304b2075b
Merge branch 'master' into interndocs.dmh 2018-06-30 13:56:36 -06:00
Dennis Heimbigner
40013b72f6 The pr does some cleanup on the internal documentation
in the docs directory.
1. Add a new internal document -- testserver.dox -- to describe
   how to set up and maintain the dap test server.
2. It moves the internal documentation (internal.dox, indexing.dox,
   and testserver.dox) to later in the documentation table of contents.
3. Cleanup the formatting of the internal documents.
4. Cleanup some minor doxygen issues in other files.
2018-06-29 15:06:27 -06:00
Ward Fisher
246b56dedd
Revert "fix cdf5 configure option" 2018-06-29 14:23:40 -06:00
Wei-keng Liao
02dca6a541 add cmake -L option help string for ENABLE_CDF5 2018-06-28 13:17:08 -05:00