Commit Graph

694 Commits

Author SHA1 Message Date
Dennis Heimbigner
65414eeaa4 Fix some protocol differences between netcdf-c and the Hyrax server.
re: Partly addresses issue https://github.com/Unidata/netcdf-c/issues/1712.

1. Turn on Hyrax Hack to accept Hyrax style attribute containers.
2. Support Url type as alias for String.
3. Accept the special attribute, "__DAP4_Checksum_CRC32",
   to control per-variable checksums.
4. Make _DAP4_xxx attributes be reserved and only accessible
   by name (ala _SuperBlock attribute).
5. Fix handling of checksums. There is a hack in the code
   that uses an extra flag in the chunk header to indicate
   that all variables have checksums. This violates the spec
   and will be removed once it is possible to regenerate the
   test cases.

Note that checksumming with the Hyrax test server has not
been tested. This, along with some other probable inconsistencies,
needs fixing when OPeNDAP and Unidata can agree on the proper
specification. Testing will be included.
2020-05-30 17:36:25 -06:00
Ward Fisher
c2ea69df23
Merge pull request #1732 from DennisHeimbigner/vsfix.dmh
Fix undefined references when using Visual Studio
2020-05-19 16:21:04 -06:00
Dennis Heimbigner
c68c4c804d Fix undefined references when using Visual Studio
Fix Issue https://github.com/Unidata/netcdf-c/issues/1725.
Replace PR https://github.com/Unidata/netcdf-c/pull/1726
Also replace PR https://github.com/Unidata/netcdf-c/pull/1694

The general problem is that under Visual Studio, we are seeing
a number of undefined reference and other scoping errors.
The reason is that the code is not properly using Visual Studio
_declspec() declarations.

The basic solution is to ensure that when compiling the code itself
one needs to ensure that _declspec(dllexport) is used. There
are several sets of macros to handle this, but they all rely
on the flag DLL_EXPORT being define when the code is compiled,
but not being defined when the code is used via a .h file.

As a test, I modified XGetOpt.c to build properly. I also
fixed the oc2 library to properly _declspec things like ocdebug.

I also made some misc. changes to get all the tests to run
if cygwin is installed (to get bash, sed, etc).

Misc. Changes:
* Put XGetOpt.c into libsrc and copy at build time
  to the other directories where it is needed.
2020-05-18 19:36:28 -06:00
Peter Hill
e1ef7a958c Fix wrong header include in testing HDF5 for zlib 2020-05-16 12:45:13 +01:00
Ward Fisher
fc1d75f842
Merge pull request #1721 from gsjaardema/patch-39
Different method of setting Parallel Filters variables
2020-05-13 15:05:34 -06:00
Ward Fisher
9f6981bc6b
Merge pull request #1719 from DennisHeimbigner/filterrepeat.dmh
Allow redefinition of variable filters
2020-05-12 16:03:51 -06:00
Greg Sjaardema
485810262d
Fix CMake generation of netcdf_meta.h
These changes seem to be correct for the generation of `netcdf_meta.h` in a CMake build.  I think this addresses #1723 

There is still an issue with the libnetcdf.settings
2020-05-11 16:53:30 -06:00
Greg Sjaardema
8d7d5ac9e5
Different method of setting Parallel Filters variables
The current method of setting the `HDF5_HAS_PAR_FILTERS` and `HAS_PAR_FILTERS` variables is done purely based on the `HDF5_VERSION` and that variable is only set inside the if block which finds the HDF5 library based on CMake package files.  If the user specifies the explicit location of the HDF5 library and include files (for example, via:
```
         -DHDF5_C_LIBRARY:PATH=${INSTALL_PATH}/lib/libhdf5.${LD_EXT} \
         -DHDF5_HL_LIBRARY:PATH=${INSTALL_PATH}/lib/libhdf5_hl.${LD_EXT} \
         -DHDF5_INCLUDE_DIR:PATH=${INSTALL_PATH}/include
```
Then, the code path which determines whether the par filters variables is set is not run.  However, later on in the file, there is another check for parallel filter support (near line 759):
```
  # Check to see if this is hdf5-1.10.3 or later.
  CHECK_LIBRARY_EXISTS(${HDF5_C_LIBRARY_hdf5} H5Dread_chunk "" HDF5_SUPPORTS_PAR_FILTERS)
```

This PR moves the code that sets the two other par filters variables down after this check and instead of setting their values based on the version, it bases it on the results of this test.

I'm not totally sure why there are three variables; it looks like the `HDF5_SUPPORTS_PAR_FILTERS` and `HDF5_HAS_PAR_FILTERS` could be combined.  I think the `HAS_PAR_FILTERS` is a string which is used to show the results of the configuration and the other two are booleans.

The new check should work for both types of HDF5 installs (cmake-based and configure-based)
2020-05-11 11:11:24 -06:00
Dennis Heimbigner
84c69afca7 Allow redefinition of variable filters
re: Github issue https://github.com/Unidata/netcdf-c/issues/1713

If nc_def_var_filter or nc_def_var_deflate or nc_def_var_szip is
called multiple times with the same filter id, but possibly with
different sets of parameters, then the first invocation is
sticky and later invocations are ignored. The desired behavior
is to have the last invocation be used.

This PR implements that desired behavior, with some special
cases.  If you call nc_def_var_deflate multiple times, then the
last invocation rule applies with respect to deflate. However,
the shuffle filter, if enabled, is always applied just before
applying deflate.

Misc unrelated changes:
1. Make client-side filters be disabled by default
2. Fix the definition of uintptr_t and use in oc2 and libdap4
3. Add some test cases
4. modify filter order tests to use plugin filters rather
   than client-side filters
2020-05-11 09:42:31 -06:00
Dennis Heimbigner
313121a229 Use proper CURLOPT values for VERIFYHOST and VERIFYPEER
re: https://github.com/Unidata/netcdf-c/issues/1684
re: e-support VZL-904142

Two issues:
1. As of libcurl 7.66, the semantics of CURLOPT_SSL_VERIFYHOST
   changed so that the non-zero values affects certificate processing.
2. The current library was forcing the values of VERIFYPEER
   and VERIFYHOST to zero instead of leaving them to the default values.

Solution was first to leave the defaults in place for VERIFYPEER and VERIFYHOST
as long as they are not set in .ocrc/.dodsrc file.
Second, the value of HTTP.SSL.VERIFYPEER or HTTP.SSL.VERIFYHOST
as set in .ocrc/.dodrc is used to set the corresponding CURLOPT flags.
So for example, adding
> HTTP.SSL.VERIFYHOST=2
will set the value of CURLOPT_SSL_VERIFYHOST to 2, the default.
Using
> HTTP.SSL.VERIFYHOST=0
will set the value of CURLOPT_SSL_VERIFYHOST to 0, which disables it.
Similarly for VERIFYPEER.

Finally the semantics of HTTP.SSL.VALIDATE is now equivalent to
> HTTP.SSL.VERIFYPEER=1
> HTTP.SSL.VERIFYHOST=2
2020-04-10 13:42:27 -06:00
Ward Fisher
65a17399b9 Corrected parallel (mpi) testing on cmake builds. 2020-04-02 10:09:57 -06:00
Ward Fisher
b0e2d78ffa Corrected an issue with parallel filter test logic in cmake-based builds. 2020-04-01 17:08:24 -06:00
Ward Fisher
9d482c3f20 Added hard failure to script, modified cmake default. 2020-04-01 16:20:33 -06:00
Ward Fisher
d9e69bbb6b Updated soversion, bump to next development version in preparation of upstream merge back into master from 4.7.4 wellspring. 2020-03-26 11:31:43 -06:00
Greg Sjaardema
aca7b5cc2e
Better finding of the hdf5.h include path
The `FIND_PATH` function has a non-intuitive order in which is searches for include paths and the original form could find a system-installed version of HDF5 even though the developer specified an explicit path via `HDF5_INCLUDE_DIR`.  See https://cmake.org/cmake/help/latest/command/find_path.html for more information.

The form in this patch is the recommended method from that page for changing the default search order to find the explicitly specified path *first*


> The default search order is designed to be most-specific to least-specific for common use cases. Projects may override the order by simply calling the command multiple times and using the NO_* options:
```
find_path (<VAR> NAMES name PATHS paths... NO_DEFAULT_PATH)
find_path (<VAR> NAMES name)
```
> Once one of the calls succeeds the result variable will be set and stored in the cache so that no call will search again.
2020-03-05 14:44:38 -07:00
Dennis Heimbigner
b488c272d5 Fix conflicts with master 2020-02-27 14:06:45 -07:00
Dennis Heimbigner
44d0dcaad2 Add support for multiple filters per variable.
re: https://github.com/Unidata/netcdf-c/issues/1584

Support has been added for multiple filters per variable.  This
affects a number of components in netcdf. The new APIs are
documented in NUG/filters.md.

The primary changes are:
* A set of new functions are provided (see __include/netcdf_filter.h__).
    - Obtain a list of the filters associated with a variable
    - Obtain the parameters for a specific filter.
* The existing __nc_inq_var_filter__ function now returns info
  about the first defined filter.
* The utilities (ncgen, ncdump, and nccopy) now support
  an extended format for specifying a sequence of filters.
  The general form is __<filter>|<filter>..._.
* The ncdump **_Filter** attribute now dumps a list of all the
  filters associated with a variable using the above new format.
* Filter specifications can now use a filter name instead of number
  for filters known to the netcdf library, which in turn is taken
  from the HDF5 filter registration page.
* New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter
  is returned if an attempt is made to access an unknown filter.
* Internally, the dispatch table has been extended to add a function
  to handle all of the filter functions.
* New, filter-related, tests were added to nc_test4.
* A new plugin was added to the plugins directory to help with testing.

Notes:
1. The shuffle and fletcher32 filters are not part of the multifilter system.

Misc. changes:
1. A debug module was added to libhdf5 to help catch error locations.
2020-02-16 12:59:33 -07:00
Ward Fisher
b01fbb058d
Merge pull request #1619 from NOAA-GSD/ejh_more_szip
fixed libnetcdf.settings output in cmake build wrt szip
2020-02-06 12:42:27 -07:00
Ward Fisher
1573fead56
Merge pull request #1606 from NOAA-GSD/ejh_no_zlib
detect if HDF5 was built without zlib, and error out of configure/cmake
2020-02-06 12:42:04 -07:00
Edward Hartnett
dfd496d3a5 fixed libnetcdf.settings output in cmake build wrt szip 2020-02-02 13:55:58 -07:00
Ward Fisher
e8c2d22ddd Cleaned up szlib status message in libnetcdf.settings. 2020-01-24 16:36:29 -07:00
Edward Hartnett
0672310411 fixed cmake check for zlib in HDF5 library 2020-01-23 06:51:44 -07:00
Edward Hartnett
adf2aa31e4 moved cmake detection of lack of zlib to be with other HDF5 stuff 2020-01-23 05:43:48 -07:00
Edward Hartnett
60cfe7782d detecting lack of zlib in cmake build 2020-01-23 05:40:44 -07:00
Edward Hartnett
0735a45178 switch cmake build to check for func H5Dread_chunk to detect 1.10.3 or later 2020-01-21 07:42:14 -07:00
Edward Hartnett
21a204c4e8 start using par filters in HDF5-1.10.3 2020-01-17 14:01:07 -07:00
Edward Hartnett
fb51e4475b adding NC_HAS_SZIP_WRITE and NC_HAS_PAR_FILTERS to cmake build 2020-01-17 10:30:14 -07:00
Ward Fisher
488b7e43d1
Merge pull request #1588 from oxelson/jsremotetest
changed/removed references to jetstream remotetest instance.
2020-01-13 16:07:17 -07:00
Ward Fisher
8771d0bdf4
Merge pull request #1582 from NOAA-GSD/ejh_parallel_zlib
Allow user to turn on zlib, shuffle, and/or fletcher32 filters with parallel I/O for HDF5-1.10.2+
2020-01-13 16:06:51 -07:00
Jennifer Oxelson
1b343324ea
Merge branch 'master' into jsremotetest 2020-01-10 17:35:34 -07:00
Ward Fisher
438119dd69
Merge pull request #1560 from NOAA-GSD/ejh_cache_docs
increase default cache size for netCDF-4/HDF5 files, also improve cache docs and add benchmarking program
2020-01-07 11:46:30 -07:00
Ward Fisher
20a8db5a7b
Merge pull request #1532 from NetCDF-World-Domination-Council/ejh_cmake_unit_tests
Adding rest of unit_tests to CMake build, but not running them on Windows
2020-01-07 11:44:56 -07:00
Jennifer Oxelson
feb7da82ec changed/removed references to jetstream remotetest instance. 2020-01-03 13:12:58 -07:00
Dennis Heimbigner
30b3401358 Forgot to create pingurl4.c for Cmake 2020-01-02 11:47:25 -07:00
Edward Hartnett
680e44f628 changed name of macro 2019-12-20 13:58:01 -07:00
Edward Hartnett
32a776badf getting parallel I/O with zlib working in cmake builds 2019-12-20 09:43:56 -07:00
James Sharpe
c5d1e4bdec Call find_package(MPI) to locate MPI paths and link to libdispatch if required 2019-12-18 16:48:40 +00:00
Edward Hartnett
4a5ae05dbf cleanup of RELAX_COORD_BOUND in cmake build 2019-11-26 06:59:03 -07:00
Edward Hartnett
8c648b5a2d changed cmake build to only allow relaxed coord builds 2019-11-26 05:40:03 -07:00
Edward Hartnett
a4465da974 new cache default in cmake build 2019-11-25 17:23:07 -07:00
Ward Fisher
dac41651ee Merge branch 'feature/cmake-targets' of https://github.com/kprussing/netcdf-c into v4.7.3-wellspring.wif 2019-11-19 14:12:41 -07:00
Keith F. Prussing
808803ceeb Add namespace for CMake targets
The usual convention with modern CMake is to namespace the targets.
This gives the netCDF namespace to the package.
2019-11-19 14:45:21 -05:00
Keith F. Prussing
adf2f4d098 Add include directories as netcdf property
Modern CMake tracks the properties of targets so that down stream
libraries do not have to worry about include directories.  However, the
include directories must be added to the target and not just at the
directory level.
2019-11-19 14:39:46 -05:00
Ward Fisher
e4efdd4cfb Bumped version to next development version. 2019-11-18 14:35:01 -07:00
Ward Fisher
a268e6d69f Ctest security stance. 2019-11-18 12:07:01 -07:00
Ward Fisher
7cbbc32dad Adding new security stance to cdash instance. 2019-11-18 10:58:44 -07:00
edwardhartnett
14f2b50f2f turned off unit tests for windows 2019-11-16 07:20:45 -07:00
edwardhartnett
f6ace74afb merged master 2019-11-16 07:19:40 -07:00
edwardhartnett
c1989b738e fixed setting of NC_DISPATCH_VERSION in cmake build 2019-11-15 14:57:07 -07:00
edwardhartnett
7b2ddf2537 trying unit tests on windows 2019-11-15 14:22:34 -07:00