1. Issue https://github.com/Unidata/netcdf-c/issues/2043
* FreeBSD build fails because of conflicts in defining the fileno() function. So removed all extern declarations of fileno.
2. Issue https://github.com/Unidata/netcdf-c/issues/2124
* There were a couple of problems here.
* I was conflating msys with mingw and they need separate handling of paths. So treat mingw like windows.
* memio.c was not always writing the full content of the memory to file. Untested fix by properly accounting for zero size writes.
* Fix bug when skipping white space in tst_xcache.c
3. Issue https://github.com/Unidata/netcdf-c/pull/2105
* On MINGW, bash and other POSIX utilities use a mounted root directory,
but executables compiled for Windows do not recognise the mount point.
Ensure that Windows paths are used in tests of Windows executables.
4. Issue https://github.com/Unidata/netcdf-c/issues/2132
* Apparently the Intel C compiler on OSX defines isnan etc.
So disable declaration in dutil.c under that condition.
5. Fix and re-enable test_rcmerge.sh by allowing override of where to
look for .rc files
6. CMakeLists.txt suppresses certain ncdump directory tests because of differences in printing floats/doubles.
* Extend the list to include those that also fail under mingw.
* Suppress the mingw tests in ncdump/Makefile.am
re: https://github.com/Unidata/netcdf-c/issues/2119
H/T to [Egbert Eich](https://github.com/e4t) and [Bas Couwenberg](https://github.com/sebastic) for this PR.
It is undesirable to make netcdf be dependent on the availability
of libxml2, but it is desirable to allow its use if available.
In order to do this, a wrapper API (include/ncxml.h) was constructed
that supports either ezxml or libxml2 as the implementation.
Additionally, the xml support code was moved to a new directory
netcdf-c/libncxml.
Primary changes:
* Create a new sub-directory named netcdf-c/libncxml to hold all the xml implementation code.
* Move ezxml.c and ezxml.h to libncxml
* Create a wrapper API -- include/ncxml.h
* Create an implementation, ncxml_ezxml.c to support use of ezxml.
* Create an implementation, ncxml_xml2.c to support use of libxml2.
* Add a check for libxml2 in configure.ac and CMakeLists.txt
* Modify libdap to use the wrapper API instead of ezxml directly.
Misc. Other Changes:
* Change include/netcdf_json.h from built source to be part of the distribution.
re: https://github.com/Unidata/netcdf-c/issues/2117
re: https://github.com/Unidata/netcdf-c/issues/2119
* Modify libsrc to allow byte-range reading of netcdf-3 files in private S3 buckets; this required using the aws sdk. Also add a test case.
* The aws sdk can sometimes cause problems if the Awd::ShutdownAPI function is not called. So at optional atexit() support to ensure it is called. This is disabled for Windows.
* Add documentation to nczarr.md on how to build and use the aws sdk under windows. Currently it builds, but testing fails.
* Switch testing from stratus to the Unidata bucket on S3.
* Improve support for the s3: url protocol.
* Add a s3 specific utility code file: ds3util.c
* Modify NC_infermodel to attempt to read the magic number of byte-ranged files in S3.
## Misc.
* Move and rename the core S3 SDK wrapper code (libnczarr/zs3sdk.cpp) to libdispatch since it now used in libsrc as well as libnczarr.
* Add calls to nc_finalize in the utilities in case atexit is disabled.
* Add header only json parser to the distribution rather than as a built source.
If the `val` passed to `findPrimeGreaterThan` is greater than the largest value (not the sentinel) in the `NC_primes`, then the routine will fall into an infinite loop. Modified to call an external routine that brute forces the finding of a prime larger than the value in this case.
The brute force routine uses the primes in `NC_primes` table in the prime test, so this will fail if given a `value > 180503 * 180503`. The `isPrime` function could be rewritten to avoid this, but assuming this won't happen for the forseeable future. If it does happen, `isPrime` will return that any value larger than this is prime...
## Examine and fix ezxml errors
re: Issue https://github.com/Unidata/netcdf-c/issues/2119
Multiple security issues were found in ezxml (see above Issue).
* CVE-2021-31598
* CVE-2021-31348 / CVE-2021-31347
* CVE-2021-31229
* CVE-2021-30485
* CVE-2021-26222
* CVE-2021-26221
* CVE-2021-26220
* CVE-2019-20202
* CVE-2019-20201
* CVE-2019-20200
* CVE-2019-20199
* CVE-2019-20198
* CVE-2019-20007
* CVE-2019-20006
* CVE-2019-20005
In addition, moved ezxml to libdispatch.
## Examine and fix selected oss-fuzz detected errors
Note that most of these errors are in the libsrc .m4 generated
code so fixing them is difficult. It would nice if we could tell
oss-fuzz to skip those files. They are old and crufty and
probably need a complete refactor.
Issue|Status
-----|------
35382|Fixed; old bug
35398|Closed by OSS-Fuzz
35442|Guarantee alloc > 0 or error; Old bug
35721|Assert failure; ok
35992|Fixed; old bug
36038|Fixed; old bug
36129|Unfixed; old bug
36229|Fixed by adding assert; old bug
37476|Unfixed; old bug
37824|Assert Failure; ok
38300|Closed by OSS-Fuzz
38537|Unfixed; old bug
38658|Unfixed; old bug
38699|Fixed maybe; old bug
38772|Nature of error is unclear, suspect that it results from using too large a type.
39248|Need more information
39394|Unfixed; old bug
HDF5 can depend on the Z library (in fact required for netCDF). Moved the detection of whether hdf5 was built with zlib up before any other tests that may require linking of the hdf5 library to determine presence/absence of symbols. These tests require that the link line include "-lz" if the hdf5 library was built with libz support.
This is typically handled somewhat automatically if shared libraries are being used, but in the static library case, the explicit dependency needs to be specified. For internal CMake checks, it uses the `CMAKE_REQUIRED_LIBRARIES` list to specify the libraries that should be used in a `CHECK_C_SOURCE_COMPILE` or a `CHECK_LIBRARY_EXISTS` call. In the current CMakeLists.txt ordering, the zlib detection is done _after_ the `CHECK_LIBRARY_EXISTS` calls which can cause them to fail and give an incorrect result about whether the function being tested for exists. With the reordering in this PR, I am able to correctly configure netCDF on a CRAY HPC system that uses static libraries by default.
