1. Issue https://github.com/Unidata/netcdf-c/issues/2043
* FreeBSD build fails because of conflicts in defining the fileno() function. So removed all extern declarations of fileno.
2. Issue https://github.com/Unidata/netcdf-c/issues/2124
* There were a couple of problems here.
* I was conflating msys with mingw and they need separate handling of paths. So treat mingw like windows.
* memio.c was not always writing the full content of the memory to file. Untested fix by properly accounting for zero size writes.
* Fix bug when skipping white space in tst_xcache.c
3. Issue https://github.com/Unidata/netcdf-c/pull/2105
* On MINGW, bash and other POSIX utilities use a mounted root directory,
but executables compiled for Windows do not recognise the mount point.
Ensure that Windows paths are used in tests of Windows executables.
4. Issue https://github.com/Unidata/netcdf-c/issues/2132
* Apparently the Intel C compiler on OSX defines isnan etc.
So disable declaration in dutil.c under that condition.
5. Fix and re-enable test_rcmerge.sh by allowing override of where to
look for .rc files
6. CMakeLists.txt suppresses certain ncdump directory tests because of differences in printing floats/doubles.
* Extend the list to include those that also fail under mingw.
* Suppress the mingw tests in ncdump/Makefile.am
re: https://github.com/Unidata/netcdf-c/issues/2119
H/T to [Egbert Eich](https://github.com/e4t) and [Bas Couwenberg](https://github.com/sebastic) for this PR.
It is undesirable to make netcdf be dependent on the availability
of libxml2, but it is desirable to allow its use if available.
In order to do this, a wrapper API (include/ncxml.h) was constructed
that supports either ezxml or libxml2 as the implementation.
Additionally, the xml support code was moved to a new directory
netcdf-c/libncxml.
Primary changes:
* Create a new sub-directory named netcdf-c/libncxml to hold all the xml implementation code.
* Move ezxml.c and ezxml.h to libncxml
* Create a wrapper API -- include/ncxml.h
* Create an implementation, ncxml_ezxml.c to support use of ezxml.
* Create an implementation, ncxml_xml2.c to support use of libxml2.
* Add a check for libxml2 in configure.ac and CMakeLists.txt
* Modify libdap to use the wrapper API instead of ezxml directly.
Misc. Other Changes:
* Change include/netcdf_json.h from built source to be part of the distribution.
re: https://github.com/Unidata/netcdf-c/issues/2117
re: https://github.com/Unidata/netcdf-c/issues/2119
* Modify libsrc to allow byte-range reading of netcdf-3 files in private S3 buckets; this required using the aws sdk. Also add a test case.
* The aws sdk can sometimes cause problems if the Awd::ShutdownAPI function is not called. So at optional atexit() support to ensure it is called. This is disabled for Windows.
* Add documentation to nczarr.md on how to build and use the aws sdk under windows. Currently it builds, but testing fails.
* Switch testing from stratus to the Unidata bucket on S3.
* Improve support for the s3: url protocol.
* Add a s3 specific utility code file: ds3util.c
* Modify NC_infermodel to attempt to read the magic number of byte-ranged files in S3.
## Misc.
* Move and rename the core S3 SDK wrapper code (libnczarr/zs3sdk.cpp) to libdispatch since it now used in libsrc as well as libnczarr.
* Add calls to nc_finalize in the utilities in case atexit is disabled.
* Add header only json parser to the distribution rather than as a built source.
If the `val` passed to `findPrimeGreaterThan` is greater than the largest value (not the sentinel) in the `NC_primes`, then the routine will fall into an infinite loop. Modified to call an external routine that brute forces the finding of a prime larger than the value in this case.
The brute force routine uses the primes in `NC_primes` table in the prime test, so this will fail if given a `value > 180503 * 180503`. The `isPrime` function could be rewritten to avoid this, but assuming this won't happen for the forseeable future. If it does happen, `isPrime` will return that any value larger than this is prime...
## Examine and fix ezxml errors
re: Issue https://github.com/Unidata/netcdf-c/issues/2119
Multiple security issues were found in ezxml (see above Issue).
* CVE-2021-31598
* CVE-2021-31348 / CVE-2021-31347
* CVE-2021-31229
* CVE-2021-30485
* CVE-2021-26222
* CVE-2021-26221
* CVE-2021-26220
* CVE-2019-20202
* CVE-2019-20201
* CVE-2019-20200
* CVE-2019-20199
* CVE-2019-20198
* CVE-2019-20007
* CVE-2019-20006
* CVE-2019-20005
In addition, moved ezxml to libdispatch.
## Examine and fix selected oss-fuzz detected errors
Note that most of these errors are in the libsrc .m4 generated
code so fixing them is difficult. It would nice if we could tell
oss-fuzz to skip those files. They are old and crufty and
probably need a complete refactor.
Issue|Status
-----|------
35382|Fixed; old bug
35398|Closed by OSS-Fuzz
35442|Guarantee alloc > 0 or error; Old bug
35721|Assert failure; ok
35992|Fixed; old bug
36038|Fixed; old bug
36129|Unfixed; old bug
36229|Fixed by adding assert; old bug
37476|Unfixed; old bug
37824|Assert Failure; ok
38300|Closed by OSS-Fuzz
38537|Unfixed; old bug
38658|Unfixed; old bug
38699|Fixed maybe; old bug
38772|Nature of error is unclear, suspect that it results from using too large a type.
39248|Need more information
39394|Unfixed; old bug
On some versions of the HDF5 find_package call, it sets `HDF5_C_LIBRARIES` and `HDF5_HL_LIBRARIES`, but does not set the `HDF5_C_LIBRARY` or `HDF5_HL_LIBRARY` to anything. Control then falls out of the if block with these unset and it falls into the default setting at line 792. This does not include the path, so then when the later `CHECK_LIBRARY_EXISTS` calls are run, they do not have the full path to the library and will not link correctly. Since the link fails, the code defaults to thinking that none of the symbols are defined.
I don't think this change will have any affect since it only sets the symbols if they are unset.
## S3 Related Fixes
* Add comprehensive support for specifying AWS profiles to provide access credentials.
* Parse the files "~/.aws/config" and "~/.aws/credentials to provide credentials for the HDF5 ROS3 driver and to locate default region.
* Add a function to obtain the currently active S3 credentials. The search rules are defined in docs/nczarr.md.
* Provide documentation for the new features.
* Modify the struct NCauth (in include/ncauth.h) to replace specific S3 credentials with a profile name.
* Add a unit test to test the operation of profile and credentials management.
* Add support for URLS of the form "s3://<bucket>/<key>"; this requires obtaining a default region.
* Allows the specification of profile and/or region in a URL of the form "#mode=nczarr,...&aws.region=...&aws.profile=..."
## Misc. Fixes
* Move the ezxml code to libdispatch so that it can be used both by DAP4 and nczarr.
* Modify nclist to provide a deep clone operation.
* Modify ncuri to provide a deep clone operation.
* Modify the .rc file format to allow the specification of a path to be tested when looking for an entry in the .rc file.
* Ensure that the NC_rcload function is called.
* Modify nchttp to support setting request headers.
Fixes an issue with strlen() reading outside the stack allocated buffer
by NC4_HDF5_inq_att, when reading a name whose length is NC_MAX_NAME.
Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=39189 found
on GDAL
==1895951== Conditional jump or move depends on uninitialised value(s)
==1895951== at 0x483EF58: strlen (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1895951== by 0x48EF73E: ncindexlookup (ncindex.c:60)
==1895951== by 0x48E81DF: nc4_find_grp_att (nc4internal.c:587)
==1895951== by 0x48E5B39: nc4_get_att_ptrs (nc4attr.c:72)
==1895951== by 0x48F98A0: NC4_HDF5_inq_att (hdf5attr.c:818)
==1895951== by 0x48847F7: nc_inq_att (dattinq.c:91)
==1895951== by 0x10D693: pr_att (ncdump.c:767)
==1895951== by 0x110ADB: do_ncdump_rec (ncdump.c:1887)
==1895951== by 0x1112F1: do_ncdump (ncdump.c:2038)
==1895951== by 0x11248B: main (ncdump.c:2478)
==1895951==
==1895951== Use of uninitialised value of size 8
==1895951== at 0x48A24E4: crc64_little (dcrc64.c:173)
==1895951== by 0x48A27F4: NC_crc64 (dcrc64.c:229)
==1895951== by 0x4892D49: NC_hashmapkey (nchashmap.c:159)
==1895951== by 0x489314B: NC_hashmapget (nchashmap.c:263)
==1895951== by 0x48EF75F: ncindexlookup (ncindex.c:60)
==1895951== by 0x48E81DF: nc4_find_grp_att (nc4internal.c:587)
==1895951== by 0x48E5B39: nc4_get_att_ptrs (nc4attr.c:72)
==1895951== by 0x48F98A0: NC4_HDF5_inq_att (hdf5attr.c:818)
==1895951== by 0x48847F7: nc_inq_att (dattinq.c:91)
==1895951== by 0x10D693: pr_att (ncdump.c:767)
==1895951== by 0x110ADB: do_ncdump_rec (ncdump.c:1887)
==1895951== by 0x1112F1: do_ncdump (ncdump.c:2038)
==1895951==
