re: Issue
The byterange handling of the following URLS fails.
### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes"
It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD).
I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time.
### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3"
When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case.
## Misc. Related Changes
1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support
2. Added the above URLS as additional test cases
## Misc. Un-Related Changes
1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD
2. Fix some minor warnings
## Open Problems
* Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
re: PR https://github.com/Unidata/netcdf-c/issues/2622
H/T Nathan Potter for finding this.
Apparently the existing library DAP code for supporting
compressed http responses was disabled.
So:
1. enable CURLOPT_ACCEPT_ENCODING by default
2. Add a new HTTP.ENCODE for .dodsrc to allow it to be disabled.
re: Issue https://github.com/Unidata/netcdf-c/issues/2573
The file type inferencer in libdispatch/dinference.c has a simple
forward inference mechanism so that the occurrence of certain mode
values in a URL fragment implies inclusion of additional mode values.
This kind of inference is notorious for leading to cycles if not
careful. Unfortunately, this occurred in the one in dinference.c.
This was fixed by providing a more complicated, but more reliable inference
mechanism.
## Misc. Other Changes
* Found and fixed a couple of memory leaks.
* There is a recent problem in building HDF4 support on github actions. Fixed by using the internal HDF4 xdr capability.
* Some filter-related code was not being properly ifdef'd with ENABLE_NCZARRA_FILTERS.
Most changes are to get plugins working.
libdispatchdreg.c went in in unidata/netcdf-c#2460,
after I'd done it here.
Summary of individual changes below.
BLD: Remove declspec(dllexport); in dreg.c.
By removing the explicit handling, the automatic handling
(equivalent to --export-all-symbols with recent GNU tools)
will be enabled again, so the generated library will have
more than one function exported.
BLD: Link plugins against libnetcdf on Cygwin.
BLD: Add AM_LDFLAGS to plugin _LDFLAGS to pass -no-undefined.
BLD: Link ncz*filters plugins against libnetcdf.
BLD: Add AM_LDFLAGS to test plugin _LDFLAGS.
Also move rpath from AM_LDFLAGS to test plugin _LDFLAGS.
TST: Don't run nczarr_test/run_specific_filters.sh on Cygwin.
It takes over half an hour to complete, where the others take a minute or less.
TST: Try to find the hanging Cygwin test.
re: Issue https://github.com/Unidata/netcdf-c/issues/2502
H/T Charlie Zender
* Fix NCZarr handling of endianness value NC_ENDIAN_NATIVE. This now matches how it is handled in libhdf5
* Fix NCZarr handling of char typed attribute with value "". This now matches how it is handled in libhdf5
* Add test for various char attribute values
* Change the mapping of NC_CHAR and NC_STRING to dtype; requires changing some test files also.
* Optimize the testing for NC_ENOTBUILT in NC_open.
* Turn off debugging left on accidentally
* Fix memory leak in tst_pnetcdf.c
* Fix blosc test
re: PR https://github.com/Unidata/netcdf-c/pull/2492
re: Issue https://github.com/Unidata/netcdf-c/issues/2494
This PR fixes some problems with the pull request https://github.com/Unidata/netcdf-c/pull/2492 in response to Issue https://github.com/Unidata/netcdf-c/issues/2494.
* Found and fixed more scalar handling problems and add a test case for scalars.
* Cleanup nczarr_test/run_string.sh test
* Document *_nczarr_default_maxstrlen* and *_nczarr_maxstrlen*.
* Support both "Nan" and *Nan* as being floating point constants
for attributes. It is unclear from the Zarr V2 spec if
unquoted *Nan* is legal or not, but support for reading.
Write the quoted versions when writing an attribute. Similar
for Infinity constants.
So NCZarr supports the following constants for use in Attributes
* *Nan*, "Nan", *-Nan*, "-Nan"
* *Nanf*, "Nanf", *-Nanf*, "-Nanf"
* *Infinity*, "Infinity", *-Infinity*, "-Infinity"
* *Infinityf*, "Infinityf", *-Infinityf*, "-Infinityf"
* re: https://github.com/Unidata/netcdf-c/pull/2278
* re: https://github.com/Unidata/netcdf-c/issues/2485
* re: https://github.com/Unidata/netcdf-c/issues/2474
This PR subsumes PR https://github.com/Unidata/netcdf-c/pull/2278.
Actually is a bit an omnibus covering several issues.
## PR https://github.com/Unidata/netcdf-c/pull/2278
Add support for the Zarr string type.
Zarr strings are restricted currently to be of fixed size.
The primary issue to be addressed is to provide a way for user to
specify the size of the fixed length strings. This is handled by providing
the following new attributes special:
1. **_nczarr_default_maxstrlen** —
This is an attribute of the root group. It specifies the default
maximum string length for string types. If not specified, then
it has the value of 64 characters.
2. **_nczarr_maxstrlen** —
This is a per-variable attribute. It specifies the maximum
string length for the string type associated with the variable.
If not specified, then it is assigned the value of
**_nczarr_default_maxstrlen**.
This PR also requires some hacking to handle the existing netcdf-c NC_CHAR
type, which does not exist in zarr. The goal was to choose numpy types for
both the netcdf-c NC_STRING type and the netcdf-c NC_CHAR type such that
if a pure zarr implementation read them, it would still work and an
NC_CHAR type would be handled by zarr as a string of length 1.
For writing variables and NCZarr attributes, the type mapping is as follows:
* "|S1" for NC_CHAR.
* ">S1" for NC_STRING && MAXSTRLEN==1
* ">Sn" for NC_STRING && MAXSTRLEN==n
Note that it is a bit of a hack to use endianness, but it should be ok since for
string/char, the endianness has no meaning.
For reading attributes with pure zarr (i.e. with no nczarr
atribute types defined), they will always be interpreted as of
type NC_CHAR.
## Issue: https://github.com/Unidata/netcdf-c/issues/2474
This PR partly fixes this issue because it provided more
comprehensive support for Zarr attributes that are JSON valued expressions.
This PR still does not address the problem in that issue where the
_ARRAY_DIMENSION attribute is incorrectly set. Than can only be
fixed by the creator of the datasets.
## Issue: https://github.com/Unidata/netcdf-c/issues/2485
This PR also fixes the scalar failure shown in this issue.
It generally cleans up scalar handling.
It also adds a note to the documentation describing that
NCZarr supports scalars while Zarr does not and also how
scalar interoperability is achieved.
## Misc. Other Changes
1. Convert the nczarr special attributes and keys to be all lower case. So "_NCZARR_ATTR" now used "_nczarr_attr. Support back compatibility for the upper case names.
2. Cleanup my too-clever-by-half handling of scalars in libnczarr.
re: Issue https://github.com/Unidata/netcdf-c/issues/2419
There are effectively two json subsystems in netcdf-c.
1. ncjson.[ch] in libnetcdf
2. netcdf_json.h for use by plugins so they can be built without need
for libnetcdf.
The netcdf_json.h file is constructed from the concatenation of
ncjson.h plus ncjson.c. It turned out that in doing this, I was
leaving some symbols externally visible so that if, for some
reason, a plugin was built and needed libnetcdf, then symbol
conflicts arose.
The solution is to prefix the declarations in ncjson.[ch] with a
macro (OPTSTATIC) that can be resolved to either nothing or to
"static". Then in netcdf_json.h, it resolves to "static" and
prevents the symbol conflicts.
Note that netcdf_json.h is constructed once in
netcdf-c/include/Makefile.am with the rule named
"makepluginjson". This means that it is included in the
distribution. However, this also means that if ncjson.[ch] is
changed, then it is necessary to invoke makepluginjson
explicitly to rebuild netcdf_json.h
re: https://github.com/Unidata/netcdf-c/issues/2337
re: https://github.com/Unidata/netcdf-c/issues/2407
Add two functions to netcdf.h to allow programs to get/set
selected entries into the internal .rc tables. This should fix
the above issues by allowing HTTP.CAINFO to be set to the
certificates directory. Note that the changes should be
performed as early as possible in the program because some of
the .rc table entries may get cached internally and changing the
entry after that caching occurs may have no effect.
The new signatures are as follows:
1. Get the value of a simple .rc entry of the form "key=value".
Note that caller must free the returned value, which might be NULL.
````
char* nc_rc_get(char* const * key);
@param key table entry key
@return value if .rc table has entry of the form key=value
@return NULL if no such entry is found.
````
2. Insert/Overwrite the specified key=value pair in the .rc table.
````
int nc_rc_set(const char* key, const char* value);
@param key table entry key -- may not be NULL
@param value table entry value -- may not be NULL
@return NC_NOERR if no error
@return NC_EINVAL if error
````
Addendum:
re: https://github.com/Unidata/netcdf-c/issues/2407
Modify dhttp.c to use the .rc entry HTTP.CAINFO if defined.
re: https://github.com/Unidata/netcdf-c/issues/2294
Ed Hartnett suggested that the netcdf library installation process
be extended to install the standard filters into a user specified
location. The user can then set HDF5_PLUGIN_PATH to that location.
This PR provides that capability using:
````
configure option: --with-plugin-dir=<absolute directory path>
cmake option: -DPLUGIN_INSTALL_DIR=<absolute directory path>
````
Currently, the following plugins are always installed, if
available: bzip2, zstd, blosc.
If NCZarr is enabled, then additional plugins are installed:
fletcher32, shuffle, deflate, szip.
Additionally, the necessary codec support is installed
for each of the above filters that is installed.
## Changes:
1. Cleanup handling of built-in bzip2.
2. Add documentation to docs/filters.md
3. Re-factor the NCZarr codec libraries
4. Add a test, although it can only be exercised after
the library is installed, so it cannot be used during
normal testing.
5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
