re: Discussion https://github.com/Unidata/netcdf-c/discussions/3085
This discussion raised the issue of the best way to distinguish
a netcdfd-c created file and an HDF5 created file. The
recommended way is to use the _NCProperties attribute. In order
for users to process this attribute, I have added a parser for
the attribute to the netcdf_aux.h file.
re: PR https://github.com/Unidata/netcdf-c/pull/3068
1. Update the ncjson and ncproplist code to lastest.
2. Fix some references to older API functions.
3. Update the netcdf_json and netcdf_proplist builds.
The new plugin path API uses char** to represent
a variable lenght vector of variable length strings.
FORTRAN is not capable of accessing such structures.
So, this PR extends the API to provide a counted string-based
API to the plugin path functionality.
The new functions are inserted in the netcdf_aux.h/daux.c files.
The new functions are just wrappers around other plugin path API
function; they are just (I hope) more convenient for FORTRAN users.
The new functions are as follows:
### *ncaux_plugin_path_stringlen(void)*
* Return the length (as in strlen) of the current plugin path directories encoded as a string. Return -1 if the request fails.
### *int ncaux_plugin_path_stringget(int pathlen, char* path)*
* Get the current sequence of directories in the internal global plugin path list encoded as a string path using ';' as a path separator. As an example, it might return "/a/b/c;/home/user/me;/tmp". The arguments are as follows:
* *pathlen* -- the length of the path argument.
* *path* -- a string into which the current plugin path as a string is stored.
* Return NC_NOERR | NC_EINVAL
### *int ncaux_plugin_path_stringset(int pathlen, const char* path)*
* Set the current sequence of directories in the internal global plugin path list. As an example, it might take "/a/b/c;/home/user/me;/tmp". The arguments are as follows:
* *pathlen* -- the length of the path argument.
* *path* -- a string that is parsed to obtain the sequence of directories for the current plugin path.
* Return NC_NOERR | NC_EINVAL
Replaces PR https://github.com/Unidata/netcdf-c/pull/3024
and PR https://github.com/Unidata/netcdf-c/pull/3033
re: https://github.com/Unidata/netcdf-c/issues/2753
As suggested by Ed Hartnett, This PR extends the netcdf.h API to support programmatic control over the search path used to locate plugins.
I created several different APIs, but finally settled on the following API as being the simplest possible. It does have the disadvantage that it requires use of a global lock (not implemented) if used in a threaded environment.
Specifically, note that modifying the plugin path must be done "atomically". That is, in a multi-threaded environment, it is important that the sequence of actions involved in setting up the plugin path must be done by a single processor or in some other way as to guarantee that two or more processors are not simultaneously accessing the plugin path get/set operations.
As an example, assume there exists a mutex lock called PLUGINLOCK. Then any processor accessing the plugin paths should operate as follows:
````
lock(PLUGINLOCK);
nc_plugin_path_get(...);
<rebuild plugin path>
nc_plugin_path_set(...);
unlock(PLUGINLOCK);
````
## Internal Architecture
It is assumed here that there only needs to be a single set of plugin path directories that is shared by all filter code and is independent of any file descriptor; it is global in other words. This means, for example, that the path list for NCZarr and for HDF5 will always be the same.
However internally, processing the set of plugin paths depends on the particular NC_FORMATX value (NC_FORMATX_NC_HDF5 and NC_FORMATX_NCZARR, currently). So the *nc_plugin_path_set* function, will take the paths it is given and propagate them to each of the NC_FORMATX dispatchers to store in a way that is appropriate to the given dispatcher.
There is a complication with respect to the *nc_plugin_path_get* function. It is possible for users to bypass the netcdf API and modify the HDF5 plugin paths directly. This can result in an inconsistent plugin path between the value used by HDF5 and the global value used by netcdf-c. Since there is no obvious fix for this, we warn the user of this possibility and otherwise ignore it.
## Test Changes
* New tests<br>
a. unit_test/run_pluginpaths.sh -- was created to test this new capability.<br>
b. A new test utility has been added as *unit_test/run_dfaltpluginpath.sh* to test the default plugin path list.
* New test support utilities<br>
a. unit_test/ncpluginpath.c -- report current state of the plugin path<br>
b. unit_test/tst_pluginpaths.c -- test program to support run_pluginpaths.sh
## Documentation
* A new file -- docs/pluginpath.md -- provides documentation of the new API. It includes some
material taken fro filters.md.
## Other Major Changes
1. Cleanup the whole plugin path decision tree. This is described in the *docs/pluginpath.md* document and summarized in Addendum 2 below.
2. I noticed that the ncdump/testpathcvt.sh had been disabled, so fixed and re-enabled it. This necessitated some significant changes to dpathmgr.c.
## Misc. Changes
1. Add some path manipulation utilities to netcf_aux.h
2. Fix some minor bugs in netcdf_json.h
3. Convert netcdf_json.h and netcdf_proplist.h to BUILT_SOURCE.
4. Add NETCDF_ENABLE_HDF5 as synonym for USE_HDF5
5. Fix some size_t <-> int conversion warnings.
6. Encountered and fixed the Windows \r\n problem in tst_pluginpaths.c.
7. Cleanup some minor CMakeLists.txt problems.
8. Provide an implementation of echo -n since it appears to not be
available on all platforms.
9. Add a property list mechanism to pass environmental information to filters.
10. Cleanup Doxyfile.in
11. Fixed a memory leak in libdap2; surprised that I did not find this earlier.
## Addendum 1: Proposed API
The API makes use of a counted vector of strings representing the sequence of directories in the path. The relevant type definition is as follows.
````
typedef struct NCPluginList {size_t ndirs; char** dirs;} NCPluginList;
````
The API proposed in this PR looks like this (from netcdf-c/include/netcdf_filter.h).
* ````int nc_plugin_path_ndirs(size_t* ndirsp);````
Arguments: *ndirsp* -- store the number of directories in this memory.
This function returns the number of directories in the sequence if internal directories of the internal plugin path list.
* ````int nc_plugin_path_get(NCPluginList* dirs);````
Arguments: *dirs* -- counted vector for storing the sequence of directies in the internal path list.
This function returns the current sequence of directories from the internal plugin path list. Since this function does not modify the plugin path, it does not need to be locked; it is only when used to get the path to be modified that locking is required. If the value of *dirs.dirs* is NULL (the normal case), then memory is allocated to hold the vector of directories. Otherwise, use the memory of *dirs.dirs* to hold the vector of directories.
* ````int nc_plugin_path_set(const NCPluginList* dirs);````
Arguments: *dirs* -- counted vector for providing the new sequence of directories in the internal path list.
This function empties the current internal path sequence and replaces it with the sequence of directories argument. Using an *ndirs* argument of 0 will clear the set of plugin paths.
## Addendum 2: Build-Time and Run-Time Constants.
### Build-Time Constants
<table style="border:2px solid black;border-collapse:collapse">
<tr style="outline: thin solid;" align="center"><td colspan="4">Table showing the build-time computation of NETCDF_PLUGIN_INSTALL_DIR and NETCDF_PLUGIN_SEARCH_PATH.</td>
<tr style="outline: thin solid" ><th>--with-plugin-dir<th>--prefix<th>NETCDF_PLUGIN_INSTALL_DIR<th>NETCDF_PLUGIN_SEARCH_PATH
<tr style="outline: thin solid" ><td>undefined<td>undefined<td>undefined<td>PLATFORMDEFALT
<tr style="outline: thin solid" ><td>undefined<td><abspath-prefix><td><abspath-prefix>/hdf5/lib/plugin<td><abspath-prefix>/hdf5/lib/plugin<SEP>PLATFORMDEFALT
<tr style="outline: thin solid" ><td><abspath-plugins><td>N.A.<td><abspath-plugins><td><abspath-plugins><SEP>PLATFORMDEFALT
</table>
<table style="border:2px solid black;border-collapse:collapse">
<tr style="outline: thin solid" align="center"><td colspan="2">Table showing the computation of the initial global plugin path</td>
<tr style="outline: thin solid"><th>HDF5_PLUGIN_PATH<th>Initial global plugin path
<tr style="outline: thin solid"><td>undefined<td>NETCDF_PLUGIN_SEARCH_PATH
<tr style="outline: thin solid"><td><path1;...pathn><td><path1;...pathn>
</table>
As suggested by Ward, I ensured that this PR supports
read backward compatibility with old key format.
This addition also adds a test case for this.
## Misc. Other Changes
* Remove some unused code
* Cleanup json error handling
* Fix some more unsigned/signed conversions warning
As discussed in a netcdf meeting, convert NCZarr V2 to store all netcdf-4 specific info as attributes. This improves interoperability with other Zarr implementations by no longer using non-standard keys.
## Other Changes
* Remove support for older NCZarr formats.
* Update anonymous dimension naming
* Begin the process of fixing the -Wconversion and -Wsign-compare warnings in libnczarr, nczarr_test, and v3_nczarr_test.
* Update docs/nczarr.md
* Rebuild using the .y and .l files
The NETCDF_ENABLE_SET_LOG_LEVEL_FUNC option is apparently not used, but
is effectively used to set NETCDF_ENABLE_SET_LOG_LEVEL.
This is not clear from the build files CMakeLists.txt and configure.ac.
So this PR cleanups the code to make it more clear what is going on.
re: Issue https://github.com/Unidata/netcdf-c/issues/2927
The NC_s3sdkinitialize NC_s3sdkfinalize functions were
misplaced. They should have been moved from ds3util.c to
ncs3sdk_h5.c. When using ncs3sdl_aws.cpp, this resulted in a
duplicate definition.
Also, found and fixed a memory leak in the NCZarr S3 code.
# Description
Remove various obsolete build options. Also do some code movement.
## Specific Changes
* The remotetest server is sometimes unstable, so provide a mechanism
to force disabling calls to remotetest.unidata.ucar.edu.
This is enabled by adding a repository variable named
REMOTETESTDOWN with the value "yes".
* Fix CMakeLists.txt to use the uname command as an alternate
to using the hostname command (which does not work under cygwin).
* Remove the JNA stuff as obsolete
* Remove the ENABLE_CLIENTSIDE_FILTERS options since it has been
disabled for a while.
* Fix bad option flag in some github action .yml files: change --disable-xml2 to --disable-libxml2
* Collect globalstate definitions into nc4internal.h
* Remove ENABLE_NCZARR_FILTERS_TESTING option as obsolete and replace
with ENABLE_NCZARR_FILTERS
* Move some dispatcher independent functions from libsrc4/nc4internal.c to libdispatch/ddispatch.c
* As a long term goal, and because it is now the case that --enable-nczarr
=> USE_NETCDF4, make the external options --enable-netcdf-4 and
--enable-netcdf4 obsolete in favor of --enable-hdf5
We will do the following for one more release cycle.
1. Make --enable-netcdf-4 be an alias for --enable-netcdf4.
2. Make --enable-netcdf4 an alias for --enable-hdf5.
3. Internally, convert most uses of USE_NETCDF_4 ad USE_NETCDF4 to USE_HDF5
After the next release, --enable-netcdf-4 and --enable-netcdf4 will
be removed.
NOTE: it is important that this fix gets into 4.9.3
re: Issue https://github.com/Unidata/netcdf-c/issues/2798
## Modifications
* This PR includes PR https://github.com/Unidata/netcdf-c/pull/2813
* Support the following AWS environment variables in the internal S3 library
(they are already supported by aws-sdk-cpp).
- AWS_REGION
- AWS_DEFAULT_REGION
- AWS_ACCESS_KEY_ID
- AWS_CONFIG_FILE
- AWS_PROFILE
- AWS_SECRET_ACCESS_KEY
- (source https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html).
* Support an empty region when specifying s3.amazonaws.com as the host.
* Move some S3/AWS related functions to ds3util.c
* Add a test case to test empty region and AWS_[DEFAULT]_REGION.
The most critical bug is in nch5s3comms.c.
I for some reason assumed that signing keys
did not contain any zero bytes. But obviously
it can, so a test was removed.
Other fixes:
1. Guarantee allocated memory is initialized to all zeros.
2. Cleanup errmsg handling in libncpoco.
3. Fix processing of aws list-objects-v2 because I misread the syntax.
Mostly just add an explicit cast when calling `malloc` and its
variants. Sometimes instead change the type of a local variable if
this would silence multiple warnings.
