2018-12-07 06:51:35 +08:00
|
|
|
# Copyright 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002,
|
|
|
|
# 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014,
|
|
|
|
# 2015, 2016, 2017, 2018
|
|
|
|
# University Corporation for Atmospheric Research/Unidata.
|
|
|
|
|
|
|
|
# See netcdf-c/COPYRIGHT file for more info.
|
|
|
|
|
2022-08-28 10:21:13 +08:00
|
|
|
# Load only once
|
|
|
|
if test "x$TEST_COMMON_SH" = x ; then
|
|
|
|
export TEST_COMMON_SH=1
|
|
|
|
|
Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.
The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.
## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
- "size" with an integer value representing the (current) size of the dimension.
- "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.
For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.
Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.
## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
AWS Transfer Utility mechanism. This is controlled by the
```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-27 06:56:48 +08:00
|
|
|
# Define various global constants
|
|
|
|
|
Extend the netcdf API to support programmatic changes to the plugin search path
Replaces PR https://github.com/Unidata/netcdf-c/pull/3024
and PR https://github.com/Unidata/netcdf-c/pull/3033
re: https://github.com/Unidata/netcdf-c/issues/2753
As suggested by Ed Hartnett, This PR extends the netcdf.h API to support programmatic control over the search path used to locate plugins.
I created several different APIs, but finally settled on the following API as being the simplest possible. It does have the disadvantage that it requires use of a global lock (not implemented) if used in a threaded environment.
Specifically, note that modifying the plugin path must be done "atomically". That is, in a multi-threaded environment, it is important that the sequence of actions involved in setting up the plugin path must be done by a single processor or in some other way as to guarantee that two or more processors are not simultaneously accessing the plugin path get/set operations.
As an example, assume there exists a mutex lock called PLUGINLOCK. Then any processor accessing the plugin paths should operate as follows:
````
lock(PLUGINLOCK);
nc_plugin_path_get(...);
<rebuild plugin path>
nc_plugin_path_set(...);
unlock(PLUGINLOCK);
````
## Internal Architecture
It is assumed here that there only needs to be a single set of plugin path directories that is shared by all filter code and is independent of any file descriptor; it is global in other words. This means, for example, that the path list for NCZarr and for HDF5 will always be the same.
However internally, processing the set of plugin paths depends on the particular NC_FORMATX value (NC_FORMATX_NC_HDF5 and NC_FORMATX_NCZARR, currently). So the *nc_plugin_path_set* function, will take the paths it is given and propagate them to each of the NC_FORMATX dispatchers to store in a way that is appropriate to the given dispatcher.
There is a complication with respect to the *nc_plugin_path_get* function. It is possible for users to bypass the netcdf API and modify the HDF5 plugin paths directly. This can result in an inconsistent plugin path between the value used by HDF5 and the global value used by netcdf-c. Since there is no obvious fix for this, we warn the user of this possibility and otherwise ignore it.
## Test Changes
* New tests<br>
a. unit_test/run_pluginpaths.sh -- was created to test this new capability.<br>
b. A new test utility has been added as *unit_test/run_dfaltpluginpath.sh* to test the default plugin path list.
* New test support utilities<br>
a. unit_test/ncpluginpath.c -- report current state of the plugin path<br>
b. unit_test/tst_pluginpaths.c -- test program to support run_pluginpaths.sh
## Documentation
* A new file -- docs/pluginpath.md -- provides documentation of the new API. It includes some
material taken fro filters.md.
## Other Major Changes
1. Cleanup the whole plugin path decision tree. This is described in the *docs/pluginpath.md* document and summarized in Addendum 2 below.
2. I noticed that the ncdump/testpathcvt.sh had been disabled, so fixed and re-enabled it. This necessitated some significant changes to dpathmgr.c.
## Misc. Changes
1. Add some path manipulation utilities to netcf_aux.h
2. Fix some minor bugs in netcdf_json.h
3. Convert netcdf_json.h and netcdf_proplist.h to BUILT_SOURCE.
4. Add NETCDF_ENABLE_HDF5 as synonym for USE_HDF5
5. Fix some size_t <-> int conversion warnings.
6. Encountered and fixed the Windows \r\n problem in tst_pluginpaths.c.
7. Cleanup some minor CMakeLists.txt problems.
8. Provide an implementation of echo -n since it appears to not be
available on all platforms.
9. Add a property list mechanism to pass environmental information to filters.
10. Cleanup Doxyfile.in
11. Fixed a memory leak in libdap2; surprised that I did not find this earlier.
## Addendum 1: Proposed API
The API makes use of a counted vector of strings representing the sequence of directories in the path. The relevant type definition is as follows.
````
typedef struct NCPluginList {size_t ndirs; char** dirs;} NCPluginList;
````
The API proposed in this PR looks like this (from netcdf-c/include/netcdf_filter.h).
* ````int nc_plugin_path_ndirs(size_t* ndirsp);````
Arguments: *ndirsp* -- store the number of directories in this memory.
This function returns the number of directories in the sequence if internal directories of the internal plugin path list.
* ````int nc_plugin_path_get(NCPluginList* dirs);````
Arguments: *dirs* -- counted vector for storing the sequence of directies in the internal path list.
This function returns the current sequence of directories from the internal plugin path list. Since this function does not modify the plugin path, it does not need to be locked; it is only when used to get the path to be modified that locking is required. If the value of *dirs.dirs* is NULL (the normal case), then memory is allocated to hold the vector of directories. Otherwise, use the memory of *dirs.dirs* to hold the vector of directories.
* ````int nc_plugin_path_set(const NCPluginList* dirs);````
Arguments: *dirs* -- counted vector for providing the new sequence of directories in the internal path list.
This function empties the current internal path sequence and replaces it with the sequence of directories argument. Using an *ndirs* argument of 0 will clear the set of plugin paths.
## Addendum 2: Build-Time and Run-Time Constants.
### Build-Time Constants
<table style="border:2px solid black;border-collapse:collapse">
<tr style="outline: thin solid;" align="center"><td colspan="4">Table showing the build-time computation of NETCDF_PLUGIN_INSTALL_DIR and NETCDF_PLUGIN_SEARCH_PATH.</td>
<tr style="outline: thin solid" ><th>--with-plugin-dir<th>--prefix<th>NETCDF_PLUGIN_INSTALL_DIR<th>NETCDF_PLUGIN_SEARCH_PATH
<tr style="outline: thin solid" ><td>undefined<td>undefined<td>undefined<td>PLATFORMDEFALT
<tr style="outline: thin solid" ><td>undefined<td><abspath-prefix><td><abspath-prefix>/hdf5/lib/plugin<td><abspath-prefix>/hdf5/lib/plugin<SEP>PLATFORMDEFALT
<tr style="outline: thin solid" ><td><abspath-plugins><td>N.A.<td><abspath-plugins><td><abspath-plugins><SEP>PLATFORMDEFALT
</table>
<table style="border:2px solid black;border-collapse:collapse">
<tr style="outline: thin solid" align="center"><td colspan="2">Table showing the computation of the initial global plugin path</td>
<tr style="outline: thin solid"><th>HDF5_PLUGIN_PATH<th>Initial global plugin path
<tr style="outline: thin solid"><td>undefined<td>NETCDF_PLUGIN_SEARCH_PATH
<tr style="outline: thin solid"><td><path1;...pathn><td><path1;...pathn>
</table>
2024-10-19 09:23:59 +08:00
|
|
|
# Define various build and install paths
|
2024-05-16 08:46:25 +08:00
|
|
|
abs_top_srcdir=@abs_top_srcdir@
|
|
|
|
abs_top_builddir=@abs_top_builddir@
|
|
|
|
TOPSRCDIR="${abs_top_srcdir}"
|
|
|
|
TOPBUILDDIR="${abs_top_builddir}"
|
Extend the netcdf API to support programmatic changes to the plugin search path
Replaces PR https://github.com/Unidata/netcdf-c/pull/3024
and PR https://github.com/Unidata/netcdf-c/pull/3033
re: https://github.com/Unidata/netcdf-c/issues/2753
As suggested by Ed Hartnett, This PR extends the netcdf.h API to support programmatic control over the search path used to locate plugins.
I created several different APIs, but finally settled on the following API as being the simplest possible. It does have the disadvantage that it requires use of a global lock (not implemented) if used in a threaded environment.
Specifically, note that modifying the plugin path must be done "atomically". That is, in a multi-threaded environment, it is important that the sequence of actions involved in setting up the plugin path must be done by a single processor or in some other way as to guarantee that two or more processors are not simultaneously accessing the plugin path get/set operations.
As an example, assume there exists a mutex lock called PLUGINLOCK. Then any processor accessing the plugin paths should operate as follows:
````
lock(PLUGINLOCK);
nc_plugin_path_get(...);
<rebuild plugin path>
nc_plugin_path_set(...);
unlock(PLUGINLOCK);
````
## Internal Architecture
It is assumed here that there only needs to be a single set of plugin path directories that is shared by all filter code and is independent of any file descriptor; it is global in other words. This means, for example, that the path list for NCZarr and for HDF5 will always be the same.
However internally, processing the set of plugin paths depends on the particular NC_FORMATX value (NC_FORMATX_NC_HDF5 and NC_FORMATX_NCZARR, currently). So the *nc_plugin_path_set* function, will take the paths it is given and propagate them to each of the NC_FORMATX dispatchers to store in a way that is appropriate to the given dispatcher.
There is a complication with respect to the *nc_plugin_path_get* function. It is possible for users to bypass the netcdf API and modify the HDF5 plugin paths directly. This can result in an inconsistent plugin path between the value used by HDF5 and the global value used by netcdf-c. Since there is no obvious fix for this, we warn the user of this possibility and otherwise ignore it.
## Test Changes
* New tests<br>
a. unit_test/run_pluginpaths.sh -- was created to test this new capability.<br>
b. A new test utility has been added as *unit_test/run_dfaltpluginpath.sh* to test the default plugin path list.
* New test support utilities<br>
a. unit_test/ncpluginpath.c -- report current state of the plugin path<br>
b. unit_test/tst_pluginpaths.c -- test program to support run_pluginpaths.sh
## Documentation
* A new file -- docs/pluginpath.md -- provides documentation of the new API. It includes some
material taken fro filters.md.
## Other Major Changes
1. Cleanup the whole plugin path decision tree. This is described in the *docs/pluginpath.md* document and summarized in Addendum 2 below.
2. I noticed that the ncdump/testpathcvt.sh had been disabled, so fixed and re-enabled it. This necessitated some significant changes to dpathmgr.c.
## Misc. Changes
1. Add some path manipulation utilities to netcf_aux.h
2. Fix some minor bugs in netcdf_json.h
3. Convert netcdf_json.h and netcdf_proplist.h to BUILT_SOURCE.
4. Add NETCDF_ENABLE_HDF5 as synonym for USE_HDF5
5. Fix some size_t <-> int conversion warnings.
6. Encountered and fixed the Windows \r\n problem in tst_pluginpaths.c.
7. Cleanup some minor CMakeLists.txt problems.
8. Provide an implementation of echo -n since it appears to not be
available on all platforms.
9. Add a property list mechanism to pass environmental information to filters.
10. Cleanup Doxyfile.in
11. Fixed a memory leak in libdap2; surprised that I did not find this earlier.
## Addendum 1: Proposed API
The API makes use of a counted vector of strings representing the sequence of directories in the path. The relevant type definition is as follows.
````
typedef struct NCPluginList {size_t ndirs; char** dirs;} NCPluginList;
````
The API proposed in this PR looks like this (from netcdf-c/include/netcdf_filter.h).
* ````int nc_plugin_path_ndirs(size_t* ndirsp);````
Arguments: *ndirsp* -- store the number of directories in this memory.
This function returns the number of directories in the sequence if internal directories of the internal plugin path list.
* ````int nc_plugin_path_get(NCPluginList* dirs);````
Arguments: *dirs* -- counted vector for storing the sequence of directies in the internal path list.
This function returns the current sequence of directories from the internal plugin path list. Since this function does not modify the plugin path, it does not need to be locked; it is only when used to get the path to be modified that locking is required. If the value of *dirs.dirs* is NULL (the normal case), then memory is allocated to hold the vector of directories. Otherwise, use the memory of *dirs.dirs* to hold the vector of directories.
* ````int nc_plugin_path_set(const NCPluginList* dirs);````
Arguments: *dirs* -- counted vector for providing the new sequence of directories in the internal path list.
This function empties the current internal path sequence and replaces it with the sequence of directories argument. Using an *ndirs* argument of 0 will clear the set of plugin paths.
## Addendum 2: Build-Time and Run-Time Constants.
### Build-Time Constants
<table style="border:2px solid black;border-collapse:collapse">
<tr style="outline: thin solid;" align="center"><td colspan="4">Table showing the build-time computation of NETCDF_PLUGIN_INSTALL_DIR and NETCDF_PLUGIN_SEARCH_PATH.</td>
<tr style="outline: thin solid" ><th>--with-plugin-dir<th>--prefix<th>NETCDF_PLUGIN_INSTALL_DIR<th>NETCDF_PLUGIN_SEARCH_PATH
<tr style="outline: thin solid" ><td>undefined<td>undefined<td>undefined<td>PLATFORMDEFALT
<tr style="outline: thin solid" ><td>undefined<td><abspath-prefix><td><abspath-prefix>/hdf5/lib/plugin<td><abspath-prefix>/hdf5/lib/plugin<SEP>PLATFORMDEFALT
<tr style="outline: thin solid" ><td><abspath-plugins><td>N.A.<td><abspath-plugins><td><abspath-plugins><SEP>PLATFORMDEFALT
</table>
<table style="border:2px solid black;border-collapse:collapse">
<tr style="outline: thin solid" align="center"><td colspan="2">Table showing the computation of the initial global plugin path</td>
<tr style="outline: thin solid"><th>HDF5_PLUGIN_PATH<th>Initial global plugin path
<tr style="outline: thin solid"><td>undefined<td>NETCDF_PLUGIN_SEARCH_PATH
<tr style="outline: thin solid"><td><path1;...pathn><td><path1;...pathn>
</table>
2024-10-19 09:23:59 +08:00
|
|
|
|
Codify cross-platform file paths
The netcdf-c code has to deal with a variety of platforms:
Windows, OSX, Linux, Cygwin, MSYS, etc. These platforms differ
significantly in the kind of file paths that they accept. So in
order to handle this, I have created a set of replacements for
the most common file system operations such as _open_ or _fopen_
or _access_ to manage the file path differences correctly.
A more limited version of this idea was already implemented via
the ncwinpath.h and dwinpath.c code. So this can be viewed as a
replacement for that code. And in path in many cases, the only
change that was required was to replace '#include <ncwinpath.h>'
with '#include <ncpathmgt.h>' and then replace file operation
calls with the NCxxx equivalent from ncpathmgr.h Note that
recently, the ncwinpath.h was renamed ncpathmgmt.h, so this pull
request should not require dealing with winpath.
The heart of the change is include/ncpathmgmt.h, which provides
alternate operations such as NCfopen or NCaccess and which properly
parse and rebuild path arguments to work for the platform on which
the code is executing. This mostly matters for Windows because of the
way that it uses backslash and drive letters, as compared to *nix*.
One important feature is that the user can do string manipulations
on a file path without having to worry too much about the platform
because the path management code will properly handle most mixed cases.
So one can for example concatenate a path suffix that uses forward
slashes to a Windows path and have it work correctly.
The conversion code is in libdispatch/dpathmgr.c, and the
important function there is NCpathcvt which does the proper
conversions to the local path format.
As a rule, most code should just replace their file operations with
the corresponding NCxxx ones defined in include/ncpathmgmt.h. These
NCxxx functions all call NCpathcvt on their path arguments before
executing the actual file operation.
In some rare cases, the client may need to directly use NCpathcvt,
but this should be avoided as much as possible. If there is a need
for supporting a new file operation not already in ncpathmgmt.h, then
use the code in dpathmgr.c as a template. Also please notify Unidata
so we can include it as a formal part or our supported operations.
Also, if you see an operation in the library that is not using the
NCxxx form, then please submit an issue so we can fix it.
Misc. Changes:
* Clean up the utf8 testing code; it is impossible to get some
tests to work under windows using shell scripts; the args do
not pass as utf8 but as some other encoding.
* Added an extra utf8 test case: test_unicode_path.sh
* Add a true test for HDF5 1.10.6 or later because as noted in
PR https://github.com/Unidata/netcdf-c/pull/1794,
HDF5 changed its Windows file path handling.
2021-03-05 04:41:31 +08:00
|
|
|
FP_ISCMAKE=@ISCMAKE@
|
|
|
|
FP_ISMSVC=@ISMSVC@
|
2022-02-09 11:53:30 +08:00
|
|
|
FP_WINVERMAJOR=@WINVERMAJOR@
|
|
|
|
FP_WINVERBUILD=@WINVERBUILD@
|
2021-05-20 07:19:33 +08:00
|
|
|
FP_ISCYGWIN=@ISCYGWIN@
|
2021-11-04 02:49:54 +08:00
|
|
|
FP_ISMINGW=@ISMINGW@
|
2021-12-24 13:18:56 +08:00
|
|
|
FP_ISMSYS=@ISMSYS@
|
2022-03-19 11:01:40 +08:00
|
|
|
FP_ISOSX=@ISOSX@
|
2022-04-08 06:57:35 +08:00
|
|
|
|
2022-02-09 11:53:30 +08:00
|
|
|
FP_ISREGEDIT=@ISREGEDIT@
|
2022-05-15 06:05:48 +08:00
|
|
|
FP_USEPLUGINS=@USEPLUGINS@
|
Codify cross-platform file paths
The netcdf-c code has to deal with a variety of platforms:
Windows, OSX, Linux, Cygwin, MSYS, etc. These platforms differ
significantly in the kind of file paths that they accept. So in
order to handle this, I have created a set of replacements for
the most common file system operations such as _open_ or _fopen_
or _access_ to manage the file path differences correctly.
A more limited version of this idea was already implemented via
the ncwinpath.h and dwinpath.c code. So this can be viewed as a
replacement for that code. And in path in many cases, the only
change that was required was to replace '#include <ncwinpath.h>'
with '#include <ncpathmgt.h>' and then replace file operation
calls with the NCxxx equivalent from ncpathmgr.h Note that
recently, the ncwinpath.h was renamed ncpathmgmt.h, so this pull
request should not require dealing with winpath.
The heart of the change is include/ncpathmgmt.h, which provides
alternate operations such as NCfopen or NCaccess and which properly
parse and rebuild path arguments to work for the platform on which
the code is executing. This mostly matters for Windows because of the
way that it uses backslash and drive letters, as compared to *nix*.
One important feature is that the user can do string manipulations
on a file path without having to worry too much about the platform
because the path management code will properly handle most mixed cases.
So one can for example concatenate a path suffix that uses forward
slashes to a Windows path and have it work correctly.
The conversion code is in libdispatch/dpathmgr.c, and the
important function there is NCpathcvt which does the proper
conversions to the local path format.
As a rule, most code should just replace their file operations with
the corresponding NCxxx ones defined in include/ncpathmgmt.h. These
NCxxx functions all call NCpathcvt on their path arguments before
executing the actual file operation.
In some rare cases, the client may need to directly use NCpathcvt,
but this should be avoided as much as possible. If there is a need
for supporting a new file operation not already in ncpathmgmt.h, then
use the code in dpathmgr.c as a template. Also please notify Unidata
so we can include it as a formal part or our supported operations.
Also, if you see an operation in the library that is not using the
NCxxx form, then please submit an issue so we can fix it.
Misc. Changes:
* Clean up the utf8 testing code; it is impossible to get some
tests to work under windows using shell scripts; the args do
not pass as utf8 but as some other encoding.
* Added an extra utf8 test case: test_unicode_path.sh
* Add a true test for HDF5 1.10.6 or later because as noted in
PR https://github.com/Unidata/netcdf-c/pull/1794,
HDF5 changed its Windows file path handling.
2021-03-05 04:41:31 +08:00
|
|
|
|
|
|
|
# Feature flags
|
|
|
|
FEATURE_HDF5=@HAS_HDF5@
|
2022-04-29 05:51:20 +08:00
|
|
|
FEATURE_PARALLEL=@HAS_PARALLEL@
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2020-08-18 09:15:47 +08:00
|
|
|
# Define selected features of the build
|
|
|
|
FEATURE_HDF5=@HAS_HDF5@
|
2023-03-03 10:51:02 +08:00
|
|
|
FEATURE_FILTERTESTS=@DO_FILTER_TESTS@
|
|
|
|
FEATURE_BYTERANGE=@HAS_BYTERANGE@
|
2023-04-26 07:15:06 +08:00
|
|
|
FEATURE_ROS3=@HAS_HDF5_ROS3@
|
|
|
|
FEATURE_S3_AWS=@HAS_S3_AWS@
|
|
|
|
FEATURE_S3_INTERNAL=@HAS_S3_INTERNAL@
|
2023-03-03 10:51:02 +08:00
|
|
|
FEATURE_S3=@HAS_S3@
|
Support installation of filters into user-specified location
re: https://github.com/Unidata/netcdf-c/issues/2294
Ed Hartnett suggested that the netcdf library installation process
be extended to install the standard filters into a user specified
location. The user can then set HDF5_PLUGIN_PATH to that location.
This PR provides that capability using:
````
configure option: --with-plugin-dir=<absolute directory path>
cmake option: -DPLUGIN_INSTALL_DIR=<absolute directory path>
````
Currently, the following plugins are always installed, if
available: bzip2, zstd, blosc.
If NCZarr is enabled, then additional plugins are installed:
fletcher32, shuffle, deflate, szip.
Additionally, the necessary codec support is installed
for each of the above filters that is installed.
## Changes:
1. Cleanup handling of built-in bzip2.
2. Add documentation to docs/filters.md
3. Re-factor the NCZarr codec libraries
4. Add a test, although it can only be exercised after
the library is installed, so it cannot be used during
normal testing.
5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
|
|
|
FEATURE_NCZARR=@HAS_NCZARR@
|
2024-03-19 04:40:19 +08:00
|
|
|
FEATURE_S3TESTS=@NETCDF_ENABLE_S3_TESTING@
|
2024-05-16 08:46:25 +08:00
|
|
|
FEATURE_NCZARR_ZIP=@HAS_NCZARR_ZIP@
|
2023-04-26 07:15:06 +08:00
|
|
|
FEATURE_LARGE_TESTS=@DO_LARGE_TESTS@
|
2020-08-18 09:15:47 +08:00
|
|
|
|
Extend the netcdf API to support programmatic changes to the plugin search path
Replaces PR https://github.com/Unidata/netcdf-c/pull/3024
and PR https://github.com/Unidata/netcdf-c/pull/3033
re: https://github.com/Unidata/netcdf-c/issues/2753
As suggested by Ed Hartnett, This PR extends the netcdf.h API to support programmatic control over the search path used to locate plugins.
I created several different APIs, but finally settled on the following API as being the simplest possible. It does have the disadvantage that it requires use of a global lock (not implemented) if used in a threaded environment.
Specifically, note that modifying the plugin path must be done "atomically". That is, in a multi-threaded environment, it is important that the sequence of actions involved in setting up the plugin path must be done by a single processor or in some other way as to guarantee that two or more processors are not simultaneously accessing the plugin path get/set operations.
As an example, assume there exists a mutex lock called PLUGINLOCK. Then any processor accessing the plugin paths should operate as follows:
````
lock(PLUGINLOCK);
nc_plugin_path_get(...);
<rebuild plugin path>
nc_plugin_path_set(...);
unlock(PLUGINLOCK);
````
## Internal Architecture
It is assumed here that there only needs to be a single set of plugin path directories that is shared by all filter code and is independent of any file descriptor; it is global in other words. This means, for example, that the path list for NCZarr and for HDF5 will always be the same.
However internally, processing the set of plugin paths depends on the particular NC_FORMATX value (NC_FORMATX_NC_HDF5 and NC_FORMATX_NCZARR, currently). So the *nc_plugin_path_set* function, will take the paths it is given and propagate them to each of the NC_FORMATX dispatchers to store in a way that is appropriate to the given dispatcher.
There is a complication with respect to the *nc_plugin_path_get* function. It is possible for users to bypass the netcdf API and modify the HDF5 plugin paths directly. This can result in an inconsistent plugin path between the value used by HDF5 and the global value used by netcdf-c. Since there is no obvious fix for this, we warn the user of this possibility and otherwise ignore it.
## Test Changes
* New tests<br>
a. unit_test/run_pluginpaths.sh -- was created to test this new capability.<br>
b. A new test utility has been added as *unit_test/run_dfaltpluginpath.sh* to test the default plugin path list.
* New test support utilities<br>
a. unit_test/ncpluginpath.c -- report current state of the plugin path<br>
b. unit_test/tst_pluginpaths.c -- test program to support run_pluginpaths.sh
## Documentation
* A new file -- docs/pluginpath.md -- provides documentation of the new API. It includes some
material taken fro filters.md.
## Other Major Changes
1. Cleanup the whole plugin path decision tree. This is described in the *docs/pluginpath.md* document and summarized in Addendum 2 below.
2. I noticed that the ncdump/testpathcvt.sh had been disabled, so fixed and re-enabled it. This necessitated some significant changes to dpathmgr.c.
## Misc. Changes
1. Add some path manipulation utilities to netcf_aux.h
2. Fix some minor bugs in netcdf_json.h
3. Convert netcdf_json.h and netcdf_proplist.h to BUILT_SOURCE.
4. Add NETCDF_ENABLE_HDF5 as synonym for USE_HDF5
5. Fix some size_t <-> int conversion warnings.
6. Encountered and fixed the Windows \r\n problem in tst_pluginpaths.c.
7. Cleanup some minor CMakeLists.txt problems.
8. Provide an implementation of echo -n since it appears to not be
available on all platforms.
9. Add a property list mechanism to pass environmental information to filters.
10. Cleanup Doxyfile.in
11. Fixed a memory leak in libdap2; surprised that I did not find this earlier.
## Addendum 1: Proposed API
The API makes use of a counted vector of strings representing the sequence of directories in the path. The relevant type definition is as follows.
````
typedef struct NCPluginList {size_t ndirs; char** dirs;} NCPluginList;
````
The API proposed in this PR looks like this (from netcdf-c/include/netcdf_filter.h).
* ````int nc_plugin_path_ndirs(size_t* ndirsp);````
Arguments: *ndirsp* -- store the number of directories in this memory.
This function returns the number of directories in the sequence if internal directories of the internal plugin path list.
* ````int nc_plugin_path_get(NCPluginList* dirs);````
Arguments: *dirs* -- counted vector for storing the sequence of directies in the internal path list.
This function returns the current sequence of directories from the internal plugin path list. Since this function does not modify the plugin path, it does not need to be locked; it is only when used to get the path to be modified that locking is required. If the value of *dirs.dirs* is NULL (the normal case), then memory is allocated to hold the vector of directories. Otherwise, use the memory of *dirs.dirs* to hold the vector of directories.
* ````int nc_plugin_path_set(const NCPluginList* dirs);````
Arguments: *dirs* -- counted vector for providing the new sequence of directories in the internal path list.
This function empties the current internal path sequence and replaces it with the sequence of directories argument. Using an *ndirs* argument of 0 will clear the set of plugin paths.
## Addendum 2: Build-Time and Run-Time Constants.
### Build-Time Constants
<table style="border:2px solid black;border-collapse:collapse">
<tr style="outline: thin solid;" align="center"><td colspan="4">Table showing the build-time computation of NETCDF_PLUGIN_INSTALL_DIR and NETCDF_PLUGIN_SEARCH_PATH.</td>
<tr style="outline: thin solid" ><th>--with-plugin-dir<th>--prefix<th>NETCDF_PLUGIN_INSTALL_DIR<th>NETCDF_PLUGIN_SEARCH_PATH
<tr style="outline: thin solid" ><td>undefined<td>undefined<td>undefined<td>PLATFORMDEFALT
<tr style="outline: thin solid" ><td>undefined<td><abspath-prefix><td><abspath-prefix>/hdf5/lib/plugin<td><abspath-prefix>/hdf5/lib/plugin<SEP>PLATFORMDEFALT
<tr style="outline: thin solid" ><td><abspath-plugins><td>N.A.<td><abspath-plugins><td><abspath-plugins><SEP>PLATFORMDEFALT
</table>
<table style="border:2px solid black;border-collapse:collapse">
<tr style="outline: thin solid" align="center"><td colspan="2">Table showing the computation of the initial global plugin path</td>
<tr style="outline: thin solid"><th>HDF5_PLUGIN_PATH<th>Initial global plugin path
<tr style="outline: thin solid"><td>undefined<td>NETCDF_PLUGIN_SEARCH_PATH
<tr style="outline: thin solid"><td><path1;...pathn><td><path1;...pathn>
</table>
2024-10-19 09:23:59 +08:00
|
|
|
FEATURE_PLUGIN_INSTALL_DIR="@NETCDF_PLUGIN_INSTALL_DIR@"
|
|
|
|
FEATURE_PLUGIN_SEARCH_PATH="@NETCDF_PLUGIN_SEARCH_PATH@"
|
|
|
|
|
2022-08-28 10:21:13 +08:00
|
|
|
# Thredds-test server is currently disabled
|
|
|
|
#FEATURE_THREDDSTEST=1
|
|
|
|
|
Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.
The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.
## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
- "size" with an integer value representing the (current) size of the dimension.
- "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.
For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.
Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.
## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
AWS Transfer Utility mechanism. This is controlled by the
```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-27 06:56:48 +08:00
|
|
|
# This is the Unidata S3 test bucket
|
|
|
|
# All S3 tests should use this to store intermediate results.
|
|
|
|
S3TESTBUCKET=@S3TESTBUCKET@
|
|
|
|
|
|
|
|
# This is the s3 path within the Unidata bucket;
|
|
|
|
# All S3 tests should use this to store intermediate results.
|
|
|
|
S3TESTSUBTREE=@S3TESTSUBTREE@
|
|
|
|
|
|
|
|
TESTUID=@TESTUID@
|
|
|
|
|
2018-01-17 02:00:09 +08:00
|
|
|
set -e
|
|
|
|
|
2017-03-09 08:01:10 +08:00
|
|
|
# Figure out various locations in the src/build tree.
|
|
|
|
# This is relatively fragile code and is essentially
|
|
|
|
# specific to netcdf-c. It does, however, have the virtue
|
|
|
|
# of isolating all this nonsense into one place.
|
|
|
|
# This will get somewhat simplified (I hope) when
|
|
|
|
# we move to a separate test_utilities directory
|
|
|
|
|
|
|
|
# This code is intended to provide constants
|
|
|
|
# for accessing various objects in the src/build
|
|
|
|
# tree(s) across multiple ways of building netcdf-c.
|
|
|
|
# Currently, the following build situations are supported.
|
|
|
|
# 1. Autoconf with make check: the src and build trees are the same
|
|
|
|
# 2. Autoconf with make distcheck: the src and build trees are distinct
|
|
|
|
# 3. Cmake on a *nix platform using e.g. gcc:
|
|
|
|
# the src and build trees are distinct.
|
2017-04-10 23:26:57 +08:00
|
|
|
# 4. Cmake on windows using cygwin or msys.
|
|
|
|
# The src and build trees are distinct.
|
2017-06-08 03:21:07 +08:00
|
|
|
#
|
2017-04-10 23:26:57 +08:00
|
|
|
# For now, an explicit build using the Visual C(++) compiler
|
|
|
|
# is not supported. The big issue is the handling of executables
|
|
|
|
# and the notion of a VS configuration/build type like Debug or Release.
|
|
|
|
# When using VS, executables are placed in a subdirectory of the
|
|
|
|
# build directory. That subdirectory is named by the configuration type.
|
2017-04-07 04:55:11 +08:00
|
|
|
# Thus one finds ncdump.exe in $top_builddir/ncdump/Debug instead of
|
2017-04-10 23:26:57 +08:00
|
|
|
# $top_builddir/ncdump. An additional issue is the extension of an
|
|
|
|
# executable: .exe vs nothing. This code attempts to figure out which is used.
|
2017-06-08 03:21:07 +08:00
|
|
|
#
|
2017-04-10 23:26:57 +08:00
|
|
|
# For possible future fixes, a placeholder is left in place in the
|
|
|
|
# following code named VS. If it were set to the build type, then,
|
|
|
|
# in theory, this code would work with Visual C. It is disabled for now.
|
2017-06-08 03:21:07 +08:00
|
|
|
#
|
2017-03-09 08:01:10 +08:00
|
|
|
# The goal, then, of this common code is to set up some useful
|
|
|
|
#constants for use in test shell scripts.
|
|
|
|
# 1. srcdir - absolute path to the source dir (e.g. ${top_srcdir}/ncgen)
|
|
|
|
# 2. top_srcdir - absolute path to the root of the source
|
|
|
|
# 3. top_builddir - absolute path to the root of the build directory;
|
|
|
|
# may be same as top_srcdir (e.g. #1).
|
|
|
|
# 4. builddir - absolute path of th the directory into which generated
|
|
|
|
# stuff (.nc, .cdl, etc) is stored.
|
|
|
|
# 5. execdir - absolute path of the directory into which executables are
|
|
|
|
# placed. For all but the VS case, execdir == builddir.
|
2017-04-07 04:55:11 +08:00
|
|
|
#
|
2017-03-09 08:01:10 +08:00
|
|
|
# The following are defined to support inter-directory references.
|
|
|
|
# 6. NCDUMP - absolute path to the ncdump.exe executable
|
|
|
|
# 7. NCCOPY - absolute path to the nccopy.exe executable
|
|
|
|
# 8. NCGEN - absolute path to ncgen.exe
|
|
|
|
# 9. NCGEN3 - absolute path to ncgen3.exe
|
2021-09-03 07:04:26 +08:00
|
|
|
#10. NCPATHCVT - absolute path to ncpathcvt.exe
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2017-04-10 23:26:57 +08:00
|
|
|
# Allow global set -x mechanism for debugging.
|
2017-03-09 08:01:10 +08:00
|
|
|
if test "x$SETX" = x1 ; then set -x ; fi
|
|
|
|
|
2021-09-05 17:30:48 +08:00
|
|
|
# On MINGW, bash and other POSIX utilities use a mounted root directory,
|
|
|
|
# but executables compiled for Windows do not recognise the mount point.
|
|
|
|
# Here we ensure that Windows paths are used in tests of Windows executables.
|
2022-04-12 00:39:10 +08:00
|
|
|
|
2021-09-05 17:30:48 +08:00
|
|
|
system=`uname`
|
|
|
|
if test "x${system##MINGW*}" = x; then
|
|
|
|
alias pwd='pwd -W'
|
|
|
|
fi
|
|
|
|
|
2017-04-07 04:55:11 +08:00
|
|
|
# We assume that TOPSRCDIR and TOPBUILDDIR are defined
|
2017-03-09 08:01:10 +08:00
|
|
|
# At the top of this shell script
|
|
|
|
top_srcdir="$TOPSRCDIR"
|
|
|
|
top_builddir="$TOPBUILDDIR"
|
|
|
|
|
2017-04-10 23:26:57 +08:00
|
|
|
# Currently not used, but left as a Visual Studio placeholder.
|
|
|
|
# VS=Debug
|
|
|
|
|
2023-10-09 01:22:52 +08:00
|
|
|
# Set when using gdb
|
|
|
|
#DL=".libs/"
|
|
|
|
|
2017-03-09 08:01:10 +08:00
|
|
|
# srcdir may or may not be defined, but if not, then create it
|
|
|
|
if test "x$srcdir" = x ; then
|
|
|
|
# we need to figure out our directory
|
|
|
|
# pick off the last component as the relative name of this directory
|
2021-09-05 17:37:02 +08:00
|
|
|
srcdir=`pwd`
|
Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.
The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.
## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
- "size" with an integer value representing the (current) size of the dimension.
- "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.
For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.
Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.
## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
AWS Transfer Utility mechanism. This is controlled by the
```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-27 06:56:48 +08:00
|
|
|
if test "x$srcdir" != "x$top_srcdir" ; then
|
|
|
|
current=`basename $srcdir`
|
|
|
|
srcdir="${top_srcdir}/$current"
|
|
|
|
fi
|
2017-03-09 08:01:10 +08:00
|
|
|
fi
|
|
|
|
|
|
|
|
# We also assume we are executing in builddir
|
2021-09-05 17:37:02 +08:00
|
|
|
builddir=`pwd`
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2024-05-16 08:46:25 +08:00
|
|
|
# execdir is usually an alias for builddir
|
2018-01-17 02:00:09 +08:00
|
|
|
execdir="${builddir}"
|
2017-03-09 08:01:10 +08:00
|
|
|
|
|
|
|
# pick off the last component as the relative name of this directory
|
|
|
|
thisdir=`basename $srcdir`
|
|
|
|
|
2021-09-05 17:37:02 +08:00
|
|
|
WD=`pwd`
|
2017-03-09 08:01:10 +08:00
|
|
|
# Absolutize paths of interest
|
2021-09-05 17:37:02 +08:00
|
|
|
cd $srcdir; srcdir=`pwd` ; cd $WD
|
|
|
|
cd $top_srcdir; top_srcdir=`pwd` ; cd $WD
|
|
|
|
cd $builddir; builddir=`pwd` ; cd $WD
|
|
|
|
cd $top_builddir; top_builddir=`pwd` ; cd $WD
|
|
|
|
cd $execdir; execdir=`pwd` ; cd $WD
|
2017-03-09 08:01:10 +08:00
|
|
|
|
|
|
|
# For sun os
|
|
|
|
export srcdir top_srcdir builddir top_builddir execdir
|
|
|
|
|
2017-04-10 23:26:57 +08:00
|
|
|
# Figure out executable extension (probably a better way)
|
2017-03-09 08:01:10 +08:00
|
|
|
if test -e "${top_builddir}/ncdump${VS}/ncdump.exe" ; then
|
|
|
|
ext=".exe"
|
|
|
|
else
|
|
|
|
ext=""
|
|
|
|
fi
|
|
|
|
|
2017-06-29 03:51:01 +08:00
|
|
|
# We need to locate certain executables (and other things),
|
|
|
|
# capture absolute paths, and make visible
|
2024-05-16 08:46:25 +08:00
|
|
|
export NCDUMP="${abs_top_builddir}/ncdump${VS}/${DL}ncdump${ext}"
|
|
|
|
export NCCOPY="${abs_top_builddir}/ncdump${VS}/${DL}nccopy${ext}"
|
|
|
|
export NCGEN="${abs_top_builddir}/ncgen${VS}/${DL}ncgen${ext}"
|
|
|
|
export NCGEN3="${abs_top_builddir}/ncgen3${VS}/${DL}ncgen3${ext}"
|
|
|
|
export NCPATHCVT="${abs_top_builddir}/ncdump${VS}/${DL}ncpathcvt${ext}"
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2017-04-10 23:26:57 +08:00
|
|
|
# Temporary hacks (until we have a test_utils directory)
|
|
|
|
# to locate certain specific test files
|
2024-05-16 08:46:25 +08:00
|
|
|
ncgen3c0="${abs_top_srcdir}/ncgen3/c0.cdl"
|
|
|
|
ncgenc0="${abs_top_srcdir}/ncgen/c0.cdl"
|
|
|
|
ncgenc04="${abs_top_srcdir}/ncgen/c0_4.cdl"
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2022-05-15 06:05:48 +08:00
|
|
|
# Set LC_ALL
|
|
|
|
if test "x$FP_ISMSVC" = xyes || test "x$FP_ISCYGWIN" = xyes; then export LC_ALL="en_US.utf8"; fi
|
|
|
|
|
2022-06-15 04:44:23 +08:00
|
|
|
# Set HOME
|
|
|
|
if test "x$FP_ISMSVC" = xyes || test "x$FP_MINGW" = xyes; then
|
|
|
|
if test "x$HOME" = x ; then
|
|
|
|
HOME=`echo $USERPROFILE |tr '\\\' '/'`
|
|
|
|
export HOME
|
|
|
|
fi
|
|
|
|
fi
|
|
|
|
|
2024-10-01 04:39:48 +08:00
|
|
|
# OS/X and windows apparently have no echo -n option, so fake it
|
|
|
|
echon() {
|
|
|
|
${execdir}/../ncdump/echon -n $@
|
|
|
|
}
|
2023-04-05 08:37:20 +08:00
|
|
|
|
Enhance/Fix filter support
re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214
The primary change is to support so-called "standard filters".
A standard filter is one that is defined by the following
netcdf-c API:
````
int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params);
int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params);
````
So for example, zstandard would be a standard filter by defining
the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*.
In order to define these functions, we need a new dispatch function:
````
int nc_inq_filter_avail(int ncid, unsigned filterid);
````
This function, combined with the existing filter API can be used
to implement arbitrary standard filters using a simple code pattern.
Note that I would have preferred that this function return a list
of all available filters, but HDF5 does not support that functionality.
So this PR implements the dispatch function and implements
the following standard functions:
+ bzip2
+ zstandard
+ blosc
Specific test cases are also provided for HDF5 and NCZarr.
Over time, other specific standard filters will be defined.
## Primary Changes
* Add nc_inq_filter_avail() to netcdf-c API.
* Add standard filter implementations to test use of *nc_inq_filter_avail*.
* Bump the dispatch table version number and add to all the relevant
dispatch tables (libsrc, libsrcp, etc).
* Create a program to invoke nc_inq_filter_avail so that it is accessible
to shell scripts.
* Cleanup szip support to properly support szip
when HDF5 is disabled. This involves detecting
libsz separately from testing if HDF5 supports szip.
* Integrate shuffle and fletcher32 into the existing
filter API. This means that, for example, nc_def_var_fletcher32
is now a wrapper around nc_def_var_filter.
* Extend the Codec defaulting to allow multiple default shared libraries.
## Misc. Changes
* Modify configure.ac/CMakeLists.txt to look for the relevant
libraries implementing standard filters.
* Modify libnetcdf.settings to list available standard filters
(including deflate and szip).
* Add CMake test modules to locate libbz2 and libzstd.
* Cleanup the HDF5 memory manager function use in the plugins.
* remove unused file include//ncfilter.h
* remove tests for the HDF5 memory operations e.g. H5allocate_memory.
* Add flag to ncdump to force use of _Filter instead of _Deflate
or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
|
|
|
# Test for filter availability
|
|
|
|
avail() {
|
|
|
|
if test yes = `${execdir}/../ncdump/ncfilteravail $1` ; then return 0 ; else echo "filter $1 not available" ; return 1; fi
|
|
|
|
}
|
|
|
|
|
2017-03-09 08:01:10 +08:00
|
|
|
# Make sure we are in builddir (not execdir)
|
|
|
|
cd $builddir
|
2022-08-28 10:21:13 +08:00
|
|
|
|
Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.
The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.
## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
- "size" with an integer value representing the (current) size of the dimension.
- "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.
For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.
Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.
## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
AWS Transfer Utility mechanism. This is controlled by the
```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-27 06:56:48 +08:00
|
|
|
# Parallel make can cause inter-test interference (mostly because of historical naming issues).
|
|
|
|
# As a protection against this, the isolate() function supports the creation of an
|
|
|
|
# isolation directory in which created products are stored.
|
|
|
|
# Cleanup can be accomplished by deleting the whole isolation directory.
|
|
|
|
# The name of the isolation directory is by convention "testdir_<random>".
|
|
|
|
# The isolation dir is created within the ${builddir} directory.
|
|
|
|
# The <random> is a generated 32 bit unsigned random integer to make the chance
|
|
|
|
# of collision very small.
|
|
|
|
# Process ID was not used because of the small but real chance of collision.
|
2023-04-26 07:15:06 +08:00
|
|
|
|
|
|
|
isolate() {
|
Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.
The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.
## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
- "size" with an integer value representing the (current) size of the dimension.
- "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.
For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.
Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.
## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
AWS Transfer Utility mechanism. This is controlled by the
```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-27 06:56:48 +08:00
|
|
|
if test "x$ISOPATH" = x ; then
|
|
|
|
ISOTESTSET="${builddir}/testset_"
|
|
|
|
if test "x$NOISOPATH" = x ; then ISOTESTSET="${ISOTESTSET}${TESTUID}"; fi
|
|
|
|
ISODIR="$1"
|
|
|
|
ISOPATH="${ISOTESTSET}/$ISODIR"
|
|
|
|
rm -fr $ISOPATH
|
|
|
|
mkdir -p $ISOPATH
|
2023-04-26 07:15:06 +08:00
|
|
|
fi
|
|
|
|
}
|
|
|
|
|
2022-08-28 10:21:13 +08:00
|
|
|
fi #TEST_COMMON_SH
|