2018-12-07 06:51:35 +08:00
|
|
|
# Copyright 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002,
|
|
|
|
# 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014,
|
|
|
|
# 2015, 2016, 2017, 2018
|
|
|
|
# University Corporation for Atmospheric Research/Unidata.
|
|
|
|
|
|
|
|
# See netcdf-c/COPYRIGHT file for more info.
|
|
|
|
|
2022-08-28 10:21:13 +08:00
|
|
|
# Load only once
|
|
|
|
if test "x$TEST_COMMON_SH" = x ; then
|
|
|
|
export TEST_COMMON_SH=1
|
|
|
|
|
Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.
The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.
## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
- "size" with an integer value representing the (current) size of the dimension.
- "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.
For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.
Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.
## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
AWS Transfer Utility mechanism. This is controlled by the
```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-27 06:56:48 +08:00
|
|
|
# Define various global constants
|
|
|
|
|
2020-08-18 09:15:47 +08:00
|
|
|
# Define location of execution
|
2018-03-01 04:40:50 +08:00
|
|
|
TOPSRCDIR='@abs_top_srcdir@'
|
2018-03-01 03:54:53 +08:00
|
|
|
TOPBUILDDIR='@abs_top_builddir@'
|
Codify cross-platform file paths
The netcdf-c code has to deal with a variety of platforms:
Windows, OSX, Linux, Cygwin, MSYS, etc. These platforms differ
significantly in the kind of file paths that they accept. So in
order to handle this, I have created a set of replacements for
the most common file system operations such as _open_ or _fopen_
or _access_ to manage the file path differences correctly.
A more limited version of this idea was already implemented via
the ncwinpath.h and dwinpath.c code. So this can be viewed as a
replacement for that code. And in path in many cases, the only
change that was required was to replace '#include <ncwinpath.h>'
with '#include <ncpathmgt.h>' and then replace file operation
calls with the NCxxx equivalent from ncpathmgr.h Note that
recently, the ncwinpath.h was renamed ncpathmgmt.h, so this pull
request should not require dealing with winpath.
The heart of the change is include/ncpathmgmt.h, which provides
alternate operations such as NCfopen or NCaccess and which properly
parse and rebuild path arguments to work for the platform on which
the code is executing. This mostly matters for Windows because of the
way that it uses backslash and drive letters, as compared to *nix*.
One important feature is that the user can do string manipulations
on a file path without having to worry too much about the platform
because the path management code will properly handle most mixed cases.
So one can for example concatenate a path suffix that uses forward
slashes to a Windows path and have it work correctly.
The conversion code is in libdispatch/dpathmgr.c, and the
important function there is NCpathcvt which does the proper
conversions to the local path format.
As a rule, most code should just replace their file operations with
the corresponding NCxxx ones defined in include/ncpathmgmt.h. These
NCxxx functions all call NCpathcvt on their path arguments before
executing the actual file operation.
In some rare cases, the client may need to directly use NCpathcvt,
but this should be avoided as much as possible. If there is a need
for supporting a new file operation not already in ncpathmgmt.h, then
use the code in dpathmgr.c as a template. Also please notify Unidata
so we can include it as a formal part or our supported operations.
Also, if you see an operation in the library that is not using the
NCxxx form, then please submit an issue so we can fix it.
Misc. Changes:
* Clean up the utf8 testing code; it is impossible to get some
tests to work under windows using shell scripts; the args do
not pass as utf8 but as some other encoding.
* Added an extra utf8 test case: test_unicode_path.sh
* Add a true test for HDF5 1.10.6 or later because as noted in
PR https://github.com/Unidata/netcdf-c/pull/1794,
HDF5 changed its Windows file path handling.
2021-03-05 04:41:31 +08:00
|
|
|
FP_ISCMAKE=@ISCMAKE@
|
|
|
|
FP_ISMSVC=@ISMSVC@
|
2022-02-09 11:53:30 +08:00
|
|
|
FP_WINVERMAJOR=@WINVERMAJOR@
|
|
|
|
FP_WINVERBUILD=@WINVERBUILD@
|
2021-05-20 07:19:33 +08:00
|
|
|
FP_ISCYGWIN=@ISCYGWIN@
|
2021-11-04 02:49:54 +08:00
|
|
|
FP_ISMINGW=@ISMINGW@
|
2021-12-24 13:18:56 +08:00
|
|
|
FP_ISMSYS=@ISMSYS@
|
2022-03-19 11:01:40 +08:00
|
|
|
FP_ISOSX=@ISOSX@
|
2022-04-08 06:57:35 +08:00
|
|
|
|
2022-02-09 11:53:30 +08:00
|
|
|
FP_ISREGEDIT=@ISREGEDIT@
|
2022-05-15 06:05:48 +08:00
|
|
|
FP_USEPLUGINS=@USEPLUGINS@
|
Codify cross-platform file paths
The netcdf-c code has to deal with a variety of platforms:
Windows, OSX, Linux, Cygwin, MSYS, etc. These platforms differ
significantly in the kind of file paths that they accept. So in
order to handle this, I have created a set of replacements for
the most common file system operations such as _open_ or _fopen_
or _access_ to manage the file path differences correctly.
A more limited version of this idea was already implemented via
the ncwinpath.h and dwinpath.c code. So this can be viewed as a
replacement for that code. And in path in many cases, the only
change that was required was to replace '#include <ncwinpath.h>'
with '#include <ncpathmgt.h>' and then replace file operation
calls with the NCxxx equivalent from ncpathmgr.h Note that
recently, the ncwinpath.h was renamed ncpathmgmt.h, so this pull
request should not require dealing with winpath.
The heart of the change is include/ncpathmgmt.h, which provides
alternate operations such as NCfopen or NCaccess and which properly
parse and rebuild path arguments to work for the platform on which
the code is executing. This mostly matters for Windows because of the
way that it uses backslash and drive letters, as compared to *nix*.
One important feature is that the user can do string manipulations
on a file path without having to worry too much about the platform
because the path management code will properly handle most mixed cases.
So one can for example concatenate a path suffix that uses forward
slashes to a Windows path and have it work correctly.
The conversion code is in libdispatch/dpathmgr.c, and the
important function there is NCpathcvt which does the proper
conversions to the local path format.
As a rule, most code should just replace their file operations with
the corresponding NCxxx ones defined in include/ncpathmgmt.h. These
NCxxx functions all call NCpathcvt on their path arguments before
executing the actual file operation.
In some rare cases, the client may need to directly use NCpathcvt,
but this should be avoided as much as possible. If there is a need
for supporting a new file operation not already in ncpathmgmt.h, then
use the code in dpathmgr.c as a template. Also please notify Unidata
so we can include it as a formal part or our supported operations.
Also, if you see an operation in the library that is not using the
NCxxx form, then please submit an issue so we can fix it.
Misc. Changes:
* Clean up the utf8 testing code; it is impossible to get some
tests to work under windows using shell scripts; the args do
not pass as utf8 but as some other encoding.
* Added an extra utf8 test case: test_unicode_path.sh
* Add a true test for HDF5 1.10.6 or later because as noted in
PR https://github.com/Unidata/netcdf-c/pull/1794,
HDF5 changed its Windows file path handling.
2021-03-05 04:41:31 +08:00
|
|
|
|
|
|
|
# Feature flags
|
|
|
|
FEATURE_HDF5=@HAS_HDF5@
|
2022-04-29 05:51:20 +08:00
|
|
|
FEATURE_PARALLEL=@HAS_PARALLEL@
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2020-08-18 09:15:47 +08:00
|
|
|
# Define selected features of the build
|
|
|
|
FEATURE_HDF5=@HAS_HDF5@
|
2023-03-03 10:51:02 +08:00
|
|
|
FEATURE_FILTERTESTS=@DO_FILTER_TESTS@
|
|
|
|
FEATURE_PLUGIN_INSTALL_DIR=@PLUGIN_INSTALL_DIR@
|
|
|
|
FEATURE_BYTERANGE=@HAS_BYTERANGE@
|
2023-04-26 07:15:06 +08:00
|
|
|
FEATURE_ROS3=@HAS_HDF5_ROS3@
|
|
|
|
FEATURE_S3_AWS=@HAS_S3_AWS@
|
|
|
|
FEATURE_S3_INTERNAL=@HAS_S3_INTERNAL@
|
2023-03-03 10:51:02 +08:00
|
|
|
FEATURE_S3=@HAS_S3@
|
Support installation of filters into user-specified location
re: https://github.com/Unidata/netcdf-c/issues/2294
Ed Hartnett suggested that the netcdf library installation process
be extended to install the standard filters into a user specified
location. The user can then set HDF5_PLUGIN_PATH to that location.
This PR provides that capability using:
````
configure option: --with-plugin-dir=<absolute directory path>
cmake option: -DPLUGIN_INSTALL_DIR=<absolute directory path>
````
Currently, the following plugins are always installed, if
available: bzip2, zstd, blosc.
If NCZarr is enabled, then additional plugins are installed:
fletcher32, shuffle, deflate, szip.
Additionally, the necessary codec support is installed
for each of the above filters that is installed.
## Changes:
1. Cleanup handling of built-in bzip2.
2. Add documentation to docs/filters.md
3. Re-factor the NCZarr codec libraries
4. Add a test, although it can only be exercised after
the library is installed, so it cannot be used during
normal testing.
5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
|
|
|
FEATURE_NCZARR=@HAS_NCZARR@
|
Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.
The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.
## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
- "size" with an integer value representing the (current) size of the dimension.
- "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.
For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.
Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.
## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
AWS Transfer Utility mechanism. This is controlled by the
```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-27 06:56:48 +08:00
|
|
|
FEATURE_S3TESTS=@ENABLE_S3_TESTING@
|
2021-01-29 11:11:01 +08:00
|
|
|
FEATURE_NCZARR_ZIP=@DO_NCZARR_ZIP_TESTS@
|
2023-04-26 07:15:06 +08:00
|
|
|
FEATURE_LARGE_TESTS=@DO_LARGE_TESTS@
|
2020-08-18 09:15:47 +08:00
|
|
|
|
2022-08-28 10:21:13 +08:00
|
|
|
# Thredds-test server is currently disabled
|
|
|
|
#FEATURE_THREDDSTEST=1
|
|
|
|
|
Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.
The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.
## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
- "size" with an integer value representing the (current) size of the dimension.
- "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.
For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.
Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.
## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
AWS Transfer Utility mechanism. This is controlled by the
```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-27 06:56:48 +08:00
|
|
|
# This is the Unidata S3 test bucket
|
|
|
|
# All S3 tests should use this to store intermediate results.
|
|
|
|
S3TESTBUCKET=@S3TESTBUCKET@
|
|
|
|
|
|
|
|
# This is the s3 path within the Unidata bucket;
|
|
|
|
# All S3 tests should use this to store intermediate results.
|
|
|
|
S3TESTSUBTREE=@S3TESTSUBTREE@
|
|
|
|
|
|
|
|
TESTUID=@TESTUID@
|
|
|
|
|
2018-01-17 02:00:09 +08:00
|
|
|
set -e
|
|
|
|
|
2017-03-09 08:01:10 +08:00
|
|
|
# Figure out various locations in the src/build tree.
|
|
|
|
# This is relatively fragile code and is essentially
|
|
|
|
# specific to netcdf-c. It does, however, have the virtue
|
|
|
|
# of isolating all this nonsense into one place.
|
|
|
|
# This will get somewhat simplified (I hope) when
|
|
|
|
# we move to a separate test_utilities directory
|
|
|
|
|
|
|
|
# This code is intended to provide constants
|
|
|
|
# for accessing various objects in the src/build
|
|
|
|
# tree(s) across multiple ways of building netcdf-c.
|
|
|
|
# Currently, the following build situations are supported.
|
|
|
|
# 1. Autoconf with make check: the src and build trees are the same
|
|
|
|
# 2. Autoconf with make distcheck: the src and build trees are distinct
|
|
|
|
# 3. Cmake on a *nix platform using e.g. gcc:
|
|
|
|
# the src and build trees are distinct.
|
2017-04-10 23:26:57 +08:00
|
|
|
# 4. Cmake on windows using cygwin or msys.
|
|
|
|
# The src and build trees are distinct.
|
2017-06-08 03:21:07 +08:00
|
|
|
#
|
2017-04-10 23:26:57 +08:00
|
|
|
# For now, an explicit build using the Visual C(++) compiler
|
|
|
|
# is not supported. The big issue is the handling of executables
|
|
|
|
# and the notion of a VS configuration/build type like Debug or Release.
|
|
|
|
# When using VS, executables are placed in a subdirectory of the
|
|
|
|
# build directory. That subdirectory is named by the configuration type.
|
2017-04-07 04:55:11 +08:00
|
|
|
# Thus one finds ncdump.exe in $top_builddir/ncdump/Debug instead of
|
2017-04-10 23:26:57 +08:00
|
|
|
# $top_builddir/ncdump. An additional issue is the extension of an
|
|
|
|
# executable: .exe vs nothing. This code attempts to figure out which is used.
|
2017-06-08 03:21:07 +08:00
|
|
|
#
|
2017-04-10 23:26:57 +08:00
|
|
|
# For possible future fixes, a placeholder is left in place in the
|
|
|
|
# following code named VS. If it were set to the build type, then,
|
|
|
|
# in theory, this code would work with Visual C. It is disabled for now.
|
2017-06-08 03:21:07 +08:00
|
|
|
#
|
2017-03-09 08:01:10 +08:00
|
|
|
# The goal, then, of this common code is to set up some useful
|
|
|
|
#constants for use in test shell scripts.
|
|
|
|
# 1. srcdir - absolute path to the source dir (e.g. ${top_srcdir}/ncgen)
|
|
|
|
# 2. top_srcdir - absolute path to the root of the source
|
|
|
|
# 3. top_builddir - absolute path to the root of the build directory;
|
|
|
|
# may be same as top_srcdir (e.g. #1).
|
|
|
|
# 4. builddir - absolute path of th the directory into which generated
|
|
|
|
# stuff (.nc, .cdl, etc) is stored.
|
|
|
|
# 5. execdir - absolute path of the directory into which executables are
|
|
|
|
# placed. For all but the VS case, execdir == builddir.
|
2017-04-07 04:55:11 +08:00
|
|
|
#
|
2017-03-09 08:01:10 +08:00
|
|
|
# The following are defined to support inter-directory references.
|
|
|
|
# 6. NCDUMP - absolute path to the ncdump.exe executable
|
|
|
|
# 7. NCCOPY - absolute path to the nccopy.exe executable
|
|
|
|
# 8. NCGEN - absolute path to ncgen.exe
|
|
|
|
# 9. NCGEN3 - absolute path to ncgen3.exe
|
2021-09-03 07:04:26 +08:00
|
|
|
#10. NCPATHCVT - absolute path to ncpathcvt.exe
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2017-04-10 23:26:57 +08:00
|
|
|
# Allow global set -x mechanism for debugging.
|
2017-03-09 08:01:10 +08:00
|
|
|
if test "x$SETX" = x1 ; then set -x ; fi
|
|
|
|
|
2021-09-05 17:30:48 +08:00
|
|
|
# On MINGW, bash and other POSIX utilities use a mounted root directory,
|
|
|
|
# but executables compiled for Windows do not recognise the mount point.
|
|
|
|
# Here we ensure that Windows paths are used in tests of Windows executables.
|
2022-04-12 00:39:10 +08:00
|
|
|
|
2021-09-05 17:30:48 +08:00
|
|
|
system=`uname`
|
|
|
|
if test "x${system##MINGW*}" = x; then
|
|
|
|
alias pwd='pwd -W'
|
2022-04-12 00:39:10 +08:00
|
|
|
|
2021-09-05 17:30:48 +08:00
|
|
|
fi
|
|
|
|
|
2017-04-07 04:55:11 +08:00
|
|
|
# We assume that TOPSRCDIR and TOPBUILDDIR are defined
|
2017-03-09 08:01:10 +08:00
|
|
|
# At the top of this shell script
|
|
|
|
top_srcdir="$TOPSRCDIR"
|
|
|
|
top_builddir="$TOPBUILDDIR"
|
|
|
|
|
2017-04-10 23:26:57 +08:00
|
|
|
# Currently not used, but left as a Visual Studio placeholder.
|
|
|
|
# VS=Debug
|
|
|
|
|
2023-10-09 01:22:52 +08:00
|
|
|
# Set when using gdb
|
|
|
|
#DL=".libs/"
|
|
|
|
|
2017-03-09 08:01:10 +08:00
|
|
|
# srcdir may or may not be defined, but if not, then create it
|
|
|
|
if test "x$srcdir" = x ; then
|
|
|
|
# we need to figure out our directory
|
|
|
|
# pick off the last component as the relative name of this directory
|
2021-09-05 17:37:02 +08:00
|
|
|
srcdir=`pwd`
|
Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.
The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.
## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
- "size" with an integer value representing the (current) size of the dimension.
- "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.
For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.
Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.
## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
AWS Transfer Utility mechanism. This is controlled by the
```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-27 06:56:48 +08:00
|
|
|
if test "x$srcdir" != "x$top_srcdir" ; then
|
|
|
|
current=`basename $srcdir`
|
|
|
|
srcdir="${top_srcdir}/$current"
|
|
|
|
fi
|
2017-03-09 08:01:10 +08:00
|
|
|
fi
|
|
|
|
|
|
|
|
# We also assume we are executing in builddir
|
2021-09-05 17:37:02 +08:00
|
|
|
builddir=`pwd`
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2018-01-17 02:00:09 +08:00
|
|
|
# execdir is an alias for builddir
|
|
|
|
execdir="${builddir}"
|
2017-03-09 08:01:10 +08:00
|
|
|
|
|
|
|
# pick off the last component as the relative name of this directory
|
|
|
|
thisdir=`basename $srcdir`
|
|
|
|
|
2021-09-05 17:37:02 +08:00
|
|
|
WD=`pwd`
|
2017-03-09 08:01:10 +08:00
|
|
|
# Absolutize paths of interest
|
2021-09-05 17:37:02 +08:00
|
|
|
cd $srcdir; srcdir=`pwd` ; cd $WD
|
|
|
|
cd $top_srcdir; top_srcdir=`pwd` ; cd $WD
|
|
|
|
cd $builddir; builddir=`pwd` ; cd $WD
|
|
|
|
cd $top_builddir; top_builddir=`pwd` ; cd $WD
|
|
|
|
cd $execdir; execdir=`pwd` ; cd $WD
|
2017-03-09 08:01:10 +08:00
|
|
|
|
|
|
|
# For sun os
|
|
|
|
export srcdir top_srcdir builddir top_builddir execdir
|
|
|
|
|
2017-04-10 23:26:57 +08:00
|
|
|
# Figure out executable extension (probably a better way)
|
2017-03-09 08:01:10 +08:00
|
|
|
if test -e "${top_builddir}/ncdump${VS}/ncdump.exe" ; then
|
|
|
|
ext=".exe"
|
|
|
|
else
|
|
|
|
ext=""
|
|
|
|
fi
|
|
|
|
|
2017-06-29 03:51:01 +08:00
|
|
|
# We need to locate certain executables (and other things),
|
|
|
|
# capture absolute paths, and make visible
|
2023-10-09 01:22:52 +08:00
|
|
|
export NCDUMP="${top_builddir}/ncdump${VS}/${DL}ncdump${ext}"
|
|
|
|
export NCCOPY="${top_builddir}/ncdump${VS}/${DL}nccopy${ext}"
|
|
|
|
export NCGEN="${top_builddir}/ncgen${VS}/${DL}ncgen${ext}"
|
|
|
|
export NCGEN3="${top_builddir}/ncgen3${VS}/${DL}ncgen3${ext}"
|
|
|
|
export NCPATHCVT="${top_builddir}/ncdump${VS}/${DL}ncpathcvt${ext}"
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2017-04-10 23:26:57 +08:00
|
|
|
# Temporary hacks (until we have a test_utils directory)
|
|
|
|
# to locate certain specific test files
|
2017-03-09 08:01:10 +08:00
|
|
|
ncgen3c0="${top_srcdir}/ncgen3/c0.cdl"
|
|
|
|
ncgenc0="${top_srcdir}/ncgen/c0.cdl"
|
|
|
|
ncgenc04="${top_srcdir}/ncgen/c0_4.cdl"
|
|
|
|
|
2022-05-15 06:05:48 +08:00
|
|
|
# Set LC_ALL
|
|
|
|
if test "x$FP_ISMSVC" = xyes || test "x$FP_ISCYGWIN" = xyes; then export LC_ALL="en_US.utf8"; fi
|
|
|
|
|
2022-06-15 04:44:23 +08:00
|
|
|
# Set HOME
|
|
|
|
if test "x$FP_ISMSVC" = xyes || test "x$FP_MINGW" = xyes; then
|
|
|
|
if test "x$HOME" = x ; then
|
|
|
|
HOME=`echo $USERPROFILE |tr '\\\' '/'`
|
|
|
|
export HOME
|
|
|
|
fi
|
|
|
|
fi
|
|
|
|
|
2023-04-05 08:37:20 +08:00
|
|
|
# OS/X apparently has no echo -n option, so fake it
|
|
|
|
echon() { echo "$@" | tr -d '\r\n' ; }
|
|
|
|
|
Enhance/Fix filter support
re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214
The primary change is to support so-called "standard filters".
A standard filter is one that is defined by the following
netcdf-c API:
````
int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params);
int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params);
````
So for example, zstandard would be a standard filter by defining
the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*.
In order to define these functions, we need a new dispatch function:
````
int nc_inq_filter_avail(int ncid, unsigned filterid);
````
This function, combined with the existing filter API can be used
to implement arbitrary standard filters using a simple code pattern.
Note that I would have preferred that this function return a list
of all available filters, but HDF5 does not support that functionality.
So this PR implements the dispatch function and implements
the following standard functions:
+ bzip2
+ zstandard
+ blosc
Specific test cases are also provided for HDF5 and NCZarr.
Over time, other specific standard filters will be defined.
## Primary Changes
* Add nc_inq_filter_avail() to netcdf-c API.
* Add standard filter implementations to test use of *nc_inq_filter_avail*.
* Bump the dispatch table version number and add to all the relevant
dispatch tables (libsrc, libsrcp, etc).
* Create a program to invoke nc_inq_filter_avail so that it is accessible
to shell scripts.
* Cleanup szip support to properly support szip
when HDF5 is disabled. This involves detecting
libsz separately from testing if HDF5 supports szip.
* Integrate shuffle and fletcher32 into the existing
filter API. This means that, for example, nc_def_var_fletcher32
is now a wrapper around nc_def_var_filter.
* Extend the Codec defaulting to allow multiple default shared libraries.
## Misc. Changes
* Modify configure.ac/CMakeLists.txt to look for the relevant
libraries implementing standard filters.
* Modify libnetcdf.settings to list available standard filters
(including deflate and szip).
* Add CMake test modules to locate libbz2 and libzstd.
* Cleanup the HDF5 memory manager function use in the plugins.
* remove unused file include//ncfilter.h
* remove tests for the HDF5 memory operations e.g. H5allocate_memory.
* Add flag to ncdump to force use of _Filter instead of _Deflate
or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
|
|
|
# Test for filter availability
|
|
|
|
avail() {
|
|
|
|
if test yes = `${execdir}/../ncdump/ncfilteravail $1` ; then return 0 ; else echo "filter $1 not available" ; return 1; fi
|
|
|
|
}
|
|
|
|
|
2022-08-28 10:21:13 +08:00
|
|
|
|
2017-03-09 08:01:10 +08:00
|
|
|
# Make sure we are in builddir (not execdir)
|
|
|
|
cd $builddir
|
2022-08-28 10:21:13 +08:00
|
|
|
|
Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.
The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.
## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
- "size" with an integer value representing the (current) size of the dimension.
- "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.
For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.
Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.
## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
AWS Transfer Utility mechanism. This is controlled by the
```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-27 06:56:48 +08:00
|
|
|
|
|
|
|
# Parallel make can cause inter-test interference (mostly because of historical naming issues).
|
|
|
|
# As a protection against this, the isolate() function supports the creation of an
|
|
|
|
# isolation directory in which created products are stored.
|
|
|
|
# Cleanup can be accomplished by deleting the whole isolation directory.
|
|
|
|
# The name of the isolation directory is by convention "testdir_<random>".
|
|
|
|
# The isolation dir is created within the ${builddir} directory.
|
|
|
|
# The <random> is a generated 32 bit unsigned random integer to make the chance
|
|
|
|
# of collision very small.
|
|
|
|
# Process ID was not used because of the small but real chance of collision.
|
2023-04-26 07:15:06 +08:00
|
|
|
|
|
|
|
isolate() {
|
Mitigate S3 test interference + Unlimited Dimensions in NCZarr
This PR started as an attempt to add unlimited dimensions to NCZarr.
It did that, but this exposed significant problems with test interference.
So this PR is mostly about fixing -- well mitigating anyway -- test
interference.
The problem of test interference is now documented in the document docs/internal.md.
The solutions implemented here are also describe in that document.
The solution is somewhat fragile but multiple cleanup mechanisms
are provided. Note that this feature requires that the
AWS command line utility must be installed.
## Unlimited Dimensions.
The existing NCZarr extensions to Zarr are modified to support unlimited dimensions.
NCzarr extends the Zarr meta-data for the ".zgroup" object to include netcdf-4 model extensions. This information is stored in ".zgroup" as dictionary named "_nczarr_group".
Inside "_nczarr_group", there is a key named "dims" that stores information about netcdf-4 named dimensions. The value of "dims" is a dictionary whose keys are the named dimensions. The value associated with each dimension name has one of two forms
Form 1 is a special case of form 2, and is kept for backward compatibility. Whenever a new file is written, it uses format 1 if possible, otherwise format 2.
* Form 1: An integer representing the size of the dimension, which is used for simple named dimensions.
* Form 2: A dictionary with the following keys and values"
- "size" with an integer value representing the (current) size of the dimension.
- "unlimited" with a value of either "1" or "0" to indicate if this dimension is an unlimited dimension.
For Unlimited dimensions, the size is initially zero, and as variables extend the length of that dimension, the size value for the dimension increases.
That dimension size is shared by all arrays referencing that dimension, so if one array extends an unlimited dimension, it is implicitly extended for all other arrays that reference that dimension.
This is the standard semantics for unlimited dimensions.
Adding unlimited dimensions required a number of other changes to the NCZarr code-base. These included the following.
* Did a partial refactor of the slice handling code in zwalk.c to clean it up.
* Added a number of tests for unlimited dimensions derived from the same test in nc_test4.
* Added several NCZarr specific unlimited tests; more are needed.
* Add test of endianness.
## Misc. Other Changes
* Modify libdispatch/ncs3sdk_aws.cpp to optionally support use of the
AWS Transfer Utility mechanism. This is controlled by the
```#define TRANSFER```` command in that file. It defaults to being disabled.
* Parameterize both the standard Unidata S3 bucket (S3TESTBUCKET) and the netcdf-c test data prefix (S3TESTSUBTREE).
* Fixed an obscure memory leak in ncdump.
* Removed some obsolete unit testing code and test cases.
* Uncovered a bug in the netcdf-c handling of big-endian floats and doubles. Have not fixed yet. See tst_h5_endians.c.
* Renamed some nczarr_tests testcases to avoid name conflicts with nc_test4.
* Modify the semantics of zmap\#ncsmap_write to only allow total rewrite of objects.
* Modify the semantics of zodom to properly handle stride > 1.
* Add a truncate operation to the libnczarr zmap code.
2023-09-27 06:56:48 +08:00
|
|
|
if test "x$ISOPATH" = x ; then
|
|
|
|
ISOTESTSET="${builddir}/testset_"
|
|
|
|
if test "x$NOISOPATH" = x ; then ISOTESTSET="${ISOTESTSET}${TESTUID}"; fi
|
|
|
|
ISODIR="$1"
|
|
|
|
ISOPATH="${ISOTESTSET}/$ISODIR"
|
|
|
|
rm -fr $ISOPATH
|
|
|
|
mkdir -p $ISOPATH
|
2023-04-26 07:15:06 +08:00
|
|
|
fi
|
|
|
|
}
|
|
|
|
|
2022-08-28 10:21:13 +08:00
|
|
|
fi #TEST_COMMON_SH
|