2018-12-07 06:51:35 +08:00
|
|
|
# Copyright 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002,
|
|
|
|
|
# 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014,
|
|
|
|
|
# 2015, 2016, 2017, 2018
|
|
|
|
|
# University Corporation for Atmospheric Research/Unidata.
|
|
|
|
|
|
|
|
|
|
# See netcdf-c/COPYRIGHT file for more info.
|
|
|
|
|
|
2020-08-18 09:15:47 +08:00
|
|
|
# Define location of execution
|
2018-03-01 04:40:50 +08:00
|
|
|
TOPSRCDIR='@abs_top_srcdir@'
|
2018-03-01 03:54:53 +08:00
|
|
|
TOPBUILDDIR='@abs_top_builddir@'
|
Codify cross-platform file paths
The netcdf-c code has to deal with a variety of platforms:
Windows, OSX, Linux, Cygwin, MSYS, etc. These platforms differ
significantly in the kind of file paths that they accept. So in
order to handle this, I have created a set of replacements for
the most common file system operations such as _open_ or _fopen_
or _access_ to manage the file path differences correctly.
A more limited version of this idea was already implemented via
the ncwinpath.h and dwinpath.c code. So this can be viewed as a
replacement for that code. And in path in many cases, the only
change that was required was to replace '#include <ncwinpath.h>'
with '#include <ncpathmgt.h>' and then replace file operation
calls with the NCxxx equivalent from ncpathmgr.h Note that
recently, the ncwinpath.h was renamed ncpathmgmt.h, so this pull
request should not require dealing with winpath.
The heart of the change is include/ncpathmgmt.h, which provides
alternate operations such as NCfopen or NCaccess and which properly
parse and rebuild path arguments to work for the platform on which
the code is executing. This mostly matters for Windows because of the
way that it uses backslash and drive letters, as compared to *nix*.
One important feature is that the user can do string manipulations
on a file path without having to worry too much about the platform
because the path management code will properly handle most mixed cases.
So one can for example concatenate a path suffix that uses forward
slashes to a Windows path and have it work correctly.
The conversion code is in libdispatch/dpathmgr.c, and the
important function there is NCpathcvt which does the proper
conversions to the local path format.
As a rule, most code should just replace their file operations with
the corresponding NCxxx ones defined in include/ncpathmgmt.h. These
NCxxx functions all call NCpathcvt on their path arguments before
executing the actual file operation.
In some rare cases, the client may need to directly use NCpathcvt,
but this should be avoided as much as possible. If there is a need
for supporting a new file operation not already in ncpathmgmt.h, then
use the code in dpathmgr.c as a template. Also please notify Unidata
so we can include it as a formal part or our supported operations.
Also, if you see an operation in the library that is not using the
NCxxx form, then please submit an issue so we can fix it.
Misc. Changes:
* Clean up the utf8 testing code; it is impossible to get some
tests to work under windows using shell scripts; the args do
not pass as utf8 but as some other encoding.
* Added an extra utf8 test case: test_unicode_path.sh
* Add a true test for HDF5 1.10.6 or later because as noted in
PR https://github.com/Unidata/netcdf-c/pull/1794,
HDF5 changed its Windows file path handling.
2021-03-05 04:41:31 +08:00
|
|
|
FP_ISCMAKE=@ISCMAKE@
|
|
|
|
|
FP_ISMSVC=@ISMSVC@
|
2022-02-09 11:53:30 +08:00
|
|
|
FP_WINVERMAJOR=@WINVERMAJOR@
|
|
|
|
|
FP_WINVERBUILD=@WINVERBUILD@
|
2021-05-20 07:19:33 +08:00
|
|
|
FP_ISCYGWIN=@ISCYGWIN@
|
2021-11-04 02:49:54 +08:00
|
|
|
FP_ISMINGW=@ISMINGW@
|
2021-12-24 13:18:56 +08:00
|
|
|
FP_ISMSYS=@ISMSYS@
|
2022-04-08 06:57:35 +08:00
|
|
|
|
2022-03-19 11:01:40 +08:00
|
|
|
FP_ISOSX=@ISOSX@
|
2021-12-24 13:18:56 +08:00
|
|
|
FP_ISREGEDIT=@ISREGEDIT@
|
2022-04-08 06:57:35 +08:00
|
|
|
|
2021-12-24 13:18:56 +08:00
|
|
|
FP_USEPLUGINS=@USEPLUGINS@
|
2022-02-09 11:53:30 +08:00
|
|
|
FP_ISREGEDIT=@ISREGEDIT@
|
Codify cross-platform file paths
The netcdf-c code has to deal with a variety of platforms:
Windows, OSX, Linux, Cygwin, MSYS, etc. These platforms differ
significantly in the kind of file paths that they accept. So in
order to handle this, I have created a set of replacements for
the most common file system operations such as _open_ or _fopen_
or _access_ to manage the file path differences correctly.
A more limited version of this idea was already implemented via
the ncwinpath.h and dwinpath.c code. So this can be viewed as a
replacement for that code. And in path in many cases, the only
change that was required was to replace '#include <ncwinpath.h>'
with '#include <ncpathmgt.h>' and then replace file operation
calls with the NCxxx equivalent from ncpathmgr.h Note that
recently, the ncwinpath.h was renamed ncpathmgmt.h, so this pull
request should not require dealing with winpath.
The heart of the change is include/ncpathmgmt.h, which provides
alternate operations such as NCfopen or NCaccess and which properly
parse and rebuild path arguments to work for the platform on which
the code is executing. This mostly matters for Windows because of the
way that it uses backslash and drive letters, as compared to *nix*.
One important feature is that the user can do string manipulations
on a file path without having to worry too much about the platform
because the path management code will properly handle most mixed cases.
So one can for example concatenate a path suffix that uses forward
slashes to a Windows path and have it work correctly.
The conversion code is in libdispatch/dpathmgr.c, and the
important function there is NCpathcvt which does the proper
conversions to the local path format.
As a rule, most code should just replace their file operations with
the corresponding NCxxx ones defined in include/ncpathmgmt.h. These
NCxxx functions all call NCpathcvt on their path arguments before
executing the actual file operation.
In some rare cases, the client may need to directly use NCpathcvt,
but this should be avoided as much as possible. If there is a need
for supporting a new file operation not already in ncpathmgmt.h, then
use the code in dpathmgr.c as a template. Also please notify Unidata
so we can include it as a formal part or our supported operations.
Also, if you see an operation in the library that is not using the
NCxxx form, then please submit an issue so we can fix it.
Misc. Changes:
* Clean up the utf8 testing code; it is impossible to get some
tests to work under windows using shell scripts; the args do
not pass as utf8 but as some other encoding.
* Added an extra utf8 test case: test_unicode_path.sh
* Add a true test for HDF5 1.10.6 or later because as noted in
PR https://github.com/Unidata/netcdf-c/pull/1794,
HDF5 changed its Windows file path handling.
2021-03-05 04:41:31 +08:00
|
|
|
|
|
|
|
|
# Feature flags
|
|
|
|
|
FEATURE_HDF5=@HAS_HDF5@
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2020-08-18 09:15:47 +08:00
|
|
|
# Define selected features of the build
|
|
|
|
|
FEATURE_HDF5=@HAS_HDF5@
|
2020-10-17 05:04:51 +08:00
|
|
|
FEATURE_S3TESTS=@DO_NCZARR_S3_TESTS@
|
2021-01-29 11:11:01 +08:00
|
|
|
FEATURE_NCZARR_ZIP=@DO_NCZARR_ZIP_TESTS@
|
2021-09-03 07:04:26 +08:00
|
|
|
FEATURE_FILTERTESTS=@DO_FILTER_TESTS@
|
2020-08-18 09:15:47 +08:00
|
|
|
|
2018-01-17 02:00:09 +08:00
|
|
|
set -e
|
|
|
|
|
|
2017-03-09 08:01:10 +08:00
|
|
|
# Figure out various locations in the src/build tree.
|
|
|
|
|
# This is relatively fragile code and is essentially
|
|
|
|
|
# specific to netcdf-c. It does, however, have the virtue
|
|
|
|
|
# of isolating all this nonsense into one place.
|
|
|
|
|
# This will get somewhat simplified (I hope) when
|
|
|
|
|
# we move to a separate test_utilities directory
|
|
|
|
|
|
|
|
|
|
# This code is intended to provide constants
|
|
|
|
|
# for accessing various objects in the src/build
|
|
|
|
|
# tree(s) across multiple ways of building netcdf-c.
|
|
|
|
|
# Currently, the following build situations are supported.
|
|
|
|
|
# 1. Autoconf with make check: the src and build trees are the same
|
|
|
|
|
# 2. Autoconf with make distcheck: the src and build trees are distinct
|
|
|
|
|
# 3. Cmake on a *nix platform using e.g. gcc:
|
|
|
|
|
# the src and build trees are distinct.
|
2017-04-10 23:26:57 +08:00
|
|
|
# 4. Cmake on windows using cygwin or msys.
|
|
|
|
|
# The src and build trees are distinct.
|
2017-06-08 03:21:07 +08:00
|
|
|
#
|
2017-04-10 23:26:57 +08:00
|
|
|
# For now, an explicit build using the Visual C(++) compiler
|
|
|
|
|
# is not supported. The big issue is the handling of executables
|
|
|
|
|
# and the notion of a VS configuration/build type like Debug or Release.
|
|
|
|
|
# When using VS, executables are placed in a subdirectory of the
|
|
|
|
|
# build directory. That subdirectory is named by the configuration type.
|
2017-04-07 04:55:11 +08:00
|
|
|
# Thus one finds ncdump.exe in $top_builddir/ncdump/Debug instead of
|
2017-04-10 23:26:57 +08:00
|
|
|
# $top_builddir/ncdump. An additional issue is the extension of an
|
|
|
|
|
# executable: .exe vs nothing. This code attempts to figure out which is used.
|
2017-06-08 03:21:07 +08:00
|
|
|
#
|
2017-04-10 23:26:57 +08:00
|
|
|
# For possible future fixes, a placeholder is left in place in the
|
|
|
|
|
# following code named VS. If it were set to the build type, then,
|
|
|
|
|
# in theory, this code would work with Visual C. It is disabled for now.
|
2017-06-08 03:21:07 +08:00
|
|
|
#
|
2017-03-09 08:01:10 +08:00
|
|
|
# The goal, then, of this common code is to set up some useful
|
|
|
|
|
#constants for use in test shell scripts.
|
|
|
|
|
# 1. srcdir - absolute path to the source dir (e.g. ${top_srcdir}/ncgen)
|
|
|
|
|
# 2. top_srcdir - absolute path to the root of the source
|
|
|
|
|
# 3. top_builddir - absolute path to the root of the build directory;
|
|
|
|
|
# may be same as top_srcdir (e.g. #1).
|
|
|
|
|
# 4. builddir - absolute path of th the directory into which generated
|
|
|
|
|
# stuff (.nc, .cdl, etc) is stored.
|
|
|
|
|
# 5. execdir - absolute path of the directory into which executables are
|
|
|
|
|
# placed. For all but the VS case, execdir == builddir.
|
2017-04-07 04:55:11 +08:00
|
|
|
#
|
2017-03-09 08:01:10 +08:00
|
|
|
# The following are defined to support inter-directory references.
|
|
|
|
|
# 6. NCDUMP - absolute path to the ncdump.exe executable
|
|
|
|
|
# 7. NCCOPY - absolute path to the nccopy.exe executable
|
|
|
|
|
# 8. NCGEN - absolute path to ncgen.exe
|
|
|
|
|
# 9. NCGEN3 - absolute path to ncgen3.exe
|
2021-09-03 07:04:26 +08:00
|
|
|
#10. NCPATHCVT - absolute path to ncpathcvt.exe
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2017-04-10 23:26:57 +08:00
|
|
|
# Allow global set -x mechanism for debugging.
|
2017-03-09 08:01:10 +08:00
|
|
|
if test "x$SETX" = x1 ; then set -x ; fi
|
|
|
|
|
|
2021-09-05 17:30:48 +08:00
|
|
|
# On MINGW, bash and other POSIX utilities use a mounted root directory,
|
|
|
|
|
# but executables compiled for Windows do not recognise the mount point.
|
|
|
|
|
# Here we ensure that Windows paths are used in tests of Windows executables.
|
2022-04-12 00:39:10 +08:00
|
|
|
|
2021-09-05 17:30:48 +08:00
|
|
|
system=`uname`
|
|
|
|
|
if test "x${system##MINGW*}" = x; then
|
|
|
|
|
alias pwd='pwd -W'
|
2022-04-12 00:39:10 +08:00
|
|
|
|
2021-09-05 17:30:48 +08:00
|
|
|
fi
|
|
|
|
|
|
2017-04-07 04:55:11 +08:00
|
|
|
# We assume that TOPSRCDIR and TOPBUILDDIR are defined
|
2017-03-09 08:01:10 +08:00
|
|
|
# At the top of this shell script
|
|
|
|
|
top_srcdir="$TOPSRCDIR"
|
|
|
|
|
top_builddir="$TOPBUILDDIR"
|
|
|
|
|
|
2017-04-10 23:26:57 +08:00
|
|
|
# Currently not used, but left as a Visual Studio placeholder.
|
|
|
|
|
# VS=Debug
|
|
|
|
|
|
2017-03-09 08:01:10 +08:00
|
|
|
# srcdir may or may not be defined, but if not, then create it
|
|
|
|
|
if test "x$srcdir" = x ; then
|
|
|
|
|
# we need to figure out our directory
|
|
|
|
|
# pick off the last component as the relative name of this directory
|
2021-09-05 17:37:02 +08:00
|
|
|
srcdir=`pwd`
|
2017-03-09 08:01:10 +08:00
|
|
|
current=`basename $srcdir`
|
|
|
|
|
srcdir="${top_srcdir}/$current"
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
|
|
# We also assume we are executing in builddir
|
2021-09-05 17:37:02 +08:00
|
|
|
builddir=`pwd`
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2018-01-17 02:00:09 +08:00
|
|
|
# execdir is an alias for builddir
|
|
|
|
|
execdir="${builddir}"
|
2017-03-09 08:01:10 +08:00
|
|
|
|
|
|
|
|
# pick off the last component as the relative name of this directory
|
|
|
|
|
thisdir=`basename $srcdir`
|
|
|
|
|
|
2021-09-05 17:37:02 +08:00
|
|
|
WD=`pwd`
|
2017-03-09 08:01:10 +08:00
|
|
|
# Absolutize paths of interest
|
2021-09-05 17:37:02 +08:00
|
|
|
cd $srcdir; srcdir=`pwd` ; cd $WD
|
|
|
|
|
cd $top_srcdir; top_srcdir=`pwd` ; cd $WD
|
|
|
|
|
cd $builddir; builddir=`pwd` ; cd $WD
|
|
|
|
|
cd $top_builddir; top_builddir=`pwd` ; cd $WD
|
|
|
|
|
cd $execdir; execdir=`pwd` ; cd $WD
|
2017-03-09 08:01:10 +08:00
|
|
|
|
|
|
|
|
# For sun os
|
|
|
|
|
export srcdir top_srcdir builddir top_builddir execdir
|
|
|
|
|
|
2017-04-10 23:26:57 +08:00
|
|
|
# Figure out executable extension (probably a better way)
|
2017-03-09 08:01:10 +08:00
|
|
|
if test -e "${top_builddir}/ncdump${VS}/ncdump.exe" ; then
|
|
|
|
|
ext=".exe"
|
|
|
|
|
else
|
|
|
|
|
ext=""
|
|
|
|
|
fi
|
|
|
|
|
|
2017-06-29 03:51:01 +08:00
|
|
|
# We need to locate certain executables (and other things),
|
|
|
|
|
# capture absolute paths, and make visible
|
|
|
|
|
export NCDUMP="${top_builddir}/ncdump${VS}/ncdump${ext}"
|
|
|
|
|
export NCCOPY="${top_builddir}/ncdump${VS}/nccopy${ext}"
|
|
|
|
|
export NCGEN="${top_builddir}/ncgen${VS}/ncgen${ext}"
|
|
|
|
|
export NCGEN3="${top_builddir}/ncgen3${VS}/ncgen3${ext}"
|
2021-09-03 07:04:26 +08:00
|
|
|
export NCPATHCVT="${top_builddir}/ncdump${VS}/ncpathcvt${ext}"
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2017-04-10 23:26:57 +08:00
|
|
|
# Temporary hacks (until we have a test_utils directory)
|
|
|
|
|
# to locate certain specific test files
|
2017-03-09 08:01:10 +08:00
|
|
|
ncgen3c0="${top_srcdir}/ncgen3/c0.cdl"
|
|
|
|
|
ncgenc0="${top_srcdir}/ncgen/c0.cdl"
|
|
|
|
|
ncgenc04="${top_srcdir}/ncgen/c0_4.cdl"
|
|
|
|
|
|
Enhance/Fix filter support
re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214
The primary change is to support so-called "standard filters".
A standard filter is one that is defined by the following
netcdf-c API:
````
int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params);
int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params);
````
So for example, zstandard would be a standard filter by defining
the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*.
In order to define these functions, we need a new dispatch function:
````
int nc_inq_filter_avail(int ncid, unsigned filterid);
````
This function, combined with the existing filter API can be used
to implement arbitrary standard filters using a simple code pattern.
Note that I would have preferred that this function return a list
of all available filters, but HDF5 does not support that functionality.
So this PR implements the dispatch function and implements
the following standard functions:
+ bzip2
+ zstandard
+ blosc
Specific test cases are also provided for HDF5 and NCZarr.
Over time, other specific standard filters will be defined.
## Primary Changes
* Add nc_inq_filter_avail() to netcdf-c API.
* Add standard filter implementations to test use of *nc_inq_filter_avail*.
* Bump the dispatch table version number and add to all the relevant
dispatch tables (libsrc, libsrcp, etc).
* Create a program to invoke nc_inq_filter_avail so that it is accessible
to shell scripts.
* Cleanup szip support to properly support szip
when HDF5 is disabled. This involves detecting
libsz separately from testing if HDF5 supports szip.
* Integrate shuffle and fletcher32 into the existing
filter API. This means that, for example, nc_def_var_fletcher32
is now a wrapper around nc_def_var_filter.
* Extend the Codec defaulting to allow multiple default shared libraries.
## Misc. Changes
* Modify configure.ac/CMakeLists.txt to look for the relevant
libraries implementing standard filters.
* Modify libnetcdf.settings to list available standard filters
(including deflate and szip).
* Add CMake test modules to locate libbz2 and libzstd.
* Cleanup the HDF5 memory manager function use in the plugins.
* remove unused file include//ncfilter.h
* remove tests for the HDF5 memory operations e.g. H5allocate_memory.
* Add flag to ncdump to force use of _Filter instead of _Deflate
or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
|
|
|
# Test for filter availability
|
|
|
|
|
avail() {
|
|
|
|
|
if test yes = `${execdir}/../ncdump/ncfilteravail $1` ; then return 0 ; else echo "filter $1 not available" ; return 1; fi
|
|
|
|
|
}
|
2022-02-09 11:53:30 +08:00
|
|
|
if test "x$FP_ISMSVC" = xyes || test "x$FP_ISCYGWIN" = xyes; then export LC_ALL="en_US.utf8"; fi
|
Enhance/Fix filter support
re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214
The primary change is to support so-called "standard filters".
A standard filter is one that is defined by the following
netcdf-c API:
````
int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params);
int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params);
````
So for example, zstandard would be a standard filter by defining
the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*.
In order to define these functions, we need a new dispatch function:
````
int nc_inq_filter_avail(int ncid, unsigned filterid);
````
This function, combined with the existing filter API can be used
to implement arbitrary standard filters using a simple code pattern.
Note that I would have preferred that this function return a list
of all available filters, but HDF5 does not support that functionality.
So this PR implements the dispatch function and implements
the following standard functions:
+ bzip2
+ zstandard
+ blosc
Specific test cases are also provided for HDF5 and NCZarr.
Over time, other specific standard filters will be defined.
## Primary Changes
* Add nc_inq_filter_avail() to netcdf-c API.
* Add standard filter implementations to test use of *nc_inq_filter_avail*.
* Bump the dispatch table version number and add to all the relevant
dispatch tables (libsrc, libsrcp, etc).
* Create a program to invoke nc_inq_filter_avail so that it is accessible
to shell scripts.
* Cleanup szip support to properly support szip
when HDF5 is disabled. This involves detecting
libsz separately from testing if HDF5 supports szip.
* Integrate shuffle and fletcher32 into the existing
filter API. This means that, for example, nc_def_var_fletcher32
is now a wrapper around nc_def_var_filter.
* Extend the Codec defaulting to allow multiple default shared libraries.
## Misc. Changes
* Modify configure.ac/CMakeLists.txt to look for the relevant
libraries implementing standard filters.
* Modify libnetcdf.settings to list available standard filters
(including deflate and szip).
* Add CMake test modules to locate libbz2 and libzstd.
* Cleanup the HDF5 memory manager function use in the plugins.
* remove unused file include//ncfilter.h
* remove tests for the HDF5 memory operations e.g. H5allocate_memory.
* Add flag to ncdump to force use of _Filter instead of _Deflate
or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
|
|
|
|
2017-03-09 08:01:10 +08:00
|
|
|
# Make sure we are in builddir (not execdir)
|
|
|
|
|
cd $builddir
|
