mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2024-11-21 03:13:42 +08:00
Code Issues Packages Projects Releases Wiki Activity
c798bb6405
netcdf-c/CMakeLists.txt

2746 lines
90 KiB
CMake
Raw Normal View History

Added a short copyright message at head of a couple CMakeLists files; this will be propegated to the others. Updated version in configure.ac to 4.3.1-rc2.
2013-08-15 23:43:59 +08:00
## This is a CMake file, part of Unidata's netCDF package.
Silenced warning in CMakeLists.txt
2018-03-27 01:53:31 +08:00
# Copyright 2012-2018, see the COPYRIGHT file for more information.
Tweaked cmake header.
2015-10-15 01:08:54 +08:00
#
Added a short copyright message at head of a couple CMakeLists files; this will be propegated to the others. Updated version in configure.ac to 4.3.1-rc2.
2013-08-15 23:43:59 +08:00
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
##################################
# Set Project Properties
##################################
2012-10-03 04:56:46 +08:00
#Minimum required CMake Version
CMake: CMP0075 requires CMake >= 3.12. The presence of cmake_policy(CMP0075 required CMake >=3.12, and errors with older CMake. Since older versions of CMake than 3.12 are already broken with NetCDF and uncommon nowadays, it may be better to explicitly require 3.12 or newer.
2022-02-08 00:14:57 +08:00
cmake_minimum_required(VERSION 3.12.0)
# CMake 3.12: Use libraries specified in CMAKE_REQUIRED_LIBRARIES for check include macros
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
Begin wiring in option to toggle automatic search for libxml2 in support of https://github.com/Unidata/netcdf-c/pulls/2135
2021-11-03 05:35:04 +08:00
#Project Name
CMake: CMP0075 requires CMake >= 3.12. The presence of cmake_policy(CMP0075 required CMake >=3.12, and errors with older CMake. Since older versions of CMake than 3.12 are already broken with NetCDF and uncommon nowadays, it may be better to explicitly require 3.12 or newer.
2022-02-08 00:14:57 +08:00
project(netCDF
LANGUAGES C CXX
HOMEPAGE_URL "https://www.unidata.ucar.edu/software/netcdf/"
DESCRIPTION "NetCDF is a set of software libraries and machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data."
)
Merged latest changes from netcdf-cmake branch. Includes a few windows fixes, plus the skeleton for 'make dist' and 'make distcheck' support in cmake-based builds.
2013-03-28 03:15:00 +08:00
set(PACKAGE "netCDF" CACHE STRING "")
Added stanza to cmake build system for CMAKE_MACOSX_RPATH. Bumped minimum cmake version required to 2.8.12
2014-07-16 00:50:58 +08:00
Added a netcdf_lib_version variable to correspond to the version computed by autotools. Also changed netcdf_so_version to 7 in cmake. This way cmake-based builds give us the same library names as autotools-based builds.
2014-05-13 06:05:39 +08:00
#####
# Version Info:
#
# Release Version
# Library Version
# SO Version
#
# SO Version is computed from library version. See:
# http://www.gnu.org/software/libtool/manual/libtool.html#Libtool-versioning
#####
Added generated file for cmake-based builds.
2014-09-11 00:49:32 +08:00
SET(NC_VERSION_MAJOR 4)
Bumped dev version in CMakeLists.txt. SO Version remains unchanged.
2022-06-04 00:55:41 +08:00
SET(NC_VERSION_MINOR 9)
Bump version strings to reflect next development version.
2023-03-14 05:43:44 +08:00
SET(NC_VERSION_PATCH 3)
Changed version to include 'development' signifier. This will be removed/changed when preparing releases. This way, there will never be a question about what is being worked on.
2015-06-20 02:29:15 +08:00
SET(NC_VERSION_NOTE "-development")
Added generated file for cmake-based builds.
2014-09-11 00:49:32 +08:00
SET(netCDF_VERSION ${NC_VERSION_MAJOR}.${NC_VERSION_MINOR}.${NC_VERSION_PATCH}${NC_VERSION_NOTE})
o Improved package_source support in CMake-based builds. o Corrected an issue related to uninitialized variables which caused test failures on 2 test systems under very specific circumstances.
2013-02-20 07:29:41 +08:00
SET(VERSION ${netCDF_VERSION})
Added generated file for cmake-based builds.
2014-09-11 00:49:32 +08:00
SET(NC_VERSION ${netCDF_VERSION})
Reorganized doxygen documents, renamed .doc files as .dox. Bumped revision to 4.3.1.2. There may not be a 4.3.1.2 release, but I didn't want to leave it as 4.3.1.1. Removed CMake-specific doxygen/documentation template, made changes required so that CMake and autotools-based builds can use the same template (Doxyfile.in).
2014-02-20 07:32:26 +08:00
SET(PACKAGE_VERSION ${VERSION})
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
# These values should match those in configure.ac
Fix typo in CMakeLists.txt At the time generated dynamic library is named `libnetcdf.so.19}` which looks like a typo.
2022-06-20 00:29:20 +08:00
SET(netCDF_LIB_VERSION 19)
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
SET(netCDF_SO_VERSION 19)
fixed dispatch version number in CMakeLists.txt
2020-07-15 20:58:16 +08:00
# Version of the dispatch table. This must match the value in
# configure.ac.
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
SET(NC_DISPATCH_VERSION 5)
fixed setting of NC_DISPATCH_VERSION in cmake build
2019-11-16 05:57:07 +08:00
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
# Get system configuration, Use it to determine osname, os release, cpu. These
# will be used when committing to CDash.
Merging latest changes from netcdf-cmake branch.
2013-03-12 05:13:51 +08:00
find_program(UNAME NAMES uname)
Merged latest from netcdf-cmake branch. Addressed several cmake-related issues, in terms of building on Windows. Also added XGetopt.c to EXTRA_dist in Makefile.am files.
2013-03-20 06:40:15 +08:00
IF(UNAME)
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
macro(getuname name flag)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
exec_program("${UNAME}" ARGS "${flag}" OUTPUT_VARIABLE "${name}")
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
endmacro(getuname)
getuname(osname -s)
getuname(osrel -r)
getuname(cpu -m)
Added BUILDNAME_PREFIX and BUILDNAME_SUFFIX options to cmake build system, to make it easier to customize the build name displayed on the CDash dashboard.
2015-04-24 01:03:26 +08:00
set(TMP_BUILDNAME "${osname}-${osrel}-${cpu}")
Merged latest from netcdf-cmake branch. Addressed several cmake-related issues, in terms of building on Windows. Also added XGetopt.c to EXTRA_dist in Makefile.am files.
2013-03-20 06:40:15 +08:00
ENDIF()
Added BUILDNAME_PREFIX and BUILDNAME_SUFFIX options to cmake build system, to make it easier to customize the build name displayed on the CDash dashboard.
2015-04-24 01:03:26 +08:00
Improve UTF8 Support On Windows re: Issue https://github.com/Unidata/netcdf-c/issues/2190 The primary purpose of this PR is to improve the utf8 support for windows. This is persuant to a change in Windows that supports utf8 natively (almost). The almost means that it is still utf16 internally and the set of characters representable by utf8 is larger than those representable by utf16. This leaves open the question in the Issue about handling the Windows 1252 character set. This required the following changes: 1. Test the Windows build and major version in order to see if native utf8 is supported. 2. If native utf8 is supported, Modify dpathmgr.c to call the 8-bit version of the windows fopen() and open() functions. 3. In support of this, programs that use XGetOpt (Windows versions) need to get the command line as utf8 and then parse to arc+argv as utf8. This requires using a homegrown command line parser named XCommandLineToArgvA. 4. Add a utility program called "acpget" that prints out the current Windows code page and locale. Additionally, some technical debt was cleaned up as follows: 1. Unify all the places which attempt to read all or a part of a file into the dutil.c#NC_readfile code. 2. Similary unify all the code that creates temp files into dutil.c#NC_mktmp code. 3. Convert almost all remaining calls to fopen() and open() to NCfopen() and NCopen3(). This is to ensure that path management is used consistently. This touches a number of files. 4. extern->EXTERNL as needed to get it to work under Windows.
2022-02-09 11:53:30 +08:00
# Define some Platforms
if(osname MATCHES "CYGWIN.*")
SET(ISCYGWIN yes)
endif()
if(osname MATCHES "Darwin.*")
SET(ISOSX yes)
endif()
if(MSVC)
SET(ISMSVC yes)
endif()
Make public a limited API for programmatic access to internal .rc tables re: https://github.com/Unidata/netcdf-c/issues/2337 re: https://github.com/Unidata/netcdf-c/issues/2407 Add two functions to netcdf.h to allow programs to get/set selected entries into the internal .rc tables. This should fix the above issues by allowing HTTP.CAINFO to be set to the certificates directory. Note that the changes should be performed as early as possible in the program because some of the .rc table entries may get cached internally and changing the entry after that caching occurs may have no effect. The new signatures are as follows: 1. Get the value of a simple .rc entry of the form "key=value". Note that caller must free the returned value, which might be NULL. ```` char* nc_rc_get(char* const * key); @param key table entry key @return value if .rc table has entry of the form key=value @return NULL if no such entry is found. ```` 2. Insert/Overwrite the specified key=value pair in the .rc table. ```` int nc_rc_set(const char* key, const char* value); @param key table entry key -- may not be NULL @param value table entry value -- may not be NULL @return NC_NOERR if no error @return NC_EINVAL if error ```` Addendum: re: https://github.com/Unidata/netcdf-c/issues/2407 Modify dhttp.c to use the .rc entry HTTP.CAINFO if defined.
2022-06-18 04:35:12 +08:00
if(osname MATCHES "MINGW.*" OR osname MATCHES "MSYS.*")
Improve UTF8 Support On Windows re: Issue https://github.com/Unidata/netcdf-c/issues/2190 The primary purpose of this PR is to improve the utf8 support for windows. This is persuant to a change in Windows that supports utf8 natively (almost). The almost means that it is still utf16 internally and the set of characters representable by utf8 is larger than those representable by utf16. This leaves open the question in the Issue about handling the Windows 1252 character set. This required the following changes: 1. Test the Windows build and major version in order to see if native utf8 is supported. 2. If native utf8 is supported, Modify dpathmgr.c to call the 8-bit version of the windows fopen() and open() functions. 3. In support of this, programs that use XGetOpt (Windows versions) need to get the command line as utf8 and then parse to arc+argv as utf8. This requires using a homegrown command line parser named XCommandLineToArgvA. 4. Add a utility program called "acpget" that prints out the current Windows code page and locale. Additionally, some technical debt was cleaned up as follows: 1. Unify all the places which attempt to read all or a part of a file into the dutil.c#NC_readfile code. 2. Similary unify all the code that creates temp files into dutil.c#NC_mktmp code. 3. Convert almost all remaining calls to fopen() and open() to NCfopen() and NCopen3(). This is to ensure that path management is used consistently. This touches a number of files. 4. extern->EXTERNL as needed to get it to work under Windows.
2022-02-09 11:53:30 +08:00
SET(ISMINGW yes)
SET(MINGW yes)
endif()
Added BUILDNAME_PREFIX and BUILDNAME_SUFFIX options to cmake build system, to make it easier to customize the build name displayed on the CDash dashboard.
2015-04-24 01:03:26 +08:00
###
# Allow for some customization of the buildname.
# This will make it easier to identify different builds,
# based on values passed from command line/shell scripts.
#
# For ctest scripts, we can use CTEST_BUILD_NAME.
###
SET(BUILDNAME_PREFIX "" CACHE STRING "")
SET(BUILDNAME_SUFFIX "" CACHE STRING "")
IF(BUILDNAME_PREFIX)
Corrected logic error in buildname prefix/suffix.
2015-04-24 01:13:50 +08:00
SET(TMP_BUILDNAME "${BUILDNAME_PREFIX}-${TMP_BUILDNAME}")
Added BUILDNAME_PREFIX and BUILDNAME_SUFFIX options to cmake build system, to make it easier to customize the build name displayed on the CDash dashboard.
2015-04-24 01:03:26 +08:00
ENDIF()
IF(BUILDNAME_SUFFIX)
Corrected logic error in buildname prefix/suffix.
2015-04-24 01:13:50 +08:00
SET(TMP_BUILDNAME "${TMP_BUILDNAME}-${BUILDNAME_SUFFIX}")
Added BUILDNAME_PREFIX and BUILDNAME_SUFFIX options to cmake build system, to make it easier to customize the build name displayed on the CDash dashboard.
2015-04-24 01:03:26 +08:00
ENDIF()
IF(NOT BUILDNAME)
SET(BUILDNAME "${TMP_BUILDNAME}" CACHE STRING "Build name variable for CDash")
ENDIF()
###
# End BUILDNAME customization.
###
Use GNUInstallDirs to install into /usr/lib64 as needed
2013-08-14 00:00:22 +08:00
# For CMAKE_INSTALL_LIBDIR
INCLUDE(GNUInstallDirs)
2012-10-03 04:56:46 +08:00
IF(MSVC)
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
SET(GLOBAL PROPERTY USE_FOLDERS ON)
Improve UTF8 Support On Windows re: Issue https://github.com/Unidata/netcdf-c/issues/2190 The primary purpose of this PR is to improve the utf8 support for windows. This is persuant to a change in Windows that supports utf8 natively (almost). The almost means that it is still utf16 internally and the set of characters representable by utf8 is larger than those representable by utf16. This leaves open the question in the Issue about handling the Windows 1252 character set. This required the following changes: 1. Test the Windows build and major version in order to see if native utf8 is supported. 2. If native utf8 is supported, Modify dpathmgr.c to call the 8-bit version of the windows fopen() and open() functions. 3. In support of this, programs that use XGetOpt (Windows versions) need to get the command line as utf8 and then parse to arc+argv as utf8. This requires using a homegrown command line parser named XCommandLineToArgvA. 4. Add a utility program called "acpget" that prints out the current Windows code page and locale. Additionally, some technical debt was cleaned up as follows: 1. Unify all the places which attempt to read all or a part of a file into the dutil.c#NC_readfile code. 2. Similary unify all the code that creates temp files into dutil.c#NC_mktmp code. 3. Convert almost all remaining calls to fopen() and open() to NCfopen() and NCopen3(). This is to ensure that path management is used consistently. This touches a number of files. 4. extern->EXTERNL as needed to get it to work under Windows.
2022-02-09 11:53:30 +08:00
ADD_COMPILE_OPTIONS("/utf-8")
2012-10-03 04:56:46 +08:00
ENDIF()
#Add custom CMake Module
Update CmakeLists.txt in support of https://github.com/Unidata/netcdf-c/issues/1295
2019-01-26 02:26:26 +08:00
SET(CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules/"
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
CACHE INTERNAL "Location of our custom CMake modules.")
2012-10-03 04:56:46 +08:00
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
# auto-configure style checks, other CMake modules.
cmake: include modules by name This allows for toolchains to override these modules.
2017-12-30 04:14:45 +08:00
INCLUDE(CheckLibraryExists)
INCLUDE(CheckIncludeFile)
INCLUDE(CheckIncludeFiles)
INCLUDE(CheckTypeSize)
INCLUDE(CheckFunctionExists)
INCLUDE(CheckCXXSourceCompiles)
INCLUDE(CheckCSourceCompiles)
INCLUDE(TestBigEndian)
INCLUDE(CheckSymbolExists)
INCLUDE(GetPrerequisites)
Added checks for CURLOPT_KEYPASSWD and CURLINFO_RESPONSE_CODE in CMakeLists.txt for DAP-enabled builds.
2014-03-12 04:31:41 +08:00
Updating cmake shared_linker_flags as suggested by Nico Schlomer: https://github.com/Unidata/netcdf-c/issues/46 Fixed indentation. Updated .gitignore. Added a macro to make it easy to check to see if the linker supports a particular flag. Debug builds now pass the -w1,--no-undefined if it is detected the linker supports it.
2014-04-03 04:26:42 +08:00
INCLUDE(CheckCCompilerFlag)
Merged latest changes from netcdf-cmake branch in preparation for next release candidate (4.3.0 rc4).
2013-03-26 04:28:40 +08:00
FIND_PACKAGE(PkgConfig QUIET)
Merged latest changes from netcdf-cmake branch. Includes a few windows fixes, plus the skeleton for 'make dist' and 'make distcheck' support in cmake-based builds.
2013-03-28 03:15:00 +08:00
Added a check for endianness in CMake, the lack of which and fix for was contributed by Wei-Keng Liao
2017-02-07 03:23:55 +08:00
# A check to see if the system is big endian
Integrating a big endian test.
2017-02-09 05:51:41 +08:00
TEST_BIG_ENDIAN(BIGENDIAN)
IF(${BIGENDIAN})
Added a check for endianness in CMake, the lack of which and fix for was contributed by Wei-Keng Liao
2017-02-07 03:23:55 +08:00
SET(WORDS_BIGENDIAN "1")
Integrating a big endian test.
2017-02-09 05:51:41 +08:00
ENDIF(${BIGENDIAN})
Fix way CMake handle -DPLUGIN_INSTALL_DIR re: Issue https://github.com/Unidata/netcdf-c/issues/2429 The CMake tests for -DPLUGIN_INSTALL_DIR only accepted "on" or "off" as boolean values. So extend to accept e.g. -DPLUGIN_INSTALL_DIR=yes or -DPLUGIN_INSTALL_DIR=1, etc.
2022-06-26 06:47:49 +08:00
# Define a function to convert various true or false values
# to either TRUE|FALSE (uppercase).
# If value is not a recognized boolean, then return NOTFOUND
#1, ON, YES, TRUE, Y,
#0, OFF, NO, FALSE, N, IGNORE, NOTFOUND -NOTFOUND ""
SET(TRUELIST "on;yes;y;true")
SET(FALSELIST "off;no;n;false;0;ignore;notfound")
function(booleanize VALUE RETVAR)
# force case
STRING(TOLOWER "${VALUE}" LCVALUE)
# Now do all the comparisons
if(LCVALUE IN_LIST TRUELIST OR LCVALUE GREATER 0)
SET(${RETVAR} TRUE PARENT_SCOPE)
elseif(LCVALUE IN_LIST FALSELIST OR LCVALUE MATCHES ".*-notfound" OR LCVALUE STREQUAL "")
SET(${RETVAR} FALSE PARENT_SCOPE)
else()
SET(${RETVAR} NOTFOUND PARENT_SCOPE)
endif()
endfunction()
Updating cmake shared_linker_flags as suggested by Nico Schlomer: https://github.com/Unidata/netcdf-c/issues/46 Fixed indentation. Updated .gitignore. Added a macro to make it easy to check to see if the linker supports a particular flag. Debug builds now pass the -w1,--no-undefined if it is detected the linker supports it.
2014-04-03 04:26:42 +08:00
# A macro to check if a C linker supports a particular flag.
MACRO(CHECK_C_LINKER_FLAG M_FLAG M_RESULT)
SET(T_REQ_FLAG "${CMAKE_REQUIRED_FLAGS}")
SET(CMAKE_REQUIRED_FLAGS "${M_FLAG}")
CHECK_C_SOURCE_COMPILES("int main() {return 0;}" ${M_RESULT})
SET(CMAKE_REQUIRED_FLAGS "${T_REQ_FLAG}")
ENDMACRO()
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
# Enable 'dist and distcheck'.
# File adapted from http://ensc.de/cmake/FindMakeDist.cmake
FIND_PACKAGE(MakeDist)
# End 'enable dist and distcheck'
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
# Set the build type.
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
IF(NOT CMAKE_BUILD_TYPE)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
SET(CMAKE_BUILD_TYPE DEBUG CACHE STRING "Choose the type of build, options are: None, Debug, Release."
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
FORCE)
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
ENDIF()
Updating cmake shared_linker_flags as suggested by Nico Schlomer: https://github.com/Unidata/netcdf-c/issues/46 Fixed indentation. Updated .gitignore. Added a macro to make it easy to check to see if the linker supports a particular flag. Debug builds now pass the -w1,--no-undefined if it is detected the linker supports it.
2014-04-03 04:26:42 +08:00
# Set build type uppercase
STRING(TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE)
Continued to flesh out cmake support for libnetcdf.settings. Added a pair of functions which will return a yes/no depending on whether or not a variable is defined. By default, cmake uses TRUE/FALSE.
2014-06-03 05:14:27 +08:00
# Determine the configure date.
Added checks for Windows systems.
2014-12-16 01:56:14 +08:00
Use SOURCE_DATE_EPOCH instead of current date. The environment variable is used to support Reproducible Builds. https://reproducible-builds.org/specs/source-date-epoch/
2016-05-15 23:20:18 +08:00
IF(DEFINED ENV{SOURCE_DATE_EPOCH})
EXECUTE_PROCESS(
COMMAND "date" "-u" "-d" "@$ENV{SOURCE_DATE_EPOCH}"
OUTPUT_VARIABLE CONFIG_DATE
)
ELSE()
EXECUTE_PROCESS(
COMMAND date
OUTPUT_VARIABLE CONFIG_DATE
)
ENDIF()
Added checks for Windows systems.
2014-12-16 01:56:14 +08:00
IF(CONFIG_DATE)
string(STRIP ${CONFIG_DATE} CONFIG_DATE)
ENDIF()
Added NC_EXTRA_DEPS option to netcdf-c
2014-09-30 02:04:35 +08:00
##
# Allow for extra dependencies.
##
SET(EXTRA_DEPS "")
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
################################
# End Project Properties
################################
################################
# Set CTest Properties
################################
ENABLE_TESTING()
INCLUDE(CTest)
# Copy the CTest customization file into binary directory, as required.
Changes to make it possible to nest this project inside of another CMake project.
2019-11-07 00:43:49 +08:00
FILE(COPY ${CMAKE_CURRENT_SOURCE_DIR}/CTestCustom.cmake DESTINATION ${CMAKE_CURRENT_BINARY_DIR})
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
# Set Memory test program for non-MSVC based builds.
# Assume valgrind for now.
BLD: Get CMake build compiling on Cygwin. Mostly changing to Markdown files to a format Doxygen recognizes.
2022-07-12 03:48:04 +08:00
IF((NOT MSVC) AND (NOT MINGW) AND (NOT ISCYGWIN))
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
SET(CTEST_MEMORYCHECK_COMMAND valgrind CACHE STRING "")
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
ENDIF()
# Set variable to define the build type.
INCLUDE(GenerateExportHeader)
################################
# End CTest Properties
################################
Fixed an issue with Windows builds not compiling due to how ssize_t was being detected/defined.
2013-05-25 05:19:07 +08:00
Merged latest changes from netcdf-cmake branch. Includes a few windows fixes, plus the skeleton for 'make dist' and 'make distcheck' support in cmake-based builds.
2013-03-28 03:15:00 +08:00
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
################################
Adding cmake infrastructure in support of https://github.com/Unidata/netcdf-c/issues/228
2016-03-08 05:04:52 +08:00
# Compiler and Linker Configuration
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
################################
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
##
Reformatted CMakeLists.txt to move compiler options into a single unified block at the top.
2014-01-25 02:10:41 +08:00
# Default building shared libraries.
Manual merge of pull request https://github.com/Unidata/netcdf-c/pull/64 contributed by nschloe. Assorted CMake improvements.
2014-06-12 05:51:31 +08:00
# BUILD_SHARED_LIBS is provided by/used by
# CMake directly.
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
##
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
OPTION(BUILD_SHARED_LIBS "Configure netCDF as a shared library." ON)
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
IF(BUILD_SHARED_LIBS)
Manual merge of pull request https://github.com/Unidata/netcdf-c/pull/64 contributed by nschloe. Assorted CMake improvements.
2014-06-12 05:51:31 +08:00
SET(CMAKE_POSITION_INDEPENDENT_CODE ON)
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
ENDIF()
Adding cmake infrastructure in support of https://github.com/Unidata/netcdf-c/issues/228
2016-03-08 05:04:52 +08:00
OPTION(NC_FIND_SHARED_LIBS "Find dynamically-built versions of dependent libraries" ${BUILD_SHARED_LIBS})
##
# We've had a request to allow for non-versioned shared libraries.
Misc. source comment typos Some are user-facing. Found via `codespell` and through the downstream FreeCAD.
2018-03-16 22:38:40 +08:00
# This seems reasonable enough to accommodate. See
Adding cmake infrastructure in support of https://github.com/Unidata/netcdf-c/issues/228
2016-03-08 05:04:52 +08:00
# https://github.com/Unidata/netcdf-c/issues/228 for more info.
##
OPTION(ENABLE_SHARED_LIBRARY_VERSION "Encode the library SO version in the file name of the generated library file." ON)
Reformatted CMakeLists.txt to move compiler options into a single unified block at the top.
2014-01-25 02:10:41 +08:00
# Set some default linux gcc & apple compiler options for
# debug builds.
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
IF(CMAKE_COMPILER_IS_GNUCC OR APPLE)
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
OPTION(ENABLE_COVERAGE_TESTS "Enable compiler flags needed to perform coverage tests." OFF)
OPTION(ENABLE_CONVERSION_WARNINGS "Enable warnings for implicit conversion from 64 to 32-bit datatypes." ON)
OPTION(ENABLE_LARGE_FILE_TESTS "Enable large file tests." OFF)
improve mechanism by which to find parallel HDF5 So far, CMakeLists.txt made the assumption that ${HDF5_LIB} is always defined in order to determine if it has parallel support. This is not the case, compare with bug #25. Make use of FIND_PACKAGE(HDF5) now; it sets HDF5_IS_PARALLEL anyways.
2014-03-07 23:40:50 +08:00
Reformatted CMakeLists.txt to move compiler options into a single unified block at the top.
2014-01-25 02:10:41 +08:00
# Debugging flags
Reorganized C_FLAG definitions to make debug, release builds a bit more configurable.
2014-03-20 03:56:09 +08:00
SET(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -Wall")
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
Corrected CMAKE_SHARED_LINKER_FLAGS_DEBUG setting, replacing 'one' with 'ell', as reported by nschloe at https://github.com/Unidata/netcdf-c/issues/58
2014-05-21 05:31:08 +08:00
# Check to see if -Wl,--no-undefined is supported.
CHECK_C_LINKER_FLAG("-Wl,--no-undefined" LIBTOOL_HAS_NO_UNDEFINED)
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
Updating cmake shared_linker_flags as suggested by Nico Schlomer: https://github.com/Unidata/netcdf-c/issues/46 Fixed indentation. Updated .gitignore. Added a macro to make it easy to check to see if the linker supports a particular flag. Debug builds now pass the -w1,--no-undefined if it is detected the linker supports it.
2014-04-03 04:26:42 +08:00
IF(LIBTOOL_HAS_NO_UNDEFINED)
Corrected CMAKE_SHARED_LINKER_FLAGS_DEBUG setting, replacing 'one' with 'ell', as reported by nschloe at https://github.com/Unidata/netcdf-c/issues/58
2014-05-21 05:31:08 +08:00
SET(CMAKE_SHARED_LINKER_FLAGS_DEBUG "${CMAKE_SHARED_LINKER_FLAGS_DEBUG} -Wl,--no-undefined")
Updating cmake shared_linker_flags as suggested by Nico Schlomer: https://github.com/Unidata/netcdf-c/issues/46 Fixed indentation. Updated .gitignore. Added a macro to make it easy to check to see if the linker supports a particular flag. Debug builds now pass the -w1,--no-undefined if it is detected the linker supports it.
2014-04-03 04:26:42 +08:00
ENDIF()
SET(CMAKE_REQUIRED_FLAGS "${TMP_CMAKE_REQUIRED_FLAGS}")
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
Reformatted CMakeLists.txt to move compiler options into a single unified block at the top.
2014-01-25 02:10:41 +08:00
# Coverage tests need to have optimization turned off.
Reorganized C_FLAG definitions to make debug, release builds a bit more configurable.
2014-03-20 03:56:09 +08:00
IF(ENABLE_COVERAGE_TESTS)
Reformatted CMakeLists.txt to move compiler options into a single unified block at the top.
2014-01-25 02:10:41 +08:00
SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g -O0 -fprofile-arcs -ftest-coverage")
SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fprofile-arcs -ftest-coverage")
improve mechanism by which to find parallel HDF5 So far, CMakeLists.txt made the assumption that ${HDF5_LIB} is always defined in order to determine if it has parallel support. This is not the case, compare with bug #25. Make use of FIND_PACKAGE(HDF5) now; it sets HDF5_IS_PARALLEL anyways.
2014-03-07 23:40:50 +08:00
MESSAGE(STATUS "Coverage Tests: On.")
Reformatted CMakeLists.txt to move compiler options into a single unified block at the top.
2014-01-25 02:10:41 +08:00
ENDIF()
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
This PR adds EXPERIMENTAL support for accessing data in the cloud using a variant of the Zarr protocol and storage format. This enhancement is generically referred to as "NCZarr". The data model supported by NCZarr is netcdf-4 minus the user-defined types and the String type. In this sense it is similar to the CDF-5 data model. More detailed information about enabling and using NCZarr is described in the document NUG/nczarr.md and in a [Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in). WARNING: this code has had limited testing, so do use this version for production work. Also, performance improvements are ongoing. Note especially the following platform matrix of successful tests: Platform | Build System | S3 support ------------------------------------ Linux+gcc | Automake | yes Linux+gcc | CMake | yes Visual Studio | CMake | no Additionally, and as a consequence of the addition of NCZarr, major changes have been made to the Filter API. NOTE: NCZarr does not yet support filters, but these changes are enablers for that support in the future. Note that it is possible (probable?) that there will be some accidental reversions if the changes here did not correctly mimic the existing filter testing. In any case, previously filter ids and parameters were of type unsigned int. In order to support the more general zarr filter model, this was all converted to char*. The old HDF5-specific, unsigned int operations are still supported but they are wrappers around the new, char* based nc_filterx_XXX functions. This entailed at least the following changes: 1. Added the files libdispatch/dfilterx.c and include/ncfilter.h 2. Some filterx utilities have been moved to libdispatch/daux.c 3. A new entry, "filter_actions" was added to the NCDispatch table and the version bumped. 4. An overly complex set of structs was created to support funnelling all of the filterx operations thru a single dispatch "filter_actions" entry. 5. Move common code to from libhdf5 to libsrc4 so that it is accessible to nczarr. Changes directly related to Zarr: 1. Modified CMakeList.txt and configure.ac to support both C and C++ -- this is in support of S3 support via the awd-sdk libraries. 2. Define a size64_t type to support nczarr. 3. More reworking of libdispatch/dinfermodel.c to support zarr and to regularize the structure of the fragments section of a URL. Changes not directly related to Zarr: 1. Make client-side filter registration be conditional, with default off. 2. Hack include/nc4internal.h to make some flags added by Ed be unique: e.g. NC_CREAT, NC_INDEF, etc. 3. cleanup include/nchttp.h and libdispatch/dhttp.c. 4. Misc. changes to support compiling under Visual Studio including: * Better testing under windows for dirent.h and opendir and closedir. 5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags and to centralize error reporting. 6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them. 7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible. Changes Left TO-DO: 1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
# Warnings for 64-to-32 bit conversions.
Reformatted CMakeLists.txt to move compiler options into a single unified block at the top.
2014-01-25 02:10:41 +08:00
IF(ENABLE_CONVERSION_WARNINGS)
Updating cmake shared_linker_flags as suggested by Nico Schlomer: https://github.com/Unidata/netcdf-c/issues/46 Fixed indentation. Updated .gitignore. Added a macro to make it easy to check to see if the linker supports a particular flag. Debug builds now pass the -w1,--no-undefined if it is detected the linker supports it.
2014-04-03 04:26:42 +08:00
CHECK_C_COMPILER_FLAG(-Wconversion CC_HAS_WCONVERSION)
CHECK_C_COMPILER_FLAG(-Wshorten-64-to-32 CC_HAS_SHORTEN_64_32)
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
Updating cmake shared_linker_flags as suggested by Nico Schlomer: https://github.com/Unidata/netcdf-c/issues/46 Fixed indentation. Updated .gitignore. Added a macro to make it easy to check to see if the linker supports a particular flag. Debug builds now pass the -w1,--no-undefined if it is detected the linker supports it.
2014-04-03 04:26:42 +08:00
IF(CC_HAS_SHORTEN_64_32)
Reorganized C_FLAG definitions to make debug, release builds a bit more configurable.
2014-03-20 03:56:09 +08:00
SET(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -Wshorten-64-to-32")
Updating cmake shared_linker_flags as suggested by Nico Schlomer: https://github.com/Unidata/netcdf-c/issues/46 Fixed indentation. Updated .gitignore. Added a macro to make it easy to check to see if the linker supports a particular flag. Debug builds now pass the -w1,--no-undefined if it is detected the linker supports it.
2014-04-03 04:26:42 +08:00
ENDIF()
IF(CC_HAS_WCONVERSION)
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
SET(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -Wconversion")
Updating cmake shared_linker_flags as suggested by Nico Schlomer: https://github.com/Unidata/netcdf-c/issues/46 Fixed indentation. Updated .gitignore. Added a macro to make it easy to check to see if the linker supports a particular flag. Debug builds now pass the -w1,--no-undefined if it is detected the linker supports it.
2014-04-03 04:26:42 +08:00
ENDIF()
Reformatted CMakeLists.txt to move compiler options into a single unified block at the top.
2014-01-25 02:10:41 +08:00
ENDIF(ENABLE_CONVERSION_WARNINGS)
ENDIF(CMAKE_COMPILER_IS_GNUCC OR APPLE)
# End default linux gcc & apple compiler options.
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
Use relative pathnames in __FILE__ macros on mingw
2021-08-27 07:27:21 +08:00
# Use relative pathnames in __FILE__ macros on MINGW:
IF(MINGW)
CHECK_C_COMPILER_FLAG("-fmacro-prefix-map='${CMAKE_SOURCE_DIR}'=." CC_HAS_MACRO_PREFIX_MAP)
IF(CC_HAS_MACRO_PREFIX_MAP)
SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fmacro-prefix-map='${CMAKE_SOURCE_DIR}'=.")
ENDIF()
ENDIF()
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
ADD_DEFINITIONS()
Fix common typos Detected by https://github.com/vlajos/misspell_fixer
2015-08-20 17:42:05 +08:00
# Suppress CRT Warnings.
2012-11-08 01:34:05 +08:00
# Only necessary for Windows
IF(MSVC)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
ADD_DEFINITIONS(-D_CRT_SECURE_NO_WARNINGS)
Merged changes from netcdf-branch. o Changed variable names 'typeid' to 'typeid1' to avoid a namespace conflict in visual studio. o Cleaned up a handful of warnings in Visual Studio. o Addressed a few Coverity-discovered issuesl o Made changes to CMake-based builds.
2012-12-14 06:09:41 +08:00
ENDIF()
Support ANSI *printf specifiers on mingw-w64
2021-08-24 17:37:34 +08:00
# Support ANSI format specifiers for *printf on MINGW:
IF(MINGW)
ADD_DEFINITIONS(-D__USE_MINGW_ANSI_STDIO=1)
ENDIF()
Add local include paths first If a netcdf.h include file exists in any of the include paths specified for HDF5 or other libraries, it was being included instead of the local netcdf.h file in the source directory since the local include paths were added at the end of the compile line instead of at the beginning. This commit puts the local include paths at the beginning of the compile line
2015-11-16 20:51:17 +08:00
#####
# System inspection checks
#####
Unify definition of NC_DISPATCH_VERSION re: Issue The netcdf dispatch table version was defined in several places. Modify to only require defining it in CMakeLists.txt and configure.ac. Fix entailed the following changes: * Up the NC_DISPATCH_VERSION from 2 to 3 in configure.ac and CMakeLists.txt * Create include/netcdf_dispatch.h.in and use it to configure include/netcdf_dispatch.h * For CMAKE, make it search CMAKE_CURRENT_BINARY_DIR so code can locate the configured netcdf_dispatch.h * Add entry to config.h.cmake.in for NC_DISPATCH_VERSION * Move NCerror from include/ncdispatch.h to libdap2/nccomon.h * Fix an API problem re nchttp.h * Fix a conversion warning in libdispatch/dinfermodel.c
2021-02-01 12:40:08 +08:00
INCLUDE_DIRECTORIES(${CMAKE_CURRENT_BINARY_DIR}/include)
Changes to make it possible to nest this project inside of another CMake project.
2019-11-07 00:43:49 +08:00
INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}/include)
INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}/oc2)
INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}/libsrc)
SET(CMAKE_REQUIRED_INCLUDES ${CMAKE_CURRENT_SOURCE_DIR}/libsrc)
Add local include paths first If a netcdf.h include file exists in any of the include paths specified for HDF5 or other libraries, it was being included instead of the local netcdf.h file in the source directory since the local include paths were added at the end of the compile line instead of at the beginning. This commit puts the local include paths at the beginning of the compile line
2015-11-16 20:51:17 +08:00
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
################################
# End Compiler Configuration
################################
Fixed issue with linked libraries losing RPATH during installation.
2013-02-27 06:11:09 +08:00
##
# Configuration for post-install RPath
# Adapted from http://www.cmake.org/Wiki/CMake_RPATH_handling
##
Avoid setting RPATH in mingw
2021-08-25 08:40:56 +08:00
IF(NOT WIN32 AND BUILD_SHARED_LIBS)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
# use, i.e. don't skip the full RPATH for the build tree
SET(CMAKE_SKIP_BUILD_RPATH FALSE)
# when building, don't use the install RPATH already
# (but later on when installing)
SET(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE)
Added stanza to cmake build system for CMAKE_MACOSX_RPATH. Bumped minimum cmake version required to 2.8.12
2014-07-16 00:50:58 +08:00
if(APPLE)
set(CMAKE_MACOSX_RPATH ON)
endif(APPLE)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
# add the automatically determined parts of the RPATH
# which point to directories outside the build tree to the install RPATH
SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
Added stanza to cmake build system for CMAKE_MACOSX_RPATH. Bumped minimum cmake version required to 2.8.12
2014-07-16 00:50:58 +08:00
# the RPATH to be used when installing,
# but only if it's not a system directory
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
LIST(FIND CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES "${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}" isSystemDir)
IF("${isSystemDir}" STREQUAL "-1")
SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}")
ENDIF("${isSystemDir}" STREQUAL "-1")
Added stanza to cmake build system for CMAKE_MACOSX_RPATH. Bumped minimum cmake version required to 2.8.12
2014-07-16 00:50:58 +08:00
Fixed issue with linked libraries losing RPATH during installation.
2013-02-27 06:11:09 +08:00
ENDIF()
##
# End configuration for post-install RPath
##
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
################################
2012-10-03 04:56:46 +08:00
# Option checks
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
################################
2012-10-03 04:56:46 +08:00
# HDF5 cache variables.
new cache default in cmake build
2019-11-26 08:23:07 +08:00
SET(DEFAULT_CHUNK_SIZE 16777216 CACHE STRING "Default Chunk Cache Size.")
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
SET(DEFAULT_CHUNKS_IN_CACHE 10 CACHE STRING "Default number of chunks in cache.")
new cache default in cmake build
2019-11-26 08:23:07 +08:00
SET(CHUNK_CACHE_SIZE 16777216 CACHE STRING "Default Chunk Cache Size.")
SET(CHUNK_CACHE_NELEMS 4133 CACHE STRING "Default maximum number of elements in cache.")
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
SET(CHUNK_CACHE_PREEMPTION 0.75 CACHE STRING "Default file chunk cache preemption policy for HDf5 files(a number between 0 and 1, inclusive.")
Provide a Number of fixes/improvements to NCZarr Primary changes: * Add an improved cache system to speed up performance. * Fix NCZarr to properly handle scalar variables. Misc. Related Changes: * Added unit tests for extendible hash and for the generic cache. * Add config parameter to set size of the NCZarr cache. * Add initial performance tests but leave them unused. * Add CRC64 support. * Move location of ncdumpchunks utility from /ncgen to /ncdump. * Refactor auth support. Misc. Unrelated Changes: * More cleanup of the S3 support * Add support for S3 authentication in .rc files: HTTP.S3.ACCESSID and HTTP.S3.SECRETKEY. * Remove the hashkey from the struct OBJHDR since it is never used.
2020-11-20 08:01:04 +08:00
SET(CHUNK_CACHE_SIZE_NCZARR 4194304 CACHE STRING "Default NCZarr Chunk Cache Size.")
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
SET(MAX_DEFAULT_CACHE_SIZE 67108864 CACHE STRING "Default maximum cache size.")
SET(NETCDF_LIB_NAME "" CACHE STRING "Default name of the netcdf library.")
SET(TEMP_LARGE "." CACHE STRING "Where to put large temp files if large file tests are run.")
Fix STRNG typo in CMakeList
2019-03-17 16:09:16 +08:00
SET(NCPROPERTIES_EXTRA "" CACHE STRING "Specify extra pairs for _NCProperties.")
Added new option to allow for custom library names.
2012-11-01 06:17:33 +08:00
IF(NOT NETCDF_LIB_NAME STREQUAL "")
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
SET(MOD_NETCDF_NAME ON)
Added new option to allow for custom library names.
2012-11-01 06:17:33 +08:00
ENDIF()
2012-10-03 04:56:46 +08:00
# Set the appropriate compiler/architecture for universal OSX binaries.
IF(${CMAKE_SYSTEM_NAME} EQUAL "Darwin")
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
SET(CMAKE_OSX_ARCHITECTURES i386;x86_64)
2012-10-03 04:56:46 +08:00
ENDIF(${CMAKE_SYSTEM_NAME} EQUAL "Darwin")
changes to detect HDF5 HL, build with CRT and detect ZLIB in HDF5
2018-02-09 02:45:29 +08:00
# Macro for replacing '/MD' with '/MT'.
# Used only on Windows, /MD tells VS to use the shared
# CRT libs, MT tells VS to use the static CRT libs.
#
# Taken From:
# http://www.cmake.org/Wiki/CMake_FAQ#How_can_I_build_my_MSVC_application_with_a_static_runtime.3F
#
MACRO(specify_static_crt_flag)
SET(vars
CMAKE_C_FLAGS
CMAKE_C_FLAGS_DEBUG
add CMAKE_C_FLAGS_RELEASE in specify_static_crt_flag
2018-04-24 20:14:11 +08:00
CMAKE_C_FLAGS_RELEASE
changes to detect HDF5 HL, build with CRT and detect ZLIB in HDF5
2018-02-09 02:45:29 +08:00
CMAKE_C_FLAGS_MINSIZEREL
CMAKE_C_FLAGS_RELWITHDEBINFO
CMAKE_CXX_FLAGS CMAKE_CXX_FLAGS_DEBUG
CMAKE_CXX_FLAGS_RELEASE CMAKE_CXX_FLAGS_MINSIZEREL
CMAKE_CXX_FLAGS_RELWITHDEBINFO)
FOREACH(flag_var ${vars})
IF(${flag_var} MATCHES "/MD")
STRING(REGEX REPLACE "/MD" "/MT" ${flag_var} "${${flag_var}}")
ENDIF()
ENDFOREACH()
FOREACH(flag_var ${vars})
MESSAGE(STATUS " '${flag_var}': ${${flag_var}}")
ENDFOREACH()
MESSAGE(STATUS "")
ENDMACRO()
Added NC_USE_STATIC_CRT option to Cmake-based builds.
2013-09-19 04:49:37 +08:00
# Option to use Static Runtimes in MSVC
IF(MSVC)
OPTION(NC_USE_STATIC_CRT "Use static CRT Libraries ('\\MT')." OFF)
IF(NC_USE_STATIC_CRT)
SET(USE_STATIC_CRT ON)
specify_static_crt_flag()
ENDIF()
ENDIF()
o Improved package_source support in CMake-based builds. o Corrected an issue related to uninitialized variables which caused test failures on 2 test systems under very specific circumstances.
2013-02-20 07:29:41 +08:00
# Option to build netCDF Version 2
Corrected an issue with disabling the V2 API support in cmake-based builds.
2014-04-04 00:45:22 +08:00
OPTION (ENABLE_V2_API "Build netCDF Version 2." ON)
Corrected an issue with toggling the V2 API, that I had introduced earlier today.
2014-04-04 07:09:38 +08:00
SET(BUILD_V2 ${ENABLE_V2_API})
Corrected an issue with disabling the V2 API support in cmake-based builds.
2014-04-04 00:45:22 +08:00
IF(NOT ENABLE_V2_API)
Infrastructure for generating a meta header file dynamically at configure time.
2014-09-05 07:20:15 +08:00
SET(NO_NETCDF_2 ON)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
ELSE(NOT ENABLE_V2_API)
SET(USE_NETCDF_2 TRUE)
ENDIF(NOT ENABLE_V2_API)
2012-10-03 04:56:46 +08:00
# Option to build utilities
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
OPTION(BUILD_UTILITIES "Build ncgen, ncgen3, ncdump." ON)
improve mechanism by which to find parallel HDF5 So far, CMakeLists.txt made the assumption that ${HDF5_LIB} is always defined in order to determine if it has parallel support. This is not the case, compare with bug #25. Make use of FIND_PACKAGE(HDF5) now; it sets HDF5_IS_PARALLEL anyways.
2014-03-07 23:40:50 +08:00
2012-10-03 04:56:46 +08:00
# Option to use MMAP
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
OPTION(ENABLE_MMAP "Use MMAP." ON)
CMake fixes for enabling MMAP tests.
2013-02-15 04:24:02 +08:00
2012-10-03 04:56:46 +08:00
# Option to use examples.
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
OPTION(ENABLE_EXAMPLES "Build Examples" ON)
2012-10-03 04:56:46 +08:00
Moved stanza for NC_EXTRA_DEPS before checking for other libraries.
2016-11-11 07:22:02 +08:00
###
# Allow the user to specify libraries
# to link against, similar to automakes 'LIBS' variable.
###
SET(NC_EXTRA_DEPS "" CACHE STRING "Additional libraries to link against.")
IF(NC_EXTRA_DEPS)
STRING(REPLACE " " ";" DEPS_LIST ${NC_EXTRA_DEPS})
FOREACH(_DEP ${DEPS_LIST})
STRING(REGEX REPLACE "^-l" "" _LIB ${_DEP})
FIND_LIBRARY("${_LIB}_DEP" NAMES "${_LIB}" "lib${_LIB}")
MESSAGE(${${_LIB}_DEP})
Fix syntax for NC_EXTRA_DEPS
2018-02-10 03:57:11 +08:00
IF("${${_LIB}_DEP}" STREQUAL "${_LIB}_DEP-NOTFOUND")
Moved stanza for NC_EXTRA_DEPS before checking for other libraries.
2016-11-11 07:22:02 +08:00
MESSAGE(FATAL_ERROR "Error finding ${_LIB}.")
ELSE()
MESSAGE(STATUS "Found ${_LIB}: ${${_LIB}_DEP}")
ENDIF()
SET(EXTRA_DEPS ${EXTRA_DEPS} "${${_LIB}_DEP}")
ENDFOREACH()
MESSAGE("Extra deps: ${EXTRA_DEPS}")
LIST(REMOVE_DUPLICATES EXTRA_DEPS)
SET(CMAKE_REQUIRED_LIBRARIES ${CMAKE_REQUIRED_LIBRARIES} ${EXTRA_DEPS})
ENDIF()
###
# End user-specified dependent libraries.
###
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
################################
# Format Option checks
################################
# We need to now treat enable-netcdf4 and enable-hdf5 as separate,
# but for back compatability, we need to treat enable-netcdf4
# as equivalent to enable-hdf5.
# We detect this using these rules:
# 1. if ENABLE_HDF5 is off then disable hdf5
# 2. if ENABLE_NETCDF4 is off then disable hdf5
# 3. else enable hdf5
OPTION(ENABLE_NETCDF_4 "Use HDF5." ON)
OPTION(ENABLE_NETCDF4 "Use HDF5." ON)
OPTION(ENABLE_HDF5 "Use HDF5." ON)
IF(NOT ENABLE_HDF5 OR NOT ENABLE_NETCDF4 OR NOT ENABLE_NETCDF_4)
SET(ENABLE_HDF5 OFF CACHE BOOL "Use HDF5" FORCE)
ENDIF()
attempting fix for cmake HDF4 build problem when netcdf-4 is not built
2018-02-09 23:26:58 +08:00
OPTION(ENABLE_HDF4 "Build netCDF-4 with HDF4 read capability(HDF4, HDF5 and Zlib required)." OFF)
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
OPTION(ENABLE_DAP "Enable DAP2 and DAP4 Client." ON)
OPTION(ENABLE_NCZARR "Enable NCZarr Client." ON)
OPTION(ENABLE_PNETCDF "Build with parallel I/O for CDF-1, 2, and 5 files using PnetCDF." OFF)
SET(ENABLE_CDF5 AUTO CACHE STRING "AUTO")
OPTION(ENABLE_CDF5 "Enable CDF5 support" ON)
# Netcdf-4 support (i.e. libsrc4) is required by more than just HDF5 (e.g. NCZarr)
# So depending on what above formats are enabled, enable netcdf-4
if(ENABLE_HDF5 OR ENABLE_HDF4 OR ENABLE_NCZARR)
Correct logic for running DAP4 tests when HDF5 is not enabled.
2023-01-10 08:40:51 +08:00
SET(ENABLE_NETCDF_4 ON CACHE BOOL "Enable netCDF-4 API" FORCE)
SET(ENABLE_NETCDF4 ON CACHE BOOL "Enable netCDF4 Alias" FORCE)
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
endif()
Provide a single option to disable all network access and testing. Add the option "--disable-network-access" (automake) or "-DENABLE_NETWORK_ACCESS=OFF" (cmake). When disabled, this option transitively disables all network access capabilities and testing. If set, this option implies the following: * --disable-dap * --disable-byterange * --disable-s3 This PR answers a request for a feature from Ed Hartnett. ## Misc. Other changes * Take the opportunity to clean up some old, unused options; e.g. --enable-multifilters. * Fix bug in using S3 urls.
2023-06-11 04:08:04 +08:00
# enable|disable all forms of network access
Change the network access nomenclature to be less ambiguous. As it stood, did it refer to networked drives? Other things that are network adjacent?
2023-06-13 05:23:44 +08:00
OPTION(ENABLE_REMOTE_FUNCTIONALITY "Enable|disable all forms remote data access (DAP, S3, etc)" ON)
MESSAGE(">>> ENABLE_REMOTE_FUNCTIONALITY=${ENABLE_REMOTE_FUNCTIONALITY}")
if(NOT ENABLE_REMOTE_FUNCTIONALITY)
MESSAGE(WARNING "ENABLE_REMOTE_FUNCTIONALITY=NO => ENABLE_DAP[4]=NO")
SET(ENABLE_DAP OFF CACHE BOOL "ENABLE_REMOTE_FUNCTIONALITY=NO => ENABLE_DAP=NO" FORCE)
SET(ENABLE_DAP4 OFF CACHE BOOL "ENABLE_REMOTE_FUNCTIONALITY=NO => ENABLE_DAP4=NO" FORCE)
Provide a single option to disable all network access and testing. Add the option "--disable-network-access" (automake) or "-DENABLE_NETWORK_ACCESS=OFF" (cmake). When disabled, this option transitively disables all network access capabilities and testing. If set, this option implies the following: * --disable-dap * --disable-byterange * --disable-s3 This PR answers a request for a feature from Ed Hartnett. ## Misc. Other changes * Take the opportunity to clean up some old, unused options; e.g. --enable-multifilters. * Fix bug in using S3 urls.
2023-06-11 04:08:04 +08:00
ENDIF()
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
IF(ENABLE_HDF4)
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
SET(USE_HDF4 ON)
# Check for include files, libraries.
FIND_PATH(MFHDF_H_INCLUDE_DIR mfhdf.h)
IF(NOT MFHDF_H_INCLUDE_DIR)
MESSAGE(FATAL_ERROR "HDF4 Support specified, cannot find file mfhdf.h")
ELSE()
INCLUDE_DIRECTORIES(${MFHDF_H_INCLUDE_DIR})
ENDIF()
FIND_LIBRARY(HDF4_DF_LIB NAMES df libdf hdf)
IF(NOT HDF4_DF_LIB)
MESSAGE(FATAL_ERROR "Can't find or link to the hdf4 df library.")
ENDIF()
FIND_LIBRARY(HDF4_MFHDF_LIB NAMES mfhdf libmfhdf)
IF(NOT HDF4_MFHDF_LIB)
MESSAGE(FATAL_ERROR "Can't find or link to the hdf4 mfhdf library.")
ENDIF()
Attempting to fix an issue.
2017-02-23 00:55:13 +08:00
SET(HAVE_LIBMFHDF TRUE)
Added a new test for h4 endianness. [NCF332]
2015-05-29 07:13:15 +08:00
SET(HDF4_LIBRARIES ${HDF4_DF_LIB} ${HDF4_MFHDF_LIB})
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
# End include files, libraries.
MESSAGE(STATUS "HDF4 libraries: ${HDF4_DF_LIB}, ${HDF4_MFHDF_LIB}")
MESSAGE(STATUS "Seeking HDF4 jpeg dependency.")
# Look for the jpeglib.h header file.
FIND_PATH(JPEGLIB_H_INCLUDE_DIR jpeglib.h)
IF(NOT JPEGLIB_H_INCLUDE_DIR)
MESSAGE(FATAL_ERROR "HDF4 Support enabled but cannot find jpeglib.h")
ELSE()
SET(HAVE_JPEGLIB_H ON CACHE BOOL "")
Attempting to fix an issue.
2017-02-23 00:55:13 +08:00
SET(HAVE_LIBJPEG TRUE)
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
INCLUDE_DIRECTORIES(${JPEGLIB_H_INCLUDE_DIR})
ENDIF()
FIND_LIBRARY(JPEG_LIB NAMES jpeg libjpeg)
IF(NOT JPEG_LIB)
MESSAGE(FATAL_ERROR "HDF4 Support enabled but cannot find libjpeg")
ENDIF()
SET(HDF4_LIBRARIES ${JPEG_LIB} ${HDF4_LIBRARIES})
MESSAGE(STATUS "Found JPEG libraries: ${JPEG_LIB}")
# Option to enable HDF4 file tests.
Changed HDF4 test check from 'wget' to 'curl', to mirror change in test scripts.
2014-08-29 07:10:19 +08:00
OPTION(ENABLE_HDF4_FILE_TESTS "Run HDF4 file tests. This fetches sample HDF4 files from the Unidata ftp site to test with (requires curl)." ON)
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
IF(ENABLE_HDF4_FILE_TESTS)
Changed HDF4 test check from 'wget' to 'curl', to mirror change in test scripts.
2014-08-29 07:10:19 +08:00
FIND_PROGRAM(PROG_CURL NAMES curl)
IF(PROG_CURL)
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
SET(USE_HDF4_FILE_TESTS ON)
ELSE()
Changed HDF4 test check from 'wget' to 'curl', to mirror change in test scripts.
2014-08-29 07:10:19 +08:00
MESSAGE(STATUS "Unable to locate 'curl'. Disabling hdf4 file tests.")
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
SET(USE_HDF4_FILE_TESTS OFF)
ENDIF()
ENDIF()
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
ENDIF()
2012-10-03 04:56:46 +08:00
# Option to Build DLL
IF(WIN32)
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
OPTION(ENABLE_DLL "Build a Windows DLL." ${BUILD_SHARED_LIBS})
IF(ENABLE_DLL)
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
SET(BUILD_DLL ON CACHE BOOL "")
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
ADD_DEFINITIONS(-DDLL_NETCDF)
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
ADD_DEFINITIONS(-DDLL_EXPORT)
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
ENDIF()
2012-10-03 04:56:46 +08:00
ENDIF()
# Did the user specify a default minimum blocksize for posixio?
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
SET(NCIO_MINBLOCKSIZE 256 CACHE STRING "Minimum I/O Blocksize for netCDF classic and 64-bit offset format files.")
2012-10-03 04:56:46 +08:00
IF(ENABLE_NETCDF_4)
Infrastructure for generating a meta header file dynamically at configure time.
2014-09-05 07:20:15 +08:00
SET(USE_NETCDF4 ON CACHE BOOL "")
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
SET(ENABLE_NETCDF_4 ON CACHE BOOL "")
SET(ENABLE_NETCDF4 ON CACHE BOOL "")
attempting fix for cmake HDF4 build problem when netcdf-4 is not built
2018-02-09 23:26:58 +08:00
ELSE()
SET(USE_HDF4_FILE_TESTS OFF)
Updated ac, cmake based builds to require parallel functionality when parallel tests are requested.
2018-02-27 08:08:26 +08:00
SET(USE_HDF4 OFF)
attempting fix for cmake HDF4 build problem when netcdf-4 is not built
2018-02-09 23:26:58 +08:00
SET(ENABLE_HDF4_FILE_TESTS OFF)
Updated ac, cmake based builds to require parallel functionality when parallel tests are requested.
2018-02-27 08:08:26 +08:00
SET(ENABLE_HDF4 OFF)
2012-10-03 04:56:46 +08:00
ENDIF()
fixed classic cmake builds without logging
2018-05-13 23:48:20 +08:00
# Option Logging, only valid for netcdf4.
OPTION(ENABLE_LOGGING "Enable Logging." OFF)
re: github issues https://github.com/Unidata/netcdf-c/issues/1168 https://github.com/Unidata/netcdf-c/issues/1163 https://github.com/Unidata/netcdf-c/issues/1162 This PR partially fixes memory leaks in the netcdf-c library, in the ncdump utility, and in some test cases. The netcdf-c library now runs memory clean with the assumption that the --disable-utilities option is used. The primary remaining problem is ncgen. Once that is fixed, I believe the netcdf-c library will run memory clean with no limitations. Notes ----------- 1. Memory checking was performed using gcc -fsanitize=address. Valgrind-based testing has yet to be performed. 2. The pnetcdf, hdf4, and examples code has not been tested. Misc. Non-leak changes 1. Make tst_diskless2 only run when netcdf4 is enabled (issue 1162) 2. Fix CmakeLists.txt to turn off logging if ENABLE_NETCDF_4 is OFF 3. Isolated all my debug scripts into a single top-level directory called debug 4. Fix some USE_NETCDF4 dependencies in nc_test and nc_test4 Makefile.am
2018-10-31 10:48:12 +08:00
IF(NOT ENABLE_NETCDF_4)
SET(ENABLE_LOGGING OFF)
ENDIF()
fixed classic cmake builds without logging
2018-05-13 23:48:20 +08:00
IF(ENABLE_LOGGING)
ADD_DEFINITIONS(-DLOGGING)
ADD_DEFINITIONS(-DENABLE_SET_LOG_LEVEL)
SET(LOGGING ON)
attempting to fix ENABLE_SET_LOG_LEVEL problem with cmake build
2018-07-16 22:39:23 +08:00
SET(ENABLE_SET_LOG_LEVEL ON)
fixed classic cmake builds without logging
2018-05-13 23:48:20 +08:00
ENDIF()
OPTION(ENABLE_SET_LOG_LEVEL_FUNC "Enable definition of nc_set_log_level()." ON)
attempting to fix ENABLE_SET_LOG_LEVEL problem with cmake build
2018-07-17 01:45:26 +08:00
IF(ENABLE_NETCDF_4 AND NOT ENABLE_LOGGING AND ENABLE_SET_LOG_LEVEL_FUNC)
fixed classic cmake builds without logging
2018-05-13 23:48:20 +08:00
ADD_DEFINITIONS(-DENABLE_SET_LOG_LEVEL)
attempting to fix ENABLE_SET_LOG_LEVEL problem with cmake build
2018-07-16 20:39:15 +08:00
SET(ENABLE_SET_LOG_LEVEL ON)
fixed classic cmake builds without logging
2018-05-13 23:48:20 +08:00
ENDIF()
Wiring in a quick test.
2017-11-21 04:52:06 +08:00
# Option to allow for strict null file padding.
# See https://github.com/Unidata/netcdf-c/issues/657 for more information
OPTION(ENABLE_STRICT_NULL_BYTE_HEADER_PADDING "Enable strict null byte header padding." OFF)
Added expected failure to cmake-based tests.
2017-11-21 08:02:16 +08:00
Wiring in a quick test.
2017-11-21 04:52:06 +08:00
IF(ENABLE_STRICT_NULL_BYTE_HEADER_PADDING)
SET(USE_STRICT_NULL_BYTE_HEADER_PADDING ON CACHE BOOL "")
ENDIF(ENABLE_STRICT_NULL_BYTE_HEADER_PADDING)
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
# Note that szip management is tricky.
# This is because we have three things to consider:
# 1. is libsz available?
# 2. is szip enabled in HDF5?
# 3. is nczarr enabled?
# We need separate flags for cases 1 and 2
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
##
2012-10-03 04:56:46 +08:00
# Option to Enable HDF5
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
#
# The HDF5 cmake variables differ between platform (linux/osx and Windows),
# as well as between HDF5 versions. As a result, this section is a bit convoluted.
#
# Note that the behavior seems much more stable across HDF5 versions under linux,
# so we do not have to do as much version-based tweaking.
#
# At the end of it, we should have the following defined:
#
# * HDF5_C_LIBRARY
# * HDF5_HL_LIBRARY
# * HDF5_LIBRARIES
# * HDF5_INCLUDE_DIR
# *
##
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
SET(USE_HDF5 ${ENABLE_HDF5})
IF(USE_HDF5)
Added logic for checking the version when the HDF5 libraries and include directories are set explicitly.
2021-08-31 05:37:21 +08:00
##
# Assert HDF5 version meets minimum required version.
##
SET(HDF5_VERSION_REQUIRED 1.8.10)
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
##
Fix common typos Detected by https://github.com/vlajos/misspell_fixer
2015-08-20 17:42:05 +08:00
# Accommodate developers who have hdf5 libraries and
Merged latest from CMake branche. Fixed an error on very small files (< 4 bytes) that was only occuring on sol. Updated CMake configuration file.
2013-05-02 07:55:24 +08:00
# headers on their system, but do not have a the hdf
# .cmake files. If this is the case, they should
adding NC_HAS_SZIP_WRITE and NC_HAS_PAR_FILTERS to cmake build
2020-01-18 01:30:14 +08:00
# specify HDF5_HL_LIBRARY, HDF5_LIBRARY, HDF5_INCLUDE_DIR manually.
Added logic for checking the version when the HDF5 libraries and include directories are set explicitly.
2021-08-31 05:37:21 +08:00
#
Begin wiring in option to toggle automatic search for libxml2 in support of https://github.com/Unidata/netcdf-c/pulls/2135
2021-11-03 05:35:04 +08:00
# This script will attempt to determine the version of the HDF5 library programatically.
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
##
IF(HDF5_C_LIBRARY AND HDF5_HL_LIBRARY AND HDF5_INCLUDE_DIR)
SET(HDF5_LIBRARIES ${HDF5_C_LIBRARY} ${HDF5_HL_LIBRARY})
Support user-specified hdf5 and hdf5_hl libraries If the netcdf cmake build is configured with explicitly specified hdf5 c and hl libraries: ``` -DHDF5_C_LIBRARY:PATH=${ACCESS}/lib/libhdf5.dylib \ -DHDF5_HL_LIBRARY:PATH=${ACCESS}/lib/libhdf5_hl.dylib \ -DHDF5_INCLUDE_DIR:PATH=${ACCESS}/include \ ``` Then it looks like the HDF5_C_LIBRARIES and HDF5_HL_LIBRARIES variables do not get set and then this causes the HDF5 libraries to be omitted from the `TLL_LIBS` symbol in liblib/CMakeLists.txt. This is subsequently used to build `ALL_TLL_LIBS` which is used for `LIBS` which is output as the `Extra libraries: @LIBS@` in libnetcdf.settings.in The output for `-- Linking against:` is also incorrect as it omits the hdf5 libraries: Output without suggested change: ``` -- Linking against: /usr/lib/libdl.dylib;/usr/lib/libm.dylib;/opt/local/lib/libz.dylib;/opt/local/lib/libcurl.dylib (later in file) Extra libraries: -ldl -lm -lz -lcurl ``` versus output with suggested change: ``` -- Linking against: /Users/gdsjaar/src/seacas-parallel/lib/libhdf5_hl.dylib;/Users/gdsjaar/src/seacas-parallel/lib/libhdf5.dylib;/usr/lib/libdl.dylib;/usr/lib/libm.dylib;/opt/local/lib/libz.dylib;/opt/local/lib/libcurl.dylib (later in file) Extra libraries: -lhdf5_hl -lhdf5 -ldl -lm -lz -lcurl ``` This also causes a link failure when building libnetcdf.dylib
2016-02-09 08:20:48 +08:00
SET(HDF5_C_LIBRARIES ${HDF5_C_LIBRARY})
Corrected an issue detecting parallel hdf5 properly.
2017-03-07 06:34:14 +08:00
SET(HDF5_C_LIBRARY_hdf5 ${HDF5_C_LIBRARY})
Support user-specified hdf5 and hdf5_hl libraries If the netcdf cmake build is configured with explicitly specified hdf5 c and hl libraries: ``` -DHDF5_C_LIBRARY:PATH=${ACCESS}/lib/libhdf5.dylib \ -DHDF5_HL_LIBRARY:PATH=${ACCESS}/lib/libhdf5_hl.dylib \ -DHDF5_INCLUDE_DIR:PATH=${ACCESS}/include \ ``` Then it looks like the HDF5_C_LIBRARIES and HDF5_HL_LIBRARIES variables do not get set and then this causes the HDF5 libraries to be omitted from the `TLL_LIBS` symbol in liblib/CMakeLists.txt. This is subsequently used to build `ALL_TLL_LIBS` which is used for `LIBS` which is output as the `Extra libraries: @LIBS@` in libnetcdf.settings.in The output for `-- Linking against:` is also incorrect as it omits the hdf5 libraries: Output without suggested change: ``` -- Linking against: /usr/lib/libdl.dylib;/usr/lib/libm.dylib;/opt/local/lib/libz.dylib;/opt/local/lib/libcurl.dylib (later in file) Extra libraries: -ldl -lm -lz -lcurl ``` versus output with suggested change: ``` -- Linking against: /Users/gdsjaar/src/seacas-parallel/lib/libhdf5_hl.dylib;/Users/gdsjaar/src/seacas-parallel/lib/libhdf5.dylib;/usr/lib/libdl.dylib;/usr/lib/libm.dylib;/opt/local/lib/libz.dylib;/opt/local/lib/libcurl.dylib (later in file) Extra libraries: -lhdf5_hl -lhdf5 -ldl -lm -lz -lcurl ``` This also causes a link failure when building libnetcdf.dylib
2016-02-09 08:20:48 +08:00
SET(HDF5_HL_LIBRARIES ${HDF5_HL_LIBRARY})
Fix setting of HDF5 include directory path when explicitly specified The HDF5 include directory needs to be updated when explicitly setting the HDF5 libraries and include path. This is done on line 561 when FindPackage finds the hdf5 libraries, but was not being done in the manual setting.
2016-02-23 01:54:37 +08:00
INCLUDE_DIRECTORIES(${HDF5_INCLUDE_DIR})
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
MESSAGE(STATUS "Using HDF5 C Library: ${HDF5_C_LIBRARY}")
MESSAGE(STATUS "Using HDF5 HL LIbrary: ${HDF5_HL_LIBRARY}")
cmake: extract the HDF5 version when finding HDF5 manually
2021-08-17 01:49:09 +08:00
if (EXISTS "${HDF5_INCLUDE_DIR}/H5pubconf.h")
file(READ "${HDF5_INCLUDE_DIR}/H5pubconf.h" _hdf5_version_lines
REGEX "#define[ \t]+H5_VERSION")
string(REGEX REPLACE ".*H5_VERSION .*\"\(.*\)\".*" "\\1" _hdf5_version "${_hdf5_version_lines}")
Caching HDF5_VERSION string for diagnostic purposes.
2021-08-31 06:29:27 +08:00
set(HDF5_VERSION "${_hdf5_version}" CACHE STRING "")
cmake: extract the HDF5 version when finding HDF5 manually
2021-08-17 01:49:09 +08:00
unset(_hdf5_version)
unset(_hdf5_version_lines)
endif ()
Added logic for checking the version when the HDF5 libraries and include directories are set explicitly.
2021-08-31 05:37:21 +08:00
MESSAGE(STATUS "Found HDF5 libraries version ${HDF5_VERSION}")
###
# If HDF5_VERSION is still empty, we have a problem.
# Error out.
###
IF("${HDF5_VERSION}" STREQUAL "")
MESSAGE(FATAL_ERR "Unable to determine HDF5 version. NetCDF requires at least version ${HDF5_VERSION_REQUIRED}. Please ensure that libhdf5 is installed and accessible.")
ENDIF()
###
# Now that we know HDF5_VERSION isn't empty, we can check for minimum required version,
# and toggle various options.
###
IF(${HDF5_VERSION} VERSION_LESS ${HDF5_VERSION_REQUIRED})
MESSAGE(FATAL_ERROR "netCDF requires at least HDF5 ${HDF5_VERSION_REQUIRED}. Found ${HDF5_VERSION}.")
ENDIF()
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
ELSE(HDF5_C_LIBRARY AND HDF5_HL_LIBRARY AND HDF5_INCLUDE_DIR) # We are seeking out HDF5 with Find Package.
###
# For now we assume that if we are building netcdf
# as a shared library, we will use hdf5 as a shared
# library. If we are building netcdf statically,
# we will use a static library. This can be toggled
Misc. source comment typos Some are user-facing. Found via `codespell` and through the downstream FreeCAD.
2018-03-16 22:38:40 +08:00
# by explicitly modifying NC_FIND_SHARED_LIBS.
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
##
IF(NC_FIND_SHARED_LIBS)
Updated CMakeLists to explicitely look for static or shared hdf5 libraries. Required as part of hdf5 1.8.16, and is documented in https://github.com/Unidata/netcdf-c/issues/186
2016-01-08 05:11:44 +08:00
SET(NC_HDF5_LINK_TYPE "shared")
Work progresses on a CMake system that will work with hdf5 1.8.15 and 1.8.16, as well as that will work with Windows and Linux.
2016-01-08 06:43:17 +08:00
SET(NC_HDF5_LINK_TYPE_UPPER "SHARED")
ADD_DEFINITIONS(-DH5_BUILT_AS_DYNAMIC_LIB)
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
ELSE(NC_FIND_SHARED_LIBS)
Updated CMakeLists to explicitely look for static or shared hdf5 libraries. Required as part of hdf5 1.8.16, and is documented in https://github.com/Unidata/netcdf-c/issues/186
2016-01-08 05:11:44 +08:00
SET(NC_HDF5_LINK_TYPE "static")
Work progresses on a CMake system that will work with hdf5 1.8.15 and 1.8.16, as well as that will work with Windows and Linux.
2016-01-08 06:43:17 +08:00
SET(NC_HDF5_LINK_TYPE_UPPER "STATIC")
ADD_DEFINITIONS(-DH5_BUILT_AS_STATIC_LIB)
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
ENDIF(NC_FIND_SHARED_LIBS)
#####
# First, find the C and HL libraries.
#
# This has been updated to reflect what is in the hdf5
# examples, even though the previous version of what we
# had worked.
#####
IF(MSVC)
SET(SEARCH_PACKAGE_NAME ${HDF5_PACKAGE_NAME})
Enable HDF5 parallel support for MSVC; add MPI dependency.
2019-09-22 09:56:39 +08:00
FIND_PACKAGE(HDF5 NAMES ${SEARCH_PACKAGE_NAME} COMPONENTS C HL CONFIG REQUIRED ${NC_HDF5_LINK_TYPE})
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
ELSE(MSVC)
FIND_PACKAGE(HDF5 COMPONENTS C HL REQUIRED)
ENDIF(MSVC)
##
# Next, check the HDF5 version. This will inform which
# HDF5 variables we need to munge.
##
Revamping the HDF5_VERSION logic.
2021-08-30 23:10:14 +08:00
# Some versions of HDF5 set HDF5_VERSION_STRING instead of HDF5_VERSION
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
IF(HDF5_VERSION_STRING AND NOT HDF5_VERSION)
SET(HDF5_VERSION ${HDF5_VERSION_STRING})
add cmake option FIND_SHARED_LIBS Defaults to ${BUILD_SHARED_LIBS} and lets the user decide if static or shared versions of third-party libraries are searched. Only applied to HDF5 for now.
2014-08-21 16:25:00 +08:00
ENDIF()
Revamping the HDF5_VERSION logic.
2021-08-30 23:10:14 +08:00
###
# If HDF5_VERSION is undefined, attempt to determine it programatically.
###
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
IF("${HDF5_VERSION}" STREQUAL "")
Revamping the HDF5_VERSION logic.
2021-08-30 23:10:14 +08:00
MESSAGE(STATUS "HDF5_VERSION not detected. Attempting to determine programatically.")
IF (EXISTS "${HDF5_INCLUDE_DIR}/H5pubconf.h")
file(READ "${HDF5_INCLUDE_DIR}/H5pubconf.h" _hdf5_version_lines
REGEX "#define[ \t]+H5_VERSION")
string(REGEX REPLACE ".*H5_VERSION .*\"\(.*\)\".*" "\\1" _hdf5_version "${_hdf5_version_lines}")
Caching HDF5_VERSION string for diagnostic purposes.
2021-08-31 06:29:27 +08:00
set(HDF5_VERSION "${_hdf5_version}" CACHE STRING "")
Revamping the HDF5_VERSION logic.
2021-08-30 23:10:14 +08:00
unset(_hdf5_version)
unset(_hdf5_version_lines)
MESSAGE(STATUS "Found HDF5 libraries version ${HDF5_VERSION}")
ENDIF()
Caching HDF5_VERSION string for diagnostic purposes.
2021-08-31 06:29:27 +08:00
ELSE()
SET(HDF5_VERSION ${HDF5_VERSION} CACHE STRING "")
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
ENDIF()
Work progresses on a CMake system that will work with hdf5 1.8.15 and 1.8.16, as well as that will work with Windows and Linux.
2016-01-08 06:43:17 +08:00
Revamping the HDF5_VERSION logic.
2021-08-30 23:10:14 +08:00
###
# If HDF5_VERSION is still empty, we have a problem.
# Error out.
###
IF("${HDF5_VERSION}" STREQUAL "")
MESSAGE(FATAL_ERR "Unable to determine HDF5 version. NetCDF requires at least version ${HDF5_VERSION_REQUIRED}. Please ensure that libhdf5 is installed and accessible.")
ENDIF()
Plugged a hole where HDF5_VERSION wouldn't be checked against minimum HDF5 required version.
2021-08-31 04:21:37 +08:00
###
# Now that we know HDF5_VERSION isn't empty, we can check for minimum required version,
# and toggle various options.
###
IF(${HDF5_VERSION} VERSION_LESS ${HDF5_VERSION_REQUIRED})
MESSAGE(FATAL_ERROR "netCDF requires at least HDF5 ${HDF5_VERSION_REQUIRED}. Found ${HDF5_VERSION}.")
ENDIF()
Begin wiring in option to toggle automatic search for libxml2 in support of https://github.com/Unidata/netcdf-c/pulls/2135
2021-11-03 05:35:04 +08:00
Revamping the HDF5_VERSION logic.
2021-08-30 23:10:14 +08:00
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
##
# Include the HDF5 include directory.
##
Fix HDF5 include directory with CMake >= 3.6.0. HDF5_INCLUDE_DIR has been deprecated some time ago, only HDF5_INCLUDE_DIRS is set for CMake >= 3.6.0.
2016-08-28 21:26:35 +08:00
IF(HDF5_INCLUDE_DIRS AND NOT HDF5_INCLUDE_DIR)
SET(HDF5_INCLUDE_DIR ${HDF5_INCLUDE_DIRS})
ENDIF()
MESSAGE(STATUS "Using HDF5 include dir: ${HDF5_INCLUDE_DIR}")
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
INCLUDE_DIRECTORIES(${HDF5_INCLUDE_DIR})
Work progresses on a CMake system that will work with hdf5 1.8.15 and 1.8.16, as well as that will work with Windows and Linux.
2016-01-08 06:43:17 +08:00
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
###
# This is the block where we figure out what the appropriate
# variables are, and we ensure that we end up with
# HDF5_C_LIBRARY, HDF5_HL_LIBRARY and HDF5_LIBRARIES.
###
Merged latest changes from netcdf-cmake branch.
2013-06-21 04:38:23 +08:00
IF(MSVC)
Make sure HDF5_C_LIBRARY and HDF5_HL_LIBRARY set On some versions of the HDF5 find_package call, it sets `HDF5_C_LIBRARIES` and `HDF5_HL_LIBRARIES`, but does not set the `HDF5_C_LIBRARY` or `HDF5_HL_LIBRARY` to anything. Control then falls out of the if block with these unset and it falls into the default setting at line 792. This does not include the path, so then when the later `CHECK_LIBRARY_EXISTS` calls are run, they do not have the full path to the library and will not link correctly. Since the link fails, the code defaults to thinking that none of the symbols are defined. I don't think this change will have any affect since it only sets the symbols if they are unset.
2021-10-02 03:34:58 +08:00
####
# Environmental variables in Windows when using MSVC
# are a hot mess between versions.
####
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
##
# HDF5 1.8.15 defined HDF5_LIBRARIES.
##
IF(${HDF5_VERSION} VERSION_LESS "1.8.16")
Work progresses on a CMake system that will work with hdf5 1.8.15 and 1.8.16, as well as that will work with Windows and Linux.
2016-01-08 06:43:17 +08:00
SET(HDF5_C_LIBRARY hdf5)
Corrected an issue detecting parallel hdf5 properly.
2017-03-07 06:34:14 +08:00
SET(HDF5_C_LIBRARY_hdf5 hdf5)
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
ENDIF(${HDF5_VERSION} VERSION_LESS "1.8.16")
IF(${HDF5_VERSION} VERSION_GREATER "1.8.15")
IF(NOT HDF5_LIBRARIES AND HDF5_C_${NC_HDF5_LINK_TYPE_UPPER}_LIBRARY AND HDF5_HL_${NC_HDF5_LINK_TYPE_UPPER}_LIBRARY)
SET(HDF5_C_LIBRARY ${HDF5_C_${NC_HDF5_LINK_TYPE_UPPER}_LIBRARY})
Updated CMakeLists.txt to work on Windows.
2017-02-18 04:00:12 +08:00
SET(HDF5_C_LIBRARY_hdf5 ${HDF5_C_${NC_HDF5_LINK_TYPE_UPPER}_LIBRARY})
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
SET(HDF5_HL_LIBRARY ${HDF5_HL_${NC_HDF5_LINK_TYPE_UPPER}_LIBRARY})
More headache between discrepancies in CMake HDF5 variables. Updated to work on Windows as well as *nix.
2017-02-17 04:00:07 +08:00
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
SET(HDF5_LIBRARIES ${HDF5_C_${NC_HDF5_LINK_TYPE_UPPER}_LIBRARY} ${HDF5_HL_${NC_HDF5_LINK_TYPE_UPPER}_LIBRARY})
ENDIF()
ENDIF(${HDF5_VERSION} VERSION_GREATER "1.8.15")
ELSE(MSVC)
Begin wiring in option to toggle automatic search for libxml2 in support of https://github.com/Unidata/netcdf-c/pulls/2135
2021-11-03 05:35:04 +08:00
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
# Depending on the install, either HDF5_hdf_library or
# HDF5_C_LIBRARIES may be defined. We must check for either.
IF(HDF5_C_LIBRARIES AND NOT HDF5_hdf5_LIBRARY)
SET(HDF5_hdf5_LIBRARY ${HDF5_C_LIBRARIES})
Work progresses on a CMake system that will work with hdf5 1.8.15 and 1.8.16, as well as that will work with Windows and Linux.
2016-01-08 06:43:17 +08:00
ENDIF()
Begin wiring in option to toggle automatic search for libxml2 in support of https://github.com/Unidata/netcdf-c/pulls/2135
2021-11-03 05:35:04 +08:00
Make sure HDF5_C_LIBRARY and HDF5_HL_LIBRARY set On some versions of the HDF5 find_package call, it sets `HDF5_C_LIBRARIES` and `HDF5_HL_LIBRARIES`, but does not set the `HDF5_C_LIBRARY` or `HDF5_HL_LIBRARY` to anything. Control then falls out of the if block with these unset and it falls into the default setting at line 792. This does not include the path, so then when the later `CHECK_LIBRARY_EXISTS` calls are run, they do not have the full path to the library and will not link correctly. Since the link fails, the code defaults to thinking that none of the symbols are defined. I don't think this change will have any affect since it only sets the symbols if they are unset.
2021-10-02 03:34:58 +08:00
# Some versions of FIND_PACKAGE set HDF5_C_LIBRARIES, but not HDF5_C_LIBRARY
# We use HDF5_C_LIBRARY below, so need to make sure it is set.
IF(HDF5_C_LIBRARIES AND NOT HDF5_C_LIBRARY)
SET(HDF5_C_LIBRARY ${HDF5_C_LIBRARIES})
ENDIF()
# Same issue as above...
IF(HDF5_HL_LIBRARIES AND NOT HDF5_HL_LIBRARY)
SET(HDF5_HL_LIBRARY ${HDF5_HL_LIBRARIES})
ENDIF()
Work progresses on a CMake system that will work with hdf5 1.8.15 and 1.8.16, as well as that will work with Windows and Linux.
2016-01-08 06:43:17 +08:00
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
ENDIF(MSVC)
Get parallel hdf5 detection working
2016-12-01 00:22:44 +08:00
IF(NOT HDF5_C_LIBRARY)
SET(HDF5_C_LIBRARY hdf5)
ENDIF()
CMake: CMP0075 requires CMake >= 3.12. The presence of cmake_policy(CMP0075 required CMake >=3.12, and errors with older CMake. Since older versions of CMake than 3.12 are already broken with NetCDF and uncommon nowadays, it may be better to explicitly require 3.12 or newer.
2022-02-08 00:14:57 +08:00
Refactoring CMakeLists.txt to make the HDF5-related logic a little more clear. Or at least, better documented.
2016-01-09 01:56:25 +08:00
ENDIF(HDF5_C_LIBRARY AND HDF5_HL_LIBRARY AND HDF5_INCLUDE_DIR)
Corrected some typos.
2017-05-25 05:49:18 +08:00
Threads doesn't need to be required.
2018-05-16 02:19:33 +08:00
FIND_PACKAGE(Threads)
Added threads package to cmake build.
2018-04-25 06:15:26 +08:00
Fix pull request https://github.com/Unidata/netcdf-c/pull/374 (dap4.dmh) 1. Cleanup test_common.sh to expunge (mostly) the use of the VS path value. This has the effect of being unable to use the Visual Studio C compiler for shell tests. 2. There is a missing case in CMakeLists.txt so add defaulting for HDF5_C_LIBRARY_hdf5 using HDF5_C_LIBRARY. Ward should probably examine this to get it fixed correctly. 3. Put back ref to esg.md in docs/Doxyfile.in 4. Fix minor warning in dut8proc.h
2017-04-10 23:26:57 +08:00
# There is a missing case in the above code so default it
Fix name of the HDF5_C_LIBRARY_hdf5 CMake variable
2021-03-03 00:27:19 +08:00
IF(NOT HDF5_C_LIBRARY_hdf5 OR "${HDF5_C_LIBRARY_hdf5}" STREQUAL "" )
Fix pull request https://github.com/Unidata/netcdf-c/pull/374 (dap4.dmh) 1. Cleanup test_common.sh to expunge (mostly) the use of the VS path value. This has the effect of being unable to use the Visual Studio C compiler for shell tests. 2. There is a missing case in CMakeLists.txt so add defaulting for HDF5_C_LIBRARY_hdf5 using HDF5_C_LIBRARY. Ward should probably examine this to get it fixed correctly. 3. Put back ref to esg.md in docs/Doxyfile.in 4. Fix minor warning in dut8proc.h
2017-04-10 23:26:57 +08:00
SET(HDF5_C_LIBRARY_hdf5 "${HDF5_C_LIBRARY}")
ENDIF()
Work progresses, we need to handle shared and static hdf5 separately it turns out. See examples provided by hdf5 group.
2016-01-08 06:22:11 +08:00
Refactor Z library detection HDF5 can depend on the Z library (in fact required for netCDF). Moved the detection of whether hdf5 was built with zlib up before any other tests that may require linking of the hdf5 library to determine presence/absence of symbols. These tests require that the link line include "-lz" if the hdf5 library was built with libz support. This is typically handled somewhat automatically if shared libraries are being used, but in the static library case, the explicit dependency needs to be specified. For internal CMake checks, it uses the `CMAKE_REQUIRED_LIBRARIES` list to specify the libraries that should be used in a `CHECK_C_SOURCE_COMPILE` or a `CHECK_LIBRARY_EXISTS` call. In the current CMakeLists.txt ordering, the zlib detection is done _after_ the `CHECK_LIBRARY_EXISTS` calls which can cause them to fail and give an incorrect result about whether the function being tested for exists. With the reordering in this PR, I am able to correctly configure netCDF on a CRAY HPC system that uses static libraries by default.
2021-10-05 00:52:23 +08:00
FIND_PATH(HAVE_HDF5_H hdf5.h PATHS ${HDF5_INCLUDE_DIR} NO_DEFAULT_PATH)
IF(NOT HAVE_HDF5_H)
MESSAGE(FATAL_ERROR "Compiling a test with hdf5 failed. Either hdf5.h cannot be found, or the log messages should be checked for another reason.")
ELSE(NOT HAVE_HDF5_H)
INCLUDE_DIRECTORIES(${HAVE_HDF5_H})
ENDIF(NOT HAVE_HDF5_H)
Fix some HDF5-related tests Some of the tests for HDF5 symbols are failing due to not finding the include file which skews the test results. Added the `CMAKE_REQUIRED_INCLUDES` command to set the path. In one place this was already set, but used `HAVE_HDF5_H` which is not an include path and isn't even set at the location where it was used. We may not need these set more than once, but I added them before each `CHECK_C_SOURCE_COMPILES` in case those blocks get moved around and the value gets unset when it should be set. Also changed the `H5Literate_vers` check to check that it is defined instead of checking the value. As best I can see, as long as it is defined, then the `H5Literate` symbol will be defined which is what is needed. Checked with 1.10.7 and 1.12.0
2021-04-01 04:08:40 +08:00
set (CMAKE_REQUIRED_INCLUDES ${HDF5_INCLUDE_DIR})
Refactor Z library detection HDF5 can depend on the Z library (in fact required for netCDF). Moved the detection of whether hdf5 was built with zlib up before any other tests that may require linking of the hdf5 library to determine presence/absence of symbols. These tests require that the link line include "-lz" if the hdf5 library was built with libz support. This is typically handled somewhat automatically if shared libraries are being used, but in the static library case, the explicit dependency needs to be specified. For internal CMake checks, it uses the `CMAKE_REQUIRED_LIBRARIES` list to specify the libraries that should be used in a `CHECK_C_SOURCE_COMPILE` or a `CHECK_LIBRARY_EXISTS` call. In the current CMakeLists.txt ordering, the zlib detection is done _after_ the `CHECK_LIBRARY_EXISTS` calls which can cause them to fail and give an incorrect result about whether the function being tested for exists. With the reordering in this PR, I am able to correctly configure netCDF on a CRAY HPC system that uses static libraries by default.
2021-10-05 00:52:23 +08:00
# Check to ensure that HDF5 was built with zlib.
CMake: CMP0075 requires CMake >= 3.12. The presence of cmake_policy(CMP0075 required CMake >=3.12, and errors with older CMake. Since older versions of CMake than 3.12 are already broken with NetCDF and uncommon nowadays, it may be better to explicitly require 3.12 or newer.
2022-02-08 00:14:57 +08:00
# This needs to be near the beginning since we
# need to know whether to add "-lz" to the symbol
Refactor Z library detection HDF5 can depend on the Z library (in fact required for netCDF). Moved the detection of whether hdf5 was built with zlib up before any other tests that may require linking of the hdf5 library to determine presence/absence of symbols. These tests require that the link line include "-lz" if the hdf5 library was built with libz support. This is typically handled somewhat automatically if shared libraries are being used, but in the static library case, the explicit dependency needs to be specified. For internal CMake checks, it uses the `CMAKE_REQUIRED_LIBRARIES` list to specify the libraries that should be used in a `CHECK_C_SOURCE_COMPILE` or a `CHECK_LIBRARY_EXISTS` call. In the current CMakeLists.txt ordering, the zlib detection is done _after_ the `CHECK_LIBRARY_EXISTS` calls which can cause them to fail and give an incorrect result about whether the function being tested for exists. With the reordering in this PR, I am able to correctly configure netCDF on a CRAY HPC system that uses static libraries by default.
2021-10-05 00:52:23 +08:00
# tests below.
CHECK_C_SOURCE_COMPILES("#include <H5public.h>
#if !H5_HAVE_ZLIB_H
#error
#endif
int main() {
int x = 1;}" HAVE_HDF5_ZLIB)
IF(NOT HAVE_HDF5_ZLIB)
MESSAGE(FATAL_ERROR "HDF5 was built without zlib. Rebuild HDF5 with zlib.")
ELSE()
Better detection of libz library Instead of just specifying "z" as a dependent library, use `find_package`
2021-10-12 04:54:21 +08:00
# If user has specified the `ZLIB_LIBRARY`, use it; otherwise try to find...
IF(NOT ZLIB_LIBRARY)
Use consistent form for find_package Remove the double quotes I had used in the find_package...
2021-10-12 04:56:52 +08:00
find_package(ZLIB)
Better detection of libz library Instead of just specifying "z" as a dependent library, use `find_package`
2021-10-12 04:54:21 +08:00
IF(ZLIB_FOUND)
SET(ZLIB_LIBRARY ${ZLIB_LIBRARIES})
ELSE()
MESSAGE(FATAL_ERROR "HDF5 Requires ZLIB, but cannot find libz.")
ENDIF()
ENDIF()
SET(CMAKE_REQUIRED_LIBRARIES ${ZLIB_LIBRARY} ${CMAKE_REQUIRED_LIBRARIES})
MESSAGE(STATUS "HDF5 has zlib.")
Refactor Z library detection HDF5 can depend on the Z library (in fact required for netCDF). Moved the detection of whether hdf5 was built with zlib up before any other tests that may require linking of the hdf5 library to determine presence/absence of symbols. These tests require that the link line include "-lz" if the hdf5 library was built with libz support. This is typically handled somewhat automatically if shared libraries are being used, but in the static library case, the explicit dependency needs to be specified. For internal CMake checks, it uses the `CMAKE_REQUIRED_LIBRARIES` list to specify the libraries that should be used in a `CHECK_C_SOURCE_COMPILE` or a `CHECK_LIBRARY_EXISTS` call. In the current CMakeLists.txt ordering, the zlib detection is done _after_ the `CHECK_LIBRARY_EXISTS` calls which can cause them to fail and give an incorrect result about whether the function being tested for exists. With the reordering in this PR, I am able to correctly configure netCDF on a CRAY HPC system that uses static libraries by default.
2021-10-05 00:52:23 +08:00
ENDIF()
#Check to see if H5Z_SZIP exists in HDF5_Libraries. If so, we must use szip library.
Fix detection and use of SZIP library Potential fix for #1915
2020-12-21 23:25:37 +08:00
CHECK_C_SOURCE_COMPILES("#include <H5public.h>
#if !H5_HAVE_FILTER_SZIP
#error
#endif
int main() {
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
int x = 1;}" USE_HDF5_SZIP)
IF(USE_HDF5_SZIP)
SET(HAVE_H5Z_SZIP yes)
Fix detection and use of SZIP library Potential fix for #1915
2020-12-21 23:25:37 +08:00
ENDIF()
Updated SO version in cmakelists.txt in preparation for the 4.8.0 release.
2021-03-20 01:30:38 +08:00
Caching HDF5_VERSION string for diagnostic purposes.
2021-08-31 06:29:27 +08:00
####
# Check to see if HDF5 library is 1.10.6 or greater.
# Used to control path name conversion
####
IF(${HDF5_VERSION} VERSION_GREATER "1.10.5")
SET(HDF5_UTF8_PATHS ON)
ELSE()
SET(HDF5_UTF8_PATHS OFF)
ENDIF()
MESSAGE("-- HDF5_UTF8_PATHS (HDF5 version 1.10.6+): ${HDF5_UTF8_PATHS}")
Enable HDF5 parallel support for MSVC; add MPI dependency.
2019-09-22 09:56:39 +08:00
# Find out if HDF5 was built with parallel support.
# Do that by checking for the targets H5Pget_fapl_mpiposx and
# H5Pget_fapl_mpio in ${HDF5_LIB}.
Updated hdf5 variable in CmakeLists file.
2017-02-04 02:04:05 +08:00
Enable HDF5 parallel support for MSVC; add MPI dependency.
2019-09-22 09:56:39 +08:00
# H5Pset_fapl_mpiposix and H5Pget_fapl_mpiposix have been removed since HDF5 1.8.12.
# Use H5Pset_fapl_mpio and H5Pget_fapl_mpio, instead.
# CHECK_LIBRARY_EXISTS(${HDF5_C_LIBRARY_hdf5} H5Pget_fapl_mpiposix "" HDF5_IS_PARALLEL_MPIPOSIX)
Ignore flags NC_MPIIO and NC_MPIPOSIX.
2018-09-23 09:22:34 +08:00
Enable HDF5 parallel support for MSVC; add MPI dependency.
2019-09-22 09:56:39 +08:00
CHECK_LIBRARY_EXISTS(${HDF5_C_LIBRARY_hdf5} H5Pget_fapl_mpio "" HDF5_IS_PARALLEL_MPIO)
IF(HDF5_IS_PARALLEL_MPIO)
SET(HDF5_PARALLEL ON)
ELSE()
SET(HDF5_PARALLEL OFF)
ENDIF()
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
Enable HDF5 parallel support for MSVC; add MPI dependency.
2019-09-22 09:56:39 +08:00
#Check to see if HDF5 library has collective metadata APIs, (HDF5 >= 1.10.0)
CHECK_LIBRARY_EXISTS(${HDF5_C_LIBRARY_hdf5} H5Pset_all_coll_metadata_ops "" HDF5_HAS_COLL_METADATA_OPS)
Progress working towards [NCF-329]
2015-04-22 03:52:43 +08:00
Github issue 152: https://github.com/Unidata/netcdf-c/issues/152 User request to have all orphaned DAP2 attributes kept as netcdf global attributes. This is primarily a change in the oc code nplus testcase dataset changes. Result may be inconsistent with netcdf-Java output.
2015-11-30 11:38:12 +08:00
IF(HDF5_PARALLEL)
The call to h5cc (used to determine the hdf5 version) was not working properly on OSX or Ubuntu 14.04. Replaced with a similar call. Also modified the call to use h5pcc in the event we are on a parallel system.
2014-10-10 06:00:01 +08:00
SET(HDF5_CC h5pcc)
ELSE()
SET(HDF5_CC h5cc)
ENDIF()
handle missing H5Dread_chunks
2020-05-15 02:59:00 +08:00
# Check to see if H5Dread_chunk is available
CHECK_LIBRARY_EXISTS(${HDF5_C_LIBRARY_hdf5} H5Dread_chunk "" HAS_READCHUNKS)
Use the built-in HDF5 byte-range reader, if available. re: Issue https://github.com/Unidata/netcdf-c/issues/1848 The existing Virtual File Driver built to support byte-range read-only file access is quite old. It turns out to be extremely slow (reason unknown at the moment). Starting with HDF5 1.10.6, the HDF5 library has its own version of such a file driver. The HDF5 developers have better knowledge about building such a driver and what incantations are needed to get good performance. This PR modifies the byte-range code in hdf5open.c so that if the HDF5 file driver is available, then it is used in preference to the one written by the Netcdf group. Misc. Other Changes: 1. Moved all of nc4print code to ncdump to keep appveyor quiet.
2020-09-25 04:33:58 +08:00
# Check to see if H5Pset_fapl_ros3 is available
CHECK_LIBRARY_EXISTS(${HDF5_C_LIBRARY_hdf5} H5Pset_fapl_ros3 "" HAS_HDF5_ROS3)
switch cmake build to check for func H5Dread_chunk to detect 1.10.3 or later
2020-01-21 22:42:14 +08:00
# Check to see if this is hdf5-1.10.3 or later.
handle missing H5Dread_chunks
2020-05-15 02:59:00 +08:00
IF(HAS_READCHUNKS)
SET(HDF5_SUPPORTS_PAR_FILTERS ON)
SET(ENABLE_NCDUMPCHUNKS ON)
ENDIF()
Improve the building of NCZARR S3 support in CMake and Autoconf There were some irregularities in the flags for handling NCZarr S3 support. The primary change is to regularize the flags controlling this to the following. 1. Automake: --enable-nczarr-s3 and CMake: ENABLE_NCZARR_S3 2. Automake: --enable-nczarr-s3-tests and CMake: ENABLE_NCZARR_S3_TESTS Flag 1 indicates that NCZarr should be built with S3 support enabled. Flag 2 indicates that the NCZarr S3 tests should be run These two flags are separate because running the NCZarr S3 tests requires access to protected S3 resources. Currently, running these tests is restricted to Unidata personnel. However, users may want to enable S3 support even if they cannot run the tests. It is, of course, an error to specify 2 without specifying 1. Additionally, if the AWS S3 SDK library is not found, then the NCZARR S3 support and testing must be disabled. Otherwise an error is signaled during the build. Some of these NCZarr and S3 changes are propagated to nc-config. Misc. Other Changes: 1. Allow testing for CYGWIN or MSVC in shell scripts. 2. Add specific test for HDF5 library version 1.10.6. This is encoded as "HDF5_UTF8_PATHS" because that is the first version where HDF5 properly supports it under Windows. This is used in hdf5internal/nc4_ndf5_ansi_to_utf8. 3. Add a AM Conditional -- AX_IGNORE -- for use in testing when it is desirable to temporarily suppress Makefile code. 4. Add MULTIFILTER flag to CMakeLists.txt
2020-10-17 05:04:51 +08:00
# Record if ROS3 Driver is available
Use the built-in HDF5 byte-range reader, if available. re: Issue https://github.com/Unidata/netcdf-c/issues/1848 The existing Virtual File Driver built to support byte-range read-only file access is quite old. It turns out to be extremely slow (reason unknown at the moment). Starting with HDF5 1.10.6, the HDF5 library has its own version of such a file driver. The HDF5 developers have better knowledge about building such a driver and what incantations are needed to get good performance. This PR modifies the byte-range code in hdf5open.c so that if the HDF5 file driver is available, then it is used in preference to the one written by the Netcdf group. Misc. Other Changes: 1. Moved all of nc4print code to ncdump to keep appveyor quiet.
2020-09-25 04:33:58 +08:00
IF(HAS_HDF5_ROS3)
SET(ENABLE_HDF5_ROS3 ON)
ENDIF()
Different method of setting Parallel Filters variables The current method of setting the `HDF5_HAS_PAR_FILTERS` and `HAS_PAR_FILTERS` variables is done purely based on the `HDF5_VERSION` and that variable is only set inside the if block which finds the HDF5 library based on CMake package files. If the user specifies the explicit location of the HDF5 library and include files (for example, via: ``` -DHDF5_C_LIBRARY:PATH=${INSTALL_PATH}/lib/libhdf5.${LD_EXT} \ -DHDF5_HL_LIBRARY:PATH=${INSTALL_PATH}/lib/libhdf5_hl.${LD_EXT} \ -DHDF5_INCLUDE_DIR:PATH=${INSTALL_PATH}/include ``` Then, the code path which determines whether the par filters variables is set is not run. However, later on in the file, there is another check for parallel filter support (near line 759): ``` # Check to see if this is hdf5-1.10.3 or later. CHECK_LIBRARY_EXISTS(${HDF5_C_LIBRARY_hdf5} H5Dread_chunk "" HDF5_SUPPORTS_PAR_FILTERS) ``` This PR moves the code that sets the two other par filters variables down after this check and instead of setting their values based on the version, it bases it on the results of this test. I'm not totally sure why there are three variables; it looks like the `HDF5_SUPPORTS_PAR_FILTERS` and `HDF5_HAS_PAR_FILTERS` could be combined. I think the `HAS_PAR_FILTERS` is a string which is used to show the results of the configuration and the other two are booleans. The new check should work for both types of HDF5 installs (cmake-based and configure-based)
2020-05-12 01:11:24 +08:00
IF (HDF5_SUPPORTS_PAR_FILTERS)
SET(HDF5_HAS_PAR_FILTERS TRUE CACHE BOOL "")
SET(HAS_PAR_FILTERS yes CACHE STRING "")
ELSE()
SET(HDF5_HAS_PAR_FILTERS FALSE CACHE BOOL "")
SET(HAS_PAR_FILTERS no CACHE STRING "")
ENDIF()
getting parallel I/O with zlib working in cmake builds
2019-12-21 00:43:56 +08:00
Better finding of the hdf5.h include path The `FIND_PATH` function has a non-intuitive order in which is searches for include paths and the original form could find a system-installed version of HDF5 even though the developer specified an explicit path via `HDF5_INCLUDE_DIR`. See https://cmake.org/cmake/help/latest/command/find_path.html for more information. The form in this patch is the recommended method from that page for changing the default search order to find the explicitly specified path *first* > The default search order is designed to be most-specific to least-specific for common use cases. Projects may override the order by simply calling the command multiple times and using the NO_* options: ``` find_path (<VAR> NAMES name PATHS paths... NO_DEFAULT_PATH) find_path (<VAR> NAMES name) ``` > Once one of the calls succeeds the result variable will be set and stored in the cache so that no call will search again.
2020-03-06 05:44:38 +08:00
FIND_PATH(HAVE_HDF5_H hdf5.h PATHS ${HDF5_INCLUDE_DIR} NO_DEFAULT_PATH)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
IF(NOT HAVE_HDF5_H)
MESSAGE(FATAL_ERROR "Compiling a test with hdf5 failed. Either hdf5.h cannot be found, or the log messages should be checked for another reason.")
ELSE(NOT HAVE_HDF5_H)
INCLUDE_DIRECTORIES(${HAVE_HDF5_H})
ENDIF(NOT HAVE_HDF5_H)
Fix syntax for NC_EXTRA_DEPS
2018-02-10 03:57:11 +08:00
#option to include HDF5 High Level header file (hdf5_hl.h) in case we are not doing a make install
changes to detect HDF5 HL, build with CRT and detect ZLIB in HDF5
2018-02-09 02:45:29 +08:00
INCLUDE_DIRECTORIES(${HDF5_HL_INCLUDE_DIR})
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
ENDIF(USE_HDF5)
Progress working towards [NCF-329]
2015-04-22 03:52:43 +08:00
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
# See if we have libcurl
FIND_PACKAGE(CURL)
ADD_DEFINITIONS(-DCURL_STATICLIB=1)
INCLUDE_DIRECTORIES(${CURL_INCLUDE_DIRS})
Fix CMake bug
2020-12-31 04:30:12 +08:00
# Define a test flag for have curl library
IF(CURL_LIBRARIES OR CURL_LIBRARY)
SET(FOUND_CURL TRUE)
ELSE()
SET(FOUND_CURL FALSE)
ENDIF()
Provide a single option to disable all network access and testing. Add the option "--disable-network-access" (automake) or "-DENABLE_NETWORK_ACCESS=OFF" (cmake). When disabled, this option transitively disables all network access capabilities and testing. If set, this option implies the following: * --disable-dap * --disable-byterange * --disable-s3 This PR answers a request for a feature from Ed Hartnett. ## Misc. Other changes * Take the opportunity to clean up some old, unused options; e.g. --enable-multifilters. * Fix bug in using S3 urls.
2023-06-11 04:08:04 +08:00
# Start disabling if curl not found
IF(NOT FOUND_CURL)
Change the network access nomenclature to be less ambiguous. As it stood, did it refer to networked drives? Other things that are network adjacent?
2023-06-13 05:23:44 +08:00
MESSAGE(WARNING "ENABLE_REMOTE_FUNCTIONALITY requires libcurl; disabling")
SET(ENABLE_REMOTE_FUNCTIONALITY OFF CACHE BOOL "ENABLE_REMOTE_FUNCTIONALITY requires libcurl; disabling" FORCE)
Provide a single option to disable all network access and testing. Add the option "--disable-network-access" (automake) or "-DENABLE_NETWORK_ACCESS=OFF" (cmake). When disabled, this option transitively disables all network access capabilities and testing. If set, this option implies the following: * --disable-dap * --disable-byterange * --disable-s3 This PR answers a request for a feature from Ed Hartnett. ## Misc. Other changes * Take the opportunity to clean up some old, unused options; e.g. --enable-multifilters. * Fix bug in using S3 urls.
2023-06-11 04:08:04 +08:00
ENDIF()
Clean up curl symbol tests * Set `CMAKE_REQUIRED_INCLUDES` to include the path found for `curl.h`. The `CHECK_C_SOURCE_COMPILES` function uses this and not the `INCLUDE_DIRECTORIES` * Make the test for version 7.66 or later match the same test in `configure.ac` * If the version is 7.66 or later, then we can skip the tests for the curl symbols which were all added in versions prior to 7.66. * If the version is earlier than 7.66, then continue to perform the tests.
2021-06-03 02:54:05 +08:00
set (CMAKE_REQUIRED_INCLUDES ${CURL_INCLUDE_DIRS})
# Check to see if we have libcurl 7.66 or later
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
CHECK_C_SOURCE_COMPILES("
#include <curl/curl.h>
Clean up curl symbol tests * Set `CMAKE_REQUIRED_INCLUDES` to include the path found for `curl.h`. The `CHECK_C_SOURCE_COMPILES` function uses this and not the `INCLUDE_DIRECTORIES` * Make the test for version 7.66 or later match the same test in `configure.ac` * If the version is 7.66 or later, then we can skip the tests for the curl symbols which were all added in versions prior to 7.66. * If the version is earlier than 7.66, then continue to perform the tests.
2021-06-03 02:54:05 +08:00
int main() {
#if LIBCURL_VERSION_NUM < 0x074200
choke me;
#endif
}" HAVE_LIBCURL_766)
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
Clean up curl symbol tests * Set `CMAKE_REQUIRED_INCLUDES` to include the path found for `curl.h`. The `CHECK_C_SOURCE_COMPILES` function uses this and not the `INCLUDE_DIRECTORIES` * Make the test for version 7.66 or later match the same test in `configure.ac` * If the version is 7.66 or later, then we can skip the tests for the curl symbols which were all added in versions prior to 7.66. * If the version is earlier than 7.66, then continue to perform the tests.
2021-06-03 02:54:05 +08:00
IF (HAVE_LIBCURL_766)
Begin wiring in option to toggle automatic search for libxml2 in support of https://github.com/Unidata/netcdf-c/pulls/2135
2021-11-03 05:35:04 +08:00
# If libcurl version is >= 7.66, then can skip tests
Clean up curl symbol tests * Set `CMAKE_REQUIRED_INCLUDES` to include the path found for `curl.h`. The `CHECK_C_SOURCE_COMPILES` function uses this and not the `INCLUDE_DIRECTORIES` * Make the test for version 7.66 or later match the same test in `configure.ac` * If the version is 7.66 or later, then we can skip the tests for the curl symbols which were all added in versions prior to 7.66. * If the version is earlier than 7.66, then continue to perform the tests.
2021-06-03 02:54:05 +08:00
# for these symbols which were added in an earlier version
set(HAVE_CURLOPT_USERNAME TRUE)
set(HAVE_CURLOPT_PASSWORD TRUE)
set(HAVE_CURLOPT_KEYPASSWD TRUE)
set(HAVE_CURLINFO_RESPONSE_CODE TRUE)
set(HAVE_CURLINFO_HTTP_CONNECTCODE TRUE)
set(HAVE_CURLOPT_BUFFERSIZE TRUE)
set(HAVE_CURLOPT_KEEPALIVE TRUE)
ELSE()
# Check to see if CURLOPT_USERNAME is defined.
# It is present starting version 7.19.1.
CHECK_C_SOURCE_COMPILES("
#include <curl/curl.h>
int main() {int x = CURLOPT_USERNAME;}" HAVE_CURLOPT_USERNAME)
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
Clean up curl symbol tests * Set `CMAKE_REQUIRED_INCLUDES` to include the path found for `curl.h`. The `CHECK_C_SOURCE_COMPILES` function uses this and not the `INCLUDE_DIRECTORIES` * Make the test for version 7.66 or later match the same test in `configure.ac` * If the version is 7.66 or later, then we can skip the tests for the curl symbols which were all added in versions prior to 7.66. * If the version is earlier than 7.66, then continue to perform the tests.
2021-06-03 02:54:05 +08:00
# Check to see if CURLOPT_PASSWORD is defined.
# It is present starting version 7.19.1.
CHECK_C_SOURCE_COMPILES("
#include <curl/curl.h>
int main() {int x = CURLOPT_PASSWORD;}" HAVE_CURLOPT_PASSWORD)
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
Clean up curl symbol tests * Set `CMAKE_REQUIRED_INCLUDES` to include the path found for `curl.h`. The `CHECK_C_SOURCE_COMPILES` function uses this and not the `INCLUDE_DIRECTORIES` * Make the test for version 7.66 or later match the same test in `configure.ac` * If the version is 7.66 or later, then we can skip the tests for the curl symbols which were all added in versions prior to 7.66. * If the version is earlier than 7.66, then continue to perform the tests.
2021-06-03 02:54:05 +08:00
# Check to see if CURLOPT_KEYPASSWD is defined.
# It is present starting version 7.16.4.
CHECK_C_SOURCE_COMPILES("
#include <curl/curl.h>
int main() {int x = CURLOPT_KEYPASSWD;}" HAVE_CURLOPT_KEYPASSWD)
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
Clean up curl symbol tests * Set `CMAKE_REQUIRED_INCLUDES` to include the path found for `curl.h`. The `CHECK_C_SOURCE_COMPILES` function uses this and not the `INCLUDE_DIRECTORIES` * Make the test for version 7.66 or later match the same test in `configure.ac` * If the version is 7.66 or later, then we can skip the tests for the curl symbols which were all added in versions prior to 7.66. * If the version is earlier than 7.66, then continue to perform the tests.
2021-06-03 02:54:05 +08:00
# Check to see if CURLINFO_RESPONSE_CODE is defined.
# It showed up in curl 7.10.7.
CHECK_C_SOURCE_COMPILES("
#include <curl/curl.h>
int main() {int x = CURLINFO_RESPONSE_CODE;}" HAVE_CURLINFO_RESPONSE_CODE)
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
Clean up curl symbol tests * Set `CMAKE_REQUIRED_INCLUDES` to include the path found for `curl.h`. The `CHECK_C_SOURCE_COMPILES` function uses this and not the `INCLUDE_DIRECTORIES` * Make the test for version 7.66 or later match the same test in `configure.ac` * If the version is 7.66 or later, then we can skip the tests for the curl symbols which were all added in versions prior to 7.66. * If the version is earlier than 7.66, then continue to perform the tests.
2021-06-03 02:54:05 +08:00
# Check to see if CURLINFO_HTTP_CONNECTCODE is defined.
# It showed up in curl 7.10.7.
CHECK_C_SOURCE_COMPILES("
#include <curl/curl.h>
int main() {int x = CURLINFO_HTTP_CONNECTCODE;}" HAVE_CURLINFO_HTTP_CONNECTCODE)
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
Clean up curl symbol tests * Set `CMAKE_REQUIRED_INCLUDES` to include the path found for `curl.h`. The `CHECK_C_SOURCE_COMPILES` function uses this and not the `INCLUDE_DIRECTORIES` * Make the test for version 7.66 or later match the same test in `configure.ac` * If the version is 7.66 or later, then we can skip the tests for the curl symbols which were all added in versions prior to 7.66. * If the version is earlier than 7.66, then continue to perform the tests.
2021-06-03 02:54:05 +08:00
# Check to see if CURLOPT_BUFFERSIZE is defined.
# It is present starting version 7.59
CHECK_C_SOURCE_COMPILES("
#include <curl/curl.h>
int main() {int x = CURLOPT_BUFFERSIZE;}" HAVE_CURLOPT_BUFFERSIZE)
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
Clean up curl symbol tests * Set `CMAKE_REQUIRED_INCLUDES` to include the path found for `curl.h`. The `CHECK_C_SOURCE_COMPILES` function uses this and not the `INCLUDE_DIRECTORIES` * Make the test for version 7.66 or later match the same test in `configure.ac` * If the version is 7.66 or later, then we can skip the tests for the curl symbols which were all added in versions prior to 7.66. * If the version is earlier than 7.66, then continue to perform the tests.
2021-06-03 02:54:05 +08:00
# Check to see if CURLOPT_TCP_KEEPALIVE is defined.
# It is present starting version 7.25
CHECK_C_SOURCE_COMPILES("
#include <curl/curl.h>
int main() {int x = CURLOPT_TCP_KEEPALIVE;}" HAVE_CURLOPT_KEEPALIVE)
ENDIF()
Use proper CURLOPT values for VERIFYHOST and VERIFYPEER re: https://github.com/Unidata/netcdf-c/issues/1684 re: e-support VZL-904142 Two issues: 1. As of libcurl 7.66, the semantics of CURLOPT_SSL_VERIFYHOST changed so that the non-zero values affects certificate processing. 2. The current library was forcing the values of VERIFYPEER and VERIFYHOST to zero instead of leaving them to the default values. Solution was first to leave the defaults in place for VERIFYPEER and VERIFYHOST as long as they are not set in .ocrc/.dodsrc file. Second, the value of HTTP.SSL.VERIFYPEER or HTTP.SSL.VERIFYHOST as set in .ocrc/.dodrc is used to set the corresponding CURLOPT flags. So for example, adding > HTTP.SSL.VERIFYHOST=2 will set the value of CURLOPT_SSL_VERIFYHOST to 2, the default. Using > HTTP.SSL.VERIFYHOST=0 will set the value of CURLOPT_SSL_VERIFYHOST to 0, which disables it. Similarly for VERIFYPEER. Finally the semantics of HTTP.SSL.VALIDATE is now equivalent to > HTTP.SSL.VERIFYPEER=1 > HTTP.SSL.VERIFYHOST=2
2020-04-11 03:42:27 +08:00
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
IF(ENABLE_DAP)
Corrected DAP4 and NC4 logic
2017-09-27 05:26:34 +08:00
SET(USE_DAP ON CACHE BOOL "")
SET(ENABLE_DAP2 ON CACHE BOOL "")
Additional DAP4 fixes This change-set modifies PR https://github.com/Unidata/netcdf-c/pull/2555 to add the changes listed below. Most of these changes are required by changes to the Java remotetest.unidata.ucar.edu server. ## DAP4 Related Changes * Add tests *dap4_test/test_constraints.sh* and *dap4_test/test_hyrax.sh*. * Provide explicit list of remotetest files to test. * Cleanup local checksum computing and verification. * Define a temporary Hyrax hack flag to deal with the way Hyrax handles checksums and add "#hyrax" fragment flag for it. * Add a hack to get past an LGTM problem with using "http:". * Improve debug support. ## Other Changes * Cleanup the recipe in *docs/nczarr.md* for building *aws-sdk-cpp* library.
2023-01-19 10:47:29 +08:00
Cleanup DAP4 testing NOTE: This PR should not be included in 4.9.1 since additional DAP4 related PRs will be forthcoming. This PR makes major changes to libdap4 and dap4_test driven by changes to TDS. * Enable DAP4 * Clean up the test input files and the test baseline comparison files. This entails: * Remove a multitude of unused test input and baseline data files; among them are dap4_test/: daptestfiles, dmrtestfiles, nctestfiles, and misctestfiles. * Define a canonical set of test input files and record in dap4_test/cdltestfiles. * Use the cdltestfiles to generate the .nc test inputs. This set of .nc files is then moved to the d4ts (DAP4 test server) war file in the tds repository. This set then becomes the canonical set of DAP4 test sources. * Scrape d4ts to obtain copies of the raw streams of DAP4 encoded data. The .dmr and .dap streams are then stored in dap4_test/rawtestfiles. * Disable some remote server tests until those servers are fixed. * Add an option to ncdump (-XF) that forces the type of the _FillValue attribute; this is primarily to simplify testing of fill mismatch. * Minor bug fixes to ncgen. * Changes to libdap4: * Replace old checksum hack with the dap4.checksum flag. * Support the dap4.XXX controls. * Cleanup _FillValue handling, especially var-attribute type mismatches. * Fix enum handling based on changes to netcdf-java. * Changes to dap4_test: * Add getopt support to various test support programs. * Remove unneeded shell scripts. * Add new scripts: test_curlopt.sh
2022-11-14 04:15:11 +08:00
IF(ENABLE_HDF5)
Correct logic for running DAP4 tests when HDF5 is not enabled.
2023-01-10 08:40:51 +08:00
MESSAGE(STATUS "Enabling DAP4")
Cleanup DAP4 testing NOTE: This PR should not be included in 4.9.1 since additional DAP4 related PRs will be forthcoming. This PR makes major changes to libdap4 and dap4_test driven by changes to TDS. * Enable DAP4 * Clean up the test input files and the test baseline comparison files. This entails: * Remove a multitude of unused test input and baseline data files; among them are dap4_test/: daptestfiles, dmrtestfiles, nctestfiles, and misctestfiles. * Define a canonical set of test input files and record in dap4_test/cdltestfiles. * Use the cdltestfiles to generate the .nc test inputs. This set of .nc files is then moved to the d4ts (DAP4 test server) war file in the tds repository. This set then becomes the canonical set of DAP4 test sources. * Scrape d4ts to obtain copies of the raw streams of DAP4 encoded data. The .dmr and .dap streams are then stored in dap4_test/rawtestfiles. * Disable some remote server tests until those servers are fixed. * Add an option to ncdump (-XF) that forces the type of the _FillValue attribute; this is primarily to simplify testing of fill mismatch. * Minor bug fixes to ncgen. * Changes to libdap4: * Replace old checksum hack with the dap4.checksum flag. * Support the dap4.XXX controls. * Cleanup _FillValue handling, especially var-attribute type mismatches. * Fix enum handling based on changes to netcdf-java. * Changes to dap4_test: * Add getopt support to various test support programs. * Remove unneeded shell scripts. * Add new scripts: test_curlopt.sh
2022-11-14 04:15:11 +08:00
SET(ENABLE_DAP4 ON CACHE BOOL "")
Correct logic for running DAP4 tests when HDF5 is not enabled.
2023-01-10 08:40:51 +08:00
ELSE()
MESSAGE(STATUS "Disabling DAP4")
Fix race condition in ncdump (and other) tests. re: Issue https://github.com/Unidata/netcdf-c/issues/2551 Ryan May identified the use of a common scratch file (tmp.cdl) across multiple test shell scripts in ncdump directory and the nczarr_test directory. This sometimes causes errors because of race conditions between those scripts. I renamed those common files to avoid the race condition. I also did some further checking and found some additional, similar conflicts and fixed those. Also did some minor cleanup of unused files. Tests fixed: ncdump: run_back_comp_tests.sh tst_bom.sh tst_nccopy4.sh tst_nccopy5.sh nczarr_test: git df master -- run_nccopyz.sh run_nczarr_fill.sh run_scalar.sh
2022-11-09 11:12:38 +08:00
SET(ENABLE_DAP4 OFF CACHE BOOL "")
Fix a mismatched if,endif block label.
2023-04-13 06:07:57 +08:00
ENDIF(ENABLE_HDF5)
Corrected DAP4 and NC4 logic
2017-09-27 05:26:34 +08:00
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
ELSE()
Fix race condition in ncdump (and other) tests. re: Issue https://github.com/Unidata/netcdf-c/issues/2551 Ryan May identified the use of a common scratch file (tmp.cdl) across multiple test shell scripts in ncdump directory and the nczarr_test directory. This sometimes causes errors because of race conditions between those scripts. I renamed those common files to avoid the race condition. I also did some further checking and found some additional, similar conflicts and fixed those. Also did some minor cleanup of unused files. Tests fixed: ncdump: run_back_comp_tests.sh tst_bom.sh tst_nccopy4.sh tst_nccopy5.sh nczarr_test: git df master -- run_nccopyz.sh run_nczarr_fill.sh run_scalar.sh
2022-11-09 11:12:38 +08:00
SET(ENABLE_DAP2 OFF CACHE BOOL "")
SET(ENABLE_DAP4 OFF CACHE BOOL "")
2012-10-03 04:56:46 +08:00
ENDIF()
Provide byte-range reading of remote datasets re: issue https://github.com/Unidata/netcdf-c/issues/1251 Assume that you have the URL to a remote dataset which is a normal netcdf-3 or netcdf-4 file. This PR allows the netcdf-c to read that dataset's contents as a netcdf file using HTTP byte ranges if the remote server supports byte-range access. Originally, this PR was set up to access Amazon S3 objects, but it can also access other remote datasets such as those provided by a Thredds server via the HTTPServer access protocol. It may also work for other kinds of servers. Note that this is not intended as a true production capability because, as is known, this kind of access to can be quite slow. In addition, the byte-range IO drivers do not currently do any sort of optimization or caching. An additional goal here is to gain some experience with the Amazon S3 REST protocol. This architecture and its use documented in the file docs/byterange.dox. There are currently two test cases: 1. nc_test/tst_s3raw.c - this does a simple open, check format, close cycle for a remote netcdf-3 file and a remote netcdf-4 file. 2. nc_test/test_s3raw.sh - this uses ncdump to investigate some remote datasets. This PR also incorporates significantly changed model inference code (see the superceded PR https://github.com/Unidata/netcdf-c/pull/1259). 1. It centralizes the code that infers the dispatcher. 2. It adds support for byte-range URLs Other changes: 1. NC_HDF5_finalize was not being properly called by nc_finalize(). 2. Fix minor bug in ncgen3.l 3. fix memory leak in nc4info.c 4. add code to walk the .daprc triples and to replace protocol= fragment tag with a more general mode= tag. Final Note: Th inference code is still way too complicated. We need to move to the validfile() model used by netcdf Java, where each dispatcher is asked if it can process the file. This decentralizes the inference code. This will be done after all the major new dispatchers (PIO, Zarr, etc) have been implemented.
2019-01-02 09:27:36 +08:00
# Option to support byte-range reading of remote datasets
Added S3 status to libnetcdf.settings, turned byterange on by default.
2023-01-10 04:11:16 +08:00
OPTION(ENABLE_BYTERANGE "Enable byte-range access to remote datasets.." ON)
Change the network access nomenclature to be less ambiguous. As it stood, did it refer to networked drives? Other things that are network adjacent?
2023-06-13 05:23:44 +08:00
if(NOT ENABLE_REMOTE_FUNCTIONALITY)
MESSAGE(WARNING "ENABLE_REMOTE_FUNCTIONALITY=NO => ENABLE_BYTERANGE=NO")
SET(ENABLE_BYTERANGE OFF CACHE BOOL "ENABLE_REMOTE_FUNCTIONALITY=NO => ENABLE_BYTERANGE=NO" FORCE)
Provide a single option to disable all network access and testing. Add the option "--disable-network-access" (automake) or "-DENABLE_NETWORK_ACCESS=OFF" (cmake). When disabled, this option transitively disables all network access capabilities and testing. If set, this option implies the following: * --disable-dap * --disable-byterange * --disable-s3 This PR answers a request for a feature from Ed Hartnett. ## Misc. Other changes * Take the opportunity to clean up some old, unused options; e.g. --enable-multifilters. * Fix bug in using S3 urls.
2023-06-11 04:08:04 +08:00
ENDIF()
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
Fix common typos Detected by https://github.com/vlajos/misspell_fixer
2015-08-20 17:42:05 +08:00
# Check for the math library so it can be explicitly linked.
Bumped version to 4.3.1.1 in preparation for upcoming bugfix release. Modified the Doxyfile templates to use 'dot', when available, to generate class diagrams instead of the inferior diagram engine bundled in doxygen.
2014-02-05 07:13:18 +08:00
IF(NOT WIN32)
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
FIND_LIBRARY(HAVE_LIBM NAMES math m libm)
IF(NOT HAVE_LIBM)
fix musl build
2020-04-17 22:45:20 +08:00
CHECK_FUNCTION_EXISTS(exp HAVE_LIBM_FUNC)
IF(NOT HAVE_LIBM_FUNC)
MESSAGE(FATAL_ERROR "Unable to find the math library.")
ELSE(NOT HAVE_LIBM_FUNC)
SET(HAVE_LIBM "")
ENDIF()
ELSE(NOT HAVE_LIBM)
MESSAGE(STATUS "Found Math library: ${HAVE_LIBM}")
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
ENDIF()
Bumped version to 4.3.1.1 in preparation for upcoming bugfix release. Modified the Doxyfile templates to use 'dot', when available, to generate class diagrams instead of the inferior diagram engine bundled in doxygen.
2014-02-05 07:13:18 +08:00
ENDIF()
2012-10-03 04:56:46 +08:00
# Option to Enable DAP long tests, remote tests.
Restore default on for enable-dap-remote-testsc The remote test server is up again. So re-enable. Also do some test cleanup.
2021-12-17 06:43:22 +08:00
OPTION(ENABLE_DAP_REMOTE_TESTS "Enable DAP remote tests." ON)
Add support for Zarr string type to NCZarr * re: https://github.com/Unidata/netcdf-c/pull/2278 * re: https://github.com/Unidata/netcdf-c/issues/2485 * re: https://github.com/Unidata/netcdf-c/issues/2474 This PR subsumes PR https://github.com/Unidata/netcdf-c/pull/2278. Actually is a bit an omnibus covering several issues. ## PR https://github.com/Unidata/netcdf-c/pull/2278 Add support for the Zarr string type. Zarr strings are restricted currently to be of fixed size. The primary issue to be addressed is to provide a way for user to specify the size of the fixed length strings. This is handled by providing the following new attributes special: 1. **_nczarr_default_maxstrlen** &mdash; This is an attribute of the root group. It specifies the default maximum string length for string types. If not specified, then it has the value of 64 characters. 2. **_nczarr_maxstrlen** &mdash; This is a per-variable attribute. It specifies the maximum string length for the string type associated with the variable. If not specified, then it is assigned the value of **_nczarr_default_maxstrlen**. This PR also requires some hacking to handle the existing netcdf-c NC_CHAR type, which does not exist in zarr. The goal was to choose numpy types for both the netcdf-c NC_STRING type and the netcdf-c NC_CHAR type such that if a pure zarr implementation read them, it would still work and an NC_CHAR type would be handled by zarr as a string of length 1. For writing variables and NCZarr attributes, the type mapping is as follows: * "|S1" for NC_CHAR. * ">S1" for NC_STRING && MAXSTRLEN==1 * ">Sn" for NC_STRING && MAXSTRLEN==n Note that it is a bit of a hack to use endianness, but it should be ok since for string/char, the endianness has no meaning. For reading attributes with pure zarr (i.e. with no nczarr atribute types defined), they will always be interpreted as of type NC_CHAR. ## Issue: https://github.com/Unidata/netcdf-c/issues/2474 This PR partly fixes this issue because it provided more comprehensive support for Zarr attributes that are JSON valued expressions. This PR still does not address the problem in that issue where the _ARRAY_DIMENSION attribute is incorrectly set. Than can only be fixed by the creator of the datasets. ## Issue: https://github.com/Unidata/netcdf-c/issues/2485 This PR also fixes the scalar failure shown in this issue. It generally cleans up scalar handling. It also adds a note to the documentation describing that NCZarr supports scalars while Zarr does not and also how scalar interoperability is achieved. ## Misc. Other Changes 1. Convert the nczarr special attributes and keys to be all lower case. So "_NCZARR_ATTR" now used "_nczarr_attr. Support back compatibility for the upper case names. 2. Cleanup my too-clever-by-half handling of scalars in libnczarr.
2022-08-28 10:21:13 +08:00
OPTION(ENABLE_EXTERNAL_SERVER_TESTS "Enable external Server remote tests." OFF)
2012-10-03 04:56:46 +08:00
OPTION(ENABLE_DAP_LONG_TESTS "Enable DAP long tests." OFF)
Added ENABLE_DAP_AUTH_TESTS flag to CMakelists files.
2014-03-08 04:58:09 +08:00
Provide a single option to disable all network access and testing. Add the option "--disable-network-access" (automake) or "-DENABLE_NETWORK_ACCESS=OFF" (cmake). When disabled, this option transitively disables all network access capabilities and testing. If set, this option implies the following: * --disable-dap * --disable-byterange * --disable-s3 This PR answers a request for a feature from Ed Hartnett. ## Misc. Other changes * Take the opportunity to clean up some old, unused options; e.g. --enable-multifilters. * Fix bug in using S3 urls.
2023-06-11 04:08:04 +08:00
if(NOT ENABLE_DAP)
SET(ENABLE_DAP_REMOTE_TESTS OFF CACHE BOOL "" FORCE)
SET(ENABLE_EXTERNAL_SERVER_TESTS OFF CACHE BOOL "" FORCE)
SET(ENABLE_DAP_LONG_TESTS OFF CACHE BOOL "" FORCE)
ENDIF()
SET(REMOTETESTSERVERS "remotetest.unidata.ucar.edu" CACHE STRING "test servers to use for remote test")
Add support for Zarr string type to NCZarr * re: https://github.com/Unidata/netcdf-c/pull/2278 * re: https://github.com/Unidata/netcdf-c/issues/2485 * re: https://github.com/Unidata/netcdf-c/issues/2474 This PR subsumes PR https://github.com/Unidata/netcdf-c/pull/2278. Actually is a bit an omnibus covering several issues. ## PR https://github.com/Unidata/netcdf-c/pull/2278 Add support for the Zarr string type. Zarr strings are restricted currently to be of fixed size. The primary issue to be addressed is to provide a way for user to specify the size of the fixed length strings. This is handled by providing the following new attributes special: 1. **_nczarr_default_maxstrlen** &mdash; This is an attribute of the root group. It specifies the default maximum string length for string types. If not specified, then it has the value of 64 characters. 2. **_nczarr_maxstrlen** &mdash; This is a per-variable attribute. It specifies the maximum string length for the string type associated with the variable. If not specified, then it is assigned the value of **_nczarr_default_maxstrlen**. This PR also requires some hacking to handle the existing netcdf-c NC_CHAR type, which does not exist in zarr. The goal was to choose numpy types for both the netcdf-c NC_STRING type and the netcdf-c NC_CHAR type such that if a pure zarr implementation read them, it would still work and an NC_CHAR type would be handled by zarr as a string of length 1. For writing variables and NCZarr attributes, the type mapping is as follows: * "|S1" for NC_CHAR. * ">S1" for NC_STRING && MAXSTRLEN==1 * ">Sn" for NC_STRING && MAXSTRLEN==n Note that it is a bit of a hack to use endianness, but it should be ok since for string/char, the endianness has no meaning. For reading attributes with pure zarr (i.e. with no nczarr atribute types defined), they will always be interpreted as of type NC_CHAR. ## Issue: https://github.com/Unidata/netcdf-c/issues/2474 This PR partly fixes this issue because it provided more comprehensive support for Zarr attributes that are JSON valued expressions. This PR still does not address the problem in that issue where the _ARRAY_DIMENSION attribute is incorrectly set. Than can only be fixed by the creator of the datasets. ## Issue: https://github.com/Unidata/netcdf-c/issues/2485 This PR also fixes the scalar failure shown in this issue. It generally cleans up scalar handling. It also adds a note to the documentation describing that NCZarr supports scalars while Zarr does not and also how scalar interoperability is achieved. ## Misc. Other Changes 1. Convert the nczarr special attributes and keys to be all lower case. So "_NCZARR_ATTR" now used "_nczarr_attr. Support back compatibility for the upper case names. 2. Cleanup my too-clever-by-half handling of scalars in libnczarr.
2022-08-28 10:21:13 +08:00
SET(REMOTETESTSERVERS "remotetest.unidata.ucar.edu" CACHE STRING "test servers to use for remote test")
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
# See if we have zlib
FIND_PACKAGE(ZLIB)
# Define a test flag for have zlib library
IF(ZLIB_FOUND)
INCLUDE_DIRECTORIES(${ZLIB_INCLUDE_DIRS})
SET(ENABLE_ZLIB TRUE)
ELSE()
SET(ENABLE_ZLIB FALSE)
ENDIF()
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
macro(set_std_filter filter)
# Upper case the filter name
string(TOUPPER "${filter}" upfilter)
string(TOLOWER "${filter}" downfilter)
String handling in CMakeLists.txt
2023-07-22 04:19:50 +08:00
if(ENABLE_FILTER_${upfilter})
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
# Define a test flag for filter
Add capability to enable/disable compression libraries
2023-06-14 07:11:53 +08:00
IF(${filter}_FOUND)
INCLUDE_DIRECTORIES(${${filter}_INCLUDE_DIRS})
SET(ENABLE_${upfilter} TRUE)
SET(HAVE_${upfilter} ON)
SET(STD_FILTERS "${STD_FILTERS} ${downfilter}")
MESSAGE(">>> Standard Filter: ${downfilter}")
ELSE()
SET(ENABLE_${upfilter} FALSE)
SET(HAVE_${upfilter} OFF)
ENDIF()
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
ELSE()
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=&lt;absolute directory path&gt; cmake option: -DPLUGIN_INSTALL_DIR=&lt;absolute directory path&gt; ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
SET(HAVE_${upfilter} OFF)
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
ENDIF()
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
endmacro(set_std_filter)
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
# Locate some compressors
Logic to ensure libsz is searched for if Zarr is enabled but enable_filter_szip is false.
2023-07-22 04:46:45 +08:00
OPTION(ENABLE_FILTER_SZIP "Enable use of Szip compression library if it is available. Required if ENABLE_NCZARR is true." ON)
Parity for enable_bz2. BZ2 cannot be disabled altogether, but can fall back to inbternal implementation.
2023-07-21 05:54:56 +08:00
OPTION(ENABLE_FILTER_BZ2 "Enable use of Bz2 compression library if it is available." ON)
OPTION(ENABLE_FILTER_BLOSC "Enable use of blosc compression library if it is available." ON)
OPTION(ENABLE_FILTER_ZSTD "Enable use of Zstd compression library if it is available." ON)
Additional tweaking of search logic.
2023-07-22 04:32:25 +08:00
IF (ENABLE_FILTER_SZIP)
Don't call find_package if not enabled The `FIND_PACKAGE` should not be called if the filter/compression library is not enabled. It was causing some inconsistencied in link libraries and CMake configure output...
2023-06-14 22:22:01 +08:00
FIND_PACKAGE(Szip)
Logic to ensure libsz is searched for if Zarr is enabled but enable_filter_szip is false.
2023-07-22 04:46:45 +08:00
ELSEIF(ENABLE_NCZARR)
Don't call find_package if not enabled The `FIND_PACKAGE` should not be called if the filter/compression library is not enabled. It was causing some inconsistencied in link libraries and CMake configure output...
2023-06-14 22:22:01 +08:00
FIND_PACKAGE(Szip)
ENDIF()
Parity for enable_bz2. BZ2 cannot be disabled altogether, but can fall back to inbternal implementation.
2023-07-21 05:54:56 +08:00
IF (ENABLE_FILTER_BZ2)
Don't call find_package if not enabled The `FIND_PACKAGE` should not be called if the filter/compression library is not enabled. It was causing some inconsistencied in link libraries and CMake configure output...
2023-06-14 22:22:01 +08:00
FIND_PACKAGE(Bz2)
ENDIF()
Small change in CMakeLists.txt
2023-07-22 04:02:30 +08:00
IF (ENABLE_FILTER_BLOSC)
Don't call find_package if not enabled The `FIND_PACKAGE` should not be called if the filter/compression library is not enabled. It was causing some inconsistencied in link libraries and CMake configure output...
2023-06-14 22:22:01 +08:00
FIND_PACKAGE(Blosc)
ENDIF()
Small change in CMakeLists.txt
2023-07-22 04:02:30 +08:00
IF (ENABLE_FILTER_ZSTD)
Don't call find_package if not enabled The `FIND_PACKAGE` should not be called if the filter/compression library is not enabled. It was causing some inconsistencied in link libraries and CMake configure output...
2023-06-14 22:22:01 +08:00
FIND_PACKAGE(Zstd)
ENDIF()
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
# Accumulate standard filters
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=&lt;absolute directory path&gt; cmake option: -DPLUGIN_INSTALL_DIR=&lt;absolute directory path&gt; ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
set(STD_FILTERS "deflate") # Always have deflate*/
Fix szip handling
2022-05-01 09:06:01 +08:00
set_std_filter(Szip)
Additional cmake-based logic.
2023-07-24 23:32:39 +08:00
SET(HAVE_SZ ${Szip_FOUND})
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
set_std_filter(Blosc)
Additional cmake-based logic.
2023-07-24 23:32:39 +08:00
IF(Zstd_FOUND)
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
set_std_filter(Zstd)
Additional cmake-based logic.
2023-07-24 23:32:39 +08:00
ENDIF()
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=&lt;absolute directory path&gt; cmake option: -DPLUGIN_INSTALL_DIR=&lt;absolute directory path&gt; ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
IF(Bz2_FOUND)
set_std_filter(Bz2)
ELSE()
# The reason we use a local version is to support a more comples test case
Use env variable USERPROFILE instead of HOME for windows and mingw. re: https://github.com/Unidata/netcdf-c/issues/2380 re: https://github.com/Unidata/netcdf-c/issues/2337 This PARTIALLY fixes some HOME problems because under Windows, the HOME environment variable may not be set. In that case, use the USERPROFILE environment variable instead.
2022-06-15 04:44:23 +08:00
MESSAGE("libbz2 not found using built-in version")
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=&lt;absolute directory path&gt; cmake option: -DPLUGIN_INSTALL_DIR=&lt;absolute directory path&gt; ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
SET(HAVE_LOCAL_BZ2 ON)
SET(HAVE_BZ2 ON)
set(STD_FILTERS "${STD_FILTERS} bz2")
ENDIF()
Update to default --with-plugin-dir to yes
2022-05-25 10:05:19 +08:00
# If user wants, then install selected plugins (default on)
Make plugin dir off by default for this release.
2022-06-11 05:03:36 +08:00
SET(PLUGIN_INSTALL_DIR "NO" CACHE STRING "Whether and where we should install plugins; defaults to yes")
Fix way CMake handle -DPLUGIN_INSTALL_DIR re: Issue https://github.com/Unidata/netcdf-c/issues/2429 The CMake tests for -DPLUGIN_INSTALL_DIR only accepted "on" or "off" as boolean values. So extend to accept e.g. -DPLUGIN_INSTALL_DIR=yes or -DPLUGIN_INSTALL_DIR=1, etc.
2022-06-26 06:47:49 +08:00
# This is ugly, but seems necessary because of CMake's boolean structure
SET(boolval FALSE)
IF(DEFINED PLUGIN_INSTALL_DIR)
booleanize(${PLUGIN_INSTALL_DIR} boolval)
IF(boolval)
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
SET(ENABLE_PLUGIN_INSTALL YES)
Fix way CMake handle -DPLUGIN_INSTALL_DIR re: Issue https://github.com/Unidata/netcdf-c/issues/2429 The CMake tests for -DPLUGIN_INSTALL_DIR only accepted "on" or "off" as boolean values. So extend to accept e.g. -DPLUGIN_INSTALL_DIR=yes or -DPLUGIN_INSTALL_DIR=1, etc.
2022-06-26 06:47:49 +08:00
# No actual value was specified
UNSET(PLUGIN_INSTALL_DIR CACHE)
ELSE()
IF(boolval STREQUAL "NOTFOUND")
# Must be an actual value
SET(ENABLE_PLUGIN_INSTALL YES)
ELSE()
SET(ENABLE_PLUGIN_INSTALL NO)
ENDIF()
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=&lt;absolute directory path&gt; cmake option: -DPLUGIN_INSTALL_DIR=&lt;absolute directory path&gt; ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
ENDIF()
Fix way CMake handle -DPLUGIN_INSTALL_DIR re: Issue https://github.com/Unidata/netcdf-c/issues/2429 The CMake tests for -DPLUGIN_INSTALL_DIR only accepted "on" or "off" as boolean values. So extend to accept e.g. -DPLUGIN_INSTALL_DIR=yes or -DPLUGIN_INSTALL_DIR=1, etc.
2022-06-26 06:47:49 +08:00
ELSE()
SET(ENABLE_PLUGIN_INSTALL NO)
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
ENDIF()
Fix way CMake handle -DPLUGIN_INSTALL_DIR re: Issue https://github.com/Unidata/netcdf-c/issues/2429 The CMake tests for -DPLUGIN_INSTALL_DIR only accepted "on" or "off" as boolean values. So extend to accept e.g. -DPLUGIN_INSTALL_DIR=yes or -DPLUGIN_INSTALL_DIR=1, etc.
2022-06-26 06:47:49 +08:00
# Ensure no defined plugin dir if not enabled
IF(NOT ENABLE_PLUGIN_INSTALL)
UNSET(PLUGIN_INSTALL_DIR CACHE)
ENDIF()
IF(ENABLE_PLUGIN_INSTALL)
IF(NOT DEFINED PLUGIN_INSTALL_DIR)
# Default to HDF5_PLUGIN_PATH or its default directories
IF(DEFINED ENV{HDF5_PLUGIN_PATH})
SET(PLUGIN_INSTALL_DIR "$ENV{HDF5_PLUGIN_PATH}")
ELSE()
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
IF(ISMSVC OR ISMINGW)
Fix way CMake handle -DPLUGIN_INSTALL_DIR re: Issue https://github.com/Unidata/netcdf-c/issues/2429 The CMake tests for -DPLUGIN_INSTALL_DIR only accepted "on" or "off" as boolean values. So extend to accept e.g. -DPLUGIN_INSTALL_DIR=yes or -DPLUGIN_INSTALL_DIR=1, etc.
2022-06-26 06:47:49 +08:00
SET(PLUGIN_INSTALL_DIR "$ENV{ALLUSERSPROFILE}\\hdf5\\lib\\plugin")
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
ELSE()
Add various newer options to nc-config
2022-09-20 05:56:16 +08:00
SET(PLUGIN_INSTALL_DIR "/usr/local/hdf5/lib/plugin")
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
ENDIF()
Fix way CMake handle -DPLUGIN_INSTALL_DIR re: Issue https://github.com/Unidata/netcdf-c/issues/2429 The CMake tests for -DPLUGIN_INSTALL_DIR only accepted "on" or "off" as boolean values. So extend to accept e.g. -DPLUGIN_INSTALL_DIR=yes or -DPLUGIN_INSTALL_DIR=1, etc.
2022-06-26 06:47:49 +08:00
ENDIF()
MESSAGE("Defaulting to -DPLUGIN_INSTALL_DIR=${PLUGIN_INSTALL_DIR}")
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
ENDIF()
ENDIF()
Update to default --with-plugin-dir to yes
2022-05-25 10:05:19 +08:00
IF(ENABLE_PLUGIN_INSTALL)
# Use the lowest priority dir in the path
IF(NOT ISMSVC AND NOT ISMINGW)
Fix way CMake handle -DPLUGIN_INSTALL_DIR re: Issue https://github.com/Unidata/netcdf-c/issues/2429 The CMake tests for -DPLUGIN_INSTALL_DIR only accepted "on" or "off" as boolean values. So extend to accept e.g. -DPLUGIN_INSTALL_DIR=yes or -DPLUGIN_INSTALL_DIR=1, etc.
2022-06-26 06:47:49 +08:00
STRING(REPLACE ":" ";" PATH_LIST ${PLUGIN_INSTALL_DIR})
Update to default --with-plugin-dir to yes
2022-05-25 10:05:19 +08:00
ELSE()
Fix way CMake handle -DPLUGIN_INSTALL_DIR re: Issue https://github.com/Unidata/netcdf-c/issues/2429 The CMake tests for -DPLUGIN_INSTALL_DIR only accepted "on" or "off" as boolean values. So extend to accept e.g. -DPLUGIN_INSTALL_DIR=yes or -DPLUGIN_INSTALL_DIR=1, etc.
2022-06-26 06:47:49 +08:00
SET(PATH_LIST ${PLUGIN_INSTALL_DIR})
Update to default --with-plugin-dir to yes
2022-05-25 10:05:19 +08:00
ENDIF()
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
Update to default --with-plugin-dir to yes
2022-05-25 10:05:19 +08:00
# Get last element
LIST(GET PATH_LIST -1 PLUGIN_INSTALL_DIR)
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
SET(PLUGIN_INSTALL_DIR_SETTING "${PLUGIN_INSTALL_DIR}")
Fix way CMake handle -DPLUGIN_INSTALL_DIR re: Issue https://github.com/Unidata/netcdf-c/issues/2429 The CMake tests for -DPLUGIN_INSTALL_DIR only accepted "on" or "off" as boolean values. So extend to accept e.g. -DPLUGIN_INSTALL_DIR=yes or -DPLUGIN_INSTALL_DIR=1, etc.
2022-06-26 06:47:49 +08:00
MESSAGE("Final value of-DPLUGIN_INSTALL_DIR=${PLUGIN_INSTALL_DIR}")
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
ELSE() # No option specified
UNSET(PLUGIN_INSTALL_DIR)
UNSET(PLUGIN_INSTALL_DIR CACHE)
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=&lt;absolute directory path&gt; cmake option: -DPLUGIN_INSTALL_DIR=&lt;absolute directory path&gt; ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
SET(PLUGIN_INSTALL_DIR_SETTING "N.A.")
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
ENDIF()
Avoid optionnaly depends on zip for NCZarr
2023-01-19 16:37:24 +08:00
# Try to enable NCZarr zip support
Turn nczarr zip support off by default in cmake, add a status line indicating whether nczarr-zip-support is available, in libnetcdf.settings.
2023-02-28 04:34:26 +08:00
OPTION(ENABLE_NCZARR_ZIP "Enable NCZarr ZIP support." OFF)
Avoid optionnaly depends on zip for NCZarr
2023-01-19 16:37:24 +08:00
IF (ENABLE_NCZARR_ZIP)
FIND_PACKAGE(Zip REQUIRED)
Add zip file support The primary change is to support the use of a zip file as a storage format. Simultaneously the .nz4 support is made obsolete Use of zip requires the libzip support library, so a number of changes to the build files (Makefile.am, CMakeLists.txt) are necessary to locate and incorporate libzip. The nczarr_tests tests are also changed to add zip testing. Other changes: * Make sure distcheck leaves no files around. * Add some functions to netcdf_aux to export some functions of libnetcdf. * Add a new error NC_EFOUND as the complement of NC_EEMPTY. * Add tracing support to nclog and use it in libnczarr. * Modify the zmap interface to support the writeonce semantics of zip. * Create a new s3util.c to support a variety of S3 auxilliary functions. * EXTERNL'ize a number of functions so they can be used in s3util. * Add support for the S3 ListObjects CommonPrefixes mechanism to improve search. * Add experimental support for running nczarr X s3 tests against the actual Amazon S3 cloud.
2021-01-29 11:11:01 +08:00
INCLUDE_DIRECTORIES(${Zip_INCLUDE_DIRS})
Avoid optionnaly depends on zip for NCZarr
2023-01-19 16:37:24 +08:00
ENDIF ()
Add zip file support The primary change is to support the use of a zip file as a storage format. Simultaneously the .nz4 support is made obsolete Use of zip requires the libzip support library, so a number of changes to the build files (Makefile.am, CMakeLists.txt) are necessary to locate and incorporate libzip. The nczarr_tests tests are also changed to add zip testing. Other changes: * Make sure distcheck leaves no files around. * Add some functions to netcdf_aux to export some functions of libnetcdf. * Add a new error NC_EFOUND as the complement of NC_EEMPTY. * Add tracing support to nclog and use it in libnczarr. * Modify the zmap interface to support the writeonce semantics of zip. * Create a new s3util.c to support a variety of S3 auxilliary functions. * EXTERNL'ize a number of functions so they can be used in s3util. * Add support for the S3 ListObjects CommonPrefixes mechanism to improve search. * Add experimental support for running nczarr X s3 tests against the actual Amazon S3 cloud.
2021-01-29 11:11:01 +08:00
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
# libdl is always available; built-in in Windows and OSX
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
OPTION(ENABLE_PLUGINS "Enable dynamically loaded plugins (default on)." ON)
IF(MINGW)
SET(ENABLE_PLUGINS OFF CACHE BOOL "Disable plugins" FORCE)
ELSE()
IF(NOT WIN32)
IF(HAVE_DLFCN_H)
INCLUDE_DIRECTORIES("dlfcn.h")
ENDIF()
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
ENDIF()
ENDIF()
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
IF(ENABLE_PLUGINS)
SET(USEPLUGINS yes)
ENDIF()
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
Merging CMake branch into trunk. Changes include: CMake related changes in CMakeLists.txt files, cmake_config.h.in. Other changes relate to Windows-specific issues, and changes made when regenerating generated source files.
2013-01-16 06:43:09 +08:00
# Enable some developer-only tests
There were a few changes that needed to be made, and we needed to disable extra tests by default or 'make distcheck' fails, for now. Will investigate in more depth later.
2015-10-09 01:02:31 +08:00
OPTION(ENABLE_EXTRA_TESTS "Enable Extra tests. Some may not work because of known issues. Developers only." OFF)
Merging CMake branch into trunk. Changes include: CMake related changes in CMakeLists.txt files, cmake_config.h.in. Other changes relate to Windows-specific issues, and changes made when regenerating generated source files.
2013-01-16 06:43:09 +08:00
IF(ENABLE_EXTRA_TESTS)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
SET(EXTRA_TESTS ON)
Merging CMake branch into trunk. Changes include: CMake related changes in CMakeLists.txt files, cmake_config.h.in. Other changes relate to Windows-specific issues, and changes made when regenerating generated source files.
2013-01-16 06:43:09 +08:00
ENDIF()
2012-10-03 04:56:46 +08:00
# Option to use bundled XGetopt in place of getopt(). This is mostly useful
Fix undefined references when using Visual Studio Fix Issue https://github.com/Unidata/netcdf-c/issues/1725. Replace PR https://github.com/Unidata/netcdf-c/pull/1726 Also replace PR https://github.com/Unidata/netcdf-c/pull/1694 The general problem is that under Visual Studio, we are seeing a number of undefined reference and other scoping errors. The reason is that the code is not properly using Visual Studio _declspec() declarations. The basic solution is to ensure that when compiling the code itself one needs to ensure that _declspec(dllexport) is used. There are several sets of macros to handle this, but they all rely on the flag DLL_EXPORT being define when the code is compiled, but not being defined when the code is used via a .h file. As a test, I modified XGetOpt.c to build properly. I also fixed the oc2 library to properly _declspec things like ocdebug. I also made some misc. changes to get all the tests to run if cygwin is installed (to get bash, sed, etc). Misc. Changes: * Put XGetOpt.c into libsrc and copy at build time to the other directories where it is needed.
2020-05-19 09:36:28 +08:00
# for MSVC builds. If not building utilities or some tests,
# getopt() isn't required at all.
2012-10-03 04:56:46 +08:00
IF(MSVC)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
OPTION(ENABLE_XGETOPT "Enable bundled XGetOpt instead of external getopt()." ON)
IF(ENABLE_XGETOPT)
SET(USE_X_GETOPT ON CACHE BOOL "")
ENDIF()
2012-10-03 04:56:46 +08:00
ENDIF()
SET(MATH "")
IF(NOT WIN32)
Added STDIO option to CMake-based build.
2015-10-10 03:06:32 +08:00
# STDIO instead of posixio.
OPTION(ENABLE_STDIO "If true, use stdio instead of posixio (ex. on the Cray)" OFF)
IF(ENABLE_STDIO)
SET(USE_STDIO ON CACHE BOOL "")
ENDIF()
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
# FFIO insteaad of PosixIO
OPTION(ENABLE_FFIO "If true, use ffio instead of posixio" OFF)
IF(ENABLE_FFIO)
SET(USE_FFIO ON CACHE BOOL "")
ENDIF()
2012-10-03 04:56:46 +08:00
ENDIF()
Fix byterange handling of some URLS re: Issue The byterange handling of the following URLS fails. ### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes" It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD). I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time. ### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3" When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case. ## Misc. Related Changes 1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support 2. Added the above URLS as additional test cases ## Misc. Un-Related Changes 1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD 2. Fix some minor warnings ## Open Problems * Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-03 10:51:02 +08:00
# Options for S3 Support
OPTION(ENABLE_S3 "Enable S3 support." OFF)
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
OPTION(ENABLE_S3_INTERNAL "Enable S3 Internal support." OFF)
Fix byterange handling of some URLS re: Issue The byterange handling of the following URLS fails. ### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes" It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD). I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time. ### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3" When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case. ## Misc. Related Changes 1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support 2. Added the above URLS as additional test cases ## Misc. Un-Related Changes 1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD 2. Fix some minor warnings ## Open Problems * Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-03 10:51:02 +08:00
OPTION(ENABLE_NCZARR_S3 "Enable NCZarr S3 support; Deprecated in favor of ENABLE_S3" OFF)
## Addendum [5/9/23] It turns out that attempting to test S3 using a github action secret is a very complex process. So, this was disabled for github actions. However, a new *run_tests_s3.yml* action file was added that will eventually encapsulate S3 testing.
2023-05-10 11:13:49 +08:00
Change the network access nomenclature to be less ambiguous. As it stood, did it refer to networked drives? Other things that are network adjacent?
2023-06-13 05:23:44 +08:00
IF(NOT ENABLE_REMOTE_FUNCTIONALITY)
Provide a single option to disable all network access and testing. Add the option "--disable-network-access" (automake) or "-DENABLE_NETWORK_ACCESS=OFF" (cmake). When disabled, this option transitively disables all network access capabilities and testing. If set, this option implies the following: * --disable-dap * --disable-byterange * --disable-s3 This PR answers a request for a feature from Ed Hartnett. ## Misc. Other changes * Take the opportunity to clean up some old, unused options; e.g. --enable-multifilters. * Fix bug in using S3 urls.
2023-06-11 04:08:04 +08:00
SET(ENABLE_S3 OFF CACHE BOOL "" FORCE)
SET(ENABLE_S3_INTERNAL OFF CACHE BOOL "" FORCE)
SET(ENABLE_NCZARR_S3 OFF CACHE BOOL "" FORCE)
ENDIF()
## Addendum [5/9/23] It turns out that attempting to test S3 using a github action secret is a very complex process. So, this was disabled for github actions. However, a new *run_tests_s3.yml* action file was added that will eventually encapsulate S3 testing.
2023-05-10 11:13:49 +08:00
# Control S3 Testing: Multi-valued option
SET(WITH_S3_TESTING OFF CACHE STRING "Control S3 Testing: ON (i.e. all) OFF (i.e. none) PUBLIC")
SET_PROPERTY(CACHE WITH_S3_TESTING PROPERTY STRINGS ON OFF PUBLIC) #
IF(WITH_S3_TESTING STREQUAL "")
SET(WITH_S3_TESTING OFF CACHE STRING "") # Default
ENDIF()
IF(WITH_S3_TESTING)
message(WARNING "**** DO NOT USE WITH_S3_TESTING=ON UNLESS YOU HAVE ACCESS TO THE UNIDATA S3 BUCKET! ***")
ENDIF()
This PR adds EXPERIMENTAL support for accessing data in the cloud using a variant of the Zarr protocol and storage format. This enhancement is generically referred to as "NCZarr". The data model supported by NCZarr is netcdf-4 minus the user-defined types and the String type. In this sense it is similar to the CDF-5 data model. More detailed information about enabling and using NCZarr is described in the document NUG/nczarr.md and in a [Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in). WARNING: this code has had limited testing, so do use this version for production work. Also, performance improvements are ongoing. Note especially the following platform matrix of successful tests: Platform | Build System | S3 support ------------------------------------ Linux+gcc | Automake | yes Linux+gcc | CMake | yes Visual Studio | CMake | no Additionally, and as a consequence of the addition of NCZarr, major changes have been made to the Filter API. NOTE: NCZarr does not yet support filters, but these changes are enablers for that support in the future. Note that it is possible (probable?) that there will be some accidental reversions if the changes here did not correctly mimic the existing filter testing. In any case, previously filter ids and parameters were of type unsigned int. In order to support the more general zarr filter model, this was all converted to char*. The old HDF5-specific, unsigned int operations are still supported but they are wrappers around the new, char* based nc_filterx_XXX functions. This entailed at least the following changes: 1. Added the files libdispatch/dfilterx.c and include/ncfilter.h 2. Some filterx utilities have been moved to libdispatch/daux.c 3. A new entry, "filter_actions" was added to the NCDispatch table and the version bumped. 4. An overly complex set of structs was created to support funnelling all of the filterx operations thru a single dispatch "filter_actions" entry. 5. Move common code to from libhdf5 to libsrc4 so that it is accessible to nczarr. Changes directly related to Zarr: 1. Modified CMakeList.txt and configure.ac to support both C and C++ -- this is in support of S3 support via the awd-sdk libraries. 2. Define a size64_t type to support nczarr. 3. More reworking of libdispatch/dinfermodel.c to support zarr and to regularize the structure of the fragments section of a URL. Changes not directly related to Zarr: 1. Make client-side filter registration be conditional, with default off. 2. Hack include/nc4internal.h to make some flags added by Ed be unique: e.g. NC_CREAT, NC_INDEF, etc. 3. cleanup include/nchttp.h and libdispatch/dhttp.c. 4. Misc. changes to support compiling under Visual Studio including: * Better testing under windows for dirent.h and opendir and closedir. 5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags and to centralize error reporting. 6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them. 7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible. Changes Left TO-DO: 1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
Fix byterange handling of some URLS re: Issue The byterange handling of the following URLS fails. ### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes" It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD). I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time. ### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3" When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case. ## Misc. Related Changes 1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support 2. Added the above URLS as additional test cases ## Misc. Un-Related Changes 1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD 2. Fix some minor warnings ## Open Problems * Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-03 10:51:02 +08:00
# ENABLE_NCZARR_S3 is now an alias for ENABLE_S3 (but...)
if (NOT ENABLE_S3 AND ENABLE_NCZARR_S3)
SET(ENABLE_S3 ON CACHE BOOL "NCARR S3" FORCE) # For back compatibility
Fix nczarr-experimental to better support CMake and find AWS libraries The primary fix is to improve CMake build support. Specific changes include: * CMake: Provide a better soln to locating the AWS SDK libraries; the new way is the preferred method as described in the aws-cpp-sdk documentation. * CMake (and Automake): allow -DENABLE_S3_SDK (default off) to suppress looking for AWS libraries. * CMake: add the complete set of nczarr tests * CMake: add EXTERNL as needed to various .h files. * Improve support for windows drive letters in paths. * Add nczarr and s3 flags to nc-config * For VisualStudio X nczarr, cleanup the NAN+INFINITY handling * Convert _MSC_VER -> _WIN32 and vice versa as needed * NCZarr - support multiple platform paths including windows, cygwin. mingw, etc. * NCZarr - sort the test outputs because different platforms produce directory contents in different orders. One big change concerns netcdf-c/CMakeLists.txt and netcdf-c/configure.ac. In the current versions, it was the case that --disable-hdf5 disabled netcdf-4 (libsrc4). With nczarr, this can no longer be the case because nczarr requires libsrc4 even if libhdf5 is disabled. So, I modified the above files to move the format options (HDF5, NCZarr, HDF4, etc) to a single place near the front of the files. Now it is the case that: * Enabling any of the formats that require libsrc4 also does an implicit --enable-netcdf4. * --disable-netcdf4 | --disable-netcdf-4 now becomes and alias for --disable-hdf5. There are probably some bugs in this change in terms of dependencies between format options. Problems: * CMake S3 support is still not working for Visual Studio * A recent issue points out that there is work to do on handling UTF8 filenames, but that will be addressed in a separate fix. Notes: * Consider converting all of our includes/.h files to use EXTERNL
2020-07-13 02:21:56 +08:00
ENDIF()
Fix byterange handling of some URLS re: Issue The byterange handling of the following URLS fails. ### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes" It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD). I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time. ### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3" When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case. ## Misc. Related Changes 1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support 2. Added the above URLS as additional test cases ## Misc. Un-Related Changes 1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD 2. Fix some minor warnings ## Open Problems * Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-03 10:51:02 +08:00
UNSET(ENABLE_NCZARR_S3)
This PR adds EXPERIMENTAL support for accessing data in the cloud using a variant of the Zarr protocol and storage format. This enhancement is generically referred to as "NCZarr". The data model supported by NCZarr is netcdf-4 minus the user-defined types and the String type. In this sense it is similar to the CDF-5 data model. More detailed information about enabling and using NCZarr is described in the document NUG/nczarr.md and in a [Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in). WARNING: this code has had limited testing, so do use this version for production work. Also, performance improvements are ongoing. Note especially the following platform matrix of successful tests: Platform | Build System | S3 support ------------------------------------ Linux+gcc | Automake | yes Linux+gcc | CMake | yes Visual Studio | CMake | no Additionally, and as a consequence of the addition of NCZarr, major changes have been made to the Filter API. NOTE: NCZarr does not yet support filters, but these changes are enablers for that support in the future. Note that it is possible (probable?) that there will be some accidental reversions if the changes here did not correctly mimic the existing filter testing. In any case, previously filter ids and parameters were of type unsigned int. In order to support the more general zarr filter model, this was all converted to char*. The old HDF5-specific, unsigned int operations are still supported but they are wrappers around the new, char* based nc_filterx_XXX functions. This entailed at least the following changes: 1. Added the files libdispatch/dfilterx.c and include/ncfilter.h 2. Some filterx utilities have been moved to libdispatch/daux.c 3. A new entry, "filter_actions" was added to the NCDispatch table and the version bumped. 4. An overly complex set of structs was created to support funnelling all of the filterx operations thru a single dispatch "filter_actions" entry. 5. Move common code to from libhdf5 to libsrc4 so that it is accessible to nczarr. Changes directly related to Zarr: 1. Modified CMakeList.txt and configure.ac to support both C and C++ -- this is in support of S3 support via the awd-sdk libraries. 2. Define a size64_t type to support nczarr. 3. More reworking of libdispatch/dinfermodel.c to support zarr and to regularize the structure of the fragments section of a URL. Changes not directly related to Zarr: 1. Make client-side filter registration be conditional, with default off. 2. Hack include/nc4internal.h to make some flags added by Ed be unique: e.g. NC_CREAT, NC_INDEF, etc. 3. cleanup include/nchttp.h and libdispatch/dhttp.c. 4. Misc. changes to support compiling under Visual Studio including: * Better testing under windows for dirent.h and opendir and closedir. 5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags and to centralize error reporting. 6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them. 7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible. Changes Left TO-DO: 1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
Change the network access nomenclature to be less ambiguous. As it stood, did it refer to networked drives? Other things that are network adjacent?
2023-06-13 05:23:44 +08:00
IF(NOT ENABLE_REMOTE_FUNCTIONALITY)
MESSAGE(WARNING "ENABLE_REMOTE_FUNCTIONALITY=NO => disable all s3 functionality")
Provide a single option to disable all network access and testing. Add the option "--disable-network-access" (automake) or "-DENABLE_NETWORK_ACCESS=OFF" (cmake). When disabled, this option transitively disables all network access capabilities and testing. If set, this option implies the following: * --disable-dap * --disable-byterange * --disable-s3 This PR answers a request for a feature from Ed Hartnett. ## Misc. Other changes * Take the opportunity to clean up some old, unused options; e.g. --enable-multifilters. * Fix bug in using S3 urls.
2023-06-11 04:08:04 +08:00
SET(ENABLE_S3 OFF CACHE BOOL "" FORCE)
SET(ENABLE_S3_INTERNAL OFF CACHE BOOL "" FORCE)
SET(ENABLE_NCZARR_S3 OFF CACHE BOOL "" FORCE)
SET(ENABLE_HDF5_ROS3 OFF CACHE BOOL "Use ROS3" FORCE)
SET(WITH_S3_TESTING OFF CACHE STRING "" FORCE)
ENDIF()
Fix byterange handling of some URLS re: Issue The byterange handling of the following URLS fails. ### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes" It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD). I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time. ### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3" When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case. ## Misc. Related Changes 1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support 2. Added the above URLS as additional test cases ## Misc. Un-Related Changes 1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD 2. Fix some minor warnings ## Open Problems * Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-03 10:51:02 +08:00
# Note we check for the library after checking for enable_s3
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
# because for some reason this screws up if we unconditionally test for sdk
# and it is not available. Fix someday
Fix byterange handling of some URLS re: Issue The byterange handling of the following URLS fails. ### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes" It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD). I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time. ### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3" When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case. ## Misc. Related Changes 1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support 2. Added the above URLS as additional test cases ## Misc. Un-Related Changes 1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD 2. Fix some minor warnings ## Open Problems * Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-03 10:51:02 +08:00
IF(ENABLE_S3)
## Addendum [5/9/23] It turns out that attempting to test S3 using a github action secret is a very complex process. So, this was disabled for github actions. However, a new *run_tests_s3.yml* action file was added that will eventually encapsulate S3 testing.
2023-05-10 11:13:49 +08:00
IF(NOT ENABLE_S3_INTERNAL)
# See if aws-s3-sdk is available
find_package(AWSSDK REQUIRED COMPONENTS s3;core)
IF(AWSSDK_FOUND)
SET(service s3;core)
AWSSDK_DETERMINE_LIBS_TO_LINK(service AWS_LINK_LIBRARIES)
SET(ENABLE_S3_AWS ON CACHE BOOL "S3 AWS" FORCE)
ELSE()
SET(ENABLE_S3_AWS OFF CACHE BOOL "S3 AWS" FORCE)
ENDIF()
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
ENDIF()
Improve the building of NCZARR S3 support in CMake and Autoconf There were some irregularities in the flags for handling NCZarr S3 support. The primary change is to regularize the flags controlling this to the following. 1. Automake: --enable-nczarr-s3 and CMake: ENABLE_NCZARR_S3 2. Automake: --enable-nczarr-s3-tests and CMake: ENABLE_NCZARR_S3_TESTS Flag 1 indicates that NCZarr should be built with S3 support enabled. Flag 2 indicates that the NCZarr S3 tests should be run These two flags are separate because running the NCZarr S3 tests requires access to protected S3 resources. Currently, running these tests is restricted to Unidata personnel. However, users may want to enable S3 support even if they cannot run the tests. It is, of course, an error to specify 2 without specifying 1. Additionally, if the AWS S3 SDK library is not found, then the NCZARR S3 support and testing must be disabled. Otherwise an error is signaled during the build. Some of these NCZarr and S3 changes are propagated to nc-config. Misc. Other Changes: 1. Allow testing for CYGWIN or MSVC in shell scripts. 2. Add specific test for HDF5 library version 1.10.6. This is encoded as "HDF5_UTF8_PATHS" because that is the first version where HDF5 properly supports it under Windows. This is used in hdf5internal/nc4_ndf5_ansi_to_utf8. 3. Add a AM Conditional -- AX_IGNORE -- for use in testing when it is desirable to temporarily suppress Makefile code. 4. Add MULTIFILTER flag to CMakeLists.txt
2020-10-17 05:04:51 +08:00
ELSE()
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
SET(ENABLE_S3_AWS OFF CACHE BOOL "S3 AWS" FORCE)
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
ENDIF()
Improve the building of NCZARR S3 support in CMake and Autoconf There were some irregularities in the flags for handling NCZarr S3 support. The primary change is to regularize the flags controlling this to the following. 1. Automake: --enable-nczarr-s3 and CMake: ENABLE_NCZARR_S3 2. Automake: --enable-nczarr-s3-tests and CMake: ENABLE_NCZARR_S3_TESTS Flag 1 indicates that NCZarr should be built with S3 support enabled. Flag 2 indicates that the NCZarr S3 tests should be run These two flags are separate because running the NCZarr S3 tests requires access to protected S3 resources. Currently, running these tests is restricted to Unidata personnel. However, users may want to enable S3 support even if they cannot run the tests. It is, of course, an error to specify 2 without specifying 1. Additionally, if the AWS S3 SDK library is not found, then the NCZARR S3 support and testing must be disabled. Otherwise an error is signaled during the build. Some of these NCZarr and S3 changes are propagated to nc-config. Misc. Other Changes: 1. Allow testing for CYGWIN or MSVC in shell scripts. 2. Add specific test for HDF5 library version 1.10.6. This is encoded as "HDF5_UTF8_PATHS" because that is the first version where HDF5 properly supports it under Windows. This is used in hdf5internal/nc4_ndf5_ansi_to_utf8. 3. Add a AM Conditional -- AX_IGNORE -- for use in testing when it is desirable to temporarily suppress Makefile code. 4. Add MULTIFILTER flag to CMakeLists.txt
2020-10-17 05:04:51 +08:00
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
# Unless/until we get aws-sdk-cpp working for Windows, force use of internal
IF(ENABLE_S3 AND MSVC)
SET(ENABLE_S3_INTERNAL ON CACHE BOOL "S3 Intern" FORCE)
ENDIF()
IF(ENABLE_S3)
IF(NOT ENABLE_S3_AWS AND NOT ENABLE_S3_INTERNAL)
Fix byterange handling of some URLS re: Issue The byterange handling of the following URLS fails. ### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes" It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD). I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time. ### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3" When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case. ## Misc. Related Changes 1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support 2. Added the above URLS as additional test cases ## Misc. Un-Related Changes 1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD 2. Fix some minor warnings ## Open Problems * Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-03 10:51:02 +08:00
message(FATAL_ERROR "S3 support library not found; please specify option -DENABLE_S3=NO")
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
SET(ENABLE_S3 OFF CACHE BOOL "S3 support" FORCE)
ENDIF()
IF(ENABLE_S3_AWS AND ENABLE_S3_INTERNAL)
message(WARNING "Both aws-sdk-cpp and s3-internal enabled => use s3-internal")
SET(ENABLE_S3_AWS OFF CACHE BOOL "S3 AWS" FORCE)
Improve the building of NCZARR S3 support in CMake and Autoconf There were some irregularities in the flags for handling NCZarr S3 support. The primary change is to regularize the flags controlling this to the following. 1. Automake: --enable-nczarr-s3 and CMake: ENABLE_NCZARR_S3 2. Automake: --enable-nczarr-s3-tests and CMake: ENABLE_NCZARR_S3_TESTS Flag 1 indicates that NCZarr should be built with S3 support enabled. Flag 2 indicates that the NCZarr S3 tests should be run These two flags are separate because running the NCZarr S3 tests requires access to protected S3 resources. Currently, running these tests is restricted to Unidata personnel. However, users may want to enable S3 support even if they cannot run the tests. It is, of course, an error to specify 2 without specifying 1. Additionally, if the AWS S3 SDK library is not found, then the NCZARR S3 support and testing must be disabled. Otherwise an error is signaled during the build. Some of these NCZarr and S3 changes are propagated to nc-config. Misc. Other Changes: 1. Allow testing for CYGWIN or MSVC in shell scripts. 2. Add specific test for HDF5 library version 1.10.6. This is encoded as "HDF5_UTF8_PATHS" because that is the first version where HDF5 properly supports it under Windows. This is used in hdf5internal/nc4_ndf5_ansi_to_utf8. 3. Add a AM Conditional -- AX_IGNORE -- for use in testing when it is desirable to temporarily suppress Makefile code. 4. Add MULTIFILTER flag to CMakeLists.txt
2020-10-17 05:04:51 +08:00
ENDIF()
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
ENDIF()
Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries"
2020-08-18 05:51:01 +08:00
## Addendum [5/9/23] It turns out that attempting to test S3 using a github action secret is a very complex process. So, this was disabled for github actions. However, a new *run_tests_s3.yml* action file was added that will eventually encapsulate S3 testing.
2023-05-10 11:13:49 +08:00
IF(NOT ENABLE_S3)
IF(WITH_S3_TESTING STREQUAL "PUBLIC" OR WITH_S3_TESTING)
message(WARNING "S3 support is disabled => WITH_S3_TESTING=OFF")
SET(WITH_S3_TESTING OFF CACHE STRING "" FORCE)
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
ENDIF()
Add zip file support The primary change is to support the use of a zip file as a storage format. Simultaneously the .nz4 support is made obsolete Use of zip requires the libzip support library, so a number of changes to the build files (Makefile.am, CMakeLists.txt) are necessary to locate and incorporate libzip. The nczarr_tests tests are also changed to add zip testing. Other changes: * Make sure distcheck leaves no files around. * Add some functions to netcdf_aux to export some functions of libnetcdf. * Add a new error NC_EFOUND as the complement of NC_EEMPTY. * Add tracing support to nclog and use it in libnczarr. * Modify the zmap interface to support the writeonce semantics of zip. * Create a new s3util.c to support a variety of S3 auxilliary functions. * EXTERNL'ize a number of functions so they can be used in s3util. * Add support for the S3 ListObjects CommonPrefixes mechanism to improve search. * Add experimental support for running nczarr X s3 tests against the actual Amazon S3 cloud.
2021-01-29 11:11:01 +08:00
ENDIF()
Replace ezxml with tinyxml2 re: PR https://github.com/Unidata/netcdf-c/pull/2139 re: PR https://github.com/Unidata/netcdf-c/pull/2169 re: PR https://github.com/Unidata/netcdf-c/pull/2146 re: Issue https://github.com/Unidata/netcdf-c/issues/2119 Found the product tinyxml2 at https://github.com/leethomason/tinyxml2.git and replaced ezxml with it. Tinyxml2 is about twice the LOC of ezxml, but at least is it still being maintained, and I can use it out of the box. It is C++ rather than C, but we seem to have reached the point that we can include C++ code with only minor compile flag changes. Untested on Mac OS. Added instructions to the end of libncxml/Makefile.am on how to upgrade to a later version of tinyxml2. This PR obsoletes the use of ezxml (re PRs https://github.com/Unidata/netcdf-c/pull/2146 and https://github.com/Unidata/netcdf-c/issue/2119).
2021-12-23 12:04:40 +08:00
OPTION(ENABLE_LIBXML2 "Link against libxml2 if it is available, use the packaged tinyxml2 parser otherwise." ON)
SET(XMLPARSER "tinyxml2 (bundled)")
Allow optional use of libxml2 re: https://github.com/Unidata/netcdf-c/issues/2119 H/T to [Egbert Eich](https://github.com/e4t) and [Bas Couwenberg](https://github.com/sebastic) for this PR. It is undesirable to make netcdf be dependent on the availability of libxml2, but it is desirable to allow its use if available. In order to do this, a wrapper API (include/ncxml.h) was constructed that supports either ezxml or libxml2 as the implementation. Additionally, the xml support code was moved to a new directory netcdf-c/libncxml. Primary changes: * Create a new sub-directory named netcdf-c/libncxml to hold all the xml implementation code. * Move ezxml.c and ezxml.h to libncxml * Create a wrapper API -- include/ncxml.h * Create an implementation, ncxml_ezxml.c to support use of ezxml. * Create an implementation, ncxml_xml2.c to support use of libxml2. * Add a check for libxml2 in configure.ac and CMakeLists.txt * Modify libdap to use the wrapper API instead of ezxml directly. Misc. Other Changes: * Change include/netcdf_json.h from built source to be part of the distribution.
2021-11-02 12:37:05 +08:00
# see if we have libxml2
Begin wiring in option to toggle automatic search for libxml2 in support of https://github.com/Unidata/netcdf-c/pulls/2135
2021-11-03 05:35:04 +08:00
IF(ENABLE_LIBXML2)
find_package(LibXml2)
IF(LibXml2_FOUND)
SET(HAVE_LIBXML2 TRUE)
INCLUDE_DIRECTORIES(${LIBXML2_INCLUDE_DIRS})
SET(XMLPARSER "libxml2")
ELSE()
SET(HAVE_LIBXML2 FALSE)
ENDIF()
ENDIF(ENABLE_LIBXML2)
Allow optional use of libxml2 re: https://github.com/Unidata/netcdf-c/issues/2119 H/T to [Egbert Eich](https://github.com/e4t) and [Bas Couwenberg](https://github.com/sebastic) for this PR. It is undesirable to make netcdf be dependent on the availability of libxml2, but it is desirable to allow its use if available. In order to do this, a wrapper API (include/ncxml.h) was constructed that supports either ezxml or libxml2 as the implementation. Additionally, the xml support code was moved to a new directory netcdf-c/libncxml. Primary changes: * Create a new sub-directory named netcdf-c/libncxml to hold all the xml implementation code. * Move ezxml.c and ezxml.h to libncxml * Create a wrapper API -- include/ncxml.h * Create an implementation, ncxml_ezxml.c to support use of ezxml. * Create an implementation, ncxml_xml2.c to support use of libxml2. * Add a check for libxml2 in configure.ac and CMakeLists.txt * Modify libdap to use the wrapper API instead of ezxml directly. Misc. Other Changes: * Change include/netcdf_json.h from built source to be part of the distribution.
2021-11-02 12:37:05 +08:00
Add zip file support The primary change is to support the use of a zip file as a storage format. Simultaneously the .nz4 support is made obsolete Use of zip requires the libzip support library, so a number of changes to the build files (Makefile.am, CMakeLists.txt) are necessary to locate and incorporate libzip. The nczarr_tests tests are also changed to add zip testing. Other changes: * Make sure distcheck leaves no files around. * Add some functions to netcdf_aux to export some functions of libnetcdf. * Add a new error NC_EFOUND as the complement of NC_EEMPTY. * Add tracing support to nclog and use it in libnczarr. * Modify the zmap interface to support the writeonce semantics of zip. * Create a new s3util.c to support a variety of S3 auxilliary functions. * EXTERNL'ize a number of functions so they can be used in s3util. * Add support for the S3 ListObjects CommonPrefixes mechanism to improve search. * Add experimental support for running nczarr X s3 tests against the actual Amazon S3 cloud.
2021-01-29 11:11:01 +08:00
IF(NOT ENABLE_BYTERANGE AND ENABLE_HDF5_ROS3)
MESSAGE(WARNING "ROS3 support requires ENABLE_BYTERANGE=TRUE; disabling ROS3 support")
SET(ENABLE_HDF5_ROS3 OFF CACHE BOOL "ROS3 support" FORCE)
ENDIF()
Added an advanced cmake option, ENABLE_FAILING_TESTS, which will be used to manage tests that are known to fail, but for which failure is not expected/desired. This way we can occasionally check to see if other fixes/changes have fixed failures we're aware of.
2015-04-30 01:04:26 +08:00
##
2012-10-03 04:56:46 +08:00
# Enable Tests
Added an advanced cmake option, ENABLE_FAILING_TESTS, which will be used to manage tests that are known to fail, but for which failure is not expected/desired. This way we can occasionally check to see if other fixes/changes have fixed failures we're aware of.
2015-04-30 01:04:26 +08:00
##
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
OPTION(ENABLE_TESTS "Enable basic tests, run with 'make test'." ON)
improve mechanism by which to find parallel HDF5 So far, CMakeLists.txt made the assumption that ${HDF5_LIB} is always defined in order to determine if it has parallel support. This is not the case, compare with bug #25. Make use of FIND_PACKAGE(HDF5) now; it sets HDF5_IS_PARALLEL anyways.
2014-03-07 23:40:50 +08:00
IF(ENABLE_TESTS)
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
SET(BUILD_TESTSETS ON CACHE BOOL "")
# Options for CTest-based tests, dashboards.
SET(NC_CTEST_PROJECT_NAME "netcdf-c" CACHE STRING "Project Name for CTest-based testing purposes.")
Adding new security stance to cdash instance.
2019-11-19 01:58:44 +08:00
SET(NC_CTEST_DROP_SITE "cdash.unidata.ucar.edu:443" CACHE STRING "Dashboard location for CTest-based testing purposes.")
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
SET(NC_CTEST_DROP_LOC_PREFIX "" CACHE STRING "Prefix for Dashboard location on remote server when using CTest-based testing.")
Ctest security stance.
2019-11-19 03:07:01 +08:00
SET(SUBMIT_URL "https://cdash.unidata.ucar.edu:443")
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
FIND_PROGRAM(HOSTNAME_CMD NAMES hostname)
hostname lacks arguments on mingw
2021-08-25 08:59:50 +08:00
IF(NOT WIN32)
Added a platform check for hostname on Windows. Updated how 'diff' is called in a diskless test to ignore whitespace when comparing files.
2015-02-05 00:11:27 +08:00
SET(HOSTNAME_ARG "-s")
Preparing for 4.3.3 release.
2015-02-13 04:16:54 +08:00
ENDIF()
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
IF(HOSTNAME_CMD)
Added a platform check for hostname on Windows. Updated how 'diff' is called in a diskless test to ignore whitespace when comparing files.
2015-02-05 00:11:27 +08:00
EXEC_PROGRAM(${HOSTNAME_CMD} ARGS "${HOSTNAME_ARG}" OUTPUT_VARIABLE HOSTNAME)
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
SET(NC_CTEST_SITE "${HOSTNAME}" CACHE STRING "Hostname of test machine.")
ENDIF()
IF(NC_CTEST_SITE)
SET(SITE "${NC_CTEST_SITE}" CACHE STRING "")
ENDIF()
Added an advanced cmake option, ENABLE_FAILING_TESTS, which will be used to manage tests that are known to fail, but for which failure is not expected/desired. This way we can occasionally check to see if other fixes/changes have fixed failures we're aware of.
2015-04-30 01:04:26 +08:00
###
# This option dictates whether or not to turn on
# tests which are known to fail. This is not the
# same thing as an 'expected failure'. Rather, these
# are tests that will need to be fixed eventually.
#
Spelling fixes
2019-09-18 10:27:43 +08:00
# By placing them here, we can occasionally turn this
Added an advanced cmake option, ENABLE_FAILING_TESTS, which will be used to manage tests that are known to fail, but for which failure is not expected/desired. This way we can occasionally check to see if other fixes/changes have fixed failures we're aware of.
2015-04-30 01:04:26 +08:00
# flag on and see if any known failures have been
# fixed in the course of code improvement/other bug
# fixes.
#
# To use this, simply add as a fencepost around tests
# which are known to fail.
###
OPTION(ENABLE_FAILING_TESTS "Run tests which are known to fail, check to see if any have been fixed." OFF)
Wiring unit test directory into cmake-based builds.
2019-08-22 04:50:09 +08:00
###
turned off unit tests for windows
2019-11-16 22:20:45 +08:00
# Option to turn on unit testing. See
Make use of clock_gettime be conditional. Re: GH Issue https://github.com/Unidata/netcdf-c/issues/1900 Apparently the clock_gettime() function is not always available. It is used in unit_test/tst_exhash.c and unit_test/tst_xcache.c. To solve this, a number of things were changed: * Move the timing code to a new file unit_tests/timer_utils.[ch] * Modify the timing code to choose one of several timing methods depending on availability. The prioritized order is as follows: 1. If Windows, use the QueryPerformanceCounter mechanism else 2. Use clock_gettime if available else 3. Use gettimeofday if available else 4. Use getrusage if available Note that the resolution of 3 and 4 is less than 1 or 2. Misc. Other Changes: * Move the test in CMakeLists.txt that disables unit tests for WIN32 to unit_test/CMakeLists.txt since some unit tests actually work under Visual Studio. * Fix some of the unit tests to work under visual studio * Fix problem with using remove() in zmap_nzf.c * Remove some warning about use of EXTERNL
2020-12-07 09:19:53 +08:00
# https://github.com/Unidata/netcdf-c/pull/1472 for more information.
Wiring unit test directory into cmake-based builds.
2019-08-22 04:50:09 +08:00
###
Added benchmarking option to cmake-based builds, turned on unit-testing by default
2023-02-17 02:20:45 +08:00
OPTION(ENABLE_UNIT_TESTS "Run Unit Tests." ON)
###
# Option to turn on performance testing.
# See https://github.com/Unidata/netcdf-c/issues/2627 for more information.
###
Turn benchmarks off by default. They require netcdf4, additional logic is required in order to have them on by default.
2023-02-17 07:41:19 +08:00
OPTION(ENABLE_BENCHMARKS "Run benchmark Tests." OFF)
Make use of clock_gettime be conditional. Re: GH Issue https://github.com/Unidata/netcdf-c/issues/1900 Apparently the clock_gettime() function is not always available. It is used in unit_test/tst_exhash.c and unit_test/tst_xcache.c. To solve this, a number of things were changed: * Move the timing code to a new file unit_tests/timer_utils.[ch] * Modify the timing code to choose one of several timing methods depending on availability. The prioritized order is as follows: 1. If Windows, use the QueryPerformanceCounter mechanism else 2. Use clock_gettime if available else 3. Use gettimeofday if available else 4. Use getrusage if available Note that the resolution of 3 and 4 is less than 1 or 2. Misc. Other Changes: * Move the test in CMakeLists.txt that disables unit tests for WIN32 to unit_test/CMakeLists.txt since some unit tests actually work under Visual Studio. * Fix some of the unit tests to work under visual studio * Fix problem with using remove() in zmap_nzf.c * Remove some warning about use of EXTERNL
2020-12-07 09:19:53 +08:00
Added an advanced cmake option, ENABLE_FAILING_TESTS, which will be used to manage tests that are known to fail, but for which failure is not expected/desired. This way we can occasionally check to see if other fixes/changes have fixed failures we're aware of.
2015-04-30 01:04:26 +08:00
###
# End known-failures.
###
MARK_AS_ADVANCED(ENABLE_FAILING_TESTS)
Merged changes from cmake branch. oc2 changes are just 'file property' changes, not sure what exactly.
2012-12-21 05:50:45 +08:00
ENDIF()
2012-10-03 04:56:46 +08:00
Added EXTREME_NUMBERS option to cmake, based on option in autotools, which had somehow not made it's way over previously.
2016-09-28 01:20:41 +08:00
###
# Option to enable extreme numbers during testing.
###
OPTION(ENABLE_EXTREME_NUMBERS "Enable extreme numbers during testing, such as MAX_INT-1" ON)
IF(ENABLE_EXTREME_NUMBERS)
SET(USE_EXTREME_NUMBERS ON)
ENDIF()
2012-10-03 04:56:46 +08:00
# Enable Large file tests
IF(ENABLE_LARGE_FILE_TESTS)
Merged changes from cmake branch. oc2 changes are just 'file property' changes, not sure what exactly.
2012-12-21 05:50:45 +08:00
SET(LARGE_FILE_TESTS ON)
2012-10-03 04:56:46 +08:00
ENDIF()
Test should be off by default
2018-07-11 00:58:52 +08:00
OPTION(ENABLE_METADATA_PERF_TESTS "Enable test of metadata performance." OFF)
Enable metadata_perf test in CMake build Currently defaults to enabled, can change top-level CMakeLists.txt file to change default to disabled
2018-07-10 23:25:09 +08:00
IF(ENABLE_METADATA_PERF_TESTS)
SET(ENABLE_METADATA_PERF ON)
ENDIF()
Merge from latest netcdf-cmake branch. - Fixes for Windows. - CMake-based changes (functionality, tests). - Smattering of coverity-informed bug-fixes.
2012-12-06 02:35:42 +08:00
# Location for large file tests.
SET(TEMP_LARGE "." CACHE STRING "Location to store large file tests.")
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
OPTION(ENABLE_FSYNC "Enable experimental fsync code." OFF)
2012-10-03 04:56:46 +08:00
IF(ENABLE_FSYNC)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
SET(USE_FSYNC ON)
2012-10-03 04:56:46 +08:00
ENDIF()
Added infrastructure to have cmake-based builds automatically generate PDF documentation when using Doxygen.
2014-06-19 04:33:49 +08:00
# Temporary
jna fixes
2014-04-10 05:20:16 +08:00
OPTION (ENABLE_JNA "Enable jna bug fix code." OFF)
IF(ENABLE_JNA)
SET(JNA ON)
ENDIF()
Merged changes from cmake branch. oc2 changes are just 'file property' changes, not sure what exactly.
2012-12-21 05:50:45 +08:00
# Linux specific large file support flags.
# Modelled after check in CMakeLists.txt for hdf5.
Updating for Windows Large File Support.
2013-10-17 06:44:11 +08:00
OPTION(ENABLE_LARGE_FILE_SUPPORT "Enable large file support." ON)
IF(ENABLE_LARGE_FILE_SUPPORT)
IF(MSVC)
SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} /LARGEADDRESSAWARE")
SET(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} /LARGEADDRESSAWARE")
SET(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} /LARGEADDRESSAWARE")
ELSE()
Corrected the way large file support was implemented in cmakelists.txt, not sure why it was like that to start with. In support of https://github.com/Unidata/netcdf-c/issues/385
2017-07-14 07:15:56 +08:00
SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64")
Merged changes from cmake branch. oc2 changes are just 'file property' changes, not sure what exactly.
2012-12-21 05:50:45 +08:00
ENDIF()
ENDIF()
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
OPTION(ENABLE_EXAMPLE_TESTS "Run extra example tests. Requires GNU Sed. Ignored if HDF5 is not Enabled" OFF)
IF(NOT ENABLE_HDF5 AND ENABLE_EXAMPLE_TESTS)
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
SET(ENABLE_EXAMPLE_TESTS OFF)
2012-10-03 04:56:46 +08:00
ENDIF()
added setting of mpiexec command to CMake build
2018-08-16 21:33:04 +08:00
# Enable Parallel IO with netCDF-4/HDF5 files using HDF5 parallel I/O.
Corrected parallel support in CMake, including library checks and parallel IO tests. Updated CMake-related documentation.
2013-02-13 06:22:54 +08:00
SET(STATUS_PARALLEL "OFF")
Github issue 152: https://github.com/Unidata/netcdf-c/issues/152 User request to have all orphaned DAP2 attributes kept as netcdf global attributes. This is primarily a change in the oc code nplus testcase dataset changes. Result may be inconsistent with netcdf-Java output.
2015-11-30 11:38:12 +08:00
OPTION(ENABLE_PARALLEL4 "Build netCDF-4 with parallel IO" "${HDF5_PARALLEL}")
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
IF(ENABLE_PARALLEL4 AND ENABLE_HDF5)
Github issue 152: https://github.com/Unidata/netcdf-c/issues/152 User request to have all orphaned DAP2 attributes kept as netcdf global attributes. This is primarily a change in the oc code nplus testcase dataset changes. Result may be inconsistent with netcdf-Java output.
2015-11-30 11:38:12 +08:00
IF(NOT HDF5_PARALLEL)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
SET(USE_PARALLEL OFF CACHE BOOL "")
MESSAGE(STATUS "Cannot find HDF5 library built with parallel support. Disabling parallel build.")
improve mechanism by which to find parallel HDF5 So far, CMakeLists.txt made the assumption that ${HDF5_LIB} is always defined in order to determine if it has parallel support. This is not the case, compare with bug #25. Make use of FIND_PACKAGE(HDF5) now; it sets HDF5_IS_PARALLEL anyways.
2014-03-07 23:40:50 +08:00
ELSE()
Call find_package(MPI) to locate MPI paths and link to libdispatch if required
2019-12-19 00:48:40 +08:00
FIND_PACKAGE(MPI REQUIRED)
Github issue 152: https://github.com/Unidata/netcdf-c/issues/152 User request to have all orphaned DAP2 attributes kept as netcdf global attributes. This is primarily a change in the oc code nplus testcase dataset changes. Result may be inconsistent with netcdf-Java output.
2015-11-30 11:38:12 +08:00
SET(HDF5_PARALLEL ON CACHE BOOL "")
Corrected placement of USE_PARALLEL, based on commit from nschloe: 5bf1848e8df6d58fa9c199a0e3d5ba30aa23302e
2014-04-04 06:57:17 +08:00
SET(USE_PARALLEL ON CACHE BOOL "")
Added PARALLEL4 flag to CMakeLists.
2015-10-10 04:13:50 +08:00
SET(USE_PARALLEL4 ON CACHE BOOL "")
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
SET(STATUS_PARALLEL "ON")
added setting of mpiexec command to CMake build
2018-08-16 21:33:04 +08:00
configure_file("${netCDF_SOURCE_DIR}/nc_test4/run_par_test.sh.in"
got par tests working in cmake build
2018-08-17 22:52:34 +08:00
"${netCDF_BINARY_DIR}/tmp/run_par_test.sh" @ONLY NEWLINE_STYLE LF)
FILE(COPY "${netCDF_BINARY_DIR}/tmp/run_par_test.sh"
DESTINATION ${netCDF_BINARY_DIR}/nc_test4
FILE_PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE)
Fix conflicts with master
2020-02-28 05:06:45 +08:00
configure_file("${netCDF_SOURCE_DIR}/h5_test/run_par_tests.sh.in"
"${netCDF_BINARY_DIR}/tmp/run_par_tests.sh" @ONLY NEWLINE_STYLE LF)
FILE(COPY "${netCDF_BINARY_DIR}/tmp/run_par_tests.sh"
DESTINATION ${netCDF_BINARY_DIR}/h5_test
FILE_PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
ENDIF()
Corrected parallel support in CMake, including library checks and parallel IO tests. Updated CMake-related documentation.
2013-02-13 06:22:54 +08:00
ENDIF()
Added new tests to CMake test system.
2013-02-07 07:09:19 +08:00
More replacement of "parallel-netcdf" with "PnetCDF".
2018-09-21 00:45:25 +08:00
# Options to enable parallel IO for classic formats with PnetCDF library.
Corrected parallel support in CMake, including library checks and parallel IO tests. Updated CMake-related documentation.
2013-02-13 06:22:54 +08:00
SET(STATUS_PNETCDF "OFF")
IF(ENABLE_PNETCDF)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
# Check for ncmpi_create in libpnetcdf, define USE_PNETCDF
Use PnetCDF instead of parallel-netcdf to avoid confusion with parallel netcdf4. Also, update PnetCDF web page.
2018-09-18 00:47:40 +08:00
# Does the user want to turn on PnetCDF read ability?
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
SET(USE_PNETCDF ON CACHE BOOL "")
FIND_LIBRARY(PNETCDF NAMES pnetcdf)
FIND_PATH(PNETCDF_INCLUDE_DIR pnetcdf.h)
IF(NOT PNETCDF)
add configure option --enable-relax-coord-bound for issue #243
2016-11-13 13:58:09 +08:00
MESSAGE(STATUS "Cannot find PnetCDF library. Disabling PnetCDF support.")
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
SET(USE_PNETCDF OFF CACHE BOOL "")
Corrected logic use of USE_PARALLEL for no-hdf5 but still using pnetcdf. Added fix for github issue http://github.com/Unidata/netcdf-c/issues/143 as suggested by Wei-keng Liao.
2015-11-10 02:21:04 +08:00
ELSE(NOT PNETCDF)
SET(USE_PARALLEL ON CACHE BOOL "")
Enable PNetCDF settings in cmake build One possible solution to finding the PNetCDF library and testing for the correct version in the cmake build.
2016-02-09 22:23:09 +08:00
add configure option --enable-relax-coord-bound for issue #243
2016-11-13 13:58:09 +08:00
# Check PnetCDF version. Must be >= 1.6.0
Enable PNetCDF settings in cmake build One possible solution to finding the PNetCDF library and testing for the correct version in the cmake build.
2016-02-09 22:23:09 +08:00
set(pnetcdf_h "${PNETCDF_INCLUDE_DIR}/pnetcdf.h" )
add configure option --enable-relax-coord-bound for issue #243
2016-11-13 13:58:09 +08:00
message(STATUS "PnetCDF include file ${pnetcdf_h} will be searched for version")
Enable PNetCDF settings in cmake build One possible solution to finding the PNetCDF library and testing for the correct version in the cmake build.
2016-02-09 22:23:09 +08:00
file(STRINGS "${pnetcdf_h}" pnetcdf_major_string REGEX "^#define PNETCDF_VERSION_MAJOR")
string(REGEX REPLACE "[^0-9]" "" pnetcdf_major "${pnetcdf_major_string}")
file(STRINGS "${pnetcdf_h}" pnetcdf_minor_string REGEX "^#define PNETCDF_VERSION_MINOR")
string(REGEX REPLACE "[^0-9]" "" pnetcdf_minor "${pnetcdf_minor_string}")
file(STRINGS "${pnetcdf_h}" pnetcdf_sub_string REGEX "^#define PNETCDF_VERSION_SUB")
string(REGEX REPLACE "[^0-9]" "" pnetcdf_sub "${pnetcdf_sub_string}")
set(pnetcdf_version "${pnetcdf_major}.${pnetcdf_minor}.${pnetcdf_sub}")
add configure option --enable-relax-coord-bound for issue #243
2016-11-13 13:58:09 +08:00
message(STATUS "Found PnetCDF version ${pnetcdf_version}")
Enable PNetCDF settings in cmake build One possible solution to finding the PNetCDF library and testing for the correct version in the cmake build.
2016-02-09 22:23:09 +08:00
Fix version dependency. Required PNetCDF version is 1.6.1 or greater; not 1.6.0 as I previously thought. Note that this is pull request may (partially) satisfy Issue #194 which I just found.
2016-02-10 02:23:33 +08:00
if(${pnetcdf_version} VERSION_GREATER "1.6.0")
Enable PNetCDF settings in cmake build One possible solution to finding the PNetCDF library and testing for the correct version in the cmake build.
2016-02-09 22:23:09 +08:00
SET(STATUS_PNETCDF "ON")
INCLUDE_DIRECTORIES(${PNETCDF_INCLUDE_DIR})
SET(HAVE_LIBPNETCDF ON)
Use PnetCDF instead of parallel-netcdf to avoid confusion with parallel netcdf4. Also, update PnetCDF web page.
2018-09-18 00:47:40 +08:00
# PnetCDF => parallel
Enable PNetCDF settings in cmake build One possible solution to finding the PNetCDF library and testing for the correct version in the cmake build.
2016-02-09 22:23:09 +08:00
SET(STATUS_PARALLEL ON)
SET(USE_PARALLEL ON)
add configure option --enable-relax-coord-bound for issue #243
2016-11-13 13:58:09 +08:00
MESSAGE(STATUS "Using PnetCDF Library: ${PNETCDF}")
Enable PNetCDF settings in cmake build One possible solution to finding the PNetCDF library and testing for the correct version in the cmake build.
2016-02-09 22:23:09 +08:00
ELSE()
add configure option --enable-relax-coord-bound for issue #243
2016-11-13 13:58:09 +08:00
MESSAGE(WARNING "ENABLE_PNETCDF requires version 1.6.1 or later; found version ${pnetcdf_version}. PnetCDF is disabled")
Added missing install for cmake-based builds, netcdf_mem.h. Also, moved install directives for header files into include/CMakeLists.txt. This is in support of https://github.com/Unidata/netcdf-c/issues/227
2016-02-24 02:12:16 +08:00
ENDIF()
Generate nc_test/run_pnetcdf_tests.sh when using cmake, in support of https://github.com/Unidata/netcdf-c/issues/2435
2022-06-29 04:56:38 +08:00
###
# Generate pnetcdf test.
###
CONFIGURE_FILE("${netCDF_SOURCE_DIR}/nc_test/run_pnetcdf_tests.sh.in"
"${netCDF_BINARY_DIR}/nc_test/run_pnetcdf_tests.sh")
Removed library version check for pnetcdf from cmake build, for the time being.
2015-09-18 04:02:15 +08:00
ENDIF(NOT PNETCDF)
Added new tests to CMake test system.
2013-02-07 07:09:19 +08:00
ENDIF()
Misc. source comment typos Some are user-facing. Found via `codespell` and through the downstream FreeCAD.
2018-03-16 22:38:40 +08:00
# Options to enable use of fill values for elements causing NC_ERANGE
error out when conflicts with PnetCDF are detected
2018-07-17 03:34:22 +08:00
SET(ENABLE_ERANGE_FILL AUTO CACHE STRING "AUTO")
OPTION(ENABLE_ERANGE_FILL "Enable use of fill value when out-of-range type conversion causes NC_ERANGE error." OFF)
IF(ENABLE_ERANGE_FILL) # enable or auto
STRING(TOUPPER ${ENABLE_ERANGE_FILL} ENABLE_ERANGE_FILL)
IF(ENABLE_ERANGE_FILL AND NOT ENABLE_ERANGE_FILL STREQUAL "AUTO")
# explicitly enabled
SET(ENABLE_ERANGE_FILL ON)
ELSE()
IF(NOT ENABLE_ERANGE_FILL STREQUAL "AUTO")
SET(ENABLE_ERANGE_FILL OFF)
ENDIF()
ENDIF()
ENDIF(ENABLE_ERANGE_FILL)
# Now ENABLE_ERANGE_FILL is either AUTO, ON, or OFF
check PnetCDF setting on erange-fill and adjust
2016-11-15 00:41:10 +08:00
cleanup of RELAX_COORD_BOUND in cmake build
2019-11-26 21:59:03 +08:00
# More relaxed coordinate check is now mandatory for all builds.
changed cmake build to only allow relaxed coord builds
2019-11-26 20:40:03 +08:00
SET(ENABLE_ZERO_LENGTH_COORD_BOUND ON)
check PnetCDF setting on erange-fill and adjust
2016-11-15 00:41:10 +08:00
# check and conform with PnetCDF settings on ERANGE_FILL and RELAX_COORD_BOUND
add configure option --enable-relax-coord-bound for issue #243
2016-11-13 13:58:09 +08:00
IF(STATUS_PNETCDF)
check PnetCDF setting on erange-fill and adjust
2016-11-15 00:41:10 +08:00
file(STRINGS "${pnetcdf_h}" enable_erange_fill_pnetcdf REGEX "^#define PNETCDF_ERANGE_FILL")
string(REGEX REPLACE "[^0-9]" "" erange_fill_pnetcdf "${enable_erange_fill_pnetcdf}")
IF("x${erange_fill_pnetcdf}" STREQUAL "x1")
error out when conflicts with PnetCDF are detected
2018-07-17 03:34:22 +08:00
SET(erange_fill_pnetcdf "ON")
ELSE()
SET(erange_fill_pnetcdf "OFF")
ENDIF()
IF(ENABLE_ERANGE_FILL STREQUAL "AUTO") # not set on command line
SET(ENABLE_ERANGE_FILL "${erange_fill_pnetcdf}")
check PnetCDF setting on erange-fill and adjust
2016-11-15 00:41:10 +08:00
ELSE()
error out when conflicts with PnetCDF are detected
2018-07-17 03:34:22 +08:00
# user explicitly set this option on command line
IF(NOT ENABLE_ERANGE_FILL STREQUAL "${erange_fill_pnetcdf}")
IF(ENABLE_ERANGE_FILL)
MESSAGE(FATAL_ERROR "Enabling erange-fill conflicts with PnetCDF setting")
ELSE()
MESSAGE(FATAL_ERROR "Disabling erange-fill conflicts with PnetCDF setting")
ENDIF()
check PnetCDF setting on erange-fill and adjust
2016-11-15 00:41:10 +08:00
ENDIF()
ENDIF()
error out when conflicts with PnetCDF are detected
2018-07-17 03:34:22 +08:00
add configure option --enable-relax-coord-bound for issue #243
2016-11-13 13:58:09 +08:00
file(STRINGS "${pnetcdf_h}" relax_coord_bound_pnetcdf REGEX "^#define PNETCDF_RELAX_COORD_BOUND")
string(REGEX REPLACE "[^0-9]" "" relax_coord_bound "${relax_coord_bound_pnetcdf}")
check PnetCDF setting on erange-fill and adjust
2016-11-15 00:41:10 +08:00
IF("x${relax_coord_bound}" STREQUAL "x1")
error out when conflicts with PnetCDF are detected
2018-07-17 03:34:22 +08:00
SET(relax_coord_bound_pnetcdf "ON")
add configure option --enable-relax-coord-bound for issue #243
2016-11-13 13:58:09 +08:00
ELSE()
error out when conflicts with PnetCDF are detected
2018-07-17 03:34:22 +08:00
SET(relax_coord_bound_pnetcdf "OFF")
ENDIF()
cleanup of RELAX_COORD_BOUND in cmake build
2019-11-26 21:59:03 +08:00
# pnetcdf must have relaxed coord bounds to build with netCDF-4
changed cmake build to only allow relaxed coord builds
2019-11-26 20:40:03 +08:00
IF(NOT ENABLE_ZERO_LENGTH_COORD_BOUND STREQUAL "${relax_coord_bound_pnetcdf}")
cleanup of RELAX_COORD_BOUND in cmake build
2019-11-26 21:59:03 +08:00
MESSAGE(FATAL_ERROR "Pnetcdf must be built with relax-coord-bound enabled")
add configure option --enable-relax-coord-bound for issue #243
2016-11-13 13:58:09 +08:00
ENDIF()
ENDIF()
check PnetCDF setting on erange-fill and adjust
2016-11-15 00:41:10 +08:00
error out when conflicts with PnetCDF are detected
2018-07-17 03:34:22 +08:00
IF(ENABLE_ERANGE_FILL)
MESSAGE(STATUS "Enabling use of fill value when NC_ERANGE")
check PnetCDF setting on erange-fill and adjust
2016-11-15 00:41:10 +08:00
SET(M4FLAGS "-DERANGE_FILL" CACHE STRING "")
ENDIF()
resolve a merge conflict
2018-08-28 06:57:07 +08:00
IF(ENABLE_ZERO_LENGTH_COORD_BOUND)
spell check
2018-07-29 05:10:07 +08:00
MESSAGE(STATUS "Enabling a more relaxed check for NC_EINVALCOORDS")
add configure option --enable-relax-coord-bound for issue #243
2016-11-13 13:58:09 +08:00
ADD_DEFINITIONS(-DRELAX_COORD_BOUND)
ENDIF()
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
# Enable Parallel Tests.
Added hard failure to script, modified cmake default.
2020-04-02 06:20:33 +08:00
OPTION(ENABLE_PARALLEL_TESTS "Enable Parallel IO Tests. Requires HDF5/NetCDF4 with parallel I/O Support." "${HDF5_PARALLEL}")
Corrected parallel support in CMake, including library checks and parallel IO tests. Updated CMake-related documentation.
2013-02-13 06:22:54 +08:00
IF(ENABLE_PARALLEL_TESTS AND USE_PARALLEL)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
SET(TEST_PARALLEL ON CACHE BOOL "")
Got distcheck working, corrected some parallel test logic in cmakelists.
2016-06-10 03:21:28 +08:00
IF(USE_NETCDF4)
SET(TEST_PARALLEL4 ON CACHE BOOL "")
ENDIF()
Corrected logic error in cmake.
2018-02-27 08:16:15 +08:00
ENDIF()
IF (ENABLE_PARALLEL_TESTS AND NOT USE_PARALLEL)
Updated ac, cmake based builds to require parallel functionality when parallel tests are requested.
2018-02-27 08:08:26 +08:00
MESSAGE(FATAL_ERROR "Parallel tests requested, but no parallel HDF5 installation detected.")
Corrected parallel support in CMake, including library checks and parallel IO tests. Updated CMake-related documentation.
2013-02-13 06:22:54 +08:00
ENDIF()
2012-10-03 04:56:46 +08:00
Begin changing over to use strlcat instead of strncat because strlcat provides better protection against buffer overflows. Code is taken from the FreeBSD project source code. Specifically: https://github.com/freebsd/freebsd/blob/master/lib/libc/string/strlcat.c License appears to be acceptable, but needs to be checked by e.g. Debian. Step 1: 1. Add to netcdf-c/include/ncconfigure.h to use our version if not already available as determined by HAVE_STRLCAT in config.h. 2. Add the strlcat code to libdispatch/dstring.c 3. Turns out that strlcat was already defined in several places. So remove it from: ncgen3/genlib.c ncdump/dumplib.c 3. Define strlcat extern definition in ncconfigure.h. 4. Modify following directories to use strlcat: libdap2 libdap4 ncdap_test dap4_test Will do others in subsequent steps.
2017-11-24 01:55:24 +08:00
# Enable special filter test; experimental when using cmake.
CMakeLists.txt use ON vs YES, indent In CMake logically ON, YES, TRUE, 1 are equivalent. But many CMake editors won't syntax highlight a lower-case "yes", and CMake-gui will change them to ON, creating unnecessary changes to CMakeCache.txt. Also indent a WIN32 if/then block.
2023-03-20 09:02:58 +08:00
OPTION(ENABLE_FILTER_TESTING "Enable filter testing. Ignored if shared libraries or netCDF4 are not enabled" ON)
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
IF(ENABLE_FILTER_TESTING)
if(NOT ENABLE_HDF5 AND NOT ENABLE_NCZARR)
MESSAGE(WARNING "ENABLE_FILTER_TESTING requires HDF5 and/or NCZarr. Disabling.")
SET(ENABLE_FILTER_TESTING OFF CACHE BOOL "Enable Filter Testing" FORCE)
ENDIF()
1. Keep up to date by merging master 2. Fixed plugin building (nc_test4/hdf5plugins) to be done properly by cmake and automake. 4. Duplicated part of the nc_test4 filter test code in examples/C An incomplete and untested set of hooks exist for OS-X in nc_test4/findplugins.in. They need testing.
2018-01-17 02:00:09 +08:00
ENDIF()
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
1. Keep up to date by merging master 2. Fixed plugin building (nc_test4/hdf5plugins) to be done properly by cmake and automake. 4. Duplicated part of the nc_test4 filter test code in examples/C An incomplete and untested set of hooks exist for OS-X in nc_test4/findplugins.in. They need testing.
2018-01-17 02:00:09 +08:00
IF(NOT BUILD_SHARED_LIBS)
MESSAGE(WARNING "ENABLE_FILTER_TESTING requires shared libraries. Disabling.")
Provide byte-range reading of remote datasets re: issue https://github.com/Unidata/netcdf-c/issues/1251 Assume that you have the URL to a remote dataset which is a normal netcdf-3 or netcdf-4 file. This PR allows the netcdf-c to read that dataset's contents as a netcdf file using HTTP byte ranges if the remote server supports byte-range access. Originally, this PR was set up to access Amazon S3 objects, but it can also access other remote datasets such as those provided by a Thredds server via the HTTPServer access protocol. It may also work for other kinds of servers. Note that this is not intended as a true production capability because, as is known, this kind of access to can be quite slow. In addition, the byte-range IO drivers do not currently do any sort of optimization or caching. An additional goal here is to gain some experience with the Amazon S3 REST protocol. This architecture and its use documented in the file docs/byterange.dox. There are currently two test cases: 1. nc_test/tst_s3raw.c - this does a simple open, check format, close cycle for a remote netcdf-3 file and a remote netcdf-4 file. 2. nc_test/test_s3raw.sh - this uses ncdump to investigate some remote datasets. This PR also incorporates significantly changed model inference code (see the superceded PR https://github.com/Unidata/netcdf-c/pull/1259). 1. It centralizes the code that infers the dispatcher. 2. It adds support for byte-range URLs Other changes: 1. NC_HDF5_finalize was not being properly called by nc_finalize(). 2. Fix minor bug in ncgen3.l 3. fix memory leak in nc4info.c 4. add code to walk the .daprc triples and to replace protocol= fragment tag with a more general mode= tag. Final Note: Th inference code is still way too complicated. We need to move to the validfile() model used by netcdf Java, where each dispatcher is asked if it can process the file. This decentralizes the inference code. This will be done after all the major new dispatchers (PIO, Zarr, etc) have been implemented.
2019-01-02 09:27:36 +08:00
SET(ENABLE_FILTER_TESTING OFF)
Begin changing over to use strlcat instead of strncat because strlcat provides better protection against buffer overflows. Code is taken from the FreeBSD project source code. Specifically: https://github.com/freebsd/freebsd/blob/master/lib/libc/string/strlcat.c License appears to be acceptable, but needs to be checked by e.g. Debian. Step 1: 1. Add to netcdf-c/include/ncconfigure.h to use our version if not already available as determined by HAVE_STRLCAT in config.h. 2. Add the strlcat code to libdispatch/dstring.c 3. Turns out that strlcat was already defined in several places. So remove it from: ncgen3/genlib.c ncdump/dumplib.c 3. Define strlcat extern definition in ncconfigure.h. 4. Modify following directories to use strlcat: libdap2 libdap4 ncdap_test dap4_test Will do others in subsequent steps.
2017-11-24 01:55:24 +08:00
ENDIF()
Allow redefinition of variable filters re: Github issue https://github.com/Unidata/netcdf-c/issues/1713 If nc_def_var_filter or nc_def_var_deflate or nc_def_var_szip is called multiple times with the same filter id, but possibly with different sets of parameters, then the first invocation is sticky and later invocations are ignored. The desired behavior is to have the last invocation be used. This PR implements that desired behavior, with some special cases. If you call nc_def_var_deflate multiple times, then the last invocation rule applies with respect to deflate. However, the shuffle filter, if enabled, is always applied just before applying deflate. Misc unrelated changes: 1. Make client-side filters be disabled by default 2. Fix the definition of uintptr_t and use in oc2 and libdap4 3. Add some test cases 4. modify filter order tests to use plugin filters rather than client-side filters
2020-05-11 23:42:31 +08:00
CMakeLists.txt use ON vs YES, indent In CMake logically ON, YES, TRUE, 1 are equivalent. But many CMake editors won't syntax highlight a lower-case "yes", and CMake-gui will change them to ON, creating unnecessary changes to CMakeCache.txt. Also indent a WIN32 if/then block.
2023-03-20 09:02:58 +08:00
OPTION(ENABLE_NCZARR_FILTERS "Enable NCZarr filters" ON)
OPTION(ENABLE_NCZARR_FILTERS_TESTING "Enable NCZarr filter testing." ON)
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
# Constraints
CMake: CMP0075 requires CMake >= 3.12. The presence of cmake_policy(CMP0075 required CMake >=3.12, and errors with older CMake. Since older versions of CMake than 3.12 are already broken with NetCDF and uncommon nowadays, it may be better to explicitly require 3.12 or newer.
2022-02-08 00:14:57 +08:00
IF (NOT ENABLE_PLUGINS)
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
MESSAGE(WARNING "ENABLE_FILTER_TESTING requires shared libraries. Disabling.")
SET(ENABLE_NCZARR_FILTERS OFF CACHE BOOL "Enable NCZarr Filters." FORCE)
ENDIF()
IF (NOT ENABLE_NCZARR)
MESSAGE(WARNING "ENABLE_NCZARR==NO => ENABLE_NCZARR_FILTERS==NO AND ENABLE_NCZARR_FILTER_TESTING==NO")
SET(ENABLE_NCZARR_FILTERS OFF CACHE BOOL "Disable NCZARR_FILTERS" FORCE)
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
ENDIF()
CMake: CMP0075 requires CMake >= 3.12. The presence of cmake_policy(CMP0075 required CMake >=3.12, and errors with older CMake. Since older versions of CMake than 3.12 are already broken with NetCDF and uncommon nowadays, it may be better to explicitly require 3.12 or newer.
2022-02-08 00:14:57 +08:00
IF (NOT ENABLE_NCZARR_FILTERS)
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
SET(ENABLE_NCZARR_FILTER_TESTING OFF CACHE BOOL "Enable NCZarr Filter Testing" FORCE)
ENDIF()
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
SET(ENABLE_CLIENTSIDE_FILTERS OFF)
Begin changing over to use strlcat instead of strncat because strlcat provides better protection against buffer overflows. Code is taken from the FreeBSD project source code. Specifically: https://github.com/freebsd/freebsd/blob/master/lib/libc/string/strlcat.c License appears to be acceptable, but needs to be checked by e.g. Debian. Step 1: 1. Add to netcdf-c/include/ncconfigure.h to use our version if not already available as determined by HAVE_STRLCAT in config.h. 2. Add the strlcat code to libdispatch/dstring.c 3. Turns out that strlcat was already defined in several places. So remove it from: ncgen3/genlib.c ncdump/dumplib.c 3. Define strlcat extern definition in ncconfigure.h. 4. Modify following directories to use strlcat: libdap2 libdap4 ncdap_test dap4_test Will do others in subsequent steps.
2017-11-24 01:55:24 +08:00
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
# Determine whether or not to generate documentation.
A bit of whitespace cleanup, clarification of messages in CMake-based builds.
2014-06-19 03:46:22 +08:00
OPTION(ENABLE_DOXYGEN "Enable generation of doxygen-based documentation." OFF)
2012-10-03 04:56:46 +08:00
IF(ENABLE_DOXYGEN)
make Doxygen a requirement if documentation is to be built
2014-07-16 16:25:27 +08:00
FIND_PACKAGE(Doxygen REQUIRED)
# Offer the option to build internal documentation.
OPTION(ENABLE_INTERNAL_DOCS "Build internal documentation. This is of interest to developers only." OFF)
IF(ENABLE_INTERNAL_DOCS)
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
SET(BUILD_INTERNAL_DOCS yes CACHE STRING "")
make Doxygen a requirement if documentation is to be built
2014-07-16 16:25:27 +08:00
ELSE()
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
SET(BUILD_INTERNAL_DOCS no CACHE STRING "")
make Doxygen a requirement if documentation is to be built
2014-07-16 16:25:27 +08:00
ENDIF()
Added infrastructure to have cmake-based builds automatically generate PDF documentation when using Doxygen.
2014-06-19 04:33:49 +08:00
Merging Jen's new search option for releases into user documentation.
2015-08-18 00:59:32 +08:00
###
#
# If we are building release documentation, we need to set some
# variables that will be used in the Doxygen.in template.
###
OPTION(ENABLE_DOXYGEN_BUILD_RELEASE_DOCS "Build release documentation. This is of interest only to the netCDF developers." OFF)
IF(ENABLE_DOXYGEN_BUILD_RELEASE_DOCS)
SET(DOXYGEN_CSS_FILE "${CMAKE_SOURCE_DIR}/docs/release.css" CACHE STRING "")
SET(DOXYGEN_HEADER_FILE "${CMAKE_SOURCE_DIR}/docs/release_header.html" CACHE STRING "")
SET(DOXYGEN_SEARCHENGINE "NO" CACHE STRING "")
Re-adding server-side search option.
2015-11-03 03:46:09 +08:00
SET(ENABLE_DOXYGEN_SERVER_BASED_SEARCH NO CACHE STRING "")
Added a new .gitignore specific to include/. Added netcdf_meta.h to user documentation generated by doxygen.
2014-09-19 05:05:08 +08:00
ELSE()
Merging Jen's new search option for releases into user documentation.
2015-08-18 00:59:32 +08:00
SET(DOXYGEN_CSS_FILE "" CACHE STRING "")
SET(DOXYGEN_HEADER_FILE "" CACHE STRING "")
SET(DOXYGEN_SEARCHENGINE "YES" CACHE STRING "")
Re-adding server-side search option.
2015-11-03 03:46:09 +08:00
# If not using release document configuration,
# provide an option for server-based search.
OPTION(ENABLE_DOXYGEN_SERVER_SIDE_SEARCH "Configure Doxygen with server-based search." OFF)
IF(ENABLE_DOXYGEN_SERVER_SIDE_SEARCH)
Reintroduced ENABLE_DOXYGEN_SERVER_BASED_SEARCH option for local webserver use.
2015-11-03 03:58:29 +08:00
SET(DOXYGEN_SERVER_BASED_SEARCH "YES" CACHE STRING "")
Re-adding server-side search option.
2015-11-03 03:46:09 +08:00
ELSE()
Reintroduced ENABLE_DOXYGEN_SERVER_BASED_SEARCH option for local webserver use.
2015-11-03 03:58:29 +08:00
SET(DOXYGEN_SERVER_BASED_SEARCH "NO" CACHE STRING "")
Re-adding server-side search option.
2015-11-03 03:46:09 +08:00
ENDIF(ENABLE_DOXYGEN_SERVER_SIDE_SEARCH)
Merging Jen's new search option for releases into user documentation.
2015-08-18 00:59:32 +08:00
ENDIF(ENABLE_DOXYGEN_BUILD_RELEASE_DOCS)
make Doxygen a requirement if documentation is to be built
2014-07-16 16:25:27 +08:00
# Option to turn on the TODO list in the doxygen-generated documentation.
Added a CMake option for use at release time. DOXYGEN_ENABLE_SERVER_SIDE_SEARCH turns on server side search in the doxygen documentation. This is off by default, but when on provides a much more powerful search than the default.
2014-09-04 02:52:10 +08:00
OPTION(DOXYGEN_ENABLE_TASKS "Turn on test, todo, bug lists in documentation. This is of interest to developers only." OFF)
IF(DOXYGEN_ENABLE_TASKS)
make Doxygen a requirement if documentation is to be built
2014-07-16 16:25:27 +08:00
SET(SHOW_DOXYGEN_TAG_LIST YES CACHE STRING "")
Added a CMake option for use at release time. DOXYGEN_ENABLE_SERVER_SIDE_SEARCH turns on server side search in the doxygen documentation. This is off by default, but when on provides a much more powerful search than the default.
2014-09-04 02:52:10 +08:00
ELSE(DOXYGEN_ENABLE_TASKS)
make Doxygen a requirement if documentation is to be built
2014-07-16 16:25:27 +08:00
SET(SHOW_DOXYGEN_TODO_LIST NO CACHE STRING "")
Added a CMake option for use at release time. DOXYGEN_ENABLE_SERVER_SIDE_SEARCH turns on server side search in the doxygen documentation. This is off by default, but when on provides a much more powerful search than the default.
2014-09-04 02:52:10 +08:00
ENDIF(DOXYGEN_ENABLE_TASKS)
Added a new cmake-based option, ENABLE_DOXYGEN_TODO_LIST. When used, the generated documentation will include a 'todo' page. This is generally for internal consumption only, so that pages and sections that need review or editing can be flagged.
2014-02-28 05:19:28 +08:00
make Doxygen a requirement if documentation is to be built
2014-07-16 16:25:27 +08:00
OPTION(ENABLE_DOXYGEN_PDF_OUTPUT "[EXPERIMENTAL] Turn on PDF output for Doxygen-generated documentation." OFF)
Added infrastructure to have cmake-based builds automatically generate PDF documentation when using Doxygen.
2014-06-19 04:33:49 +08:00
make Doxygen a requirement if documentation is to be built
2014-07-16 16:25:27 +08:00
IF(ENABLE_DOXYGEN_PDF_OUTPUT)
SET(NC_ENABLE_DOXYGEN_PDF_OUTPUT "YES" CACHE STRING "")
ELSE()
SET(NC_ENABLE_DOXYGEN_PDF_OUTPUT "NO" CACHE STRING "")
ENDIF()
Added infrastructure to have cmake-based builds automatically generate PDF documentation when using Doxygen.
2014-06-19 04:33:49 +08:00
Reorganized some stuff in CMakeLists.txt
2014-08-13 06:34:11 +08:00
FIND_PROGRAM(NC_DOT NAMES dot)
make Doxygen a requirement if documentation is to be built
2014-07-16 16:25:27 +08:00
# Specify whether or not 'dot' was found on the system path.
IF(NC_DOT)
SET(HAVE_DOT YES CACHE STRING "")
ELSE(NC_DOT)
SET(HAVE_DOT NO CACHE STRING "")
ENDIF(NC_DOT)
2012-10-03 04:56:46 +08:00
ENDIF()
Fix https://github.com/Unidata/netcdf-c/issues/963 Fix https://github.com/Unidata/netcdf-c/issues/962 1. remove the --disable-diskless option since it is no longer needed. Similarly for CMakeLists.txt. 2. Fixed nc4files.c where BAIL and return were mixed leading to situation where cleanup code was not being invoked. This probably occurs elsewhere, but I did not find any specifically.
2018-05-12 05:30:19 +08:00
# Always enable DISKLESS
OPTION(ENABLE_DISKLESS "Enable in-memory files" ON)
getting ready for next try at quantization code
2021-08-24 14:45:38 +08:00
# Always enable quantization.
OPTION(ENABLE_QUANTIZE "Enable variable quantization" ON)
Added new variable to MSVC-based builds, NC_MSVC_STACK_SIZE. This variable allows the user to change the default stack size of 1000000.
2013-10-10 06:02:13 +08:00
# By default, MSVC has a stack size of 1000000.
# Allow a user to override this.
IF(MSVC)
Added definition for Windows systems to use 64-bit fstat.
2013-11-01 01:10:20 +08:00
SET(NC_MSVC_STACK_SIZE 40000000 CACHE STRING "Default stack size for MSVC-based projects.")
Removed detritus from CMakeLists.txt
2013-10-10 06:05:50 +08:00
# By default, CMake sets the stack to 1000000.
# Remove this limitation.
# See here for more details:
# http://www.cmake.org/pipermail/cmake/2009-April/028710.html
Added new variable to MSVC-based builds, NC_MSVC_STACK_SIZE. This variable allows the user to change the default stack size of 1000000.
2013-10-10 06:02:13 +08:00
SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} /STACK:${NC_MSVC_STACK_SIZE}")
SET(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} /STACK:${NC_MSVC_STACK_SIZE}")
SET(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} /STACK:${NC_MSVC_STACK_SIZE}")
ENDIF()
2012-10-03 04:56:46 +08:00
# Set some of the options as advanced.
Reintroduced ENABLE_DOXYGEN_SERVER_BASED_SEARCH option for local webserver use.
2015-11-03 03:58:29 +08:00
MARK_AS_ADVANCED(ENABLE_INTERNAL_DOCS VALGRIND_TESTS ENABLE_COVERAGE_TESTS )
Add support for Zarr string type to NCZarr * re: https://github.com/Unidata/netcdf-c/pull/2278 * re: https://github.com/Unidata/netcdf-c/issues/2485 * re: https://github.com/Unidata/netcdf-c/issues/2474 This PR subsumes PR https://github.com/Unidata/netcdf-c/pull/2278. Actually is a bit an omnibus covering several issues. ## PR https://github.com/Unidata/netcdf-c/pull/2278 Add support for the Zarr string type. Zarr strings are restricted currently to be of fixed size. The primary issue to be addressed is to provide a way for user to specify the size of the fixed length strings. This is handled by providing the following new attributes special: 1. **_nczarr_default_maxstrlen** &mdash; This is an attribute of the root group. It specifies the default maximum string length for string types. If not specified, then it has the value of 64 characters. 2. **_nczarr_maxstrlen** &mdash; This is a per-variable attribute. It specifies the maximum string length for the string type associated with the variable. If not specified, then it is assigned the value of **_nczarr_default_maxstrlen**. This PR also requires some hacking to handle the existing netcdf-c NC_CHAR type, which does not exist in zarr. The goal was to choose numpy types for both the netcdf-c NC_STRING type and the netcdf-c NC_CHAR type such that if a pure zarr implementation read them, it would still work and an NC_CHAR type would be handled by zarr as a string of length 1. For writing variables and NCZarr attributes, the type mapping is as follows: * "|S1" for NC_CHAR. * ">S1" for NC_STRING && MAXSTRLEN==1 * ">Sn" for NC_STRING && MAXSTRLEN==n Note that it is a bit of a hack to use endianness, but it should be ok since for string/char, the endianness has no meaning. For reading attributes with pure zarr (i.e. with no nczarr atribute types defined), they will always be interpreted as of type NC_CHAR. ## Issue: https://github.com/Unidata/netcdf-c/issues/2474 This PR partly fixes this issue because it provided more comprehensive support for Zarr attributes that are JSON valued expressions. This PR still does not address the problem in that issue where the _ARRAY_DIMENSION attribute is incorrectly set. Than can only be fixed by the creator of the datasets. ## Issue: https://github.com/Unidata/netcdf-c/issues/2485 This PR also fixes the scalar failure shown in this issue. It generally cleans up scalar handling. It also adds a note to the documentation describing that NCZarr supports scalars while Zarr does not and also how scalar interoperability is achieved. ## Misc. Other Changes 1. Convert the nczarr special attributes and keys to be all lower case. So "_NCZARR_ATTR" now used "_nczarr_attr. Support back compatibility for the upper case names. 2. Cleanup my too-clever-by-half handling of scalars in libnczarr.
2022-08-28 10:21:13 +08:00
MARK_AS_ADVANCED(ENABLE_DAP_REMOTE_TESTS ENABLE_DAP_LONG_TESTS USE_REMOTE_CDASH ENABLE_EXTERNAL_SERVER_TESTS)
Reintroduced ENABLE_DOXYGEN_SERVER_BASED_SEARCH option for local webserver use.
2015-11-03 03:58:29 +08:00
MARK_AS_ADVANCED(ENABLE_DOXYGEN_BUILD_RELEASE_DOCS DOXYGEN_ENABLE_TASKS ENABLE_DOXYGEN_SERVER_SIDE_SEARCH)
Adding cmake infrastructure in support of https://github.com/Unidata/netcdf-c/issues/228
2016-03-08 05:04:52 +08:00
MARK_AS_ADVANCED(ENABLE_SHARED_LIBRARY_VERSION)
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
Reorganizing CMakeLists.txt for clarity and expanded options. Attempting to clean up some warnings on output, and to make it easier to find things within the file.
2013-10-01 05:51:34 +08:00
################################
# Option checks
################################
2012-10-03 04:56:46 +08:00
# Library include checks
CHECK_INCLUDE_FILE("math.h" HAVE_MATH_H)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_INCLUDE_FILE("unistd.h" HAVE_UNISTD_H)
Merging CMake branch into trunk. Changes include: CMake related changes in CMakeLists.txt files, cmake_config.h.in. Other changes relate to Windows-specific issues, and changes made when regenerating generated source files.
2013-01-16 06:43:09 +08:00
# Solve a compatibility issue in ncgen/, which checks
# for NO_UNISTD_H
IF(NOT HAVE_UNISTD_H)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
SET(YY_NO_UNISTD_H TRUE)
Merging CMake branch into trunk. Changes include: CMake related changes in CMakeLists.txt files, cmake_config.h.in. Other changes relate to Windows-specific issues, and changes made when regenerating generated source files.
2013-01-16 06:43:09 +08:00
ENDIF()
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_INCLUDE_FILE("alloca.h" HAVE_ALLOCA_H)
Merged latest changes from netcdf-cmake branch. Includes a few windows fixes, plus the skeleton for 'make dist' and 'make distcheck' support in cmake-based builds.
2013-03-28 03:15:00 +08:00
CHECK_INCLUDE_FILE("malloc.h" HAVE_MALLOC_H)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_INCLUDE_FILE("fcntl.h" HAVE_FCNTL_H)
CHECK_INCLUDE_FILE("getopt.h" HAVE_GETOPT_H)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
CHECK_INCLUDE_FILE("locale.h" HAVE_LOCALE_H)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_INCLUDE_FILE("stdint.h" HAVE_STDINT_H)
CHECK_INCLUDE_FILE("stdio.h" HAVE_STDIO_H)
Updates for Visual-Studio-based builds.
2017-03-17 04:34:33 +08:00
IF(MSVC)
CHECK_INCLUDE_FILE("io.h" HAVE_IO_H)
ENDIF(MSVC)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_INCLUDE_FILE("stdlib.h" HAVE_STDLIB_H)
Merged changes from cmake branch. oc2 changes are just 'file property' changes, not sure what exactly.
2012-12-21 05:50:45 +08:00
CHECK_INCLUDE_FILE("stdarg.h" HAVE_STDARG_H)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_INCLUDE_FILE("strings.h" HAVE_STRINGS_H)
CHECK_INCLUDE_FILE("signal.h" HAVE_SIGNAL_H)
2012-10-03 04:56:46 +08:00
CHECK_INCLUDE_FILE("sys/param.h" HAVE_SYS_PARAM_H)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_INCLUDE_FILE("sys/stat.h" HAVE_SYS_STAT_H)
CHECK_INCLUDE_FILE("sys/time.h" HAVE_SYS_TIME_H)
2012-10-03 04:56:46 +08:00
CHECK_INCLUDE_FILE("sys/types.h" HAVE_SYS_TYPES_H)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
CHECK_INCLUDE_FILE("sys/mman.h" HAVE_SYS_MMAN_H)
2012-10-03 04:56:46 +08:00
CHECK_INCLUDE_FILE("sys/resource.h" HAVE_SYS_RESOURCE_H)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_INCLUDE_FILE("fcntl.h" HAVE_FCNTL_H)
2012-10-03 04:56:46 +08:00
CHECK_INCLUDE_FILE("inttypes.h" HAVE_INTTYPES_H)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_INCLUDE_FILE("pstdint.h" HAVE_PSTDINT_H)
CHECK_INCLUDE_FILE("endian.h" HAVE_ENDIAN_H)
Merged latest changes from cmake development branch. Addressed a number of memory-related problems reported by Coverity.
2013-04-03 06:09:31 +08:00
CHECK_INCLUDE_FILE("BaseTsd.h" HAVE_BASETSD_H)
Added a check for stddef.h in CMakeLists.txt; some linux versions do not store stddef in /usr/include, but rather /usr/include/linux.
2014-02-08 00:51:47 +08:00
CHECK_INCLUDE_FILE("stddef.h" HAVE_STDDEF_H)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
CHECK_INCLUDE_FILE("string.h" HAVE_STRING_H)
Accomodating Windows winsock issue.
2017-07-27 03:40:03 +08:00
CHECK_INCLUDE_FILE("winsock2.h" HAVE_WINSOCK2_H)
re: hithub issue https://github.com/Unidata/netcdf-c/issues/435 Some temporary files are being left in a tempdir (e.g. /tmp under *nix*). The situation is described tersely in netcdf-c/docs/auth.html#REDIR Basically, when a url is used that requires redirection, a physical cookiejar file is required to exist in the file system in order for this to work. Since it was difficult to figure out when redirection was being used (it was internal to libcurl) I needed to be prepared for that eventuality. The result was that I always created a cookiejar file if one was not specified in the rc file. This actually occurs in two places: one inside oc2 and one inside libdap4. The solution was two-fold: 1. do not use a cookiejar directory -- create cookiejar file directly 2. ensure that all cookiejar related files are reclaimed by nc_close(). Note that if nc_close (or nc_abort) is not called for whatever reason, then reclamation will not occur.
2017-07-06 00:03:48 +08:00
CHECK_INCLUDE_FILE("ftw.h" HAVE_FTW_H)
Updated typo.
2018-02-03 11:27:06 +08:00
CHECK_INCLUDE_FILE("libgen.h" HAVE_LIBGEN_H)
This PR adds EXPERIMENTAL support for accessing data in the cloud using a variant of the Zarr protocol and storage format. This enhancement is generically referred to as "NCZarr". The data model supported by NCZarr is netcdf-4 minus the user-defined types and the String type. In this sense it is similar to the CDF-5 data model. More detailed information about enabling and using NCZarr is described in the document NUG/nczarr.md and in a [Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in). WARNING: this code has had limited testing, so do use this version for production work. Also, performance improvements are ongoing. Note especially the following platform matrix of successful tests: Platform | Build System | S3 support ------------------------------------ Linux+gcc | Automake | yes Linux+gcc | CMake | yes Visual Studio | CMake | no Additionally, and as a consequence of the addition of NCZarr, major changes have been made to the Filter API. NOTE: NCZarr does not yet support filters, but these changes are enablers for that support in the future. Note that it is possible (probable?) that there will be some accidental reversions if the changes here did not correctly mimic the existing filter testing. In any case, previously filter ids and parameters were of type unsigned int. In order to support the more general zarr filter model, this was all converted to char*. The old HDF5-specific, unsigned int operations are still supported but they are wrappers around the new, char* based nc_filterx_XXX functions. This entailed at least the following changes: 1. Added the files libdispatch/dfilterx.c and include/ncfilter.h 2. Some filterx utilities have been moved to libdispatch/daux.c 3. A new entry, "filter_actions" was added to the NCDispatch table and the version bumped. 4. An overly complex set of structs was created to support funnelling all of the filterx operations thru a single dispatch "filter_actions" entry. 5. Move common code to from libhdf5 to libsrc4 so that it is accessible to nczarr. Changes directly related to Zarr: 1. Modified CMakeList.txt and configure.ac to support both C and C++ -- this is in support of S3 support via the awd-sdk libraries. 2. Define a size64_t type to support nczarr. 3. More reworking of libdispatch/dinfermodel.c to support zarr and to regularize the structure of the fragments section of a URL. Changes not directly related to Zarr: 1. Make client-side filter registration be conditional, with default off. 2. Hack include/nc4internal.h to make some flags added by Ed be unique: e.g. NC_CREAT, NC_INDEF, etc. 3. cleanup include/nchttp.h and libdispatch/dhttp.c. 4. Misc. changes to support compiling under Visual Studio including: * Better testing under windows for dirent.h and opendir and closedir. 5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags and to centralize error reporting. 6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them. 7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible. Changes Left TO-DO: 1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
CHECK_INCLUDE_FILE("execinfo.h" HAVE_EXECINFO_H)
CHECK_INCLUDE_FILE("dirent.h" HAVE_DIRENT_H)
Provide a Number of fixes/improvements to NCZarr Primary changes: * Add an improved cache system to speed up performance. * Fix NCZarr to properly handle scalar variables. Misc. Related Changes: * Added unit tests for extendible hash and for the generic cache. * Add config parameter to set size of the NCZarr cache. * Add initial performance tests but leave them unused. * Add CRC64 support. * Move location of ncdumpchunks utility from /ncgen to /ncdump. * Refactor auth support. Misc. Unrelated Changes: * More cleanup of the S3 support * Add support for S3 authentication in .rc files: HTTP.S3.ACCESSID and HTTP.S3.SECRETKEY. * Remove the hashkey from the struct OBJHDR since it is never used.
2020-11-20 08:01:04 +08:00
CHECK_INCLUDE_FILE("time.h" HAVE_TIME_H)
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
CHECK_INCLUDE_FILE("dlfcn.h" HAVE_DLFCN_H)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
# Symbol Exists
CHECK_SYMBOL_EXISTS(isfinite "math.h" HAVE_DECL_ISFINITE)
CHECK_SYMBOL_EXISTS(isnan "math.h" HAVE_DECL_ISNAN)
CHECK_SYMBOL_EXISTS(isinf "math.h" HAVE_DECL_ISINF)
CHECK_SYMBOL_EXISTS(st_blksize "sys/stat.h" HAVE_STRUCT_STAT_ST_BLKSIZE)
CHECK_SYMBOL_EXISTS(alloca "alloca.h" HAVE_ALLOCA)
Working out an error with unistd.h in windows.
2017-01-31 05:54:00 +08:00
CHECK_SYMBOL_EXISTS(snprintf "stdio.h" HAVE_SNPRINTF)
2012-10-03 04:56:46 +08:00
# Type checks
Fix pull request https://github.com/Unidata/netcdf-c/pull/374 (dap4.dmh) 1. When running under windows (as opposed to cygwin) we need to make sure to not user /cygdrive/ file paths. This was ocurring in libdap4/d4read.c, but may occur elsewhere. 2. Shell scripts in the git repo are not being checked-out with the executable mode set. Had core.filemode set to false. Was a major hassle to fix.
2017-04-04 11:39:44 +08:00
# Aliases for automake consistency
SET(SIZEOF_VOIDSTAR ${CMAKE_SIZEOF_VOID_P})
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
SET(SIZEOF_VOIDP ${SIZEOF_VOIDSTAR})
Addressed several minor issues revealed by static analysis (coverity, clang). Mostly to do with extreme corner cases in which a null pointer would be dereferenced, or dereferencing a non-null (but non-initialized) pointer would result in garbage.
2014-03-29 07:11:26 +08:00
CHECK_TYPE_SIZE("char" SIZEOF_CHAR)
CHECK_TYPE_SIZE("double" SIZEOF_DOUBLE)
CHECK_TYPE_SIZE("float" SIZEOF_FLOAT)
CHECK_TYPE_SIZE("int" SIZEOF_INT)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
CHECK_TYPE_SIZE("uint" SIZEOF_UINT)
IF(SIZEOF_UINT)
SET(HAVE_UINT TRUE)
ENDIF(SIZEOF_UINT)
a clean commit for #383
2017-12-21 10:53:30 +08:00
CHECK_TYPE_SIZE("schar" SIZEOF_SCHAR)
IF(SIZEOF_SCHAR)
SET(HAVE_SCHAR TRUE)
ENDIF(SIZEOF_SCHAR)
Addressed several minor issues revealed by static analysis (coverity, clang). Mostly to do with extreme corner cases in which a null pointer would be dereferenced, or dereferencing a non-null (but non-initialized) pointer would result in garbage.
2014-03-29 07:11:26 +08:00
CHECK_TYPE_SIZE("long" SIZEOF_LONG)
CHECK_TYPE_SIZE("long long" SIZEOF_LONG_LONG)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
IF(SIZEOF_LONG_LONG)
SET(HAVE_LONG_LONG_INT TRUE)
ENDIF(SIZEOF_LONG_LONG)
CHECK_TYPE_SIZE("unsigned long long" SIZEOF_UNSIGNED_LONG_LONG)
Addressed several minor issues revealed by static analysis (coverity, clang). Mostly to do with extreme corner cases in which a null pointer would be dereferenced, or dereferencing a non-null (but non-initialized) pointer would result in garbage.
2014-03-29 07:11:26 +08:00
CHECK_TYPE_SIZE("off_t" SIZEOF_OFF_T)
CHECK_TYPE_SIZE("off64_t" SIZEOF_OFF64_T)
CHECK_TYPE_SIZE("short" SIZEOF_SHORT)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
CHECK_TYPE_SIZE("ushort" SIZEOF_USHORT)
IF(SIZEOF_USHORT)
SET(HAVE_USHORT TRUE)
ENDIF(SIZEOF_USHORT)
CHECK_TYPE_SIZE("_Bool" SIZEOF__BOOL)
Addressed several minor issues revealed by static analysis (coverity, clang). Mostly to do with extreme corner cases in which a null pointer would be dereferenced, or dereferencing a non-null (but non-initialized) pointer would result in garbage.
2014-03-29 07:11:26 +08:00
CHECK_TYPE_SIZE("size_t" SIZEOF_SIZE_T)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
revise #1033 and emulate CMAKE OPTION to use 3rd option AUTO
2018-07-02 08:42:03 +08:00
# Check whether to turn on or off CDF5 support.
IF(SIZEOF_SIZE_T EQUAL 4)
IF(ENABLE_CDF5) # enable or auto
STRING(TOUPPER ${ENABLE_CDF5} ENABLE_CDF5)
IF(ENABLE_CDF5 AND NOT ENABLE_CDF5 STREQUAL "AUTO") # explicitly enabled
MESSAGE(FATAL_ERROR "Unable to support CDF5 feature because size_t is less than 8 bytes")
ENDIF(ENABLE_CDF5 AND NOT ENABLE_CDF5 STREQUAL "AUTO")
SET(ENABLE_CDF5 OFF) # cannot support CDF5
SET(USE_CDF5 OFF CACHE BOOL "") # cannot support CDF5
ENDIF(ENABLE_CDF5)
ELSE(SIZEOF_SIZE_T EQUAL 4)
IF(ENABLE_CDF5) # explicitly set by user or not set
SET(USE_CDF5 ON CACHE BOOL "")
ELSE(ENABLE_CDF5) # explicitly disabled by user
SET(USE_CDF5 OFF CACHE BOOL "")
ENDIF(ENABLE_CDF5)
ENDIF(SIZEOF_SIZE_T EQUAL 4)
This completes (for now) the refactoring of libsrc4. The file docs/indexing.dox tries to provide design information for the refactoring. The primary change is to replace all walking of linked lists with the use of the NCindex data structure. Ncindex is a combination of a hash table (for name-based lookup) and a vector (for walking the elements in the index). Additionally, global vectors are added to NC_HDF5_FILE_INFO_T to support direct mapping of an e.g. dimid to the NC_DIM_INFO_T object. These global vectors exist for dimensions, types, and groups because they have globally unique id numbers. WARNING: 1. since libsrc4 and libsrchdf4 share code, there are also changes in libsrchdf4. 2. Any outstanding pull requests that change libsrc4 or libhdf4 are likely to cause conflicts with this code. 3. The original reason for doing this was for performance improvements, but as noted elsewhere, this may not be significant because the meta-data read performance apparently is being dominated by the hdf5 library because we do bulk meta-data reading rather than lazy reading.
2018-03-17 01:46:18 +08:00
CHECK_TYPE_SIZE("ssize_t" SIZEOF_SSIZE_T)
IF(SIZEOF_SSIZE_T)
SET(HAVE_SSIZE_T TRUE)
ENDIF(SIZEOF_SSIZE_T)
CHECK_TYPE_SIZE("ptrdiff_t" SIZEOF_PTRDIFF_T)
IF(SIZEOF_PTRDIFF_T)
SET(HAVE_PTRDIFF_T TRUE)
ENDIF(SIZEOF_PTRDIFF_T)
CHECK_TYPE_SIZE("uintptr_t" SIZEOF_UINTPTR_T)
IF(SIZEOF_UINTPTR_T)
SET(HAVE_UINTPTR_T TRUE)
ENDIF(SIZEOF_UINTPTR_T)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
Debug LFS issue on Windows.
2013-08-31 05:16:17 +08:00
# __int64 is used on Windows for large file support.
Addressed several minor issues revealed by static analysis (coverity, clang). Mostly to do with extreme corner cases in which a null pointer would be dereferenced, or dereferencing a non-null (but non-initialized) pointer would result in garbage.
2014-03-29 07:11:26 +08:00
CHECK_TYPE_SIZE("__int64" SIZEOF___INT_64)
CHECK_TYPE_SIZE("int64_t" SIZEOF_INT64_T)
Provide a Number of fixes/improvements to NCZarr Primary changes: * Add an improved cache system to speed up performance. * Fix NCZarr to properly handle scalar variables. Misc. Related Changes: * Added unit tests for extendible hash and for the generic cache. * Add config parameter to set size of the NCZarr cache. * Add initial performance tests but leave them unused. * Add CRC64 support. * Move location of ncdumpchunks utility from /ncgen to /ncdump. * Refactor auth support. Misc. Unrelated Changes: * More cleanup of the S3 support * Add support for S3 authentication in .rc files: HTTP.S3.ACCESSID and HTTP.S3.SECRETKEY. * Remove the hashkey from the struct OBJHDR since it is never used.
2020-11-20 08:01:04 +08:00
CHECK_TYPE_SIZE("uint64" SIZEOF_UINT64)
add sizeof checks for unsigned data types
2016-10-29 07:16:49 +08:00
CHECK_TYPE_SIZE("unsigned char" SIZEOF_UCHAR)
Attempting to fix visual studio errors in support of https://github.com/Unidata/netcdf-c/pull/375
2017-03-14 05:12:47 +08:00
CHECK_TYPE_SIZE("unsigned short int" SIZEOF_UNSIGNED_SHORT_INT)
CHECK_TYPE_SIZE("unsigned int" SIZEOF_UNSIGNED_INT)
add sizeof checks for unsigned data types
2016-10-29 07:16:49 +08:00
CHECK_TYPE_SIZE("long long" SIZEOF_LONGLONG)
CHECK_TYPE_SIZE("unsigned long long" SIZEOF_ULONGLONG)
2012-10-03 04:56:46 +08:00
Fix a number of CMake problems
2020-11-20 13:24:13 +08:00
CHECK_TYPE_SIZE("uint64_t" SIZEOF_UINT64_T)
IF(SIZEOF_UINT64_T)
SET(HAVE_UINT64_T TRUE)
ENDIF(SIZEOF_UINT64_T)
Added note in CMakeLists.txt to specify why we're doing something on Windows platforms.
2013-08-31 05:25:28 +08:00
# On windows systems, we redefine off_t as __int64
# to enable LFS. This is true on 32 and 64 bit system.s
# We must redefine SIZEOF_OFF_T to match.
Debug LFS issue on Windows.
2013-08-31 05:16:17 +08:00
IF(MSVC AND SIZEOF___INT_64)
Added note in CMakeLists.txt to specify why we're doing something on Windows platforms.
2013-08-31 05:25:28 +08:00
SET(SIZEOF_OFF_T ${SIZEOF___INT_64})
Debug LFS issue on Windows.
2013-08-31 05:16:17 +08:00
ENDIF()
improve mechanism by which to find parallel HDF5 So far, CMakeLists.txt made the assumption that ${HDF5_LIB} is always defined in order to determine if it has parallel support. This is not the case, compare with bug #25. Make use of FIND_PACKAGE(HDF5) now; it sets HDF5_IS_PARALLEL anyways.
2014-03-07 23:40:50 +08:00
# Check for various functions.
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_FUNCTION_EXISTS(fsync HAVE_FSYNC)
CHECK_FUNCTION_EXISTS(strlcat HAVE_STRLCAT)
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
CHECK_FUNCTION_EXISTS(strlcpy HAVE_STRLCPY)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_FUNCTION_EXISTS(strdup HAVE_STRDUP)
Corrected issues where functions were not available in Windows. Specifically strndup. Also accomodated an included needed for getcwd.
2014-08-08 07:03:27 +08:00
CHECK_FUNCTION_EXISTS(strndup HAVE_STRNDUP)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_FUNCTION_EXISTS(strtoll HAVE_STRTOLL)
1. Fix conflicts with current master. 2. There is a bug in building tinyxml2 under OSX, so as a hack, the absence of an installed libxml2 under OSX will disable libxml2 and DAP4.
2022-03-16 05:33:13 +08:00
CHECK_FUNCTION_EXISTS(strcasecmp HAVE_STRCASECMP)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
CHECK_FUNCTION_EXISTS(strtoull HAVE_STRTOULL)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_FUNCTION_EXISTS(mkstemp HAVE_MKSTEMP)
This pr should probably be delayed until after Version 4.5. Primary change is to cleanup code and remove duplicated code. 1. Unify the rc file reading into libdispatch/drc.c. Eventually extend if we need rc file for netcdf itself as opposed to the dap code. 2. Unify the extraction from the rc file of DAP authorization info. 3. Misc. other small unifications: make temp file, read file. 4. Avoid use of libcurl when reading file:// because there is some kind of problem with the Visual Studio version. Might be related to the winpath problem. In any case, do direct read instead. 5. Add new error code NC_ERCFILE for errors in reading RC file. 6. Complete documentation cleanup as indicated in this comment https://github.com/Unidata/netcdf-c/pull/472#issuecomment-325926426 7. Convert some occurrences of #ifdef _WIN32 to #ifdef _MSC_VER
2017-09-03 08:09:36 +08:00
CHECK_FUNCTION_EXISTS(mktemp HAVE_MKTEMP)
Corrected an issue with hashmap and missing random() function on Windows using Visual Studio 2013
2016-04-12 06:07:27 +08:00
CHECK_FUNCTION_EXISTS(random HAVE_RANDOM)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_FUNCTION_EXISTS(gettimeofday HAVE_GETTIMEOFDAY)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
CHECK_FUNCTION_EXISTS(MPI_Comm_f2c HAVE_MPI_COMM_F2C)
add a check for MPI_Info_f2c
2018-09-22 06:32:36 +08:00
CHECK_FUNCTION_EXISTS(MPI_Info_f2c HAVE_MPI_INFO_F2C)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
CHECK_FUNCTION_EXISTS(memmove HAVE_MEMMOVE)
CHECK_FUNCTION_EXISTS(getpagesize HAVE_GETPAGESIZE)
CHECK_FUNCTION_EXISTS(sysconf HAVE_SYSCONF)
CHECK_FUNCTION_EXISTS(getrlimit HAVE_GETRLIMIT)
Added a check for _filelengthi64 in configuration. When found, this will be used instead of fstat when computing file length.
2013-11-05 03:58:33 +08:00
CHECK_FUNCTION_EXISTS(_filelengthi64 HAVE_FILE_LENGTH_I64)
cmake: fix detection of the mmap function
2018-10-16 21:44:20 +08:00
CHECK_FUNCTION_EXISTS(mmap HAVE_MMAP)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
CHECK_FUNCTION_EXISTS(mremap HAVE_MREMAP)
Fix minor --ansi warnings in dinfermodel.c and bzlib.c re: Needed to provide centralized definitions of fileno and fdopen; also need to #include sys/types.h
2019-03-23 05:16:47 +08:00
CHECK_FUNCTION_EXISTS(fileno HAVE_FILENO)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
Make use of clock_gettime be conditional. Re: GH Issue https://github.com/Unidata/netcdf-c/issues/1900 Apparently the clock_gettime() function is not always available. It is used in unit_test/tst_exhash.c and unit_test/tst_xcache.c. To solve this, a number of things were changed: * Move the timing code to a new file unit_tests/timer_utils.[ch] * Modify the timing code to choose one of several timing methods depending on availability. The prioritized order is as follows: 1. If Windows, use the QueryPerformanceCounter mechanism else 2. Use clock_gettime if available else 3. Use gettimeofday if available else 4. Use getrusage if available Note that the resolution of 3 and 4 is less than 1 or 2. Misc. Other Changes: * Move the test in CMakeLists.txt that disables unit tests for WIN32 to unit_test/CMakeLists.txt since some unit tests actually work under Visual Studio. * Fix some of the unit tests to work under visual studio * Fix problem with using remove() in zmap_nzf.c * Remove some warning about use of EXTERNL
2020-12-07 09:19:53 +08:00
CHECK_FUNCTION_EXISTS(clock_gettime HAVE_CLOCK_GETTIME)
CHECK_SYMBOL_EXISTS("struct timespec" "time.h" HAVE_STRUCT_TIMESPEC)
Fix additional S3 support issues re: https://github.com/Unidata/netcdf-c/issues/2117 re: https://github.com/Unidata/netcdf-c/issues/2119 * Modify libsrc to allow byte-range reading of netcdf-3 files in private S3 buckets; this required using the aws sdk. Also add a test case. * The aws sdk can sometimes cause problems if the Awd::ShutdownAPI function is not called. So at optional atexit() support to ensure it is called. This is disabled for Windows. * Add documentation to nczarr.md on how to build and use the aws sdk under windows. Currently it builds, but testing fails. * Switch testing from stratus to the Unidata bucket on S3. * Improve support for the s3: url protocol. * Add a s3 specific utility code file: ds3util.c * Modify NC_infermodel to attempt to read the magic number of byte-ranged files in S3. ## Misc. * Move and rename the core S3 SDK wrapper code (libnczarr/zs3sdk.cpp) to libdispatch since it now used in libsrc as well as libnczarr. * Add calls to nc_finalize in the utilities in case atexit is disabled. * Add header only json parser to the distribution rather than as a built source.
2021-10-30 10:06:37 +08:00
CHECK_FUNCTION_EXISTS(atexit HAVE_ATEXIT)
Make use of clock_gettime be conditional. Re: GH Issue https://github.com/Unidata/netcdf-c/issues/1900 Apparently the clock_gettime() function is not always available. It is used in unit_test/tst_exhash.c and unit_test/tst_xcache.c. To solve this, a number of things were changed: * Move the timing code to a new file unit_tests/timer_utils.[ch] * Modify the timing code to choose one of several timing methods depending on availability. The prioritized order is as follows: 1. If Windows, use the QueryPerformanceCounter mechanism else 2. Use clock_gettime if available else 3. Use gettimeofday if available else 4. Use getrusage if available Note that the resolution of 3 and 4 is less than 1 or 2. Misc. Other Changes: * Move the test in CMakeLists.txt that disables unit tests for WIN32 to unit_test/CMakeLists.txt since some unit tests actually work under Visual Studio. * Fix some of the unit tests to work under visual studio * Fix problem with using remove() in zmap_nzf.c * Remove some warning about use of EXTERNL
2020-12-07 09:19:53 +08:00
Fix additional S3 support issues re: https://github.com/Unidata/netcdf-c/issues/2117 re: https://github.com/Unidata/netcdf-c/issues/2119 * Modify libsrc to allow byte-range reading of netcdf-3 files in private S3 buckets; this required using the aws sdk. Also add a test case. * The aws sdk can sometimes cause problems if the Awd::ShutdownAPI function is not called. So at optional atexit() support to ensure it is called. This is disabled for Windows. * Add documentation to nczarr.md on how to build and use the aws sdk under windows. Currently it builds, but testing fails. * Switch testing from stratus to the Unidata bucket on S3. * Improve support for the s3: url protocol. * Add a s3 specific utility code file: ds3util.c * Modify NC_infermodel to attempt to read the magic number of byte-ranged files in S3. ## Misc. * Move and rename the core S3 SDK wrapper code (libnczarr/zs3sdk.cpp) to libdispatch since it now used in libsrc as well as libnczarr. * Add calls to nc_finalize in the utilities in case atexit is disabled. * Add header only json parser to the distribution rather than as a built source.
2021-10-30 10:06:37 +08:00
# Control invoking nc_finalize at exit
OPTION(ENABLE_ATEXIT_FINALIZE "Invoke nc_finalize at exit." ON)
IF(NOT HAVE_ATEXIT)
IF(ENABLE_ATEXIT_FINALIZE AND NOT HAVE_ATEXIT)
SET(ENABLE_ATEXIT_FINALIZE OFF CACHE BOOL "Enable ATEXIT" FORCE)
MESSAGE(WARNING "ENABLE_ATEXIT_FINALIZE set but atexit() function not defined")
ENDIF()
ENDIF()
Begin wiring in option to toggle automatic search for libxml2 in support of https://github.com/Unidata/netcdf-c/pulls/2135
2021-11-03 05:35:04 +08:00
Make the netcdf-c library compile with gcc -ansi. Primary fixes to get -ansi to work. 1. Convert all '//' C++ style comments to /*...*/ or to use #if 0...#endif 2. It turns out that when -ansi is specified, then a number of functions no longer are defined in the header -- but they are still in the .so file.<br> The big example is strdup(). So, added code to include/ncconfig.h to define externs for those missing functions that occur in more than one place. These are enabled if !_WIN32 && __STDC__ == 1 (__STDC__ is supposed to be the equivalent compile time flag to -ansi). Note that this requires config.h (which references ncconfig.h) to be included in files where it is currently not included. Single uses will be only in the file that uses them. 3. Added mmap test for the MAP_ANONYMOUS flag to configure.ac. Apparently this is not always defined with -ansi. 4. fix some large integer constants in nc_test4/tst_atts3.c and nc_test4/tst_filterparser.c to avoid compiler complaints. 5. fix a double constant in nc_test4/tst_filterparser.c to avoid compiler complaints. [Note I suspect #4 and #5 will be a problem on big-endian machines, but we have no way to test] Misc. Changes: 1. convert more instances of _MSC_VER to _WIN32. 2. added some debugging code to include/nctestserver.h 3. added comment about libdispatch/drc.c always being compiled. 4. modify parser generation in ncgen to remove unneeded files.
2018-12-06 10:20:43 +08:00
# Check to see if MAP_ANONYMOUS is defined.
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
IF(MSVC)
MESSAGE(WARNING "mmap not supported under visual studio: disabling MMAP support.")
SET(ENABLE_MMAP OFF)
ELSE()
Make the netcdf-c library compile with gcc -ansi. Primary fixes to get -ansi to work. 1. Convert all '//' C++ style comments to /*...*/ or to use #if 0...#endif 2. It turns out that when -ansi is specified, then a number of functions no longer are defined in the header -- but they are still in the .so file.<br> The big example is strdup(). So, added code to include/ncconfig.h to define externs for those missing functions that occur in more than one place. These are enabled if !_WIN32 && __STDC__ == 1 (__STDC__ is supposed to be the equivalent compile time flag to -ansi). Note that this requires config.h (which references ncconfig.h) to be included in files where it is currently not included. Single uses will be only in the file that uses them. 3. Added mmap test for the MAP_ANONYMOUS flag to configure.ac. Apparently this is not always defined with -ansi. 4. fix some large integer constants in nc_test4/tst_atts3.c and nc_test4/tst_filterparser.c to avoid compiler complaints. 5. fix a double constant in nc_test4/tst_filterparser.c to avoid compiler complaints. [Note I suspect #4 and #5 will be a problem on big-endian machines, but we have no way to test] Misc. Changes: 1. convert more instances of _MSC_VER to _WIN32. 2. added some debugging code to include/nctestserver.h 3. added comment about libdispatch/drc.c always being compiled. 4. modify parser generation in ncgen to remove unneeded files.
2018-12-06 10:20:43 +08:00
CHECK_C_SOURCE_COMPILES("
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
#include <sys/mman.h>
Make the netcdf-c library compile with gcc -ansi. Primary fixes to get -ansi to work. 1. Convert all '//' C++ style comments to /*...*/ or to use #if 0...#endif 2. It turns out that when -ansi is specified, then a number of functions no longer are defined in the header -- but they are still in the .so file.<br> The big example is strdup(). So, added code to include/ncconfig.h to define externs for those missing functions that occur in more than one place. These are enabled if !_WIN32 && __STDC__ == 1 (__STDC__ is supposed to be the equivalent compile time flag to -ansi). Note that this requires config.h (which references ncconfig.h) to be included in files where it is currently not included. Single uses will be only in the file that uses them. 3. Added mmap test for the MAP_ANONYMOUS flag to configure.ac. Apparently this is not always defined with -ansi. 4. fix some large integer constants in nc_test4/tst_atts3.c and nc_test4/tst_filterparser.c to avoid compiler complaints. 5. fix a double constant in nc_test4/tst_filterparser.c to avoid compiler complaints. [Note I suspect #4 and #5 will be a problem on big-endian machines, but we have no way to test] Misc. Changes: 1. convert more instances of _MSC_VER to _WIN32. 2. added some debugging code to include/nctestserver.h 3. added comment about libdispatch/drc.c always being compiled. 4. modify parser generation in ncgen to remove unneeded files.
2018-12-06 10:20:43 +08:00
int main() {int x = MAP_ANONYMOUS;}" HAVE_MAPANON)
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
IF(NOT HAVE_MMAP OR NOT HAVE_MAPANON)
MESSAGE(WARNING "mmap or MAP_ANONYMOUS not found: disabling MMAP support.")
Revert/Improve nc_create + NC_DISKLESS behavior re: https://github.com/Unidata/netcdf-c/issues/1154 Inadvertently, the behavior of NC_DISKLESS with nc_create() was changed in release 4.6.1. Previously, the NC_WRITE flag needed to be explicitly used with NC_DISKLESS in order to cause the created file to be persisted to disk. Additional analyis indicated that the current NC_DISKLESS implementation was seriously flawed. This PR attempts to clean up and regularize the situation with respect to NC_DISKLESS control. One important aspect of diskless operation is that there are two different notions of write. 1. The file is read-write vs read-only when using the netcdf API. 2. The file is persisted or not to disk at nc_close(). Previously, these two were conflated. The rules now are as follows. 1. NC_DISKLESS + NC_WRITE means that the file is read/write using the netcdf API 2. NC_DISKLESS + NC_PERSIST means that the file is persisted to a disk file at nc_close. 3. NC_DISKLESS + NC_PERSIST + NC_WRITE means both 1 and 2. The NC_PERSIST flag is new and takes over the obsolete NC_MPIPOSIX flag. NC_MPIPOSIX is still defined, but is now an alias for the NC_MPIIO flag. It is also now the case that for netcdf-4, NC_DISKLESS is independent of NC_INMEMORY and in fact it is an error to specify both flags simultaneously. Finally, the MMAP code was fixed to use NC_PERSIST as well. Also marked MMAP as deprecated. Also added a test case to test various combinations of NC_DISKLESS, NC_PERSIST, and NC_WRITE. This PR affects a number of files and especially test cases that used NC_DISKLESS. Misc. Unrelated fixes 1. fixed some warnings in ncdump/dumplib.c
2018-10-11 03:32:17 +08:00
SET(ENABLE_MMAP OFF)
ENDIF()
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
ENDIF()
Revert/Improve nc_create + NC_DISKLESS behavior re: https://github.com/Unidata/netcdf-c/issues/1154 Inadvertently, the behavior of NC_DISKLESS with nc_create() was changed in release 4.6.1. Previously, the NC_WRITE flag needed to be explicitly used with NC_DISKLESS in order to cause the created file to be persisted to disk. Additional analyis indicated that the current NC_DISKLESS implementation was seriously flawed. This PR attempts to clean up and regularize the situation with respect to NC_DISKLESS control. One important aspect of diskless operation is that there are two different notions of write. 1. The file is read-write vs read-only when using the netcdf API. 2. The file is persisted or not to disk at nc_close(). Previously, these two were conflated. The rules now are as follows. 1. NC_DISKLESS + NC_WRITE means that the file is read/write using the netcdf API 2. NC_DISKLESS + NC_PERSIST means that the file is persisted to a disk file at nc_close. 3. NC_DISKLESS + NC_PERSIST + NC_WRITE means both 1 and 2. The NC_PERSIST flag is new and takes over the obsolete NC_MPIPOSIX flag. NC_MPIPOSIX is still defined, but is now an alias for the NC_MPIIO flag. It is also now the case that for netcdf-4, NC_DISKLESS is independent of NC_INMEMORY and in fact it is an error to specify both flags simultaneously. Finally, the MMAP code was fixed to use NC_PERSIST as well. Also marked MMAP as deprecated. Also added a test case to test various combinations of NC_DISKLESS, NC_PERSIST, and NC_WRITE. This PR affects a number of files and especially test cases that used NC_DISKLESS. Misc. Unrelated fixes 1. fixed some warnings in ncdump/dumplib.c
2018-10-11 03:32:17 +08:00
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
IF(ENABLE_MMAP)
Fix cmake X mmap supercede PR: https://github.com/Unidata/netcdf-c/pull/1384 Since we have an mmap user, undeprecate it and make sure it works. Other changes: * fix test cases to work with make -j * fix exposed ncgen error.
2019-04-20 10:32:26 +08:00
# Aliases
Revert/Improve nc_create + NC_DISKLESS behavior re: https://github.com/Unidata/netcdf-c/issues/1154 Inadvertently, the behavior of NC_DISKLESS with nc_create() was changed in release 4.6.1. Previously, the NC_WRITE flag needed to be explicitly used with NC_DISKLESS in order to cause the created file to be persisted to disk. Additional analyis indicated that the current NC_DISKLESS implementation was seriously flawed. This PR attempts to clean up and regularize the situation with respect to NC_DISKLESS control. One important aspect of diskless operation is that there are two different notions of write. 1. The file is read-write vs read-only when using the netcdf API. 2. The file is persisted or not to disk at nc_close(). Previously, these two were conflated. The rules now are as follows. 1. NC_DISKLESS + NC_WRITE means that the file is read/write using the netcdf API 2. NC_DISKLESS + NC_PERSIST means that the file is persisted to a disk file at nc_close. 3. NC_DISKLESS + NC_PERSIST + NC_WRITE means both 1 and 2. The NC_PERSIST flag is new and takes over the obsolete NC_MPIPOSIX flag. NC_MPIPOSIX is still defined, but is now an alias for the NC_MPIIO flag. It is also now the case that for netcdf-4, NC_DISKLESS is independent of NC_INMEMORY and in fact it is an error to specify both flags simultaneously. Finally, the MMAP code was fixed to use NC_PERSIST as well. Also marked MMAP as deprecated. Also added a test case to test various combinations of NC_DISKLESS, NC_PERSIST, and NC_WRITE. This PR affects a number of files and especially test cases that used NC_DISKLESS. Misc. Unrelated fixes 1. fixed some warnings in ncdump/dumplib.c
2018-10-11 03:32:17 +08:00
SET(BUILD_MMAP ON)
SET(USE_MMAP ON)
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
ENDIF(ENABLE_MMAP)
Merged latest changes from netcdf-cmake branch. Includes a few windows fixes, plus the skeleton for 'make dist' and 'make distcheck' support in cmake-based builds.
2013-03-28 03:15:00 +08:00
Bringing config.h as generagted by cmake in line with config.h as generated by autotools.
2017-01-28 08:41:03 +08:00
#CHECK_FUNCTION_EXISTS(alloca HAVE_ALLOCA)
Improve UTF8 Support On Windows re: Issue https://github.com/Unidata/netcdf-c/issues/2190 The primary purpose of this PR is to improve the utf8 support for windows. This is persuant to a change in Windows that supports utf8 natively (almost). The almost means that it is still utf16 internally and the set of characters representable by utf8 is larger than those representable by utf16. This leaves open the question in the Issue about handling the Windows 1252 character set. This required the following changes: 1. Test the Windows build and major version in order to see if native utf8 is supported. 2. If native utf8 is supported, Modify dpathmgr.c to call the 8-bit version of the windows fopen() and open() functions. 3. In support of this, programs that use XGetOpt (Windows versions) need to get the command line as utf8 and then parse to arc+argv as utf8. This requires using a homegrown command line parser named XCommandLineToArgvA. 4. Add a utility program called "acpget" that prints out the current Windows code page and locale. Additionally, some technical debt was cleaned up as follows: 1. Unify all the places which attempt to read all or a part of a file into the dutil.c#NC_readfile code. 2. Similary unify all the code that creates temp files into dutil.c#NC_mktmp code. 3. Convert almost all remaining calls to fopen() and open() to NCfopen() and NCopen3(). This is to ensure that path management is used consistently. This touches a number of files. 4. extern->EXTERNL as needed to get it to work under Windows.
2022-02-09 11:53:30 +08:00
# Used in the `configure_file` calls below
Improve filter installation process to avoid use of an extra shell script re: https://github.com/Unidata/netcdf-c/issues/2338 re: https://github.com/Unidata/netcdf-c/issues/2294 In issue https://github.com/Unidata/netcdf-c/issues/2338, Ed Hartnett suggested a better way to install filters to a user defined location -- for Automake, anyway. This PR implements that suggestion. It turns out to be more complicated than it appears, so there are fair number of changes; mostly to shell scripts. Most of the change is in plugins/Makefile.am. NOTE: this PR still does NOT address the use of HDF5_PLUGIN_PATH as the default; this turns out to be complex when dealing with NCZarr. So this will be addressed in a subsequent post 4.9.0 PR. ## Misc. Changes 1. Record the occurrences of incomplete codecs in libnczarr so that they can be included in _Codecs attribute correctly. This allows users to see what missing filters are referenced in the Zarr file. Primarily affects libnczarr/zfilter.[ch]. Also required creating a new no-effect filter: H5Zunknown.c. 2. Move the unknown filter test to a separate test file. 3. Incorporates PR https://github.com/Unidata/netcdf-c/pull/2343
2022-05-15 06:05:48 +08:00
SET(ISCMAKE "yes")
Improve UTF8 Support On Windows re: Issue https://github.com/Unidata/netcdf-c/issues/2190 The primary purpose of this PR is to improve the utf8 support for windows. This is persuant to a change in Windows that supports utf8 natively (almost). The almost means that it is still utf16 internally and the set of characters representable by utf8 is larger than those representable by utf16. This leaves open the question in the Issue about handling the Windows 1252 character set. This required the following changes: 1. Test the Windows build and major version in order to see if native utf8 is supported. 2. If native utf8 is supported, Modify dpathmgr.c to call the 8-bit version of the windows fopen() and open() functions. 3. In support of this, programs that use XGetOpt (Windows versions) need to get the command line as utf8 and then parse to arc+argv as utf8. This requires using a homegrown command line parser named XCommandLineToArgvA. 4. Add a utility program called "acpget" that prints out the current Windows code page and locale. Additionally, some technical debt was cleaned up as follows: 1. Unify all the places which attempt to read all or a part of a file into the dutil.c#NC_readfile code. 2. Similary unify all the code that creates temp files into dutil.c#NC_mktmp code. 3. Convert almost all remaining calls to fopen() and open() to NCfopen() and NCopen3(). This is to ensure that path management is used consistently. This touches a number of files. 4. extern->EXTERNL as needed to get it to work under Windows.
2022-02-09 11:53:30 +08:00
IF(MSVC)
CMakeLists.txt use ON vs YES, indent In CMake logically ON, YES, TRUE, 1 are equivalent. But many CMake editors won't syntax highlight a lower-case "yes", and CMake-gui will change them to ON, creating unnecessary changes to CMakeCache.txt. Also indent a WIN32 if/then block.
2023-03-20 09:02:58 +08:00
SET(ISMSVC ON CACHE BOOL "" FORCE)
SET(REGEDIT ON CACHE BOOL "" FORCE)
# Get windows major version and build number
EXECUTE_PROCESS(COMMAND "systeminfo" OUTPUT_VARIABLE WININFO)
IF(WININFO STREQUAL "")
SET(WVM 0)
SET(WVB 0)
ELSE()
STRING(REGEX MATCH "\nOS Version:[ \t]+[0-9.]+" WINVERLINE "${WININFO}")
STRING(REGEX REPLACE "[^0-9]*([0-9]+)[.]([0-9])+[.]([0-9]+)" "\\1" WVM "${WINVERLINE}")
STRING(REGEX REPLACE "[^0-9]*([0-9]+)[.]([0-9])+[.]([0-9]+)" "\\3" WVB "${WINVERLINE}")
ENDIF()
SET(WINVERMAJOR ${WVM} CACHE STRING "" FORCE)
SET(WINVERBUILD ${WVB} CACHE STRING "" FORCE)
Improve UTF8 Support On Windows re: Issue https://github.com/Unidata/netcdf-c/issues/2190 The primary purpose of this PR is to improve the utf8 support for windows. This is persuant to a change in Windows that supports utf8 natively (almost). The almost means that it is still utf16 internally and the set of characters representable by utf8 is larger than those representable by utf16. This leaves open the question in the Issue about handling the Windows 1252 character set. This required the following changes: 1. Test the Windows build and major version in order to see if native utf8 is supported. 2. If native utf8 is supported, Modify dpathmgr.c to call the 8-bit version of the windows fopen() and open() functions. 3. In support of this, programs that use XGetOpt (Windows versions) need to get the command line as utf8 and then parse to arc+argv as utf8. This requires using a homegrown command line parser named XCommandLineToArgvA. 4. Add a utility program called "acpget" that prints out the current Windows code page and locale. Additionally, some technical debt was cleaned up as follows: 1. Unify all the places which attempt to read all or a part of a file into the dutil.c#NC_readfile code. 2. Similary unify all the code that creates temp files into dutil.c#NC_mktmp code. 3. Convert almost all remaining calls to fopen() and open() to NCfopen() and NCopen3(). This is to ensure that path management is used consistently. This touches a number of files. 4. extern->EXTERNL as needed to get it to work under Windows.
2022-02-09 11:53:30 +08:00
ENDIF()
2012-10-03 04:56:46 +08:00
#####
# End system inspection checks.
#####
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
################################
# Define Utility Macros
################################
# Macro to append files to the EXTRA_DIST files.
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
# Note: can only be used in subdirectories because of the use of PARENT_SCOPE
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
SET(EXTRA_DIST "")
MACRO(ADD_EXTRA_DIST files)
FOREACH(F ${files})
SET(EXTRA_DIST ${EXTRA_DIST} ${CMAKE_CURRENT_SOURCE_DIR}/${F})
SET(EXTRA_DIST ${EXTRA_DIST} PARENT_SCOPE)
ENDFOREACH()
ENDMACRO()
# A basic script used to convert m4 files
Typo3.
2018-04-26 04:23:25 +08:00
FIND_PROGRAM(NC_M4 NAMES m4 m4.exe)
Modified how m4 is detected/handled, made it less a straight platform check.
2015-06-12 04:28:39 +08:00
IF(NC_M4)
Added output from looking for m4 executable.
2018-04-26 04:02:10 +08:00
MESSAGE(STATUS "Found m4: ${NC_M4}")
SET(HAVE_M4 TRUE)
ELSE()
Typo2.
2018-04-26 04:13:45 +08:00
MESSAGE(STATUS "m4 not found.")
More m4 fenceposting.
2020-05-27 06:09:10 +08:00
SET(HAVE_M4 FALSE)
Modified how m4 is detected/handled, made it less a straight platform check.
2015-06-12 04:28:39 +08:00
ENDIF()
Made m4 check with cmake linux/unix only.
2015-01-13 01:42:33 +08:00
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
MACRO(GEN_m4 filename)
Removed generated tests from source control. Ensuring generated files are regenerated if m4 is present.
2016-01-23 03:03:28 +08:00
Prevent cmake writing to source dir
2022-07-19 21:26:04 +08:00
set(fallbackdest "${CMAKE_CURRENT_SOURCE_DIR}/${filename}.c")
set(dest "${CMAKE_CURRENT_BINARY_DIR}/${filename}.c")
More tweaking.
2020-12-08 05:45:14 +08:00
Added a check for m4 and throw an error if m4 isn't on the path and the generated file isn't present (as it would be in a point release). In support of https://github.com/Unidata/netcdf-c/issues/1739
2020-05-27 04:46:23 +08:00
# If m4 isn't present, and the generated file doesn't exist,
# it cannot be generated and an error should be thrown.
IF(NOT HAVE_M4)
Prevent cmake writing to source dir
2022-07-19 21:26:04 +08:00
IF(NOT EXISTS ${fallbackdest})
MESSAGE(FATAL_ERROR "m4 is required to generate ${filename}.c. Please install m4 so that it is on the PATH and try again.")
ELSE()
Fix setting dest for non-m4 path
2023-02-15 18:43:14 +08:00
SET(dest ${fallbackdest})
Added a check for m4 and throw an error if m4 isn't on the path and the generated file isn't present (as it would be in a point release). In support of https://github.com/Unidata/netcdf-c/issues/1739
2020-05-27 04:46:23 +08:00
ENDIF()
Prevent cmake writing to source dir
2022-07-19 21:26:04 +08:00
ELSE()
Generate m4 files out of tree
2020-08-12 05:46:18 +08:00
ADD_CUSTOM_COMMAND(
More tweaking.
2020-12-08 05:45:14 +08:00
OUTPUT ${dest}
Generate m4 files out of tree
2020-08-12 05:46:18 +08:00
COMMAND ${NC_M4}
More tweaking.
2020-12-08 05:45:14 +08:00
ARGS ${M4FLAGS} ${CMAKE_CURRENT_SOURCE_DIR}/${filename}.m4 > ${dest}
Generate m4 files out of tree
2020-08-12 05:46:18 +08:00
VERBATIM
Added a check for m4 and throw an error if m4 isn't on the path and the generated file isn't present (as it would be in a point release). In support of https://github.com/Unidata/netcdf-c/issues/1739
2020-05-27 04:46:23 +08:00
)
Removed generated tests from source control. Ensuring generated files are regenerated if m4 is present.
2016-01-23 03:03:28 +08:00
Prevent cmake writing to source dir
2022-07-19 21:26:04 +08:00
ENDIF()
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
ENDMACRO(GEN_m4)
# Binary tests, but ones which depend on value of 'TEMP_LARGE' being defined.
MACRO(add_bin_env_temp_large_test prefix F)
ADD_EXECUTABLE(${prefix}_${F} ${F}.c)
TARGET_LINK_LIBRARIES(${prefix}_${F} netcdf)
IF(MSVC)
SET_TARGET_PROPERTIES(${prefix}_${F}
PROPERTIES LINK_FLAGS_DEBUG " /NODEFAULTLIB:MSVCRT"
)
ENDIF()
ADD_TEST(${prefix}_${F} bash "-c" "TEMP_LARGE=${CMAKE_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR}/${prefix}_${F}")
IF(MSVC)
SET_PROPERTY(TARGET ${prefix}_${F} PROPERTY FOLDER "tests")
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
SET_TARGET_PROPERTIES(${prefix}_${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY
More work on getting the shell scripts working properly with cmake under MSYS.
2015-01-31 03:55:46 +08:00
${CMAKE_CURRENT_BINARY_DIR})
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
SET_TARGET_PROPERTIES(${prefix}_${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG
More work on getting the shell scripts working properly with cmake under MSYS.
2015-01-31 03:55:46 +08:00
${CMAKE_CURRENT_BINARY_DIR})
SET_TARGET_PROPERTIES(${prefix}_${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE
${CMAKE_CURRENT_BINARY_DIR})
Working around a couple of problematic calls to fstat in support of https://github.com/Unidata/netcdf-c/issues/188 . No idea why these were a problem in VS14 (Visual Studio 2015) and not in VS12 (Visual Studio 2013). It's possible there are other, similar issues out there. We'll see what turns up when we run all the tests. These changes were tested against nc_test4/tst_large2.c
2016-03-03 04:03:29 +08:00
SET_TARGET_PROPERTIES(${prefix}_${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELWITHDEBINFO
${CMAKE_CURRENT_BINARY_DIR})
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
ENDIF()
ENDMACRO()
# Tests which are binary, but depend on a particular environmental variable.
MACRO(add_bin_env_test prefix F)
ADD_EXECUTABLE(${prefix}_${F} ${F}.c)
TARGET_LINK_LIBRARIES(${prefix}_${F} netcdf)
IF(MSVC)
SET_TARGET_PROPERTIES(${prefix}_${F}
PROPERTIES LINK_FLAGS_DEBUG " /NODEFAULTLIB:MSVCRT"
)
ENDIF()
Changes to make it possible to nest this project inside of another CMake project.
2019-11-07 00:43:49 +08:00
ADD_TEST(${prefix}_${F} bash "-c" "TOPSRCDIR=${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR}/${prefix}_${F}")
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
IF(MSVC)
SET_PROPERTY(TARGET ${prefix}_${F} PROPERTY FOLDER "tests")
ENDIF()
ENDMACRO()
# Build a binary used by a script, but don't make a test out of it.
MACRO(build_bin_test F)
Move generated file to BINARY_DIR
2020-08-12 22:38:00 +08:00
if(EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/${F}.c")
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
ADD_EXECUTABLE(${F} "${CMAKE_CURRENT_SOURCE_DIR}/${F}.c" ${ARGN})
Move generated file to BINARY_DIR
2020-08-12 22:38:00 +08:00
else()
# File should have been copied to the binary directory
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
ADD_EXECUTABLE(${F} "${CMAKE_CURRENT_BINARY_DIR}/${F}.c" ${ARGN})
Move generated file to BINARY_DIR
2020-08-12 22:38:00 +08:00
endif()
Added a new test for h4 endianness. [NCF332]
2015-05-29 07:13:15 +08:00
TARGET_LINK_LIBRARIES(${F} netcdf ${ALL_TLL_LIBS})
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
IF(MSVC)
SET_TARGET_PROPERTIES(${F}
PROPERTIES LINK_FLAGS_DEBUG " /NODEFAULTLIB:MSVCRT"
)
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
SET_TARGET_PROPERTIES(${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY
More work on getting the shell scripts working properly with cmake under MSYS.
2015-01-31 03:55:46 +08:00
${CMAKE_CURRENT_BINARY_DIR})
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
SET_TARGET_PROPERTIES(${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG
More work on getting the shell scripts working properly with cmake under MSYS.
2015-01-31 03:55:46 +08:00
${CMAKE_CURRENT_BINARY_DIR})
SET_TARGET_PROPERTIES(${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE
${CMAKE_CURRENT_BINARY_DIR})
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
ENDIF()
ENDMACRO()
# Binary tests which are used by a script looking for a specific name.
MACRO(add_bin_test_no_prefix F)
Make use of clock_gettime be conditional. Re: GH Issue https://github.com/Unidata/netcdf-c/issues/1900 Apparently the clock_gettime() function is not always available. It is used in unit_test/tst_exhash.c and unit_test/tst_xcache.c. To solve this, a number of things were changed: * Move the timing code to a new file unit_tests/timer_utils.[ch] * Modify the timing code to choose one of several timing methods depending on availability. The prioritized order is as follows: 1. If Windows, use the QueryPerformanceCounter mechanism else 2. Use clock_gettime if available else 3. Use gettimeofday if available else 4. Use getrusage if available Note that the resolution of 3 and 4 is less than 1 or 2. Misc. Other Changes: * Move the test in CMakeLists.txt that disables unit tests for WIN32 to unit_test/CMakeLists.txt since some unit tests actually work under Visual Studio. * Fix some of the unit tests to work under visual studio * Fix problem with using remove() in zmap_nzf.c * Remove some warning about use of EXTERNL
2020-12-07 09:19:53 +08:00
build_bin_test(${F} ${ARGN})
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
ADD_TEST(${F} ${EXECUTABLE_OUTPUT_PATH}/${F})
IF(MSVC)
SET_PROPERTY(TEST ${F} PROPERTY FOLDER "tests/")
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
SET_TARGET_PROPERTIES(${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY
More work on getting the shell scripts working properly with cmake under MSYS.
2015-01-31 03:55:46 +08:00
${CMAKE_CURRENT_BINARY_DIR})
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
SET_TARGET_PROPERTIES(${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG
More work on getting the shell scripts working properly with cmake under MSYS.
2015-01-31 03:55:46 +08:00
${CMAKE_CURRENT_BINARY_DIR})
SET_TARGET_PROPERTIES(${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
${CMAKE_CURRENT_BINARY_DIR})
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
ENDIF()
ENDMACRO()
# Binary tests which are used by a script looking for a specific name.
MACRO(build_bin_test_no_prefix F)
build_bin_test(${F})
IF(MSVC)
Fixed an issue when building binary tests that are invoked indirectly, with visual studio.
2017-11-30 03:17:57 +08:00
#SET_PROPERTY(TEST ${F} PROPERTY FOLDER "tests/")
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
SET_TARGET_PROPERTIES(${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY
More work on getting the shell scripts working properly with cmake under MSYS.
2015-01-31 03:55:46 +08:00
${CMAKE_CURRENT_BINARY_DIR})
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
SET_TARGET_PROPERTIES(${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG
More work on getting the shell scripts working properly with cmake under MSYS.
2015-01-31 03:55:46 +08:00
${CMAKE_CURRENT_BINARY_DIR})
SET_TARGET_PROPERTIES(${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE
${CMAKE_CURRENT_BINARY_DIR})
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
ENDIF()
ENDMACRO()
MACRO(add_bin_test prefix F)
Make use of clock_gettime be conditional. Re: GH Issue https://github.com/Unidata/netcdf-c/issues/1900 Apparently the clock_gettime() function is not always available. It is used in unit_test/tst_exhash.c and unit_test/tst_xcache.c. To solve this, a number of things were changed: * Move the timing code to a new file unit_tests/timer_utils.[ch] * Modify the timing code to choose one of several timing methods depending on availability. The prioritized order is as follows: 1. If Windows, use the QueryPerformanceCounter mechanism else 2. Use clock_gettime if available else 3. Use gettimeofday if available else 4. Use getrusage if available Note that the resolution of 3 and 4 is less than 1 or 2. Misc. Other Changes: * Move the test in CMakeLists.txt that disables unit tests for WIN32 to unit_test/CMakeLists.txt since some unit tests actually work under Visual Studio. * Fix some of the unit tests to work under visual studio * Fix problem with using remove() in zmap_nzf.c * Remove some warning about use of EXTERNL
2020-12-07 09:19:53 +08:00
ADD_EXECUTABLE(${prefix}_${F} ${F}.c ${ARGN})
tests are using HDF5 explicitly, so link against HDF5
2014-10-08 18:50:55 +08:00
TARGET_LINK_LIBRARIES(${prefix}_${F}
Added a new test for h4 endianness. [NCF332]
2015-05-29 07:13:15 +08:00
${ALL_TLL_LIBS}
tests are using HDF5 explicitly, so link against HDF5
2014-10-08 18:50:55 +08:00
netcdf
)
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
IF(MSVC)
SET_TARGET_PROPERTIES(${prefix}_${F}
PROPERTIES LINK_FLAGS_DEBUG " /NODEFAULTLIB:MSVCRT"
)
ENDIF()
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
ADD_TEST(${prefix}_${F}
${EXECUTABLE_OUTPUT_PATH}/${prefix}_${F}
)
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
IF(MSVC)
SET_PROPERTY(TEST ${prefix}_${F} PROPERTY FOLDER "tests/")
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
SET_TARGET_PROPERTIES(${prefix}_${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY
More work on getting the shell scripts working properly with cmake under MSYS.
2015-01-31 03:55:46 +08:00
${CMAKE_CURRENT_BINARY_DIR})
Cleaning up ctest site identification.
2015-02-03 06:14:22 +08:00
SET_TARGET_PROPERTIES(${prefix}_${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG
More work on getting the shell scripts working properly with cmake under MSYS.
2015-01-31 03:55:46 +08:00
${CMAKE_CURRENT_BINARY_DIR})
SET_TARGET_PROPERTIES(${prefix}_${F} PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE
${CMAKE_CURRENT_BINARY_DIR})
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
ENDIF()
ENDMACRO()
# A cmake script to print out information at the end of the configuration step.
MACRO(print_conf_summary)
MESSAGE("")
MESSAGE("")
MESSAGE("Configuration Summary:")
MESSAGE("")
MESSAGE(STATUS "Building Shared Libraries: ${BUILD_SHARED_LIBS}")
MESSAGE(STATUS "Building netCDF-4: ${ENABLE_NETCDF_4}")
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
MESSAGE(STATUS "Building DAP2 Support: ${ENABLE_DAP2}")
MESSAGE(STATUS "Building DAP4 Support: ${ENABLE_DAP4}")
Master merge, conflict resolution, cleanup
2019-02-25 07:54:13 +08:00
MESSAGE(STATUS "Building Byte-range Support: ${ENABLE_BYTERANGE}")
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
MESSAGE(STATUS "Building Utilities: ${BUILD_UTILITIES}")
IF(CMAKE_PREFIX_PATH)
MESSAGE(STATUS "CMake Prefix Path: ${CMAKE_PREFIX_PATH}")
ENDIF()
MESSAGE("")
IF(${STATUS_PNETCDF} OR ${STATUS_PARALLEL})
MESSAGE("Building Parallel NetCDF")
Use PnetCDF instead of parallel-netcdf to avoid confusion with parallel netcdf4. Also, update PnetCDF web page.
2018-09-18 00:47:40 +08:00
MESSAGE(STATUS "Using PnetCDF: ${STATUS_PNETCDF}")
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
MESSAGE(STATUS "Using Parallel IO: ${STATUS_PARALLEL}")
MESSAGE("")
ENDIF()
MESSAGE("Tests Enabled: ${ENABLE_TESTS}")
IF(ENABLE_TESTS)
MESSAGE(STATUS "DAP Remote Tests: ${ENABLE_DAP_REMOTE_TESTS}")
MESSAGE(STATUS "Extra Tests: ${ENABLE_EXTRA_TESTS}")
MESSAGE(STATUS "Coverage Tests: ${ENABLE_COVERAGE_TESTS}")
MESSAGE(STATUS "Parallel Tests: ${ENABLE_PARALLEL_TESTS}")
MESSAGE(STATUS "Large File Tests: ${ENABLE_LARGE_FILE_TESTS}")
Added EXTREME_NUMBERS option to cmake, based on option in autotools, which had somehow not made it's way over previously.
2016-09-28 01:20:41 +08:00
MESSAGE(STATUS "Extreme Numbers: ${ENABLE_EXTREME_NUMBERS}")
Wiring unit test directory into cmake-based builds.
2019-08-22 04:50:09 +08:00
MESSAGE(STATUS "Unit Tests: ${ENABLE_UNIT_TESTS}")
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
ENDIF()
MESSAGE("")
MESSAGE("Compiler:")
MESSAGE("")
MESSAGE(STATUS "Build Type: ${CMAKE_BUILD_TYPE}")
MESSAGE(STATUS "CMAKE_C_COMPILER: ${CMAKE_C_COMPILER}")
MESSAGE(STATUS "CMAKE_C_FLAGS: ${CMAKE_C_FLAGS}")
IF("${CMAKE_BUILD_TYPE}" STREQUAL "DEBUG")
MESSAGE(STATUS "CMAKE_C_FLAGS_DEBUG: ${CMAKE_C_FLAGS_DEBUG}")
ENDIF()
IF("${CMAKE_BUILD_TYPE}" STREQUAL "RELEASE")
MESSAGE(STATUS "CMAKE_C_FLAGS_RELEASE: ${CMAKE_C_FLAGS_RELEASE}")
ENDIF()
MESSAGE(STATUS "Linking against: ${ALL_TLL_LIBS}")
MESSAGE("")
ENDMACRO()
Adding a workaround for older versions of cmake that are still common in the wild, rather than force manual upgrades.
2023-05-26 02:23:21 +08:00
##specific
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
# Shell script Macro
Updated CMakelists to remove debug, release subdirectories on Windows. Updating shell scripts to work with MSYS paths.
2015-01-28 04:57:51 +08:00
##
# Determine if 'bash' is on the system.
##
Added a cmake-based option to explicitly turn off shell script tests.
2018-04-27 06:27:20 +08:00
OPTION(ENABLE_BASH_SCRIPT_TESTING "Detection is typically automatic, but this option can be used to force enable/disable bash-script based tests." ON)
IF(ENABLE_BASH_SCRIPT_TESTING)
FIND_PROGRAM(HAVE_BASH bash)
IF(HAVE_BASH)
STRING(COMPARE EQUAL "${HAVE_BASH}" "C:/Windows/System32/bash.exe" IS_BASH_EXE)
IF(NOT IS_BASH_EXE)
MESSAGE(STATUS "Found bash: ${HAVE_BASH}")
ELSE()
MESSAGE(STATUS "Ignoring ${HAVE_BASH}")
SET(HAVE_BASH "")
ENDIF()
Added fenceposts around bash scripts, excluding the bash.exe included with Windows 10 (for now) when searching to see if bash is available.
2018-04-27 04:38:56 +08:00
ELSE()
Added a cmake-based option to explicitly turn off shell script tests.
2018-04-27 06:27:20 +08:00
MESSAGE(STATUS "Bash shell not found; disabling shell script tests.")
Added fenceposts around bash scripts, excluding the bash.exe included with Windows 10 (for now) when searching to see if bash is available.
2018-04-27 04:38:56 +08:00
ENDIF()
Added a cmake-based option to explicitly turn off shell script tests.
2018-04-27 06:27:20 +08:00
ELSE(ENABLE_BASH_SCRIPT_TESTING)
SET(HAVE_BASH "")
ENDIF(ENABLE_BASH_SCRIPT_TESTING)
Updated CMakelists to remove debug, release subdirectories on Windows. Updating shell scripts to work with MSYS paths.
2015-01-28 04:57:51 +08:00
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
MACRO(add_sh_test prefix F)
Updated CMakelists to remove debug, release subdirectories on Windows. Updating shell scripts to work with MSYS paths.
2015-01-28 04:57:51 +08:00
IF(HAVE_BASH)
Adding a workaround for older versions of cmake that are still common in the wild, rather than force manual upgrades.
2023-05-26 02:23:21 +08:00
ADD_TEST(${prefix}_${F} bash "-c" "export srcdir=${CMAKE_CURRENT_SOURCE_DIR};export TOPSRCDIR=${CMAKE_SOURCE_DIR};bash ${CMAKE_CURRENT_BINARY_DIR}/${F}.sh ${ARGN}")
move macro defintions behind option settings This allows us to access HDF5 variables in the macros.
2014-10-08 18:47:58 +08:00
ENDIF()
ENDMACRO()
# A function used to create autotools-style 'yes/no' definitions.
# If a variable is set, it 'yes' is returned. Otherwise, 'no' is
# returned.
#
# Also creates a version of the ret_val prepended with 'NC',
# when feature is true, which is used to generate netcdf_meta.h.
FUNCTION(is_enabled feature ret_val)
IF(${feature})
SET(${ret_val} "yes" PARENT_SCOPE)
SET("NC_${ret_val}" 1 PARENT_SCOPE)
ELSE()
SET(${ret_val} "no" PARENT_SCOPE)
SET("NC_${ret_val}" 0 PARENT_SCOPE)
ENDIF(${feature})
ENDFUNCTION()
# A function used to create autotools-style 'yes/no' definitions.
# If a variable is set, it 'yes' is returned. Otherwise, 'no' is
# returned.
#
# Also creates a version of the ret_val prepended with 'NC',
# when feature is true, which is used to generate netcdf_meta.h.
FUNCTION(is_disabled feature ret_val)
IF(${feature})
SET(${ret_val} "no" PARENT_SCOPE)
ELSE()
SET(${ret_val} "yes" PARENT_SCOPE)
SET("NC_${ret_val}" 1 PARENT_SCOPE)
ENDIF(${feature})
ENDFUNCTION()
################################
# End Macro Definitions
################################
Fixed netcdf-c issue #30 (on github) related to cmake-generated nc-config script.
2014-03-09 08:39:09 +08:00
# Create config.h file.
Made template naming convention more uniform.
2014-05-30 03:23:24 +08:00
configure_file("${netCDF_SOURCE_DIR}/config.h.cmake.in"
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
"${netCDF_BINARY_DIR}/config.h")
Added new option to enable/disable 'Data Conversion warnings', ENABLE_CONVERSION_WARNINGS, when using CMake-based builds. Added a custom cmake test configuration file to override the default '50' warnings when submitting to a remote dashboard. Updated Building-with-CMake file.
2013-09-17 01:45:34 +08:00
ADD_DEFINITIONS(-DHAVE_CONFIG_H)
o Improved package_source support in CMake-based builds. o Corrected an issue related to uninitialized variables which caused test failures on 2 test systems under very specific circumstances.
2013-02-20 07:29:41 +08:00
INCLUDE_DIRECTORIES(${netCDF_BINARY_DIR})
2012-10-03 04:56:46 +08:00
# End autotools-style checs for config.h
#####
# Set core names of the libraries.
#####
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
SET(netCDF_LIB_CORENAME "netcdf")
2012-10-03 04:56:46 +08:00
#####
# Set the true names of all the libraries, if customized by external project
#####
# Recurse into other subdirectories.
add_subdirectory("include")
add_subdirectory(libdispatch)
add_subdirectory(libsrc)
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
IF(USE_PNETCDF)
Moved libsrc5 to libsrcp because pnetcdf is used for more than CDF-5 files.
2015-08-15 10:38:30 +08:00
add_subdirectory(libsrcp)
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
ENDIF(USE_PNETCDF)
merge trunk into this branch
2013-03-16 04:31:07 +08:00
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
IF(USE_NETCDF4)
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
add_subdirectory(libsrc4)
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
ENDIF()
IF(USE_HDF5)
created libhdf5, moved some files
2018-05-09 01:58:01 +08:00
add_subdirectory(libhdf5)
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
ENDIF(USE_HDF5)
2012-10-03 04:56:46 +08:00
attempting fix for cmake HDF4 build problem when netcdf-4 is not built
2018-02-09 23:26:58 +08:00
IF(USE_HDF4)
moved HDF4 to its own dispatch layer
2018-02-08 21:20:58 +08:00
add_subdirectory(libhdf4)
fixed name
2018-03-05 19:22:15 +08:00
add_subdirectory(hdf4_test)
attempting fix for cmake HDF4 build problem when netcdf-4 is not built
2018-02-09 23:26:58 +08:00
ENDIF(USE_HDF4)
moved HDF4 to its own dispatch layer
2018-02-08 21:20:58 +08:00
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
IF(ENABLE_DAP2)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
ADD_SUBDIRECTORY(oc2)
ADD_SUBDIRECTORY(libdap2)
2012-10-03 04:56:46 +08:00
ENDIF()
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
IF(ENABLE_DAP4)
ADD_SUBDIRECTORY(libdap4)
Allow optional use of libxml2 re: https://github.com/Unidata/netcdf-c/issues/2119 H/T to [Egbert Eich](https://github.com/e4t) and [Bas Couwenberg](https://github.com/sebastic) for this PR. It is undesirable to make netcdf be dependent on the availability of libxml2, but it is desirable to allow its use if available. In order to do this, a wrapper API (include/ncxml.h) was constructed that supports either ezxml or libxml2 as the implementation. Additionally, the xml support code was moved to a new directory netcdf-c/libncxml. Primary changes: * Create a new sub-directory named netcdf-c/libncxml to hold all the xml implementation code. * Move ezxml.c and ezxml.h to libncxml * Create a wrapper API -- include/ncxml.h * Create an implementation, ncxml_ezxml.c to support use of ezxml. * Create an implementation, ncxml_xml2.c to support use of libxml2. * Add a check for libxml2 in configure.ac and CMakeLists.txt * Modify libdap to use the wrapper API instead of ezxml directly. Misc. Other Changes: * Change include/netcdf_json.h from built source to be part of the distribution.
2021-11-02 12:37:05 +08:00
ADD_SUBDIRECTORY(libncxml)
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
ELSE()
IF(ENABLE_S3_INTERNAL)
ADD_SUBDIRECTORY(libncxml)
ENDIF()
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
ENDIF()
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
IF(ENABLE_PLUGINS)
ADD_SUBDIRECTORY(libncpoco)
ENDIF()
This PR adds EXPERIMENTAL support for accessing data in the cloud using a variant of the Zarr protocol and storage format. This enhancement is generically referred to as "NCZarr". The data model supported by NCZarr is netcdf-4 minus the user-defined types and the String type. In this sense it is similar to the CDF-5 data model. More detailed information about enabling and using NCZarr is described in the document NUG/nczarr.md and in a [Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in). WARNING: this code has had limited testing, so do use this version for production work. Also, performance improvements are ongoing. Note especially the following platform matrix of successful tests: Platform | Build System | S3 support ------------------------------------ Linux+gcc | Automake | yes Linux+gcc | CMake | yes Visual Studio | CMake | no Additionally, and as a consequence of the addition of NCZarr, major changes have been made to the Filter API. NOTE: NCZarr does not yet support filters, but these changes are enablers for that support in the future. Note that it is possible (probable?) that there will be some accidental reversions if the changes here did not correctly mimic the existing filter testing. In any case, previously filter ids and parameters were of type unsigned int. In order to support the more general zarr filter model, this was all converted to char*. The old HDF5-specific, unsigned int operations are still supported but they are wrappers around the new, char* based nc_filterx_XXX functions. This entailed at least the following changes: 1. Added the files libdispatch/dfilterx.c and include/ncfilter.h 2. Some filterx utilities have been moved to libdispatch/daux.c 3. A new entry, "filter_actions" was added to the NCDispatch table and the version bumped. 4. An overly complex set of structs was created to support funnelling all of the filterx operations thru a single dispatch "filter_actions" entry. 5. Move common code to from libhdf5 to libsrc4 so that it is accessible to nczarr. Changes directly related to Zarr: 1. Modified CMakeList.txt and configure.ac to support both C and C++ -- this is in support of S3 support via the awd-sdk libraries. 2. Define a size64_t type to support nczarr. 3. More reworking of libdispatch/dinfermodel.c to support zarr and to regularize the structure of the fragments section of a URL. Changes not directly related to Zarr: 1. Make client-side filter registration be conditional, with default off. 2. Hack include/nc4internal.h to make some flags added by Ed be unique: e.g. NC_CREAT, NC_INDEF, etc. 3. cleanup include/nchttp.h and libdispatch/dhttp.c. 4. Misc. changes to support compiling under Visual Studio including: * Better testing under windows for dirent.h and opendir and closedir. 5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags and to centralize error reporting. 6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them. 7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible. Changes Left TO-DO: 1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
IF(ENABLE_NCZARR)
ADD_SUBDIRECTORY(libnczarr)
Additional Fixes to NCZarr Primary Fixes: * Add a whole variable optimization -- used in the rare case that nc_get/put_vara covers the whole of a variable and the variable has a single chunk. * Fix chunking error when stride causes whole chunks to be skipped. * Fix some memory leaks * Add test cases * Add one performance test to nczarr_test/. This uses the timer utils from unit_test: timer_utils.[ch]. * Move ncdumpchunks utility from ncdump to nczarr_test Misc. Other Changes: * Make check for aws libraries conditional on --enable-nczarr-s3 * Remove all but one bm tests from nczarr_test until they are working. * Remove another dependency on HDF5 from supposedly non-HDF5 specific code; specifically hdf5_log_hdf5. * Make the BAIL2 macro be hdf5 specific and replace elsewhere with an HDF5 independent equivalent. * Move hdf5cache.c to libsrc4/nc4cache.c because it is used by nczarr. * Modify unit_tests so that some of them are run even if using Windows. * Misc. small bug fixes and refactors and memory leaks. * Rename some conflicting tests for cmake. * Attempted to make nc_perf work with cmake and failed.
2020-12-17 11:48:02 +08:00
FILE(COPY ${netCDF_SOURCE_DIR}/unit_test/timer_utils.h
DESTINATION ${netCDF_BINARY_DIR}/nczarr_test/)
FILE(COPY ${netCDF_SOURCE_DIR}/unit_test/timer_utils.c
DESTINATION ${netCDF_BINARY_DIR}/nczarr_test/)
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
FILE(COPY ${netCDF_SOURCE_DIR}/nc_test4/test_filter.c
DESTINATION ${netCDF_BINARY_DIR}/nczarr_test/)
FILE(COPY ${netCDF_SOURCE_DIR}/nc_test4/test_filter_misc.c
DESTINATION ${netCDF_BINARY_DIR}/nczarr_test/)
FILE(COPY ${netCDF_SOURCE_DIR}/nc_test4/test_filter_repeat.c
DESTINATION ${netCDF_BINARY_DIR}/nczarr_test/)
FILE(COPY ${netCDF_SOURCE_DIR}/nc_test4/test_filter_order.c
DESTINATION ${netCDF_BINARY_DIR}/nczarr_test/)
FILE(COPY ${netCDF_SOURCE_DIR}/nc_test4/tst_multifilter.c
DESTINATION ${netCDF_BINARY_DIR}/nczarr_test/)
This PR adds EXPERIMENTAL support for accessing data in the cloud using a variant of the Zarr protocol and storage format. This enhancement is generically referred to as "NCZarr". The data model supported by NCZarr is netcdf-4 minus the user-defined types and the String type. In this sense it is similar to the CDF-5 data model. More detailed information about enabling and using NCZarr is described in the document NUG/nczarr.md and in a [Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in). WARNING: this code has had limited testing, so do use this version for production work. Also, performance improvements are ongoing. Note especially the following platform matrix of successful tests: Platform | Build System | S3 support ------------------------------------ Linux+gcc | Automake | yes Linux+gcc | CMake | yes Visual Studio | CMake | no Additionally, and as a consequence of the addition of NCZarr, major changes have been made to the Filter API. NOTE: NCZarr does not yet support filters, but these changes are enablers for that support in the future. Note that it is possible (probable?) that there will be some accidental reversions if the changes here did not correctly mimic the existing filter testing. In any case, previously filter ids and parameters were of type unsigned int. In order to support the more general zarr filter model, this was all converted to char*. The old HDF5-specific, unsigned int operations are still supported but they are wrappers around the new, char* based nc_filterx_XXX functions. This entailed at least the following changes: 1. Added the files libdispatch/dfilterx.c and include/ncfilter.h 2. Some filterx utilities have been moved to libdispatch/daux.c 3. A new entry, "filter_actions" was added to the NCDispatch table and the version bumped. 4. An overly complex set of structs was created to support funnelling all of the filterx operations thru a single dispatch "filter_actions" entry. 5. Move common code to from libhdf5 to libsrc4 so that it is accessible to nczarr. Changes directly related to Zarr: 1. Modified CMakeList.txt and configure.ac to support both C and C++ -- this is in support of S3 support via the awd-sdk libraries. 2. Define a size64_t type to support nczarr. 3. More reworking of libdispatch/dinfermodel.c to support zarr and to regularize the structure of the fragments section of a URL. Changes not directly related to Zarr: 1. Make client-side filter registration be conditional, with default off. 2. Hack include/nc4internal.h to make some flags added by Ed be unique: e.g. NC_CREAT, NC_INDEF, etc. 3. cleanup include/nchttp.h and libdispatch/dhttp.c. 4. Misc. changes to support compiling under Visual Studio including: * Better testing under windows for dirent.h and opendir and closedir. 5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags and to centralize error reporting. 6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them. 7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible. Changes Left TO-DO: 1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
ENDIF()
2012-10-03 04:56:46 +08:00
add_subdirectory(liblib)
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
IF(ENABLE_PLUGINS)
Fixed plugin compilation using MSVC in support of https://github.com/Unidata/netcdf-c/issues/1245
2018-12-19 04:26:49 +08:00
add_subdirectory(plugins)
ENDIF()
2012-10-10 04:56:27 +08:00
# For tests and utilities, we are no longer
# exporting symbols but rather importing them.
IF(BUILD_DLL)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
REMOVE_DEFINITIONS(-DDLL_EXPORT)
2012-10-10 04:56:27 +08:00
ENDIF()
2012-10-03 04:56:46 +08:00
2012-10-10 04:56:27 +08:00
# Enable Utilities.
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
IF(BUILD_UTILITIES)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
INCLUDE_DIRECTORIES(ncdump)
ADD_SUBDIRECTORY(ncgen)
ADD_SUBDIRECTORY(ncgen3)
ADD_SUBDIRECTORY(ncdump)
2012-10-03 04:56:46 +08:00
ENDIF()
# Enable tests
IF(ENABLE_TESTS)
Corrected an issue with disabling the V2 API support in cmake-based builds.
2014-04-04 00:45:22 +08:00
IF(ENABLE_V2_API)
Corrected an issue with toggling the V2 API, that I had introduced earlier today.
2014-04-04 07:09:38 +08:00
ADD_SUBDIRECTORY(nctest)
Corrected an issue with disabling the V2 API support in cmake-based builds.
2014-04-04 00:45:22 +08:00
ENDIF()
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
ADD_SUBDIRECTORY(nc_test)
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
IF(USE_HDF5)
Regarding [NCF-331], the issues appears to have been in nc4file.c:get_type_info2(). The call to H5Tget_order() was only happening for H5T_Float. According to the documentation, this works for all datatypes as of hdf5 1.8.6. This tentatively fixes the bug reported, but additional testing (and cleanup from debugging) is needed.
2015-05-16 04:24:27 +08:00
INCLUDE_DIRECTORIES(h5_test)
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
ADD_SUBDIRECTORY(nc_test4)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
ADD_SUBDIRECTORY(h5_test)
ENDIF()
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
IF(ENABLE_DAP2)
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
ADD_SUBDIRECTORY(ncdap_test)
ENDIF()
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
IF(ENABLE_DAP4)
ADD_SUBDIRECTORY(dap4_test)
ENDIF()
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
IF(ENABLE_EXAMPLES)
ADD_SUBDIRECTORY(examples)
ENDIF()
Added benchmarking option to cmake-based builds, turned on unit-testing by default
2023-02-17 02:20:45 +08:00
IF(ENABLE_BENCHMARKS)
ADD_SUBDIRECTORY(nc_perf)
ENDIF(ENABLE_BENCHMARKS)
Wiring unit test directory into cmake-based builds.
2019-08-22 04:50:09 +08:00
IF(ENABLE_UNIT_TESTS)
ADD_SUBDIRECTORY(unit_test)
ENDIF(ENABLE_UNIT_TESTS)
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
IF(ENABLE_NCZARR)
ADD_SUBDIRECTORY(nczarr_test)
ENDIF()
2012-10-03 04:56:46 +08:00
ENDIF()
# Code to generate an export header
Modified CMake configuration to fix some problems present on 64-bit linux.
2012-10-04 04:47:34 +08:00
#GENERATE_EXPORT_HEADER(netcdf
remove tabs from CMakeLists.txt
2014-03-07 23:46:26 +08:00
# BASE_NAME netcdf
# EXPORT_MACRO_NAME netcdf_EXPORT
# EXPORT_FILE_NAME netcdf_Export.h
# STATIC_DEFINE netcdf_BUILT_AS_STATIC
Modified CMake configuration to fix some problems present on 64-bit linux.
2012-10-04 04:47:34 +08:00
#)
2012-10-03 04:56:46 +08:00
#####
# Build doxygen documentation, if need be.
#####
Renamed man4 directory to docs.
2014-05-22 04:40:39 +08:00
ADD_SUBDIRECTORY(docs)
Modified Settings used to generate Binary installer for Windows, so that the user is given the option of adding the binaries to the system path.
2014-12-03 04:58:23 +08:00
Moved dependencies into proper location in relation to netcdf.dll to avoid 'can't find hdf5.dll, etc' errors.
2014-12-06 02:19:02 +08:00
##
Misc. source comment typos Some are user-facing. Found via `codespell` and through the downstream FreeCAD.
2018-03-16 22:38:40 +08:00
# Brute force, grab all of the dlls from the dependency directory,
Moved dependencies into proper location in relation to netcdf.dll to avoid 'can't find hdf5.dll, etc' errors.
2014-12-06 02:19:02 +08:00
# install them in the binary dir. Grab all of the .libs, put them
# in the libdir.
##
Merged latest changes from cmake branch. o Improved CMake support. o Cmake module compatibility (netcdf-config*.cmake.in).
2012-11-20 05:43:12 +08:00
IF(MSVC)
Moved dependencies into proper location in relation to netcdf.dll to avoid 'can't find hdf5.dll, etc' errors.
2014-12-06 02:19:02 +08:00
FILE(GLOB COPY_FILES ${CMAKE_PREFIX_PATH}/lib/*.lib)
INSTALL(FILES ${COPY_FILES}
DESTINATION ${CMAKE_INSTALL_LIBDIR}
COMPONENT dependencies)
FILE(GLOB COPY_FILES ${CMAKE_PREFIX_PATH}/bin/*.dll)
STRING(REGEX REPLACE "msv[.*].dll" "" COPY_FILES "${COPY_FILES}")
INSTALL(FILES ${COPY_FILES}
DESTINATION ${CMAKE_INSTALL_BINDIR}
COMPONENT dependencies)
Merged latest changes from cmake branch. o Improved CMake support. o Cmake module compatibility (netcdf-config*.cmake.in).
2012-11-20 05:43:12 +08:00
ENDIF()
2012-10-03 04:56:46 +08:00
# Subdirectory CMakeLists.txt files should specify their own
# 'install' files.
# Including 'CPack' kicks everything off.
INCLUDE(InstallRequiredSystemLibraries)
CONFIGURE_FILE(
${CMAKE_CURRENT_SOURCE_DIR}/FixBundle.cmake.in
${CMAKE_CURRENT_BINARY_DIR}/FixBundle.cmake
@ONLY
Cleaned up indentation, white space in multiple CMakeLists.txt files.
2014-04-22 01:15:33 +08:00
)
2012-10-03 04:56:46 +08:00
Stopped tracking a cmake-specific pkgconfig template; using the same one as autotools. Tightened up cmake summary a little, now a pkg-config file is generated.
2014-02-07 07:17:30 +08:00
###
# Create pkgconfig files.
###
IF(NOT DEFINED CMAKE_INSTALL_LIBDIR)
SET(CMAKE_INSTALL_LIBDIR lib)
ENDIF(NOT DEFINED CMAKE_INSTALL_LIBDIR)
improve mechanism by which to find parallel HDF5 So far, CMakeLists.txt made the assumption that ${HDF5_LIB} is always defined in order to determine if it has parallel support. This is not the case, compare with bug #25. Make use of FIND_PACKAGE(HDF5) now; it sets HDF5_IS_PARALLEL anyways.
2014-03-07 23:40:50 +08:00
# Set
Stopped tracking a cmake-specific pkgconfig template; using the same one as autotools. Tightened up cmake summary a little, now a pkg-config file is generated.
2014-02-07 07:17:30 +08:00
SET(prefix ${CMAKE_INSTALL_PREFIX})
SET(exec_prefix ${CMAKE_INSTALL_PREFIX})
SET(libdir ${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR})
Manually merging pull request https://github.com/Unidata/netcdf-c/pull/74 contributed by Nico Schlömer
2014-08-21 02:30:11 +08:00
SET(includedir ${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_INCLUDEDIR})
Stopped tracking a cmake-specific pkgconfig template; using the same one as autotools. Tightened up cmake summary a little, now a pkg-config file is generated.
2014-02-07 07:17:30 +08:00
SET(CC ${CMAKE_C_COMPILER})
Reorganized doxygen documents, renamed .doc files as .dox. Bumped revision to 4.3.1.2. There may not be a 4.3.1.2 release, but I didn't want to leave it as 4.3.1.1. Removed CMake-specific doxygen/documentation template, made changes required so that CMake and autotools-based builds can use the same template (Doxyfile.in).
2014-02-20 07:32:26 +08:00
Stopped tracking a cmake-specific pkgconfig template; using the same one as autotools. Tightened up cmake summary a little, now a pkg-config file is generated.
2014-02-07 07:17:30 +08:00
# Process all dependency libraries and create a string
# used when parsing netcdf.pc.in
Corrected a case in CMake Syntax where an empty string would cause a failure.
2019-04-27 06:00:28 +08:00
SET(NC_LIBS "")
Continued to flesh out cmake support for libnetcdf.settings. Added a pair of functions which will return a yes/no depending on whether or not a variable is defined. By default, cmake uses TRUE/FALSE.
2014-06-03 05:14:27 +08:00
FOREACH(_LIB ${ALL_TLL_LIBS})
GET_FILENAME_COMPONENT(_LIB_NAME ${_LIB} NAME_WE)
STRING(REGEX REPLACE "^lib" "" _NAME ${_LIB_NAME})
LIST(APPEND NC_LIBS "-l${_NAME}")
GET_FILENAME_COMPONENT(_LIB_DIR ${_LIB} PATH)
LIST(APPEND LINKFLAGS "-L${_LIB_DIR}")
ENDFOREACH()
Stopped tracking a cmake-specific pkgconfig template; using the same one as autotools. Tightened up cmake summary a little, now a pkg-config file is generated.
2014-02-07 07:17:30 +08:00
Bringing netcdf.pc generated by cmake in line with netcdf.pc generated by configure, in support of https://github.com/Unidata/netcdf-c/issues/gh1324
2019-02-28 05:36:42 +08:00
#SET(NC_LIBS "-lnetcdf ${NC_LIBS}")
Corrected cmake syntax, working with a list not a string.
2019-04-27 06:28:53 +08:00
IF(NC_LIBS)
Corrected a case in CMake Syntax where an empty string would cause a failure.
2019-04-27 06:00:28 +08:00
STRING(REPLACE ";" " " NC_LIBS "${NC_LIBS}")
STRING(REPLACE "-lhdf5::hdf5-shared" "-lhdf5" NC_LIBS ${NC_LIBS})
STRING(REPLACE "-lhdf5::hdf5_hl-shared" "-lhdf5_hl" NC_LIBS ${NC_LIBS})
STRING(REPLACE "-lhdf5::hdf5-static" "-lhdf5" NC_LIBS ${NC_LIBS})
STRING(REPLACE "-lhdf5::hdf5_hl-static" "-lhdf5_hl" NC_LIBS ${NC_LIBS})
ENDIF()
Corrected for string reported by FindHDF5.cmake on Windows.
2017-12-05 03:50:43 +08:00
Stopped tracking a cmake-specific pkgconfig template; using the same one as autotools. Tightened up cmake summary a little, now a pkg-config file is generated.
2014-02-07 07:17:30 +08:00
STRING(REPLACE ";" " " LINKFLAGS "${LINKFLAGS}")
LIST(REMOVE_DUPLICATES NC_LIBS)
LIST(REMOVE_DUPLICATES LINKFLAGS)
Bringing netcdf.pc generated by cmake in line with netcdf.pc generated by configure, in support of https://github.com/Unidata/netcdf-c/issues/gh1324
2019-02-28 05:36:42 +08:00
SET(LIBS ${NC_LIBS})
SET(NC_LIBS "-lnetcdf")
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
configure_file(
Stopped tracking a cmake-specific pkgconfig template; using the same one as autotools. Tightened up cmake summary a little, now a pkg-config file is generated.
2014-02-07 07:17:30 +08:00
${netCDF_SOURCE_DIR}/netcdf.pc.in
${netCDF_BINARY_DIR}/netcdf.pc @ONLY)
The pr does some cleanup on the internal documentation in the docs directory. 1. Add a new internal document -- testserver.dox -- to describe how to set up and maintain the dap test server. 2. It moves the internal documentation (internal.dox, indexing.dox, and testserver.dox) to later in the documentation table of contents. 3. Cleanup the formatting of the internal documents. 4. Cleanup some minor doxygen issues in other files.
2018-06-30 05:06:27 +08:00
IF(NOT IS_DIRECTORY ${netCDF_BINARY_DIR}/tmp)
FILE(MAKE_DIRECTORY ${netCDF_BINARY_DIR}/tmp)
ENDIF()
Made template naming convention more uniform.
2014-05-30 03:23:24 +08:00
configure_file("${netCDF_SOURCE_DIR}/nc-config.cmake.in"
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
"${netCDF_BINARY_DIR}/tmp/nc-config" @ONLY
NEWLINE_STYLE LF)
Fixed netcdf-c issue #30 (on github) related to cmake-generated nc-config script.
2014-03-09 08:39:09 +08:00
FILE(COPY "${netCDF_BINARY_DIR}/tmp/nc-config"
DESTINATION ${netCDF_BINARY_DIR}/
FILE_PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE)
Merged pull requestion from nschloe, #48.
2014-04-22 06:11:48 +08:00
INSTALL(FILES ${netCDF_BINARY_DIR}/netcdf.pc
Roughed in skeleton of cmake support for libnetcdf.settings.
2014-06-03 04:03:42 +08:00
DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig
COMPONENT utilities)
Stopped tracking a cmake-specific pkgconfig template; using the same one as autotools. Tightened up cmake summary a little, now a pkg-config file is generated.
2014-02-07 07:17:30 +08:00
Added infrastructure to have cmake-based builds automatically generate PDF documentation when using Doxygen.
2014-06-19 04:33:49 +08:00
INSTALL(PROGRAMS ${netCDF_BINARY_DIR}/nc-config
Merged pull request 'make better use of GNUInstallDirs' contributed by nschloe, https://github.com/Unidata/netcdf-c/pull/57
2014-05-22 05:15:56 +08:00
DESTINATION ${CMAKE_INSTALL_BINDIR}
COMPONENT utilities)
Fixed netcdf-c issue #30 (on github) related to cmake-generated nc-config script.
2014-03-09 08:39:09 +08:00
Stopped tracking a cmake-specific pkgconfig template; using the same one as autotools. Tightened up cmake summary a little, now a pkg-config file is generated.
2014-02-07 07:17:30 +08:00
###
Fixed netcdf-c issue #30 (on github) related to cmake-generated nc-config script.
2014-03-09 08:39:09 +08:00
# End pkgconfig, nc-config file creation.
Stopped tracking a cmake-specific pkgconfig template; using the same one as autotools. Tightened up cmake summary a little, now a pkg-config file is generated.
2014-02-07 07:17:30 +08:00
###
2012-10-03 04:56:46 +08:00
##
Moved dependencies into proper location in relation to netcdf.dll to avoid 'can't find hdf5.dll, etc' errors.
2014-12-06 02:19:02 +08:00
# Print the configuration summary
improve mechanism by which to find parallel HDF5 So far, CMakeLists.txt made the assumption that ${HDF5_LIB} is always defined in order to determine if it has parallel support. This is not the case, compare with bug #25. Make use of FIND_PACKAGE(HDF5) now; it sets HDF5_IS_PARALLEL anyways.
2014-03-07 23:40:50 +08:00
##
2012-10-03 04:56:46 +08:00
print_conf_summary()
Fixed an issue with Windows builds not compiling due to how ssize_t was being detected/defined.
2013-05-25 05:19:07 +08:00
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
# Enable Makedist files.
ADD_MAKEDIST()
ENABLE_MAKEDIST(README.md COPYRIGHT RELEASE_NOTES.md INSTALL INSTALL.cmake test_prog.c lib_flags.am cmake CMakeLists.txt COMPILE.cmake.txt config.h.cmake.in cmake_uninstall.cmake.in netcdf-config-version.cmake.in netcdf-config.cmake.in FixBundle.cmake.in nc-config.cmake.in configure configure.ac install-sh config.h.in config.sub CTestConfig.cmake.in)
Fixed an issue with Windows builds not compiling due to how ssize_t was being detected/defined.
2013-05-25 05:19:07 +08:00
Roughed in skeleton of cmake support for libnetcdf.settings.
2014-06-03 04:03:42 +08:00
#####
# Configure and print the libnetcdf.settings file.
#####
Continued to flesh out cmake support for libnetcdf.settings. Added a pair of functions which will return a yes/no depending on whether or not a variable is defined. By default, cmake uses TRUE/FALSE.
2014-06-03 05:14:27 +08:00
# Set variables to mirror those used by autoconf.
Roughed in skeleton of cmake support for libnetcdf.settings.
2014-06-03 04:03:42 +08:00
# This way we don't need to maintain two separate template
# files.
Continued to flesh out cmake support for libnetcdf.settings. Added a pair of functions which will return a yes/no depending on whether or not a variable is defined. By default, cmake uses TRUE/FALSE.
2014-06-03 05:14:27 +08:00
SET(host_cpu "${cpu}")
SET(host_vendor "${osname}")
SET(host_os "${osrel}")
Changes to make it possible to nest this project inside of another CMake project.
2019-11-07 00:43:49 +08:00
SET(abs_top_builddir "${CMAKE_CURRENT_BINARY_DIR}")
SET(abs_top_srcdir "${CMAKE_CURRENT_SOURCE_DIR}")
Continued to flesh out cmake support for libnetcdf.settings. Added a pair of functions which will return a yes/no depending on whether or not a variable is defined. By default, cmake uses TRUE/FALSE.
2014-06-03 05:14:27 +08:00
SET(CC_VERSION "${CMAKE_C_COMPILER}")
Manual merge of pull request https://github.com/Unidata/netcdf-c/pull/64 contributed by nschloe. Assorted CMake improvements.
2014-06-12 05:51:31 +08:00
# Build *FLAGS for libnetcdf.settings.
SET(CFLAGS "${CMAKE_C_FLAGS} ${CMAKE_C_FLAGS_${CMAKE_BUILD_TYPE}}")
SET(CPPFLAGS "${CMAKE_CPP_FLAGS} ${CMAKE_CPP_FLAGS_${CMAKE_BUILD_TYPE}}")
SET(LDFLAGS "${CMAKE_SHARED_LINKER_FLAGS} ${CMAKE_SHARED_LINKER_FLAGS_${CMAKE_BUILD_TYPE}}")
When creating the libnetcdf.settings file, CMake-based builds now include compiler and linker debug or release flags.
2014-06-04 00:47:01 +08:00
is_disabled(BUILD_SHARED_LIBS enable_static)
is_enabled(BUILD_SHARED_LIBS enable_shared)
Continued to flesh out cmake support for libnetcdf.settings. Added a pair of functions which will return a yes/no depending on whether or not a variable is defined. By default, cmake uses TRUE/FALSE.
2014-06-03 05:14:27 +08:00
is_enabled(ENABLE_V2_API HAS_NC2)
is_enabled(ENABLE_NETCDF_4 HAS_NC4)
is_enabled(ENABLE_HDF4 HAS_HDF4)
Revert "Revert "Fix nczarr-experimental: improve build support, disengage hdf5 vs netcdf4 flags, and find AWS libraries""
2020-08-18 09:15:47 +08:00
is_enabled(USE_HDF5 HAS_HDF5)
added BENCHMARKS to the summary
2022-04-26 20:18:52 +08:00
is_enabled(OFF HAS_BENCHMARKS)
Continued to flesh out cmake support for libnetcdf.settings. Added a pair of functions which will return a yes/no depending on whether or not a variable is defined. By default, cmake uses TRUE/FALSE.
2014-06-03 05:14:27 +08:00
is_enabled(STATUS_PNETCDF HAS_PNETCDF)
is_enabled(STATUS_PARALLEL HAS_PARALLEL)
More cleaning up of configuration status messages.
2015-11-18 08:14:22 +08:00
is_enabled(ENABLE_PARALLEL4 HAS_PARALLEL4)
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
is_enabled(ENABLE_DAP HAS_DAP)
Upgrade the nczarr code to match Zarr V2 Re: https://github.com/zarr-developers/zarr-python/pull/716 The Zarr version 2 spec has been extended to include the ability to choose the dimension separator in chunk name keys. The legal separators has been extended from {'.'} to {'.' '/'}. So now it is possible to use a key like "0/1/2/0" for chunk names. This PR implements this for NCZarr. The V2 spec now says that this separator can be set on a per-variable basis. For now, I have chosen to allow this be set only globally by adding a key named "ZARR.DIMENSION_SEPARATOR=<char>" in the .daprc/.dodsrc/ncrc file. Currently, the only legal separator characters are '.' (the default) and '/'. On writing, this key will only be written if its value is different than the default. This change caused problems because supporting a separator of '/' is difficult to parse when keys/paths use '/' as the path separator. A test case was added for this. Additionally, make nczarr be enabled default by default. This required some additional changes so that if zip and/or AWS S3 sdk are unavailable, then they are disabled for NCZarr. In addition the following unrelated changes were made. 1. Tested that pure-zarr mode could read an nczarr formatted store. 1. The .rc file handling now merges all known .rc files (.ncrc,.daprc, and .dodsrc) in that order and using those in HOME first, then in current directory. For duplicate entries, the later ones override the earlier ones. This change is to remove some of the conflicts inherent in the current .rc file load process. A set of test cases was also added. 1. Re-order tests in configure.ac and CMakeLists.txt so that if libcurl is not found then the other options that depend upon it properly are disabled. 1. I decided that xarray support should be enabled by default for pure zarr. In order to allow disabling, I added a new mode flag "noxarray". 1. Certain test in nczarr_test depend on use of .dodsrc. In order for these to work when testing in parallel, some inter-test dependencies needed to be added. 1. Improved authorization testing to use changes in thredds.ucar.edu
2021-04-25 09:48:15 +08:00
is_enabled(ENABLE_DAP2 HAS_DAP2)
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
is_enabled(ENABLE_DAP4 HAS_DAP4)
Master merge, conflict resolution, cleanup
2019-02-25 07:54:13 +08:00
is_enabled(ENABLE_BYTERANGE HAS_BYTERANGE)
Fix https://github.com/Unidata/netcdf-c/issues/963 Fix https://github.com/Unidata/netcdf-c/issues/962 1. remove the --disable-diskless option since it is no longer needed. Similarly for CMakeLists.txt. 2. Fixed nc4files.c where BAIL and return were mixed leading to situation where cleanup code was not being invoked. This probably occurs elsewhere, but I did not find any specifically.
2018-05-12 05:30:19 +08:00
is_enabled(ENABLE_DISKLESS HAS_DISKLESS)
Continued to flesh out cmake support for libnetcdf.settings. Added a pair of functions which will return a yes/no depending on whether or not a variable is defined. By default, cmake uses TRUE/FALSE.
2014-06-03 05:14:27 +08:00
is_enabled(USE_MMAP HAS_MMAP)
is_enabled(JNA HAS_JNA)
error out when conflicts with PnetCDF are detected
2018-07-17 03:34:22 +08:00
is_enabled(ENABLE_ZERO_LENGTH_COORD_BOUND RELAX_COORD_BOUND)
Syntax corrections.
2017-09-27 04:01:21 +08:00
is_enabled(USE_CDF5 HAS_CDF5)
Fixes in support of https://github.com/Unidata/netcdf-c/issues/805
2018-01-26 04:07:33 +08:00
is_enabled(ENABLE_ERANGE_FILL HAS_ERANGE_FILL)
Corrected an issue with parallel filter test logic in cmake-based builds.
2020-04-02 07:08:24 +08:00
is_enabled(HDF5_HAS_PAR_FILTERS HAS_PAR_FILTERS)
Fix byterange handling of some URLS re: Issue The byterange handling of the following URLS fails. ### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes" It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD). I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time. ### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3" When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case. ## Misc. Related Changes 1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support 2. Added the above URLS as additional test cases ## Misc. Un-Related Changes 1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD 2. Fix some minor warnings ## Open Problems * Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-03 10:51:02 +08:00
is_enabled(ENABLE_S3 HAS_S3)
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
is_enabled(ENABLE_S3_AWS HAS_S3_AWS)
is_enabled(ENABLE_S3_INTERNAL HAS_S3_INTERNAL)
is_enabled(HAS_HDF5_ROS3 HAS_HDF5_ROS3)
This PR adds EXPERIMENTAL support for accessing data in the cloud using a variant of the Zarr protocol and storage format. This enhancement is generically referred to as "NCZarr". The data model supported by NCZarr is netcdf-4 minus the user-defined types and the String type. In this sense it is similar to the CDF-5 data model. More detailed information about enabling and using NCZarr is described in the document NUG/nczarr.md and in a [Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in). WARNING: this code has had limited testing, so do use this version for production work. Also, performance improvements are ongoing. Note especially the following platform matrix of successful tests: Platform | Build System | S3 support ------------------------------------ Linux+gcc | Automake | yes Linux+gcc | CMake | yes Visual Studio | CMake | no Additionally, and as a consequence of the addition of NCZarr, major changes have been made to the Filter API. NOTE: NCZarr does not yet support filters, but these changes are enablers for that support in the future. Note that it is possible (probable?) that there will be some accidental reversions if the changes here did not correctly mimic the existing filter testing. In any case, previously filter ids and parameters were of type unsigned int. In order to support the more general zarr filter model, this was all converted to char*. The old HDF5-specific, unsigned int operations are still supported but they are wrappers around the new, char* based nc_filterx_XXX functions. This entailed at least the following changes: 1. Added the files libdispatch/dfilterx.c and include/ncfilter.h 2. Some filterx utilities have been moved to libdispatch/daux.c 3. A new entry, "filter_actions" was added to the NCDispatch table and the version bumped. 4. An overly complex set of structs was created to support funnelling all of the filterx operations thru a single dispatch "filter_actions" entry. 5. Move common code to from libhdf5 to libsrc4 so that it is accessible to nczarr. Changes directly related to Zarr: 1. Modified CMakeList.txt and configure.ac to support both C and C++ -- this is in support of S3 support via the awd-sdk libraries. 2. Define a size64_t type to support nczarr. 3. More reworking of libdispatch/dinfermodel.c to support zarr and to regularize the structure of the fragments section of a URL. Changes not directly related to Zarr: 1. Make client-side filter registration be conditional, with default off. 2. Hack include/nc4internal.h to make some flags added by Ed be unique: e.g. NC_CREAT, NC_INDEF, etc. 3. cleanup include/nchttp.h and libdispatch/dhttp.c. 4. Misc. changes to support compiling under Visual Studio including: * Better testing under windows for dirent.h and opendir and closedir. 5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags and to centralize error reporting. 6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them. 7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible. Changes Left TO-DO: 1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
is_enabled(ENABLE_NCZARR HAS_NCZARR)
Turn nczarr zip support off by default in cmake, add a status line indicating whether nczarr-zip-support is available, in libnetcdf.settings.
2023-02-28 04:34:26 +08:00
is_enabled(ENABLE_NCZARR_ZIP HAS_NCZARR_ZIP)
Add zip file support The primary change is to support the use of a zip file as a storage format. Simultaneously the .nz4 support is made obsolete Use of zip requires the libzip support library, so a number of changes to the build files (Makefile.am, CMakeLists.txt) are necessary to locate and incorporate libzip. The nczarr_tests tests are also changed to add zip testing. Other changes: * Make sure distcheck leaves no files around. * Add some functions to netcdf_aux to export some functions of libnetcdf. * Add a new error NC_EFOUND as the complement of NC_EEMPTY. * Add tracing support to nclog and use it in libnczarr. * Modify the zmap interface to support the writeonce semantics of zip. * Create a new s3util.c to support a variety of S3 auxilliary functions. * EXTERNL'ize a number of functions so they can be used in s3util. * Add support for the S3 ListObjects CommonPrefixes mechanism to improve search. * Add experimental support for running nczarr X s3 tests against the actual Amazon S3 cloud.
2021-01-29 11:11:01 +08:00
is_enabled(ENABLE_NCZARR_ZIP DO_NCZARR_ZIP_TESTS)
getting ready for next try at quantization code
2021-08-24 14:45:38 +08:00
is_enabled(ENABLE_QUANTIZE HAS_QUANTIZE)
added info about logging to build summary, fixed info about byterange in build summary
2021-08-24 14:17:03 +08:00
is_enabled(ENABLE_LOGGING HAS_LOGGING)
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
is_enabled(ENABLE_FILTER_TESTING DO_FILTER_TESTS)
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
is_enabled(HAVE_SZ HAS_SZIP)
is_enabled(HAVE_SZ HAS_SZLIB_WRITE)
added ZSTD to netcdf_meta.h and libnetcdf.settings
2022-04-11 22:03:24 +08:00
is_enabled(HAVE_ZSTD HAS_ZSTD)
Update cmakelists.txt in support of https://github.com/Unidata/netcdf-c/issues/2511
2022-09-21 05:24:33 +08:00
is_enabled(HAVE_BLOSC HAS_BLOSC)
is_enabled(HAVE_BZ2 HAS_BZ2)
Change the network access nomenclature to be less ambiguous. As it stood, did it refer to networked drives? Other things that are network adjacent?
2023-06-13 05:23:44 +08:00
is_enabled(ENABLE_REMOTE_FUNCTIONALITY DO_REMOTE_FUNCTIONALITY)
## Addendum [5/9/23] It turns out that attempting to test S3 using a github action secret is a very complex process. So, this was disabled for github actions. However, a new *run_tests_s3.yml* action file was added that will eventually encapsulate S3 testing.
2023-05-10 11:13:49 +08:00
if(ENABLE_S3_INTERNAL)
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
SET(WHICH_S3_SDK "internal")
SET(NC_WHICH_S3_SDK "internal")
## Addendum [5/9/23] It turns out that attempting to test S3 using a github action secret is a very complex process. So, this was disabled for github actions. However, a new *run_tests_s3.yml* action file was added that will eventually encapsulate S3 testing.
2023-05-10 11:13:49 +08:00
elseif(ENABLE_S3_AWS)
SET(WHICH_S3_SDK "aws-sdk-cpp")
SET(NC_WHICH_S3_SDK "aws-sdk-cpp")
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
else()
SET(WHICH_S3_SDK "none")
SET(NC_WHICH_S3_SDK "none")
endif()
## Addendum [5/9/23] It turns out that attempting to test S3 using a github action secret is a very complex process. So, this was disabled for github actions. However, a new *run_tests_s3.yml* action file was added that will eventually encapsulate S3 testing.
2023-05-10 11:13:49 +08:00
if(WITH_S3_TESTING STREQUAL PUBLIC)
SET(DO_S3_TESTING "public")
elseif(WITH_S3_TESTING)
SET(DO_S3_TESTING "yes")
elseif(NOT WITH_S3_TESTING)
SET(DO_S3_TESTING "no")
else()
SET(DO_S3_TESTING "no")
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
endif()
Turn nczarr zip support off by default in cmake, add a status line indicating whether nczarr-zip-support is available, in libnetcdf.settings.
2023-02-28 04:34:26 +08:00
Continued to flesh out cmake support for libnetcdf.settings. Added a pair of functions which will return a yes/no depending on whether or not a variable is defined. By default, cmake uses TRUE/FALSE.
2014-06-03 05:14:27 +08:00
# Generate file from template.
Roughed in skeleton of cmake support for libnetcdf.settings.
2014-06-03 04:03:42 +08:00
CONFIGURE_FILE("${CMAKE_CURRENT_SOURCE_DIR}/libnetcdf.settings.in"
"${CMAKE_CURRENT_BINARY_DIR}/libnetcdf.settings"
Manually merging pull request https://github.com/Unidata/netcdf-c/pull/74 contributed by Nico Schlömer
2014-08-21 02:30:11 +08:00
@ONLY)
Roughed in skeleton of cmake support for libnetcdf.settings.
2014-06-03 04:03:42 +08:00
# Read in settings file, print out.
# Avoid using system-specific calls so that this
# might also work on Windows.
FILE(READ "${CMAKE_CURRENT_BINARY_DIR}/libnetcdf.settings"
LIBNETCDF_SETTINGS)
This PR adds EXPERIMENTAL support for accessing data in the cloud using a variant of the Zarr protocol and storage format. This enhancement is generically referred to as "NCZarr". The data model supported by NCZarr is netcdf-4 minus the user-defined types and the String type. In this sense it is similar to the CDF-5 data model. More detailed information about enabling and using NCZarr is described in the document NUG/nczarr.md and in a [Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in). WARNING: this code has had limited testing, so do use this version for production work. Also, performance improvements are ongoing. Note especially the following platform matrix of successful tests: Platform | Build System | S3 support ------------------------------------ Linux+gcc | Automake | yes Linux+gcc | CMake | yes Visual Studio | CMake | no Additionally, and as a consequence of the addition of NCZarr, major changes have been made to the Filter API. NOTE: NCZarr does not yet support filters, but these changes are enablers for that support in the future. Note that it is possible (probable?) that there will be some accidental reversions if the changes here did not correctly mimic the existing filter testing. In any case, previously filter ids and parameters were of type unsigned int. In order to support the more general zarr filter model, this was all converted to char*. The old HDF5-specific, unsigned int operations are still supported but they are wrappers around the new, char* based nc_filterx_XXX functions. This entailed at least the following changes: 1. Added the files libdispatch/dfilterx.c and include/ncfilter.h 2. Some filterx utilities have been moved to libdispatch/daux.c 3. A new entry, "filter_actions" was added to the NCDispatch table and the version bumped. 4. An overly complex set of structs was created to support funnelling all of the filterx operations thru a single dispatch "filter_actions" entry. 5. Move common code to from libhdf5 to libsrc4 so that it is accessible to nczarr. Changes directly related to Zarr: 1. Modified CMakeList.txt and configure.ac to support both C and C++ -- this is in support of S3 support via the awd-sdk libraries. 2. Define a size64_t type to support nczarr. 3. More reworking of libdispatch/dinfermodel.c to support zarr and to regularize the structure of the fragments section of a URL. Changes not directly related to Zarr: 1. Make client-side filter registration be conditional, with default off. 2. Hack include/nc4internal.h to make some flags added by Ed be unique: e.g. NC_CREAT, NC_INDEF, etc. 3. cleanup include/nchttp.h and libdispatch/dhttp.c. 4. Misc. changes to support compiling under Visual Studio including: * Better testing under windows for dirent.h and opendir and closedir. 5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags and to centralize error reporting. 6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them. 7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible. Changes Left TO-DO: 1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
MESSAGE(STATUS ${LIBNETCDF_SETTINGS})
Roughed in skeleton of cmake support for libnetcdf.settings.
2014-06-03 04:03:42 +08:00
# Install libnetcdf.settings file into same location
# as the libraries.
INSTALL(FILES "${netCDF_BINARY_DIR}/libnetcdf.settings"
Fixed installation location for libnetcdf.settings file.
2014-06-18 04:25:49 +08:00
DESTINATION "${CMAKE_INSTALL_LIBDIR}"
Roughed in skeleton of cmake support for libnetcdf.settings.
2014-06-03 04:03:42 +08:00
COMPONENT libraries)
#####
# End libnetcdf.settings section.
#####
Initial version of template header file.
2014-09-05 05:42:11 +08:00
#####
# Create 'netcdf_meta.h' include file.
#####
configure_file(
Have finished with autotools-based netcdf_meta.h template. Need to redo cmake-based template to match behavior. Modified cmake to use the same netcdf_meta template as autotools. Removed trace of old cmake template file.
2014-09-11 06:50:45 +08:00
${netCDF_SOURCE_DIR}/include/netcdf_meta.h.in
Configure netcdf_meta.h directly to BINARY_DIR
2020-08-12 22:09:28 +08:00
${netCDF_BINARY_DIR}/include/netcdf_meta.h @ONLY)
Initial version of template header file.
2014-09-05 05:42:11 +08:00
Unify definition of NC_DISPATCH_VERSION re: Issue The netcdf dispatch table version was defined in several places. Modify to only require defining it in CMakeLists.txt and configure.ac. Fix entailed the following changes: * Up the NC_DISPATCH_VERSION from 2 to 3 in configure.ac and CMakeLists.txt * Create include/netcdf_dispatch.h.in and use it to configure include/netcdf_dispatch.h * For CMAKE, make it search CMAKE_CURRENT_BINARY_DIR so code can locate the configured netcdf_dispatch.h * Add entry to config.h.cmake.in for NC_DISPATCH_VERSION * Move NCerror from include/ncdispatch.h to libdap2/nccomon.h * Fix an API problem re nchttp.h * Fix a conversion warning in libdispatch/dinfermodel.c
2021-02-01 12:40:08 +08:00
#####
# Create 'netcdf_dispatch.h' include file.
#####
configure_file(
${netCDF_SOURCE_DIR}/include/netcdf_dispatch.h.in
${netCDF_BINARY_DIR}/include/netcdf_dispatch.h @ONLY NEWLINE_STYLE LF)
Fix use of ininitialized CMake variable The `ISCMAKE` was defined after its first use in a `configure_file` call. Moved to before its first use.
2021-04-01 22:13:13 +08:00
####
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
# Build test_common.sh
#####
Changes to make it possible to nest this project inside of another CMake project.
2019-11-07 00:43:49 +08:00
SET(EXTRA_DIST ${EXTRA_DIST} ${CMAKE_CURRENT_SOURCE_DIR}/test_common.in)
SET(TOPSRCDIR "${CMAKE_CURRENT_SOURCE_DIR}")
SET(TOPBUILDDIR "${CMAKE_CURRENT_BINARY_DIR}")
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/test_common.in ${CMAKE_CURRENT_BINARY_DIR}/test_common.sh @ONLY NEWLINE_STYLE LF)
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
1. Keep up to date by merging master 2. Fixed plugin building (nc_test4/hdf5plugins) to be done properly by cmake and automake. 4. Duplicated part of the nc_test4 filter test code in examples/C An incomplete and untested set of hooks exist for OS-X in nc_test4/findplugins.in. They need testing.
2018-01-17 02:00:09 +08:00
#####
Improve filter installation process to avoid use of an extra shell script re: https://github.com/Unidata/netcdf-c/issues/2338 re: https://github.com/Unidata/netcdf-c/issues/2294 In issue https://github.com/Unidata/netcdf-c/issues/2338, Ed Hartnett suggested a better way to install filters to a user defined location -- for Automake, anyway. This PR implements that suggestion. It turns out to be more complicated than it appears, so there are fair number of changes; mostly to shell scripts. Most of the change is in plugins/Makefile.am. NOTE: this PR still does NOT address the use of HDF5_PLUGIN_PATH as the default; this turns out to be complex when dealing with NCZarr. So this will be addressed in a subsequent post 4.9.0 PR. ## Misc. Changes 1. Record the occurrences of incomplete codecs in libnczarr so that they can be included in _Codecs attribute correctly. This allows users to see what missing filters are referenced in the Zarr file. Primarily affects libnczarr/zfilter.[ch]. Also required creating a new no-effect filter: H5Zunknown.c. 2. Move the unknown filter test to a separate test file. 3. Incorporates PR https://github.com/Unidata/netcdf-c/pull/2343
2022-05-15 06:05:48 +08:00
# Build and copy nc_test4/findplugin.sh to various places
1. Keep up to date by merging master 2. Fixed plugin building (nc_test4/hdf5plugins) to be done properly by cmake and automake. 4. Duplicated part of the nc_test4 filter test code in examples/C An incomplete and untested set of hooks exist for OS-X in nc_test4/findplugins.in. They need testing.
2018-01-17 02:00:09 +08:00
#####
Changes to make it possible to nest this project inside of another CMake project.
2019-11-07 00:43:49 +08:00
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/nc_test4/findplugin.in ${CMAKE_CURRENT_BINARY_DIR}/nc_test4/findplugin.sh @ONLY NEWLINE_STYLE LF)
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/nc_test4/findplugin.in ${CMAKE_CURRENT_BINARY_DIR}/nczarr_test/findplugin.sh @ONLY NEWLINE_STYLE LF)
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=&lt;absolute directory path&gt; cmake option: -DPLUGIN_INSTALL_DIR=&lt;absolute directory path&gt; ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/nc_test4/findplugin.in ${CMAKE_CURRENT_BINARY_DIR}/plugins/findplugin.sh @ONLY NEWLINE_STYLE LF)
Changes to make it possible to nest this project inside of another CMake project.
2019-11-07 00:43:49 +08:00
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/nc_test4/findplugin.in ${CMAKE_CURRENT_BINARY_DIR}/examples/C/findplugin.sh @ONLY NEWLINE_STYLE LF)
1. Keep up to date by merging master 2. Fixed plugin building (nc_test4/hdf5plugins) to be done properly by cmake and automake. 4. Duplicated part of the nc_test4 filter test code in examples/C An incomplete and untested set of hooks exist for OS-X in nc_test4/findplugins.in. They need testing.
2018-01-17 02:00:09 +08:00
Add generated parallel tests for nc_perf, cmake-based build system.
2023-02-17 06:52:43 +08:00
IF(ENABLE_BENCHMARKS)
IF(ENABLE_PARALLEL4)
CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/nc_perf/run_par_bm_test.sh.in ${CMAKE_CURRENT_BINARY_DIR}/nc_perf/run_par_bm_test.sh @ONLY NEWLINE_STYLE LF)
CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/nc_perf/run_gfs_test.sh.in ${CMAKE_CURRENT_BINARY_DIR}/nc_perf/run_gfs_test.sh @ONLY NEWLINE_STYLE LF)
ENDIF(ENABLE_PARALLEL4)
Fix missing endif statement
2023-02-17 06:56:25 +08:00
ENDIF(ENABLE_BENCHMARKS)
Move generated file to BINARY_DIR
2020-08-12 22:38:00 +08:00
IF(ENABLE_TESTS)
#####
# Build ncdap_test|dap4_test/findtestserver[4].c
#####
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/ncdap_test/findtestserver.c.in ${CMAKE_CURRENT_BINARY_DIR}/ncdap_test/findtestserver.c @ONLY NEWLINE_STYLE LF)
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/ncdap_test/findtestserver.c.in ${CMAKE_CURRENT_BINARY_DIR}/dap4_test/findtestserver4.c @ONLY NEWLINE_STYLE LF)
Fix the NC_INMEMORY code to work in all cases with HDF5 1.10. re: github issue https://github.com/Unidata/netcdf-c/issues/1111 One of the less common use cases for the in-memory feature is apparently failing with HDF5-1.10.x. The fix is complicated and requires significant changes to libhdf5/nc4memcb.c. The current setup is detailed in the file docs/inmeminternal.dox. Additionally, it was discovered that the program nc_test/tst_inmemory.c, which is invoked by nc_test/run_inmemory.sh, actually was failing because of the above problem. But the failure is not detected since the script does not return non-zero value. Other Changes: 1. Fix nc_test_tst_inmemory to return errors correctly. 2. Make ncdap_tests/findtestserver.c and dap4_tests/findtestserver4.c be generated from ncdap_test/findtestserver.c.in. 3. Make LOG() print output to stderr instead of stdout to avoid contaminating e.g. ncdump output. 4. Modify the handling of NC_INMEMORY and NC_DISKLESS flags to properly handle that NC_DISKLESS => NC_INMEMORY. This affects a number of code pieces, especially memio.c.
2018-09-05 01:27:47 +08:00
Move generated file to BINARY_DIR
2020-08-12 22:38:00 +08:00
#####
# Build dap4_test/pingurl4.c
#####
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/ncdap_test/pingurl.c ${CMAKE_CURRENT_BINARY_DIR}/dap4_test/pingurl4.c @ONLY NEWLINE_STYLE LF)
ENDIF()
Forgot to create pingurl4.c for Cmake
2020-01-03 02:47:25 +08:00
Cleaning up make dist, it's generating tarballs that don't work with cmake.
2022-02-02 02:40:01 +08:00
if(DEFINED ENV{LIB_FUZZING_ENGINE})
add_subdirectory(fuzz)
endif(DEFINED ENV{LIB_FUZZING_ENGINE})
Adds fuzz target for oss-fuzz integration
2021-03-31 17:27:27 +08:00
move export functionality towards the end of CMakeLists.txt This ensure that all referenced variables are actually defined.
2014-08-22 01:09:18 +08:00
####
# Export files
####
# Create CMake package configuration files. With these, other packages using
# cmake should be able to find netcdf using find_package and find_library.
# The EXPORT call is paired with one in liblib.
set(ConfigPackageLocation ${CMAKE_INSTALL_LIBDIR}/cmake/netCDF)
Moved dependencies into proper location in relation to netcdf.dll to avoid 'can't find hdf5.dll, etc' errors.
2014-12-06 02:19:02 +08:00
move export functionality towards the end of CMakeLists.txt This ensure that all referenced variables are actually defined.
2014-08-22 01:09:18 +08:00
install(EXPORT netCDFTargets
DESTINATION ${ConfigPackageLocation}
Moved dependencies into proper location in relation to netcdf.dll to avoid 'can't find hdf5.dll, etc' errors.
2014-12-06 02:19:02 +08:00
COMPONENT headers
Add namespace for CMake targets The usual convention with modern CMake is to namespace the targets. This gives the netCDF namespace to the package.
2019-11-20 03:45:21 +08:00
NAMESPACE netCDF::
move export functionality towards the end of CMakeLists.txt This ensure that all referenced variables are actually defined.
2014-08-22 01:09:18 +08:00
)
include(CMakePackageConfigHelpers)
CONFIGURE_PACKAGE_CONFIG_FILE(
"${CMAKE_CURRENT_SOURCE_DIR}/netCDFConfig.cmake.in"
"${CMAKE_CURRENT_BINARY_DIR}/netCDFConfig.cmake"
INSTALL_DESTINATION "${ConfigPackageLocation}"
NO_CHECK_REQUIRED_COMPONENTS_MACRO
PATH_VARS
CMAKE_INSTALL_PREFIX
CMAKE_INSTALL_INCLUDEDIR
CMAKE_INSTALL_LIBDIR
)
INSTALL(
FILES "${CMAKE_CURRENT_BINARY_DIR}/netCDFConfig.cmake"
DESTINATION "${ConfigPackageLocation}"
Moved dependencies into proper location in relation to netcdf.dll to avoid 'can't find hdf5.dll, etc' errors.
2014-12-06 02:19:02 +08:00
COMPONENT headers
move export functionality towards the end of CMakeLists.txt This ensure that all referenced variables are actually defined.
2014-08-22 01:09:18 +08:00
)
Add namespace for CMake targets The usual convention with modern CMake is to namespace the targets. This gives the netCDF namespace to the package.
2019-11-20 03:45:21 +08:00
add_library(netCDF::netcdf ALIAS netcdf)
Add include directories as netcdf property Modern CMake tracks the properties of targets so that down stream libraries do not have to worry about include directories. However, the include directories must be added to the target and not just at the directory level.
2019-11-20 03:39:46 +08:00
target_include_directories(netcdf
PUBLIC
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>
)
move export functionality towards the end of CMakeLists.txt This ensure that all referenced variables are actually defined.
2014-08-22 01:09:18 +08:00
# Create export configuration
write_basic_package_version_file(
"${CMAKE_CURRENT_BINARY_DIR}/netCDF/netCDFConfigVersion.cmake"
VERSION ${netCDF_VERSION}
COMPATIBILITY SameMajorVersion
)
install(
FILES
"${CMAKE_CURRENT_BINARY_DIR}/netCDF/netCDFConfigVersion.cmake"
DESTINATION ${ConfigPackageLocation}
Moved dependencies into proper location in relation to netcdf.dll to avoid 'can't find hdf5.dll, etc' errors.
2014-12-06 02:19:02 +08:00
COMPONENT headers
move export functionality towards the end of CMakeLists.txt This ensure that all referenced variables are actually defined.
2014-08-22 01:09:18 +08:00
)
####
# End export files
####
2012-10-03 04:56:46 +08:00
# CPack inclusion must come last.
Modified Settings used to generate Binary installer for Windows, so that the user is given the option of adding the binaries to the system path.
2014-12-03 04:58:23 +08:00
# INCLUDE(CPack)
INCLUDE(CMakeInstallation.cmake)
Reference in New Issue Copy Permalink