mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2025-01-12 15:45:21 +08:00
Code Issues Packages Projects Releases Wiki Activity
320161f2e6
netcdf-c/plugins/CMakeLists.txt

145 lines
4.6 KiB
CMake
Raw Normal View History

Misc. files updated with copyright stanza.
2018-12-07 06:51:35 +08:00
# Copyright 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002,
# 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014,
# 2015, 2016, 2017, 2018
# University Corporation for Atmospheric Research/Unidata.
# See netcdf-c/COPYRIGHT file for more info.
1. Keep up to date by merging master 2. Fixed plugin building (nc_test4/hdf5plugins) to be done properly by cmake and automake. 4. Duplicated part of the nc_test4 filter test code in examples/C An incomplete and untested set of hooks exist for OS-X in nc_test4/findplugins.in. They need testing.
2018-01-17 02:00:09 +08:00
SET(CMAKE_BUILD_TYPE "")
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
IF(MSVC)
SET(PLUGINEXT "dll")
SET(PLUGINPRE "__nc")
ELSE()
Update to default --with-plugin-dir to yes
2022-05-25 10:05:19 +08:00
SET(PLUGINPRE "lib__nc")
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
if(APPLE)
SET(PLUGINEXT "dylib")
ELSE()
SET(PLUGINEXT "so")
ENDIF()
ENDIF()
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
FILE(READ H5Znoop.c NOOP_SOURCE)
FILE(WRITE ${CMAKE_CURRENT_BINARY_DIR}/H5Znoop1.c "#define NOOP_INSTANCE 1\n")
FILE(APPEND ${CMAKE_CURRENT_BINARY_DIR}/H5Znoop1.c "${NOOP_SOURCE}")
H5Znoop1.c copied to BINARY_DIR instead of SOURCE_DIR
2020-08-12 23:03:33 +08:00
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
SET(h5misc_SOURCES H5Zmisc.c H5Zutil.c h5misc.h)
1. Keep up to date by merging master 2. Fixed plugin building (nc_test4/hdf5plugins) to be done properly by cmake and automake. 4. Duplicated part of the nc_test4 filter test code in examples/C An incomplete and untested set of hooks exist for OS-X in nc_test4/findplugins.in. They need testing.
2018-01-17 02:00:09 +08:00
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
SET(h5noop_SOURCES H5Znoop.c H5Zutil.c h5noop.h)
SET_SOURCE_FILES_PROPERTIES(H5Znoop.c PROPERTIES COMPILE_OPTIONS -DNOOP_INSTANCE=0)
SET(h5noop1_SOURCES ${CMAKE_CURRENT_BINARY_DIR}/H5Znoop1.c H5Zutil.c h5noop.h)
SET_SOURCE_FILES_PROPERTIES(${CMAKE_CURRENT_BINARY_DIR}/H5Znoop1.c PROPERTIES COMPILE_OPTIONS -DNOOP_INSTANCE=1)
Improve filter installation process to avoid use of an extra shell script re: https://github.com/Unidata/netcdf-c/issues/2338 re: https://github.com/Unidata/netcdf-c/issues/2294 In issue https://github.com/Unidata/netcdf-c/issues/2338, Ed Hartnett suggested a better way to install filters to a user defined location -- for Automake, anyway. This PR implements that suggestion. It turns out to be more complicated than it appears, so there are fair number of changes; mostly to shell scripts. Most of the change is in plugins/Makefile.am. NOTE: this PR still does NOT address the use of HDF5_PLUGIN_PATH as the default; this turns out to be complex when dealing with NCZarr. So this will be addressed in a subsequent post 4.9.0 PR. ## Misc. Changes 1. Record the occurrences of incomplete codecs in libnczarr so that they can be included in _Codecs attribute correctly. This allows users to see what missing filters are referenced in the Zarr file. Primarily affects libnczarr/zfilter.[ch]. Also required creating a new no-effect filter: H5Zunknown.c. 2. Move the unknown filter test to a separate test file. 3. Incorporates PR https://github.com/Unidata/netcdf-c/pull/2343
2022-05-15 06:05:48 +08:00
SET(h5unknown_SOURCES H5Zunknown.c)
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
SET(h5shuffle_SOURCES H5Zshuffle.c)
SET(h5fletcher32_SOURCES H5Zfletcher32.c H5checksum.c)
SET(h5deflate_SOURCES H5Zdeflate.c)
SET(nczmisc_SOURCES NCZmisc.c)
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=<absolute directory path> cmake option: -DPLUGIN_INSTALL_DIR=<absolute directory path> ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
SET(nczhdf5filters_SOURCES NCZhdf5filters.c)
SET(nczstdfilters_SOURCES NCZstdfilters.c)
1. Keep up to date by merging master 2. Fixed plugin building (nc_test4/hdf5plugins) to be done properly by cmake and automake. 4. Duplicated part of the nc_test4 filter test code in examples/C An incomplete and untested set of hooks exist for OS-X in nc_test4/findplugins.in. They need testing.
2018-01-17 02:00:09 +08:00
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=<absolute directory path> cmake option: -DPLUGIN_INSTALL_DIR=<absolute directory path> ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
IF(ENABLE_PLUGINS)
1. Keep up to date by merging master 2. Fixed plugin building (nc_test4/hdf5plugins) to be done properly by cmake and automake. 4. Duplicated part of the nc_test4 filter test code in examples/C An incomplete and untested set of hooks exist for OS-X in nc_test4/findplugins.in. They need testing.
2018-01-17 02:00:09 +08:00
# LDFLAGS = -module -avoid-version -shared -export-dynamic -no-undefined
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
SET(CMAKE_LIBRARY_OUTPUT_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}")
SET(CMAKE_LIBRARY_OUTPUT_DIRECTORY_DEBUG "${CMAKE_CURRENT_BINARY_DIR}")
SET(CMAKE_LIBRARY_OUTPUT_DIRECTORY_RELEASE "${CMAKE_CURRENT_BINARY_DIR}")
MACRO(buildplugin TARGET TARGETLIB)
Update to default --with-plugin-dir to yes
2022-05-25 10:05:19 +08:00
SET(MANGLELIB "${PLUGINPRE}${TARGETLIB}")
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
ADD_LIBRARY(${TARGET} MODULE ${${TARGET}_SOURCES})
Update to default --with-plugin-dir to yes
2022-05-25 10:05:19 +08:00
SET_TARGET_PROPERTIES(${TARGET} PROPERTIES LIBRARY_OUTPUT_NAME ${MANGLELIB})
SET_TARGET_PROPERTIES(${TARGET} PROPERTIES ARCHIVE_OUTPUT_NAME ${MANGLELIB})
SET_TARGET_PROPERTIES(${TARGET} PROPERTIES RUNTIME_OUTPUT_NAME ${MANGLELIB})
SET_TARGET_PROPERTIES(${TARGET} PROPERTIES PREFIX "") # Critical that this be set to ""
Specify correct plugin suffix CMake by default uses the ".so" suffix for libraries with the `MODULE` attribute. Specify the desired suffix with by setting the `SUFFIX` property explicitly on the plugin targets. Fixes plugin install issue on Macs.
2022-06-08 21:45:03 +08:00
SET_TARGET_PROPERTIES(${TARGET} PROPERTIES SUFFIX ".${PLUGINEXT}")
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
TARGET_LINK_LIBRARIES(${TARGET} ${ALL_TLL_LIBS};${ARGN})
IF(MSVC)
target_compile_options(${TARGET} PRIVATE /Zi)
# Tell linker to include symbol data
set_target_properties(${TARGET} PROPERTIES LINK_FLAGS "/INCREMENTAL:NO /DEBUG /OPT:REF /OPT:ICF")
# Set file name & location
set_target_properties(${TARGET} PROPERTIES COMPILE_PDB_NAME ${TARGET} COMPILE_PDB_OUTPUT_DIR ${CMAKE_BINARY_DIR})
ENDIF()
ENDMACRO()
With Ward's help, I figured out how to obviate the need for build type under cmake+Visual-studio. This simplified findplugin.in.
2018-01-17 04:28:30 +08:00
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
buildplugin(h5misc "h5misc")
buildplugin(h5noop "h5noop")
buildplugin(h5noop1 "h5noop1")
Improve filter installation process to avoid use of an extra shell script re: https://github.com/Unidata/netcdf-c/issues/2338 re: https://github.com/Unidata/netcdf-c/issues/2294 In issue https://github.com/Unidata/netcdf-c/issues/2338, Ed Hartnett suggested a better way to install filters to a user defined location -- for Automake, anyway. This PR implements that suggestion. It turns out to be more complicated than it appears, so there are fair number of changes; mostly to shell scripts. Most of the change is in plugins/Makefile.am. NOTE: this PR still does NOT address the use of HDF5_PLUGIN_PATH as the default; this turns out to be complex when dealing with NCZarr. So this will be addressed in a subsequent post 4.9.0 PR. ## Misc. Changes 1. Record the occurrences of incomplete codecs in libnczarr so that they can be included in _Codecs attribute correctly. This allows users to see what missing filters are referenced in the Zarr file. Primarily affects libnczarr/zfilter.[ch]. Also required creating a new no-effect filter: H5Zunknown.c. 2. Move the unknown filter test to a separate test file. 3. Incorporates PR https://github.com/Unidata/netcdf-c/pull/2343
2022-05-15 06:05:48 +08:00
buildplugin(h5unknown "h5unknown")
Add support for multiple filters per variable. re: https://github.com/Unidata/netcdf-c/issues/1584 Support has been added for multiple filters per variable. This affects a number of components in netcdf. The new APIs are documented in NUG/filters.md. The primary changes are: * A set of new functions are provided (see __include/netcdf_filter.h__). - Obtain a list of the filters associated with a variable - Obtain the parameters for a specific filter. * The existing __nc_inq_var_filter__ function now returns info about the first defined filter. * The utilities (ncgen, ncdump, and nccopy) now support an extended format for specifying a sequence of filters. The general form is __<filter>|<filter>..._. * The ncdump **_Filter** attribute now dumps a list of all the filters associated with a variable using the above new format. * Filter specifications can now use a filter name instead of number for filters known to the netcdf library, which in turn is taken from the HDF5 filter registration page. * New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter is returned if an attempt is made to access an unknown filter. * Internally, the dispatch table has been extended to add a function to handle all of the filter functions. * New, filter-related, tests were added to nc_test4. * A new plugin was added to the plugins directory to help with testing. Notes: 1. The shuffle and fletcher32 filters are not part of the multifilter system. Misc. changes: 1. A debug module was added to libhdf5 to help catch error locations.
2020-02-17 03:59:33 +08:00
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
buildplugin(h5shuffle "h5shuffle")
buildplugin(h5fletcher32 "h5fletcher32")
buildplugin(h5deflate "h5deflate")
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
buildplugin(nczmisc "zmisc")
buildplugin(nczhdf5filters "zhdf5filters" netcdf)
buildplugin(nczstdfilters "zstdfilters" netcdf)
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
IF(ENABLE_BLOSC)
SET(h5blosc_SOURCES H5Zblosc.c)
buildplugin(h5blosc "h5blosc" netcdf;${Blosc_LIBRARIES})
ENDIF()
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
IF(ENABLE_ZSTD)
SET(h5zstd_SOURCES H5Zzstd.c H5Zzstd.h)
buildplugin(h5zstd "h5zstd" netcdf;${Zstd_LIBRARIES})
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
ENDIF()
Mostly revert the filter code to reduce its complexity of use. re: https://github.com/Unidata/netcdf-c/issues/1836 Revert the internal filter code to simplify it. From the user's point of view, the only visible changes should be: 1. The functions that convert text to filter specs have had their signature reverted and have been moved to netcdf_aux.h 2. Some filter API functions now return NC_ENOFILTER when inquiry is made about some filter. Internally,the dispatch table has been modified to get rid of the filter_actions entry and associated complex structures. It has been replaced with inq_var_filter_ids and inq_var_filter_info entries and the dispatch table version has been bumped to 3. Corresponding NOOP and NOTNC4 functions were added to libdispatch/dnotnc4.c. Also, the filter_action entries in dispatch tables were replaced for all dispatch code bases (HDF5, DAP2, etc). This should only impact UDF users. In the process, it became clear that the form of the filters field in NC_VAR_INFO_T was format dependent, so I converted it to be of type void* and pushed its management into the various dispatch code bases. Specifically libhdf5 and libnczarr now manage the filters field in their own way. The auxilliary functions for parsing textual filter specifications were moved to netcdf_aux.h and were renamed to the following: * ncaux_h5filterspec_parse * ncaux_h5filterspec_parselist * ncaux_h5filterspec_free * ncaux_h5filter_fix8 Misc. Other Changes: 1. Document NUG/filters.md updated to reflect the changes above. 2. All the old data types (structs and enums) used by filter_actions actions were deleted. The exception is the NC_H5_Filterspec because it is needed by ncaux_h5filterspec_parselist. 3. Clientside filters were removed -- another enhancement for which no-one ever asked. 4. The ability to remove filters was itself removed. 5. Some functionality needed by nczarr was moved from libhdf5 to libsrc4 e.g. nc4_find_default_chunksizes 6. All the filterx code was removed 7. ncfilter.h and nc4filter.c no longer used Misc. Unrelated Changes: 1. The nczarr_test makefile clean was leaving some directories; so add clean-local to take care of them.
2020-09-28 02:43:46 +08:00
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
# Need our version of szip if libsz available and we are not using HDF5
IF(HAVE_SZ)
SET(h5szip_SOURCES H5Zszip.c H5Zszip.h)
buildplugin(h5szip "h5szip" ${Szip_LIBRARIES})
ENDIF()
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=&lt;absolute directory path&gt; cmake option: -DPLUGIN_INSTALL_DIR=&lt;absolute directory path&gt; ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
IF(HAVE_LOCAL_BZ2)
SET(h5bzip2_SOURCES H5Zbzip2.c blocksort.c huffman.c crctable.c randtable.c compress.c decompress.c bzlib.c bzlib.h bzlib_private.h)
buildplugin(h5bzip2 "h5bzip2")
ELSE()
SET(h5bzip2_SOURCES H5Zbzip2.c)
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
buildplugin(h5bzip2 "h5bzip2" ${Bzip2_LIBRARIES})
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=&lt;absolute directory path&gt; cmake option: -DPLUGIN_INSTALL_DIR=&lt;absolute directory path&gt; ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
ENDIF()
# Installation
IF(ENABLE_PLUGIN_INSTALL)
MACRO(installplugin PLUG)
Update to default --with-plugin-dir to yes
2022-05-25 10:05:19 +08:00
SET(INSTALLED_PLUGIN_LIB "${PLUGINPRE}${PLUG}.${PLUGINEXT}")
MESSAGE(STATUS "Installing: ${INSTALLED_PLUGIN_LIB} into ${PLUGIN_INSTALL_DIR}")
install(PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/${INSTALLED_PLUGIN_LIB} DESTINATION ${PLUGIN_INSTALL_DIR})
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=&lt;absolute directory path&gt; cmake option: -DPLUGIN_INSTALL_DIR=&lt;absolute directory path&gt; ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
ENDMACRO()
install(DIRECTORY DESTINATION ${PLUGIN_INSTALL_DIR})
IF(Bzip2_FOUND)
installplugin(h5bzip2)
ENDIF()
IF(Zstd_FOUND)
installplugin(h5zstd)
ENDIF()
IF(Blosc_FOUND)
installplugin(h5blosc)
ENDIF()
IF(ENABLE_NCZARR)
installplugin(h5fletcher32)
installplugin(h5shuffle)
installplugin(h5deflate)
I made a major update to this PR with the following changes: ## Overwriting I think I solved the file overwrite problem by doing light name mangling of the shared library names. With this change the probabilty is very small that installing our filter wrappers in a directory will overwrite code produced by others. ## Default Install Location I have setup the --with-plugin-dir option default to install in the following locations in order of preference 1. If HDF5_PLUGIN_PATH is defined (at build time remember), then the last directory in that path will be where the filter wrapper shared libraries will be installed. 2. Otherwise the default is "/usr/local/hdf5/lib/plugin" (on *nix*) or "%ALLUSERSPROFILE%\\hdf5\\lib\\plugin" for Windows or Mingw. Currently, --with-plugin-dir is disabled by default. I should note that even if I enable it by default, installing netcdf-c will still not run "out of the box" because the hypothetical naive user will not know which compressor libraries need to be pre-installed before netcdf is installed. Nor will that user have any way to find out what needs to be installed.
2022-05-20 12:00:40 +08:00
installplugin(zhdf5filters)
installplugin(zstdfilters)
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=&lt;absolute directory path&gt; cmake option: -DPLUGIN_INSTALL_DIR=&lt;absolute directory path&gt; ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
IF(Szip_FOUND)
installplugin(h5szip)
ENDIF()
ENDIF()
ENDIF(ENABLE_PLUGIN_INSTALL)
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=&lt;absolute directory path&gt; cmake option: -DPLUGIN_INSTALL_DIR=&lt;absolute directory path&gt; ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
ENDIF(ENABLE_PLUGINS)
1. Keep up to date by merging master 2. Fixed plugin building (nc_test4/hdf5plugins) to be done properly by cmake and automake. 4. Duplicated part of the nc_test4 filter test code in examples/C An incomplete and untested set of hooks exist for OS-X in nc_test4/findplugins.in. They need testing.
2018-01-17 02:00:09 +08:00
# Copy some test files from current source dir to out-of-tree build dir.
FILE(COPY ${COPY_FILES} DESTINATION ${CMAKE_CURRENT_BINARY_DIR}/)
IF(MSVC)
FILE(COPY ${COPY_FILES} DESTINATION ${RUNTIME_OUTPUT_DIRECTORY}/)
ENDIF()
Reference in New Issue Copy Permalink