mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2024-12-27 08:49:16 +08:00
Code Issues Packages Projects Releases Wiki Activity
53887b2295
netcdf-c/libnetcdf.settings.in

60 lines
1.5 KiB
Plaintext
Raw Normal View History

Add ability to programmatically extract info from libnetcdf.settings API is below. I have made this API public yet by adding it to netcdf.h. I will do that when everyone is agreed on the proper API. extern const char* nc_settings(const char* key); /*get value of a specific key */ extern const char** nc_settings_all(); /*get all settings in envv format */ extern void nc_settings_reclaim(); /* reclaim all space and clean up */ Envv format is {key,value}*,NULL Also added test: nc_test/tst_settings.c
2014-08-30 04:51:14 +08:00
# NetCDF C Configuration Summary
Regularized libnetcdf.settings.in to make parsing settings file easier
2014-08-25 06:29:44 +08:00
==============================
Added skeleton for libnetcdf.settings. See JIRA ticket at https://bugtracking.unidata.ucar.edu/browse/NCF-303
2014-05-31 03:36:35 +08:00
Add ability to programmatically extract info from libnetcdf.settings API is below. I have made this API public yet by adding it to netcdf.h. I will do that when everyone is agreed on the proper API. extern const char* nc_settings(const char* key); /*get value of a specific key */ extern const char** nc_settings_all(); /*get all settings in envv format */ extern void nc_settings_reclaim(); /* reclaim all space and clean up */ Envv format is {key,value}*,NULL Also added test: nc_test/tst_settings.c
2014-08-30 04:51:14 +08:00
# General
Added skeleton for libnetcdf.settings. See JIRA ticket at https://bugtracking.unidata.ucar.edu/browse/NCF-303
2014-05-31 03:36:35 +08:00
-------
First pass at fleshing out libnetcdf.settings for autotools-based builds.
2014-05-31 05:30:58 +08:00
NetCDF Version: @PACKAGE_VERSION@
Fixed tabs.
2019-11-20 06:14:26 +08:00
Dispatch Version: @NC_DISPATCH_VERSION@
First pass at fleshing out libnetcdf.settings for autotools-based builds.
2014-05-31 05:30:58 +08:00
Configured On: @CONFIG_DATE@
Host System: @host_cpu@-@host_vendor@-@host_os@
Build Directory: @abs_top_builddir@
Fixed tabs.
2019-11-20 06:14:26 +08:00
Install Prefix: @prefix@
Support installation of filters into user-specified location re: https://github.com/Unidata/netcdf-c/issues/2294 Ed Hartnett suggested that the netcdf library installation process be extended to install the standard filters into a user specified location. The user can then set HDF5_PLUGIN_PATH to that location. This PR provides that capability using: ```` configure option: --with-plugin-dir=<absolute directory path> cmake option: -DPLUGIN_INSTALL_DIR=<absolute directory path> ```` Currently, the following plugins are always installed, if available: bzip2, zstd, blosc. If NCZarr is enabled, then additional plugins are installed: fletcher32, shuffle, deflate, szip. Additionally, the necessary codec support is installed for each of the above filters that is installed. ## Changes: 1. Cleanup handling of built-in bzip2. 2. Add documentation to docs/filters.md 3. Re-factor the NCZarr codec libraries 4. Add a test, although it can only be exercised after the library is installed, so it cannot be used during normal testing. 5. Cleanup use of HDF5_PLUGIN_PATH in the filter test cases.
2022-04-30 04:31:55 +08:00
Plugin Install Prefix: @PLUGIN_INSTALL_DIR_SETTING@
Added skeleton for libnetcdf.settings. See JIRA ticket at https://bugtracking.unidata.ucar.edu/browse/NCF-303
2014-05-31 03:36:35 +08:00
Add ability to programmatically extract info from libnetcdf.settings API is below. I have made this API public yet by adding it to netcdf.h. I will do that when everyone is agreed on the proper API. extern const char* nc_settings(const char* key); /*get value of a specific key */ extern const char** nc_settings_all(); /*get all settings in envv format */ extern void nc_settings_reclaim(); /* reclaim all space and clean up */ Envv format is {key,value}*,NULL Also added test: nc_test/tst_settings.c
2014-08-30 04:51:14 +08:00
# Compiling Options
Added skeleton for libnetcdf.settings. See JIRA ticket at https://bugtracking.unidata.ucar.edu/browse/NCF-303
2014-05-31 03:36:35 +08:00
-----------------
Regularized libnetcdf.settings.in to make parsing settings file easier
2014-08-25 06:29:44 +08:00
C Compiler: @CC_VERSION@
CFLAGS: @CFLAGS@
CPPFLAGS: @CPPFLAGS@
LDFLAGS: @LDFLAGS@
AM_CFLAGS: @AM_CFLAGS@
AM_CPPFLAGS: @AM_CPPFLAGS@
AM_LDFLAGS: @AM_LDFLAGS@
Shared Library: @enable_shared@
Static Library: @enable_static@
Extra libraries: @LIBS@
Begin wiring in option to toggle automatic search for libxml2 in support of https://github.com/Unidata/netcdf-c/pulls/2135
2021-11-03 05:35:04 +08:00
XML Parser: @XMLPARSER@
Added skeleton for libnetcdf.settings. See JIRA ticket at https://bugtracking.unidata.ucar.edu/browse/NCF-303
2014-05-31 03:36:35 +08:00
Add ability to programmatically extract info from libnetcdf.settings API is below. I have made this API public yet by adding it to netcdf.h. I will do that when everyone is agreed on the proper API. extern const char* nc_settings(const char* key); /*get value of a specific key */ extern const char** nc_settings_all(); /*get all settings in envv format */ extern void nc_settings_reclaim(); /* reclaim all space and clean up */ Envv format is {key,value}*,NULL Also added test: nc_test/tst_settings.c
2014-08-30 04:51:14 +08:00
# Features
Regularized libnetcdf.settings.in to make parsing settings file easier
2014-08-25 06:29:44 +08:00
--------
Added S3 status to libnetcdf.settings, turned byterange on by default.
2023-01-10 04:11:16 +08:00
Benchmarks: @HAS_BENCHMARKS@
Regularized libnetcdf.settings.in to make parsing settings file easier
2014-08-25 06:29:44 +08:00
NetCDF-2 API: @HAS_NC2@
HDF4 Support: @HAS_HDF4@
updated libnetcdf.settings.in and changing makefile.am to adjust to possibility of netcdf-4 build without hdf5
2018-11-29 06:50:38 +08:00
HDF5 Support: @HAS_HDF5@
Cleaning up netcdf status message.
2015-11-18 07:31:29 +08:00
NetCDF-4 API: @HAS_NC4@
Added S3 status to libnetcdf.settings, turned byterange on by default.
2023-01-10 04:11:16 +08:00
CDF5 Support: @HAS_CDF5@
merge-squash
2015-08-16 06:26:35 +08:00
NC-4 Parallel Support: @HAS_PARALLEL4@
Use PnetCDF instead of parallel-netcdf to avoid confusion with parallel netcdf4. Also, update PnetCDF web page.
2018-09-18 00:47:40 +08:00
PnetCDF Support: @HAS_PNETCDF@
Added S3 status to libnetcdf.settings, turned byterange on by default.
2023-01-10 04:11:16 +08:00
Upgrade the nczarr code to match Zarr V2 Re: https://github.com/zarr-developers/zarr-python/pull/716 The Zarr version 2 spec has been extended to include the ability to choose the dimension separator in chunk name keys. The legal separators has been extended from {'.'} to {'.' '/'}. So now it is possible to use a key like "0/1/2/0" for chunk names. This PR implements this for NCZarr. The V2 spec now says that this separator can be set on a per-variable basis. For now, I have chosen to allow this be set only globally by adding a key named "ZARR.DIMENSION_SEPARATOR=<char>" in the .daprc/.dodsrc/ncrc file. Currently, the only legal separator characters are '.' (the default) and '/'. On writing, this key will only be written if its value is different than the default. This change caused problems because supporting a separator of '/' is difficult to parse when keys/paths use '/' as the path separator. A test case was added for this. Additionally, make nczarr be enabled default by default. This required some additional changes so that if zip and/or AWS S3 sdk are unavailable, then they are disabled for NCZarr. In addition the following unrelated changes were made. 1. Tested that pure-zarr mode could read an nczarr formatted store. 1. The .rc file handling now merges all known .rc files (.ncrc,.daprc, and .dodsrc) in that order and using those in HOME first, then in current directory. For duplicate entries, the later ones override the earlier ones. This change is to remove some of the conflicts inherent in the current .rc file load process. A set of test cases was also added. 1. Re-order tests in configure.ac and CMakeLists.txt so that if libcurl is not found then the other options that depend upon it properly are disabled. 1. I decided that xarray support should be enabled by default for pure zarr. In order to allow disabling, I added a new mode flag "noxarray". 1. Certain test in nczarr_test depend on use of .dodsrc. In order for these to work when testing in parallel, some inter-test dependencies needed to be added. 1. Improved authorization testing to use changes in thredds.ucar.edu
2021-04-25 09:48:15 +08:00
DAP2 Support: @HAS_DAP2@
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
DAP4 Support: @HAS_DAP4@
Master merge, conflict resolution, cleanup
2019-02-25 07:54:13 +08:00
Byte-Range Support: @HAS_BYTERANGE@
Turn nczarr zip support off by default in cmake, add a status line indicating whether nczarr-zip-support is available, in libnetcdf.settings.
2023-02-28 04:34:26 +08:00
Fix byterange handling of some URLS re: Issue The byterange handling of the following URLS fails. ### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes" It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD). I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time. ### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3" When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case. ## Misc. Related Changes 1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support 2. Added the above URLS as additional test cases ## Misc. Un-Related Changes 1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD 2. Fix some minor warnings ## Open Problems * Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-03 10:51:02 +08:00
S3 Support: @HAS_S3@
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
S3 SDK: @WHICH_S3_SDK@
Fix byterange handling of some URLS re: Issue The byterange handling of the following URLS fails. ### Problem 1: "https://crudata.uea.ac.uk/cru/data/temperature/HadCRUT.4.6.0.0.median.nc#mode=bytes" It turns out that byterange in hdf5 has two possible targets: S3 and not-S3 (e.g. a thredds server or the crudata URL above). Each uses a different HDF5 Virtual File Driver (VFD). I incorrectly set up the byterange code in libhdf5 so that it would choose one or the other of the two VFD's for any netcdf-c library build. The fix is to allow it to choose either one at run-time. ### Problem 2: "https://noaa-goes16.s3.amazonaws.com/ABI-L1b-RadF/2022/001/18/OR_ABI-L1b-RadF-M6C01_G16_s20220011800205_e20220011809513_c20220011809562.nc#mode=bytes,s3" When given what appears to be an S3-related URL, the netcdf-c library code converts it into a canonical, so-called "path" format. In casing out the possible input URL formats, I missed the case where the host contains the bucket ("noaa-goes16"), but not the region. So the fix was to check for this case. ## Misc. Related Changes 1. Since S3 is used in more than just NCZarr, I changed the automake/cmake options to replace "--enable-nczarr-s3" with "--enable-s3", but keeping the former option as a synonym for the latter. This also entailed cleaning up libnetcdf.settings WRT S3 support 2. Added the above URLS as additional test cases ## Misc. Un-Related Changes 1. CURLOPT_PUT is deprecated in favor to CURLOPT_UPLOAD 2. Fix some minor warnings ## Open Problems * Under Ubuntu, either libcrypto or aws-sdk-cpp has a memory leak.
2023-03-03 10:51:02 +08:00
Added S3 status to libnetcdf.settings, turned byterange on by default.
2023-01-10 04:11:16 +08:00
NCZarr Support: @HAS_NCZARR@
Regularized libnetcdf.settings.in to make parsing settings file easier
2014-08-25 06:29:44 +08:00
Diskless Support: @HAS_DISKLESS@
MMap Support: @HAS_MMAP@
JNA Support: @HAS_JNA@
Fix the NC_INMEMORY code to work in all cases with HDF5 1.10. re: github issue https://github.com/Unidata/netcdf-c/issues/1111 One of the less common use cases for the in-memory feature is apparently failing with HDF5-1.10.x. The fix is complicated and requires significant changes to libhdf5/nc4memcb.c. The current setup is detailed in the file docs/inmeminternal.dox. Additionally, it was discovered that the program nc_test/tst_inmemory.c, which is invoked by nc_test/run_inmemory.sh, actually was failing because of the above problem. But the failure is not detected since the script does not return non-zero value. Other Changes: 1. Fix nc_test_tst_inmemory to return errors correctly. 2. Make ncdap_tests/findtestserver.c and dap4_tests/findtestserver4.c be generated from ncdap_test/findtestserver.c.in. 3. Make LOG() print output to stderr instead of stdout to avoid contaminating e.g. ncdump output. 4. Modify the handling of NC_INMEMORY and NC_DISKLESS flags to properly handle that NC_DISKLESS => NC_INMEMORY. This affects a number of code pieces, especially memio.c.
2018-09-05 01:27:47 +08:00
ERANGE Fill Support: @HAS_ERANGE_FILL@
Cleaned up szlib status message in libnetcdf.settings.
2020-01-25 07:36:29 +08:00
Relaxed Boundary Check: @RELAX_COORD_BOUND@
Added S3 status to libnetcdf.settings, turned byterange on by default.
2023-01-10 04:11:16 +08:00
getting ready for next try at quantization code
2021-08-24 14:45:38 +08:00
Quantization: @HAS_QUANTIZE@
added info about logging to build summary, fixed info about byterange in build summary
2021-08-24 14:19:41 +08:00
Logging: @HAS_LOGGING@
Enhance/Fix filter support re: Discussion https://github.com/Unidata/netcdf-c/discussions/2214 The primary change is to support so-called "standard filters". A standard filter is one that is defined by the following netcdf-c API: ```` int nc_def_var_XXX(int ncid, int varid, size_t nparams, unsigned* params); int nc_inq_var_XXXX(int ncid, int varid, int* usefilterp, unsigned* params); ```` So for example, zstandard would be a standard filter by defining the functions *nc_def_var_zstandard* and *nc_inq_var_zstandard*. In order to define these functions, we need a new dispatch function: ```` int nc_inq_filter_avail(int ncid, unsigned filterid); ```` This function, combined with the existing filter API can be used to implement arbitrary standard filters using a simple code pattern. Note that I would have preferred that this function return a list of all available filters, but HDF5 does not support that functionality. So this PR implements the dispatch function and implements the following standard functions: + bzip2 + zstandard + blosc Specific test cases are also provided for HDF5 and NCZarr. Over time, other specific standard filters will be defined. ## Primary Changes * Add nc_inq_filter_avail() to netcdf-c API. * Add standard filter implementations to test use of *nc_inq_filter_avail*. * Bump the dispatch table version number and add to all the relevant dispatch tables (libsrc, libsrcp, etc). * Create a program to invoke nc_inq_filter_avail so that it is accessible to shell scripts. * Cleanup szip support to properly support szip when HDF5 is disabled. This involves detecting libsz separately from testing if HDF5 supports szip. * Integrate shuffle and fletcher32 into the existing filter API. This means that, for example, nc_def_var_fletcher32 is now a wrapper around nc_def_var_filter. * Extend the Codec defaulting to allow multiple default shared libraries. ## Misc. Changes * Modify configure.ac/CMakeLists.txt to look for the relevant libraries implementing standard filters. * Modify libnetcdf.settings to list available standard filters (including deflate and szip). * Add CMake test modules to locate libbz2 and libzstd. * Cleanup the HDF5 memory manager function use in the plugins. * remove unused file include//ncfilter.h * remove tests for the HDF5 memory operations e.g. H5allocate_memory. * Add flag to ncdump to force use of _Filter instead of _Deflate or _Shuffle or _Fletcher32. Used for testing.
2022-03-15 02:39:37 +08:00
SZIP Write Support: @HAS_SZLIB_WRITE@
Standard Filters: @STD_FILTERS@
added BENCHMARKS to the summary
2022-04-26 20:18:52 +08:00
ZSTD Support: @HAS_ZSTD@
Added S3 status to libnetcdf.settings, turned byterange on by default.
2023-01-10 04:11:16 +08:00
Parallel Filters: @HAS_PAR_FILTERS@
Reference in New Issue Copy Permalink