Commit Graph

7384 Commits

Author SHA1 Message Date
Ward Fisher
7d2a646f25 Merge branch 'ejh_fix_redef' of https://github.com/NOAA-GSD/netcdf-c into NOAA-GSD-ejh_fix_redef 2020-07-09 13:55:37 -06:00
Ward Fisher
b59429d5bc
Merge pull request #1782 from NOAA-GSD/ejh_fix_warning
fixed warning in hdf5filter.c
2020-07-09 13:34:12 -06:00
Ward Fisher
ed241d740b
Merge pull request #1784 from NOAA-GSD/ejh_h5_par_zip_test
Add a HDF5 only test of writing and reading a file with parallel I/O and the zlib filter
2020-07-09 13:32:39 -06:00
Edward Hartnett
3f7197cc07 updated RELEASE_NOTES 2020-07-08 13:28:15 -06:00
Edward Hartnett
b1c0fa07af added tst_h_par_compress to CMake build 2020-07-08 13:26:50 -06:00
Edward Hartnett
73e21b4fd6 took out timing code 2020-07-08 13:21:33 -06:00
Edward Hartnett
c19aa14eaf turned off szip testing while HDF5 issue is resolved 2020-07-08 13:14:58 -06:00
Edward Hartnett
4936edd840 adding other settings of dataset property list to match what is in nc4hdf5.c 2020-07-08 12:56:34 -06:00
Edward Hartnett
d1a13b28cc adding property list for dataset creation to tst_h_par_compress.c 2020-07-08 12:39:35 -06:00
Edward Hartnett
c0b6792515 changed tst_h_par_compress to use H5Dcreate2() just like code in nc4hdf5.c does 2020-07-08 12:36:02 -06:00
Edward Hartnett
1d79a47c6d fixed warning in tst_parallel5.c 2020-07-08 12:24:48 -06:00
Edward Hartnett
90777d6465 added new test tst_h_par_compress.c 2020-07-08 12:17:37 -06:00
Edward Hartnett
3e60a863de fixed warning in hdf5filter.c 2020-07-08 11:24:54 -06:00
Edward Hartnett
5ff171a2fe updated release notes 2020-07-08 11:11:28 -06:00
Edward Hartnett
832fbf19c8 now dont return error on second redef call for netcdf/HDF5 files 2020-07-08 11:10:15 -06:00
Edward Hartnett
5e41f33685 starting to test repeated redefs 2020-07-08 11:04:16 -06:00
Ward Fisher
933a8979c1
Merge pull request #1775 from NOAA-GSD/ejh_test_szip_unlim
Allow szip to be used with unlimited dim
2020-07-06 16:40:53 -06:00
Edward Hartnett
4cc06b683b added test cdl files to cleanfiles 2020-07-03 12:56:22 -06:00
Edward Hartnett
64264190ba merged in some ncdump tests 2020-07-02 14:30:35 -06:00
Edward Hartnett
4103faf0d5 added some ncdump tests for szip when it is present 2020-07-02 13:59:37 -06:00
Edward Hartnett
8b6a477151 updated RELEASE_NOTES 2020-07-02 10:58:26 -06:00
Edward Hartnett
6c112efb8e fixed problem setting szip on var with unlimited dim and added test 2020-07-02 10:55:34 -06:00
Edward Hartnett
88dd5c5c65 starting to add test for unlimited dim 2020-07-02 10:34:15 -06:00
Ward Fisher
dd0609b153
Merge pull request #1769 from DennisHeimbigner/nczarr-experimental.dmh
First experimental release of nczarr
2020-07-01 13:03:14 -06:00
Dennis Heimbigner
5c6fb13172 Merge branch 'nczarr-experimental.dmh' of https://github.com/DennisHeimbigner/netcdf-c into nczarr-experimental.dmh 2020-06-29 09:39:56 -06:00
Dennis Heimbigner
eeb2b8d843 Fixed documentation 2020-06-29 09:39:38 -06:00
Dennis Heimbigner
b47bee443f turn on some tests 2020-06-29 09:25:28 -06:00
Dennis Heimbigner
d21c0170d7 Semi-disable S3 tests.
They rely on access to the unidata S3 bucket on stratus,
so only Unidata NetCDF personnel can execute them,
To enable them, one needs to do this at command level:
export NETCDF_S3_TESTS=1
2020-06-29 09:14:47 -06:00
Dennis Heimbigner
9e1ff1555e Merge branch 'nczarr-experimental.dmh' of https://github.com/DennisHeimbigner/netcdf-c into nczarr-experimental.dmh 2020-06-28 19:53:56 -06:00
Dennis Heimbigner
93ecc49129 Fix LGTM errors and S3 mis-configuration 2020-06-28 19:53:40 -06:00
Dennis Heimbigner
c7cf0d3807 Fix LGTM errors 2020-06-28 19:07:08 -06:00
Dennis Heimbigner
999c291020 Fix LGTM detected errors 2020-06-28 18:59:53 -06:00
Dennis Heimbigner
59e04ae071 This PR adds EXPERIMENTAL support for accessing data in the
cloud using a variant of the Zarr protocol and storage
format. This enhancement is generically referred to as "NCZarr".

The data model supported by NCZarr is netcdf-4 minus the user-defined
types and the String type. In this sense it is similar to the CDF-5
data model.

More detailed information about enabling and using NCZarr is
described in the document NUG/nczarr.md and in a
[Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in).

WARNING: this code has had limited testing, so do use this version
for production work. Also, performance improvements are ongoing.
Note especially the following platform matrix of successful tests:

Platform | Build System | S3 support
------------------------------------
Linux+gcc      | Automake     | yes
Linux+gcc      | CMake        | yes
Visual Studio  | CMake        | no

Additionally, and as a consequence of the addition of NCZarr,
major changes have been made to the Filter API. NOTE: NCZarr
does not yet support filters, but these changes are enablers for
that support in the future.  Note that it is possible
(probable?) that there will be some accidental reversions if the
changes here did not correctly mimic the existing filter testing.

In any case, previously filter ids and parameters were of type
unsigned int. In order to support the more general zarr filter
model, this was all converted to char*.  The old HDF5-specific,
unsigned int operations are still supported but they are
wrappers around the new, char* based nc_filterx_XXX functions.
This entailed at least the following changes:
1. Added the files libdispatch/dfilterx.c and include/ncfilter.h
2. Some filterx utilities have been moved to libdispatch/daux.c
3. A new entry, "filter_actions" was added to the NCDispatch table
   and the version bumped.
4. An overly complex set of structs was created to support funnelling
   all of the filterx operations thru a single dispatch
   "filter_actions" entry.
5. Move common code to from libhdf5 to libsrc4 so that it is accessible
   to nczarr.

Changes directly related to Zarr:
1. Modified CMakeList.txt and configure.ac to support both C and C++
   -- this is in support of S3 support via the awd-sdk libraries.
2. Define a size64_t type to support nczarr.
3. More reworking of libdispatch/dinfermodel.c to
   support zarr and to regularize the structure of the fragments
   section of a URL.

Changes not directly related to Zarr:
1. Make client-side filter registration be conditional, with default off.
2. Hack include/nc4internal.h to make some flags added by Ed be unique:
   e.g. NC_CREAT, NC_INDEF, etc.
3. cleanup include/nchttp.h and libdispatch/dhttp.c.
4. Misc. changes to support compiling under Visual Studio including:
   * Better testing under windows for dirent.h and opendir and closedir.
5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags
   and to centralize error reporting.
6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them.
7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible.

Changes Left TO-DO:
1. fix provenance code, it is too HDF5 specific.
2020-06-28 18:02:47 -06:00
Ward Fisher
ba24e3c08e
Merge pull request #1765 from opoplawski/hdf5_zlib
Fix wrong header include in testing HDF5 for zlib
2020-06-25 16:31:43 -06:00
Ward Fisher
7c4745508a
Merge pull request #1768 from NOAA-GSD/ejh_fix_plugins_makefile
fixed the setting of AM_LDFLAGS
2020-06-25 16:04:34 -06:00
Ward Fisher
e07e24b7a0
Merge pull request #1764 from DennisHeimbigner/dfalt.dmh
Fix nccopy chunking to use default chunking
2020-06-25 15:39:36 -06:00
Dennis Heimbigner
04dbf30a5e Update release notes 2020-06-25 11:24:57 -06:00
Edward Hartnett
1ce19f15b3 fixed the setting of AM_LDFLAGS 2020-06-25 10:22:33 -06:00
Orion Poplawski
b3e42ac54d Fix wrong header include in testing HDF5 for zlib 2020-06-24 20:30:55 -06:00
Dennis Heimbigner
c195e24f2a Fix nccopy chunking to use default chunking
re: https://github.com/Unidata/netcdf-c/issues/1763

The nccopy program incorrectly set default chunking parameters
to use full dimension lengths. Instead, it should use the
values computed by the default chunking values as defined
in nc4_default_chunksizes2 in the netcdf library.

Solution: Revert to the old behavior.
2020-06-24 16:34:32 -06:00
Ward Fisher
ed421f2233
Merge pull request #1754 from LProx2020/README-LINK-PATCH
Updated README.md Links
2020-06-11 13:38:35 -06:00
Ward Fisher
1bf99871bd
Merge pull request #1755 from gsjaardema/patch-43
Avoid potential integer overrun
2020-06-11 12:38:44 -06:00
Greg Sjaardema
edf0ca6c98
Avoid potential integer overrun
It is possible for the values stored to `file_value_size` to overrun the storage capacity of a 32-bit integer.  The value does need to store negative values potentially, so can be `size_t` or `hsize_t`, so use the `hssize_t` which is a signed 64-bit value.  Could also use `ssize_t`, but that is not used in this routine...
2020-06-10 15:42:22 -06:00
LProx2020
e4e0e93402
Updated README.md Links
Reference: Issue #1711
-The proper link was provided for the Language Independent User's Guide.
-The proper link was provided for the NetCDF-Java/Common Data Model Library
2020-06-04 14:16:30 -04:00
Ward Fisher
b36c9c5b3f
Merge pull request #1747 from NOAA-GSD/ejh_fix
fixed LDFLAFS in plugins Makefile.am, also fixed warning in tst_h_vars.c, restore NC_NOTNC3_get|put_varm() functions
2020-06-03 13:47:39 -06:00
Ward Fisher
386e269528
Merge pull request #1752 from sayerhs/master
Fix parallel NetCDF checks in configure.ac for spectrum MPI
2020-06-03 13:25:01 -06:00
Shreyas Ananthan
e11bc3509d Fix parallel NetCDF checks in configure.ac for spectrum MPI
Closes #1751
2020-06-03 13:13:35 -04:00
Edward Hartnett
87226c4879 readded NOTNC3 varm functions to dispatch 2020-06-03 05:55:30 -06:00
Edward Hartnett
9f6fc00f98 changed back to totally dead 2020-06-02 13:59:08 -06:00
Edward Hartnett
97c2335adc changed to only MOSTLY dead 2020-06-02 07:51:59 -06:00