Commit Graph

7815 Commits

Author SHA1 Message Date
Dennis Heimbigner
91168e33a0 Fix JSON quoted string processing in libnczarr
re: github issue https://github.com/Unidata/netcdf-c/issues/1982

The problem was that the libnczarr/zsjon.c handling of strings with
embedded double quotes was wrong; a one line fix.
Also added a test case.

Misc. other changes:

1. I Discovered, en passant, that the handling of 64 bit constants
had an error that was fixed.
2. cleanup of the constant conversion code to recurse on arrays of values.
2021-05-06 16:39:44 -06:00
Ward Fisher
e6bdee5930
Merge pull request #1990 from DennisHeimbigner/nczarr_enhance1.dmh
Upgrade the nczarr code to match Zarr V2
2021-04-26 16:13:41 -06:00
Dennis Heimbigner
5873c627db remove push signal 2021-04-25 22:03:03 -06:00
Dennis Heimbigner
1243c3d866 Allow .rc tests to work in parallel by isolation 2021-04-25 22:02:29 -06:00
Dennis Heimbigner
dec9e74107 logdepend 2021-04-25 13:41:50 -06:00
Dennis Heimbigner
1f51ec56aa LGTM cleanup 2021-04-24 21:44:57 -06:00
Dennis Heimbigner
ed70a2e463 try 2: fix test properties 2021-04-24 20:49:03 -06:00
Dennis Heimbigner
05b4046173 try 1: fix test properties 2021-04-24 20:41:59 -06:00
Dennis Heimbigner
f3d118bfa1 reset action signals 2021-04-24 19:54:41 -06:00
Dennis Heimbigner
6ef02261f0 Update RELEASE NOTES 2021-04-24 19:53:18 -06:00
Dennis Heimbigner
74b40fd788 Upgrade the nczarr code to match Zarr V2
Re: https://github.com/zarr-developers/zarr-python/pull/716

The Zarr version 2 spec has been extended to include the ability
to choose the dimension separator in chunk name keys. The legal
separators has been extended from {'.'} to {'.' '/'}.  So now it
is possible to use a key like "0/1/2/0" for chunk names.

This PR implements this for NCZarr. The V2 spec now says that
this separator can be set on a per-variable basis. For now, I
have chosen to allow this be set only globally by adding a key
named "ZARR.DIMENSION_SEPARATOR=<char>" in the
.daprc/.dodsrc/ncrc file. Currently, the only legal separator
characters are '.' (the default) and '/'. On writing, this key
will only be written if its value is different than the default.
This change caused problems because supporting a separator of '/'
is difficult to parse when keys/paths use '/' as the path separator.
A test case was added for this.

Additionally, make nczarr be enabled default by default. This required
some additional changes so that if zip and/or AWS S3 sdk are unavailable,
then they are disabled for NCZarr.

In addition the following unrelated changes were made.

1. Tested that pure-zarr mode could read an nczarr formatted store.
1. The .rc file handling now merges all known .rc files (.ncrc,.daprc, and .dodsrc) in that order and using those in HOME first, then in current directory. For duplicate entries, the later ones override the earlier ones. This change is to remove some of the conflicts inherent in the current .rc file load process. A set of test cases was also added.
1. Re-order tests in configure.ac and CMakeLists.txt so that if libcurl
   is not found then the other options that depend upon it properly
   are disabled.
1. I decided that xarray support should be enabled by default for pure
   zarr. In order to allow disabling, I added a new mode flag "noxarray".
1. Certain test in nczarr_test depend on use of .dodsrc. In order for these
   to work when testing in parallel, some inter-test dependencies needed to
   be added.
1. Improved authorization testing to use changes in thredds.ucar.edu
2021-04-24 19:48:15 -06:00
Ward Fisher
706e33b056
Merge pull request #1989 from DennisHeimbigner/shellfix.dmh
Fix shell handling of escapes
2021-04-22 16:14:23 -06:00
Dennis Heimbigner
2bdc00cf4d Update RELEASENOTES 2021-04-21 15:03:50 -06:00
Dennis Heimbigner
efd1be5d62 Fix shell handling of escapes
re: https://github.com/Unidata/netcdf-c/issues/1988

There was an issue with certain shell programs (bash notably).
For certain platforms and when given a url that had an escaped
'#' character (e.g. \\#) bash would not remove the backslash. So I
had to add a hack for this. Unfortunately I overdid it and it
removed all '' characters. This is ok for non-windows platforms,
but obviously fails for windows.

The fix is this.

1. In a utility program (ncgen, ncdump, nccopy, etc) there is probably a call (or calls) to NC_backslashUnescape(xxx) where xxx is a path argument from the command line.
2. Replace each such call with NC_shellUnescape(xxx).

The NC_shellUnescape function was added and searched only for occurrences of "\#" and replaces them with "#".
2021-04-21 14:59:15 -06:00
Ward Fisher
67c640f1b8
Merge pull request #1984 from DennisHeimbigner/kwaliasfix.dmh
Fix the handling of certain alias types in CDL files.
2021-04-14 14:20:56 -06:00
Dennis Heimbigner
30ee9914e2 Update release notes 2021-04-14 11:23:33 -06:00
Dennis Heimbigner
ac421620b3 Fix the handling of certain alias types on CDL files.
re: https://github.com/Unidata/netcdf-c/issues/1977

PR https://github.com/Unidata/netcdf-c/pull/1753, changed ncgen
to allows certain type names to be used as identifiers in
selected situations.

An unwanted side effect was that existing type aliases no longer
were accepted by ncgen. Specifically, using the "long" type
caused an error.

I was able to figure out a better solution to the original
problem (https://github.com/Unidata/netcdf-c/issues/1750)
that also fixes this problem as well.

This PR fixes that problem in ncgen/ncgen.l,
and adds tests to ncdump/test_keywords.sh
2021-04-13 16:56:43 -06:00
Ward Fisher
1e99bb4a7c
Merge pull request #1979 from opoplawski/builddir
Fix out of tree builds with configure
2021-04-12 16:07:54 -06:00
Ward Fisher
bcadaf3839
Merge pull request #1980 from opoplawski/test
Fix tst_h_atts3 for hdf5 1.12
2021-04-12 16:07:24 -06:00
Orion Poplawski
9fc8ae62a8 Fix tst_h_atts3 for hdf5 1.12 2021-04-10 19:51:58 -06:00
Orion Poplawski
031b93a0f8 Fix out of tree builds with configure - add $(top_builddir)/include to include directory search 2021-04-10 17:57:55 -06:00
Ward Fisher
e38078d14f
Merge pull request #1952 from DennisHeimbigner/nczarr_xarray.dmh
Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute
2021-04-08 13:01:29 -06:00
Dennis Heimbigner
02bca8fae7 force tests to rerun 2021-04-02 19:05:58 -06:00
Dennis Heimbigner
0454d8e235 Addendum: This PR has been extended to include
interoperability fixed. We were given a Zarr format dataset
stored as a directory+file tree. This dataset uses the XArray
conventions and was generated by some non-Unidata Zarr implementation.
In attempting to process it with NCZarr, several interoperability
problems were discovered and fixed. This gives us more confidence
that NCZarr -- using pure zarr -- can interoperate with other
Zarr implementations.

Specific changes:
* Add test nczarr_test/run_interop.sh
* Support attributes with single value not enclosed in JSON array tags.
* Add mode inferencing and use it in nczarr_test/run_purezarr.sh
* Reduce size of tst_err_enddef.nc because it is more than 3 GB.
2021-04-02 18:39:50 -06:00
Dennis Heimbigner
727dd6861e Merge branch 'master' into nczarr_xarray.dmh 2021-04-02 18:29:39 -06:00
Ward Fisher
9bf69804a1
Merge pull request #1976 from Unidata/gh1969.wif
PR1970 + Release Notes
2021-04-02 11:22:34 -06:00
Ward Fisher
37f69e1f74 Updated release notes. 2021-04-02 10:54:24 -06:00
Ward Fisher
acc8e769ee Merge branch 'bzip2108.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1969.wif 2021-04-02 10:53:31 -06:00
Dennis Heimbigner
e038553abe Update RELEASE_NOTES.md 2021-04-01 14:12:49 -06:00
Dennis Heimbigner
e7c4e7ead1 add zjson fix 2021-04-01 13:56:04 -06:00
Ward Fisher
f01e4f1027
Merge pull request #1975 from Unidata/gh1962.wif
Refactored the HDF5 version detection logic in CMakeLists.txt
2021-04-01 13:04:41 -06:00
Ward Fisher
ad51b0a79a Updated release notes, made if-block syntax more verbose. 2021-04-01 11:46:44 -06:00
Ward Fisher
35bf5f4b79 Corrected a typo. 2021-04-01 11:43:15 -06:00
Ward Fisher
17b4c2bebb Refactored the HDF5 version detection logic in CMakeLists.txt in support of https://github.com/Unidata/netcdf-c/issues/1962 2021-04-01 11:24:11 -06:00
Ward Fisher
5b40de1267
Merge pull request #1974 from gsjaardema/patch-49
Fix use of ininitialized CMake variable
2021-04-01 09:43:00 -06:00
Greg Sjaardema
6118776d60
Fix use of ininitialized CMake variable
The `ISCMAKE` was defined after its first use in a `configure_file` call.  Moved to before its first use.
2021-04-01 08:13:13 -06:00
Ward Fisher
790a1a2a06
Merge pull request #1961 from Unidata/4.8.0-wellspring.wif
Selectively merge relevant changes and updates back upstream from the 4.8.0 prep branch.
2021-03-30 16:40:55 -06:00
Dennis Heimbigner
395f47a9b3 Update the bzip library in plugins to version 1.0.8
re: Issue https://github.com/Unidata/netcdf-c/issues/1969
2021-03-30 15:27:24 -06:00
Ward Fisher
670fd2482f Corrected typo preventing success when running make dist or make distcheck 2021-03-30 14:08:00 -06:00
Ward Fisher
cf5fc9b0f7 Updated doxyfile to include libnczarr/zarr.c, also removed references to files that have been removed. 2021-03-30 13:53:16 -06:00
Ward Fisher
90c662c58f Corrected release date in preparation for today's release. 2021-03-30 10:08:05 -06:00
Ward Fisher
95719addd2 Added 1.0.1 to test matrix in support of https://github.com/Unidata/netcdf-c/pull/1931#issuecomment-804312933 2021-03-26 14:50:22 -06:00
Ward Fisher
85f3abb515 Updated expected release date. 2021-03-26 14:47:53 -06:00
Ward Fisher
ac4ade4f18 Merge branch '4.8.0-wellspring-prs.wif' into 4.8.0-wellspring.wif 2021-03-22 16:59:15 -06:00
Ward Fisher
4a7f5aeef2 Added check for H5Literate symbol (hdf5 1.8.x, 1.10.x) or macro (1.12.x)) in support of https://github.com/Unidata/netcdf-c/issues/1965 2021-03-22 13:40:11 -06:00
Ward Fisher
094e2a4ea3 Amend check for H5Literate to work with 1.10.x 2021-03-22 13:24:03 -06:00
Ward Fisher
3e0b36a355 Added check for H5Literate in support of https://github.com/Unidata/netcdf-c/issues/1965 2021-03-22 13:05:23 -06:00
Ward Fisher
fc4bfc6196 Merge branch 'dispatchversion.dmh' of https://github.com/DennisHeimbigner/netcdf-c into 4.8.0-wellspring-prs.wif 2021-03-22 12:41:03 -06:00
Ward Fisher
f75c1c4d7b
Merge branch 'master' into dispatchversion.dmh 2021-03-22 12:40:09 -06:00
Ward Fisher
dab186b688 Merge branch '4.8.0-wellspring.wif' of github.com:Unidata/netcdf-c into 4.8.0-wellspring.wif 2021-03-22 12:38:24 -06:00