mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2025-01-06 15:34:44 +08:00
Code Issues Packages Projects Releases Wiki Activity
7939ec559f
netcdf-c/nczarr_test/run_purezarr.sh

48 lines
1.3 KiB
Bash
Raw Normal View History

Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute The XArray implementation that uses Zarr for storage provides a mechanism to simulate named dimensions. It does this by adding a per-variable attribute called _ARRAY_DIMENSIONS. This attribute contains a list of names to be matched against the shape values of the variable. In effect a named dimension is created with the name _ARRAY_DIMENSIONS(i) and length shape(i) for all i in range 0..rank(variable). Both read and write support is provided. This XArray support is only invoked if the mode value of "xarray" is defined. So for example, as in this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,xarray,s3 ```` Note that the "xarray" mode flag also implies mode flag "zarr", so the above is equivalent to this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,zarr,xarray,s3 ```` The primary change to implement this was to unify the handling of dimension references in libnczarr/zsync. A test for this and other pure-zarr features was added as nczarr_test/run_purezarr.sh Other changes: * Make sure distcheck leaves no files around. * Change the special attribute flag DIMSCALEFLAG to HIDDENATTRFLAG to support the xarray attribute. * Annotate the zmap implementations with feature flags such as WRITEONCE (for zip files).
2021-02-25 04:46:11 +08:00
#!/bin/sh
if test "x$srcdir" = x ; then srcdir=`pwd`; fi
. ../test_common.sh
. "$srcdir/test_nczarr.sh"
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
set -e
s3isolate "testdir_purezarr"
THISDIR=`pwd`
cd $ISOPATH
Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute The XArray implementation that uses Zarr for storage provides a mechanism to simulate named dimensions. It does this by adding a per-variable attribute called _ARRAY_DIMENSIONS. This attribute contains a list of names to be matched against the shape values of the variable. In effect a named dimension is created with the name _ARRAY_DIMENSIONS(i) and length shape(i) for all i in range 0..rank(variable). Both read and write support is provided. This XArray support is only invoked if the mode value of "xarray" is defined. So for example, as in this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,xarray,s3 ```` Note that the "xarray" mode flag also implies mode flag "zarr", so the above is equivalent to this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,zarr,xarray,s3 ```` The primary change to implement this was to unify the handling of dimension references in libnczarr/zsync. A test for this and other pure-zarr features was added as nczarr_test/run_purezarr.sh Other changes: * Make sure distcheck leaves no files around. * Change the special attribute flag DIMSCALEFLAG to HIDDENATTRFLAG to support the xarray attribute. * Annotate the zmap implementations with feature flags such as WRITEONCE (for zip files).
2021-02-25 04:46:11 +08:00
# This shell script tests support for:
Upgrade the nczarr code to match Zarr V2 Re: https://github.com/zarr-developers/zarr-python/pull/716 The Zarr version 2 spec has been extended to include the ability to choose the dimension separator in chunk name keys. The legal separators has been extended from {'.'} to {'.' '/'}. So now it is possible to use a key like "0/1/2/0" for chunk names. This PR implements this for NCZarr. The V2 spec now says that this separator can be set on a per-variable basis. For now, I have chosen to allow this be set only globally by adding a key named "ZARR.DIMENSION_SEPARATOR=<char>" in the .daprc/.dodsrc/ncrc file. Currently, the only legal separator characters are '.' (the default) and '/'. On writing, this key will only be written if its value is different than the default. This change caused problems because supporting a separator of '/' is difficult to parse when keys/paths use '/' as the path separator. A test case was added for this. Additionally, make nczarr be enabled default by default. This required some additional changes so that if zip and/or AWS S3 sdk are unavailable, then they are disabled for NCZarr. In addition the following unrelated changes were made. 1. Tested that pure-zarr mode could read an nczarr formatted store. 1. The .rc file handling now merges all known .rc files (.ncrc,.daprc, and .dodsrc) in that order and using those in HOME first, then in current directory. For duplicate entries, the later ones override the earlier ones. This change is to remove some of the conflicts inherent in the current .rc file load process. A set of test cases was also added. 1. Re-order tests in configure.ac and CMakeLists.txt so that if libcurl is not found then the other options that depend upon it properly are disabled. 1. I decided that xarray support should be enabled by default for pure zarr. In order to allow disabling, I added a new mode flag "noxarray". 1. Certain test in nczarr_test depend on use of .dodsrc. In order for these to work when testing in parallel, some inter-test dependencies needed to be added. 1. Improved authorization testing to use changes in thredds.ucar.edu
2021-04-25 09:48:15 +08:00
# 1. pure zarr (noxarray) read/write
Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute The XArray implementation that uses Zarr for storage provides a mechanism to simulate named dimensions. It does this by adding a per-variable attribute called _ARRAY_DIMENSIONS. This attribute contains a list of names to be matched against the shape values of the variable. In effect a named dimension is created with the name _ARRAY_DIMENSIONS(i) and length shape(i) for all i in range 0..rank(variable). Both read and write support is provided. This XArray support is only invoked if the mode value of "xarray" is defined. So for example, as in this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,xarray,s3 ```` Note that the "xarray" mode flag also implies mode flag "zarr", so the above is equivalent to this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,zarr,xarray,s3 ```` The primary change to implement this was to unify the handling of dimension references in libnczarr/zsync. A test for this and other pure-zarr features was added as nczarr_test/run_purezarr.sh Other changes: * Make sure distcheck leaves no files around. * Change the special attribute flag DIMSCALEFLAG to HIDDENATTRFLAG to support the xarray attribute. * Annotate the zmap implementations with feature flags such as WRITEONCE (for zip files).
2021-02-25 04:46:11 +08:00
# 2. xarray read/write
testcase() {
zext=$1
Upgrade the nczarr code to match Zarr V2 Re: https://github.com/zarr-developers/zarr-python/pull/716 The Zarr version 2 spec has been extended to include the ability to choose the dimension separator in chunk name keys. The legal separators has been extended from {'.'} to {'.' '/'}. So now it is possible to use a key like "0/1/2/0" for chunk names. This PR implements this for NCZarr. The V2 spec now says that this separator can be set on a per-variable basis. For now, I have chosen to allow this be set only globally by adding a key named "ZARR.DIMENSION_SEPARATOR=<char>" in the .daprc/.dodsrc/ncrc file. Currently, the only legal separator characters are '.' (the default) and '/'. On writing, this key will only be written if its value is different than the default. This change caused problems because supporting a separator of '/' is difficult to parse when keys/paths use '/' as the path separator. A test case was added for this. Additionally, make nczarr be enabled default by default. This required some additional changes so that if zip and/or AWS S3 sdk are unavailable, then they are disabled for NCZarr. In addition the following unrelated changes were made. 1. Tested that pure-zarr mode could read an nczarr formatted store. 1. The .rc file handling now merges all known .rc files (.ncrc,.daprc, and .dodsrc) in that order and using those in HOME first, then in current directory. For duplicate entries, the later ones override the earlier ones. This change is to remove some of the conflicts inherent in the current .rc file load process. A set of test cases was also added. 1. Re-order tests in configure.ac and CMakeLists.txt so that if libcurl is not found then the other options that depend upon it properly are disabled. 1. I decided that xarray support should be enabled by default for pure zarr. In order to allow disabling, I added a new mode flag "noxarray". 1. Certain test in nczarr_test depend on use of .dodsrc. In order for these to work when testing in parallel, some inter-test dependencies needed to be added. 1. Improved authorization testing to use changes in thredds.ucar.edu
2021-04-25 09:48:15 +08:00
echo "*** Test: pure zarr write then read; format=$zext"
Turn on the xarray convention for NCZarr format re: https://github.com/pydata/xarray/issues/6374 As a result of a discussion about Xarray (see above issue), I decided to turn on the xarray convention for NCZarr datasets where possible so that xarray can read a larger set of nczarr generated datasets. This causes the following changes: * If the user wants to generate a pure zarr file, then the mode "zarr" must be explicitly used; it is no longer the case that "mode=xarray" or mode="noxarray" implies "mode=zarr". * It is still the case that "mode=noxarray" will turn off the XArray convention. The following conditions will cause ''_ARRAY_DIMENSIONS'' to not be written. * The variable is not in the root group, * Any dimension referenced by the variable is not in the root group.
2022-03-24 09:28:06 +08:00
fileargs tmp_purezarr "mode=zarr,noxarray,$zext"
Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute The XArray implementation that uses Zarr for storage provides a mechanism to simulate named dimensions. It does this by adding a per-variable attribute called _ARRAY_DIMENSIONS. This attribute contains a list of names to be matched against the shape values of the variable. In effect a named dimension is created with the name _ARRAY_DIMENSIONS(i) and length shape(i) for all i in range 0..rank(variable). Both read and write support is provided. This XArray support is only invoked if the mode value of "xarray" is defined. So for example, as in this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,xarray,s3 ```` Note that the "xarray" mode flag also implies mode flag "zarr", so the above is equivalent to this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,zarr,xarray,s3 ```` The primary change to implement this was to unify the handling of dimension references in libnczarr/zsync. A test for this and other pure-zarr features was added as nczarr_test/run_purezarr.sh Other changes: * Make sure distcheck leaves no files around. * Change the special attribute flag DIMSCALEFLAG to HIDDENATTRFLAG to support the xarray attribute. * Annotate the zmap implementations with feature flags such as WRITEONCE (for zip files).
2021-02-25 04:46:11 +08:00
deletemap $zext $file
${NCGEN} -4 -b -o "$fileurl" $srcdir/ref_purezarr_base.cdl
${NCDUMP} $fileurl > tmp_purezarr_${zext}.cdl
diff -b ${srcdir}/ref_purezarr.cdl tmp_purezarr_${zext}.cdl
Upgrade the nczarr code to match Zarr V2 Re: https://github.com/zarr-developers/zarr-python/pull/716 The Zarr version 2 spec has been extended to include the ability to choose the dimension separator in chunk name keys. The legal separators has been extended from {'.'} to {'.' '/'}. So now it is possible to use a key like "0/1/2/0" for chunk names. This PR implements this for NCZarr. The V2 spec now says that this separator can be set on a per-variable basis. For now, I have chosen to allow this be set only globally by adding a key named "ZARR.DIMENSION_SEPARATOR=<char>" in the .daprc/.dodsrc/ncrc file. Currently, the only legal separator characters are '.' (the default) and '/'. On writing, this key will only be written if its value is different than the default. This change caused problems because supporting a separator of '/' is difficult to parse when keys/paths use '/' as the path separator. A test case was added for this. Additionally, make nczarr be enabled default by default. This required some additional changes so that if zip and/or AWS S3 sdk are unavailable, then they are disabled for NCZarr. In addition the following unrelated changes were made. 1. Tested that pure-zarr mode could read an nczarr formatted store. 1. The .rc file handling now merges all known .rc files (.ncrc,.daprc, and .dodsrc) in that order and using those in HOME first, then in current directory. For duplicate entries, the later ones override the earlier ones. This change is to remove some of the conflicts inherent in the current .rc file load process. A set of test cases was also added. 1. Re-order tests in configure.ac and CMakeLists.txt so that if libcurl is not found then the other options that depend upon it properly are disabled. 1. I decided that xarray support should be enabled by default for pure zarr. In order to allow disabling, I added a new mode flag "noxarray". 1. Certain test in nczarr_test depend on use of .dodsrc. In order for these to work when testing in parallel, some inter-test dependencies needed to be added. 1. Improved authorization testing to use changes in thredds.ucar.edu
2021-04-25 09:48:15 +08:00
echo "*** Test: xarray zarr write then read; format=$zext"
fileargs tmp_xarray "mode=zarr,$zext"
Significantly Improve Amazon S3 Cloud Storage Support ## S3 Related Fixes * Add comprehensive support for specifying AWS profiles to provide access credentials. * Parse the files "~/.aws/config" and "~/.aws/credentials to provide credentials for the HDF5 ROS3 driver and to locate default region. * Add a function to obtain the currently active S3 credentials. The search rules are defined in docs/nczarr.md. * Provide documentation for the new features. * Modify the struct NCauth (in include/ncauth.h) to replace specific S3 credentials with a profile name. * Add a unit test to test the operation of profile and credentials management. * Add support for URLS of the form "s3://<bucket>/<key>"; this requires obtaining a default region. * Allows the specification of profile and/or region in a URL of the form "#mode=nczarr,...&aws.region=...&aws.profile=..." ## Misc. Fixes * Move the ezxml code to libdispatch so that it can be used both by DAP4 and nczarr. * Modify nclist to provide a deep clone operation. * Modify ncuri to provide a deep clone operation. * Modify the .rc file format to allow the specification of a path to be tested when looking for an entry in the .rc file. * Ensure that the NC_rcload function is called. * Modify nchttp to support setting request headers.
2021-09-28 08:36:33 +08:00
#deletemap $zext $file
Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute The XArray implementation that uses Zarr for storage provides a mechanism to simulate named dimensions. It does this by adding a per-variable attribute called _ARRAY_DIMENSIONS. This attribute contains a list of names to be matched against the shape values of the variable. In effect a named dimension is created with the name _ARRAY_DIMENSIONS(i) and length shape(i) for all i in range 0..rank(variable). Both read and write support is provided. This XArray support is only invoked if the mode value of "xarray" is defined. So for example, as in this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,xarray,s3 ```` Note that the "xarray" mode flag also implies mode flag "zarr", so the above is equivalent to this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,zarr,xarray,s3 ```` The primary change to implement this was to unify the handling of dimension references in libnczarr/zsync. A test for this and other pure-zarr features was added as nczarr_test/run_purezarr.sh Other changes: * Make sure distcheck leaves no files around. * Change the special attribute flag DIMSCALEFLAG to HIDDENATTRFLAG to support the xarray attribute. * Annotate the zmap implementations with feature flags such as WRITEONCE (for zip files).
2021-02-25 04:46:11 +08:00
${NCGEN} -4 -b -o "$fileurl" $srcdir/ref_purezarr_base.cdl
${NCDUMP} $fileurl > tmp_xarray_${zext}.cdl
diff -b ${srcdir}/ref_xarray.cdl tmp_xarray_${zext}.cdl
Significantly Improve Amazon S3 Cloud Storage Support ## S3 Related Fixes * Add comprehensive support for specifying AWS profiles to provide access credentials. * Parse the files "~/.aws/config" and "~/.aws/credentials to provide credentials for the HDF5 ROS3 driver and to locate default region. * Add a function to obtain the currently active S3 credentials. The search rules are defined in docs/nczarr.md. * Provide documentation for the new features. * Modify the struct NCauth (in include/ncauth.h) to replace specific S3 credentials with a profile name. * Add a unit test to test the operation of profile and credentials management. * Add support for URLS of the form "s3://<bucket>/<key>"; this requires obtaining a default region. * Allows the specification of profile and/or region in a URL of the form "#mode=nczarr,...&aws.region=...&aws.profile=..." ## Misc. Fixes * Move the ezxml code to libdispatch so that it can be used both by DAP4 and nczarr. * Modify nclist to provide a deep clone operation. * Modify ncuri to provide a deep clone operation. * Modify the .rc file format to allow the specification of a path to be tested when looking for an entry in the .rc file. * Ensure that the NC_rcload function is called. * Modify nchttp to support setting request headers.
2021-09-28 08:36:33 +08:00
Turn on the xarray convention for NCZarr format re: https://github.com/pydata/xarray/issues/6374 As a result of a discussion about Xarray (see above issue), I decided to turn on the xarray convention for NCZarr datasets where possible so that xarray can read a larger set of nczarr generated datasets. This causes the following changes: * If the user wants to generate a pure zarr file, then the mode "zarr" must be explicitly used; it is no longer the case that "mode=xarray" or mode="noxarray" implies "mode=zarr". * It is still the case that "mode=noxarray" will turn off the XArray convention. The following conditions will cause ''_ARRAY_DIMENSIONS'' to not be written. * The variable is not in the root group, * Any dimension referenced by the variable is not in the root group.
2022-03-24 09:28:06 +08:00
echo "*** Test: pure zarr reading nczarr; format=$zext"
fileargs tmp_nczarr "mode=nczarr,noxarray,$zext"
Upgrade the nczarr code to match Zarr V2 Re: https://github.com/zarr-developers/zarr-python/pull/716 The Zarr version 2 spec has been extended to include the ability to choose the dimension separator in chunk name keys. The legal separators has been extended from {'.'} to {'.' '/'}. So now it is possible to use a key like "0/1/2/0" for chunk names. This PR implements this for NCZarr. The V2 spec now says that this separator can be set on a per-variable basis. For now, I have chosen to allow this be set only globally by adding a key named "ZARR.DIMENSION_SEPARATOR=<char>" in the .daprc/.dodsrc/ncrc file. Currently, the only legal separator characters are '.' (the default) and '/'. On writing, this key will only be written if its value is different than the default. This change caused problems because supporting a separator of '/' is difficult to parse when keys/paths use '/' as the path separator. A test case was added for this. Additionally, make nczarr be enabled default by default. This required some additional changes so that if zip and/or AWS S3 sdk are unavailable, then they are disabled for NCZarr. In addition the following unrelated changes were made. 1. Tested that pure-zarr mode could read an nczarr formatted store. 1. The .rc file handling now merges all known .rc files (.ncrc,.daprc, and .dodsrc) in that order and using those in HOME first, then in current directory. For duplicate entries, the later ones override the earlier ones. This change is to remove some of the conflicts inherent in the current .rc file load process. A set of test cases was also added. 1. Re-order tests in configure.ac and CMakeLists.txt so that if libcurl is not found then the other options that depend upon it properly are disabled. 1. I decided that xarray support should be enabled by default for pure zarr. In order to allow disabling, I added a new mode flag "noxarray". 1. Certain test in nczarr_test depend on use of .dodsrc. In order for these to work when testing in parallel, some inter-test dependencies needed to be added. 1. Improved authorization testing to use changes in thredds.ucar.edu
2021-04-25 09:48:15 +08:00
deletemap $zext $file
${NCGEN} -4 -b -o "$fileurl" $srcdir/ref_whole.cdl
fileargs tmp_nczarr "mode=zarr,$zext"
${NCDUMP} -n nczarr2zarr $fileurl > tmp_nczarr_${zext}.cdl
diff -b ${srcdir}/ref_nczarr2zarr.cdl tmp_nczarr_${zext}.cdl
Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute The XArray implementation that uses Zarr for storage provides a mechanism to simulate named dimensions. It does this by adding a per-variable attribute called _ARRAY_DIMENSIONS. This attribute contains a list of names to be matched against the shape values of the variable. In effect a named dimension is created with the name _ARRAY_DIMENSIONS(i) and length shape(i) for all i in range 0..rank(variable). Both read and write support is provided. This XArray support is only invoked if the mode value of "xarray" is defined. So for example, as in this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,xarray,s3 ```` Note that the "xarray" mode flag also implies mode flag "zarr", so the above is equivalent to this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,zarr,xarray,s3 ```` The primary change to implement this was to unify the handling of dimension references in libnczarr/zsync. A test for this and other pure-zarr features was added as nczarr_test/run_purezarr.sh Other changes: * Make sure distcheck leaves no files around. * Change the special attribute flag DIMSCALEFLAG to HIDDENATTRFLAG to support the xarray attribute. * Annotate the zmap implementations with feature flags such as WRITEONCE (for zip files).
2021-02-25 04:46:11 +08:00
}
testcase file
if test "x$FEATURE_NCZARR_ZIP" = xyes ; then testcase zip; fi
if test "x$FEATURE_S3TESTS" = xyes ; then testcase s3; fi
Reference in New Issue Copy Permalink