netcdf-c/nc_test4/tst_bloscfail.sh
Dennis Heimbigner 1552d894a2 Cleanup a number of issues.
re: Issue https://github.com/Unidata/netcdf-c/issues/2748

This PR fixes a number of issues and bugs.

## s3cleanup fixes
* Delete extraneous s3cleanup.sh related files.
* Remove duplicate s3cleanup.uids entries.

## Support the Google S3 API
* Add code to recognize "storage.gooleapis.com"
* Add extra code to track the kind of server being accessed: unknown, Amazon, Google.
* Add a new mode flag "gs3" (analog to "s3") to support this api.
* Modify the S3 URL code to support this case.
* Modify the listobjects result parsing because Google returns some non-standard XML elements.
* Change signature and calls for NC_s3urlrebuild.

## Handle corrupt Zarr files where shape is empty for a variable.
Modify behavior when a variable's "shape" dictionary entry.
Previously it returned an error, but now it suppresses such a variable.
This change makes it possible to read non-corrupt data from the file.
Also added a test case.

## Misc. Other Changes
* Fix the nclog level handling to suppress output by default.
* Fix de-duplicates code in ncuri.c
* Restore testing of iridl.ldeo.columbia.edu.
* Fix bug in define_vars() which did not always do a proper reclaim between variables.
2023-10-08 11:22:52 -06:00

70 lines
1.6 KiB
Bash
Executable File

#!/bin/bash
if test "x$srcdir" = x ; then srcdir=`pwd`; fi
. ../test_common.sh
set -e
# Load the findplugins function
. ${builddir}/findplugin.sh
echo "findplugin.sh loaded"
# Function to remove selected -s attributes from file;
# These attributes might be platform dependent
sclean() {
cat $1 \
| sed -e '/:_IsNetcdf4/d' \
| sed -e '/:_Endianness/d' \
| sed -e '/_NCProperties/d' \
| sed -e '/_SuperblockVersion/d' \
| cat > $2
}
# Function to extract _Filter attribute from a file
# These attributes might be platform dependent
getfilterattr() {
sed -e '/var.*:_Filter/p' -ed <$1 >$2
}
trimleft() {
sed -e 's/[ ]*\([^ ].*\)/\1/' <$1 >$2
}
if test "x$TESTNCZARR" = x1 ; then
. "$srcdir/test_nczarr.sh"
fi
if ! avail blosc; then echo "Blosc compressor not found"; exit 0; fi
# Locate the plugin dir and the library names; argument order is critical
# Find bzip2 and capture
findplugin h5blosc
BLOSCLIB="${HDF5_PLUGIN_LIB}"
BLOSCDIR="${HDF5_PLUGIN_DIR}/${BZIP2LIB}"
echo "final HDF5_PLUGIN_DIR=${HDF5_PLUGIN_DIR}"
export HDF5_PLUGIN_DIR
export HDF5_PLUGIN_PATH="$HDF5_PLUGIN_DIR"
localclean() {
rm -f ./tmp_bloscx3.nc ./tmp_bloscx4.nc ./tmp_bloscx4_fail.nc
}
# Execute the specified tests
echo "*** Testing dynamic filters using API"
localclean
${NCGEN} -3 -o tmp_bloscx3.nc ${srcdir}/ref_bloscx.cdl
# This should pass
${NCCOPY} -4 -V three_dmn_rec_var -F *,32001,0,0,0,0,1,1,0 ./tmp_bloscx3.nc ./tmp_bloscx4.nc
# This should fail because shuffle is off
if ${NCCOPY} -4 -V three_dmn_rec_var -F *,32001,0,0,0,0,1,0,0 ./tmp_bloscx3.nc ./tmp_bloscx4_fail.nc ; then
echo "*** not xfail: nccopy "
else
echo "*** xfail: nccopy "
fi
localclean
exit 0