netcdf-c/dap4_test/maketests.sh

105 lines
2.9 KiB
Bash
Raw Normal View History

#!/bin/sh
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
# The purpose of this program is to fill in the
# testdata files: dmrtestfiles, daptestfiles, and
# cdltestfiles.
#
# It obtains that data from the output of a thredds
# test called GenerateRaw.java. See the description
# in that program of its actions.
#
# It is assumed that the SRC environment variable
# (below) points to the directory in the thredds
# tree containing the directories named
# rawtestfiles and testfiles.
#
# This program copies the files
# ${SRC}/dmrtestfiles/*.dmr to the dmrtestfiles
# directory in this (dap4_test) directory.
# Similarly it copies ${SRC}/daptestfiles/*.dap to
# the local daptestfiles directory. Additionally,
# it looks at ${SRC}/testfiles/*.cdl to see if
# there are any .cdl files corresponding (by name)
# to the files in daptestfiles.
if test $# != 0 ; then
for arg in "$@"; do
case "${arg}" in
debug) DEBUG=1 ;;
git) GITADD=1 ;;
*) echo unknown argument $arg ;;
esac
done
fi
SRC="d:/git/thredds/dap4/d4tests/src/test/data/resources"
# Compare files instead of recreating
if test "x$DEBUG" = x1 ; then
pushd ./daptestfiles
F=`ls -1 *.dap | sed -e 's/[.]dap//' |tr '\r\n' ' '`
popd
for f in ${F} ; do
Improve S3 Documentation and Support ## Improvements to S3 Documentation * Create a new document *quickstart_paths.md* that give a summary of the legal path formats used by netcdf-c. This includes both file paths and URL paths. * Modify *nczarr.md* to remove most of the S3 related text. * Move the S3 text from *nczarr.md* to a new document *cloud.md*. * Add some S3-related text to the *byterange.md* document. Hopefully, this will make it easier for users to find the information they want. ## Rebuild NCZarr Testing In order to avoid problems with running make check in parallel, two changes were made: 1. The *nczarr_test* test system was rebuilt. Now, for each test. any generated files are kept in a test-specific directory, isolated from all other test executions. 2. Similarly, since the S3 test bucket is shared, any generated S3 objects are isolated using a test-specific key path. ## Other S3 Related Changes * Add code to ensure that files created on S3 are reclaimed at end of testing. * Used the bash "trap" command to ensure S3 cleanup even if the test fails. * Cleanup the S3 related configure.ac flag set since S3 is used in several places. So now one should use the option *--enable-s3* instead of *--enable-nczarr-s3*, although the latter is still kept as a deprecated alias for the former. * Get some of the github actions yml to work with S3; required fixing various test scripts adding a secret to access the Unidata S3 bucket. * Cleanup S3 portion of libnetcdf.settings.in and netcdf_meta.h.in and test_common.in. * Merge partial S3 support into dhttp.c. * Create an experimental s3 access library especially for use with Windows. It is enabled by using the options *--enable-s3-internal* (automake) or *-DENABLE_S3_INTERNAL=ON* (CMake). Also add a unit-test for it. * Move some definitions from ncrc.h to ncs3sdk.h ## Other Changes * Provide a default implementation of strlcpy and move this and similar defaults into *dmissing.c*.
2023-04-26 07:15:06 +08:00
if ! diff -wBb ${SRC}/dmrtestfiles/${f}.dmr ./dmrtestfiles/${f}.dmr &> /dev/null ; then
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
echo diff -wBb ${SRC}/dmrtestfiles/${f}.dmr ./dmrtestfiles/${f}.dmr
diff -wBb ${SRC}/dmrtestfiles/${f}.dmr ./dmrtestfiles/${f}.dmr
fi
done
for f in ${F} ; do
diff -wBb ${SRC}/daptestfiles/${f}.dapr ./dmrtestfiles/${f}.dap
done
exit
fi
# Recreate the <testfiles> directories
rm -fr ./daptestfiles ./cdltestfiles ./dmrtestfiles
mkdir ./daptestfiles
mkdir ./dmrtestfiles
mkdir ./cdltestfiles
# Copy the .dap and .dmr files
cp ${SRC}/daptestfiles/*.dap ./daptestfiles
cp ${SRC}/dmrtestfiles/*.dmr ./dmrtestfiles
# Get the list of daptestfiles with the extensions removed
pushd ./daptestfiles
F=`ls -1 *.nc.dap | sed -e 's/[.]nc[.]dap//' |tr '\r\n' ' '`
popd
# For each .dap file, see if a corresponding .cdl file
# exists in ${SRC}/testfiles; if so, copy it over.
for f in ${F} ; do
if test -e ${SRC}/testfiles/${f}.cdl ; then
cp ${SRC}/testfiles/${f}.cdl ./cdltestfiles
else
echo "Not found: ${SRC}/testfiles/${f}.cdl"
fi
done
# Compile the cdl files
pushd ./cdltestfiles
F=`ls -1 *.cdl | sed -e 's/[.]cdl//' |tr '\r\n' ' '`
popd
if ! test -e nctestfiles ; then mkdir nctestfiles; fi
for f in $F ; do
../ncgen/ncgen -4 -o nctestfiles/${f}.nc cdltestfiles/${f}.cdl
done
# Fix the permissions; this is cosmetic
pushd daptestfiles ; chmod a-x * ; popd
pushd dmrtestfiles ; chmod a-x * ; popd
pushd cdltestfiles ; chmod a-x * ; popd
pushd nctestfiles ; chmod a-x * ; popd
# If invoked with the argument "git", this
# program will add the files to the git repo.
if test "x${GITADD}" = x1 ; then
git add daptestfiles/*.dap
git add daptestfiles/*.dmr
git add cdltestfiles/*.cdl
git add nctestfiles/*.nc
fi
exit