2018-08-13 13:01:08 +08:00
|
|
|
#!/bin/sh
|
2017-11-17 04:03:35 +08:00
|
|
|
# This shell script tests ncdump for netcdf-4
|
2017-12-19 05:26:33 +08:00
|
|
|
# Ed Hartnett, Dennis Heimbigner, Ward Fisher
|
2017-03-09 08:01:10 +08:00
|
|
|
|
2017-12-13 00:44:22 +08:00
|
|
|
if test "x$srcdir" = x ; then srcdir=`pwd`; fi
|
2017-03-09 08:01:10 +08:00
|
|
|
. ../test_common.sh
|
|
|
|
|
|
2010-06-03 21:24:43 +08:00
|
|
|
set -e
|
2012-05-12 04:26:11 +08:00
|
|
|
|
2018-08-26 11:44:41 +08:00
|
|
|
# Remove the version information from _NCProperties
|
|
|
|
|
cleanncprops() {
|
|
|
|
|
src="$1"
|
|
|
|
|
dst="$2"
|
|
|
|
|
rm -f $dst
|
|
|
|
|
cat $src \
|
This PR adds EXPERIMENTAL support for accessing data in the
cloud using a variant of the Zarr protocol and storage
format. This enhancement is generically referred to as "NCZarr".
The data model supported by NCZarr is netcdf-4 minus the user-defined
types and the String type. In this sense it is similar to the CDF-5
data model.
More detailed information about enabling and using NCZarr is
described in the document NUG/nczarr.md and in a
[Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in).
WARNING: this code has had limited testing, so do use this version
for production work. Also, performance improvements are ongoing.
Note especially the following platform matrix of successful tests:
Platform | Build System | S3 support
------------------------------------
Linux+gcc | Automake | yes
Linux+gcc | CMake | yes
Visual Studio | CMake | no
Additionally, and as a consequence of the addition of NCZarr,
major changes have been made to the Filter API. NOTE: NCZarr
does not yet support filters, but these changes are enablers for
that support in the future. Note that it is possible
(probable?) that there will be some accidental reversions if the
changes here did not correctly mimic the existing filter testing.
In any case, previously filter ids and parameters were of type
unsigned int. In order to support the more general zarr filter
model, this was all converted to char*. The old HDF5-specific,
unsigned int operations are still supported but they are
wrappers around the new, char* based nc_filterx_XXX functions.
This entailed at least the following changes:
1. Added the files libdispatch/dfilterx.c and include/ncfilter.h
2. Some filterx utilities have been moved to libdispatch/daux.c
3. A new entry, "filter_actions" was added to the NCDispatch table
and the version bumped.
4. An overly complex set of structs was created to support funnelling
all of the filterx operations thru a single dispatch
"filter_actions" entry.
5. Move common code to from libhdf5 to libsrc4 so that it is accessible
to nczarr.
Changes directly related to Zarr:
1. Modified CMakeList.txt and configure.ac to support both C and C++
-- this is in support of S3 support via the awd-sdk libraries.
2. Define a size64_t type to support nczarr.
3. More reworking of libdispatch/dinfermodel.c to
support zarr and to regularize the structure of the fragments
section of a URL.
Changes not directly related to Zarr:
1. Make client-side filter registration be conditional, with default off.
2. Hack include/nc4internal.h to make some flags added by Ed be unique:
e.g. NC_CREAT, NC_INDEF, etc.
3. cleanup include/nchttp.h and libdispatch/dhttp.c.
4. Misc. changes to support compiling under Visual Studio including:
* Better testing under windows for dirent.h and opendir and closedir.
5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags
and to centralize error reporting.
6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them.
7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible.
Changes Left TO-DO:
1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
|
|
|
| sed -e '/:_Endianness/d' \
|
2021-01-12 05:13:21 +08:00
|
|
|
| sed -e 's/_SuperblockVersion = [12]/_SuperblockVersion = 0/' \
|
2018-08-26 11:44:41 +08:00
|
|
|
| sed -e 's/\(netcdflibversion\|netcdf\)=.*|/\1=NNNN|/' \
|
|
|
|
|
| sed -e 's/\(hdf5libversion\|hdf5\)=.*"/\1=HHHH"/' \
|
2018-09-07 05:53:25 +08:00
|
|
|
| grep -v '_NCProperties' \
|
2018-08-26 11:44:41 +08:00
|
|
|
| cat >$dst
|
|
|
|
|
}
|
|
|
|
|
|
2018-08-13 14:01:53 +08:00
|
|
|
ERR() {
|
2017-12-19 05:26:33 +08:00
|
|
|
RES=$?
|
|
|
|
|
if [ $RES -ne 0 ]; then
|
|
|
|
|
echo "Error found: $RES"
|
|
|
|
|
exit $RES
|
|
|
|
|
fi
|
|
|
|
|
}
|
|
|
|
|
|
2010-06-03 21:24:43 +08:00
|
|
|
echo ""
|
2017-11-25 19:31:58 +08:00
|
|
|
echo "*** Testing ncgen and ncdump for netCDF-4 format."
|
2017-12-19 05:26:33 +08:00
|
|
|
${NCGEN} -k nc4 -b -o tst_netcdf4_c0_4.nc ${ncgenc04} ;ERR
|
|
|
|
|
${NCDUMP} -n c1 tst_netcdf4_c0_4.nc | sed 's/e+0/e+/g' > tst_netcdf4_c1_4.cdl ; ERR
|
|
|
|
|
diff -b tst_netcdf4_c1_4.cdl $srcdir/ref_ctest1_nc4.cdl ; ERR
|
2010-06-03 21:24:43 +08:00
|
|
|
|
2017-11-25 20:19:11 +08:00
|
|
|
echo "*** Creating test output tst_netcdf4_c0.nc."
|
2017-12-19 05:26:33 +08:00
|
|
|
${NCGEN} -k nc7 -b -o tst_netcdf4_c0.nc ${ncgenc0} ; ERR
|
2014-09-19 08:26:06 +08:00
|
|
|
|
2017-11-25 20:19:11 +08:00
|
|
|
echo "*** Testing that program tst_h_rdc0 can read tst_netcdf4_c0.nc."
|
2017-12-19 05:26:33 +08:00
|
|
|
${execdir}/tst_h_rdc0 ; ERR
|
2017-11-17 04:03:35 +08:00
|
|
|
|
|
|
|
|
echo "*** Running tst_create_files.c to create test files."
|
2017-12-19 05:26:33 +08:00
|
|
|
${execdir}/tst_create_files ; ERR
|
2017-11-17 04:03:35 +08:00
|
|
|
echo "*** Testing tst_create_files output for netCDF-4 features."
|
2017-12-19 05:26:33 +08:00
|
|
|
${NCDUMP} tst_solar_1.nc | sed 's/e+0/e+/g' > tst_solar_1.cdl ; ERR
|
|
|
|
|
diff -b tst_solar_1.cdl $srcdir/ref_tst_solar_1.cdl ; ERR
|
|
|
|
|
${NCDUMP} tst_solar_2.nc | sed 's/e+0/e+/g' > tst_solar_2.cdl ; ERR
|
|
|
|
|
diff -b tst_solar_2.cdl $srcdir/ref_tst_solar_2.cdl ; ERR
|
2017-11-17 04:03:35 +08:00
|
|
|
|
2020-07-03 03:59:37 +08:00
|
|
|
if test -f tst_roman_szip_simple.nc; then
|
|
|
|
|
echo "*** Testing szip compression."
|
|
|
|
|
${NCDUMP} tst_roman_szip_simple.nc | sed 's/e+0/e+/g' > tst_roman_szip_simple.cdl ; ERR
|
|
|
|
|
diff -b tst_roman_szip_simple.cdl $srcdir/ref_roman_szip_simple.cdl ; ERR
|
|
|
|
|
${NCDUMP} tst_roman_szip_unlim.nc | sed 's/e+0/e+/g' > tst_roman_szip_unlim.cdl ; ERR
|
|
|
|
|
diff -b tst_roman_szip_unlim.cdl $srcdir/ref_roman_szip_unlim.cdl ; ERR
|
|
|
|
|
fi
|
|
|
|
|
|
2017-11-17 04:03:35 +08:00
|
|
|
echo "*** Running tst_group_data.c to create test files."
|
2017-12-19 05:26:33 +08:00
|
|
|
${execdir}/tst_group_data ; ERR
|
|
|
|
|
${NCDUMP} tst_group_data.nc | sed 's/e+0/e+/g' > tst_group_data.cdl ; ERR
|
|
|
|
|
diff -b tst_group_data.cdl $srcdir/ref_tst_group_data.cdl ; ERR
|
2012-05-12 04:26:11 +08:00
|
|
|
|
2012-06-13 05:50:02 +08:00
|
|
|
echo "*** Testing -v option with absolute name and groups..."
|
2017-12-19 05:26:33 +08:00
|
|
|
${NCDUMP} -v g2/g3/var tst_group_data.nc | sed 's/e+0/e+/g' > tst_group_data.cdl ; ERR
|
|
|
|
|
diff -b tst_group_data.cdl $srcdir/ref_tst_group_data_v23.cdl ; ERR
|
2012-05-12 04:26:11 +08:00
|
|
|
|
2012-06-13 05:50:02 +08:00
|
|
|
echo "*** Testing -v option with relative name and groups..."
|
2017-12-19 05:26:33 +08:00
|
|
|
${NCDUMP} -v var,var2 tst_group_data.nc | sed 's/e+0/e+/g' > tst_group_data.cdl ; ERR
|
|
|
|
|
diff -b tst_group_data.cdl $srcdir/ref_tst_group_data.cdl ; ERR
|
2017-11-17 04:03:35 +08:00
|
|
|
|
|
|
|
|
echo "*** Running tst_enum_data.c to create test files."
|
2017-12-19 05:26:33 +08:00
|
|
|
${execdir}/tst_enum_data ; ERR
|
|
|
|
|
${NCDUMP} tst_enum_data.nc | sed 's/e+0/e+/g' > tst_enum_data.cdl ; ERR
|
|
|
|
|
diff -b tst_enum_data.cdl $srcdir/ref_tst_enum_data.cdl ; ERR
|
2017-11-17 04:03:35 +08:00
|
|
|
|
2022-07-18 04:32:31 +08:00
|
|
|
echo "*** Running tst_enum_undef.c to create test files."
|
|
|
|
|
${execdir}/tst_enum_undef ; ERR
|
|
|
|
|
${NCDUMP} tst_enum_undef.nc | sed 's/e+0/e+/g' > tst_enum_undef.cdl ; ERR
|
|
|
|
|
diff -b tst_enum_undef.cdl $srcdir/ref_tst_enum_undef.cdl ; ERR
|
|
|
|
|
|
2017-11-17 04:03:35 +08:00
|
|
|
echo "*** Running tst_opaque_data.c to create test files."
|
2017-12-19 05:26:33 +08:00
|
|
|
${execdir}/tst_opaque_data ; ERR
|
|
|
|
|
${NCDUMP} tst_opaque_data.nc | sed 's/e+0/e+/g' > tst_opaque_data.cdl ; ERR
|
|
|
|
|
diff -b tst_opaque_data.cdl $srcdir/ref_tst_opaque_data.cdl ; ERR
|
2017-11-17 04:03:35 +08:00
|
|
|
|
|
|
|
|
echo "*** Running tst_comp.c to create test files."
|
2017-12-19 05:26:33 +08:00
|
|
|
${execdir}/tst_comp ; ERR
|
|
|
|
|
${NCDUMP} tst_comp.nc | sed 's/e+0/e+/g' > tst_comp.cdl ; ERR
|
|
|
|
|
diff -b tst_comp.cdl $srcdir/ref_tst_comp.cdl ; ERR
|
2017-11-17 04:03:35 +08:00
|
|
|
|
|
|
|
|
echo "*** Running tst_nans.c to create test files."
|
2017-12-19 05:26:33 +08:00
|
|
|
${execdir}/tst_nans ; ERR
|
|
|
|
|
${NCDUMP} tst_nans.nc | sed 's/e+0/e+/g' > tst_nans.cdl ; ERR
|
|
|
|
|
diff -b tst_nans.cdl $srcdir/ref_tst_nans.cdl ; ERR
|
2017-11-17 04:03:35 +08:00
|
|
|
|
|
|
|
|
echo "*** Running tst_special_atts.c to create test files."
|
2017-12-19 05:26:33 +08:00
|
|
|
${execdir}/tst_special_atts ; ERR
|
2018-08-26 11:44:41 +08:00
|
|
|
${NCDUMP} -c -s tst_special_atts.nc > tst_special_atts.cdl ; ERR
|
|
|
|
|
cleanncprops tst_special_atts.cdl tst_special_atts.tmp
|
|
|
|
|
cleanncprops $srcdir/ref_tst_special_atts.cdl ref_tst_special_atts.tmp
|
2010-06-03 21:24:43 +08:00
|
|
|
echo "*** comparing tst_special_atts.cdl with ref_tst_special_atts.cdl..."
|
2018-08-26 11:44:41 +08:00
|
|
|
diff -b tst_special_atts.tmp ref_tst_special_atts.tmp ; ERR
|
2017-12-19 05:26:33 +08:00
|
|
|
|
2022-01-11 13:26:19 +08:00
|
|
|
# This creates a memory leak
|
|
|
|
|
if test 0 = 1 ; then
|
Fix various problem around VLEN's
re: https://github.com/Unidata/netcdf-c/issues/541
re: https://github.com/Unidata/netcdf-c/issues/1208
re: https://github.com/Unidata/netcdf-c/issues/2078
re: https://github.com/Unidata/netcdf-c/issues/2041
re: https://github.com/Unidata/netcdf-c/issues/2143
For a long time, there have been known problems with the
management of complex types containing VLENs. This also
involves the string type because it is stored as a VLEN of
chars.
This PR (mostly) fixes this problem. But note that it adds new
functions to netcdf.h (see below) and this may require bumping
the .so number. These new functions can be removed, if desired,
in favor of functions in netcdf_aux.h, but netcdf.h seems the
better place for them because they are intended as alternatives
to the nc_free_vlen and nc_free_string functions already in
netcdf.h.
The term complex type refers to any type that directly or
transitively references a VLEN type. So an array of VLENS, a
compound with a VLEN field, and so on.
In order to properly handle instances of these complex types, it
is necessary to have function that can recursively walk
instances of such types to perform various actions on them. The
term "deep" is also used to mean recursive.
At the moment, the two operations needed by the netcdf library are:
* free'ing an instance of the complex type
* copying an instance of the complex type.
The current library does only shallow free and shallow copy of
complex types. This means that only the top level is properly
free'd or copied, but deep internal blocks in the instance are
not touched.
Note that the term "vector" will be used to mean a contiguous (in
memory) sequence of instances of some type. Given an array with,
say, dimensions 2 X 3 X 4, this will be stored in memory as a
vector of length 2*3*4=24 instances.
The use cases are primarily these.
## nc_get_vars
Suppose one is reading a vector of instances using nc_get_vars
(or nc_get_vara or nc_get_var, etc.). These functions will
return the vector in the top-level memory provided. All
interior blocks (form nested VLEN or strings) will have been
dynamically allocated.
After using this vector of instances, it is necessary to free
(aka reclaim) the dynamically allocated memory, otherwise a
memory leak occurs. So, the recursive reclaim function is used
to walk the returned instance vector and do a deep reclaim of
the data.
Currently functions are defined in netcdf.h that are supposed to
handle this: nc_free_vlen(), nc_free_vlens(), and
nc_free_string(). Unfortunately, these functions only do a
shallow free, so deeply nested instances are not properly
handled by them.
Note that internally, the provided data is immediately written so
there is no need to copy it. But the caller may need to reclaim the
data it passed into the function.
## nc_put_att
Suppose one is writing a vector of instances as the data of an attribute
using, say, nc_put_att.
Internally, the incoming attribute data must be copied and stored
so that changes/reclamation of the input data will not affect
the attribute.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. As a result, one sees effects such as described
in Github Issue https://github.com/Unidata/netcdf-c/issues/2143.
Also, after defining the attribute, it may be necessary for the user
to free the data that was provided as input to nc_put_att().
## nc_get_att
Suppose one is reading a vector of instances as the data of an attribute
using, say, nc_get_att.
Internally, the existing attribute data must be copied and returned
to the caller, and the caller is responsible for reclaiming
the returned data.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. So this can lead to memory leaks and errors
because the deep data is shared between the library and the user.
# Solution
The solution is to build properly recursive reclaim and copy
functions and use those as needed.
These recursive functions are defined in libdispatch/dinstance.c
and their signatures are defined in include/netcdf.h.
For back compatibility, corresponding "ncaux_XXX" functions
are defined in include/netcdf_aux.h.
````
int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy);
int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp);
````
There are two variants. The first two, nc_reclaim_data() and
nc_copy_data(), assume the top-level vector is managed by the
caller. For reclaim, this is so the user can use, for example, a
statically allocated vector. For copy, it assumes the user
provides the space into which the copy is stored.
The second two, nc_reclaim_data_all() and
nc_copy_data_all(), allows the functions to manage the
top-level. So for nc_reclaim_data_all, the top level is
assumed to be dynamically allocated and will be free'd by
nc_reclaim_data_all(). The nc_copy_data_all() function
will allocate the top level and return a pointer to it to the
user. The user can later pass that pointer to
nc_reclaim_data_all() to reclaim the instance(s).
# Internal Changes
The netcdf-c library internals are changed to use the proper
reclaim and copy functions. It turns out that the places where
these functions are needed is quite pervasive in the netcdf-c
library code. Using these functions also allows some
simplification of the code since the stdata and vldata fields of
NC_ATT_INFO are no longer needed. Currently this is commented
out using the SEPDATA \#define macro. When any bugs are largely
fixed, all this code will be removed.
# Known Bugs
1. There is still one known failure that has not been solved.
All the failures revolve around some variant of this .cdl file.
The proximate cause of failure is the use of a VLEN FillValue.
````
netcdf x {
types:
float(*) row_of_floats ;
dimensions:
m = 5 ;
variables:
row_of_floats ragged_array(m) ;
row_of_floats ragged_array:_FillValue = {-999} ;
data:
ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32},
{40, 41}, _ ;
}
````
When a solution is found, I will either add it to this PR or post a new PR.
# Related Changes
* Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data.
* Remove the --enable-unfixed-memory-leaks option.
* Remove the NC_VLENS_NOTEST code that suppresses some vlen tests.
* Document this change in docs/internal.md
* Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh.
* Mark types as fixed size or not (transitively) to optimize the reclaim
and copy functions.
# Misc. Changes
* Make Doxygen process libdispatch/daux.c
* Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
|
|
|
echo "*** Running tst_vlen_data.c to create test files."
|
|
|
|
|
if ! ${execdir}/tst_vlen_data ; then if test $? != 027 ; then ERR; fi; fi
|
|
|
|
|
${NCDUMP} tst_vlen_data.nc | sed 's/e+0/e+/g' > tst_vlen_data.cdl ; ERR
|
|
|
|
|
diff -b tst_vlen_data.cdl $srcdir/ref_tst_vlen_data.cdl ; ERR
|
2022-01-11 13:26:19 +08:00
|
|
|
fi
|
Fix various problem around VLEN's
re: https://github.com/Unidata/netcdf-c/issues/541
re: https://github.com/Unidata/netcdf-c/issues/1208
re: https://github.com/Unidata/netcdf-c/issues/2078
re: https://github.com/Unidata/netcdf-c/issues/2041
re: https://github.com/Unidata/netcdf-c/issues/2143
For a long time, there have been known problems with the
management of complex types containing VLENs. This also
involves the string type because it is stored as a VLEN of
chars.
This PR (mostly) fixes this problem. But note that it adds new
functions to netcdf.h (see below) and this may require bumping
the .so number. These new functions can be removed, if desired,
in favor of functions in netcdf_aux.h, but netcdf.h seems the
better place for them because they are intended as alternatives
to the nc_free_vlen and nc_free_string functions already in
netcdf.h.
The term complex type refers to any type that directly or
transitively references a VLEN type. So an array of VLENS, a
compound with a VLEN field, and so on.
In order to properly handle instances of these complex types, it
is necessary to have function that can recursively walk
instances of such types to perform various actions on them. The
term "deep" is also used to mean recursive.
At the moment, the two operations needed by the netcdf library are:
* free'ing an instance of the complex type
* copying an instance of the complex type.
The current library does only shallow free and shallow copy of
complex types. This means that only the top level is properly
free'd or copied, but deep internal blocks in the instance are
not touched.
Note that the term "vector" will be used to mean a contiguous (in
memory) sequence of instances of some type. Given an array with,
say, dimensions 2 X 3 X 4, this will be stored in memory as a
vector of length 2*3*4=24 instances.
The use cases are primarily these.
## nc_get_vars
Suppose one is reading a vector of instances using nc_get_vars
(or nc_get_vara or nc_get_var, etc.). These functions will
return the vector in the top-level memory provided. All
interior blocks (form nested VLEN or strings) will have been
dynamically allocated.
After using this vector of instances, it is necessary to free
(aka reclaim) the dynamically allocated memory, otherwise a
memory leak occurs. So, the recursive reclaim function is used
to walk the returned instance vector and do a deep reclaim of
the data.
Currently functions are defined in netcdf.h that are supposed to
handle this: nc_free_vlen(), nc_free_vlens(), and
nc_free_string(). Unfortunately, these functions only do a
shallow free, so deeply nested instances are not properly
handled by them.
Note that internally, the provided data is immediately written so
there is no need to copy it. But the caller may need to reclaim the
data it passed into the function.
## nc_put_att
Suppose one is writing a vector of instances as the data of an attribute
using, say, nc_put_att.
Internally, the incoming attribute data must be copied and stored
so that changes/reclamation of the input data will not affect
the attribute.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. As a result, one sees effects such as described
in Github Issue https://github.com/Unidata/netcdf-c/issues/2143.
Also, after defining the attribute, it may be necessary for the user
to free the data that was provided as input to nc_put_att().
## nc_get_att
Suppose one is reading a vector of instances as the data of an attribute
using, say, nc_get_att.
Internally, the existing attribute data must be copied and returned
to the caller, and the caller is responsible for reclaiming
the returned data.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. So this can lead to memory leaks and errors
because the deep data is shared between the library and the user.
# Solution
The solution is to build properly recursive reclaim and copy
functions and use those as needed.
These recursive functions are defined in libdispatch/dinstance.c
and their signatures are defined in include/netcdf.h.
For back compatibility, corresponding "ncaux_XXX" functions
are defined in include/netcdf_aux.h.
````
int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy);
int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp);
````
There are two variants. The first two, nc_reclaim_data() and
nc_copy_data(), assume the top-level vector is managed by the
caller. For reclaim, this is so the user can use, for example, a
statically allocated vector. For copy, it assumes the user
provides the space into which the copy is stored.
The second two, nc_reclaim_data_all() and
nc_copy_data_all(), allows the functions to manage the
top-level. So for nc_reclaim_data_all, the top level is
assumed to be dynamically allocated and will be free'd by
nc_reclaim_data_all(). The nc_copy_data_all() function
will allocate the top level and return a pointer to it to the
user. The user can later pass that pointer to
nc_reclaim_data_all() to reclaim the instance(s).
# Internal Changes
The netcdf-c library internals are changed to use the proper
reclaim and copy functions. It turns out that the places where
these functions are needed is quite pervasive in the netcdf-c
library code. Using these functions also allows some
simplification of the code since the stdata and vldata fields of
NC_ATT_INFO are no longer needed. Currently this is commented
out using the SEPDATA \#define macro. When any bugs are largely
fixed, all this code will be removed.
# Known Bugs
1. There is still one known failure that has not been solved.
All the failures revolve around some variant of this .cdl file.
The proximate cause of failure is the use of a VLEN FillValue.
````
netcdf x {
types:
float(*) row_of_floats ;
dimensions:
m = 5 ;
variables:
row_of_floats ragged_array(m) ;
row_of_floats ragged_array:_FillValue = {-999} ;
data:
ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32},
{40, 41}, _ ;
}
````
When a solution is found, I will either add it to this PR or post a new PR.
# Related Changes
* Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data.
* Remove the --enable-unfixed-memory-leaks option.
* Remove the NC_VLENS_NOTEST code that suppresses some vlen tests.
* Document this change in docs/internal.md
* Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh.
* Mark types as fixed size or not (transitively) to optimize the reclaim
and copy functions.
# Misc. Changes
* Make Doxygen process libdispatch/daux.c
* Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
|
|
|
|
2017-12-19 05:26:33 +08:00
|
|
|
#echo ""
|
|
|
|
|
#echo "*** Testing ncdump on file with corrupted header "
|
|
|
|
|
#rm -f ./ignore_tst_netcdf4
|
|
|
|
|
#if ${NCDUMP} ${srcdir}/ref_test_corrupt_magic.nc > ./ignore_tst_netcdf4 2>&1 ; then
|
|
|
|
|
#echo "***Fail: ncdump should have failed on ref_test_corrupt_magic.nc"
|
|
|
|
|
#else
|
|
|
|
|
#echo "***XFail: ncdump properly failed on ref_test_corrupt_magic.nc"
|
|
|
|
|
#fi
|
|
|
|
|
#rm -fr ./ignore_tst_netcdf4
|
re e-support UBS-599337
re pull request https://github.com/Unidata/netcdf-c/pull/405
re pull request https://github.com/Unidata/netcdf-c/pull/446
Notes:
1. This branch is a cleanup of the magic.dmh branch.
2. magic.dmh was originally merged, but caused problems with parallel IO.
It was re-issued as pull request https://github.com/Unidata/netcdf-c/pull/446.
3. This branch + pull request replace any previous pull requests and magic.dmh branch.
Given an otherwise valid netCDF file that has a corrupted header,
the netcdf library currently crashes. Instead, it should return
NC_ENOTNC.
Additionally, the NC_check_file_type code does not do the
forward search required by hdf5 files. It currently only looks
at file position 0 instead of 512, 1024, 2048,... Also, it turns
out that the HDF4 magic number is assumed to always be at the
beginning of the file (unlike HDF5).
The change is localized to libdispatch/dfile.c See
https://support.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Also, it turns out that the code in NC_check_file_type is duplicated
(mostly) in the function libsrc4/nc4file.c#nc_check_for_hdf.
This branch does the following.
1. Make NC_check_file_type return NC_ENOTNC instead of crashing.
2. Remove nc_check_for_hdf and centralize all file format checking
NC_check_file_type.
3. Add proper forward search for HDF5 files (but not HDF4 files)
to look for the magic number at offsets of 0, 512, 1024...
4. Add test tst_hdf5_offset.sh. This tests that hdf5 files with
an offset are properly recognized. It does so by prefixing
a legal file with some number of zero bytes: 512, 1024, etc.
5. Off-topic: Added -N flag to ncdump to force a specific output dataset name.
2017-10-25 06:25:09 +08:00
|
|
|
|
2018-05-15 22:24:27 +08:00
|
|
|
# This should work, but does not. See github issue 982.
|
|
|
|
|
#echo "*** creating tst_output_irish_rover.cdl from ref_tst_irish_rover.nc..."
|
2018-05-15 22:13:25 +08:00
|
|
|
#${NCDUMP} ref_tst_irish_rover.nc > tst_output_irish_rover.cdl
|
|
|
|
|
|
2018-08-26 11:44:41 +08:00
|
|
|
rm -f *.tmp
|
2010-06-03 21:24:43 +08:00
|
|
|
echo "*** All ncgen and ncdump test output for netCDF-4 format passed!"
|
|
|
|
|
exit 0
|
