re: https://github.com/Unidata/netcdf-c/issues/1584
Support has been added for multiple filters per variable. This
affects a number of components in netcdf. The new APIs are
documented in NUG/filters.md.
The primary changes are:
* A set of new functions are provided (see __include/netcdf_filter.h__).
- Obtain a list of the filters associated with a variable
- Obtain the parameters for a specific filter.
* The existing __nc_inq_var_filter__ function now returns info
about the first defined filter.
* The utilities (ncgen, ncdump, and nccopy) now support
an extended format for specifying a sequence of filters.
The general form is __<filter>|<filter>..._.
* The ncdump **_Filter** attribute now dumps a list of all the
filters associated with a variable using the above new format.
* Filter specifications can now use a filter name instead of number
for filters known to the netcdf library, which in turn is taken
from the HDF5 filter registration page.
* New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter
is returned if an attempt is made to access an unknown filter.
* Internally, the dispatch table has been extended to add a function
to handle all of the filter functions.
* New, filter-related, tests were added to nc_test4.
* A new plugin was added to the plugins directory to help with testing.
Notes:
1. The shuffle and fletcher32 filters are not part of the multifilter system.
Misc. changes:
1. A debug module was added to libhdf5 to help catch error locations.
Some versions of some servers are returning malformed responses.
Make the library either handle them or gracefully fail.
The three server errors "fixed" here are as follows.
1. The attribute _NCProperties sometimes has a trailing nul character
in its value. Soln is to elide the nul(s).
2. Sometimes a DAP response has no data part, only a DMR.
Soln is to detect and return an error code instead of crashing.
3. Sometimes a server returns a redirection, but our current
openmagic() function was not following the redirect. Soln
is to follow redirects.
Also because of #2, I am temporarily making --disable-dap-remote-tests
be the default.
1) We have to use H5Tequal() to compare HDF5 type IDs.
2) When checking if we can re-use an NC_CHAR attribute it is enough to
compare data types (H5Tequal() takes care of the size comparison).
3) This commit adds missing code (reuse_att was set but not used).
Now an attribute in a NetCDF-4 file can be modified as many times as
necessary, as long its type and length remain the same.
Modifications changing either type or length of an attribute require
deleting and re-creating an attribute which increments the attribute
order creation index. Once this index reaches 65535 all attribute
modifications (for a particular group or variable) will fail.
For reference:
Issue 350 title: NetCDF-4 limits the number of times an attribute can
be modified
Pull request 1119 title: Fix checking for HDF5 max dims, no longer
re-create atts if not needed, confirm behavior for HDF5 cyclical
files, allow user to set mpiexec
* For URL paths, the new approach essentially centralizes all information
in the URL into the "#mode=" fragment key and uses that value
to determine the dispatcher for (most) URLs.
* The new approach has the following steps:
1. canonicalize the path if it is a URL.
2. use the mode= fragment key to determine the dispatcher
3. if dispatcher still not determined, then use the mode flags
argument to nc_open/nc_create to determine the dispatcher.
4. if the path points to something readable, attempt to read the
magic number at the front, and use that to determine the dispatcher.
this case may override all previous cases.
* Misc changes.
1. Update documentation
2. Moved some unit tests from libdispatch to unit_test directory.
3. Fixed use of wrong #ifdef macro in test_filter_reg.c
[I think this may fix an previously reported esupport query].
Partially address: https://github.com/Unidata/netcdf-c/issues/1056
Currently, some of the entries in the dispatch table
are conditional'd on USE_NETCDF4.
As a step in upgrading the dispatch table for use
with user-defined tables, we remove that conditional.
This means that all dispatch tables must implement the
netcdf-4 specific functions even if only to make them
return NC_ENOTNC4. To simplify this, a set of default
functions are defined in libdispatch/dnotnc4.c to provide this
behavior. The file libdispatch/dnotnc3.c is also relevant to
this.
The primary fix is to modify the various dispatch tables to
remove the conditional and use the functions in
libdispatch/dnotnc4.c as appropriate. In practice, all of the
existing tables are prepared to handle this, so the only
real change is to remove the conditionals.
Misc. Unrelated fixes
1. Fix some annoying warnings in ncvalidator.
Notes:
1. This has not been tested with either pnetcdf or hdf4 enabled.
When those are enabled, it is possible that there are still
some conditionals that need to be fixed.
This fixes an issue hit by GDAL, and that is found in netcdf 4.6.3
and 4.7.0
git bisect pointed the problem to have started with
```
77ab979c5f is the first bad commit
commit 77ab979c5f
Author: Ed Hartnett <edwardjameshartnett@gmail.com>
Date: Sat Jun 16 09:58:48 2018 -0600
using get_vars but not put_vars
:040000 040000 8611e77aaefc9ffd1d13 M libsrc4
```
where nc_get_vara_double() started using nc4_get_vars() underneath.
It turns out that nc4_get_vars() was buggy in the situation exercised by GDAL.
This can be reproduced with the following simple test case:
```
int main()
{
int status;
int cdfid = -1;
int first_dim;
int varid;
int other_var;
size_t anStart[NC_MAX_DIMS];
size_t anCount[NC_MAX_DIMS];
double* val = (double*)calloc(3, sizeof(double));
status = nc_create("foo.nc", NC_NETCDF4, &cdfid);
assert( status == NC_NOERR );
status = nc_def_dim(cdfid, "unlimited_dim", NC_UNLIMITED, &first_dim);
assert( status == NC_NOERR );
status = nc_def_var(cdfid, "my_var", NC_DOUBLE, 1, &first_dim, &varid);
assert( status == NC_NOERR );
status = nc_def_var(cdfid, "other_var", NC_DOUBLE, 1, &first_dim, &other_var);
assert( status == NC_NOERR );
status = nc_enddef(cdfid);
assert( status == NC_NOERR );
/* Write 3 elements to set the size of the unlimited dim to 3 */
anStart[0] = 0;
anCount[0] = 3;
status = nc_put_vara_double(cdfid, other_var, anStart, anCount, val);
assert( status == NC_NOERR );
/* Read 2 elements starting with index=1 */
anStart[0] = 1;
anCount[0] = 2;
status = nc_get_vara_double(cdfid, varid, anStart, anCount, val);
assert( status == NC_NOERR );
status = nc_close(cdfid);
assert( status == NC_NOERR );
free(val);
return 0;
}
```
Running it under Valgrind without this patch leads to
```
==19637==
==19637== Invalid write of size 8
==19637== at 0x4C326CB: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==19637== by 0x4EDBE3D: NC4_get_vars (hdf5var.c:2131)
==19637== by 0x4EDA24C: NC4_get_vara (hdf5var.c:1342)
==19637== by 0x4E68878: NC_get_vara (dvarget.c:104)
==19637== by 0x4E69FDB: nc_get_vara_double (dvarget.c:815)
==19637== by 0x400C08: main (in /home/even/netcdf-c/build/test)
==19637== Address 0xb70e3e8 is 8 bytes before a block of size 24 alloc'd
==19637== at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==19637== by 0x4009E8: main (in /home/even/netcdf-c/build/test)
==19637==
```
re: issue https://github.com/Unidata/netcdf-c/issues/1398
re: esupport NDY-294972
The new chunking code added to nccopy missed one case.
In the event that there are no chunking specifications
of any kind, and the input is not netcdf-4, and the output
is netcdf-4 and must be chunked, then use the default chunking
that the library computes as part of the nc_def_var() function.
Misc. changes:
1. add some chunking debug code to hdf5var.c
re: https://github.com/Unidata/netcdf-c/issues/1388
1. Centralize calls to curl_global_init and curl_global_cleanup
to libdispatch/ddispatch.c
2. Make the above calls if options require curl: currently
any of DAP2, DAP4, or byterange.
3. Side issue: Fix obscure bug in mmapio.c involving non-persistent mmap.
Anyway, I repaired it as follows:
1. Created NC4_write_provenance as parallel to NC4_read_provenance
2. Modified hdf5file.c to use NC4_write_provenance
3. Modified hdf5open.c to use NC4_read_provenance (was NC4_read_ncproperties).
4. The creation of the _NCProperties string was seriously hosed:
was using all the wrong fields.
