1) We have to use H5Tequal() to compare HDF5 type IDs.
2) When checking if we can re-use an NC_CHAR attribute it is enough to
compare data types (H5Tequal() takes care of the size comparison).
3) This commit adds missing code (reuse_att was set but not used).
Now an attribute in a NetCDF-4 file can be modified as many times as
necessary, as long its type and length remain the same.
Modifications changing either type or length of an attribute require
deleting and re-creating an attribute which increments the attribute
order creation index. Once this index reaches 65535 all attribute
modifications (for a particular group or variable) will fail.
For reference:
Issue 350 title: NetCDF-4 limits the number of times an attribute can
be modified
Pull request 1119 title: Fix checking for HDF5 max dims, no longer
re-create atts if not needed, confirm behavior for HDF5 cyclical
files, allow user to set mpiexec
* For URL paths, the new approach essentially centralizes all information
in the URL into the "#mode=" fragment key and uses that value
to determine the dispatcher for (most) URLs.
* The new approach has the following steps:
1. canonicalize the path if it is a URL.
2. use the mode= fragment key to determine the dispatcher
3. if dispatcher still not determined, then use the mode flags
argument to nc_open/nc_create to determine the dispatcher.
4. if the path points to something readable, attempt to read the
magic number at the front, and use that to determine the dispatcher.
this case may override all previous cases.
* Misc changes.
1. Update documentation
2. Moved some unit tests from libdispatch to unit_test directory.
3. Fixed use of wrong #ifdef macro in test_filter_reg.c
[I think this may fix an previously reported esupport query].
Partially address: https://github.com/Unidata/netcdf-c/issues/1056
Currently, some of the entries in the dispatch table
are conditional'd on USE_NETCDF4.
As a step in upgrading the dispatch table for use
with user-defined tables, we remove that conditional.
This means that all dispatch tables must implement the
netcdf-4 specific functions even if only to make them
return NC_ENOTNC4. To simplify this, a set of default
functions are defined in libdispatch/dnotnc4.c to provide this
behavior. The file libdispatch/dnotnc3.c is also relevant to
this.
The primary fix is to modify the various dispatch tables to
remove the conditional and use the functions in
libdispatch/dnotnc4.c as appropriate. In practice, all of the
existing tables are prepared to handle this, so the only
real change is to remove the conditionals.
Misc. Unrelated fixes
1. Fix some annoying warnings in ncvalidator.
Notes:
1. This has not been tested with either pnetcdf or hdf4 enabled.
When those are enabled, it is possible that there are still
some conditionals that need to be fixed.
This fixes an issue hit by GDAL, and that is found in netcdf 4.6.3
and 4.7.0
git bisect pointed the problem to have started with
```
77ab979c5f is the first bad commit
commit 77ab979c5f
Author: Ed Hartnett <edwardjameshartnett@gmail.com>
Date: Sat Jun 16 09:58:48 2018 -0600
using get_vars but not put_vars
:040000 040000 8611e77aaefc9ffd1d13 M libsrc4
```
where nc_get_vara_double() started using nc4_get_vars() underneath.
It turns out that nc4_get_vars() was buggy in the situation exercised by GDAL.
This can be reproduced with the following simple test case:
```
int main()
{
int status;
int cdfid = -1;
int first_dim;
int varid;
int other_var;
size_t anStart[NC_MAX_DIMS];
size_t anCount[NC_MAX_DIMS];
double* val = (double*)calloc(3, sizeof(double));
status = nc_create("foo.nc", NC_NETCDF4, &cdfid);
assert( status == NC_NOERR );
status = nc_def_dim(cdfid, "unlimited_dim", NC_UNLIMITED, &first_dim);
assert( status == NC_NOERR );
status = nc_def_var(cdfid, "my_var", NC_DOUBLE, 1, &first_dim, &varid);
assert( status == NC_NOERR );
status = nc_def_var(cdfid, "other_var", NC_DOUBLE, 1, &first_dim, &other_var);
assert( status == NC_NOERR );
status = nc_enddef(cdfid);
assert( status == NC_NOERR );
/* Write 3 elements to set the size of the unlimited dim to 3 */
anStart[0] = 0;
anCount[0] = 3;
status = nc_put_vara_double(cdfid, other_var, anStart, anCount, val);
assert( status == NC_NOERR );
/* Read 2 elements starting with index=1 */
anStart[0] = 1;
anCount[0] = 2;
status = nc_get_vara_double(cdfid, varid, anStart, anCount, val);
assert( status == NC_NOERR );
status = nc_close(cdfid);
assert( status == NC_NOERR );
free(val);
return 0;
}
```
Running it under Valgrind without this patch leads to
```
==19637==
==19637== Invalid write of size 8
==19637== at 0x4C326CB: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==19637== by 0x4EDBE3D: NC4_get_vars (hdf5var.c:2131)
==19637== by 0x4EDA24C: NC4_get_vara (hdf5var.c:1342)
==19637== by 0x4E68878: NC_get_vara (dvarget.c:104)
==19637== by 0x4E69FDB: nc_get_vara_double (dvarget.c:815)
==19637== by 0x400C08: main (in /home/even/netcdf-c/build/test)
==19637== Address 0xb70e3e8 is 8 bytes before a block of size 24 alloc'd
==19637== at 0x4C2FB55: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==19637== by 0x4009E8: main (in /home/even/netcdf-c/build/test)
==19637==
```
re: issue https://github.com/Unidata/netcdf-c/issues/1398
re: esupport NDY-294972
The new chunking code added to nccopy missed one case.
In the event that there are no chunking specifications
of any kind, and the input is not netcdf-4, and the output
is netcdf-4 and must be chunked, then use the default chunking
that the library computes as part of the nc_def_var() function.
Misc. changes:
1. add some chunking debug code to hdf5var.c
re: https://github.com/Unidata/netcdf-c/issues/1388
1. Centralize calls to curl_global_init and curl_global_cleanup
to libdispatch/ddispatch.c
2. Make the above calls if options require curl: currently
any of DAP2, DAP4, or byterange.
3. Side issue: Fix obscure bug in mmapio.c involving non-persistent mmap.
Anyway, I repaired it as follows:
1. Created NC4_write_provenance as parallel to NC4_read_provenance
2. Modified hdf5file.c to use NC4_write_provenance
3. Modified hdf5open.c to use NC4_read_provenance (was NC4_read_ncproperties).
4. The creation of the _NCProperties string was seriously hosed:
was using all the wrong fields.
re: https://github.com/Unidata/netcdf-c/issues/1373 (partial)
* Mark some global constants be const to indicate to make them easier to track.
* Hide direct access to the ncrc_globalstate behind a function call.
* Convert dispatch tables to constants (except the user defined ones)
This has some consequences in terms of function arguments needing to be marked
as const also.
* Remove some no longer needed global fields
* Aggregate all the globals in nclog.c
* Uniformly replace nc_sizevector{0,1} with NC_coord_{zero,one}
* Uniformly replace nc_ptrdffvector1 with NC_stride_one
* Remove some obsolete code
Priority: Low
re: issue https://github.com/Unidata/netcdf-c/issues/1329
HDF5 has the ability to programmatically define new filters,
as opposed to using HDF5_PLUGIN_PATH env variable.
This PR adds support for that feature.
Not clear how useful this is, though.
See docs/filters.md for details.
re: https://github.com/Unidata/netcdf-c/issues/1352
When nc4info.c encounters an _NCProperties attribute
with a version number it does not recognize, it does not
show it correctly.
Solution chosen is to arrange so that accessing the attribute
returns the raw value of the Attribute from the file. This way,
even if the version is unrecognized, it will return something
usable.
The changes were primarily to never attempt to parse the value
of _NCProperties until actually required. Which since they
are currently not used means that parsing never occurs.
Also modified ncdump/tst_fileinfo.sh to include some extra testing
I tested the original failure by changing the value of NCPROPS to 3.
However, there is no way to test this at build time.
Misc. Changes
* Inlined the provenance info in the NC_FILE_INFO_T structure
* Centralized stuff from elsewhere into include/nc_provenance.h
Misc. Unrelated Changes
* Removed/turned off some misc debug output left on by accident
* Fix CPPFLAGS name error in libhdf5/Makefile.am
