2018-05-25 04:27:16 +08:00
|
|
|
/* Copyright 2005, University Corporation for Atmospheric Research. See
|
|
|
|
* the COPYRIGHT file for copying and redistribution conditions. */
|
2017-12-04 06:37:56 +08:00
|
|
|
/**
|
2019-02-19 20:56:22 +08:00
|
|
|
* @file
|
2018-06-06 01:30:59 +08:00
|
|
|
* @internal This file is part of netcdf-4, a netCDF-like interface
|
|
|
|
* for HDF5, or a HDF5 backend for netCDF, depending on your point of
|
|
|
|
* view.
|
2017-12-04 06:37:56 +08:00
|
|
|
*
|
|
|
|
* This file handles the nc4 user-defined type functions
|
|
|
|
* (i.e. compound and opaque types).
|
|
|
|
*
|
|
|
|
* @author Ed Hartnett
|
|
|
|
*/
|
2010-06-03 21:24:43 +08:00
|
|
|
#include "nc4internal.h"
|
2012-12-13 04:05:06 +08:00
|
|
|
#include "nc4dispatch.h"
|
2023-11-27 19:36:03 +08:00
|
|
|
#include <stddef.h>
|
2010-06-03 21:24:43 +08:00
|
|
|
|
Improve performance of the nc_reclaim_data and nc_copy_data functions.
re: Issue https://github.com/Unidata/netcdf-c/issues/2685
re: PR https://github.com/Unidata/netcdf-c/pull/2179
As noted in PR https://github.com/Unidata/netcdf-c/pull/2179,
the old code did not allow for reclaiming instances of types,
nor for properly copying them. That PR provided new functions
capable of reclaiming/copying instances of arbitrary types.
However, as noted by Issue https://github.com/Unidata/netcdf-c/issues/2685, using these
most general functions resulted in a significant performance
degradation, even for common cases.
This PR attempts to mitigate the cost of using the general
reclaim/copy functions in two ways.
First, the previous functions operating at the top level by
using ncid and typeid arguments. These functions were augmented
with equivalent versions that used the netcdf-c library internal
data structures to allow direct access to needed information.
These new functions are used internally to the library.
The second mitigation involves optimizing the internal functions
by providing early tests for common cases. This avoids
unnecessary recursive function calls.
The overall result is a significant improvement in speed by a
factor of roughly twenty -- your mileage may vary. These
optimized functions are still not as fast as the original (more
limited) functions, but they are getting close. Additional optimizations are
possible. But the cost is a significant "uglification" of the
code that I deemed a step too far, at least for now.
## Misc. Changes
1. Added a test case to check the proper reclamation/copy of complex types.
2. Found and fixed some places where nc_reclaim/copy should have been used.
3. Replaced, in the netcdf-c library, (almost all) occurrences of nc_reclaim_copy with calls to NC_reclaim/copy. This plus the optimizations is the primary speed-up mechanism.
4. In DAP4, the metadata is held in a substrate in-memory file; this required some changes so that the reclaim/copy code accessed that substrate dispatcher rather than the DAP4 dispatcher.
5. Re-factored and isolated the code that computes if a type is (transitively) variable-sized or not.
6. Clean up the reclamation code in ncgen; adding the use of nc_reclaim exposed some memory problems.
2023-05-21 07:11:25 +08:00
|
|
|
#if 0
|
Fix various problem around VLEN's
re: https://github.com/Unidata/netcdf-c/issues/541
re: https://github.com/Unidata/netcdf-c/issues/1208
re: https://github.com/Unidata/netcdf-c/issues/2078
re: https://github.com/Unidata/netcdf-c/issues/2041
re: https://github.com/Unidata/netcdf-c/issues/2143
For a long time, there have been known problems with the
management of complex types containing VLENs. This also
involves the string type because it is stored as a VLEN of
chars.
This PR (mostly) fixes this problem. But note that it adds new
functions to netcdf.h (see below) and this may require bumping
the .so number. These new functions can be removed, if desired,
in favor of functions in netcdf_aux.h, but netcdf.h seems the
better place for them because they are intended as alternatives
to the nc_free_vlen and nc_free_string functions already in
netcdf.h.
The term complex type refers to any type that directly or
transitively references a VLEN type. So an array of VLENS, a
compound with a VLEN field, and so on.
In order to properly handle instances of these complex types, it
is necessary to have function that can recursively walk
instances of such types to perform various actions on them. The
term "deep" is also used to mean recursive.
At the moment, the two operations needed by the netcdf library are:
* free'ing an instance of the complex type
* copying an instance of the complex type.
The current library does only shallow free and shallow copy of
complex types. This means that only the top level is properly
free'd or copied, but deep internal blocks in the instance are
not touched.
Note that the term "vector" will be used to mean a contiguous (in
memory) sequence of instances of some type. Given an array with,
say, dimensions 2 X 3 X 4, this will be stored in memory as a
vector of length 2*3*4=24 instances.
The use cases are primarily these.
## nc_get_vars
Suppose one is reading a vector of instances using nc_get_vars
(or nc_get_vara or nc_get_var, etc.). These functions will
return the vector in the top-level memory provided. All
interior blocks (form nested VLEN or strings) will have been
dynamically allocated.
After using this vector of instances, it is necessary to free
(aka reclaim) the dynamically allocated memory, otherwise a
memory leak occurs. So, the recursive reclaim function is used
to walk the returned instance vector and do a deep reclaim of
the data.
Currently functions are defined in netcdf.h that are supposed to
handle this: nc_free_vlen(), nc_free_vlens(), and
nc_free_string(). Unfortunately, these functions only do a
shallow free, so deeply nested instances are not properly
handled by them.
Note that internally, the provided data is immediately written so
there is no need to copy it. But the caller may need to reclaim the
data it passed into the function.
## nc_put_att
Suppose one is writing a vector of instances as the data of an attribute
using, say, nc_put_att.
Internally, the incoming attribute data must be copied and stored
so that changes/reclamation of the input data will not affect
the attribute.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. As a result, one sees effects such as described
in Github Issue https://github.com/Unidata/netcdf-c/issues/2143.
Also, after defining the attribute, it may be necessary for the user
to free the data that was provided as input to nc_put_att().
## nc_get_att
Suppose one is reading a vector of instances as the data of an attribute
using, say, nc_get_att.
Internally, the existing attribute data must be copied and returned
to the caller, and the caller is responsible for reclaiming
the returned data.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. So this can lead to memory leaks and errors
because the deep data is shared between the library and the user.
# Solution
The solution is to build properly recursive reclaim and copy
functions and use those as needed.
These recursive functions are defined in libdispatch/dinstance.c
and their signatures are defined in include/netcdf.h.
For back compatibility, corresponding "ncaux_XXX" functions
are defined in include/netcdf_aux.h.
````
int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy);
int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp);
````
There are two variants. The first two, nc_reclaim_data() and
nc_copy_data(), assume the top-level vector is managed by the
caller. For reclaim, this is so the user can use, for example, a
statically allocated vector. For copy, it assumes the user
provides the space into which the copy is stored.
The second two, nc_reclaim_data_all() and
nc_copy_data_all(), allows the functions to manage the
top-level. So for nc_reclaim_data_all, the top level is
assumed to be dynamically allocated and will be free'd by
nc_reclaim_data_all(). The nc_copy_data_all() function
will allocate the top level and return a pointer to it to the
user. The user can later pass that pointer to
nc_reclaim_data_all() to reclaim the instance(s).
# Internal Changes
The netcdf-c library internals are changed to use the proper
reclaim and copy functions. It turns out that the places where
these functions are needed is quite pervasive in the netcdf-c
library code. Using these functions also allows some
simplification of the code since the stdata and vldata fields of
NC_ATT_INFO are no longer needed. Currently this is commented
out using the SEPDATA \#define macro. When any bugs are largely
fixed, all this code will be removed.
# Known Bugs
1. There is still one known failure that has not been solved.
All the failures revolve around some variant of this .cdl file.
The proximate cause of failure is the use of a VLEN FillValue.
````
netcdf x {
types:
float(*) row_of_floats ;
dimensions:
m = 5 ;
variables:
row_of_floats ragged_array(m) ;
row_of_floats ragged_array:_FillValue = {-999} ;
data:
ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32},
{40, 41}, _ ;
}
````
When a solution is found, I will either add it to this PR or post a new PR.
# Related Changes
* Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data.
* Remove the --enable-unfixed-memory-leaks option.
* Remove the NC_VLENS_NOTEST code that suppresses some vlen tests.
* Document this change in docs/internal.md
* Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh.
* Mark types as fixed size or not (transitively) to optimize the reclaim
and copy functions.
# Misc. Changes
* Make Doxygen process libdispatch/daux.c
* Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
|
|
|
#ifdef ENABLE_DAP4
|
Improve performance of the nc_reclaim_data and nc_copy_data functions.
re: Issue https://github.com/Unidata/netcdf-c/issues/2685
re: PR https://github.com/Unidata/netcdf-c/pull/2179
As noted in PR https://github.com/Unidata/netcdf-c/pull/2179,
the old code did not allow for reclaiming instances of types,
nor for properly copying them. That PR provided new functions
capable of reclaiming/copying instances of arbitrary types.
However, as noted by Issue https://github.com/Unidata/netcdf-c/issues/2685, using these
most general functions resulted in a significant performance
degradation, even for common cases.
This PR attempts to mitigate the cost of using the general
reclaim/copy functions in two ways.
First, the previous functions operating at the top level by
using ncid and typeid arguments. These functions were augmented
with equivalent versions that used the netcdf-c library internal
data structures to allow direct access to needed information.
These new functions are used internally to the library.
The second mitigation involves optimizing the internal functions
by providing early tests for common cases. This avoids
unnecessary recursive function calls.
The overall result is a significant improvement in speed by a
factor of roughly twenty -- your mileage may vary. These
optimized functions are still not as fast as the original (more
limited) functions, but they are getting close. Additional optimizations are
possible. But the cost is a significant "uglification" of the
code that I deemed a step too far, at least for now.
## Misc. Changes
1. Added a test case to check the proper reclamation/copy of complex types.
2. Found and fixed some places where nc_reclaim/copy should have been used.
3. Replaced, in the netcdf-c library, (almost all) occurrences of nc_reclaim_copy with calls to NC_reclaim/copy. This plus the optimizations is the primary speed-up mechanism.
4. In DAP4, the metadata is held in a substrate in-memory file; this required some changes so that the reclaim/copy code accessed that substrate dispatcher rather than the DAP4 dispatcher.
5. Re-factored and isolated the code that computes if a type is (transitively) variable-sized or not.
6. Clean up the reclamation code in ncgen; adding the use of nc_reclaim exposed some memory problems.
2023-05-21 07:11:25 +08:00
|
|
|
EXTERNL NC* NCD4_get_substrate_nc(int ncid);
|
|
|
|
#endif
|
Fix various problem around VLEN's
re: https://github.com/Unidata/netcdf-c/issues/541
re: https://github.com/Unidata/netcdf-c/issues/1208
re: https://github.com/Unidata/netcdf-c/issues/2078
re: https://github.com/Unidata/netcdf-c/issues/2041
re: https://github.com/Unidata/netcdf-c/issues/2143
For a long time, there have been known problems with the
management of complex types containing VLENs. This also
involves the string type because it is stored as a VLEN of
chars.
This PR (mostly) fixes this problem. But note that it adds new
functions to netcdf.h (see below) and this may require bumping
the .so number. These new functions can be removed, if desired,
in favor of functions in netcdf_aux.h, but netcdf.h seems the
better place for them because they are intended as alternatives
to the nc_free_vlen and nc_free_string functions already in
netcdf.h.
The term complex type refers to any type that directly or
transitively references a VLEN type. So an array of VLENS, a
compound with a VLEN field, and so on.
In order to properly handle instances of these complex types, it
is necessary to have function that can recursively walk
instances of such types to perform various actions on them. The
term "deep" is also used to mean recursive.
At the moment, the two operations needed by the netcdf library are:
* free'ing an instance of the complex type
* copying an instance of the complex type.
The current library does only shallow free and shallow copy of
complex types. This means that only the top level is properly
free'd or copied, but deep internal blocks in the instance are
not touched.
Note that the term "vector" will be used to mean a contiguous (in
memory) sequence of instances of some type. Given an array with,
say, dimensions 2 X 3 X 4, this will be stored in memory as a
vector of length 2*3*4=24 instances.
The use cases are primarily these.
## nc_get_vars
Suppose one is reading a vector of instances using nc_get_vars
(or nc_get_vara or nc_get_var, etc.). These functions will
return the vector in the top-level memory provided. All
interior blocks (form nested VLEN or strings) will have been
dynamically allocated.
After using this vector of instances, it is necessary to free
(aka reclaim) the dynamically allocated memory, otherwise a
memory leak occurs. So, the recursive reclaim function is used
to walk the returned instance vector and do a deep reclaim of
the data.
Currently functions are defined in netcdf.h that are supposed to
handle this: nc_free_vlen(), nc_free_vlens(), and
nc_free_string(). Unfortunately, these functions only do a
shallow free, so deeply nested instances are not properly
handled by them.
Note that internally, the provided data is immediately written so
there is no need to copy it. But the caller may need to reclaim the
data it passed into the function.
## nc_put_att
Suppose one is writing a vector of instances as the data of an attribute
using, say, nc_put_att.
Internally, the incoming attribute data must be copied and stored
so that changes/reclamation of the input data will not affect
the attribute.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. As a result, one sees effects such as described
in Github Issue https://github.com/Unidata/netcdf-c/issues/2143.
Also, after defining the attribute, it may be necessary for the user
to free the data that was provided as input to nc_put_att().
## nc_get_att
Suppose one is reading a vector of instances as the data of an attribute
using, say, nc_get_att.
Internally, the existing attribute data must be copied and returned
to the caller, and the caller is responsible for reclaiming
the returned data.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. So this can lead to memory leaks and errors
because the deep data is shared between the library and the user.
# Solution
The solution is to build properly recursive reclaim and copy
functions and use those as needed.
These recursive functions are defined in libdispatch/dinstance.c
and their signatures are defined in include/netcdf.h.
For back compatibility, corresponding "ncaux_XXX" functions
are defined in include/netcdf_aux.h.
````
int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy);
int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp);
````
There are two variants. The first two, nc_reclaim_data() and
nc_copy_data(), assume the top-level vector is managed by the
caller. For reclaim, this is so the user can use, for example, a
statically allocated vector. For copy, it assumes the user
provides the space into which the copy is stored.
The second two, nc_reclaim_data_all() and
nc_copy_data_all(), allows the functions to manage the
top-level. So for nc_reclaim_data_all, the top level is
assumed to be dynamically allocated and will be free'd by
nc_reclaim_data_all(). The nc_copy_data_all() function
will allocate the top level and return a pointer to it to the
user. The user can later pass that pointer to
nc_reclaim_data_all() to reclaim the instance(s).
# Internal Changes
The netcdf-c library internals are changed to use the proper
reclaim and copy functions. It turns out that the places where
these functions are needed is quite pervasive in the netcdf-c
library code. Using these functions also allows some
simplification of the code since the stdata and vldata fields of
NC_ATT_INFO are no longer needed. Currently this is commented
out using the SEPDATA \#define macro. When any bugs are largely
fixed, all this code will be removed.
# Known Bugs
1. There is still one known failure that has not been solved.
All the failures revolve around some variant of this .cdl file.
The proximate cause of failure is the use of a VLEN FillValue.
````
netcdf x {
types:
float(*) row_of_floats ;
dimensions:
m = 5 ;
variables:
row_of_floats ragged_array(m) ;
row_of_floats ragged_array:_FillValue = {-999} ;
data:
ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32},
{40, 41}, _ ;
}
````
When a solution is found, I will either add it to this PR or post a new PR.
# Related Changes
* Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data.
* Remove the --enable-unfixed-memory-leaks option.
* Remove the NC_VLENS_NOTEST code that suppresses some vlen tests.
* Document this change in docs/internal.md
* Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh.
* Mark types as fixed size or not (transitively) to optimize the reclaim
and copy functions.
# Misc. Changes
* Make Doxygen process libdispatch/daux.c
* Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
|
|
|
#endif
|
|
|
|
|
2017-12-04 06:37:56 +08:00
|
|
|
/* The sizes of types may vary from platform to platform, but within
|
|
|
|
* netCDF files, type sizes are fixed. */
|
2017-12-05 03:21:14 +08:00
|
|
|
#define NC_CHAR_LEN sizeof(char) /**< @internal Size of char. */
|
|
|
|
#define NC_STRING_LEN sizeof(char *) /**< @internal Size of char *. */
|
|
|
|
#define NC_BYTE_LEN 1 /**< @internal Size of byte. */
|
|
|
|
#define NC_SHORT_LEN 2 /**< @internal Size of short. */
|
|
|
|
#define NC_INT_LEN 4 /**< @internal Size of int. */
|
|
|
|
#define NC_FLOAT_LEN 4 /**< @internal Size of float. */
|
|
|
|
#define NC_DOUBLE_LEN 8 /**< @internal Size of double. */
|
|
|
|
#define NC_INT64_LEN 8 /**< @internal Size of int64. */
|
2017-12-04 06:37:56 +08:00
|
|
|
|
This PR adds EXPERIMENTAL support for accessing data in the
cloud using a variant of the Zarr protocol and storage
format. This enhancement is generically referred to as "NCZarr".
The data model supported by NCZarr is netcdf-4 minus the user-defined
types and the String type. In this sense it is similar to the CDF-5
data model.
More detailed information about enabling and using NCZarr is
described in the document NUG/nczarr.md and in a
[Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in).
WARNING: this code has had limited testing, so do use this version
for production work. Also, performance improvements are ongoing.
Note especially the following platform matrix of successful tests:
Platform | Build System | S3 support
------------------------------------
Linux+gcc | Automake | yes
Linux+gcc | CMake | yes
Visual Studio | CMake | no
Additionally, and as a consequence of the addition of NCZarr,
major changes have been made to the Filter API. NOTE: NCZarr
does not yet support filters, but these changes are enablers for
that support in the future. Note that it is possible
(probable?) that there will be some accidental reversions if the
changes here did not correctly mimic the existing filter testing.
In any case, previously filter ids and parameters were of type
unsigned int. In order to support the more general zarr filter
model, this was all converted to char*. The old HDF5-specific,
unsigned int operations are still supported but they are
wrappers around the new, char* based nc_filterx_XXX functions.
This entailed at least the following changes:
1. Added the files libdispatch/dfilterx.c and include/ncfilter.h
2. Some filterx utilities have been moved to libdispatch/daux.c
3. A new entry, "filter_actions" was added to the NCDispatch table
and the version bumped.
4. An overly complex set of structs was created to support funnelling
all of the filterx operations thru a single dispatch
"filter_actions" entry.
5. Move common code to from libhdf5 to libsrc4 so that it is accessible
to nczarr.
Changes directly related to Zarr:
1. Modified CMakeList.txt and configure.ac to support both C and C++
-- this is in support of S3 support via the awd-sdk libraries.
2. Define a size64_t type to support nczarr.
3. More reworking of libdispatch/dinfermodel.c to
support zarr and to regularize the structure of the fragments
section of a URL.
Changes not directly related to Zarr:
1. Make client-side filter registration be conditional, with default off.
2. Hack include/nc4internal.h to make some flags added by Ed be unique:
e.g. NC_CREAT, NC_INDEF, etc.
3. cleanup include/nchttp.h and libdispatch/dhttp.c.
4. Misc. changes to support compiling under Visual Studio including:
* Better testing under windows for dirent.h and opendir and closedir.
5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags
and to centralize error reporting.
6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them.
7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible.
Changes Left TO-DO:
1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
|
|
|
/** @internal Names of atomic types. */
|
|
|
|
const char* nc4_atomic_name[NUM_ATOMIC_TYPES] = {"none", "byte", "char",
|
|
|
|
"short", "int", "float",
|
|
|
|
"double", "ubyte",
|
|
|
|
"ushort", "uint",
|
|
|
|
"int64", "uint64", "string"};
|
|
|
|
static const int nc4_atomic_size[NUM_ATOMIC_TYPES] = {0, NC_BYTE_LEN, NC_CHAR_LEN, NC_SHORT_LEN,
|
|
|
|
NC_INT_LEN, NC_FLOAT_LEN, NC_DOUBLE_LEN,
|
|
|
|
NC_BYTE_LEN, NC_SHORT_LEN, NC_INT_LEN, NC_INT64_LEN,
|
|
|
|
NC_INT64_LEN, NC_STRING_LEN};
|
|
|
|
|
2017-12-04 06:37:56 +08:00
|
|
|
/**
|
|
|
|
* @internal Find all user-defined types for a location. This finds
|
|
|
|
* all user-defined types in a group.
|
|
|
|
*
|
|
|
|
* @param ncid File and group ID.
|
|
|
|
* @param ntypes Pointer that gets the number of user-defined
|
|
|
|
* types. Ignored if NULL
|
|
|
|
* @param typeids Array that gets the typeids. Ignored if NULL.
|
|
|
|
*
|
|
|
|
* @return ::NC_NOERR No error.
|
|
|
|
* @return ::NC_EBADID Bad ncid.
|
|
|
|
* @author Ed Hartnett
|
|
|
|
*/
|
2019-02-19 20:56:22 +08:00
|
|
|
int
|
2010-06-03 21:24:43 +08:00
|
|
|
NC4_inq_typeids(int ncid, int *ntypes, int *typeids)
|
|
|
|
{
|
2019-02-19 20:56:22 +08:00
|
|
|
NC_GRP_INFO_T *grp;
|
|
|
|
NC_FILE_INFO_T *h5;
|
|
|
|
NC_TYPE_INFO_T *type;
|
|
|
|
int num = 0;
|
|
|
|
int retval;
|
|
|
|
|
|
|
|
LOG((2, "nc_inq_typeids: ncid 0x%x", ncid));
|
|
|
|
|
|
|
|
/* Find info for this file and group, and set pointer to each. */
|
|
|
|
if ((retval = nc4_find_grp_h5(ncid, &grp, &h5)))
|
|
|
|
return retval;
|
|
|
|
assert(h5 && grp);
|
|
|
|
|
|
|
|
/* Count types. */
|
|
|
|
if (grp->type) {
|
|
|
|
int i;
|
|
|
|
for(i=0;i<ncindexsize(grp->type);i++)
|
|
|
|
{
|
|
|
|
if((type = (NC_TYPE_INFO_T*)ncindexith(grp->type,i)) == NULL) continue;
|
|
|
|
if (typeids)
|
|
|
|
typeids[num] = type->hdr.id;
|
|
|
|
num++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Give the count to the user. */
|
|
|
|
if (ntypes)
|
|
|
|
*ntypes = num;
|
|
|
|
|
|
|
|
return NC_NOERR;
|
2010-06-03 21:24:43 +08:00
|
|
|
}
|
|
|
|
|
2017-12-04 06:37:56 +08:00
|
|
|
/**
|
This PR adds EXPERIMENTAL support for accessing data in the
cloud using a variant of the Zarr protocol and storage
format. This enhancement is generically referred to as "NCZarr".
The data model supported by NCZarr is netcdf-4 minus the user-defined
types and the String type. In this sense it is similar to the CDF-5
data model.
More detailed information about enabling and using NCZarr is
described in the document NUG/nczarr.md and in a
[Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in).
WARNING: this code has had limited testing, so do use this version
for production work. Also, performance improvements are ongoing.
Note especially the following platform matrix of successful tests:
Platform | Build System | S3 support
------------------------------------
Linux+gcc | Automake | yes
Linux+gcc | CMake | yes
Visual Studio | CMake | no
Additionally, and as a consequence of the addition of NCZarr,
major changes have been made to the Filter API. NOTE: NCZarr
does not yet support filters, but these changes are enablers for
that support in the future. Note that it is possible
(probable?) that there will be some accidental reversions if the
changes here did not correctly mimic the existing filter testing.
In any case, previously filter ids and parameters were of type
unsigned int. In order to support the more general zarr filter
model, this was all converted to char*. The old HDF5-specific,
unsigned int operations are still supported but they are
wrappers around the new, char* based nc_filterx_XXX functions.
This entailed at least the following changes:
1. Added the files libdispatch/dfilterx.c and include/ncfilter.h
2. Some filterx utilities have been moved to libdispatch/daux.c
3. A new entry, "filter_actions" was added to the NCDispatch table
and the version bumped.
4. An overly complex set of structs was created to support funnelling
all of the filterx operations thru a single dispatch
"filter_actions" entry.
5. Move common code to from libhdf5 to libsrc4 so that it is accessible
to nczarr.
Changes directly related to Zarr:
1. Modified CMakeList.txt and configure.ac to support both C and C++
-- this is in support of S3 support via the awd-sdk libraries.
2. Define a size64_t type to support nczarr.
3. More reworking of libdispatch/dinfermodel.c to
support zarr and to regularize the structure of the fragments
section of a URL.
Changes not directly related to Zarr:
1. Make client-side filter registration be conditional, with default off.
2. Hack include/nc4internal.h to make some flags added by Ed be unique:
e.g. NC_CREAT, NC_INDEF, etc.
3. cleanup include/nchttp.h and libdispatch/dhttp.c.
4. Misc. changes to support compiling under Visual Studio including:
* Better testing under windows for dirent.h and opendir and closedir.
5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags
and to centralize error reporting.
6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them.
7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible.
Changes Left TO-DO:
1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
|
|
|
* @internal Get the name and size of an atomic type. For strings, 1 is
|
|
|
|
* returned.
|
|
|
|
*
|
|
|
|
* @param typeid1 Type ID.
|
|
|
|
* @param name Gets the name of the type.
|
|
|
|
* @param size Gets the size of one element of the type in bytes.
|
|
|
|
*
|
|
|
|
* @return ::NC_NOERR No error.
|
|
|
|
* @return ::NC_EBADID Bad ncid.
|
|
|
|
* @return ::NC_EBADTYPE Type not found.
|
|
|
|
* @author Dennis Heimbigner
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
NC4_inq_atomic_type(nc_type typeid1, char *name, size_t *size)
|
|
|
|
{
|
|
|
|
LOG((2, "nc_inq_atomic_type: typeid %d", typeid1));
|
|
|
|
|
|
|
|
if (typeid1 >= NUM_ATOMIC_TYPES)
|
|
|
|
return NC_EBADTYPE;
|
|
|
|
if (name)
|
|
|
|
strcpy(name, nc4_atomic_name[typeid1]);
|
|
|
|
if (size)
|
|
|
|
*size = nc4_atomic_size[typeid1];
|
|
|
|
return NC_NOERR;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @internal Get the id and size of an atomic type by name.
|
|
|
|
*
|
|
|
|
* @param name [in] the name of the type.
|
2022-11-16 11:29:21 +08:00
|
|
|
* @param idp [out] the type index of the type.
|
|
|
|
* @param sizep [out] the size of one element of the type in bytes.
|
This PR adds EXPERIMENTAL support for accessing data in the
cloud using a variant of the Zarr protocol and storage
format. This enhancement is generically referred to as "NCZarr".
The data model supported by NCZarr is netcdf-4 minus the user-defined
types and the String type. In this sense it is similar to the CDF-5
data model.
More detailed information about enabling and using NCZarr is
described in the document NUG/nczarr.md and in a
[Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in).
WARNING: this code has had limited testing, so do use this version
for production work. Also, performance improvements are ongoing.
Note especially the following platform matrix of successful tests:
Platform | Build System | S3 support
------------------------------------
Linux+gcc | Automake | yes
Linux+gcc | CMake | yes
Visual Studio | CMake | no
Additionally, and as a consequence of the addition of NCZarr,
major changes have been made to the Filter API. NOTE: NCZarr
does not yet support filters, but these changes are enablers for
that support in the future. Note that it is possible
(probable?) that there will be some accidental reversions if the
changes here did not correctly mimic the existing filter testing.
In any case, previously filter ids and parameters were of type
unsigned int. In order to support the more general zarr filter
model, this was all converted to char*. The old HDF5-specific,
unsigned int operations are still supported but they are
wrappers around the new, char* based nc_filterx_XXX functions.
This entailed at least the following changes:
1. Added the files libdispatch/dfilterx.c and include/ncfilter.h
2. Some filterx utilities have been moved to libdispatch/daux.c
3. A new entry, "filter_actions" was added to the NCDispatch table
and the version bumped.
4. An overly complex set of structs was created to support funnelling
all of the filterx operations thru a single dispatch
"filter_actions" entry.
5. Move common code to from libhdf5 to libsrc4 so that it is accessible
to nczarr.
Changes directly related to Zarr:
1. Modified CMakeList.txt and configure.ac to support both C and C++
-- this is in support of S3 support via the awd-sdk libraries.
2. Define a size64_t type to support nczarr.
3. More reworking of libdispatch/dinfermodel.c to
support zarr and to regularize the structure of the fragments
section of a URL.
Changes not directly related to Zarr:
1. Make client-side filter registration be conditional, with default off.
2. Hack include/nc4internal.h to make some flags added by Ed be unique:
e.g. NC_CREAT, NC_INDEF, etc.
3. cleanup include/nchttp.h and libdispatch/dhttp.c.
4. Misc. changes to support compiling under Visual Studio including:
* Better testing under windows for dirent.h and opendir and closedir.
5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags
and to centralize error reporting.
6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them.
7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible.
Changes Left TO-DO:
1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
|
|
|
*
|
|
|
|
* @return ::NC_NOERR No error.
|
|
|
|
* @return ::NC_EBADID Bad ncid.
|
|
|
|
* @return ::NC_EBADTYPE Type not found.
|
|
|
|
* @author Dennis Heimbigner
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
NC4_lookup_atomic_type(const char *name, nc_type* idp, size_t *sizep)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
LOG((2, "nc_lookup_atomic_type: name %s ", name));
|
|
|
|
|
|
|
|
if (name == NULL || strlen(name) == 0)
|
|
|
|
return NC_EBADTYPE;
|
|
|
|
for(i=0;i<NUM_ATOMIC_TYPES;i++) {
|
|
|
|
if(strcasecmp(name,nc4_atomic_name[i])==0) {
|
|
|
|
if(idp) *idp = i;
|
|
|
|
if(sizep) *sizep = nc4_atomic_size[i];
|
|
|
|
return NC_NOERR;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return NC_EBADTYPE;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @internal Get the name and size of a type.
|
|
|
|
* For VLEN the base type len is returned.
|
2017-12-04 06:37:56 +08:00
|
|
|
*
|
|
|
|
* @param ncid File and group ID.
|
|
|
|
* @param typeid1 Type ID.
|
|
|
|
* @param name Gets the name of the type.
|
|
|
|
* @param size Gets the size of one element of the type in bytes.
|
|
|
|
*
|
|
|
|
* @return ::NC_NOERR No error.
|
|
|
|
* @return ::NC_EBADID Bad ncid.
|
|
|
|
* @return ::NC_EBADTYPE Type not found.
|
|
|
|
* @author Ed Hartnett
|
2019-02-19 20:56:22 +08:00
|
|
|
*/
|
2010-06-03 21:24:43 +08:00
|
|
|
int
|
2012-12-14 06:09:41 +08:00
|
|
|
NC4_inq_type(int ncid, nc_type typeid1, char *name, size_t *size)
|
2010-06-03 21:24:43 +08:00
|
|
|
{
|
2019-02-19 20:56:22 +08:00
|
|
|
NC_GRP_INFO_T *grp;
|
|
|
|
NC_TYPE_INFO_T *type;
|
|
|
|
|
|
|
|
int retval;
|
|
|
|
|
|
|
|
LOG((2, "nc_inq_type: ncid 0x%x typeid %d", ncid, typeid1));
|
|
|
|
|
|
|
|
/* If this is an atomic type, the answer is easy. */
|
|
|
|
if (typeid1 < NUM_ATOMIC_TYPES)
|
|
|
|
{
|
|
|
|
if (name)
|
|
|
|
strcpy(name, nc4_atomic_name[typeid1]);
|
|
|
|
if (size)
|
This PR adds EXPERIMENTAL support for accessing data in the
cloud using a variant of the Zarr protocol and storage
format. This enhancement is generically referred to as "NCZarr".
The data model supported by NCZarr is netcdf-4 minus the user-defined
types and the String type. In this sense it is similar to the CDF-5
data model.
More detailed information about enabling and using NCZarr is
described in the document NUG/nczarr.md and in a
[Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in).
WARNING: this code has had limited testing, so do use this version
for production work. Also, performance improvements are ongoing.
Note especially the following platform matrix of successful tests:
Platform | Build System | S3 support
------------------------------------
Linux+gcc | Automake | yes
Linux+gcc | CMake | yes
Visual Studio | CMake | no
Additionally, and as a consequence of the addition of NCZarr,
major changes have been made to the Filter API. NOTE: NCZarr
does not yet support filters, but these changes are enablers for
that support in the future. Note that it is possible
(probable?) that there will be some accidental reversions if the
changes here did not correctly mimic the existing filter testing.
In any case, previously filter ids and parameters were of type
unsigned int. In order to support the more general zarr filter
model, this was all converted to char*. The old HDF5-specific,
unsigned int operations are still supported but they are
wrappers around the new, char* based nc_filterx_XXX functions.
This entailed at least the following changes:
1. Added the files libdispatch/dfilterx.c and include/ncfilter.h
2. Some filterx utilities have been moved to libdispatch/daux.c
3. A new entry, "filter_actions" was added to the NCDispatch table
and the version bumped.
4. An overly complex set of structs was created to support funnelling
all of the filterx operations thru a single dispatch
"filter_actions" entry.
5. Move common code to from libhdf5 to libsrc4 so that it is accessible
to nczarr.
Changes directly related to Zarr:
1. Modified CMakeList.txt and configure.ac to support both C and C++
-- this is in support of S3 support via the awd-sdk libraries.
2. Define a size64_t type to support nczarr.
3. More reworking of libdispatch/dinfermodel.c to
support zarr and to regularize the structure of the fragments
section of a URL.
Changes not directly related to Zarr:
1. Make client-side filter registration be conditional, with default off.
2. Hack include/nc4internal.h to make some flags added by Ed be unique:
e.g. NC_CREAT, NC_INDEF, etc.
3. cleanup include/nchttp.h and libdispatch/dhttp.c.
4. Misc. changes to support compiling under Visual Studio including:
* Better testing under windows for dirent.h and opendir and closedir.
5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags
and to centralize error reporting.
6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them.
7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible.
Changes Left TO-DO:
1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
|
|
|
*size = nc4_atomic_size[typeid1];
|
2019-02-19 20:56:22 +08:00
|
|
|
return NC_NOERR;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Not an atomic type - so find group. */
|
|
|
|
if ((retval = nc4_find_nc4_grp(ncid, &grp)))
|
|
|
|
return retval;
|
|
|
|
|
|
|
|
/* Find this type. */
|
2023-11-27 19:36:03 +08:00
|
|
|
if (!(type = nclistget(grp->nc4_info->alltypes, (size_t)typeid1)))
|
2019-02-19 20:56:22 +08:00
|
|
|
return NC_EBADTYPE;
|
|
|
|
|
|
|
|
if (name)
|
|
|
|
strcpy(name, type->hdr.name);
|
|
|
|
|
|
|
|
if (size)
|
|
|
|
{
|
|
|
|
if (type->nc_type_class == NC_VLEN)
|
|
|
|
*size = sizeof(nc_vlen_t);
|
|
|
|
else if (type->nc_type_class == NC_STRING)
|
This PR adds EXPERIMENTAL support for accessing data in the
cloud using a variant of the Zarr protocol and storage
format. This enhancement is generically referred to as "NCZarr".
The data model supported by NCZarr is netcdf-4 minus the user-defined
types and the String type. In this sense it is similar to the CDF-5
data model.
More detailed information about enabling and using NCZarr is
described in the document NUG/nczarr.md and in a
[Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in).
WARNING: this code has had limited testing, so do use this version
for production work. Also, performance improvements are ongoing.
Note especially the following platform matrix of successful tests:
Platform | Build System | S3 support
------------------------------------
Linux+gcc | Automake | yes
Linux+gcc | CMake | yes
Visual Studio | CMake | no
Additionally, and as a consequence of the addition of NCZarr,
major changes have been made to the Filter API. NOTE: NCZarr
does not yet support filters, but these changes are enablers for
that support in the future. Note that it is possible
(probable?) that there will be some accidental reversions if the
changes here did not correctly mimic the existing filter testing.
In any case, previously filter ids and parameters were of type
unsigned int. In order to support the more general zarr filter
model, this was all converted to char*. The old HDF5-specific,
unsigned int operations are still supported but they are
wrappers around the new, char* based nc_filterx_XXX functions.
This entailed at least the following changes:
1. Added the files libdispatch/dfilterx.c and include/ncfilter.h
2. Some filterx utilities have been moved to libdispatch/daux.c
3. A new entry, "filter_actions" was added to the NCDispatch table
and the version bumped.
4. An overly complex set of structs was created to support funnelling
all of the filterx operations thru a single dispatch
"filter_actions" entry.
5. Move common code to from libhdf5 to libsrc4 so that it is accessible
to nczarr.
Changes directly related to Zarr:
1. Modified CMakeList.txt and configure.ac to support both C and C++
-- this is in support of S3 support via the awd-sdk libraries.
2. Define a size64_t type to support nczarr.
3. More reworking of libdispatch/dinfermodel.c to
support zarr and to regularize the structure of the fragments
section of a URL.
Changes not directly related to Zarr:
1. Make client-side filter registration be conditional, with default off.
2. Hack include/nc4internal.h to make some flags added by Ed be unique:
e.g. NC_CREAT, NC_INDEF, etc.
3. cleanup include/nchttp.h and libdispatch/dhttp.c.
4. Misc. changes to support compiling under Visual Studio including:
* Better testing under windows for dirent.h and opendir and closedir.
5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags
and to centralize error reporting.
6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them.
7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible.
Changes Left TO-DO:
1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
|
|
|
*size = NC_STRING_LEN;
|
2019-02-19 20:56:22 +08:00
|
|
|
else
|
|
|
|
*size = type->size;
|
|
|
|
}
|
|
|
|
|
|
|
|
return NC_NOERR;
|
2010-06-03 21:24:43 +08:00
|
|
|
}
|
|
|
|
|
2017-12-04 06:37:56 +08:00
|
|
|
/**
|
|
|
|
* @internal Find info about any user defined type.
|
|
|
|
*
|
|
|
|
* @param ncid File and group ID.
|
|
|
|
* @param typeid1 Type ID.
|
|
|
|
* @param name Gets name of the type.
|
|
|
|
* @param size Gets size in bytes of one element of type.
|
|
|
|
* @param base_nc_typep Gets the base nc_type.
|
|
|
|
* @param nfieldsp Gets the number of fields.
|
|
|
|
* @param classp Gets the type class (NC_COMPOUND, NC_ENUM, NC_VLEN).
|
|
|
|
*
|
|
|
|
* @return ::NC_NOERR No error.
|
|
|
|
* @return ::NC_EBADID Bad ncid.
|
|
|
|
* @return ::NC_EBADTYPE Type not found.
|
|
|
|
* @author Ed Hartnett
|
2019-02-19 20:56:22 +08:00
|
|
|
*/
|
2010-06-03 21:24:43 +08:00
|
|
|
int
|
2019-02-19 20:56:22 +08:00
|
|
|
NC4_inq_user_type(int ncid, nc_type typeid1, char *name, size_t *size,
|
|
|
|
nc_type *base_nc_typep, size_t *nfieldsp, int *classp)
|
2010-06-03 21:24:43 +08:00
|
|
|
{
|
2019-02-19 20:56:22 +08:00
|
|
|
NC_GRP_INFO_T *grp;
|
|
|
|
NC_TYPE_INFO_T *type;
|
|
|
|
int retval;
|
|
|
|
|
|
|
|
LOG((2, "nc_inq_user_type: ncid 0x%x typeid %d", ncid, typeid1));
|
|
|
|
|
|
|
|
/* Find group metadata. */
|
|
|
|
if ((retval = nc4_find_nc4_grp(ncid, &grp)))
|
|
|
|
return retval;
|
|
|
|
|
|
|
|
/* Find this type. */
|
2023-11-27 19:36:03 +08:00
|
|
|
if (!(type = nclistget(grp->nc4_info->alltypes, (size_t)typeid1)))
|
2019-02-19 20:56:22 +08:00
|
|
|
return NC_EBADTYPE;
|
|
|
|
|
|
|
|
/* Count the number of fields. */
|
|
|
|
if (nfieldsp)
|
|
|
|
{
|
|
|
|
if (type->nc_type_class == NC_COMPOUND)
|
|
|
|
*nfieldsp = nclistlength(type->u.c.field);
|
|
|
|
else if (type->nc_type_class == NC_ENUM)
|
|
|
|
*nfieldsp = nclistlength(type->u.e.enum_member);
|
|
|
|
else
|
|
|
|
*nfieldsp = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Fill in size and name info, if desired. */
|
|
|
|
if (size)
|
|
|
|
{
|
|
|
|
if (type->nc_type_class == NC_VLEN)
|
|
|
|
*size = sizeof(nc_vlen_t);
|
|
|
|
else if (type->nc_type_class == NC_STRING)
|
This PR adds EXPERIMENTAL support for accessing data in the
cloud using a variant of the Zarr protocol and storage
format. This enhancement is generically referred to as "NCZarr".
The data model supported by NCZarr is netcdf-4 minus the user-defined
types and the String type. In this sense it is similar to the CDF-5
data model.
More detailed information about enabling and using NCZarr is
described in the document NUG/nczarr.md and in a
[Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in).
WARNING: this code has had limited testing, so do use this version
for production work. Also, performance improvements are ongoing.
Note especially the following platform matrix of successful tests:
Platform | Build System | S3 support
------------------------------------
Linux+gcc | Automake | yes
Linux+gcc | CMake | yes
Visual Studio | CMake | no
Additionally, and as a consequence of the addition of NCZarr,
major changes have been made to the Filter API. NOTE: NCZarr
does not yet support filters, but these changes are enablers for
that support in the future. Note that it is possible
(probable?) that there will be some accidental reversions if the
changes here did not correctly mimic the existing filter testing.
In any case, previously filter ids and parameters were of type
unsigned int. In order to support the more general zarr filter
model, this was all converted to char*. The old HDF5-specific,
unsigned int operations are still supported but they are
wrappers around the new, char* based nc_filterx_XXX functions.
This entailed at least the following changes:
1. Added the files libdispatch/dfilterx.c and include/ncfilter.h
2. Some filterx utilities have been moved to libdispatch/daux.c
3. A new entry, "filter_actions" was added to the NCDispatch table
and the version bumped.
4. An overly complex set of structs was created to support funnelling
all of the filterx operations thru a single dispatch
"filter_actions" entry.
5. Move common code to from libhdf5 to libsrc4 so that it is accessible
to nczarr.
Changes directly related to Zarr:
1. Modified CMakeList.txt and configure.ac to support both C and C++
-- this is in support of S3 support via the awd-sdk libraries.
2. Define a size64_t type to support nczarr.
3. More reworking of libdispatch/dinfermodel.c to
support zarr and to regularize the structure of the fragments
section of a URL.
Changes not directly related to Zarr:
1. Make client-side filter registration be conditional, with default off.
2. Hack include/nc4internal.h to make some flags added by Ed be unique:
e.g. NC_CREAT, NC_INDEF, etc.
3. cleanup include/nchttp.h and libdispatch/dhttp.c.
4. Misc. changes to support compiling under Visual Studio including:
* Better testing under windows for dirent.h and opendir and closedir.
5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags
and to centralize error reporting.
6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them.
7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible.
Changes Left TO-DO:
1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
|
|
|
*size = NC_STRING_LEN;
|
2019-02-19 20:56:22 +08:00
|
|
|
else
|
|
|
|
*size = type->size;
|
|
|
|
}
|
|
|
|
if (name)
|
|
|
|
strcpy(name, type->hdr.name);
|
|
|
|
|
|
|
|
/* VLENS and ENUMs have a base type - that is, they type they are
|
|
|
|
* arrays of or enums of. */
|
|
|
|
if (base_nc_typep)
|
|
|
|
{
|
|
|
|
if (type->nc_type_class == NC_ENUM)
|
|
|
|
*base_nc_typep = type->u.e.base_nc_typeid;
|
|
|
|
else if (type->nc_type_class == NC_VLEN)
|
|
|
|
*base_nc_typep = type->u.v.base_nc_typeid;
|
|
|
|
else
|
|
|
|
*base_nc_typep = NC_NAT;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* If the user wants it, tell whether this is a compound, opaque,
|
|
|
|
* vlen, enum, or string class of type. */
|
|
|
|
if (classp)
|
|
|
|
*classp = type->nc_type_class;
|
|
|
|
|
|
|
|
return NC_NOERR;
|
2010-06-03 21:24:43 +08:00
|
|
|
}
|
|
|
|
|
2017-12-04 06:37:56 +08:00
|
|
|
/**
|
|
|
|
* @internal Given the ncid, typeid and fieldid, get info about the
|
|
|
|
* field.
|
|
|
|
*
|
|
|
|
* @param ncid File and group ID.
|
|
|
|
* @param typeid1 Type ID.
|
|
|
|
* @param fieldid Field ID.
|
|
|
|
* @param name Gets name of field.
|
|
|
|
* @param offsetp Gets offset of field.
|
|
|
|
* @param field_typeidp Gets field type ID.
|
|
|
|
* @param ndimsp Gets number of dims for this field.
|
|
|
|
* @param dim_sizesp Gets the dim sizes for this field.
|
|
|
|
*
|
|
|
|
* @return ::NC_NOERR No error.
|
|
|
|
* @return ::NC_EBADID Bad ncid.
|
|
|
|
* @author Ed Hartnett
|
2019-02-19 20:56:22 +08:00
|
|
|
*/
|
2010-06-03 21:24:43 +08:00
|
|
|
int
|
2019-02-19 20:56:22 +08:00
|
|
|
NC4_inq_compound_field(int ncid, nc_type typeid1, int fieldid, char *name,
|
|
|
|
size_t *offsetp, nc_type *field_typeidp, int *ndimsp,
|
|
|
|
int *dim_sizesp)
|
2010-06-03 21:24:43 +08:00
|
|
|
{
|
2019-02-19 20:56:22 +08:00
|
|
|
NC_GRP_INFO_T *grp;
|
|
|
|
NC_TYPE_INFO_T *type;
|
|
|
|
NC_FIELD_INFO_T *field;
|
|
|
|
int d, retval;
|
|
|
|
|
|
|
|
/* Find file metadata. */
|
|
|
|
if ((retval = nc4_find_nc4_grp(ncid, &grp)))
|
|
|
|
return retval;
|
|
|
|
|
|
|
|
/* Find this type. */
|
2023-11-27 19:36:03 +08:00
|
|
|
if (!(type = nclistget(grp->nc4_info->alltypes, (size_t)typeid1)))
|
2019-02-19 20:56:22 +08:00
|
|
|
return NC_EBADTYPE;
|
|
|
|
|
|
|
|
/* Find the field. */
|
2023-11-27 19:36:03 +08:00
|
|
|
if (!(field = nclistget(type->u.c.field, (size_t)fieldid)))
|
2019-02-19 20:56:22 +08:00
|
|
|
return NC_EBADFIELD;
|
|
|
|
|
|
|
|
if (name)
|
|
|
|
strcpy(name, field->hdr.name);
|
|
|
|
if (offsetp)
|
|
|
|
*offsetp = field->offset;
|
|
|
|
if (field_typeidp)
|
|
|
|
*field_typeidp = field->nc_typeid;
|
|
|
|
if (ndimsp)
|
|
|
|
*ndimsp = field->ndims;
|
|
|
|
if (dim_sizesp)
|
|
|
|
for (d = 0; d < field->ndims; d++)
|
|
|
|
dim_sizesp[d] = field->dim_size[d];
|
|
|
|
|
|
|
|
return NC_NOERR;
|
2010-06-03 21:24:43 +08:00
|
|
|
}
|
|
|
|
|
2017-12-04 06:37:56 +08:00
|
|
|
/**
|
|
|
|
* @internal Given the typeid and the name, get the fieldid.
|
|
|
|
*
|
|
|
|
* @param ncid File and group ID.
|
|
|
|
* @param typeid1 Type ID.
|
|
|
|
* @param name Name of field.
|
|
|
|
* @param fieldidp Pointer that gets new field ID.
|
|
|
|
*
|
|
|
|
* @return ::NC_NOERR No error.
|
|
|
|
* @return ::NC_EBADID Bad ncid.
|
|
|
|
* @return ::NC_EBADTYPE Type not found.
|
2017-12-05 03:21:14 +08:00
|
|
|
* @return ::NC_EBADFIELD Field not found.
|
2017-12-04 06:37:56 +08:00
|
|
|
* @author Ed Hartnett
|
2019-02-19 20:56:22 +08:00
|
|
|
*/
|
2010-06-03 21:24:43 +08:00
|
|
|
int
|
2012-12-14 06:09:41 +08:00
|
|
|
NC4_inq_compound_fieldindex(int ncid, nc_type typeid1, const char *name, int *fieldidp)
|
2010-06-03 21:24:43 +08:00
|
|
|
{
|
2019-02-19 20:56:22 +08:00
|
|
|
NC_FILE_INFO_T *h5;
|
|
|
|
NC_TYPE_INFO_T *type;
|
|
|
|
NC_FIELD_INFO_T *field;
|
|
|
|
char norm_name[NC_MAX_NAME + 1];
|
|
|
|
int retval;
|
2023-11-27 19:36:03 +08:00
|
|
|
size_t i;
|
2019-02-19 20:56:22 +08:00
|
|
|
|
|
|
|
LOG((2, "nc_inq_compound_fieldindex: ncid 0x%x typeid %d name %s",
|
|
|
|
ncid, typeid1, name));
|
|
|
|
|
|
|
|
/* Find file metadata. */
|
|
|
|
if ((retval = nc4_find_grp_h5(ncid, NULL, &h5)))
|
|
|
|
return retval;
|
|
|
|
|
|
|
|
/* Find the type. */
|
|
|
|
if ((retval = nc4_find_type(h5, typeid1, &type)))
|
|
|
|
return retval;
|
|
|
|
|
|
|
|
/* Did the user give us a good compound type typeid? */
|
|
|
|
if (!type || type->nc_type_class != NC_COMPOUND)
|
|
|
|
return NC_EBADTYPE;
|
|
|
|
|
|
|
|
/* Normalize name. */
|
|
|
|
if ((retval = nc4_normalize_name(name, norm_name)))
|
|
|
|
return retval;
|
|
|
|
|
|
|
|
/* Find the field with this name. */
|
|
|
|
for (i = 0; i < nclistlength(type->u.c.field); i++)
|
|
|
|
{
|
|
|
|
field = nclistget(type->u.c.field, i);
|
|
|
|
assert(field);
|
|
|
|
if (!strcmp(field->hdr.name, norm_name))
|
|
|
|
break;
|
|
|
|
field = NULL; /* because this is the indicator of not found */
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!field)
|
|
|
|
return NC_EBADFIELD;
|
|
|
|
|
|
|
|
if (fieldidp)
|
|
|
|
*fieldidp = field->hdr.id;
|
|
|
|
return NC_NOERR;
|
2010-06-03 21:24:43 +08:00
|
|
|
}
|
|
|
|
|
2017-12-04 06:37:56 +08:00
|
|
|
/**
|
2022-07-18 04:32:31 +08:00
|
|
|
* @internal Get enum name from enum value. Name size will be <= NC_MAX_NAME.
|
|
|
|
* If the value is not a legitimate enum identifier and the value is zero
|
|
|
|
* (the default HDF5 enum fill value), then return the identifier "_UNDEFINED".
|
2017-12-04 06:37:56 +08:00
|
|
|
*
|
|
|
|
* @param ncid File and group ID.
|
|
|
|
* @param xtype Type ID.
|
|
|
|
* @param value Value of enum.
|
|
|
|
* @param identifier Gets the identifier for this enum value.
|
|
|
|
*
|
|
|
|
* @return ::NC_NOERR No error.
|
|
|
|
* @return ::NC_EBADID Bad ncid.
|
|
|
|
* @return ::NC_EBADTYPE Type not found.
|
2022-07-18 04:32:31 +08:00
|
|
|
* @return ::NC_EINVAL Invalid type data or no matching enum value is found
|
2017-12-04 06:37:56 +08:00
|
|
|
* @author Ed Hartnett
|
2019-02-19 20:56:22 +08:00
|
|
|
*/
|
2010-06-03 21:24:43 +08:00
|
|
|
int
|
|
|
|
NC4_inq_enum_ident(int ncid, nc_type xtype, long long value, char *identifier)
|
|
|
|
{
|
2019-02-19 20:56:22 +08:00
|
|
|
NC_GRP_INFO_T *grp;
|
|
|
|
NC_TYPE_INFO_T *type;
|
|
|
|
NC_ENUM_MEMBER_INFO_T *enum_member;
|
|
|
|
long long ll_val;
|
2023-11-27 19:36:03 +08:00
|
|
|
size_t i;
|
2019-02-19 20:56:22 +08:00
|
|
|
int retval;
|
|
|
|
int found;
|
|
|
|
|
|
|
|
LOG((3, "nc_inq_enum_ident: xtype %d value %d\n", xtype, value));
|
|
|
|
|
|
|
|
/* Find group metadata. */
|
|
|
|
if ((retval = nc4_find_nc4_grp(ncid, &grp)))
|
|
|
|
return retval;
|
|
|
|
|
|
|
|
/* Find this type. */
|
2023-11-27 19:36:03 +08:00
|
|
|
if (!(type = nclistget(grp->nc4_info->alltypes, (size_t)xtype)))
|
2019-02-19 20:56:22 +08:00
|
|
|
return NC_EBADTYPE;
|
|
|
|
|
|
|
|
/* Complain if they are confused about the type. */
|
|
|
|
if (type->nc_type_class != NC_ENUM)
|
|
|
|
return NC_EBADTYPE;
|
|
|
|
|
|
|
|
/* Move to the desired enum member in the list. */
|
|
|
|
for (found = 0, i = 0; i < nclistlength(type->u.e.enum_member); i++)
|
|
|
|
{
|
|
|
|
enum_member = nclistget(type->u.e.enum_member, i);
|
|
|
|
assert(enum_member);
|
|
|
|
switch (type->u.e.base_nc_typeid)
|
|
|
|
{
|
|
|
|
case NC_BYTE:
|
|
|
|
ll_val = *(char *)enum_member->value;
|
|
|
|
break;
|
|
|
|
case NC_UBYTE:
|
|
|
|
ll_val = *(unsigned char *)enum_member->value;
|
|
|
|
break;
|
|
|
|
case NC_SHORT:
|
|
|
|
ll_val = *(short *)enum_member->value;
|
|
|
|
break;
|
|
|
|
case NC_USHORT:
|
|
|
|
ll_val = *(unsigned short *)enum_member->value;
|
|
|
|
break;
|
|
|
|
case NC_INT:
|
|
|
|
ll_val = *(int *)enum_member->value;
|
|
|
|
break;
|
|
|
|
case NC_UINT:
|
|
|
|
ll_val = *(unsigned int *)enum_member->value;
|
|
|
|
break;
|
|
|
|
case NC_INT64:
|
|
|
|
case NC_UINT64:
|
|
|
|
ll_val = *(long long *)enum_member->value;
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
return NC_EINVAL;
|
|
|
|
}
|
|
|
|
LOG((4, "ll_val=%d", ll_val));
|
|
|
|
if (ll_val == value)
|
|
|
|
{
|
|
|
|
if (identifier)
|
|
|
|
strcpy(identifier, enum_member->name);
|
|
|
|
found = 1;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* If we didn't find it, life sucks for us. :-( */
|
2022-07-18 04:32:31 +08:00
|
|
|
if(!found) {
|
|
|
|
if(value == 0) /* Special case for HDF5 default Fill Value*/
|
|
|
|
strcpy(identifier, NC_UNDEFINED_ENUM_IDENT);
|
|
|
|
else
|
|
|
|
return NC_EINVAL;
|
|
|
|
}
|
|
|
|
|
2019-02-19 20:56:22 +08:00
|
|
|
return NC_NOERR;
|
2010-06-03 21:24:43 +08:00
|
|
|
}
|
|
|
|
|
2017-12-04 06:37:56 +08:00
|
|
|
/**
|
|
|
|
* @internal Get information about an enum member: an identifier and
|
|
|
|
* value. Identifier size will be <= NC_MAX_NAME.
|
|
|
|
*
|
|
|
|
* @param ncid File and group ID.
|
|
|
|
* @param typeid1 Type ID.
|
|
|
|
* @param idx Enum member index.
|
|
|
|
* @param identifier Gets the identifier.
|
|
|
|
* @param value Gets the enum value.
|
|
|
|
*
|
|
|
|
* @return ::NC_NOERR No error.
|
|
|
|
* @return ::NC_EBADID Bad ncid.
|
2017-12-05 03:21:14 +08:00
|
|
|
* @return ::NC_EBADTYPE Type not found.
|
|
|
|
* @return ::NC_EINVAL Bad idx.
|
2017-12-04 06:37:56 +08:00
|
|
|
* @author Ed Hartnett
|
2019-02-19 20:56:22 +08:00
|
|
|
*/
|
2010-06-03 21:24:43 +08:00
|
|
|
int
|
2019-02-19 20:56:22 +08:00
|
|
|
NC4_inq_enum_member(int ncid, nc_type typeid1, int idx, char *identifier,
|
|
|
|
void *value)
|
2010-06-03 21:24:43 +08:00
|
|
|
{
|
2019-02-19 20:56:22 +08:00
|
|
|
NC_GRP_INFO_T *grp;
|
|
|
|
NC_TYPE_INFO_T *type;
|
|
|
|
NC_ENUM_MEMBER_INFO_T *enum_member;
|
|
|
|
int retval;
|
2010-06-03 21:24:43 +08:00
|
|
|
|
2019-02-19 20:56:22 +08:00
|
|
|
LOG((2, "nc_inq_enum_member: ncid 0x%x typeid %d", ncid, typeid1));
|
|
|
|
|
|
|
|
/* Find group metadata. */
|
|
|
|
if ((retval = nc4_find_nc4_grp(ncid, &grp)))
|
|
|
|
return retval;
|
|
|
|
|
|
|
|
/* Find this type. */
|
2023-11-27 19:36:03 +08:00
|
|
|
if (!(type = nclistget(grp->nc4_info->alltypes, (size_t)typeid1)))
|
2019-02-19 20:56:22 +08:00
|
|
|
return NC_EBADTYPE;
|
|
|
|
|
|
|
|
/* Complain if they are confused about the type. */
|
|
|
|
if (type->nc_type_class != NC_ENUM)
|
|
|
|
return NC_EBADTYPE;
|
|
|
|
|
|
|
|
/* Move to the desired enum member in the list. */
|
2023-11-27 19:36:03 +08:00
|
|
|
if (!(enum_member = nclistget(type->u.e.enum_member, (size_t)idx)))
|
2019-02-19 20:56:22 +08:00
|
|
|
return NC_EINVAL;
|
|
|
|
|
|
|
|
/* Give the people what they want. */
|
|
|
|
if (identifier)
|
|
|
|
strcpy(identifier, enum_member->name);
|
|
|
|
if (value)
|
|
|
|
memcpy(value, enum_member->value, type->size);
|
|
|
|
|
|
|
|
return NC_NOERR;
|
|
|
|
}
|
2020-08-18 09:15:47 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* @internal Get the id of a type from the name.
|
|
|
|
*
|
|
|
|
* @param ncid File and group ID.
|
Regularize the scoping of dimensions
This is a follow-on to pull request
````https://github.com/Unidata/netcdf-c/pull/1959````,
which fixed up type scoping.
The primary changes are to _nc\_inq\_dimid()_ and to ncdump.
The _nc\_inq\_dimid()_ function is supposed to allow the name to be
and FQN, but this apparently never got implemented. So if was modified
to support FQNs.
The ncdump program is supposed to output fully qualified dimension names
in its generated CDL file under certain conditions.
Suppose ncdump has a netcdf-4 file F with variable V, and V's parent group
is G. For each dimension id D referenced by V, ncdump needs to determine
whether to print its name as a simple name or as a fully qualified name (FQN).
The algorithm is as follows:
1. Search up the tree of ancestor groups.
2. If one of those ancestor groups contains the dimid, then call it dimgrp.
3. If one of those ancestor groups contains a dim with the same name as the dimid, but with a different dimid, then record that as duplicate=true.
4. If dimgrp is defined and duplicate == false, then we do not need an fqn.
5. If dimgrp is defined and duplicate == true, then we do need an fqn to avoid incorrectly using the duplicate.
6. If dimgrp is undefined, then do a preorder breadth-first search of all the groups looking for the dimid.
7. If found, then use the fqn of the first found such dimension location.
8. If not found, then fail.
Test case ncdump/test_scope.sh was modified to test the proper
operation of ncdump and _nc\_inq\_dimid()_.
Misc. Other Changes:
* Fix nc_inq_ncid (NC4_inq_ncid actually) to return root group id if the name argument is NULL.
* Modify _ncdump/printfqn_ to print out a dimid FQN; this supports verification that the resulting .nc files were properly created.
2021-06-01 05:51:12 +08:00
|
|
|
* @param name Name of type; might be fully qualified.
|
2020-08-18 09:15:47 +08:00
|
|
|
* @param typeidp Pointer that will get the type ID.
|
|
|
|
*
|
|
|
|
* @return ::NC_NOERR No error.
|
|
|
|
* @return ::NC_ENOMEM Out of memory.
|
|
|
|
* @return ::NC_EINVAL Bad size.
|
|
|
|
* @return ::NC_ENOTNC4 User types in netCDF-4 files only.
|
|
|
|
* @return ::NC_EBADTYPE Type not found.
|
|
|
|
* @author Ed Hartnett
|
|
|
|
*/
|
|
|
|
EXTERNL int
|
|
|
|
NC4_inq_typeid(int ncid, const char *name, nc_type *typeidp)
|
|
|
|
{
|
|
|
|
NC_GRP_INFO_T *grp;
|
|
|
|
NC_GRP_INFO_T *grptwo;
|
|
|
|
NC_FILE_INFO_T *h5;
|
|
|
|
NC_TYPE_INFO_T *type = NULL;
|
2021-03-07 05:09:37 +08:00
|
|
|
char *norm_name = NULL;
|
|
|
|
int i, retval = NC_NOERR;
|
2020-08-18 09:15:47 +08:00
|
|
|
|
|
|
|
/* Handle atomic types. */
|
|
|
|
for (i = 0; i < NUM_ATOMIC_TYPES; i++)
|
|
|
|
if (!strcmp(name, nc4_atomic_name[i]))
|
|
|
|
{
|
|
|
|
if (typeidp)
|
|
|
|
*typeidp = i;
|
2021-03-07 05:09:37 +08:00
|
|
|
goto done;
|
2020-08-18 09:15:47 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Find info for this file and group, and set pointer to each. */
|
|
|
|
if ((retval = nc4_find_grp_h5(ncid, &grp, &h5)))
|
2021-03-07 05:09:37 +08:00
|
|
|
goto done;
|
2020-08-18 09:15:47 +08:00
|
|
|
assert(h5 && grp);
|
|
|
|
|
|
|
|
/* If the first char is a /, this is a fully-qualified
|
|
|
|
* name. Otherwise, this had better be a local name (i.e. no / in
|
|
|
|
* the middle). */
|
|
|
|
if (name[0] != '/' && strstr(name, "/"))
|
2021-03-07 05:09:37 +08:00
|
|
|
{retval = NC_EINVAL; goto done;}
|
2020-08-18 09:15:47 +08:00
|
|
|
|
|
|
|
/* Normalize name. */
|
|
|
|
if (!(norm_name = (char*)malloc(strlen(name) + 1)))
|
2021-03-07 05:09:37 +08:00
|
|
|
{retval = NC_ENOMEM; goto done;}
|
|
|
|
if ((retval = nc4_normalize_name(name, norm_name)))
|
|
|
|
goto done;
|
|
|
|
|
|
|
|
/* If this is a fqn, then walk the sequence of parent groups to the last group
|
|
|
|
and see if that group has a type of the right name */
|
|
|
|
if(name[0] == '/') { /* FQN */
|
|
|
|
int rootncid = (grp->nc4_info->root_grp->hdr.id | grp->nc4_info->controller->ext_ncid);
|
|
|
|
int parent = 0;
|
|
|
|
char* lastname = strrchr(norm_name,'/'); /* break off the last segment: the type name */
|
|
|
|
if(lastname == norm_name)
|
|
|
|
{retval = NC_EINVAL; goto done;}
|
|
|
|
*lastname++ = '\0'; /* break off the lastsegment */
|
|
|
|
if((retval = NC4_inq_grp_full_ncid(rootncid,norm_name,&parent)))
|
|
|
|
goto done;
|
|
|
|
/* Get parent info */
|
|
|
|
if((retval=nc4_find_nc4_grp(parent,&grp)))
|
|
|
|
goto done;
|
|
|
|
/* See if type exists in this group */
|
|
|
|
type = (NC_TYPE_INFO_T*)ncindexlookup(grp->type,lastname);
|
|
|
|
if(type == NULL)
|
|
|
|
{retval = NC_EBADTYPE; goto done;}
|
|
|
|
goto done;
|
2020-08-18 09:15:47 +08:00
|
|
|
}
|
2021-03-07 05:09:37 +08:00
|
|
|
|
2020-08-18 09:15:47 +08:00
|
|
|
/* Is the type in this group? If not, search parents. */
|
|
|
|
for (grptwo = grp; grptwo; grptwo = grptwo->parent) {
|
|
|
|
type = (NC_TYPE_INFO_T*)ncindexlookup(grptwo->type,norm_name);
|
|
|
|
if(type)
|
|
|
|
{
|
|
|
|
if (typeidp)
|
|
|
|
*typeidp = type->hdr.id;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Still didn't find type? Search file recursively, starting at the
|
|
|
|
* root group. */
|
|
|
|
if (!type)
|
|
|
|
if ((type = nc4_rec_find_named_type(grp->nc4_info->root_grp, norm_name)))
|
|
|
|
if (typeidp)
|
|
|
|
*typeidp = type->hdr.id;
|
|
|
|
|
|
|
|
/* OK, I give up already! */
|
|
|
|
if (!type)
|
2021-03-07 05:09:37 +08:00
|
|
|
{retval = NC_EBADTYPE; goto done;}
|
2020-08-18 09:15:47 +08:00
|
|
|
|
2021-03-07 05:09:37 +08:00
|
|
|
done:
|
|
|
|
nullfree(norm_name);
|
|
|
|
return retval;
|
2020-08-18 09:15:47 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @internal Get the class of a type
|
|
|
|
*
|
|
|
|
* @param h5 Pointer to the HDF5 file info struct.
|
|
|
|
* @param xtype NetCDF type ID.
|
|
|
|
* @param type_class Pointer that gets class of type, NC_INT,
|
|
|
|
* NC_FLOAT, NC_CHAR, or NC_STRING, NC_ENUM, NC_VLEN, NC_COMPOUND, or
|
|
|
|
* NC_OPAQUE.
|
|
|
|
*
|
|
|
|
* @return ::NC_NOERR No error.
|
|
|
|
* @author Ed Hartnett
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
nc4_get_typeclass(const NC_FILE_INFO_T *h5, nc_type xtype, int *type_class)
|
|
|
|
{
|
|
|
|
int retval = NC_NOERR;
|
|
|
|
|
|
|
|
LOG((4, "%s xtype: %d", __func__, xtype));
|
|
|
|
assert(type_class);
|
|
|
|
|
|
|
|
/* If this is an atomic type, the answer is easy. */
|
|
|
|
if (xtype <= NC_STRING)
|
|
|
|
{
|
|
|
|
switch (xtype)
|
|
|
|
{
|
|
|
|
case NC_BYTE:
|
|
|
|
case NC_UBYTE:
|
|
|
|
case NC_SHORT:
|
|
|
|
case NC_USHORT:
|
|
|
|
case NC_INT:
|
|
|
|
case NC_UINT:
|
|
|
|
case NC_INT64:
|
|
|
|
case NC_UINT64:
|
|
|
|
/* NC_INT is class used for all integral types */
|
|
|
|
*type_class = NC_INT;
|
|
|
|
break;
|
|
|
|
|
|
|
|
case NC_FLOAT:
|
|
|
|
case NC_DOUBLE:
|
|
|
|
/* NC_FLOAT is class used for all floating-point types */
|
|
|
|
*type_class = NC_FLOAT;
|
|
|
|
break;
|
|
|
|
|
|
|
|
case NC_CHAR:
|
|
|
|
*type_class = NC_CHAR;
|
|
|
|
break;
|
|
|
|
|
|
|
|
case NC_STRING:
|
|
|
|
*type_class = NC_STRING;
|
|
|
|
break;
|
|
|
|
|
|
|
|
default:
|
|
|
|
BAIL(NC_EBADTYPE);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
NC_TYPE_INFO_T *type;
|
|
|
|
|
|
|
|
/* See if it's a used-defined type */
|
|
|
|
if ((retval = nc4_find_type(h5, xtype, &type)))
|
|
|
|
BAIL(retval);
|
|
|
|
if (!type)
|
|
|
|
BAIL(NC_EBADTYPE);
|
|
|
|
|
|
|
|
*type_class = type->nc_type_class;
|
|
|
|
}
|
|
|
|
|
|
|
|
exit:
|
|
|
|
return retval;
|
|
|
|
}
|
Fix various problem around VLEN's
re: https://github.com/Unidata/netcdf-c/issues/541
re: https://github.com/Unidata/netcdf-c/issues/1208
re: https://github.com/Unidata/netcdf-c/issues/2078
re: https://github.com/Unidata/netcdf-c/issues/2041
re: https://github.com/Unidata/netcdf-c/issues/2143
For a long time, there have been known problems with the
management of complex types containing VLENs. This also
involves the string type because it is stored as a VLEN of
chars.
This PR (mostly) fixes this problem. But note that it adds new
functions to netcdf.h (see below) and this may require bumping
the .so number. These new functions can be removed, if desired,
in favor of functions in netcdf_aux.h, but netcdf.h seems the
better place for them because they are intended as alternatives
to the nc_free_vlen and nc_free_string functions already in
netcdf.h.
The term complex type refers to any type that directly or
transitively references a VLEN type. So an array of VLENS, a
compound with a VLEN field, and so on.
In order to properly handle instances of these complex types, it
is necessary to have function that can recursively walk
instances of such types to perform various actions on them. The
term "deep" is also used to mean recursive.
At the moment, the two operations needed by the netcdf library are:
* free'ing an instance of the complex type
* copying an instance of the complex type.
The current library does only shallow free and shallow copy of
complex types. This means that only the top level is properly
free'd or copied, but deep internal blocks in the instance are
not touched.
Note that the term "vector" will be used to mean a contiguous (in
memory) sequence of instances of some type. Given an array with,
say, dimensions 2 X 3 X 4, this will be stored in memory as a
vector of length 2*3*4=24 instances.
The use cases are primarily these.
## nc_get_vars
Suppose one is reading a vector of instances using nc_get_vars
(or nc_get_vara or nc_get_var, etc.). These functions will
return the vector in the top-level memory provided. All
interior blocks (form nested VLEN or strings) will have been
dynamically allocated.
After using this vector of instances, it is necessary to free
(aka reclaim) the dynamically allocated memory, otherwise a
memory leak occurs. So, the recursive reclaim function is used
to walk the returned instance vector and do a deep reclaim of
the data.
Currently functions are defined in netcdf.h that are supposed to
handle this: nc_free_vlen(), nc_free_vlens(), and
nc_free_string(). Unfortunately, these functions only do a
shallow free, so deeply nested instances are not properly
handled by them.
Note that internally, the provided data is immediately written so
there is no need to copy it. But the caller may need to reclaim the
data it passed into the function.
## nc_put_att
Suppose one is writing a vector of instances as the data of an attribute
using, say, nc_put_att.
Internally, the incoming attribute data must be copied and stored
so that changes/reclamation of the input data will not affect
the attribute.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. As a result, one sees effects such as described
in Github Issue https://github.com/Unidata/netcdf-c/issues/2143.
Also, after defining the attribute, it may be necessary for the user
to free the data that was provided as input to nc_put_att().
## nc_get_att
Suppose one is reading a vector of instances as the data of an attribute
using, say, nc_get_att.
Internally, the existing attribute data must be copied and returned
to the caller, and the caller is responsible for reclaiming
the returned data.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. So this can lead to memory leaks and errors
because the deep data is shared between the library and the user.
# Solution
The solution is to build properly recursive reclaim and copy
functions and use those as needed.
These recursive functions are defined in libdispatch/dinstance.c
and their signatures are defined in include/netcdf.h.
For back compatibility, corresponding "ncaux_XXX" functions
are defined in include/netcdf_aux.h.
````
int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy);
int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp);
````
There are two variants. The first two, nc_reclaim_data() and
nc_copy_data(), assume the top-level vector is managed by the
caller. For reclaim, this is so the user can use, for example, a
statically allocated vector. For copy, it assumes the user
provides the space into which the copy is stored.
The second two, nc_reclaim_data_all() and
nc_copy_data_all(), allows the functions to manage the
top-level. So for nc_reclaim_data_all, the top level is
assumed to be dynamically allocated and will be free'd by
nc_reclaim_data_all(). The nc_copy_data_all() function
will allocate the top level and return a pointer to it to the
user. The user can later pass that pointer to
nc_reclaim_data_all() to reclaim the instance(s).
# Internal Changes
The netcdf-c library internals are changed to use the proper
reclaim and copy functions. It turns out that the places where
these functions are needed is quite pervasive in the netcdf-c
library code. Using these functions also allows some
simplification of the code since the stdata and vldata fields of
NC_ATT_INFO are no longer needed. Currently this is commented
out using the SEPDATA \#define macro. When any bugs are largely
fixed, all this code will be removed.
# Known Bugs
1. There is still one known failure that has not been solved.
All the failures revolve around some variant of this .cdl file.
The proximate cause of failure is the use of a VLEN FillValue.
````
netcdf x {
types:
float(*) row_of_floats ;
dimensions:
m = 5 ;
variables:
row_of_floats ragged_array(m) ;
row_of_floats ragged_array:_FillValue = {-999} ;
data:
ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32},
{40, 41}, _ ;
}
````
When a solution is found, I will either add it to this PR or post a new PR.
# Related Changes
* Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data.
* Remove the --enable-unfixed-memory-leaks option.
* Remove the NC_VLENS_NOTEST code that suppresses some vlen tests.
* Document this change in docs/internal.md
* Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh.
* Mark types as fixed size or not (transitively) to optimize the reclaim
and copy functions.
# Misc. Changes
* Make Doxygen process libdispatch/daux.c
* Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* @internal return 1 if type is fixed size; 0 otherwise.
|
|
|
|
*
|
|
|
|
* @param ncid file id
|
|
|
|
* @param xtype type id
|
|
|
|
* @param fixedsizep pointer into which 1/0 is stored
|
|
|
|
*
|
|
|
|
* @return ::NC_NOERR
|
|
|
|
* @return ::NC_EBADTYPE if bad type
|
|
|
|
* @author Dennis Heimbigner
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
NC4_inq_type_fixed_size(int ncid, nc_type xtype, int* fixedsizep)
|
|
|
|
{
|
|
|
|
int stat = NC_NOERR;
|
|
|
|
int f = 0;
|
Improve performance of the nc_reclaim_data and nc_copy_data functions.
re: Issue https://github.com/Unidata/netcdf-c/issues/2685
re: PR https://github.com/Unidata/netcdf-c/pull/2179
As noted in PR https://github.com/Unidata/netcdf-c/pull/2179,
the old code did not allow for reclaiming instances of types,
nor for properly copying them. That PR provided new functions
capable of reclaiming/copying instances of arbitrary types.
However, as noted by Issue https://github.com/Unidata/netcdf-c/issues/2685, using these
most general functions resulted in a significant performance
degradation, even for common cases.
This PR attempts to mitigate the cost of using the general
reclaim/copy functions in two ways.
First, the previous functions operating at the top level by
using ncid and typeid arguments. These functions were augmented
with equivalent versions that used the netcdf-c library internal
data structures to allow direct access to needed information.
These new functions are used internally to the library.
The second mitigation involves optimizing the internal functions
by providing early tests for common cases. This avoids
unnecessary recursive function calls.
The overall result is a significant improvement in speed by a
factor of roughly twenty -- your mileage may vary. These
optimized functions are still not as fast as the original (more
limited) functions, but they are getting close. Additional optimizations are
possible. But the cost is a significant "uglification" of the
code that I deemed a step too far, at least for now.
## Misc. Changes
1. Added a test case to check the proper reclamation/copy of complex types.
2. Found and fixed some places where nc_reclaim/copy should have been used.
3. Replaced, in the netcdf-c library, (almost all) occurrences of nc_reclaim_copy with calls to NC_reclaim/copy. This plus the optimizations is the primary speed-up mechanism.
4. In DAP4, the metadata is held in a substrate in-memory file; this required some changes so that the reclaim/copy code accessed that substrate dispatcher rather than the DAP4 dispatcher.
5. Re-factored and isolated the code that computes if a type is (transitively) variable-sized or not.
6. Clean up the reclamation code in ncgen; adding the use of nc_reclaim exposed some memory problems.
2023-05-21 07:11:25 +08:00
|
|
|
NC_FILE_INFO_T* h5 = NULL;
|
|
|
|
NC_TYPE_INFO_T* typ = NULL;
|
Fix various problem around VLEN's
re: https://github.com/Unidata/netcdf-c/issues/541
re: https://github.com/Unidata/netcdf-c/issues/1208
re: https://github.com/Unidata/netcdf-c/issues/2078
re: https://github.com/Unidata/netcdf-c/issues/2041
re: https://github.com/Unidata/netcdf-c/issues/2143
For a long time, there have been known problems with the
management of complex types containing VLENs. This also
involves the string type because it is stored as a VLEN of
chars.
This PR (mostly) fixes this problem. But note that it adds new
functions to netcdf.h (see below) and this may require bumping
the .so number. These new functions can be removed, if desired,
in favor of functions in netcdf_aux.h, but netcdf.h seems the
better place for them because they are intended as alternatives
to the nc_free_vlen and nc_free_string functions already in
netcdf.h.
The term complex type refers to any type that directly or
transitively references a VLEN type. So an array of VLENS, a
compound with a VLEN field, and so on.
In order to properly handle instances of these complex types, it
is necessary to have function that can recursively walk
instances of such types to perform various actions on them. The
term "deep" is also used to mean recursive.
At the moment, the two operations needed by the netcdf library are:
* free'ing an instance of the complex type
* copying an instance of the complex type.
The current library does only shallow free and shallow copy of
complex types. This means that only the top level is properly
free'd or copied, but deep internal blocks in the instance are
not touched.
Note that the term "vector" will be used to mean a contiguous (in
memory) sequence of instances of some type. Given an array with,
say, dimensions 2 X 3 X 4, this will be stored in memory as a
vector of length 2*3*4=24 instances.
The use cases are primarily these.
## nc_get_vars
Suppose one is reading a vector of instances using nc_get_vars
(or nc_get_vara or nc_get_var, etc.). These functions will
return the vector in the top-level memory provided. All
interior blocks (form nested VLEN or strings) will have been
dynamically allocated.
After using this vector of instances, it is necessary to free
(aka reclaim) the dynamically allocated memory, otherwise a
memory leak occurs. So, the recursive reclaim function is used
to walk the returned instance vector and do a deep reclaim of
the data.
Currently functions are defined in netcdf.h that are supposed to
handle this: nc_free_vlen(), nc_free_vlens(), and
nc_free_string(). Unfortunately, these functions only do a
shallow free, so deeply nested instances are not properly
handled by them.
Note that internally, the provided data is immediately written so
there is no need to copy it. But the caller may need to reclaim the
data it passed into the function.
## nc_put_att
Suppose one is writing a vector of instances as the data of an attribute
using, say, nc_put_att.
Internally, the incoming attribute data must be copied and stored
so that changes/reclamation of the input data will not affect
the attribute.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. As a result, one sees effects such as described
in Github Issue https://github.com/Unidata/netcdf-c/issues/2143.
Also, after defining the attribute, it may be necessary for the user
to free the data that was provided as input to nc_put_att().
## nc_get_att
Suppose one is reading a vector of instances as the data of an attribute
using, say, nc_get_att.
Internally, the existing attribute data must be copied and returned
to the caller, and the caller is responsible for reclaiming
the returned data.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. So this can lead to memory leaks and errors
because the deep data is shared between the library and the user.
# Solution
The solution is to build properly recursive reclaim and copy
functions and use those as needed.
These recursive functions are defined in libdispatch/dinstance.c
and their signatures are defined in include/netcdf.h.
For back compatibility, corresponding "ncaux_XXX" functions
are defined in include/netcdf_aux.h.
````
int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy);
int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp);
````
There are two variants. The first two, nc_reclaim_data() and
nc_copy_data(), assume the top-level vector is managed by the
caller. For reclaim, this is so the user can use, for example, a
statically allocated vector. For copy, it assumes the user
provides the space into which the copy is stored.
The second two, nc_reclaim_data_all() and
nc_copy_data_all(), allows the functions to manage the
top-level. So for nc_reclaim_data_all, the top level is
assumed to be dynamically allocated and will be free'd by
nc_reclaim_data_all(). The nc_copy_data_all() function
will allocate the top level and return a pointer to it to the
user. The user can later pass that pointer to
nc_reclaim_data_all() to reclaim the instance(s).
# Internal Changes
The netcdf-c library internals are changed to use the proper
reclaim and copy functions. It turns out that the places where
these functions are needed is quite pervasive in the netcdf-c
library code. Using these functions also allows some
simplification of the code since the stdata and vldata fields of
NC_ATT_INFO are no longer needed. Currently this is commented
out using the SEPDATA \#define macro. When any bugs are largely
fixed, all this code will be removed.
# Known Bugs
1. There is still one known failure that has not been solved.
All the failures revolve around some variant of this .cdl file.
The proximate cause of failure is the use of a VLEN FillValue.
````
netcdf x {
types:
float(*) row_of_floats ;
dimensions:
m = 5 ;
variables:
row_of_floats ragged_array(m) ;
row_of_floats ragged_array:_FillValue = {-999} ;
data:
ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32},
{40, 41}, _ ;
}
````
When a solution is found, I will either add it to this PR or post a new PR.
# Related Changes
* Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data.
* Remove the --enable-unfixed-memory-leaks option.
* Remove the NC_VLENS_NOTEST code that suppresses some vlen tests.
* Document this change in docs/internal.md
* Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh.
* Mark types as fixed size or not (transitively) to optimize the reclaim
and copy functions.
# Misc. Changes
* Make Doxygen process libdispatch/daux.c
* Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
|
|
|
|
|
|
|
if(xtype < NC_STRING) {f = 1; goto done;}
|
|
|
|
if(xtype == NC_STRING) {f = 0; goto done;}
|
2022-01-11 06:27:16 +08:00
|
|
|
|
Fix various problem around VLEN's
re: https://github.com/Unidata/netcdf-c/issues/541
re: https://github.com/Unidata/netcdf-c/issues/1208
re: https://github.com/Unidata/netcdf-c/issues/2078
re: https://github.com/Unidata/netcdf-c/issues/2041
re: https://github.com/Unidata/netcdf-c/issues/2143
For a long time, there have been known problems with the
management of complex types containing VLENs. This also
involves the string type because it is stored as a VLEN of
chars.
This PR (mostly) fixes this problem. But note that it adds new
functions to netcdf.h (see below) and this may require bumping
the .so number. These new functions can be removed, if desired,
in favor of functions in netcdf_aux.h, but netcdf.h seems the
better place for them because they are intended as alternatives
to the nc_free_vlen and nc_free_string functions already in
netcdf.h.
The term complex type refers to any type that directly or
transitively references a VLEN type. So an array of VLENS, a
compound with a VLEN field, and so on.
In order to properly handle instances of these complex types, it
is necessary to have function that can recursively walk
instances of such types to perform various actions on them. The
term "deep" is also used to mean recursive.
At the moment, the two operations needed by the netcdf library are:
* free'ing an instance of the complex type
* copying an instance of the complex type.
The current library does only shallow free and shallow copy of
complex types. This means that only the top level is properly
free'd or copied, but deep internal blocks in the instance are
not touched.
Note that the term "vector" will be used to mean a contiguous (in
memory) sequence of instances of some type. Given an array with,
say, dimensions 2 X 3 X 4, this will be stored in memory as a
vector of length 2*3*4=24 instances.
The use cases are primarily these.
## nc_get_vars
Suppose one is reading a vector of instances using nc_get_vars
(or nc_get_vara or nc_get_var, etc.). These functions will
return the vector in the top-level memory provided. All
interior blocks (form nested VLEN or strings) will have been
dynamically allocated.
After using this vector of instances, it is necessary to free
(aka reclaim) the dynamically allocated memory, otherwise a
memory leak occurs. So, the recursive reclaim function is used
to walk the returned instance vector and do a deep reclaim of
the data.
Currently functions are defined in netcdf.h that are supposed to
handle this: nc_free_vlen(), nc_free_vlens(), and
nc_free_string(). Unfortunately, these functions only do a
shallow free, so deeply nested instances are not properly
handled by them.
Note that internally, the provided data is immediately written so
there is no need to copy it. But the caller may need to reclaim the
data it passed into the function.
## nc_put_att
Suppose one is writing a vector of instances as the data of an attribute
using, say, nc_put_att.
Internally, the incoming attribute data must be copied and stored
so that changes/reclamation of the input data will not affect
the attribute.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. As a result, one sees effects such as described
in Github Issue https://github.com/Unidata/netcdf-c/issues/2143.
Also, after defining the attribute, it may be necessary for the user
to free the data that was provided as input to nc_put_att().
## nc_get_att
Suppose one is reading a vector of instances as the data of an attribute
using, say, nc_get_att.
Internally, the existing attribute data must be copied and returned
to the caller, and the caller is responsible for reclaiming
the returned data.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. So this can lead to memory leaks and errors
because the deep data is shared between the library and the user.
# Solution
The solution is to build properly recursive reclaim and copy
functions and use those as needed.
These recursive functions are defined in libdispatch/dinstance.c
and their signatures are defined in include/netcdf.h.
For back compatibility, corresponding "ncaux_XXX" functions
are defined in include/netcdf_aux.h.
````
int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy);
int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp);
````
There are two variants. The first two, nc_reclaim_data() and
nc_copy_data(), assume the top-level vector is managed by the
caller. For reclaim, this is so the user can use, for example, a
statically allocated vector. For copy, it assumes the user
provides the space into which the copy is stored.
The second two, nc_reclaim_data_all() and
nc_copy_data_all(), allows the functions to manage the
top-level. So for nc_reclaim_data_all, the top level is
assumed to be dynamically allocated and will be free'd by
nc_reclaim_data_all(). The nc_copy_data_all() function
will allocate the top level and return a pointer to it to the
user. The user can later pass that pointer to
nc_reclaim_data_all() to reclaim the instance(s).
# Internal Changes
The netcdf-c library internals are changed to use the proper
reclaim and copy functions. It turns out that the places where
these functions are needed is quite pervasive in the netcdf-c
library code. Using these functions also allows some
simplification of the code since the stdata and vldata fields of
NC_ATT_INFO are no longer needed. Currently this is commented
out using the SEPDATA \#define macro. When any bugs are largely
fixed, all this code will be removed.
# Known Bugs
1. There is still one known failure that has not been solved.
All the failures revolve around some variant of this .cdl file.
The proximate cause of failure is the use of a VLEN FillValue.
````
netcdf x {
types:
float(*) row_of_floats ;
dimensions:
m = 5 ;
variables:
row_of_floats ragged_array(m) ;
row_of_floats ragged_array:_FillValue = {-999} ;
data:
ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32},
{40, 41}, _ ;
}
````
When a solution is found, I will either add it to this PR or post a new PR.
# Related Changes
* Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data.
* Remove the --enable-unfixed-memory-leaks option.
* Remove the NC_VLENS_NOTEST code that suppresses some vlen tests.
* Document this change in docs/internal.md
* Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh.
* Mark types as fixed size or not (transitively) to optimize the reclaim
and copy functions.
# Misc. Changes
* Make Doxygen process libdispatch/daux.c
* Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
|
|
|
#ifdef USE_NETCDF4
|
|
|
|
/* Must be user type */
|
Improve performance of the nc_reclaim_data and nc_copy_data functions.
re: Issue https://github.com/Unidata/netcdf-c/issues/2685
re: PR https://github.com/Unidata/netcdf-c/pull/2179
As noted in PR https://github.com/Unidata/netcdf-c/pull/2179,
the old code did not allow for reclaiming instances of types,
nor for properly copying them. That PR provided new functions
capable of reclaiming/copying instances of arbitrary types.
However, as noted by Issue https://github.com/Unidata/netcdf-c/issues/2685, using these
most general functions resulted in a significant performance
degradation, even for common cases.
This PR attempts to mitigate the cost of using the general
reclaim/copy functions in two ways.
First, the previous functions operating at the top level by
using ncid and typeid arguments. These functions were augmented
with equivalent versions that used the netcdf-c library internal
data structures to allow direct access to needed information.
These new functions are used internally to the library.
The second mitigation involves optimizing the internal functions
by providing early tests for common cases. This avoids
unnecessary recursive function calls.
The overall result is a significant improvement in speed by a
factor of roughly twenty -- your mileage may vary. These
optimized functions are still not as fast as the original (more
limited) functions, but they are getting close. Additional optimizations are
possible. But the cost is a significant "uglification" of the
code that I deemed a step too far, at least for now.
## Misc. Changes
1. Added a test case to check the proper reclamation/copy of complex types.
2. Found and fixed some places where nc_reclaim/copy should have been used.
3. Replaced, in the netcdf-c library, (almost all) occurrences of nc_reclaim_copy with calls to NC_reclaim/copy. This plus the optimizations is the primary speed-up mechanism.
4. In DAP4, the metadata is held in a substrate in-memory file; this required some changes so that the reclaim/copy code accessed that substrate dispatcher rather than the DAP4 dispatcher.
5. Re-factored and isolated the code that computes if a type is (transitively) variable-sized or not.
6. Clean up the reclamation code in ncgen; adding the use of nc_reclaim exposed some memory problems.
2023-05-21 07:11:25 +08:00
|
|
|
if ((stat = nc4_find_grp_h5(ncid, NULL, &h5)))
|
|
|
|
goto done;
|
|
|
|
if((stat = nc4_find_type(h5,xtype,&typ))) goto done;
|
|
|
|
f = !typ->varsized;
|
Fix various problem around VLEN's
re: https://github.com/Unidata/netcdf-c/issues/541
re: https://github.com/Unidata/netcdf-c/issues/1208
re: https://github.com/Unidata/netcdf-c/issues/2078
re: https://github.com/Unidata/netcdf-c/issues/2041
re: https://github.com/Unidata/netcdf-c/issues/2143
For a long time, there have been known problems with the
management of complex types containing VLENs. This also
involves the string type because it is stored as a VLEN of
chars.
This PR (mostly) fixes this problem. But note that it adds new
functions to netcdf.h (see below) and this may require bumping
the .so number. These new functions can be removed, if desired,
in favor of functions in netcdf_aux.h, but netcdf.h seems the
better place for them because they are intended as alternatives
to the nc_free_vlen and nc_free_string functions already in
netcdf.h.
The term complex type refers to any type that directly or
transitively references a VLEN type. So an array of VLENS, a
compound with a VLEN field, and so on.
In order to properly handle instances of these complex types, it
is necessary to have function that can recursively walk
instances of such types to perform various actions on them. The
term "deep" is also used to mean recursive.
At the moment, the two operations needed by the netcdf library are:
* free'ing an instance of the complex type
* copying an instance of the complex type.
The current library does only shallow free and shallow copy of
complex types. This means that only the top level is properly
free'd or copied, but deep internal blocks in the instance are
not touched.
Note that the term "vector" will be used to mean a contiguous (in
memory) sequence of instances of some type. Given an array with,
say, dimensions 2 X 3 X 4, this will be stored in memory as a
vector of length 2*3*4=24 instances.
The use cases are primarily these.
## nc_get_vars
Suppose one is reading a vector of instances using nc_get_vars
(or nc_get_vara or nc_get_var, etc.). These functions will
return the vector in the top-level memory provided. All
interior blocks (form nested VLEN or strings) will have been
dynamically allocated.
After using this vector of instances, it is necessary to free
(aka reclaim) the dynamically allocated memory, otherwise a
memory leak occurs. So, the recursive reclaim function is used
to walk the returned instance vector and do a deep reclaim of
the data.
Currently functions are defined in netcdf.h that are supposed to
handle this: nc_free_vlen(), nc_free_vlens(), and
nc_free_string(). Unfortunately, these functions only do a
shallow free, so deeply nested instances are not properly
handled by them.
Note that internally, the provided data is immediately written so
there is no need to copy it. But the caller may need to reclaim the
data it passed into the function.
## nc_put_att
Suppose one is writing a vector of instances as the data of an attribute
using, say, nc_put_att.
Internally, the incoming attribute data must be copied and stored
so that changes/reclamation of the input data will not affect
the attribute.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. As a result, one sees effects such as described
in Github Issue https://github.com/Unidata/netcdf-c/issues/2143.
Also, after defining the attribute, it may be necessary for the user
to free the data that was provided as input to nc_put_att().
## nc_get_att
Suppose one is reading a vector of instances as the data of an attribute
using, say, nc_get_att.
Internally, the existing attribute data must be copied and returned
to the caller, and the caller is responsible for reclaiming
the returned data.
Again, the code inside the netcdf library does only shallow copying
rather than deep copy. So this can lead to memory leaks and errors
because the deep data is shared between the library and the user.
# Solution
The solution is to build properly recursive reclaim and copy
functions and use those as needed.
These recursive functions are defined in libdispatch/dinstance.c
and their signatures are defined in include/netcdf.h.
For back compatibility, corresponding "ncaux_XXX" functions
are defined in include/netcdf_aux.h.
````
int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy);
int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp);
````
There are two variants. The first two, nc_reclaim_data() and
nc_copy_data(), assume the top-level vector is managed by the
caller. For reclaim, this is so the user can use, for example, a
statically allocated vector. For copy, it assumes the user
provides the space into which the copy is stored.
The second two, nc_reclaim_data_all() and
nc_copy_data_all(), allows the functions to manage the
top-level. So for nc_reclaim_data_all, the top level is
assumed to be dynamically allocated and will be free'd by
nc_reclaim_data_all(). The nc_copy_data_all() function
will allocate the top level and return a pointer to it to the
user. The user can later pass that pointer to
nc_reclaim_data_all() to reclaim the instance(s).
# Internal Changes
The netcdf-c library internals are changed to use the proper
reclaim and copy functions. It turns out that the places where
these functions are needed is quite pervasive in the netcdf-c
library code. Using these functions also allows some
simplification of the code since the stdata and vldata fields of
NC_ATT_INFO are no longer needed. Currently this is commented
out using the SEPDATA \#define macro. When any bugs are largely
fixed, all this code will be removed.
# Known Bugs
1. There is still one known failure that has not been solved.
All the failures revolve around some variant of this .cdl file.
The proximate cause of failure is the use of a VLEN FillValue.
````
netcdf x {
types:
float(*) row_of_floats ;
dimensions:
m = 5 ;
variables:
row_of_floats ragged_array(m) ;
row_of_floats ragged_array:_FillValue = {-999} ;
data:
ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32},
{40, 41}, _ ;
}
````
When a solution is found, I will either add it to this PR or post a new PR.
# Related Changes
* Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data.
* Remove the --enable-unfixed-memory-leaks option.
* Remove the NC_VLENS_NOTEST code that suppresses some vlen tests.
* Document this change in docs/internal.md
* Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh.
* Mark types as fixed size or not (transitively) to optimize the reclaim
and copy functions.
# Misc. Changes
* Make Doxygen process libdispatch/daux.c
* Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
|
|
|
#endif
|
|
|
|
done:
|
|
|
|
if(fixedsizep) *fixedsizep = f;
|
|
|
|
return stat;
|
|
|
|
}
|
Improve performance of the nc_reclaim_data and nc_copy_data functions.
re: Issue https://github.com/Unidata/netcdf-c/issues/2685
re: PR https://github.com/Unidata/netcdf-c/pull/2179
As noted in PR https://github.com/Unidata/netcdf-c/pull/2179,
the old code did not allow for reclaiming instances of types,
nor for properly copying them. That PR provided new functions
capable of reclaiming/copying instances of arbitrary types.
However, as noted by Issue https://github.com/Unidata/netcdf-c/issues/2685, using these
most general functions resulted in a significant performance
degradation, even for common cases.
This PR attempts to mitigate the cost of using the general
reclaim/copy functions in two ways.
First, the previous functions operating at the top level by
using ncid and typeid arguments. These functions were augmented
with equivalent versions that used the netcdf-c library internal
data structures to allow direct access to needed information.
These new functions are used internally to the library.
The second mitigation involves optimizing the internal functions
by providing early tests for common cases. This avoids
unnecessary recursive function calls.
The overall result is a significant improvement in speed by a
factor of roughly twenty -- your mileage may vary. These
optimized functions are still not as fast as the original (more
limited) functions, but they are getting close. Additional optimizations are
possible. But the cost is a significant "uglification" of the
code that I deemed a step too far, at least for now.
## Misc. Changes
1. Added a test case to check the proper reclamation/copy of complex types.
2. Found and fixed some places where nc_reclaim/copy should have been used.
3. Replaced, in the netcdf-c library, (almost all) occurrences of nc_reclaim_copy with calls to NC_reclaim/copy. This plus the optimizations is the primary speed-up mechanism.
4. In DAP4, the metadata is held in a substrate in-memory file; this required some changes so that the reclaim/copy code accessed that substrate dispatcher rather than the DAP4 dispatcher.
5. Re-factored and isolated the code that computes if a type is (transitively) variable-sized or not.
6. Clean up the reclamation code in ncgen; adding the use of nc_reclaim exposed some memory problems.
2023-05-21 07:11:25 +08:00
|
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
For types with one or more subtypes (e.g. basetype or
|
|
|
|
fieldtype), determine the varsizedness of the type based on the
|
|
|
|
basetype. The idea is to inform the code of the fact that
|
|
|
|
parenttype has addedtype "inserted" into it.
|
|
|
|
@param parenttype
|
|
|
|
@param subtype
|
|
|
|
*/
|
|
|
|
|
|
|
|
int
|
|
|
|
NC4_recheck_varsize(NC_TYPE_INFO_T* parenttype, nc_type subtype)
|
|
|
|
{
|
|
|
|
int stat = NC_NOERR;
|
|
|
|
NC_FILE_INFO_T* file = NULL;
|
|
|
|
NC_TYPE_INFO_T* utype = NULL;
|
|
|
|
if(subtype < NC_STRING) goto done; /* will not change the "variable-sizedness" of parenttype */
|
|
|
|
if(subtype == NC_STRING) {parenttype->varsized = 1; goto done;}
|
|
|
|
/* Get the inferred user-type */
|
|
|
|
file = parenttype->container->nc4_info;
|
|
|
|
if((stat = nc4_find_type(file,subtype,&utype))) goto done;
|
|
|
|
switch (utype->nc_type_class) {
|
|
|
|
case NC_OPAQUE: case NC_ENUM: break; /* no change */
|
|
|
|
case NC_VLEN: parenttype->varsized = 1; break;
|
|
|
|
case NC_COMPOUND: if(utype->varsized) parenttype->varsized = 1; break;
|
|
|
|
}
|
|
|
|
|
|
|
|
done:
|
|
|
|
return stat;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
When creating a type, mark it as variable-sized if known for sure.
|
|
|
|
@param typ
|
|
|
|
*/
|
|
|
|
|
|
|
|
int
|
|
|
|
NC4_set_varsize(NC_TYPE_INFO_T* typ)
|
|
|
|
{
|
|
|
|
int stat = NC_NOERR;
|
|
|
|
if(typ->hdr.id < NC_STRING) goto done; /* will not change the "variable-sizedness" of typ */
|
|
|
|
if(typ->hdr.id == NC_STRING) {typ->varsized = 1; goto done;}
|
|
|
|
switch (typ->nc_type_class) {
|
|
|
|
case NC_OPAQUE: case NC_ENUM: break; /* no change */
|
|
|
|
case NC_VLEN: typ->varsized = 1; break;
|
|
|
|
case NC_COMPOUND: typ->varsized = 0; break; /* until proven otherwise */
|
|
|
|
}
|
|
|
|
done:
|
|
|
|
return stat;
|
|
|
|
}
|
2023-06-22 04:46:22 +08:00
|
|
|
|
|
|
|
/**
|
|
|
|
* Test if a variable's type is fixed sized or not.
|
|
|
|
* @param var - to test
|
|
|
|
* @return 0 if fixed size, 1 otherwise.
|
|
|
|
*/
|
|
|
|
int
|
|
|
|
NC4_var_varsized(NC_VAR_INFO_T* var)
|
|
|
|
{
|
|
|
|
NC_TYPE_INFO_T* vtype = NULL;
|
|
|
|
|
|
|
|
/* Check the variable type */
|
|
|
|
vtype = var->type_info;
|
|
|
|
if(vtype->hdr.id < NC_STRING) return 0;
|
|
|
|
if(vtype->hdr.id == NC_STRING) return 1;
|
|
|
|
return vtype->varsized;
|
|
|
|
}
|
|
|
|
|