mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2025-04-12 18:10:24 +08:00
Code Issues Packages Projects Releases Wiki Activity

Fix various problem around VLEN's

Browse Source 
re: https://github.com/Unidata/netcdf-c/issues/541
re: https://github.com/Unidata/netcdf-c/issues/1208
re: https://github.com/Unidata/netcdf-c/issues/2078
re: https://github.com/Unidata/netcdf-c/issues/2041
re: https://github.com/Unidata/netcdf-c/issues/2143

For a long time, there have been known problems with the
management of complex types containing VLENs.  This also
involves the string type because it is stored as a VLEN of
chars.

This PR (mostly) fixes this problem. But note that it adds new
functions to netcdf.h (see below) and this may require bumping
the .so number.  These new functions can be removed, if desired,
in favor of functions in netcdf_aux.h, but netcdf.h seems the
better place for them because they are intended as alternatives
to the nc_free_vlen and nc_free_string functions already in
netcdf.h.

The term complex type refers to any type that directly or
transitively references a VLEN type. So an array of VLENS, a
compound with a VLEN field, and so on.

In order to properly handle instances of these complex types, it
is necessary to have function that can recursively walk
instances of such types to perform various actions on them.  The
term "deep" is also used to mean recursive.

At the moment, the two operations needed by the netcdf library are:
* free'ing an instance of the complex type
* copying an instance of the complex type.

The current library does only shallow free and shallow copy of
complex types. This means that only the top level is properly
free'd or copied, but deep internal blocks in the instance are
not touched.

Note that the term "vector" will be used to mean a contiguous (in
memory) sequence of instances of some type. Given an array with,
say, dimensions 2 X 3 X 4, this will be stored in memory as a
vector of length 2*3*4=24 instances.

The use cases are primarily these.

## nc_get_vars
Suppose one is reading a vector of instances using nc_get_vars
(or nc_get_vara or nc_get_var, etc.).  These functions will
return the vector in the top-level memory provided.  All
interior blocks (form nested VLEN or strings) will have been
dynamically allocated.

After using this vector of instances, it is necessary to free
(aka reclaim) the dynamically allocated memory, otherwise a
memory leak occurs.  So, the recursive reclaim function is used
to walk the returned instance vector and do a deep reclaim of
the data.

Currently functions are defined in netcdf.h that are supposed to
handle this: nc_free_vlen(), nc_free_vlens(), and
nc_free_string().  Unfortunately, these functions only do a
shallow free, so deeply nested instances are not properly
handled by them.

Note that internally, the provided data is immediately written so
there is no need to copy it. But the caller may need to reclaim the
data it passed into the function.

## nc_put_att
Suppose one is writing a vector of instances as the data of an attribute
using, say, nc_put_att.

Internally, the incoming attribute data must be copied and stored
so that changes/reclamation of the input data will not affect
the attribute.

Again, the code inside the netcdf library does only shallow copying
rather than deep copy. As a result, one sees effects such as described
in Github Issue https://github.com/Unidata/netcdf-c/issues/2143.

Also, after defining the attribute, it may be necessary for the user
to free the data that was provided as input to nc_put_att().

## nc_get_att
Suppose one is reading a vector of instances as the data of an attribute
using, say, nc_get_att.

Internally, the existing attribute data must be copied and returned
to the caller, and the caller is responsible for reclaiming
the returned data.

Again, the code inside the netcdf library does only shallow copying
rather than deep copy. So this can lead to memory leaks and errors
because the deep data is shared between the library and the user.

# Solution

The solution is to build properly recursive reclaim and copy
functions and use those as needed.
These recursive functions are defined in libdispatch/dinstance.c
and their signatures are defined in include/netcdf.h.
For back compatibility, corresponding "ncaux_XXX" functions
are defined in include/netcdf_aux.h.
````
int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count);
int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy);
int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp);
````
There are two variants. The first two, nc_reclaim_data() and
nc_copy_data(), assume the top-level vector is managed by the
caller. For reclaim, this is so the user can use, for example, a
statically allocated vector. For copy, it assumes the user
provides the space into which the copy is stored.

The second two, nc_reclaim_data_all() and
nc_copy_data_all(), allows the functions to manage the
top-level.  So for nc_reclaim_data_all, the top level is
assumed to be dynamically allocated and will be free'd by
nc_reclaim_data_all().  The nc_copy_data_all() function
will allocate the top level and return a pointer to it to the
user. The user can later pass that pointer to
nc_reclaim_data_all() to reclaim the instance(s).

# Internal Changes
The netcdf-c library internals are changed to use the proper
reclaim and copy functions.  It turns out that the places where
these functions are needed is quite pervasive in the netcdf-c
library code.  Using these functions also allows some
simplification of the code since the stdata and vldata fields of
NC_ATT_INFO are no longer needed.  Currently this is commented
out using the SEPDATA \#define macro.  When any bugs are largely
fixed, all this code will be removed.

# Known Bugs

1. There is still one known failure that has not been solved.
   All the failures revolve around some variant of this .cdl file.
   The proximate cause of failure is the use of a VLEN FillValue.
````
        netcdf x {
        types:
          float(*) row_of_floats ;
        dimensions:
          m = 5 ;
        variables:
          row_of_floats ragged_array(m) ;
              row_of_floats ragged_array:_FillValue = {-999} ;
        data:
          ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32},
                         {40, 41}, _ ;
        }
````
When a solution is found, I will either add it to this PR or post a new PR.

# Related Changes

* Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data.
* Remove the --enable-unfixed-memory-leaks option.
* Remove the NC_VLENS_NOTEST code that suppresses some vlen tests.
* Document this change in docs/internal.md
* Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh.
* Mark types as fixed size or not (transitively) to optimize the reclaim
  and copy functions.

# Misc. Changes

* Make Doxygen process libdispatch/daux.c
* Make sure the NC_ATT_INFO_T.container field is set.
This commit is contained in:
Dennis Heimbigner 2022-01-08 18:30:00 -07:00
parent 988e771a9e
commit 8b9253fef2
67 changed files with 5339 additions and 4361 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/mingw.yml vendored
View File

@ -1,5 +1,5 @@
name: NetCDF-Build MinGW
on: [workflow_dispatch]
on: [workflow_dispatch,push]
jobs:
build:

2
.github/workflows/run_tests.yml vendored
View File

@ -4,7 +4,7 @@
name: Run netCDF Tests
on: [pull_request]
on: [pull_request,push]
jobs:

4
CMakeLists.txt
View File

@ -1270,10 +1270,6 @@ IF(ENABLE_TESTS)
OPTION(ENABLE_FAILING_TESTS "Run tests which are known to fail, check to see if any have been fixed." OFF)
# There is a special class of tests that currently fail because of memory leaks.
# As with failing tests, they can be enabled to work on them
OPTION(ENABLE_UNFIXED_MEMORY_LEAKS "Run tests with unfixed memory leaks" OFF)
###
# Option to turn on unit testing. See
# https://github.com/Unidata/netcdf-c/pull/1472 for more information.

1
RELEASE_NOTES.md
View File

@ -7,6 +7,7 @@ This file contains a high-level description of this package's evolution. Release
## 4.8.2 - TBD
* [Bug Fix] Clean up the handling of deeply nested VLEN types. Marks nc_free_vlen() and nc_free_string as deprecated in favor of ncaux_reclaim_data(). See [Github #????(https://github.com/Unidata/netcdf-c/pull/????).
* [Enhancement] Improve support for msys2+mingw platform. See [Github #2171](https://github.com/Unidata/netcdf-c/pull/2171).
* [Bug Fix] Clean up the various inter-test dependencies in ncdump for CMake. See [Github #2168](https://github.com/Unidata/netcdf-c/pull/2168).
* [Enhancement] Added options to suppress the new behavior from [Github #2135](https://github.com/Unidata/netcdf-c/pull/2135). The options for `cmake` and `configure` are, respectively `-DENABLE_LIBXML2` and `--(enable/disable)-libxml2`. Both of these options defaul to 'on/enabled'. When disabled, the bundled `ezxml` XML interpreter is used regardless of whether `libxml2` is present on the system.

9
configure.ac
View File

@ -299,15 +299,6 @@ if test "x$enable_jna" = xyes ; then
AC_DEFINE([JNA], [1], [if true, include jna bug workaround code])
fi
# Should tests with unfixed memory leaks be run?
AC_MSG_CHECKING([if unfixed tests with memory leaks should be enabled])
AC_ARG_ENABLE([unfixed-memory-leaks],
[AS_HELP_STRING([--enable-unfixed-memory-leaks],
[Enable tests in the ncdump directory that have unfixed memory leaks])])
test "x$enable_unfixed_memory_leaks" = xyes || enable_unfixed_memory_leaks=no
AC_MSG_RESULT($enable_unfixed_memory_leaks)
AM_CONDITIONAL([ENABLE_UNFIXED_MEMORY_LEAKS], [test "x$enable_unfixed_memory_leaks" = xyes])
# Does the user want to turn off unit tests (useful for test coverage
# analysis).
AC_MSG_CHECKING([if unit tests should be disabled])

1
docs/Doxyfile.in
View File

@ -782,6 +782,7 @@ INPUT = \
@abs_top_srcdir@/libdispatch/derror.c \
@abs_top_srcdir@/libdispatch/dv2i.c \
@abs_top_srcdir@/libdispatch/dcopy.c \
@abs_top_srcdir@/libdispatch/daux.c \
@abs_top_srcdir@/libsrc4/nc4var.c \
@abs_top_srcdir@/libhdf5/nc4hdf.c \
@abs_top_srcdir@/libsrc4/nc4internal.c \

10
include/nc4internal.h
View File

@ -170,8 +170,10 @@ typedef struct NC_ATT_INFO
nc_type nc_typeid; /**< NetCDF type of attribute's data. */
void *format_att_info; /**< Pointer to format-specific att info. */
void *data; /**< The attribute data. */
#ifdef SEPDATA
nc_vlen_t *vldata; /**< VLEN data (only used for vlen types). */
char **stdata; /**< String data (only for string type). */
#endif
} NC_ATT_INFO_T;
/** This is a struct to handle the var metadata. */
@ -252,6 +254,7 @@ typedef struct NC_TYPE_INFO
} e; /**< Enum */
struct Fields {
NClist* field; /**< <! NClist<NC_FIELD_INFO_T*> */
int varsized; /**< <! 1 if this compound is variable sized; 0 if fixed size */
} c; /**< Compound */
struct {
nc_type base_nc_typeid; /**< Typeid of the base type. */
@ -287,7 +290,7 @@ typedef struct NC_GRP_INFO
#define NC_HDIRTY 0x10080 /**< header info has changed */
/** This is the metadata we need to keep track of for each
* netcdf-4/HDF5 file. */
* netcdf-4/ file; used by libhdf5, libnczarr, and libdap4 */
typedef struct NC_FILE_INFO
{
@ -302,7 +305,7 @@ typedef struct NC_FILE_INFO
nc_bool_t parallel; /**< True if file is open for parallel access */
nc_bool_t redef; /**< True if redefining an existing file */
nc_bool_t no_attr_create_order; /**< True if the creation order tracking of attributes is disabled (netcdf-4 only) */
int fill_mode; /**< Fill mode for vars - Unused internally currently */
int fill_mode; /**< Fill mode for vars */
nc_bool_t no_write; /**< true if nc_open has mode NC_NOWRITE. */
NC_GRP_INFO_T *root_grp; /**< Pointer to root group. */
short next_nc_grpid; /**< Next available group ID. */
@ -433,6 +436,9 @@ extern int nc4_get_att_ptrs(NC_FILE_INFO_T *h5, NC_GRP_INFO_T *grp, NC_VAR_INFO_
const char *name, nc_type *xtype, nc_type mem_type,
size_t *lenp, int *attnum, void *data);
/* Get variable/fixed size flag for type */
extern int NC4_inq_type_fixed_size(int ncid, nc_type xtype, int* isfixedsizep);
/* Close the file. */
extern int nc4_close_netcdf4_file(NC_FILE_INFO_T *h5, int abort, NC_memio *memio);

5
include/ncoffsets.h
View File

@ -51,7 +51,10 @@ typedef struct NCtypealignset {
NCalignment ncvlenalign; /* nc_vlen_t*/
} NCtypealignset;
EXTERNL size_t NC_class_alignment(int ncclass);
EXTERNL int NC_class_alignment(int ncclass, size_t*);
EXTERNL void NC_compute_alignments(void);
/* From libdispatch/dinstance.c */
EXTERNL int NC_type_alignment(int ncid, nc_type xtype, size_t*);
#endif /*NCOFFSETS_H*/

58
include/netcdf.h
View File

@ -740,7 +740,10 @@ nc_inq_vlen(int ncid, nc_type xtype, char *name, size_t *datum_sizep,
/* When you read VLEN type the library will actually allocate the
* storage space for the data. This storage space must be freed, so
* pass the pointer back to this function, when you're done with the
* data, and it will free the vlen memory. */
* data, and it will free the vlen memory.
* These two functions are deprecated in favor of the nc_reclaim_data function.
*/
EXTERNL int
nc_free_vlen(nc_vlen_t *vl);
@ -759,7 +762,9 @@ nc_get_vlen_element(int ncid, int typeid1, const void *vlen_element,
/* When you read the string type the library will allocate the storage
* space for the data. This storage space must be freed, so pass the
* pointer back to this function, when you're done with the data, and
* it will free the string memory. */
* it will free the string memory.
* This function is deprecated in favor of the nc_reclaim_data function.
*/
EXTERNL int
nc_free_string(size_t len, char **data);
@ -1751,6 +1756,53 @@ nc_put_var_string(int ncid, int varid, const char **op);
EXTERNL int
nc_get_var_string(int ncid, int varid, char **ip);
/* Begin recursive instance walking functions */
/**
Reclaim a vector of instances of arbitrary type. Intended for
use with e.g. nc_get_vara or the input to e.g. nc_put_vara.
This recursively walks the top-level instances to reclaim any
nested data such as vlen or strings or such.
Assumes it is passed a pointer to count instances of xtype.
Reclaims any nested data.
WARNING: nc_reclaim_data does not reclaim the top-level
memory because we do not know how it was allocated. However
nc_reclaim_data_all does reclaim top-level memory.
WARNING: all data blocks below the top-level (e.g. string
instances) will be reclaimed, so do not call if there is any
static data in the instance.
Should work for any netcdf format.
*/
EXTERNL int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count);
EXTERNL int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count);
/**
Copy vector of arbitrary type instances. This recursively walks
the top-level instances to copy any nested data such as vlen or
strings or such.
Assumes it is passed a pointer to count instances of xtype.
WARNING: nc_copy_data does not copy the top-level memory, but
assumes a block of proper size was passed in. However
nc_copy_data_all does allocate top-level memory copy.
Should work for any netcdf format.
*/
EXTERNL int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy);
EXTERNL int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp);
/* Instance dumper for debugging */
EXTERNL int nc_dump_data(int ncid, nc_type xtypeid, void* memory, size_t count, char** buf);
/* end recursive instance walking functions */
/* Begin Deprecated, same as functions with "_ubyte" replaced by "_uchar" */
EXTERNL int
nc_put_att_ubyte(int ncid, int varid, const char *name, nc_type xtype,
@ -1788,8 +1840,10 @@ nc_get_varm_ubyte(int ncid, int varid, const size_t *startp,
const ptrdiff_t * imapp, unsigned char *ip);
EXTERNL int
nc_put_var_ubyte(int ncid, int varid, const unsigned char *op);
EXTERNL int
nc_get_var_ubyte(int ncid, int varid, unsigned char *ip);
/* End Deprecated */
/* Set the log level. 0 shows only errors, 1 only major messages,

54
include/netcdf_aux.h
View File

@ -24,35 +24,30 @@ extern "C" {
/**
Reclaim the output tree of data from a call
to e.g. nc_get_vara or the input to e.g. nc_put_vara.
Reclaim a vector of instances of arbitrary type.
Intended for use with e.g. nc_get_vara or the input to e.g. nc_put_vara.
This recursively walks the top-level instances to
reclaim any nested data such as vlen or strings or such.
Assumes it is passed a pointer to count instances of xtype.
Reclaims any nested data.
WARNING: does not reclaim the top-level memory because
we do not know how it was allocated.
WARNING: ncaux_reclaim_data does not reclaim the top-level memory
because we do not know how it was allocated.
However ncaux_reclaim_data_all does reclaim top-level memory.
WARNING: all data blocks below the top-level (e.g. string instances)
will be reclaimed, so do not call if there is any static data in the instance.
Should work for any netcdf format.
WARNING: deprecated in favor the corresponding function in netcdf.h.
These are just wrappers for nc_reclaim_data and
nc_reclaim_data_all and nc_copy_data and nc_copy_data_all and
are here for back compatibilty.
*/
EXTERNL int ncaux_reclaim_data(int ncid, int xtype, void* memory, size_t count);
EXTERNL int ncaux_begin_compound(int ncid, const char *name, int alignmode, void** tag);
EXTERNL int ncaux_end_compound(void* tag, nc_type* typeid);
EXTERNL int ncaux_abort_compound(void* tag);
EXTERNL int ncaux_add_field(void* tag, const char *name, nc_type field_type,
int ndims, const int* dimsizes);
/* Takes any type */
EXTERNL size_t ncaux_type_alignment(int xtype, int ncid);
/* Takes type classes only */
EXTERNL size_t ncaux_class_alignment(int ncclass);
EXTERNL int ncaux_reclaim_data_all(int ncid, int xtype, void* memory, size_t count);
/**************************************************/
/* Capture the id and parameters for a filter
@ -70,11 +65,30 @@ EXTERNL int ncaux_h5filterspec_parse_parameter(const char* txt, size_t* nuiparam
EXTERNL void ncaux_h5filterspec_free(struct NC_H5_Filterspec* f);
EXTERNL void ncaux_h5filterspec_fix8(unsigned char* mem, int decode);
/**************************************************/
/* Wrappers to export selected functions from libnetcdf */
EXTERNL int ncaux_readfile(const char* filename, size_t* sizep, void** content);
EXTERNL int ncaux_writefile(const char* filename, size_t size, void* content);
/**************************************************/
/* Takes any type */
EXTERNL int ncaux_type_alignment(int xtype, int ncid, size_t* alignp);
EXTERNL int ncaux_class_alignment(int ncclass, size_t* alignp);
/**************************************************/
/* Takes type classes only */
/* Build compound types and properly handle offset and alignment */
EXTERNL int ncaux_class_alignment(int ncclass, size_t* alignp);
EXTERNL int ncaux_begin_compound(int ncid, const char *name, int alignmode, void** tag);
EXTERNL int ncaux_end_compound(void* tag, nc_type* xtypeid);
EXTERNL int ncaux_abort_compound(void* tag);
EXTERNL int ncaux_add_field(void* tag, const char *name, nc_type field_type,
int ndims, const int* dimsizes);
#if defined(__cplusplus)
}
#endif

1389
libdap2/dcetab.c
View File

File diff suppressed because it is too large Load Diff

24
libdap2/dcetab.h
View File

@ -1,9 +1,8 @@
/* A Bison parser, made by GNU Bison 3.7.6. */
/* A Bison parser, made by GNU Bison 3.0.4. */
/* Bison interface for Yacc-like parsers in C
Copyright (C) 1984, 1989-1990, 2000-2015, 2018-2021 Free Software Foundation,
Inc.
Copyright (C) 1984, 1989-1990, 2000-2015 Free Software Foundation, Inc.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@ -16,7 +15,7 @@
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>. */
along with this program. If not, see <http://www.gnu.org/licenses/>. */
/* As a special exception, you may create a larger work that contains
part or all of the Bison parser skeleton and distribute that work
@ -31,10 +30,6 @@
This special exception was added by the Free Software Foundation in
version 2.2 of Bison. */
/* DO NOT RELY ON FEATURES THAT ARE NOT DOCUMENTED in the manual,
especially those whose name start with YY_ or yy_. They are
private implementation details that can be changed or removed. */
#ifndef YY_DCE_DCE_TAB_H_INCLUDED
# define YY_DCE_DCE_TAB_H_INCLUDED
/* Debug traces. */
@ -45,20 +40,15 @@
extern int dcedebug;
#endif
/* Token kinds. */
/* Token type. */
#ifndef YYTOKENTYPE
# define YYTOKENTYPE
enum yytokentype
{
YYEMPTY = -2,
YYEOF = 0, /* "end of file" */
YYerror = 256, /* error */
YYUNDEF = 257, /* "invalid token" */
SCAN_WORD = 258, /* SCAN_WORD */
SCAN_STRINGCONST = 259, /* SCAN_STRINGCONST */
SCAN_NUMBERCONST = 260 /* SCAN_NUMBERCONST */
SCAN_WORD = 258,
SCAN_STRINGCONST = 259,
SCAN_NUMBERCONST = 260
};
typedef enum yytokentype yytoken_kind_t;
#endif
/* Value type. */

14
libdap4/d4file.c
View File

@ -525,7 +525,7 @@ makesubstrate(NCD4INFO* d4info)
int ncid = 0;
int ncflags = NC_NETCDF4|NC_CLOBBER;
if(d4info->substrate.filename != NULL) {
if(d4info->substrate.nc4id != 0) {
/* reset the substrate */
nc_abort(d4info->substrate.nc4id);
d4info->substrate.nc4id = 0;
@ -550,3 +550,15 @@ makesubstrate(NCD4INFO* d4info)
return THROW(ret);
}
int
NCD4_get_substrate(int ncid)
{
NC* nc = NULL;
NCD4INFO* d4 = NULL;
int subncid = 0;
/* Find pointer to NC struct for this file. */
(void)NC_check_id(ncid,&nc);
d4 = (NCD4INFO*)nc->dispatchdata;
subncid = d4->substrate.nc4id;
return subncid;
}

2
libdap4/ncd4.h
View File

@ -234,5 +234,7 @@ EXTERNL d4size_t NCD4_getcounter(void* p);
*/
#define HYRAXHACK
EXTERNL int NCD4_get_substrate(int ncid);
#endif /*NCD4_H*/

3
libdispatch/CMakeLists.txt
View File

@ -4,7 +4,8 @@
# University Corporation for Atmospheric Research/Unidata.
# See netcdf-c/COPYRIGHT file for more info.
SET(libdispatch_SOURCES dparallel.c dcopy.c dfile.c ddim.c datt.c dattinq.c dattput.c dattget.c derror.c dvar.c dvarget.c dvarput.c dvarinq.c ddispatch.c nclog.c dstring.c dutf8.c dinternal.c doffsets.c ncuri.c nclist.c ncbytes.c nchashmap.c nctime.c nc.c nclistmgr.c utf8proc.h utf8proc.c dpathmgr.c dutil.c drc.c dauth.c dreadonly.c dnotnc4.c dnotnc3.c daux.c dinfermodel.c
SET(libdispatch_SOURCES dparallel.c dcopy.c dfile.c ddim.c datt.c dattinq.c dattput.c dattget.c derror.c dvar.c dvarget.c dvarput.c dvarinq.c ddispatch.c nclog.c dstring.c dutf8.c dinternal.c doffsets.c ncuri.c nclist.c ncbytes.c nchashmap.c nctime.c nc.c nclistmgr.c utf8proc.h utf8proc.c dpathmgr.c dutil.c drc.c dauth.c dreadonly.c dnotnc4.c dnotnc3.c dinfermodel.c
daux.c dinstance.c
dcrc32.c dcrc32.h dcrc64.c ncexhash.c ncxcache.c ncjson.c ds3util.c)
# Netcdf-4 only functions. Must be defined even if not used

2
libdispatch/Makefile.am
View File

@ -20,7 +20,7 @@ dattinq.c dattput.c dattget.c derror.c dvar.c dvarget.c dvarput.c \
dvarinq.c dinternal.c ddispatch.c dutf8.c nclog.c dstring.c ncuri.c \
nclist.c ncbytes.c nchashmap.c nctime.c nc.c nclistmgr.c dauth.c \
doffsets.c dpathmgr.c dutil.c dreadonly.c dnotnc4.c dnotnc3.c \
daux.c dinfermodel.c \
dinfermodel.c daux.c dinstance.c \
dcrc32.c dcrc32.h dcrc64.c ncexhash.c ncxcache.c ncjson.c ds3util.c
# Add the utf8 codebase

326
libdispatch/daux.c
View File

@ -51,225 +51,11 @@ struct NCAUX_CMPD {
size_t alignment;
};
/* It is helpful to have a structure that contains memory and an offset */
typedef struct Position{char* memory; ptrdiff_t offset;} Position;
/* Forward */
static int reclaim_datar(int ncid, int xtype, size_t typesize, Position*);
static int ncaux_initialized = 0;
#ifdef USE_NETCDF4
static int reclaim_usertype(int ncid, int xtype, Position* offset);
static int reclaim_compound(int ncid, int xtype, size_t size, size_t nfields, Position* offset);
static int reclaim_vlen(int ncid, int xtype, int basetype, Position* offset);
static int reclaim_enum(int ncid, int xtype, int basetype, size_t, Position* offset);
static int reclaim_opaque(int ncid, int xtype, size_t size, Position* offset);
static int computefieldinfo(struct NCAUX_CMPD* cmpd);
#endif /* USE_NETCDF4 */
static int filterspec_cvt(const char* txt, size_t* nparamsp, unsigned int* params);
/**************************************************/
/**
Reclaim the output tree of data from a call
to e.g. nc_get_vara or the input to e.g. nc_put_vara.
This recursively walks the top-level instances to
reclaim any nested data such as vlen or strings or such.
Assumes it is passed a pointer to count instances of xtype.
Reclaims any nested data.
WARNING: does not reclaim the top-level memory because
we do not know how it was allocated.
Should work for any netcdf format.
@param ncid file ncid
@param xtype type id
@param memory to reclaim
@param count number of instances of the type in memory
@return error code
*/
EXTERNL int
ncaux_reclaim_data(int ncid, int xtype, void* memory, size_t count)
{
int stat = NC_NOERR;
size_t typesize = 0;
size_t i;
Position offset;
if(ncid < 0 || xtype < 0
|| (memory == NULL && count > 0)
|| xtype == NC_NAT)
{stat = NC_EINVAL; goto done;}
if(memory == NULL || count == 0)
goto done; /* ok, do nothing */
if((stat=nc_inq_type(ncid,xtype,NULL,&typesize))) goto done;
offset.memory = (char*)memory; /* use char* so we can do pointer arithmetic */
offset.offset = 0;
for(i=0;i<count;i++) {
if((stat=reclaim_datar(ncid,xtype,typesize,&offset))) /* reclaim one instance */
break;
}
done:
return stat;
}
/* Recursive type walker: reclaim a single instance */
static int
reclaim_datar(int ncid, int xtype, size_t typesize, Position* offset)
{
int stat = NC_NOERR;
switch (xtype) {
case NC_CHAR: case NC_BYTE: case NC_UBYTE:
case NC_SHORT: case NC_USHORT:
case NC_INT: case NC_UINT: case NC_FLOAT:
case NC_INT64: case NC_UINT64: case NC_DOUBLE:
offset->offset += typesize;
break;
#ifdef USE_NETCDF4
case NC_STRING: {
char** sp = (char**)(offset->memory + offset->offset);
/* Need to reclaim string */
if(*sp != NULL) free(*sp);
offset->offset += typesize;
} break;
default:
/* reclaim a user type */
stat = reclaim_usertype(ncid,xtype,offset);
#else
default:
stat = NC_ENOTNC4;
#endif
break;
}
return stat;
}
#ifdef USE_NETCDF4
static ptrdiff_t
read_align(ptrdiff_t offset, size_t alignment)
{
size_t loc_align = (alignment == 0 ? 1 : alignment);
size_t delta = (offset % loc_align);
if(delta == 0) return offset;
return offset + (alignment - delta);
}
static int
reclaim_usertype(int ncid, int xtype, Position* offset)
{
int stat = NC_NOERR;
size_t size;
nc_type basetype;
size_t nfields;
int klass;
/* Get info about the xtype */
stat = nc_inq_user_type(ncid, xtype, NULL, &size, &basetype, &nfields, &klass);
switch (klass) {
case NC_OPAQUE: stat = reclaim_opaque(ncid,xtype,size,offset); break;
case NC_ENUM: stat = reclaim_enum(ncid,xtype,basetype,size,offset); break;
case NC_COMPOUND: stat = reclaim_compound(ncid,xtype,size,nfields,offset); break;
case NC_VLEN: stat = reclaim_vlen(ncid,xtype,basetype,offset); break;
default:
stat = NC_EINVAL;
break;
}
return stat;
}
static int
reclaim_vlen(int ncid, int xtype, int basetype, Position* offset)
{
int stat = NC_NOERR;
size_t i, basesize;
nc_vlen_t* vl = (nc_vlen_t*)(offset->memory+offset->offset);
/* Get size of the basetype */
if((stat=nc_inq_type(ncid,basetype,NULL,&basesize))) goto done;
/* Free up each entry in the vlen list */
if(vl->p != NULL) {
Position voffset;
unsigned int alignment = ncaux_type_alignment(basetype,ncid);
voffset.memory = vl->p;
voffset.offset = 0;
for(i=0;i<vl->len;i++) {
voffset.offset = read_align(voffset.offset,alignment);
if((stat = reclaim_datar(ncid,basetype,basesize,&voffset))) goto done;
}
offset->offset += sizeof(nc_vlen_t);
free(vl->p);
}
done:
return stat;
}
static int
reclaim_enum(int ncid, int xtype, int basetype, size_t basesize, Position* offset)
{
/* basically same as an instance of the enum's integer basetype */
return reclaim_datar(ncid,basetype,basesize,offset);
}
static int
reclaim_opaque(int ncid, int xtype, size_t opsize, Position* offset)
{
/* basically a fixed size sequence of bytes */
offset->offset += opsize;
return NC_NOERR;
}
static int
reclaim_compound(int ncid, int xtype, size_t cmpdsize, size_t nfields, Position* offset)
{
int stat = NC_NOERR;
size_t fid, fieldoffset, i, fieldsize, arraycount;
int dimsizes[NC_MAX_VAR_DIMS];
int ndims;
nc_type fieldtype;
ptrdiff_t saveoffset;
saveoffset = offset->offset;
/* Get info about each field in turn and reclaim it */
for(fid=0;fid<nfields;fid++) {
unsigned int fieldalignment;
/* Get all relevant info about the field */
if((stat = nc_inq_compound_field(ncid,xtype,fid,NULL,&fieldoffset, &fieldtype, &ndims, dimsizes))) goto done;
fieldalignment = ncaux_type_alignment(fieldtype,ncid);
if((stat = nc_inq_type(ncid,fieldtype,NULL,&fieldsize))) goto done;
if(ndims == 0) {ndims=1; dimsizes[0]=1;} /* fake the scalar case */
/* Align to this field */
offset->offset = read_align(offset->offset,fieldalignment);
/* compute the total number of elements in the field array */
arraycount = 1;
for(i=0;i<ndims;i++) arraycount *= dimsizes[i];
for(i=0;i<arraycount;i++) {
if((stat = reclaim_datar(ncid, fieldtype, fieldsize, offset))) goto done;
}
}
/* Return to beginning of the compound and move |compound| */
offset->offset = saveoffset;
offset->offset += cmpdsize;
done:
return stat;
}
#endif /*USE_NETCDF4*/
/**************************************************/
/*
This code is a variant of the H5detect.c code from HDF5.
Author: D. Heimbigner 10/7/2008
@ -411,49 +197,22 @@ done:
/**
@param ncclass - type class for which alignment is requested; excludes ENUM|COMPOUND
*/
size_t
ncaux_class_alignment(int ncclass)
int
ncaux_class_alignment(int ncclass, size_t* alignp)
{
if(ncclass <= NC_MAX_ATOMIC_TYPE || ncclass == NC_VLEN || ncclass == NC_OPAQUE)
return NC_class_alignment(ncclass);
nclog(NCLOGERR,"ncaux_class_alignment: class %d; alignment cannot be determermined",ncclass);
return 0;
int stat = NC_NOERR;
size_t align = 0;
if(ncclass <= NC_MAX_ATOMIC_TYPE || ncclass == NC_VLEN || ncclass == NC_OPAQUE) {
stat = NC_class_alignment(ncclass,&align);
} else {
nclog(NCLOGERR,"ncaux_class_alignment: class %d; alignment cannot be determermined",ncclass);
}
if(alignp) *alignp = align;
if(align == 0) stat = NC_EINVAL;
return stat;
}
/**
@param ncid - only needed for a compound type
@param xtype - type for which alignment is requested
*/
size_t
ncaux_type_alignment(int xtype, int ncid)
{
if(!ncaux_initialized) {
NC_compute_alignments();
ncaux_initialized = 1;
}
if(xtype <= NC_MAX_ATOMIC_TYPE)
return NC_class_alignment(xtype); /* type == class */
#ifdef USE_NETCDF4
else {/* Presumably a user type */
int klass = NC_NAT;
int stat = nc_inq_user_type(ncid,xtype,NULL,NULL,NULL,NULL,&klass);
if(stat) goto done;
switch(klass) {
case NC_VLEN: return NC_class_alignment(klass);
case NC_OPAQUE: return NC_class_alignment(klass);
case NC_COMPOUND: {/* get alignment of the first field of the compound */
int fieldtype = NC_NAT;
if((stat=nc_inq_compound_fieldtype(ncid,xtype,0,&fieldtype))) goto done;
return ncaux_type_alignment(fieldtype,ncid); /* may recurse repeatedly */
} break;
default: break;
}
}
done:
#endif /*USE_NETCDF4 */
return 0; /* fail */
}
#ifdef USE_NETCDF4
/* Find first primitive field of a possibly nested sequence of compounds */
@ -517,14 +276,14 @@ computefieldinfo(struct NCAUX_CMPD* cmpd)
field->alignment = 1;
break;
case NC_ENUM:
field->alignment = ncaux_type_alignment(firsttype,cmpd->ncid);
status = ncaux_type_alignment(firsttype,cmpd->ncid,&field->alignment);
break;
case NC_VLEN: /*fall thru*/
case NC_COMPOUND:
field->alignment = ncaux_type_alignment(firsttype,cmpd->ncid);
status = ncaux_type_alignment(firsttype,cmpd->ncid,&field->alignment);
break;
default:
field->alignment = ncaux_type_alignment(field->fieldtype,cmpd->ncid);
status = ncaux_type_alignment(field->fieldtype,cmpd->ncid,&field->alignment);
break;
}
@ -979,9 +738,6 @@ done:
return stat;
}
#if 0
/*
Parse a filter spec string into a NC_H5_Filterspec*
@ -1118,3 +874,53 @@ ncaux_writefile(const char* filename, size_t size, void* content)
{
return NC_writefile(filename,size,content);
}
/**************************************************/
/**
Reclaim the output tree of data from a call
to e.g. nc_get_vara or the input to e.g. nc_put_vara.
This recursively walks the top-level instances to
reclaim any nested data such as vlen or strings or such.
This function is just a wrapper around nc_reclaim_data.
@param ncid file ncid
@param xtype type id
@param memory to reclaim
@param count number of instances of the type in memory
@return error code
*/
EXTERNL int
ncaux_reclaim_data(int ncid, int xtype, void* memory, size_t count)
{
/* Defer to the internal version */
return nc_reclaim_data(ncid, xtype, memory, count);
}
/*
This function is just a wrapper around nc_reclaim_data_all.
@param ncid file ncid
@param xtype type id
@param memory to reclaim
@param count number of instances of the type in memory
@return error code
*/
EXTERNL int
ncaux_reclaim_data_all(int ncid, int xtype, void* memory, size_t count)
{
/* Defer to the internal version */
return nc_reclaim_data_all(ncid, xtype, memory, count);
}
/**
@param ncid - only needed for a compound type
@param xtype - type for which alignment is requested
*/
int
ncaux_type_alignment(int xtype, int ncid, size_t* alignp)
{
/* Defer to the internal version */
return NC_type_alignment(ncid, xtype, alignp);
}

31
libdispatch/dcopy.c
View File

@ -520,6 +520,7 @@ NC_copy_att(int ncid_in, int varid_in, const char *name,
if ((res = nc_inq_att(ncid_in, varid_in, name, &xtype, &len)))
return res;
#ifdef SEPDATA
if (xtype < NC_STRING)
{
/* Handle non-string atomic types. */
@ -600,6 +601,36 @@ NC_copy_att(int ncid_in, int varid_in, const char *name,
}
}
#endif /*!USE_NETCDF4*/
#else /*!SEPDATA*/
{
/* Copy arbitrary attributes. */
int class;
size_t size;
nc_type xtype_out = NC_NAT;
if(xtype <= NC_MAX_ATOMIC_TYPE) {
xtype_out = xtype;
if((res = nc_inq_type(ncid_out,xtype_out,NULL,&size))) return res;
} else { /* User defined type */
/* Find out if there is an equal type in the output file. */
/* Note: original code used a libsrc4 specific internal function
which we had to "duplicate" here */
if ((res = NC_find_equal_type(ncid_in, xtype, ncid_out, &xtype_out)))
return res;
if (xtype_out) {
/* We found an equal type! */
if ((res = nc_inq_user_type(ncid_in, xtype, NULL, &size, NULL, NULL, &class)))
return res;
}
}
if((data = malloc(size * len))==NULL) {return NC_ENOMEM;}
res = nc_get_att(ncid_in, varid_in, name, data);
if(!res)
res = nc_put_att(ncid_out, varid_out, name, xtype_out, len, data);
(void)nc_reclaim_data_all(ncid_out,xtype_out,data,len);
}
#endif /*SEPDATA*/
return res;
}

883
libdispatch/dinstance.c Normal file
View File

@ -0,0 +1,883 @@
/*
Copyright (c) 1998-2018 University Corporation for Atmospheric Research/Unidata
See COPYRIGHT for license information.
*/
/*
This file contains various instance operations that operate
on a deep level rather than the shallow level of e.g. nc_free_vlen_t.
Currently two operations are defined:
1. reclaim a vector of instances
2. copy a vector of instances
*/
#include "config.h"
#include <stdlib.h>
#include <string.h>
#include <assert.h>
#include "netcdf.h"
#include "nc4internal.h"
#include "nc4dispatch.h"
#include "ncoffsets.h"
#include "ncbytes.h"
#undef REPORT
#undef DEBUG
/* It is helpful to have a structure that contains memory and an offset */
typedef struct Position{char* memory; ptrdiff_t offset;} Position;
static int type_alignment_initialized = 0;
/* Forward */
#ifdef USE_NETCDF4
static int reclaim_datar(int ncid, nc_type xtype, Position*);
static int reclaim_compound(int ncid, nc_type xtype, size_t size, size_t nfields, Position* offset);
static int reclaim_vlen(int ncid, nc_type xtype, nc_type basetype, Position* offset);
static int reclaim_enum(int ncid, nc_type xtype, nc_type basetype, Position* offset);
static int reclaim_opaque(int ncid, nc_type xtype, size_t size, Position* offset);
static int copy_datar(int ncid, nc_type xtype, Position* src, Position* dst);
static int copy_compound(int ncid, nc_type xtype, size_t size, size_t nfields, Position* src, Position* dst);
static int copy_vlen(int ncid, nc_type xtype, nc_type basetype, Position* src, Position* dst);
static int copy_enum(int ncid, nc_type xtype, nc_type basetype, Position* src, Position* dst);
static int copy_opaque(int ncid, nc_type xtype, size_t size, Position* src,Position* dst);
static int dump_datar(int ncid, nc_type xtype, Position*, NCbytes* buf);
static int dump_compound(int ncid, nc_type xtype, size_t size, size_t nfields, Position* offset, NCbytes* buf);
static int dump_vlen(int ncid, nc_type xtype, nc_type basetype, Position* offset, NCbytes* buf);
static int dump_enum(int ncid, nc_type xtype, nc_type basetype, Position* offset, NCbytes* buf);
static int dump_opaque(int ncid, nc_type xtype, size_t size, Position* offset, NCbytes* buf);
static ptrdiff_t read_align(ptrdiff_t offset, size_t alignment);
#endif
static int NC4_inq_any_type(int ncid, nc_type typeid, char *name, size_t *size, nc_type *basetypep, size_t *nfieldsp, int *classp);
/**
Reclaim a vector of instances of a type. This improves upon
e.g. nc_free_vlen. This recursively walks the top-level
instances to reclaim any nested data such as vlen or strings or such.
Assumes it is passed a pointer to count instances of xtype.
Reclaims any nested data.
WARNING: This needs access to the type metadata of the file, so
a valid ncid and typeid must be available, which means the file
must not have been closed or aborted.
WARNING: DOES NOT RECLAIM THE TOP-LEVEL MEMORY (see the
nc_reclaim_data_all function). The reason is that we do not
know how it was allocated (e.g. static vs dynamic); only the
caller can know that. But note that it assumes all memory
blocks other than the top were dynamically allocated, so they
will be free'd.
Should work for any netcdf format.
@param ncid root id
@param xtype type id
@param memory ptr to top-level memory to reclaim
@param count number of instances of the type in memory block
@return error code
*/
int
nc_reclaim_data(int ncid, nc_type xtype, void* memory, size_t count)
{
int stat = NC_NOERR;
size_t i;
Position offset;
int isf;
if(ncid < 0 || xtype <= 0)
{stat = NC_EINVAL; goto done;}
if(memory == NULL && count > 0)
{stat = NC_EINVAL; goto done;}
if(memory == NULL || count == 0)
goto done; /* ok, do nothing */
#ifdef REPORT
fprintf(stderr,">>> reclaim: memory=%p count=%lu ncid=%d xtype=%d\n",memory,(unsigned long)count,ncid,xtype);
#endif
/* Optimizations */
/* 1. Vector of fixed size types */
if((stat = NC4_inq_type_fixed_size(ncid,xtype,&isf))) goto done;
if(isf) goto done; /* no need to reclaim anything */
#ifdef USE_NETCDF4
/* 2.Vector of strings */
if(xtype == NC_STRING) {
char** ss = (char**)memory;
for(i=0;i<count;i++) {
nullfree(ss[i]);
}
goto done;
}
offset.memory = (char*)memory; /* use char* so we can do pointer arithmetic */
offset.offset = 0;
for(i=0;i<count;i++) {
if((stat=reclaim_datar(ncid,xtype,&offset))) /* reclaim one instance */
break;
}
#else
stat = NC_EBADTYPE;
#endif
done:
return stat;
}
/* Alternate entry point: includes recovering the top-level memory */
int
nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count)
{
int stat = NC_NOERR;
stat = nc_reclaim_data(ncid,xtypeid,memory,count);
if(stat == NC_NOERR && memory != NULL)
free(memory);
return stat;
}
#ifdef USE_NETCDF4
/* Recursive type walker: reclaim a single instance */
static int
reclaim_datar(int ncid, nc_type xtype, Position* offset)
{
int stat = NC_NOERR;
size_t xsize;
nc_type basetype;
size_t nfields;
int klass, isf;
if((stat = NC4_inq_type_fixed_size(ncid,xtype,&isf))) goto done;
if(isf) { /* no need to reclaim anything */
offset->offset += xsize;
goto done;
}
/* Get relevant type info */
if((stat = NC4_inq_any_type(ncid,xtype,NULL,&xsize,&basetype,&nfields,&klass))) goto done;
switch (xtype) {
case NC_STRING: {
char** sp = (char**)(offset->memory + offset->offset);
/* Need to reclaim string */
if(*sp != NULL) free(*sp);
offset->offset += xsize;
} break;
default:
/* reclaim a user type */
switch (klass) {
case NC_OPAQUE: stat = reclaim_opaque(ncid,xtype,xsize,offset); break;
case NC_ENUM: stat = reclaim_enum(ncid,xtype,basetype,offset); break;
case NC_COMPOUND: stat = reclaim_compound(ncid,xtype,xsize,nfields,offset); break;
case NC_VLEN:
stat = reclaim_vlen(ncid,xtype,basetype,offset);
break;
default:
stat = NC_EINVAL;
break;
}
break;
}
done:
return stat;
}
static int
reclaim_vlen(int ncid, nc_type xtype, nc_type basetype, Position* offset)
{
int stat = NC_NOERR;
size_t i;
nc_vlen_t* vl = (nc_vlen_t*)(offset->memory+offset->offset);
if(vl->len > 0 && vl->p == NULL)
{stat = NC_EINVAL; goto done;}
/* Free up each entry in the vlen list */
if(vl->len > 0) {
Position voffset;
size_t alignment = 0;
if((stat = NC_type_alignment(ncid,basetype,&alignment))) goto done;;
voffset.memory = vl->p;
voffset.offset = 0;
for(i=0;i<vl->len;i++) {
voffset.offset = read_align(voffset.offset,alignment);
if((stat = reclaim_datar(ncid,basetype,&voffset))) goto done;
}
free(vl->p);
}
offset->offset += sizeof(nc_vlen_t);
done:
return stat;
}
static int
reclaim_enum(int ncid, nc_type xtype, nc_type basetype, Position* offset)
{
int stat = NC_NOERR;
abort();
/* basically same as an instance of the enum's integer basetype */
stat = reclaim_datar(ncid,basetype,offset);
return stat;
}
static int
reclaim_opaque(int ncid, nc_type xtype, size_t size, Position* offset)
{
abort();
/* basically a fixed size sequence of bytes */
offset->offset += size;
return NC_NOERR;
}
static int
reclaim_compound(int ncid, nc_type xtype, size_t size, size_t nfields, Position* offset)
{
int stat = NC_NOERR;
size_t fid, i, arraycount;
ptrdiff_t saveoffset;
int ndims;
int dimsizes[NC_MAX_VAR_DIMS];
saveoffset = offset->offset;
/* Get info about each field in turn and reclaim it */
for(fid=0;fid<nfields;fid++) {
size_t fieldalignment;
nc_type fieldtype;
/* Get all relevant info about the field */
if((stat = nc_inq_compound_field(ncid,xtype,fid,NULL,&fieldalignment,&fieldtype,&ndims,dimsizes))) goto done;
if(ndims == 0) {ndims=1; dimsizes[0]=1;} /* fake the scalar case */
/* Align to this field */
offset->offset = saveoffset + fieldalignment;
/* compute the total number of elements in the field array */
arraycount = 1;
for(i=0;i<ndims;i++) arraycount *= dimsizes[i];
for(i=0;i<arraycount;i++) {
if((stat = reclaim_datar(ncid, fieldtype, offset))) goto done;
}
}
/* Return to beginning of the compound and move |compound| */
offset->offset = saveoffset;
offset->offset += size;
done:
return stat;
}
#endif
/**************************************************/
/**
Copy a vector of instances of a type. This recursively walks
the top-level instances to copy any nested data such as vlen or
strings or such.
Assumes it is passed a pointer to count instances of xtype and a
space into which to copy the instance. Copys any nested data.
WARNING: This needs access to the type metadata of the file, so
a valid ncid and typeid must be available, which means the file
must not have been closed or aborted.
WARNING: DOES NOT ALLOCATE THE TOP-LEVEL MEMORY (see the
nc_copy_data_all function). Note that all memory blocks other
than the top are dynamically allocated.
Should work for any netcdf format.
@param ncid root id
@param xtype type id
@param memory ptr to top-level memory to copy
@param count number of instances of the type in memory block
@param copy top-level space into which to copy the instance
@return error code
*/
int
nc_copy_data(int ncid, nc_type xtype, const void* memory, size_t count, void* copy)
{
int stat = NC_NOERR;
size_t i;
Position src;
Position dst;
size_t xsize;
int isf;
if(ncid < 0 || xtype <= 0)
{stat = NC_EINVAL; goto done;}
if(memory == NULL && count > 0)
{stat = NC_EINVAL; goto done;}
if(copy == NULL && count > 0)
{stat = NC_EINVAL; goto done;}
if(memory == NULL || count == 0)
goto done; /* ok, do nothing */
#ifdef REPORT
fprintf(stderr,">>> copy : copy =%p memory=%p count=%lu ncid=%d xtype=%d\n",copy,memory,(unsigned long)count,ncid,xtype);
#endif
/* Get type size */
if((stat = NC4_inq_any_type(ncid,xtype,NULL,&xsize,NULL,NULL,NULL))) goto done;
/* Optimizations */
/* 1. Vector of fixed sized objects */
if((stat = NC4_inq_type_fixed_size(ncid,xtype,&isf))) goto done;
if(isf) {
memcpy(copy,memory,xsize*count);
goto done;
}
#ifdef USE_NETCDF4
src.memory = (char*)memory; /* use char* so we can do pointer arithmetic */
src.offset = 0;
dst.memory = (char*)copy; /* use char* so we can do pointer arithmetic */
dst.offset = 0;
for(i=0;i<count;i++) {
if((stat=copy_datar(ncid,xtype,&src,&dst))) /* copy one instance copy_datar will increment src and dst*/
break;
}
#else
stat = NC_EBADTYPE;
#endif
done:
return stat;
}
/* Alternate entry point: includes recovering the top-level memory */
int
nc_copy_data_all(int ncid, nc_type xtype, const void* memory, size_t count, void** copyp)
{
int stat = NC_NOERR;
size_t xsize = 0;
void* copy = NULL;
/* Get type size */
if((stat = NC4_inq_any_type(ncid,xtype,NULL,&xsize,NULL,NULL,NULL))) goto done;
/* allocate the top-level */
if(count > 0) {
if((copy = calloc(xsize,count))==NULL)
{stat = NC_ENOMEM; goto done;}
}
stat = nc_copy_data(ncid,xtype,memory,count,copy);
if(copyp) {*copyp = copy; copy = NULL;}
done:
if(copy)
stat = nc_reclaim_data_all(ncid,xtype,copy,count);
return stat;
}
#ifdef USE_NETCDF4
/* Recursive type walker: copy a single instance */
static int
copy_datar(int ncid, nc_type xtype, Position* src, Position* dst)
{
int stat = NC_NOERR;
size_t xsize;
nc_type basetype;
size_t nfields;
int xclass,isf;
if((stat = NC4_inq_any_type(ncid,xtype,NULL,&xsize,&basetype,&nfields,&xclass))) goto done;
/* Optimizations */
/* 1. Vector of fixed size types */
if((stat = NC4_inq_type_fixed_size(ncid,xtype,&isf))) goto done;
if(isf) {
memcpy(dst->memory+dst->offset,src->memory+src->offset,xsize*1);
src->offset += xsize;
dst->offset += xsize;
goto done;
}
switch (xtype) {
case NC_STRING: {
char** sp = (char**)(src->memory + src->offset);
char* copy = NULL;
/* Need to copy string */
if(*sp != NULL) {
if((copy = strdup(*sp))==NULL) {stat = NC_ENOMEM; goto done;}
}
memcpy(dst->memory+dst->offset,(void*)&copy,sizeof(char*));
src->offset += xsize;
dst->offset += xsize;
} break;
default:
/* copy a user type */
switch (xclass) {
case NC_OPAQUE: stat = copy_opaque(ncid,xtype,xsize,src,dst); break;
case NC_ENUM: stat = copy_enum(ncid,xtype,basetype,src,dst); break;
case NC_COMPOUND: stat = copy_compound(ncid,xtype,xsize,nfields,src,dst); break;
case NC_VLEN: stat = copy_vlen(ncid,xtype,basetype,src,dst); break;
default: stat = NC_EINVAL; break;
}
break;
}
done:
return stat;
}
static int
copy_vlen(int ncid, nc_type xtype, nc_type basetype, Position* src, Position* dst)
{
int stat = NC_NOERR;
size_t i, basetypesize;
nc_vlen_t* vl = (nc_vlen_t*)(src->memory+src->offset);
nc_vlen_t copy = {0,NULL};
if(vl->len > 0 && vl->p == NULL)
{stat = NC_EINVAL; goto done;}
/* Get basetype info */
if((stat = NC4_inq_any_type(ncid,basetype,NULL,&basetypesize,NULL,NULL,NULL))) goto done;
/* Make space in the copy vlen */
if(vl->len > 0) {
copy.len = vl->len;
if((copy.p = calloc(copy.len,basetypesize))==NULL) {stat = NC_ENOMEM; goto done;}
}
/* Copy each entry in the vlen list */
if(vl->len > 0) {
Position vsrc, vdst;
size_t alignment = 0;
if((stat = NC_type_alignment(ncid,basetype,&alignment))) goto done;;
vsrc.memory = vl->p;
vsrc.offset = 0;
vdst.memory = copy.p;
vdst.offset = 0;
for(i=0;i<vl->len;i++) {
vsrc.offset= read_align(vsrc.offset,alignment);
vdst.offset= read_align(vdst.offset,alignment);
if((stat = copy_datar(ncid,basetype,&vsrc,&vdst))) goto done;
}
}
/* Move into place */
memcpy(dst->memory+dst->offset,&copy,sizeof(nc_vlen_t));
src->offset += sizeof(nc_vlen_t);
dst->offset += sizeof(nc_vlen_t);
done:
if(stat) {
nullfree(copy.p);
}
return stat;
}
static int
copy_enum(int ncid, nc_type xtype, nc_type basetype, Position* src, Position* dst)
{
int stat = NC_NOERR;
abort();
/* basically same as an instance of the enum's integer basetype */
stat = copy_datar(ncid,basetype,src,dst);
return stat;
}
static int
copy_opaque(int ncid, nc_type xtype, size_t size, Position* src, Position* dst)
{
abort();
/* basically a fixed size sequence of bytes */
memcpy(dst->memory+dst->offset,src->memory+src->offset,size);
src->offset += size;
dst->offset += size;
return NC_NOERR;
}
static int
copy_compound(int ncid, nc_type xtype, size_t size, size_t nfields, Position* src, Position* dst)
{
int stat = NC_NOERR;
size_t fid, i, arraycount;
ptrdiff_t savesrcoffset, savedstoffset;
int ndims;
int dimsizes[NC_MAX_VAR_DIMS];
savesrcoffset = src->offset;
savedstoffset = dst->offset;
/* Get info about each field in turn and copy it */
for(fid=0;fid<nfields;fid++) {
size_t fieldoffset;
nc_type fieldtype;
char name[NC_MAX_NAME];
/* Get all relevant info about the field */
if((stat = nc_inq_compound_field(ncid,xtype,fid,name,&fieldoffset,&fieldtype,&ndims,dimsizes))) goto done;
if(ndims == 0) {ndims=1; dimsizes[0]=1;} /* fake the scalar case */
/* Set offset for this field */
#ifdef DEBUG
fprintf(stderr,"before: offset = %d after: offset = %d\n",(int)src->offset,(int)(savesrcoffset+fieldoffset));
#endif
src->offset = savesrcoffset+fieldoffset;
dst->offset = savedstoffset+fieldoffset;
#ifdef DEBUG
fprintf(stderr,"field %s(%d) = %d\n",name,(int)fieldoffset,(int)src->offset);
#endif
/* compute the total number of elements in the field array */
arraycount = 1;
for(i=0;i<ndims;i++) arraycount *= dimsizes[i];
for(i=0;i<arraycount;i++) {
if((stat = copy_datar(ncid, fieldtype, src, dst))) goto done;
}
#ifdef DEBUG
fprintf(stderr,"src=(%d,%p)\n",(int)src->offset,src->memory);
#endif
}
#ifdef DEBUG
fprintf(stderr,"\n");
#endif
/* Return to beginning of the compound and move |compound| */
src->offset = savesrcoffset;
dst->offset = savedstoffset;
src->offset += size;
dst->offset += size;
done:
return stat;
}
#endif
/**************************************************/
/* Alignment functions */
#ifdef USE_NETCDF4
static ptrdiff_t
read_align(ptrdiff_t offset, size_t alignment)
{
size_t loc_align = (alignment == 0 ? 1 : alignment);
size_t delta = (offset % loc_align);
if(delta == 0) return offset;
return offset + (alignment - delta);
}
/**
@param ncid - only needed for a compound type
@param xtype - type for which alignment is requested
@return 0 if not found
*/
int
NC_type_alignment(int ncid, nc_type xtype, size_t* alignp)
{
int stat = NC_NOERR;
size_t align = 0;
int klass;
if(!type_alignment_initialized) {
NC_compute_alignments();
type_alignment_initialized = 1;
}
if(xtype <= NC_MAX_ATOMIC_TYPE)
{stat = NC_class_alignment(xtype,&align); goto done;}
else {/* Presumably a user type */
if((stat = NC4_inq_any_type(ncid,xtype,NULL,NULL,NULL,NULL,&klass))) goto done;
switch(klass) {
case NC_VLEN: stat = NC_class_alignment(klass,&align); break;
case NC_OPAQUE: stat = NC_class_alignment(klass,&align); break;
case NC_COMPOUND: {/* get alignment of the first field of the compound */
nc_type fieldtype;
/* Get all relevant info about the first field */
if((stat = nc_inq_compound_field(ncid,xtype,0,NULL,NULL,&fieldtype,NULL,NULL))) goto done;
stat = NC_type_alignment(ncid,fieldtype,&align); /* may recurse repeatedly */
} break;
default: break;
}
}
if(alignp) *alignp = align;
done:
#if 0
Why was this here?
if(stat == NC_NOERR && align == 0) stat = NC_EINVAL;
#endif
return stat;
}
#endif
/**************************************************/
/* Dump an instance into a bytebuffer
@param ncid root id
@param xtype type id
@param memory ptr to top-level memory to dump
@param count number of instances of the type in memory block
@return error code
*/
int
nc_dump_data(int ncid, nc_type xtype, void* memory, size_t count, char** bufp)
{
int stat = NC_NOERR;
size_t i;
Position offset;
NCbytes* buf = ncbytesnew();
if(ncid < 0 || xtype <= 0)
{stat = NC_EINVAL; goto done;}
if(memory == NULL && count > 0)
{stat = NC_EINVAL; goto done;}
if(memory == NULL || count == 0)
goto done; /* ok, do nothing */
#ifdef REPORT
fprintf(stderr,">>> dump: memory=%p count=%lu ncid=%d xtype=%d\n",memory,(unsigned long)count,ncid,xtype);
#endif
offset.memory = (char*)memory; /* use char* so we can do pointer arithmetic */
offset.offset = 0;
for(i=0;i<count;i++) {
if(i > 0) ncbytescat(buf," ");
if((stat=dump_datar(ncid,xtype,&offset,buf))) /* dump one instance */
break;
}
if(bufp) *bufp = ncbytesextract(buf);
done:
ncbytesfree(buf);
return stat;
}
int
nc_print_data(int ncid, nc_type xtype, void* memory, size_t count)
{
char* s = NULL;
int stat = NC_NOERR;
if((stat=nc_dump_data(ncid,xtype,memory,count,&s))) return stat;
fprintf(stderr,"%s\n",s);
nullfree(s)
return stat;
}
/* Recursive type walker: dump a single instance */
static int
dump_datar(int ncid, nc_type xtype, Position* offset, NCbytes* buf)
{
int stat = NC_NOERR;
size_t xsize;
nc_type basetype;
size_t nfields;
int klass;
char s[128];
/* Get relevant type info */
if((stat = NC4_inq_any_type(ncid,xtype,NULL,&xsize,&basetype,&nfields,&klass))) goto done;
switch (xtype) {
case NC_CHAR:
snprintf(s,sizeof(s),"'%c'",*(char*)(offset->memory+offset->offset));
ncbytescat(buf,s);
break;
case NC_BYTE:
snprintf(s,sizeof(s),"%d",*(char*)(offset->memory+offset->offset));
ncbytescat(buf,s);
break;
case NC_UBYTE:
snprintf(s,sizeof(s),"%u",*(unsigned char*)(offset->memory+offset->offset));
ncbytescat(buf,s);
break;
case NC_SHORT:
snprintf(s,sizeof(s),"%d",*(short*)(offset->memory+offset->offset));
ncbytescat(buf,s);
break;
case NC_USHORT:
snprintf(s,sizeof(s),"%d",*(unsigned short*)(offset->memory+offset->offset));
ncbytescat(buf,s);
break;
case NC_INT:
snprintf(s,sizeof(s),"%d",*(int*)(offset->memory+offset->offset));
ncbytescat(buf,s);
break;
case NC_UINT:
snprintf(s,sizeof(s),"%d",*(unsigned int*)(offset->memory+offset->offset));
ncbytescat(buf,s);
break;
case NC_FLOAT:
snprintf(s,sizeof(s),"%f",*(float*)(offset->memory+offset->offset));
ncbytescat(buf,s);
break;
case NC_INT64:
snprintf(s,sizeof(s),"%lld",*(long long*)(offset->memory+offset->offset));
ncbytescat(buf,s);
break;
case NC_UINT64:
snprintf(s,sizeof(s),"%llu",*(unsigned long long*)(offset->memory+offset->offset));
ncbytescat(buf,s);
break;
case NC_DOUBLE:
snprintf(s,sizeof(s),"%lf",*(double*)(offset->memory+offset->offset));
ncbytescat(buf,s);
break;
#ifdef USE_NETCDF4
case NC_STRING: {
char* s = *(char**)(offset->memory + offset->offset);
ncbytescat(buf,"\"");
ncbytescat(buf,s);
ncbytescat(buf,"\"");
} break;
#endif
default:
#ifdef USE_NETCDF4
/* dump a user type */
switch (klass) {
case NC_OPAQUE: stat = dump_opaque(ncid,xtype,xsize,offset,buf); break;
case NC_ENUM: stat = dump_enum(ncid,xtype,basetype,offset,buf); break;
case NC_COMPOUND: stat = dump_compound(ncid,xtype,xsize,nfields,offset,buf); break;
case NC_VLEN: stat = dump_vlen(ncid,xtype,basetype,offset,buf); break;
default: stat = NC_EBADTYPE; break;
}
#else
stat = NC_EBADTYPE;
#endif
break;
}
if(xtype <= NC_MAX_ATOMIC_TYPE)
offset->offset += xsize;
done:
return stat;
}
#ifdef USE_NETCDF4
static int
dump_vlen(int ncid, nc_type xtype, nc_type basetype, Position* offset, NCbytes* buf)
{
int stat = NC_NOERR;
size_t i;
nc_vlen_t* vl = (nc_vlen_t*)(offset->memory+offset->offset);
char s[128];
if(vl->len > 0 && vl->p == NULL)
{stat = NC_EINVAL; goto done;}
snprintf(s,sizeof(s),"{len=%u,p=(",(unsigned)vl->len);
ncbytescat(buf,s);
/* dump each entry in the vlen list */
if(vl->len > 0) {
Position voffset;
size_t alignment = 0;
if((stat = NC_type_alignment(ncid,basetype,&alignment))) goto done;;
voffset.memory = vl->p;
voffset.offset = 0;
for(i=0;i<vl->len;i++) {
if(i > 0) ncbytescat(buf," ");
voffset.offset = read_align(voffset.offset,alignment);
if((stat = dump_datar(ncid,basetype,&voffset,buf))) goto done;
}
}
ncbytescat(buf,")}");
offset->offset += sizeof(nc_vlen_t);
done:
return stat;
}
static int
dump_enum(int ncid, nc_type xtype, nc_type basetype, Position* offset, NCbytes* buf)
{
int stat = NC_NOERR;
/* basically same as an instance of the enum's integer basetype */
stat = dump_datar(ncid,basetype,offset,buf);
return stat;
}
static int
dump_opaque(int ncid, nc_type xtype, size_t size, Position* offset, NCbytes* buf)
{
size_t i;
char sx[16];
/* basically a fixed size sequence of bytes */
ncbytescat(buf,"|");
for(i=0;i<size;i++) {
unsigned char x = *(offset->memory+offset->offset+i);
snprintf(sx,sizeof(sx),"%2x",x);
ncbytescat(buf,sx);
}
ncbytescat(buf,"|");
offset->offset += size;
return NC_NOERR;
}
static int
dump_compound(int ncid, nc_type xtype, size_t size, size_t nfields, Position* offset, NCbytes* buf)
{
int stat = NC_NOERR;
size_t fid, i, arraycount;
ptrdiff_t saveoffset;
int ndims;
int dimsizes[NC_MAX_VAR_DIMS];
saveoffset = offset->offset;
ncbytescat(buf,"<");
/* Get info about each field in turn and dump it */
for(fid=0;fid<nfields;fid++) {
size_t fieldalignment;
nc_type fieldtype;
char name[NC_MAX_NAME];
char sd[128];
/* Get all relevant info about the field */
if((stat = nc_inq_compound_field(ncid,xtype,fid,name,&fieldalignment,&fieldtype,&ndims,dimsizes))) goto done;
if(fid > 0) ncbytescat(buf,";");
ncbytescat(buf,name);
if(ndims > 0) {
int j;
for(j=0;j<ndims;j++) {
snprintf(sd,sizeof(sd),"[%d]",(int)dimsizes[j]);
ncbytescat(buf,sd);
}
}
if(ndims == 0) {ndims=1; dimsizes[0]=1;} /* fake the scalar case */
/* Align to this field */
offset->offset = saveoffset + fieldalignment;
/* compute the total number of elements in the field array */
arraycount = 1;
for(i=0;i<ndims;i++) arraycount *= dimsizes[i];
for(i=0;i<arraycount;i++) {
if(i > 0) ncbytescat(buf," ");
if((stat = dump_datar(ncid, fieldtype, offset,buf))) goto done;
}
}
ncbytescat(buf,">");
/* Return to beginning of the compound and move |compound| */
offset->offset = saveoffset;
offset->offset += size;
done:
return stat;
}
#endif
/* Utilities */
/* Extended version that can handle atomic typeids */
static int
NC4_inq_any_type(int ncid, nc_type typeid, char *name, size_t *size,
nc_type *basetypep, size_t *nfieldsp, int *classp)
{
int stat = NC_NOERR;
if(typeid >= NC_FIRSTUSERTYPEID) {
stat = nc_inq_user_type(ncid,typeid,name,size,basetypep,nfieldsp,classp);
} else if(typeid > NC_NAT && typeid <= NC_MAX_ATOMIC_TYPE) {
if(basetypep) *basetypep = NC_NAT;
if(nfieldsp) *nfieldsp = 0;
if(classp) *classp = typeid;
stat = NC4_inq_atomic_type(typeid,name,size);
} else
stat = NC_EBADTYPE;
return stat;
}

11
libdispatch/doffsets.c
View File

@ -108,9 +108,10 @@ static NCtypealignset set;
static int NC_alignments_computed = 0;
/* Argument is a netcdf type class, except compound|ENUM */
size_t
NC_class_alignment(int ncclass)
int
NC_class_alignment(int ncclass, size_t* alignp)
{
int stat = NC_NOERR;
NCalignment* align = NULL;
int index = 0;
if(!NC_alignments_computed)
@ -135,10 +136,12 @@ NC_class_alignment(int ncclass)
case NC_COMPOUND: /* fall thru */
default:
nclog(NCLOGERR,"nc_class_alignment: class code %d cannot be aligned",ncclass);
return 0;
goto done;
}
align = &vec[index];
return align->alignment;
if(alignp) *alignp = align->alignment;
done:
return stat;
}
void

3
libdispatch/dvar.c
View File

@ -260,6 +260,9 @@ nc_def_var(int ncid, const char *name, nc_type xtype,
to set fill value after data are written.
@return ::NC_EGLOBAL Attempt to set fill value on NC_GLOBAL.
Warning: Using a vlen type as the fill value may lead to a memory
leak.
@section nc_def_var_fill_example Example
In this example from libsrc4/tst_vars.c, a variable is defined, and

9
libdispatch/dvlen.c
View File

@ -25,14 +25,16 @@ because it can free an array of VLEN objects.
WARNING: this code is incorrect because it will only
work if the basetype of the vlen is
- atomic
- atomic (excluding string basetype)
- + enum
- + opaque
- excluding string basetype,
The reason is that to operate properly, it needs to recurse when
the basetype is a complex object such as another vlen or compound.
This function is deprecated in favor of the function "nc_reclaim_data".
See include/netcdf.h.
\param vl pointer to the vlen object.
\returns ::NC_NOERR No error.
@ -63,6 +65,9 @@ work if the basetype of the vlen is
The reason is that to operate properly, it needs to recurse when
the basetype is a complex object such as another vlen or compound.
This function is deprecated in favor of the function "nc_reclaim_data".
See include/netcdf.h.
\param len number of elements in the array.
\param vlens pointer to the vlen object.

171
libhdf5/hdf5attr.c
View File

@ -314,6 +314,12 @@ NC4_HDF5_del_att(int ncid, int varid, const char *name)
if (!(att = (NC_ATT_INFO_T*)ncindexlookup(attlist, name)))
return NC_ENOTATT;
/* Reclaim the content of the attribute */
if(att->data)
if((retval = nc_reclaim_data_all(ncid,att->nc_typeid,att->data,att->len))) return retval;
att->data = NULL;
att->len = 0;
/* Delete it from the HDF5 file, if it's been created. */
if (att->created)
{
@ -324,8 +330,10 @@ NC4_HDF5_del_att(int ncid, int varid, const char *name)
deletedid = att->hdr.id;
/* Remove this attribute in this list */
/* reclaim associated HDF5 info */
if((retval=nc4_HDF5_close_att(att))) return retval;
/* Remove this attribute in this list */
if ((retval = nc4_att_list_del(attlist, att)))
return retval;
@ -413,9 +421,16 @@ nc4_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
nc_bool_t new_att = NC_FALSE;
int retval = NC_NOERR, range_error = 0;
size_t type_size;
int i;
int ret;
int ncid;
void* copy = NULL;
/* Save the old att data and length and old fillvalue in case we need to rollback on error */
struct Save {
size_t len;
void* data;
nc_type type; /* In case we change the type of the attribute */
} attsave = {0,NULL,-1};
struct Save fillsave = {0,NULL,-1};
h5 = grp->nc4_info;
nc = h5->controller;
@ -497,10 +512,6 @@ nc4_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
if (file_type == NC_NAT || mem_type == NC_NAT)
return NC_EBADTYPE;
/* Get information about this type. */
if ((retval = nc4_get_typelen_mem(h5, file_type, &type_size)))
return retval;
/* No character conversions are allowed. */
if (file_type != mem_type &&
(file_type == NC_CHAR || mem_type == NC_CHAR ||
@ -523,14 +534,29 @@ nc4_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
/* Allocate storage for the HDF5 specific att info. */
if (!(att->format_att_info = calloc(1, sizeof(NC_HDF5_ATT_INFO_T))))
BAIL(NC_ENOMEM);
if(varid == NC_GLOBAL)
att->container = (NC_OBJ*)grp;
else
att->container = (NC_OBJ*)var;
}
/* Now fill in the metadata. */
att->dirty = NC_TRUE;
/* When we reclaim existing data, make sure to use the right type */
if(new_att) attsave.type = file_type; else attsave.type = att->nc_typeid;
att->nc_typeid = file_type;
/* Get information about this (possibly new) type. */
if ((retval = nc4_get_typelen_mem(h5, file_type, &type_size)))
return retval;
#ifdef SEPDATA
/* If this att has vlen or string data, release it before we lose the length value. */
if (att->stdata)
{
int i;
for (i = 0; i < att->len; i++)
if(att->stdata[i])
free(att->stdata[i]);
@ -539,41 +565,53 @@ nc4_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
}
if (att->vldata)
{
for (i = 0; i < att->len; i++)
int i;
for (i = 0; i < att->len; i++) {
nc_free_vlen(&att->vldata[i]); /* FIX: see warning of nc_free_vlen */
free(att->vldata);
att->vldata = NULL;
}
att->len = len;
#else
if (att->data)
{
assert(attsave.data == NULL);
attsave.data = att->data;
attsave.len = att->len;
att->data = NULL;
}
#endif
/* If this is the _FillValue attribute, then we will also have to
* copy the value to the fill_vlue pointer of the NC_VAR_INFO_T
* struct for this var. (But ignore a global _FillValue
* attribute). */
* attribute).
* Since fill mismatch is no longer required, we need to convert the
* att's type to the vars's type as part of storing.
*/
if (!strcmp(att->hdr.name, _FillValue) && varid != NC_GLOBAL)
{
int size;
/* Fill value must be same type and have exactly one value */
if (att->nc_typeid != var->type_info->hdr.id)
return NC_EBADTYPE;
if (att->len != 1)
/* Fill value must have exactly one value */
if (len != 1)
BAIL(NC_EINVAL);
/* If we already wrote to the dataset, then return an error. */
if (var->written_to)
BAIL(NC_ELATEFILL);
/* Get the length of the veriable data type. */
/* Get the length of the variable's data type. */
if ((retval = nc4_get_typelen_mem(grp->nc4_info, var->type_info->hdr.id,
&type_size)))
BAIL(retval);
/* Fill value must be same type and have exactly one value */
if (att->nc_typeid != var->type_info->hdr.id)
return NC_EBADTYPE;
/* Already set a fill value? Now I'll have to free the old
* one. Make up your damn mind, would you? */
if (var->fill_value)
{
#ifdef SEPDATA
if (var->type_info->nc_type_class == NC_VLEN)
{
if ((retval = nc_free_vlen(var->fill_value)))
@ -585,16 +623,26 @@ nc4_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
free(*(char **)var->fill_value);
}
free(var->fill_value);
#else
/* reclaim later */
fillsave.data = var->fill_value;
fillsave.type = var->type_info->hdr.id;
fillsave.len = 1;
#endif
var->fill_value = NULL;
}
/* Determine the size of the fill value in bytes. */
#ifdef SEPDATA
if (var->type_info->nc_type_class == NC_VLEN)
size = sizeof(hvl_t);
else if (var->type_info->nc_type_class == NC_STRING)
size = sizeof(char *);
else
size = type_size;
#endif
#ifdef SEPDATA
/* Allocate space for the fill value. */
if (!(var->fill_value = calloc(1, size)))
BAIL(NC_ENOMEM);
@ -630,16 +678,41 @@ nc4_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
}
else
memcpy(var->fill_value, data, type_size);
#else
{
nc_type var_type = var->type_info->hdr.id;
size_t var_type_size = var->type_info->size;
/* The old code used the var's type as opposed to the att's type; normally same,
but not required. Now we need to convert from the att's type to the var's type.
Note that we use mem_type rather than file_type because our data is in the form
of the memory data. When we later capture the memory data for the actual
attribute, we will use file_type as the target of the conversion. */
if(mem_type != var_type && mem_type < NC_STRING && var_type < NC_STRING) {
/* Need to convert from memory data into copy buffer */
if((copy = malloc(len*var_type_size))==NULL) BAIL(NC_ENOMEM);
if ((retval = nc4_convert_type(data, copy, mem_type, var_type,
len, &range_error, NULL,
(h5->cmode & NC_CLASSIC_MODEL),
NC_NOQUANTIZE, 0)))
BAIL(retval);
} else { /* no conversion */
/* Still need a copy of the input data */
copy = NULL;
if((retval = nc_copy_data_all(h5->controller->ext_ncid, mem_type, data, 1, &copy)))
BAIL(retval);
}
var->fill_value = copy;
copy = NULL;
}
#endif
/* Indicate that the fill value was changed, if the variable has already
* been created in the file, so the dataset gets deleted and re-created. */
if (var->created)
var->fill_val_changed = NC_TRUE;
}
/* Copy the attribute data, if there is any. VLENs and string
* arrays have to be handled specially. */
if (att->len)
/* Copy the attribute data, if there is any. */
if (len)
{
nc_type type_class; /* Class of attribute's type */
@ -648,6 +721,7 @@ nc4_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
BAIL(retval);
assert(data);
#ifdef SEPDATA
if (type_class == NC_VLEN)
{
const hvl_t *vldata1;
@ -685,6 +759,7 @@ nc4_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
specifically an NC_CHAR, we need to clean up
the pre-existing att->data. */
if (!new_att && att->data) {
free(att->data);
att->data = NULL;
}
@ -721,21 +796,71 @@ nc4_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
BAIL(retval);
}
}
#else
{
/* Allocate top level of the copy */
if (!(copy = malloc(len * type_size)))
BAIL(NC_ENOMEM);
/* Special case conversion from memory to file type */
if(mem_type != file_type && mem_type < NC_STRING && file_type < NC_STRING) {
if ((retval = nc4_convert_type(data, copy, mem_type, file_type,
len, &range_error, NULL,
(h5->cmode & NC_CLASSIC_MODEL),
NC_NOQUANTIZE, 0)))
BAIL(retval);
} else if(mem_type == file_type) { /* General case: no conversion */
if((retval = nc_copy_data(h5->controller->ext_ncid,file_type,data,len,copy)))
BAIL(retval);
} else
BAIL(NC_EURL);
/* Store it */
att->data = copy; copy = NULL;
}
#endif
}
att->dirty = NC_TRUE;
att->created = NC_FALSE;
att->len = len;
/* Mark attributes on variable dirty, so they get written */
if(var)
var->attr_dirty = NC_TRUE;
/* Reclaim saved data */
if(attsave.data != NULL) {
assert(attsave.len > 0);
(void)nc_reclaim_data_all(h5->controller->ext_ncid,attsave.type,attsave.data,attsave.len);
attsave.len = 0; attsave.data = NULL;
}
if(fillsave.data != NULL) {
assert(fillsave.len > 0);
(void)nc_reclaim_data_all(h5->controller->ext_ncid,fillsave.type,fillsave.data,fillsave.len);
fillsave.len = 0; fillsave.data = NULL;
}
exit:
if(copy)
(void)nc_reclaim_data_all(h5->controller->ext_ncid,file_type,copy,len);
if(retval) {
/* Rollback */
if(attsave.data != NULL) {
assert(attsave.len > 0);
if(att->data)
(void)nc_reclaim_data_all(h5->controller->ext_ncid,attsave.type,att->data,att->len);
att->len = attsave.len; att->data = attsave.data;
}
if(fillsave.data != NULL) {
assert(fillsave.len > 0);
if(att->data)
(void)nc_reclaim_data_all(h5->controller->ext_ncid,fillsave.type,var->fill_value,1);
var->fill_value = fillsave.data;
}
}
/* If there was an error return it, otherwise return any potential
range error value. If none, return NC_NOERR as usual.*/
if (retval)
return retval;
if (range_error)
return NC_ERANGE;
if (retval)
return retval;
return NC_NOERR;
}

6
libhdf5/hdf5file.c
View File

@ -489,9 +489,9 @@ NC4_enddef(int ncid)
{
NC_FILE_INFO_T *nc4_info;
NC_GRP_INFO_T *grp;
NC_VAR_INFO_T *var;
int i;
int retval;
int i;
NC_VAR_INFO_T* var = NULL;
LOG((1, "%s: ncid 0x%x", __func__, ncid));
@ -499,6 +499,8 @@ NC4_enddef(int ncid)
if ((retval = nc4_find_grp_h5(ncid, &grp, &nc4_info)))
return retval;
/* Why is this here? Especially since it is not recursive so it
only applies to the this grp */
/* When exiting define mode, mark all variable written. */
for (i = 0; i < ncindexsize(grp->vars); i++)
{

12
libhdf5/hdf5internal.c
View File

@ -564,7 +564,7 @@ nc4_HDF5_close_att(NC_ATT_INFO_T *att)
nullfree(hdf5_att);
att->format_att_info = NULL;
return NC_NOERR;
return NC_NOERR;
}
/**
@ -601,11 +601,19 @@ close_vars(NC_GRP_INFO_T *grp)
{
if (var->type_info)
{
if (var->type_info->nc_type_class == NC_VLEN)
#ifdef SEPDATA
if (var->type_info->nc_type_class == NC_VLEN)
nc_free_vlen((nc_vlen_t *)var->fill_value);
else if (var->type_info->nc_type_class == NC_STRING && *(char **)var->fill_value)
free(*(char **)var->fill_value);
#else
int stat = NC_NOERR;
if((stat = nc_reclaim_data(grp->nc4_info->controller->ext_ncid,var->type_info->hdr.id,var->fill_value,1)))
return stat;
nullfree(var->fill_value);
}
#endif
var->fill_value = NULL;
}
}

77
libhdf5/hdf5open.c
View File

@ -1151,6 +1151,8 @@ static int get_fill_info(hid_t propid, NC_VAR_INFO_T *var)
/* Allocate space to hold the fill value. */
if (!var->fill_value)
{
#ifdef SEPDATA
if (var->type_info->nc_type_class == NC_VLEN)
{
if (!(var->fill_value = malloc(sizeof(nc_vlen_t))))
@ -1162,6 +1164,7 @@ static int get_fill_info(hid_t propid, NC_VAR_INFO_T *var)
return NC_ENOMEM;
}
else
#endif
{
assert(var->type_info->size);
if (!(var->fill_value = malloc(var->type_info->size)))
@ -1746,7 +1749,7 @@ read_hdf5_att(NC_GRP_INFO_T *grp, hid_t attid, NC_ATT_INFO_T *att)
LOG((5, "%s: att->hdr.id %d att->hdr.name %s att->nc_typeid %d att->len %d",
__func__, att->hdr.id, att->hdr.name, (int)att->nc_typeid, att->len));
/* Get HDF5-sepecific info struct for this attribute. */
/* Get HDF5-specific info struct for this attribute. */
hdf5_att = (NC_HDF5_ATT_INFO_T *)att->format_att_info;
/* Get type of attribute in file. */
@ -1768,7 +1771,6 @@ read_hdf5_att(NC_GRP_INFO_T *grp, hid_t attid, NC_ATT_INFO_T *att)
&(att->nc_typeid))))
BAIL(retval);
/* Get len. */
if ((spaceid = H5Aget_space(attid)) < 0)
BAIL(NC_EATTMETA);
@ -1837,6 +1839,7 @@ read_hdf5_att(NC_GRP_INFO_T *grp, hid_t attid, NC_ATT_INFO_T *att)
if ((retval = nc4_get_typelen_mem(grp->nc4_info, att->nc_typeid,
&type_size)))
return retval;
#ifdef SEPDATA
if (att_class == H5T_VLEN)
{
if (!(att->vldata = malloc((unsigned int)(att->len * sizeof(hvl_t)))))
@ -1853,11 +1856,11 @@ read_hdf5_att(NC_GRP_INFO_T *grp, hid_t attid, NC_ATT_INFO_T *att)
* nc_free_string be called on string arrays, which would not
* work if one contiguous memory block were used. So here I
* convert the contiguous block of strings into an array of
* malloced strings (each string with its own malloc). Then I
* malloced strings -- each string with its own malloc. Then I
* copy the data and free the contiguous memory. This
* involves copying the data, which is bad, but this only
* occurs for fixed length string attributes, and presumably
* these are small. (And netCDF-4 does not create them - it
* these are small. Note also that netCDF-4 does not create them - it
* always uses variable length strings. */
if (fixed_len_string)
{
@ -1899,12 +1902,64 @@ read_hdf5_att(NC_GRP_INFO_T *grp, hid_t attid, NC_ATT_INFO_T *att)
}
}
else
#else
{
if (!(att->data = malloc((unsigned int)(att->len * type_size))))
BAIL(NC_ENOMEM);
if (H5Aread(attid, hdf5_att->native_hdf_typeid, att->data) < 0)
BAIL(NC_EATTMETA);
/* For a fixed length HDF5 string, the read requires
* contiguous memory. Meanwhile, the netCDF API requires that
* nc_free_string be called on string arrays, which would not
* work if one contiguous memory block were used. So here I
* convert the contiguous block of strings into an array of
* malloced strings -- each string with its own malloc. Then I
* copy the data and free the contiguous memory. This
* involves copying the data, which is bad, but this only
* occurs for fixed length string attributes, and presumably
* these are small. Note also that netCDF-4 does not create them - it
* always uses variable length strings. */
if (att->nc_typeid == NC_STRING && fixed_len_string)
{
int i;
char *contig_buf, *cur;
char** dst = NULL;
/* Alloc space for the contiguous memory read. */
if (!(contig_buf = malloc(att->len * fixed_size * sizeof(char))))
BAIL(NC_ENOMEM);
/* Read the fixed-len strings as one big block. */
if (H5Aread(attid, hdf5_att->native_hdf_typeid, contig_buf) < 0) {
free(contig_buf);
BAIL(NC_EATTMETA);
}
/* Copy strings, one at a time, into their new home. Alloc
space for each string. The user will later free this
space with nc_free_string. */
cur = contig_buf;
dst = (char**)att->data;
for (i = 0; i < att->len; i++)
{
char* s = NULL;
if (!(s = malloc(fixed_size+1))) {
free(contig_buf);
BAIL(NC_ENOMEM);
}
memcpy(s,cur,fixed_size);
s[fixed_size] = '\0';
dst[i] = s;
cur += fixed_size;
}
/* Free contiguous memory buffer. */
free(contig_buf);
} else { /* not fixed string */
/* Just read the data */
if (H5Aread(attid, hdf5_att->native_hdf_typeid, att->data) < 0)
BAIL(NC_EATTMETA);
}
}
#endif
}
if (H5Tclose(file_typeid) < 0)
@ -2106,6 +2161,12 @@ read_type(NC_GRP_INFO_T *grp, hid_t hdf_typeid, char *type_name)
return retval;
}
{ /* See if this changes from fixed size to variable size */
int fixedsize;
if((retval = NC4_inq_type_fixed_size(grp->nc4_info->controller->ext_ncid,member_xtype,&fixedsize))) return retval;
if(!fixedsize) type->u.c.varsized = 1;
}
hdf5free(member_name);
}
}
@ -2281,6 +2342,10 @@ att_read_callbk(hid_t loc_id, const char *att_name, const H5A_info_t *ainfo,
if ((retval = nc4_att_list_add(list, att_name, &att)))
BAIL(-1);
/* Remember container */
att->container = (att_info->var ? (NC_OBJ*)att_info->var: (NC_OBJ*)att_info->grp);
/* Allocate storage for the HDF5 specific att info. */
if (!(att->format_att_info = calloc(1, sizeof(NC_HDF5_ATT_INFO_T))))
BAIL(-1);

17
libhdf5/hdf5type.c
View File

@ -146,12 +146,18 @@ add_user_type(int ncid, size_t size, const char *name, nc_type base_typeid,
if ((retval = NC4_redef(ncid)))
return retval;
/* No size is provided for vlens or enums, get it from the base type. */
if (type_class == NC_VLEN || type_class == NC_ENUM)
/* No size is provided for vlens; use the size of nc_vlen_t */
if (type_class == NC_VLEN)
size = sizeof(nc_vlen_t);
/* No size is provided for enums, get it from the base type. */
else if(type_class == NC_ENUM)
{
if ((retval = nc4_get_typelen_mem(grp->nc4_info, base_typeid, &size)))
return retval;
}
/* Else better be defined */
else if (size <= 0)
return NC_EINVAL;
@ -257,6 +263,7 @@ NC4_insert_array_compound(int ncid, int typeid1, const char *name,
NC_TYPE_INFO_T *type;
char norm_name[NC_MAX_NAME + 1];
int retval;
int fixedsize = 0;
LOG((2, "nc_insert_array_compound: ncid 0x%x, typeid %d name %s "
"offset %d field_typeid %d ndims %d", ncid, typeid1,
@ -288,6 +295,12 @@ NC4_insert_array_compound(int ncid, int typeid1, const char *name,
ndims, dim_sizesp)))
return retval;
/* See if this changes from fixed size to variable size */
if((retval = NC4_inq_type_fixed_size(ncid,field_typeid,&fixedsize)))
return retval;
if(!fixedsize)
type->u.c.varsized = 1;
return NC_NOERR;
}

18
libhdf5/hdf5var.c
View File

@ -666,7 +666,7 @@ nc_def_var_extra(int ncid, int varid, int *shuffle, int *unused1,
}
/* Are we setting a fill value? */
if (fill_value && !var->no_fill)
if (fill_value && no_fill && !(*no_fill))
{
/* Copy the fill_value. */
LOG((4, "Copying fill value into metadata for variable %s",
@ -677,10 +677,16 @@ nc_def_var_extra(int ncid, int varid, int *shuffle, int *unused1,
if (retval && retval != NC_ENOTATT)
return retval;
/* Create a _FillValue attribute. */
/* Create a _FillValue attribute; will also fill in var->fill_value */
if ((retval = nc_put_att(ncid, varid, _FillValue, var->type_info->hdr.id,
1, fill_value)))
return retval;
} else if (var->fill_value && no_fill && (*no_fill)) { /* Turning off fill value? */
/* If there's a _FillValue attribute, delete it. */
retval = NC4_HDF5_del_att(ncid, varid, _FillValue);
if (retval && retval != NC_ENOTATT) return retval;
if((retval = nc_reclaim_data_all(ncid,var->type_info->hdr.id,var->fill_value,1))) return retval;
var->fill_value = NULL;
}
/* Is the user setting the endianness? */
@ -2111,6 +2117,7 @@ NC4_get_vars(int ncid, int varid, const size_t *startp, const size_t *countp,
for (i = 0; i < fill_len; i++)
{
#ifdef SEPDATA
if (var->type_info->nc_type_class == NC_STRING)
{
if (*(char **)fillvalue)
@ -2132,6 +2139,13 @@ NC4_get_vars(int ncid, int varid, const size_t *startp, const size_t *countp,
}
else
memcpy(filldata, fillvalue, file_type_size);
#else
{
/* Copy one instance of the fill_value */
if((retval = nc_copy_data(ncid,var->type_info->hdr.id,fillvalue,1,filldata)))
BAIL(retval);
}
#endif
filldata = (char *)filldata + file_type_size;
}
}

8
libhdf5/nc4hdf.c
View File

@ -470,12 +470,14 @@ put_att_grpa(NC_GRP_INFO_T *grp, int varid, NC_ATT_INFO_T *att)
* some phoney data (which won't be written anyway.)*/
if (!dims[0])
data = &phoney_data;
else if (att->data)
data = att->data;
#ifdef SEPDATA
else if (att->vldata)
data = att->vldata;
else if (att->stdata)
data = att->stdata;
#endif
else
data = att->vldata;
data = att->data;
/* NC_CHAR types require some extra work. The space ID is set to
* scalar, and the type is told how long the string is. If it's

283
libnczarr/zattr.c
View File

@ -322,6 +322,12 @@ NCZ_del_att(int ncid, int varid, const char *name)
if (!(att = (NC_ATT_INFO_T*)ncindexlookup(attlist, name)))
return NC_ENOTATT;
/* Reclaim the content of the attribute */
if(att->data)
if((retval = nc_reclaim_data_all(ncid,att->nc_typeid,att->data,att->len))) return retval;
att->data = NULL;
att->len = 0;
/* Delete it from the ZARR file, if it's been created. */
if (att->created)
{
@ -334,6 +340,12 @@ NCZ_del_att(int ncid, int varid, const char *name)
deletedid = att->hdr.id;
/* reclaim associated NCZarr info */
{
NCZ_ATT_INFO_T* za = (NCZ_ATT_INFO_T*)att->format_att_info;
nullfree(za);
}
/* Remove this attribute in this list */
if ((retval = nc4_att_list_del(attlist, att)))
return retval;
@ -422,9 +434,16 @@ ncz_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
nc_bool_t new_att = NC_FALSE;
int retval = NC_NOERR, range_error = 0;
size_t type_size;
int i;
int ret;
int ncid;
void* copy = NULL;
/* Save the old att data and length and old fillvalue in case we need to rollback on error */
struct Save {
size_t len;
void* data;
nc_type type; /* In case we change the type of the attribute */
} attsave = {0,NULL,-1};
struct Save fillsave = {0,NULL,-1};
h5 = grp->nc4_info;
nc = h5->controller;
@ -507,10 +526,6 @@ ncz_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
if (file_type == NC_NAT || mem_type == NC_NAT)
return NC_EBADTYPE;
/* Get information about this type. */
if ((retval = nc4_get_typelen_mem(h5, file_type, &type_size)))
return retval;
/* No character conversions are allowed. */
if (file_type != mem_type &&
(file_type == NC_CHAR || mem_type == NC_CHAR ||
@ -533,30 +548,31 @@ ncz_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
/* Allocate storage for the ZARR specific att info. */
if (!(att->format_att_info = calloc(1, sizeof(NCZ_ATT_INFO_T))))
BAIL(NC_ENOMEM);
if(varid == NC_GLOBAL)
att->container = (NC_OBJ*)grp;
else
att->container = (NC_OBJ*)var;
}
/* Now fill in the metadata. */
att->dirty = NC_TRUE;
/* When we reclaim existing data, make sure to use the right type */
if(new_att) attsave.type = file_type; else attsave.type = att->nc_typeid;
att->nc_typeid = file_type;
/* If this att has vlen or string data, release it before we lose the length value. */
if (att->stdata)
{
for (i = 0; i < att->len; i++)
if(att->stdata[i])
free(att->stdata[i]);
free(att->stdata);
att->stdata = NULL;
}
if (att->vldata)
{
for (i = 0; i < att->len; i++)
nc_free_vlen(&att->vldata[i]); /* FIX: see warning of nc_free_vlen */
free(att->vldata);
att->vldata = NULL;
}
/* Get information about this (possibly new) type. */
if ((retval = nc4_get_typelen_mem(h5, file_type, &type_size)))
return retval;
att->len = len;
if (att->data)
{
assert(attsave.data == NULL);
attsave.data = att->data;
attsave.len = att->len;
att->data = NULL;
}
/* If this is the _FillValue attribute, then we will also have to
* copy the value to the fill_vlue pointer of the NC_VAR_INFO_T
@ -564,12 +580,8 @@ ncz_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
* attribute). */
if (!strcmp(att->hdr.name, _FillValue) && varid != NC_GLOBAL)
{
int size;
/* Fill value must be same type and have exactly one value */
if (att->nc_typeid != var->type_info->hdr.id)
return NC_EBADTYPE;
if (att->len != 1)
/* Fill value must have exactly one value */
if (len != 1)
return NC_EINVAL;
/* If we already wrote to the dataset, then return an error. */
@ -585,65 +597,40 @@ ncz_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
* one. Make up your damn mind, would you? */
if (var->fill_value)
{
if (var->type_info->nc_type_class == NC_VLEN)
{
if ((retval = nc_free_vlen(var->fill_value)))
return retval;
}
else if (var->type_info->nc_type_class == NC_STRING)
{
if (*(char **)var->fill_value)
free(*(char **)var->fill_value);
}
free(var->fill_value); var->fill_value = NULL;
/* reclaim later */
fillsave.data = var->fill_value;
fillsave.type = var->type_info->hdr.id;
fillsave.len = 1;
var->fill_value = NULL;
}
#ifdef LOOK
/* Determine the size of the fill value in bytes. */
if (var->type_info->nc_type_class == NC_VLEN)
size = sizeof(hvl_t);
else
#endif
if (var->type_info->nc_type_class == NC_STRING)
size = sizeof(char *);
else
size = type_size;
/* Allocate space for the fill value. */
if (!(var->fill_value = calloc(1, size)))
return NC_ENOMEM;
/* Copy the fill_value. */
LOG((4, "Copying fill value into metadata for variable %s", var->hdr.name));
if (var->type_info->nc_type_class == NC_VLEN)
{
nc_vlen_t *in_vlen = (nc_vlen_t *)data, *fv_vlen = (nc_vlen_t *)(var->fill_value);
NC_TYPE_INFO_T* basetype;
size_t basetypesize = 0;
/* get the basetype and its size */
basetype = var->type_info;
if ((retval = nc4_get_typelen_mem(grp->nc4_info, basetype->hdr.id, &basetypesize)))
return retval;
/* shallow clone the content of the vlen; shallow because it has only a temporary existence */
fv_vlen->len = in_vlen->len;
if (!(fv_vlen->p = malloc(basetypesize * in_vlen->len)))
return NC_ENOMEM;
memcpy(fv_vlen->p, in_vlen->p, in_vlen->len * basetypesize);
}
else if (var->type_info->nc_type_class == NC_STRING)
{
if (*(char **)data)
{
if (!(*(char **)(var->fill_value) = malloc(strlen(*(char **)data) + 1)))
return NC_ENOMEM;
strcpy(*(char **)var->fill_value, *(char **)data);
}
else
*(char **)var->fill_value = NULL;
}
else
memcpy(var->fill_value, data, type_size);
{
nc_type var_type = var->type_info->hdr.id;
size_t var_type_size = var->type_info->size;
/* The old code used the var's type as opposed to the att's type; normally same,
but not required. Now we need to convert from the att's type to the var's type.
Note that we use mem_type rather than file_type because our data is in the form
of the memory data. When we later capture the memory data for the actual
attribute, we will use file_type as the target of the conversion. */
if(mem_type != var_type && mem_type < NC_STRING && var_type < NC_STRING) {
/* Need to convert from memory data into copy buffer */
if((copy = malloc(len*var_type_size))==NULL) BAIL(NC_ENOMEM);
if ((retval = nc4_convert_type(data, copy, mem_type, var_type,
len, &range_error, NULL,
(h5->cmode & NC_CLASSIC_MODEL),
NC_NOQUANTIZE, 0)))
BAIL(retval);
} else { /* no conversion */
/* Still need a copy of the input data */
copy = NULL;
if((retval = nc_copy_data_all(h5->controller->ext_ncid, mem_type, data, 1, &copy)))
BAIL(retval);
}
var->fill_value = copy;
copy = NULL;
}
/* Indicate that the fill value was changed, if the variable has already
* been created in the file, so the dataset gets deleted and re-created. */
@ -651,9 +638,8 @@ ncz_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
var->fill_val_changed = NC_TRUE;
}
/* Copy the attribute data, if there is any. VLENs and string
* arrays have to be handled specially. */
if (att->len)
/* Copy the attribute data, if there is any. */
if (len)
{
nc_type type_class; /* Class of attribute's type */
@ -662,102 +648,69 @@ ncz_put_att(NC_GRP_INFO_T* grp, int varid, const char *name, nc_type file_type,
return retval;
assert(data);
#ifdef LOOK
if (type_class == NC_VLEN)
{
const hvl_t *vldata1;
NC_TYPE_INFO_T *vltype;
size_t base_typelen;
/* Get the type object for the attribute's type */
if ((retval = nc4_find_type(h5, file_type, &vltype)))
BAIL(retval);
/* Retrieve the size of the base type */
if ((retval = nc4_get_typelen_mem(h5, vltype->u.v.base_nc_typeid, &base_typelen)))
BAIL(retval);
vldata1 = data;
if (!(att->vldata = (nc_vlen_t*)malloc(att->len * sizeof(hvl_t))))
/* Allocate top level of the copy */
if (!(copy = malloc(len * type_size)))
BAIL(NC_ENOMEM);
for (i = 0; i < att->len; i++)
{
att->vldata[i].len = vldata1[i].len;
/* Warning, this only works for cases described for nc_free_vlen() */
if (!(att->vldata[i].p = malloc(base_typelen * att->vldata[i].len)))
BAIL(NC_ENOMEM);
memcpy(att->vldata[i].p, vldata1[i].p, base_typelen * att->vldata[i].len);
}
}
else
#endif
if (type_class == NC_STRING)
{
LOG((4, "copying array of NC_STRING"));
if (!(att->stdata = malloc(sizeof(char *) * att->len))) {
BAIL(NC_ENOMEM);
}
/* If we are overwriting an existing attribute,
specifically an NC_CHAR, we need to clean up
the pre-existing att->data. */
if (!new_att && att->data) {
free(att->data);
att->data = NULL;
}
for (i = 0; i < att->len; i++)
{
if(NULL != ((char **)data)[i]) {
LOG((5, "copying string %d of size %d", i, strlen(((char **)data)[i]) + 1));
if (!(att->stdata[i] = strdup(((char **)data)[i])))
BAIL(NC_ENOMEM);
}
else
att->stdata[i] = ((char **)data)[i];
}
}
else
{
/* [Re]allocate memory for the attribute data */
if (!new_att)
free (att->data);
if (!(att->data = malloc(att->len * type_size)))
BAIL(NC_ENOMEM);
#ifdef ADEBUG
fprintf(stderr,"new attr: %s: len=%d alloc=%d\n",att->hdr.name,(int)att->len,(int)(att->len*type_size));
#endif
/* Just copy the data, for non-atomic types */
if (type_class == NC_OPAQUE || type_class == NC_COMPOUND || type_class == NC_ENUM)
memcpy(att->data, data, len * type_size);
else if(mem_type == file_type) {
memcpy(att->data, data, len * type_size);
} else /* need to convert */
{
/* Data types are like religions, in that one can convert. */
if ((retval = nc4_convert_type(data, att->data, mem_type, file_type,
/* Special case conversion from memory to file type */
if(mem_type != file_type && mem_type < NC_STRING && file_type < NC_STRING) {
if ((retval = nc4_convert_type(data, copy, mem_type, file_type,
len, &range_error, NULL,
(h5->cmode & NC_CLASSIC_MODEL),
NC_NOQUANTIZE, 0)))
BAIL(retval);
}
}
} else if(mem_type == file_type) { /* General case: no conversion */
if((retval = nc_copy_data(h5->controller->ext_ncid,file_type,data,len,copy)))
BAIL(retval);
} else
BAIL(NC_EURL);
/* Store it */
att->data = copy; copy = NULL;
}
}
att->dirty = NC_TRUE;
att->created = NC_FALSE;
att->len = len;
/* Mark attributes on variable dirty, so they get written */
if(var)
var->attr_dirty = NC_TRUE;
/* Reclaim saved data */
if(attsave.data != NULL) {
assert(attsave.len > 0);
(void)nc_reclaim_data_all(h5->controller->ext_ncid,attsave.type,attsave.data,attsave.len);
attsave.len = 0; attsave.data = NULL;
}
if(fillsave.data != NULL) {
assert(fillsave.len > 0);
(void)nc_reclaim_data_all(h5->controller->ext_ncid,fillsave.type,fillsave.data,fillsave.len);
fillsave.len = 0; fillsave.data = NULL;
}
exit:
if(copy)
(void)nc_reclaim_data_all(h5->controller->ext_ncid,file_type,copy,len);
if(retval) {
/* Rollback */
if(attsave.data != NULL) {
assert(attsave.len > 0);
if(att->data)
(void)nc_reclaim_data_all(h5->controller->ext_ncid,attsave.type,att->data,att->len);
att->len = attsave.len; att->data = attsave.data;
}
if(fillsave.data != NULL) {
assert(fillsave.len > 0);
if(att->data)
(void)nc_reclaim_data_all(h5->controller->ext_ncid,fillsave.type,var->fill_value,1);
var->fill_value = fillsave.data;
}
}
/* If there was an error return it, otherwise return any potential
range error value. If none, return NC_NOERR as usual.*/
if (retval)
return retval;
if (range_error)
return NC_ERANGE;
if (retval)
return retval;
return NC_NOERR;
}

2
libnczarr/zfile.c
View File

@ -108,6 +108,8 @@ NCZ_enddef(int ncid)
if ((stat = nc4_find_grp_h5(ncid, &grp, &h5)))
goto done;
/* Why is this here? Especially since it is not recursive so it
only applies to the this grp */
/* When exiting define mode, process all variables */
for (i = 0; i < nclistlength(h5->allgroups); i++) {
NC_GRP_INFO_T* g = nclistget(h5->allgroups,i);

8
libnczarr/zinternal.c
View File

@ -335,10 +335,10 @@ close_vars(NC_GRP_INFO_T *grp)
{
if (var->type_info)
{
if (var->type_info->nc_type_class == NC_VLEN)
nc_free_vlen((nc_vlen_t *)var->fill_value);
else if (var->type_info->nc_type_class == NC_STRING && *(char **)var->fill_value)
free(*(char **)var->fill_value);
int stat = NC_NOERR;
if((stat = nc_reclaim_data(grp->nc4_info->controller->ext_ncid,var->type_info->hdr.id,var->fill_value,1)))
return stat;
nullfree(var->fill_value);
}
}
}

8
libnczarr/ztype.c
View File

@ -190,8 +190,12 @@ add_user_type(int ncid, size_t size, const char *name, nc_type base_typeid,
if ((retval = NCZ_redef(ncid)))
return retval;
/* No size is provided for vlens or enums, get it from the base type. */
if (type_class == NC_VLEN || type_class == NC_ENUM)
/* No size is provided for vlens; use the size of nc_vlen_t */
if (type_class == NC_VLEN)
size = sizeof(nc_vlen_t);
/* No size is provided for enums, get it from the base type. */
else if(type_class == NC_ENUM)
{
if ((retval = ncz_get_typelen_mem(grp->ncz_info, base_typeid, &size)))
return retval;

35
libnczarr/zvar.c
View File

@ -707,7 +707,7 @@ ncz_def_var_extra(int ncid, int varid, int *shuffle, int *unused1,
}
/* Are we setting a fill value? */
if (fill_value && !var->no_fill)
if (fill_value && no_fill && !(*no_fill))
{
/* Copy the fill_value. */
LOG((4, "Copying fill value into metadata for variable %s",
@ -718,10 +718,16 @@ ncz_def_var_extra(int ncid, int varid, int *shuffle, int *unused1,
if (retval && retval != NC_ENOTATT)
goto done;
/* Create a _FillValue attribute. */
/* Create a _FillValue attribute; will also fill in var->fill_value */
if ((retval = nc_put_att(ncid, varid, _FillValue, var->type_info->hdr.id,
1, fill_value)))
goto done;
} else if (var->fill_value && no_fill && (*no_fill)) { /* Turning off fill value? */
/* If there's a _FillValue attribute, delete it. */
retval = NCZ_del_att(ncid, varid, _FillValue);
if (retval && retval != NC_ENOTATT) return retval;
if((retval = nc_reclaim_data_all(ncid,var->type_info->hdr.id,var->fill_value,1))) return retval;
var->fill_value = NULL;
}
/* Is the user setting the endianness? */
@ -1872,28 +1878,9 @@ NCZ_get_vars(int ncid, int varid, const size_t *startp, const size_t *countp,
filldata = (char *)data + real_data_size;
for (i = 0; i < fill_len; i++)
{
if (var->type_info->nc_type_class == NC_STRING)
{
if (*(char **)fillvalue)
{
if (!(*(char **)filldata = strdup(*(char **)fillvalue)))
BAIL(NC_ENOMEM);
}
else
*(char **)filldata = NULL;
}
else if (var->type_info->nc_type_class == NC_VLEN)
{
if (fillvalue)
{
memcpy(filldata,fillvalue,file_type_size);
} else {
*(char **)filldata = NULL;
}
}
else
memcpy(filldata, fillvalue, file_type_size);
/* Copy one instance of the fill_value */
if((retval = nc_copy_data(ncid,var->type_info->hdr.id,fillvalue,1,filldata)))
BAIL(retval);
filldata = (char *)filldata + file_type_size;
}
}

10
libsrc4/nc4attr.c
View File

@ -54,7 +54,6 @@ nc4_get_att_ptrs(NC_FILE_INFO_T *h5, NC_GRP_INFO_T *grp, NC_VAR_INFO_T *var,
void *bufr = NULL;
size_t type_size;
int varid;
int i;
int retval;
LOG((3, "%s: mem_type %d", __func__, mem_type));
@ -139,11 +138,13 @@ nc4_get_att_ptrs(NC_FILE_INFO_T *h5, NC_GRP_INFO_T *grp, NC_VAR_INFO_T *var,
bugs! */
if (data)
{
#ifdef SEPDATA
if (att->vldata)
{
size_t base_typelen;
nc_hvl_t *vldest = data;
NC_TYPE_INFO_T *type;
int i;
/* Get the type object for the attribute's type */
if ((retval = nc4_find_type(h5, att->nc_typeid, &type)))
@ -163,6 +164,7 @@ nc4_get_att_ptrs(NC_FILE_INFO_T *h5, NC_GRP_INFO_T *grp, NC_VAR_INFO_T *var,
}
else if (att->stdata)
{
int i;
for (i = 0; i < att->len; i++)
{
/* Check for NULL pointer for string (valid in HDF5) */
@ -179,6 +181,12 @@ nc4_get_att_ptrs(NC_FILE_INFO_T *h5, NC_GRP_INFO_T *grp, NC_VAR_INFO_T *var,
{
memcpy(data, bufr, (size_t)(att->len * type_size));
}
#else
{
if((retval = nc_copy_data(h5->controller->ext_ncid,mem_type,bufr,att->len,data)))
BAIL(retval);
}
#endif
}
exit:

97
libsrc4/nc4internal.c
View File

@ -22,7 +22,6 @@
#include "nc.h" /* from libsrc */
#include "ncdispatch.h" /* from libdispatch */
#include "ncutf8.h"
#include "netcdf_aux.h"
/** @internal Number of reserved attributes. These attributes are
* hidden from the netcdf user, but exist in the implementation
@ -1260,20 +1259,22 @@ nc4_type_free(NC_TYPE_INFO_T *type)
int
nc4_att_free(NC_ATT_INFO_T *att)
{
int i;
int stat = NC_NOERR;
assert(att);
LOG((3, "%s: name %s ", __func__, att->hdr.name));
/* Free memory that was malloced to hold data for this
* attribute. */
if (att->data)
free(att->data);
/* Free the name. */
if (att->hdr.name)
free(att->hdr.name);
#ifdef SEPDATA
/* Free memory that was malloced to hold data for this
* attribute. */
if (att->data) {
free(att->data);
}
/* If this is a string array attribute, delete all members of the
* string array, then delete the array of pointers to strings. (The
* array was filled with pointers by HDF5 when the att was read,
@ -1282,6 +1283,7 @@ nc4_att_free(NC_ATT_INFO_T *att)
* allocate the memory that is being freed.) */
if (att->stdata)
{
int i;
for (i = 0; i < att->len; i++)
if(att->stdata[i])
free(att->stdata[i]);
@ -1291,13 +1293,31 @@ nc4_att_free(NC_ATT_INFO_T *att)
/* If this att has vlen data, release it. */
if (att->vldata)
{
int i;
for (i = 0; i < att->len; i++)
nc_free_vlen(&att->vldata[i]);
free(att->vldata);
}
#else
if (att->data) {
NC_OBJ* parent;
NC_FILE_INFO_T* h5 = NULL;
/* Locate relevant objects */
parent = att->container;
if(parent->sort == NCVAR) parent = (NC_OBJ*)(((NC_VAR_INFO_T*)parent)->container);
assert(parent->sort == NCGRP);
h5 = ((NC_GRP_INFO_T*)parent)->nc4_info;
/* Reclaim the attribute data */
if((stat = nc_reclaim_data(h5->controller->ext_ncid,att->nc_typeid,att->data,att->len))) goto done;
free(att->data); /* reclaim top level */
att->data = NULL;
}
#endif
done:
free(att);
return NC_NOERR;
return stat;
}
/**
@ -1485,6 +1505,57 @@ nc4_rec_grp_del(NC_GRP_INFO_T *grp)
return NC_NOERR;
}
/**
* @internal Recursively delete the data for a group (and everything
* it contains) in our internal metadata store.
*
* @param grp Pointer to group info struct.
*
* @return ::NC_NOERR No error.
* @author Ed Hartnett, Dennis Heimbigner
*/
int
nc4_rec_grp_del_att_data(NC_GRP_INFO_T *grp)
{
int i;
int retval;
assert(grp);
LOG((3, "%s: grp->name %s", __func__, grp->hdr.name));
/* Recursively call this function for each child, if any, stopping
* if there is an error. */
for (i = 0; i < ncindexsize(grp->children); i++)
if ((retval = nc4_rec_grp_del_att_data((NC_GRP_INFO_T *)ncindexith(grp->children, i))))
return retval;
/* Free attribute data in this group */
for (i = 0; i < ncindexsize(grp->att); i++) {
NC_ATT_INFO_T * att = (NC_ATT_INFO_T*)ncindexith(grp->att, i);
if((retval = nc_reclaim_data_all(grp->nc4_info->controller->ext_ncid,att->nc_typeid,att->data,att->len)))
return retval;
att->data = NULL;
att->len = 0;
att->dirty = 0;
}
/* Delete att data from all contained vars in this group */
for (i = 0; i < ncindexsize(grp->vars); i++) {
int j;
NC_VAR_INFO_T* v = (NC_VAR_INFO_T *)ncindexith(grp->vars, i);
for(j=0;j<ncindexsize(v->att);j++) {
NC_ATT_INFO_T* att = (NC_ATT_INFO_T*)ncindexith(v->att, j);
if((retval = nc_reclaim_data_all(grp->nc4_info->controller->ext_ncid,att->nc_typeid,att->data,att->len)))
return retval;
att->data = NULL;
att->len = 0;
att->dirty = 0;
}
}
return NC_NOERR;
}
/**
* @internal Remove a NC_ATT_INFO_T from an index.
* This will nc_free the memory too.
@ -1550,6 +1621,15 @@ nc4_nc4f_list_del(NC_FILE_INFO_T *h5)
assert(h5);
/* Order is important here. We must delete the attribute contents
before deleteing any metadata because nc_reclaim_data depends
on the existence of the type info.
*/
/* Delete all the attribute data contents in each group and variable. */
if ((retval = nc4_rec_grp_del_att_data(h5->root_grp)))
return retval;
/* Delete all the list contents for vars, dims, and atts, in each
* group. */
if ((retval = nc4_rec_grp_del(h5->root_grp)))
@ -1866,3 +1946,4 @@ NC4_move_in_NCList(NC* nc, int new_id)
}
return stat;
}

55
libsrc4/nc4type.c
View File

@ -14,6 +14,10 @@
#include "nc4internal.h"
#include "nc4dispatch.h"
#ifdef ENABLE_DAP4
EXTERNL int NCD4_get_substrate(int ncid);
#endif
/* The sizes of types may vary from platform to platform, but within
* netCDF files, type sizes are fixed. */
#define NC_CHAR_LEN sizeof(char) /**< @internal Size of char. */
@ -702,3 +706,54 @@ nc4_get_typeclass(const NC_FILE_INFO_T *h5, nc_type xtype, int *type_class)
exit:
return retval;
}
/**
* @internal return 1 if type is fixed size; 0 otherwise.
*
* @param ncid file id
* @param xtype type id
* @param fixedsizep pointer into which 1/0 is stored
*
* @return ::NC_NOERR
* @return ::NC_EBADTYPE if bad type
* @author Dennis Heimbigner
*/
int
NC4_inq_type_fixed_size(int ncid, nc_type xtype, int* fixedsizep)
{
int stat = NC_NOERR;
NC* nc = NULL;
int f = 0;
int xclass;
if ((stat = NC_check_id(ncid, &nc))) goto done;
if(xtype < NC_STRING) {f = 1; goto done;}
if(xtype == NC_STRING) {f = 0; goto done;}
#ifdef USE_NETCDF4
/* Must be user type */
if((stat = nc_inq_user_type(ncid,xtype,NULL,NULL,NULL,NULL,&xclass))) goto done;
switch (xclass) {
case NC_ENUM: case NC_OPAQUE: f = 1; break;
case NC_VLEN: f = 0; break;
case NC_COMPOUND: {
NC_FILE_INFO_T* h5 = NULL;
NC_TYPE_INFO_T* typ = NULL;
#ifdef ENABLE_DAP4
int xformat = xformat = nc->dispatch->model;
if(xformat == NC_FORMATX_DAP4) {
ncid = NCD4_get_substrate(ncid);
} /* Fall thru */
#endif
if ((stat = nc4_find_grp_h5(ncid, NULL, &h5)))
goto done;
if((stat = nc4_find_type(h5,xtype,&typ))) goto done;
f = !typ->u.c.varsized;
} break;
default: stat = NC_EBADTYPE; goto done;
}
#endif
done:
if(fixedsizep) *fixedsizep = f;
return stat;
}

9
libsrc4/nc4var.c
View File

@ -264,6 +264,7 @@ NC4_inq_var_all(int ncid, int varid, char *name, nc_type *xtypep,
{
/* Do we have a fill value for this var? */
if (var->fill_value)
#ifdef SEPDATA
{
if (var->type_info->nc_type_class == NC_STRING)
{
@ -281,9 +282,15 @@ NC4_inq_var_all(int ncid, int varid, char *name, nc_type *xtypep,
memcpy(fill_valuep, var->fill_value, var->type_info->size);
}
}
#else
{
int xtype = var->type_info->hdr.id;
if((retval = nc_copy_data(ncid,xtype,var->fill_value,1,fill_valuep))) return retval;
}
#endif
else
{
if (var->type_info->nc_type_class == NC_STRING)
if (var->type_info->nc_type_class == NC_STRING)
{
if (!(*(char **)fill_valuep = calloc(1, sizeof(char *))))
return NC_ENOMEM;

1
nc_test4/CMakeLists.txt
View File

@ -31,6 +31,7 @@ IF(BUILD_UTILITIES)
build_bin_test(renamegroup)
add_sh_test(nc_test4 run_grp_rename)
build_bin_test(tst_charvlenbug)
build_bin_test(tst_vlenstr)
ADD_SH_TEST(nc_test4 tst_misc)
build_bin_test(tst_fillonly)
ADD_SH_TEST(nc_test4 test_fillonly)

2
nc_test4/Makefile.am
View File

@ -43,7 +43,7 @@ NC4_TESTS += tst_h_strbug tst_h_refs
endif
# Build test programs plus programs used in test scripts.
check_PROGRAMS = $(NC4_TESTS) tst_empty_vlen_unlim tst_charvlenbug
check_PROGRAMS = $(NC4_TESTS) tst_empty_vlen_unlim tst_charvlenbug tst_vlenstr
TESTS = $(NC4_TESTS) run_empty_vlen_test.sh
# Add these if large file tests are turned on.

30
nc_test4/tst_charvlenbug.c
View File

@ -5,6 +5,10 @@
#include <string.h>
#include "netcdf.h"
#define AFIRST
#undef DEBUG
#define FILE "tst_charvlenbug.nc"
void checkErrorCode(int status, const char* message){
@ -29,6 +33,7 @@ main(int argc, const char * argv[])
ptrdiff_t stride[2] = {1, 1};
nc_vlen_t vlenPointers[4];
nc_vlen_t* readVlenPointers;
int i;
#ifdef AFIRST
size_t start[2] = {1, 0};
@ -92,8 +97,8 @@ main(int argc, const char * argv[])
// calculate num elements to read
dimlen = 0;
num_items = 1;
for ( int j = 0; j < ndims; ++j ) {
retval = nc_inq_dimlen(ncid, dimids_read[j], &dimlen);
for (i = 0; i < ndims; i++) {
retval = nc_inq_dimlen(ncid, dimids_read[i], &dimlen);
checkErrorCode(retval, "nc_inq_dimlen");
num_items *= dimlen;
}
@ -103,9 +108,26 @@ main(int argc, const char * argv[])
retval = nc_get_var(ncid, varid, readVlenPointers);
checkErrorCode(retval, "nc_get_var");
#ifdef DEBUG
for(i=0;i<num_items;i++) {
int j;
char* s;
nc_vlen_t* v = &readVlenPointers[i];
fprintf(stderr,"readVlenPointers[%d] = ",i);
fprintf(stderr,"(%d,%p)",(int)v->len,v->p);
fprintf(stderr," \"");
s = (char*)v->p;
for(j=0;j<v->len;j++) fprintf(stderr,"%c",s[j]);
fprintf(stderr,"\"\n");
}
#endif
if((retval = nc_reclaim_data(ncid,typeid,readVlenPointers,num_items))) goto done;
free(readVlenPointers);
retval = nc_close(ncid);
checkErrorCode(retval, "nc_close(2)");
free(readVlenPointers);
done:
return retval;
}

6
nc_test4/tst_misc.sh
View File

@ -16,7 +16,6 @@ echo "*** Fail: phony dimension creation"
ECODE=1
fi
if test "x$NC_VLEN_NOTEST" = x1 ; then
echo "*** Testing char(*) type printout error in ncdump"
rm -f ./tst_charvlenbug.nc ./tmp
${execdir}/tst_charvlenbug
@ -26,7 +25,10 @@ else
echo "*** Fail: char(*) ncdump printout"
ECODE=1
fi
fi
echo "*** Testing char(*) type "
rm -f ./tst_vlenstr.nc ./tmp
${execdir}/tst_vlenstr
rm -f tmp

166
nc_test4/tst_vlenstr.c Normal file
View File

@ -0,0 +1,166 @@
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "netcdf.h"
#define DEBUG
#ifdef DEBUG
static char* buf = NULL;
#endif
#define totalNumStrings 13
static const char* charPointers[totalNumStrings] =
{"banana", "kiwi", "apple", "grape", "apple", "cherry", "lemon", "melon", "watermelon", "strawberry", "peach", "raspberry", "pineapple"};
#define first_size 2
#define second_size 1
#define third_size 5
#define fourth_size 3
#define fifth_size 0
#define sixth_size 2
#define attlength 6
void
checkErrorCode(int status, const char* message)
{
if (status != NC_NOERR){
fprintf(stderr,"Error code: %d from %s\n",status,message);
fprintf(stderr,"\t%s\n\n",nc_strerror(status));
}
}
void
writeVariable(int dimlength, int ncid, nc_type vlen_typeID)
{
int retval = NC_NOERR;
int dimid;
int varid;
/* Setup data */
nc_vlen_t* data = calloc(sizeof(nc_vlen_t),dimlength);
/* create six variable length arrays of strings */
int stringIndex = 0;
data[0].len = first_size;
data[0].p = charPointers+stringIndex;
stringIndex += first_size;
data[1].len = second_size;
data[1].p = charPointers+stringIndex;
stringIndex += second_size;
data[2].len = third_size;
data[2].p = charPointers+stringIndex;
stringIndex += third_size;
data[3].len = fourth_size;
data[3].p = charPointers+stringIndex;
stringIndex += fourth_size;
data[4].len = fifth_size;
data[4].p = charPointers+stringIndex;
stringIndex += fifth_size;
data[5].len = sixth_size;
data[5].p = charPointers+stringIndex;
stringIndex += sixth_size;
/* Write vlen variable named Fruits */
/* Define dimension */
retval = nc_def_dim(ncid, "D1", dimlength, &dimid);
checkErrorCode(retval, "nc_def_dim");
/* Define variable */
retval = nc_def_var(ncid, "Fruits", vlen_typeID, 1, &dimid, &varid);
checkErrorCode(retval, "nc_def_var");
/* Write variable */
retval = nc_put_var(ncid, varid, data);
checkErrorCode(retval, "nc_put_var");
free(data);
}
void
writeAttribute(int len, int ncid, nc_type vlen_typeID)
{
int retval = NC_NOERR;
/* Setup data */
nc_vlen_t* data = calloc(sizeof(nc_vlen_t),len);
/* create six variable length arrays of strings */
int stringIndex = 0;
data[0].len = first_size;
data[0].p = charPointers+stringIndex;
stringIndex += first_size;
data[1].len = second_size;
data[1].p = charPointers+stringIndex;
stringIndex += second_size;
data[2].len = third_size;
data[2].p = charPointers+stringIndex;
stringIndex += third_size;
data[3].len = fourth_size;
data[3].p = charPointers+stringIndex;
stringIndex += fourth_size;
data[4].len = fifth_size;
data[4].p = charPointers+stringIndex;
stringIndex += fifth_size;
data[5].len = sixth_size;
data[5].p = charPointers+stringIndex;
stringIndex += sixth_size;
#ifdef DEBUG
if(buf) free(buf); buf = NULL;
nc_dump_data(ncid,vlen_typeID,data,len,&buf);
fprintf(stderr,">>> attribute = %s\n",buf);
#endif
/* Write vlen attribute named Fruits */
retval = nc_put_att(ncid, NC_GLOBAL, "Fruit", vlen_typeID, len, data);
checkErrorCode(retval, "nc_put_att");
free(data);
}
int
main(int argc, const char * argv[])
{
/* Open file */
int ncid;
int retval;
nc_type vlen_typeID;
retval = nc_create("tst_vlenstr.nc", NC_NETCDF4, &ncid);
checkErrorCode(retval, "nc_create");
/* Define vlen type named MY_VLEN_STRING */
retval = nc_def_vlen(ncid, "MY_VLEN_STRING", NC_STRING, &vlen_typeID);
checkErrorCode(retval, "nc_def_vlen");
/* write variable - will be able to read it back fine */
writeVariable(attlength, ncid, vlen_typeID);
/* write attribute - will read back garbled values */
writeAttribute(attlength, ncid, vlen_typeID);
retval = nc_close(ncid);
checkErrorCode(retval, "nc_close");
#ifdef DEBUG
if(buf) free(buf);
#endif
return retval;
}

14
ncdump/CMakeLists.txt
View File

@ -17,6 +17,7 @@ SET(ocprint_FILES ocprint.c)
SET(ncvalidator_FILES ncvalidator.c)
SET(printfqn_FILES printfqn.c)
SET(ncpathcvt_FILES ncpathcvt.c)
SET(nchdf5version_FILES nchdf5version.c)
IF(USE_X_GETOPT)
SET(ncdump_FILES ${ncdump_FILES} XGetopt.c)
@ -35,6 +36,7 @@ ADD_EXECUTABLE(ncpathcvt ${ncpathcvt_FILES})
IF(USE_HDF5)
ADD_EXECUTABLE(nc4print nc4print.c nc4printer.c)
ADD_EXECUTABLE(printfqn ${printfqn_FILES})
ADD_EXECUTABLE(nchdf5version ${nchdf5version_FILES})
ENDIF(USE_HDF5)
IF(ENABLE_DAP)
@ -49,6 +51,7 @@ TARGET_LINK_LIBRARIES(ncpathcvt netcdf ${ALL_TLL_LIBS})
IF(USE_HDF5)
TARGET_LINK_LIBRARIES(nc4print netcdf ${ALL_TLL_LIBS})
TARGET_LINK_LIBRARIES(printfqn netcdf ${ALL_TLL_LIBS})
TARGET_LINK_LIBRARIES(nchdf5version netcdf ${ALL_TLL_LIBS})
ENDIF(USE_HDF5)
IF(ENABLE_DAP)
@ -97,6 +100,12 @@ IF(MSVC)
${CMAKE_CURRENT_BINARY_DIR})
SET_TARGET_PROPERTIES(printfqn PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE
${CMAKE_CURRENT_BINARY_DIR})
SET_TARGET_PROPERTIES(nchdf5version PROPERTIES RUNTIME_OUTPUT_DIRECTORY
${CMAKE_CURRENT_BINARY_DIR})
SET_TARGET_PROPERTIES(nchdf5version PROPERTIES RUNTIME_OUTPUT_DIRECTORY_DEBUG
${CMAKE_CURRENT_BINARY_DIR})
SET_TARGET_PROPERTIES(nchdf5version PROPERTIES RUNTIME_OUTPUT_DIRECTORY_RELEASE
${CMAKE_CURRENT_BINARY_DIR})
ENDIF(USE_HDF5)
IF(ENABLE_DAP)
@ -295,11 +304,6 @@ endif()
ENDIF()
ENDIF(EXTRA_TESTS)
IF(ENABLE_UNFIXED_MEMORY_LEAKS)
SET_TESTS_PROPERTIES(ncdump_run_ncgen_nc4_tests.sh ncdump_tst_nccopy4.sh ncdump_tst_ncgen_shared.sh ncdump_tst_netcdf4.sh
PROPERTIES ENVIRONMENT NC_VLEN_NOTEST=1)
ENDIF()
# The unicode tests are complicated
IF(USE_HDF5)
IF(NOT MSVC AND NOT MINGW)

13
ncdump/Makefile.am
View File

@ -17,10 +17,6 @@ noinst_PROGRAMS=
TEST_EXTENSIONS = .sh
XFAIL_TESTS=""
if ! ENABLE_UNFIXED_MEMORY_LEAKS
AM_TESTS_ENVIRONMENT += export NC_VLEN_NOTEST=1;
endif
# This is the program we're building, and it's sources.
bin_PROGRAMS = ncdump
ncdump_SOURCES = ncdump.c vardata.c dumplib.c indent.c nctime0.c \
@ -57,6 +53,13 @@ nc4print_SOURCES = nc4print.c nc4printer.c
noinst_PROGRAMS += printfqn
printfqn_SOURCES = printfqn.c
# A non-installed utility program to return
# the HDF5 Version as an integer of the form XXXYYYZZZ with no newline.
# Where XXX is the major version number, YYY is the minor version number
# and ZZZ is the patch level.
noinst_PROGRAMS += nchdf5version
nchdf5version_SOURCES = nchdf5version.c
endif
# Conditionally build the ocprint program, but do not install
@ -117,7 +120,7 @@ tst_ncgen4.sh test_scope.sh
# Record interscript dependencies so parallel builds work.
tst_nccopy4.log: run_ncgen_tests.log tst_output.log tst_ncgen4.log \
tst_fillbug.log tst_netcdf4_4.log tst_h_scalar.log
tst_fillbug.log tst_netcdf4_4.log tst_h_scalar.log tst_netcdf4.log
tst_nccopy5.log: tst_nccopy4.log
endif #USE_HDF5

2
ncdump/nccopy.c
View File

@ -244,7 +244,7 @@ computeFQN(VarID vid, char** fqnp)
*q++ = '\0'; /* guarantee */
strcat(fqn,escname);
done:
if(stat == NC_NOERR && fqnp != NULL) *fqnp = fqn;
if(stat == NC_NOERR && fqnp != NULL) {*fqnp = fqn; fqn = NULL;}
return stat;
}

8
ncdump/ncdump.c
View File

@ -917,8 +917,7 @@ pr_att(
default:
error("unrecognized class of user defined type: %d", class);
}
ncaux_reclaim_data(ncid,att.type,data,att.len);
free(data);
NC_CHECK(nc_reclaim_data_all(ncid,att.type,data,att.len));
printf (" ;"); /* terminator for user defined types */
}
#endif /* USE_NETCDF4 */
@ -1957,7 +1956,7 @@ do_ncdump_rec(int ncid, const char *path)
goto done;
}
if(var.fillvalp != NULL)
{ncaux_reclaim_data(ncid,var.tinfo->tid,var.fillvalp,1); free(var.fillvalp); var.fillvalp = NULL;}
{NC_CHECK(nc_reclaim_data_all(ncid,var.tinfo->tid,var.fillvalp,1)); var.fillvalp = NULL;}
}
if (vdims) {
free(vdims);
@ -2010,8 +2009,7 @@ done:
if(var.dims != NULL) free(var.dims);
if(var.fillvalp != NULL) {
/* Release any data hanging off of fillvalp */
ncaux_reclaim_data(ncid,var.tinfo->tid,var.fillvalp,1);
free(var.fillvalp);
nc_reclaim_data_all(ncid,var.tinfo->tid,var.fillvalp,1);
var.fillvalp = NULL;
}
if(var.timeinfo != NULL) {

40
ncdump/nchdf5version.c Normal file
View File

@ -0,0 +1,40 @@
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <assert.h>
#include <hdf5.h>
#include <netcdf.h>
#define LEN 3
int
main(int argc, char** argv)
{
char v5[256];
char vn[256];
char* major, *minor, *patch;
char* p;
int i;
strcpy(v5,H5_VERSION);
major = v5; minor = NULL; patch = NULL;
if((p == strchr(major,'.'))) {*p++ = '\0'; minor = p;}
if((p == strchr(minor,'.'))) {*p++ = '\0'; patch = p;}
vn[0]='\0';
i = LEN - strlen(major);
while(i-- > 0) strcat(vn,"0");
strcat(vn,major);
i = LEN - strlen(minor);
while(i-- > 0) strcat(vn,"0");
strcat(vn,minor);
i = LEN - strlen(patch);
while(i-- > 0) strcat(vn,"0");
strcat(vn,patch);
printf("%s",vn);
return 0;
}

1
ncdump/ref_tst_utf8.cdl
View File

@ -4,6 +4,7 @@ dimensions:
㼿y = 2 ;
variables:
int 􍐪(xā, 㼿y) ;
􍐪:_FillValue = -2147483647 ;
// global attributes:
:Gā = "ā㼿y􍐪" ;

2
ncdump/run_ncgen_nc4_tests.sh
View File

@ -46,11 +46,9 @@ ${NCGEN} -lc $top_srcdir/ncgen/ref_camrun.cdl > camrun.c
echo "*** test for jira NCF-199 bug"
validateNC "ncf199" "ncf199" -k nc4
if test "x$NC_VLEN_NOTEST" = x ; then
echo "*** creating binary files for github issue 323..."
echo "*** github issue 323 test 1"
validateNC "compound_datasize_test" "compound_datasize_test" -k nc4
fi
echo "*** github issue 323 test 2"
validateNC "compound_datasize_test2" "compound_datasize_test2" -k nc4

5
ncdump/tst_nccopy4.sh
View File

@ -24,9 +24,8 @@ TESTFILES='tst_comp tst_comp2 tst_enum_data tst_fillbug
tst_group_data tst_nans tst_opaque_data tst_solar_1 tst_solar_2
tst_solar_cmp tst_special_atts tst_string_data'
if test "x$NC_VLEN_NOTEST" = x ; then
TESTFILES="$TESTFILES tst_vlen_data"
fi
# Causes memory leak; source unknown
MEMLEAK="tst_vlen_data"
echo "*** Testing netCDF-4 features of nccopy on ncdump/*.nc files"
for i in $TESTFILES ; do

40
ncdump/tst_nccopy5.sh
View File

@ -70,7 +70,7 @@ cleanup() {
# remove all created files
reset() {
cleanup
rm -fr tst_nc5.nc tst_nc5.cdl
rm -fr tst_nc5.nc tst_nc5.cdl tmp.cdl
rm -f tst_nc5_omit.nc tst_nc5_omit.cdl
}
@ -79,21 +79,21 @@ reset
if test "x$T1" = x1 ; then
# Create a simple classic input file
./tst_chunking tst_nc5.nc
${execdir}/tst_chunking tmp_nc5_base.nc
# Save a .cdl version
${NCDUMP} tst_nc5.nc > tst_nc5.cdl
${NCDUMP} tmp_nc5_base.nc > tmp_nc5_base.cdl
echo "*** Test nccopy -c with per-variable chunking; classic->enhanced"
# This should produce same as -c dim0/,dim1/1,dim2/,dim3/1,dim4/,dim5/1,dim6/
# But note that the chunk product is less than default, so we need to reduce it (-M)
${NCCOPY} -M1000 -c ivar:7,1,2,1,5,1,9 tst_nc5.nc tmp_nc5.nc
${NCDUMP} -n tst_nc5 tmp_nc5.nc > tmp_nc5.cdl
${NCCOPY} -M1000 -c ivar:7,1,2,1,5,1,9 tmp_nc5_base.nc tmp_nc5.nc
${NCDUMP} -n tmp_nc5_base tmp_nc5.nc > tmp_nc5.cdl
# Verify that the core cdl is the same
diff tst_nc5.cdl tmp_nc5.cdl
diff tmp_nc5_base.cdl tmp_nc5.cdl
# Look at the output chunking of ivar
rm -f tmp_nc5a.cdl # reuse
${NCDUMP} -hs -n tst_nc5 tmp_nc5.nc > tmp_nc5.cdl
${NCDUMP} -hs -n tmp_nc5_base tmp_nc5.nc > tmp_nc5.cdl
# extract the chunking line
TESTLINE=`sed -e '/ivar:_ChunkSizes/p' -e d <tmp_nc5.cdl`
# track line to match
@ -108,16 +108,16 @@ if test "x$T2" = x1 ; then
echo "*** Test nccopy -c with per-variable chunking; enhanced->enhanced"
reset
./tst_chunking tst_nc5.nc deflate
${NCDUMP} -n tst_nc5 tst_nc5.nc > tst_nc5.cdl
${execdir}/tst_chunking tst_nc5.nc deflate
${NCDUMP} -n tmp_nc5_base tst_nc5.nc > tst_nc5.cdl
# Use -M to ensure that chunking takes effect
${NCCOPY} -M500 -c ivar:4,1,2,1,5,2,3 tst_nc5.nc tmp_nc5.nc
${NCDUMP} -n tst_nc5 tmp_nc5.nc > tmp_nc5.cdl
${NCDUMP} -n tmp_nc5_base tmp_nc5.nc > tmp_nc5.cdl
diff tst_nc5.cdl tmp_nc5.cdl
# Look at the output chunking
rm -f tmp_nc5.cdl # reuse
${NCDUMP} -hs -n tst_nc5 tmp_nc5.nc > tmp_nc5.cdl
${NCDUMP} -hs -n tmp_nc5_base tmp_nc5.nc > tmp_nc5.cdl
# extract the chunking line
TESTLINE=`sed -e '/ivar:_ChunkSizes/p' -e d <tmp_nc5.cdl`
# track line to match
@ -132,14 +132,14 @@ if test "x$T3" = x1 ; then
echo "*** Test nccopy -c with FQN var name; enhanced ->enhanced"
reset
./tst_chunking tst_nc5.nc group
${NCDUMP} -n tst_nc5 tst_nc5.nc > tst_nc5.cdl
${execdir}/tst_chunking tst_nc5.nc group
${NCDUMP} -n tmp_nc5_base tst_nc5.nc > tst_nc5.cdl
${NCCOPY} -M500 -c /g/ivar:4,1,2,1,5,2,3 tst_nc5.nc tmp_nc5.nc
${NCDUMP} -n tst_nc5 tmp_nc5.nc > tmp_nc5.cdl
${NCDUMP} -n tmp_nc5_base tmp_nc5.nc > tmp_nc5.cdl
diff tst_nc5.cdl tmp_nc5.cdl
# Verify chunking
${NCDUMP} -hs -n tst_nc5 tmp_nc5.nc > tmp_nc5.cdl
${NCDUMP} -hs -n tmp_nc5_base tmp_nc5.nc > tmp_nc5.cdl
# extract the chunking line
TESTLINE=`sed -e '/ivar:_ChunkSizes/p' -e d <tmp_nc5.cdl`
# track line to match
@ -152,14 +152,14 @@ if test "x$T4" = x1 ; then
echo "*** Test nccopy -c with unlimited dimension; classic ->enhanced"
reset
./tst_chunking tst_nc5.nc unlimited # should produce modified tmp_nc5.nc with ivar of rank 2
${NCDUMP} -n tst_nc5 tst_nc5.nc > tst_nc5.cdl
${execdir}/tst_chunking tst_nc5.nc unlimited # should produce modified tmp_nc5.nc with ivar of rank 2
${NCDUMP} -n tmp_nc5_base tst_nc5.nc > tst_nc5.cdl
${NCCOPY} -M500 -c ivar:5,3 tst_nc5.nc tmp_nc5.nc
${NCDUMP} -n tst_nc5 tmp_nc5.nc > tmp_nc5.cdl
${NCDUMP} -n tmp_nc5_base tmp_nc5.nc > tmp_nc5.cdl
diff tst_nc5.cdl tmp_nc5.cdl
# Verify chunking
${NCDUMP} -hs -n tst_nc5 tmp_nc5.nc > tmp_nc5.cdl
${NCDUMP} -hs -n tmp_nc5_base tmp_nc5.nc > tmp_nc5.cdl
# extract the chunking line
TESTLINE=`sed -e '/ivar:_ChunkSizes/p' -e d <tmp_nc5.cdl`
# track line to match
@ -175,7 +175,7 @@ if test "x$T5" = x1 ; then
echo "*** Test nccopy -c fvar: to suppress chunking; classic ->enhanced"
reset
./tst_chunking tst_nc5_omit.nc
${execdir}/tst_chunking tst_nc5_omit.nc
${NCDUMP} -n tst_nc5_omit tst_nc5_omit.nc > tst_nc5_omit.cdl
${NCCOPY} -M500 -c ivar:7,1,2,1,5,1,9 -c fvar: tst_nc5_omit.nc tmp_nc5_omit.nc
${NCDUMP} -n tst_nc5_omit tmp_nc5_omit.nc > tmp_nc5_omit.cdl

7
ncdump/tst_ncgen_shared.sh
View File

@ -101,9 +101,10 @@ ref_tst_econst \
ref_tst_econst2 \
"
if test "x$NC_VLEN_NOTEST" = x ; then
TESTS4="$TESTS4 ref_tst_vlen_data ref_tst_vlen_data2"
fi
TESTS4="$TESTS4 ref_tst_vlen_data2"
# This causes an unfixed memory leak
MEMLEAK="ref_tst_vlen_data"
# These tests should not be cycle tested because ncdump loses information
NOCYCLE="ref_tst_econst"

12
ncdump/tst_netcdf4.sh
View File

@ -80,13 +80,6 @@ ${execdir}/tst_opaque_data ; ERR
${NCDUMP} tst_opaque_data.nc | sed 's/e+0/e+/g' > tst_opaque_data.cdl ; ERR
diff -b tst_opaque_data.cdl $srcdir/ref_tst_opaque_data.cdl ; ERR
if test "x$NC_VLEN_NOTEST" = x ; then
echo "*** Running tst_vlen_data.c to create test files."
${execdir}/tst_vlen_data ; ERR
${NCDUMP} tst_vlen_data.nc | sed 's/e+0/e+/g' > tst_vlen_data.cdl ; ERR
diff -b tst_vlen_data.cdl $srcdir/ref_tst_vlen_data.cdl ; ERR
fi
echo "*** Running tst_comp.c to create test files."
${execdir}/tst_comp ; ERR
${NCDUMP} tst_comp.nc | sed 's/e+0/e+/g' > tst_comp.cdl ; ERR
@ -105,6 +98,11 @@ cleanncprops $srcdir/ref_tst_special_atts.cdl ref_tst_special_atts.tmp
echo "*** comparing tst_special_atts.cdl with ref_tst_special_atts.cdl..."
diff -b tst_special_atts.tmp ref_tst_special_atts.tmp ; ERR
echo "*** Running tst_vlen_data.c to create test files."
if ! ${execdir}/tst_vlen_data ; then if test $? != 027 ; then ERR; fi; fi
${NCDUMP} tst_vlen_data.nc | sed 's/e+0/e+/g' > tst_vlen_data.cdl ; ERR
diff -b tst_vlen_data.cdl $srcdir/ref_tst_vlen_data.cdl ; ERR
#echo ""
#echo "*** Testing ncdump on file with corrupted header "
#rm -f ./ignore_tst_netcdf4

19
ncdump/tst_vlen_data.c
View File

@ -7,10 +7,15 @@
$Id: tst_vlen_data.c,v 1.12 2009/11/15 00:17:59 dmh Exp $
*/
/* WARNING: this test leaks memory.
The leak may be in HDF5.
*/
#include <nc_tests.h>
#include "err_macros.h"
#include <netcdf.h>
#include <stdlib.h>
#include <netcdf.h>
#include <netcdf_aux.h>
#include "err_macros.h"
#define FILE5_NAME "tst_vlen_data.nc"
#define TYPE5_NAME "row_of_floats"
@ -59,10 +64,20 @@ main(int argc, char **argv)
if (nc_def_var(ncid, VAR5_NAME, typeid, VAR5_RANK, var_dims, &varid)) ERR;
/* Create and write a variable attribute of the vlen type */
#if 0
/* In order to use ncaux_reclaim_data, all the interior nodes must have been alloc'd */
missing_val.p = (float*)malloc(sizeof(missing_value));
memcpy((void*)missing_val.p,&missing_value,sizeof(missing_value));
#else
missing_val.p = &missing_value;
#endif
missing_val.len = 1;
if (nc_put_att(ncid, varid, ATT5_NAME, typeid, ATT5_LEN, (void *) &missing_val)) ERR;
if (nc_enddef(ncid)) ERR;
#if 0
/* reclaim */
if(ncaux_reclaim_data(ncid,typeid,&missing_val,1)) ERR;
#endif
/* fill in pointers to data rows in preparation for writing */
array = (float **) malloc(NROWS * sizeof(float *));

2
ncdump/vardata.c
View File

@ -495,7 +495,7 @@ print_rows(
print_any_val(sb, vp, (void *)valp);
}
/* In case vals has memory hanging off e.g. vlen or string, make sure to reclaim it */
(void)ncaux_reclaim_data(ncid,vp->type,vals,ncols);
NC_CHECK(nc_reclaim_data(ncid,vp->type,vals,ncols));
/* determine if this is the last row */
lastrow = true;

21
ncgen/data.c
View File

@ -306,22 +306,25 @@ static const char zeros[] =
void
alignbuffer(NCConstant* prim, Bytebuffer* buf)
{
int alignment,pad,offset;
int stat = NC_NOERR;
size_t alignment;
int pad,offset;
ASSERT(prim->nctype != NC_COMPOUND);
if(prim->nctype == NC_ECONST)
alignment = ncaux_class_alignment(prim->value.enumv->typ.typecode);
stat = ncaux_class_alignment(prim->value.enumv->typ.typecode,&alignment);
else if(usingclassic && prim->nctype == NC_STRING)
alignment = ncaux_class_alignment(NC_CHAR);
stat = ncaux_class_alignment(NC_CHAR,&alignment);
else if(prim->nctype == NC_CHAR)
alignment = ncaux_class_alignment(NC_CHAR);
stat = ncaux_class_alignment(NC_CHAR,&alignment);
else
alignment = ncaux_class_alignment(prim->nctype);
offset = bbLength(buf);
pad = getpadding(offset,alignment);
if(pad > 0) {
bbAppendn(buf,(void*)zeros,pad);
stat = ncaux_class_alignment(prim->nctype,&alignment);
if(!stat) {
offset = bbLength(buf);
pad = getpadding(offset,alignment);
if(pad > 0)
bbAppendn(buf,(void*)zeros,pad);
}
}

10
ncgen/genbin.c
View File

@ -370,9 +370,7 @@ genbin_defineattr(Symbol* asym)
Bytebuffer* databuf = bbNew();
generator_reset(bin_generator,NULL);
generate_attrdata(asym,bin_generator,(Writer)genbin_write,databuf);
if((stat = ncaux_reclaim_data(asym->container->nc_id,asym->typ.basetype->nc_id,bbContents(databuf),datalistlen(asym->data))))
goto done;
done:
stat = nc_reclaim_data(asym->container->nc_id,asym->typ.basetype->nc_id,bbContents(databuf),datalistlen(asym->data));
bbFree(databuf);
return stat;
}
@ -388,7 +386,7 @@ genbin_definevardata(Symbol* vsym)
databuf = bbNew();
generator_reset(bin_generator,NULL);
generate_vardata(vsym,bin_generator,(Writer)genbin_write,databuf);
ncaux_reclaim_data(vsym->container->nc_id,vsym->typ.basetype->nc_id,bbContents(databuf),datalistlen(vsym->data));
stat = nc_reclaim_data_all(vsym->container->nc_id,vsym->typ.basetype->nc_id,bbExtract(databuf),datalistlen(vsym->data));
done:
bbFree(databuf);
return stat;
@ -443,7 +441,7 @@ genbin_writevar(Generator* generator, Symbol* vsym, Bytebuffer* memory,
CHECK_ERR(stat);
#if 0
/* Reclaim the data */
stat = ncaux_reclaim_data(vsym->container->nc_id, vsym->typ.basetype->nc_id, data, nelems);
stat = nc_reclaim_data(vsym->container->nc_id, vsym->typ.basetype->nc_id, data, nelems);
CHECK_ERR(stat);
bbClear(memory); /* reclaim top-level memory */
#endif
@ -468,7 +466,7 @@ genbin_writeattr(Generator* generator, Symbol* asym, Bytebuffer* databuf,
len = list->length;
/* Use the specialized put_att_XX routines if possible*/
if(isprim(basetype->typ.typecode)) {
if(isprim(typid)) {
switch (basetype->typ.typecode) {
case NC_BYTE: {
signed char* data = (signed char*)bbContents(databuf);

6
ncgen/ncgen.y
View File

@ -372,7 +372,7 @@ opaquedecl: OPAQUE_ '(' INT_CONST ')' typename
$5->subclass=NC_OPAQUE;
$5->typ.typecode=NC_OPAQUE;
$5->typ.size=int32_val;
$5->typ.alignment=ncaux_class_alignment(NC_OPAQUE);
(void)ncaux_class_alignment(NC_OPAQUE,&$5->typ.alignment);
}
;
@ -386,7 +386,7 @@ vlendecl: typeref '(' '*' ')' typename
$5->typ.basetype=basetype;
$5->typ.typecode=NC_VLEN;
$5->typ.size=VLENSIZE;
$5->typ.alignment=ncaux_class_alignment(NC_VLEN);
(void)ncaux_class_alignment(NC_VLEN,&$5->typ.alignment);
}
;
@ -971,7 +971,7 @@ makeprimitivetype(nc_type nctype)
sym->typ.typecode = nctype;
sym->typ.size = ncsize(nctype);
sym->typ.nelems = 1;
sym->typ.alignment = ncaux_class_alignment(nctype);
(void)ncaux_class_alignment(nctype,&sym->typ.alignment);
/* Make the basetype circular so we can always ask for it */
sym->typ.basetype = sym;
sym->prefix = listnew();

870
ncgen/ncgenl.c
View File

File diff suppressed because it is too large Load Diff

2241
ncgen/ncgeny.c
View File

File diff suppressed because it is too large Load Diff

135
ncgen/ncgeny.h
View File

@ -1,9 +1,8 @@
/* A Bison parser, made by GNU Bison 3.8.2. */
/* A Bison parser, made by GNU Bison 3.0.4. */
/* Bison interface for Yacc-like parsers in C
Copyright (C) 1984, 1989-1990, 2000-2015, 2018-2021 Free Software Foundation,
Inc.
Copyright (C) 1984, 1989-1990, 2000-2015 Free Software Foundation, Inc.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@ -16,7 +15,7 @@
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>. */
along with this program. If not, see <http://www.gnu.org/licenses/>. */
/* As a special exception, you may create a larger work that contains
part or all of the Bison parser skeleton and distribute that work
@ -31,10 +30,6 @@
This special exception was added by the Free Software Foundation in
version 2.2 of Bison. */
/* DO NOT RELY ON FEATURES THAT ARE NOT DOCUMENTED in the manual,
especially those whose name start with YY_ or yy_. They are
private implementation details that can be changed or removed. */
#ifndef YY_NCG_NCGEN_TAB_H_INCLUDED
# define YY_NCG_NCGEN_TAB_H_INCLUDED
/* Debug traces. */
@ -45,78 +40,74 @@
extern int ncgdebug;
#endif
/* Token kinds. */
/* Token type. */
#ifndef YYTOKENTYPE
# define YYTOKENTYPE
enum yytokentype
{
YYEMPTY = -2,
YYEOF = 0, /* "end of file" */
YYerror = 256, /* error */
YYUNDEF = 257, /* "invalid token" */
NC_UNLIMITED_K = 258, /* NC_UNLIMITED_K */
CHAR_K = 259, /* CHAR_K */
BYTE_K = 260, /* BYTE_K */
SHORT_K = 261, /* SHORT_K */
INT_K = 262, /* INT_K */
FLOAT_K = 263, /* FLOAT_K */
DOUBLE_K = 264, /* DOUBLE_K */
UBYTE_K = 265, /* UBYTE_K */
USHORT_K = 266, /* USHORT_K */
UINT_K = 267, /* UINT_K */
INT64_K = 268, /* INT64_K */
UINT64_K = 269, /* UINT64_K */
STRING_K = 270, /* STRING_K */
IDENT = 271, /* IDENT */
TERMSTRING = 272, /* TERMSTRING */
CHAR_CONST = 273, /* CHAR_CONST */
BYTE_CONST = 274, /* BYTE_CONST */
SHORT_CONST = 275, /* SHORT_CONST */
INT_CONST = 276, /* INT_CONST */
INT64_CONST = 277, /* INT64_CONST */
UBYTE_CONST = 278, /* UBYTE_CONST */
USHORT_CONST = 279, /* USHORT_CONST */
UINT_CONST = 280, /* UINT_CONST */
UINT64_CONST = 281, /* UINT64_CONST */
FLOAT_CONST = 282, /* FLOAT_CONST */
DOUBLE_CONST = 283, /* DOUBLE_CONST */
DIMENSIONS = 284, /* DIMENSIONS */
VARIABLES = 285, /* VARIABLES */
NETCDF = 286, /* NETCDF */
DATA = 287, /* DATA */
TYPES = 288, /* TYPES */
COMPOUND = 289, /* COMPOUND */
ENUM = 290, /* ENUM */
OPAQUE_ = 291, /* OPAQUE_ */
OPAQUESTRING = 292, /* OPAQUESTRING */
GROUP = 293, /* GROUP */
PATH = 294, /* PATH */
FILLMARKER = 295, /* FILLMARKER */
NIL = 296, /* NIL */
_FILLVALUE = 297, /* _FILLVALUE */
_FORMAT = 298, /* _FORMAT */
_STORAGE = 299, /* _STORAGE */
_CHUNKSIZES = 300, /* _CHUNKSIZES */
_DEFLATELEVEL = 301, /* _DEFLATELEVEL */
_SHUFFLE = 302, /* _SHUFFLE */
_ENDIANNESS = 303, /* _ENDIANNESS */
_NOFILL = 304, /* _NOFILL */
_FLETCHER32 = 305, /* _FLETCHER32 */
_NCPROPS = 306, /* _NCPROPS */
_ISNETCDF4 = 307, /* _ISNETCDF4 */
_SUPERBLOCK = 308, /* _SUPERBLOCK */
_FILTER = 309, /* _FILTER */
_CODECS = 310, /* _CODECS */
DATASETID = 311 /* DATASETID */
NC_UNLIMITED_K = 258,
CHAR_K = 259,
BYTE_K = 260,
SHORT_K = 261,
INT_K = 262,
FLOAT_K = 263,
DOUBLE_K = 264,
UBYTE_K = 265,
USHORT_K = 266,
UINT_K = 267,
INT64_K = 268,
UINT64_K = 269,
STRING_K = 270,
IDENT = 271,
TERMSTRING = 272,
CHAR_CONST = 273,
BYTE_CONST = 274,
SHORT_CONST = 275,
INT_CONST = 276,
INT64_CONST = 277,
UBYTE_CONST = 278,
USHORT_CONST = 279,
UINT_CONST = 280,
UINT64_CONST = 281,
FLOAT_CONST = 282,
DOUBLE_CONST = 283,
DIMENSIONS = 284,
VARIABLES = 285,
NETCDF = 286,
DATA = 287,
TYPES = 288,
COMPOUND = 289,
ENUM = 290,
OPAQUE_ = 291,
OPAQUESTRING = 292,
GROUP = 293,
PATH = 294,
FILLMARKER = 295,
NIL = 296,
_FILLVALUE = 297,
_FORMAT = 298,
_STORAGE = 299,
_CHUNKSIZES = 300,
_DEFLATELEVEL = 301,
_SHUFFLE = 302,
_ENDIANNESS = 303,
_NOFILL = 304,
_FLETCHER32 = 305,
_NCPROPS = 306,
_ISNETCDF4 = 307,
_SUPERBLOCK = 308,
_FILTER = 309,
_CODECS = 310,
DATASETID = 311
};
typedef enum yytokentype yytoken_kind_t;
#endif
/* Value type. */
#if ! defined YYSTYPE && ! defined YYSTYPE_IS_DECLARED
union YYSTYPE
{
#line 156 "ncgen.y"
#line 156 "ncgen.y" /* yacc.c:1909 */
Symbol* sym;
unsigned long size; /* allow for zero size to indicate e.g. UNLIMITED*/
@ -125,9 +116,9 @@ int nctype; /* for tracking attribute list type*/
Datalist* datalist;
NCConstant* constant;
#line 129 "ncgeny.h"
#line 120 "ncgeny.h" /* yacc.c:1909 */
};
typedef union YYSTYPE YYSTYPE;
# define YYSTYPE_IS_TRIVIAL 1
# define YYSTYPE_IS_DECLARED 1
@ -136,8 +127,6 @@ typedef union YYSTYPE YYSTYPE;
extern YYSTYPE ncglval;
int ncgparse (void);
#endif /* !YY_NCG_NCGEN_TAB_H_INCLUDED */

4
ncgen/semantics.c
View File

@ -602,12 +602,12 @@ computesize(Symbol* tsym)
case NC_VLEN: /* actually two sizes for vlen*/
computesize(tsym->typ.basetype); /* first size*/
tsym->typ.size = ncsize(tsym->typ.typecode);
tsym->typ.alignment = ncaux_class_alignment(tsym->typ.typecode);
(void)ncaux_class_alignment(tsym->typ.typecode,&tsym->typ.alignment);
tsym->typ.nelems = 1; /* always a single compound datalist */
break;
case NC_PRIM:
tsym->typ.size = ncsize(tsym->typ.typecode);
tsym->typ.alignment = ncaux_class_alignment(tsym->typ.typecode);
(void)ncaux_class_alignment(tsym->typ.typecode,&tsym->typ.alignment);
tsym->typ.nelems = 1;
break;
case NC_OPAQUE:

857
ncgen3/ncgenl.c
View File

File diff suppressed because it is too large Load Diff

1463
ncgen3/ncgeny.c
View File

File diff suppressed because it is too large Load Diff

60
ncgen3/ncgeny.h
View File

@ -1,9 +1,8 @@
/* A Bison parser, made by GNU Bison 3.8.2. */
/* A Bison parser, made by GNU Bison 3.0.4. */
/* Bison interface for Yacc-like parsers in C
Copyright (C) 1984, 1989-1990, 2000-2015, 2018-2021 Free Software Foundation,
Inc.
Copyright (C) 1984, 1989-1990, 2000-2015 Free Software Foundation, Inc.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
@ -16,7 +15,7 @@
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>. */
along with this program. If not, see <http://www.gnu.org/licenses/>. */
/* As a special exception, you may create a larger work that contains
part or all of the Bison parser skeleton and distribute that work
@ -31,10 +30,6 @@
This special exception was added by the Free Software Foundation in
version 2.2 of Bison. */
/* DO NOT RELY ON FEATURES THAT ARE NOT DOCUMENTED in the manual,
especially those whose name start with YY_ or yy_. They are
private implementation details that can be changed or removed. */
#ifndef YY_NCG_NCGEN_TAB_H_INCLUDED
# define YY_NCG_NCGEN_TAB_H_INCLUDED
/* Debug traces. */
@ -45,37 +40,32 @@
extern int ncgdebug;
#endif
/* Token kinds. */
/* Token type. */
#ifndef YYTOKENTYPE
# define YYTOKENTYPE
enum yytokentype
{
YYEMPTY = -2,
YYEOF = 0, /* "end of file" */
YYerror = 256, /* error */
YYUNDEF = 257, /* "invalid token" */
NC_UNLIMITED_K = 258, /* NC_UNLIMITED_K */
BYTE_K = 259, /* BYTE_K */
CHAR_K = 260, /* CHAR_K */
SHORT_K = 261, /* SHORT_K */
INT_K = 262, /* INT_K */
FLOAT_K = 263, /* FLOAT_K */
DOUBLE_K = 264, /* DOUBLE_K */
IDENT = 265, /* IDENT */
TERMSTRING = 266, /* TERMSTRING */
BYTE_CONST = 267, /* BYTE_CONST */
CHAR_CONST = 268, /* CHAR_CONST */
SHORT_CONST = 269, /* SHORT_CONST */
INT_CONST = 270, /* INT_CONST */
FLOAT_CONST = 271, /* FLOAT_CONST */
DOUBLE_CONST = 272, /* DOUBLE_CONST */
DIMENSIONS = 273, /* DIMENSIONS */
VARIABLES = 274, /* VARIABLES */
NETCDF = 275, /* NETCDF */
DATA = 276, /* DATA */
FILLVALUE = 277 /* FILLVALUE */
NC_UNLIMITED_K = 258,
BYTE_K = 259,
CHAR_K = 260,
SHORT_K = 261,
INT_K = 262,
FLOAT_K = 263,
DOUBLE_K = 264,
IDENT = 265,
TERMSTRING = 266,
BYTE_CONST = 267,
CHAR_CONST = 268,
SHORT_CONST = 269,
INT_CONST = 270,
FLOAT_CONST = 271,
DOUBLE_CONST = 272,
DIMENSIONS = 273,
VARIABLES = 274,
NETCDF = 275,
DATA = 276,
FILLVALUE = 277
};
typedef enum yytokentype yytoken_kind_t;
#endif
/* Value type. */
@ -88,8 +78,6 @@ typedef int YYSTYPE;
extern YYSTYPE ncglval;
int ncgparse (void);
#endif /* !YY_NCG_NCGEN_TAB_H_INCLUDED */

4
oc2/Makefile.am
View File

@ -39,9 +39,7 @@ liboc_la_CPPFLAGS = $(AM_CPPFLAGS)
# BTW: note that renaming is essential because otherwise
# autoconf will forcibly delete files of the name *.tab.*
.PHONEY: bison
bison:: dap.y
makeparser::
rm -f dap.tab.c dap.tab.h dapy.c dapy.h
bison --debug -d -p dap dap.y
sed -e 's/dap[.]tab[.]c/dapy.c/g' -e 's/dap[.]tab[.]h/dapy.h/g' <dap.tab.c >dapy.c