mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2025-02-11 16:40:36 +08:00
Code Issues Packages Projects Releases Wiki Activity
375beee9f1
netcdf-c/ncgen/ncgeny.c

3914 lines
130 KiB
C
Raw Normal View History

Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* A Bison parser, made by GNU Bison 3.7.5. */
Resolves Github issue https://github.com/Unidata/netcdf-c/issues/349. Update utf8proc.[ch] to use the version now maintained by the Julia Language project (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). The license for the previous version was unacceptable for the Debian and Ubuntu release systems. The new version both updates the code and addresses the license issue. It turns out that the utf8proc software we are using was turned over to the Julia Language developers and the license terms changed to allow modification. (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). So the fix here is as follows: 1. Wrap the library with a fixed interface: libdispatch/dutf8.c and include/ncutf8.h. 2. Replace the existing utf8proc code with the new version from https://github.com/JuliaLang/utf8proc. 3. Add a couple more test cases: nc_test/tst_utf8_validate.c and nc_test_utf8_phrases.c. If/when I can find a usable normalization test, I will incorporate that later.
2017-02-17 05:27:54 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
/* Bison implementation for Yacc-like parsers in C
Resolves Github issue https://github.com/Unidata/netcdf-c/issues/349. Update utf8proc.[ch] to use the version now maintained by the Julia Language project (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). The license for the previous version was unacceptable for the Debian and Ubuntu release systems. The new version both updates the code and addresses the license issue. It turns out that the utf8proc software we are using was turned over to the Julia Language developers and the license terms changed to allow modification. (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). So the fix here is as follows: 1. Wrap the library with a fixed interface: libdispatch/dutf8.c and include/ncutf8.h. 2. Replace the existing utf8proc code with the new version from https://github.com/JuliaLang/utf8proc. 3. Add a couple more test cases: nc_test/tst_utf8_validate.c and nc_test_utf8_phrases.c. If/when I can find a usable normalization test, I will incorporate that later.
2017-02-17 05:27:54 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
Copyright (C) 1984, 1989-1990, 2000-2015, 2018-2021 Free Software Foundation,
Inc.
Addressed an API-related usage of strncmp.
2015-11-25 07:19:36 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
This program is free software: you can redistribute it and/or modify
moving to trunk subdir
2010-06-03 21:24:43 +08:00
it under the terms of the GNU General Public License as published by
More tweaking.
2020-12-08 05:45:14 +08:00
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
Resolves Github issue https://github.com/Unidata/netcdf-c/issues/349. Update utf8proc.[ch] to use the version now maintained by the Julia Language project (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). The license for the previous version was unacceptable for the Debian and Ubuntu release systems. The new version both updates the code and addresses the license issue. It turns out that the utf8proc software we are using was turned over to the Julia Language developers and the license terms changed to allow modification. (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). So the fix here is as follows: 1. Wrap the library with a fixed interface: libdispatch/dutf8.c and include/ncutf8.h. 2. Replace the existing utf8proc code with the new version from https://github.com/JuliaLang/utf8proc. 3. Add a couple more test cases: nc_test/tst_utf8_validate.c and nc_test_utf8_phrases.c. If/when I can find a usable normalization test, I will incorporate that later.
2017-02-17 05:27:54 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
Resolves Github issue https://github.com/Unidata/netcdf-c/issues/349. Update utf8proc.[ch] to use the version now maintained by the Julia Language project (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). The license for the previous version was unacceptable for the Debian and Ubuntu release systems. The new version both updates the code and addresses the license issue. It turns out that the utf8proc software we are using was turned over to the Julia Language developers and the license terms changed to allow modification. (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). So the fix here is as follows: 1. Wrap the library with a fixed interface: libdispatch/dutf8.c and include/ncutf8.h. 2. Replace the existing utf8proc code with the new version from https://github.com/JuliaLang/utf8proc. 3. Add a couple more test cases: nc_test/tst_utf8_validate.c and nc_test_utf8_phrases.c. If/when I can find a usable normalization test, I will incorporate that later.
2017-02-17 05:27:54 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
You should have received a copy of the GNU General Public License
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
along with this program. If not, see <http://www.gnu.org/licenses/>. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* As a special exception, you may create a larger work that contains
part or all of the Bison parser skeleton and distribute that work
under terms of your choice, so long as that work isn't itself a
parser generator using the skeleton or a modified version thereof
as a parser skeleton. Alternatively, if you modify or redistribute
the parser skeleton itself, you may (at your option) remove this
special exception, which will cause the skeleton and the resulting
Bison output files to be licensed under the GNU General Public
License without this special exception.
Resolves Github issue https://github.com/Unidata/netcdf-c/issues/349. Update utf8proc.[ch] to use the version now maintained by the Julia Language project (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). The license for the previous version was unacceptable for the Debian and Ubuntu release systems. The new version both updates the code and addresses the license issue. It turns out that the utf8proc software we are using was turned over to the Julia Language developers and the license terms changed to allow modification. (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). So the fix here is as follows: 1. Wrap the library with a fixed interface: libdispatch/dutf8.c and include/ncutf8.h. 2. Replace the existing utf8proc code with the new version from https://github.com/JuliaLang/utf8proc. 3. Add a couple more test cases: nc_test/tst_utf8_validate.c and nc_test_utf8_phrases.c. If/when I can find a usable normalization test, I will incorporate that later.
2017-02-17 05:27:54 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
This special exception was added by the Free Software Foundation in
version 2.2 of Bison. */
/* C LALR(1) parser skeleton written by Richard Stallman, by
simplifying the original so-called "semantic" parser. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* DO NOT RELY ON FEATURES THAT ARE NOT DOCUMENTED in the manual,
especially those whose name start with YY_ or yy_. They are
private implementation details that can be changed or removed. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* All symbols defined below should begin with yy or YY, to avoid
infringing on user name space. This should be done even for local
variables, as they might otherwise be expanded by user macros.
There are some unavoidable exceptions within include files to
define necessary library symbols; they are noted "INFRINGES ON
USER NAME SPACE" below. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Identify Bison output, and Bison version. */
#define YYBISON 30705
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Bison version string. */
#define YYBISON_VERSION "3.7.5"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Skeleton name. */
#define YYSKELETON_NAME "yacc.c"
/* Pure parsers. */
#define YYPURE 0
More tweaking.
2020-12-08 05:45:14 +08:00
/* Push parsers. */
#define YYPUSH 0
Working out an error with unistd.h in windows.
2017-01-31 05:54:00 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
/* Pull parsers. */
#define YYPULL 1
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
/* Substitute the variable and function names. */
#define yyparse ncgparse
#define yylex ncglex
#define yyerror ncgerror
#define yydebug ncgdebug
#define yynerrs ncgnerrs
#define yylval ncglval
#define yychar ncgchar
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* First part of user prologue. */
#line 11 "ncgen.y"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/*
Changed calls in tests from deprecated _ubyte functions to equivalent _uchar functions.
2010-06-23 06:41:04 +08:00
static char SccsId[] = "$Id: ncgen.y,v 1.42 2010/05/18 21:32:46 dmh Exp $";
moving to trunk subdir
2010-06-03 21:24:43 +08:00
*/
#include "includes.h"
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
#include "netcdf_aux.h"
Things are working?
2016-05-12 05:31:17 +08:00
#include "ncgeny.h"
#include "ncgen.h"
changed macro STREQ to NCSTREQ to avoid name collusion with HDF4 library
2018-05-12 22:55:51 +08:00
#ifdef USE_NETCDF4
Fix errors when building on big-endian machine re: issue https://github.com/Unidata/netcdf-c/issues/1278 re: issue https://github.com/Unidata/netcdf-c/issues/876 re: issue https://github.com/Unidata/netcdf-c/issues/806 * Major change to the handling of 8-byte parameters for nc_def_var_filter. The old code was not well thought out. * The new algorithm is documented in docs/filters.md. * Added new utility file plugins/H5Zutil.c to support * Modified plugins/H5Zmisc.c to use new algorithm the new algorithm. * Renamed include/ncfilter.h to include/netcdf_filter.h and made it an installed header so clients can access the new algorithm utility. * Fixed nc_test4/tst_filterparser.c and nc_test4/test_filter_misc.c to use the new algorithm * libdap4/ fixes: * d4swap.c has an error in the endian pre-processing such that record counts were not being swapped correctly. * d4data.c had an error in that checksums were being computed after endian swapping rather than before. * ocinitialize() was never being called, so xxdr bigendian handling was never set correctly. * Required adding debug statements to occompile * Found and fixed memory leak in ncdump.c Not tested: * HDF4 * Pnetcdf * parallel HDF5
2019-02-01 12:13:06 +08:00
#include "netcdf_filter.h"
changed macro STREQ to NCSTREQ to avoid name collusion with HDF4 library
2018-05-12 22:55:51 +08:00
#endif
fix NCF-120
2011-09-16 00:57:16 +08:00
2011-09-17 02:36:08 +08:00
/* Following are in ncdump (for now)*/
/* Need some (unused) definitions to get it to compile */
#define ncatt_t void*
#define ncvar_t void
1. synch changes with oc2.0 2. Fix a bug in ncgen to catch a special case of a malformed datalist.
2013-06-27 02:55:30 +08:00
#include "nctime.h"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Fix the handling of certain alias types on CDL files. re: https://github.com/Unidata/netcdf-c/issues/1977 PR https://github.com/Unidata/netcdf-c/pull/1753, changed ncgen to allows certain type names to be used as identifiers in selected situations. An unwanted side effect was that existing type aliases no longer were accepted by ncgen. Specifically, using the "long" type caused an error. I was able to figure out a better solution to the original problem (https://github.com/Unidata/netcdf-c/issues/1750) that also fixes this problem as well. This PR fixes that problem in ncgen/ncgen.l, and adds tests to ncdump/test_keywords.sh
2021-04-14 06:56:43 +08:00
#undef GENLIB1
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* parser controls */
#define YY_NO_INPUT 1
/* True if string a equals string b*/
changed macro STREQ to NCSTREQ to avoid name collusion with HDF4 library
2018-05-12 22:55:51 +08:00
#ifndef NCSTREQ
#define NCSTREQ(a, b) (*(a) == *(b) && strcmp((a), (b)) == 0)
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
#endif
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#define VLENSIZE (sizeof(nc_vlen_t))
#define MAXFLOATDIM 4294967295.0
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
/* mnemonics */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
typedef enum Attrkind {ATTRVAR, ATTRGLOBAL, DONTKNOW} Attrkind;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
#define ISCONST 1
#define ISLIST 0
moving to trunk subdir
2010-06-03 21:24:43 +08:00
typedef nc_vlen_t vlen_t;
/* We retain the old representation of the symbol list
as a linked list.
*/
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
List* symlist;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Track rootgroup separately*/
Symbol* rootgroup;
/* Track the group sequence */
static List* groupstack;
/* Provide a separate sequence for accumulating values
during the parse.
*/
static List* stack;
/* track homogeneity of types for data lists*/
static nc_type consttype;
/* Misc. */
static int stackbase;
static int stacklen;
static int count;
static int opaqueid; /* counter for opaque constants*/
static int arrayuid; /* counter for pseudo-array types*/
char* primtypenames[PRIMNO] = {
"nat",
"byte", "char", "short",
"int", "float", "double",
"ubyte", "ushort", "uint",
"int64", "uint64",
"string"
};
Finalize the compression support. This relies on the HDF5 capability to dynamically load compression filters. Note that a compression filter is just a subcase of filters. The primary user-visible changes are as follows: 1. Add a standard header "netcdf_filter.h" that defines the necessary API extensions 2. Modify ncgen to support two new special attributes "_Filter_ID" and "_Filter_Parameters" so that compression can be turned on when creating a file using ncgen. 4. Add a detailed description of filtering support to the user's guide; see the file filters.md 5. Add a test case directory for this: nc_test4/filter_test. It is fragile and a ./configure flags (-enable-filter-test) is defined (default disabled) to shut this off this test to avoid spurious 'make check' failures. Note that the HDF5 documentation is not up-to-date, so much of what is encoded here comes from examining the actual code in the file H5PL.c in the HDF5 source code.
2017-04-28 03:01:59 +08:00
static int GLOBAL_SPECIAL = _NCPROPS_FLAG
| _ISNETCDF4_FLAG
| _SUPERBLOCK_FLAG
| _FORMAT_FLAG ;
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/*Defined in ncgen.l*/
extern int lineno; /* line number for error messages */
Resolve Lynton's vlen problems (partially) re: jira NCF-145
2012-02-14 08:25:32 +08:00
extern Bytebuffer* lextext; /* name or string with escapes removed */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
extern double double_val; /* last double value read */
extern float float_val; /* last float value read */
extern long long int64_val; /* last int64 value read */
extern int int32_val; /* last int32 value read */
extern short int16_val; /* last short value read */
extern unsigned long long uint64_val; /* last int64 value read */
extern unsigned int uint32_val; /* last int32 value read */
extern unsigned short uint16_val; /* last short value read */
extern char char_val; /* last char value read */
extern signed char byte_val; /* last byte value read */
extern unsigned char ubyte_val; /* last byte value read */
/* Track definitions of dims, types, attributes, and vars*/
List* grpdefs;
List* dimdefs;
List* attdefs; /* variable-specific attributes*/
List* gattdefs; /* global attributes only*/
List* xattdefs; /* unknown attributes*/
List* typdefs;
List* vardefs;
List* tmp;
/* Forward */
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
static NCConstant* makeconstdata(nc_type);
static NCConstant* evaluate(Symbol* fcn, Datalist* arglist);
static NCConstant* makeenumconstref(Symbol*);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
static void addtogroup(Symbol*);
static Symbol* currentgroup(void);
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
static Symbol* createrootgroup(const char*);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
static Symbol* creategroup(Symbol*);
static int dupobjectcheck(nc_class,Symbol*);
static void setpathcurrent(Symbol* sym);
static Symbol* makeattribute(Symbol*,Symbol*,Symbol*,Datalist*,Attrkind);
static Symbol* makeprimitivetype(nc_type i);
static Symbol* makespecial(int tag, Symbol* vsym, Symbol* tsym, void* data, int isconst);
static int containsfills(Datalist* list);
Fixed the following Jira minor bugs: NCF-42: _Format attribute sometimes being ignored NCF-43: Fixed unsigned long long parsing. NCF-47: Make opendap code properly handle illegal names like "x.y" by supressing them NCF-49: check for uint type NCF-50: properly handle username:pwd embedded in urls.
2011-05-13 01:51:32 +08:00
static void vercheck(int ncid);
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
static long long extractint(NCConstant* con);
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
#ifdef USE_NETCDF4
Add new capabilities to filter code: 1. Allow nccopy to apply filters, especially on the output file. This provides a third way to do this other than using ncgen or programatically 2. Make sure that even if the filter code is not available, it is possible to see the filter id and parameters for variables using e.g ncdump -hs. 3. Fix bug in nccopy so that the input file does not necessarily have to be netcdf-4. 4. At last minute decided to change to using a single "_Filter" attribute for ncgen 5. Added a test to tst_filter.sh to generate C code using ncgen.
2017-05-15 08:10:02 +08:00
static int parsefilterflag(const char* sdata0, Specialdata* special);
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
static int parsecodecsflag(const char* sdata0, Specialdata* special);
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
static Symbol* identkeyword(const Symbol*);
Add support for multiple filters per variable. re: https://github.com/Unidata/netcdf-c/issues/1584 Support has been added for multiple filters per variable. This affects a number of components in netcdf. The new APIs are documented in NUG/filters.md. The primary changes are: * A set of new functions are provided (see __include/netcdf_filter.h__). - Obtain a list of the filters associated with a variable - Obtain the parameters for a specific filter. * The existing __nc_inq_var_filter__ function now returns info about the first defined filter. * The utilities (ncgen, ncdump, and nccopy) now support an extended format for specifying a sequence of filters. The general form is __<filter>|<filter>..._. * The ncdump **_Filter** attribute now dumps a list of all the filters associated with a variable using the above new format. * Filter specifications can now use a filter name instead of number for filters known to the netcdf library, which in turn is taken from the HDF5 filter registration page. * New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter is returned if an attempt is made to access an unknown filter. * Internally, the dispatch table has been extended to add a function to handle all of the filter functions. * New, filter-related, tests were added to nc_test4. * A new plugin was added to the plugins directory to help with testing. Notes: 1. The shuffle and fletcher32 filters are not part of the multifilter system. Misc. changes: 1. A debug module was added to libhdf5 to help catch error locations.
2020-02-17 03:59:33 +08:00
#ifdef GENDEBUG1
Mostly revert the filter code to reduce its complexity of use. re: https://github.com/Unidata/netcdf-c/issues/1836 Revert the internal filter code to simplify it. From the user's point of view, the only visible changes should be: 1. The functions that convert text to filter specs have had their signature reverted and have been moved to netcdf_aux.h 2. Some filter API functions now return NC_ENOFILTER when inquiry is made about some filter. Internally,the dispatch table has been modified to get rid of the filter_actions entry and associated complex structures. It has been replaced with inq_var_filter_ids and inq_var_filter_info entries and the dispatch table version has been bumped to 3. Corresponding NOOP and NOTNC4 functions were added to libdispatch/dnotnc4.c. Also, the filter_action entries in dispatch tables were replaced for all dispatch code bases (HDF5, DAP2, etc). This should only impact UDF users. In the process, it became clear that the form of the filters field in NC_VAR_INFO_T was format dependent, so I converted it to be of type void* and pushed its management into the various dispatch code bases. Specifically libhdf5 and libnczarr now manage the filters field in their own way. The auxilliary functions for parsing textual filter specifications were moved to netcdf_aux.h and were renamed to the following: * ncaux_h5filterspec_parse * ncaux_h5filterspec_parselist * ncaux_h5filterspec_free * ncaux_h5filter_fix8 Misc. Other Changes: 1. Document NUG/filters.md updated to reflect the changes above. 2. All the old data types (structs and enums) used by filter_actions actions were deleted. The exception is the NC_H5_Filterspec because it is needed by ncaux_h5filterspec_parselist. 3. Clientside filters were removed -- another enhancement for which no-one ever asked. 4. The ability to remove filters was itself removed. 5. Some functionality needed by nczarr was moved from libhdf5 to libsrc4 e.g. nc4_find_default_chunksizes 6. All the filterx code was removed 7. ncfilter.h and nc4filter.c no longer used Misc. Unrelated Changes: 1. The nczarr_test makefile clean was leaving some directories; so add clean-local to take care of them.
2020-09-28 02:43:46 +08:00
static void printfilters(int nfilters, NC_ParsedFilterSpec** filters);
Add support for multiple filters per variable. re: https://github.com/Unidata/netcdf-c/issues/1584 Support has been added for multiple filters per variable. This affects a number of components in netcdf. The new APIs are documented in NUG/filters.md. The primary changes are: * A set of new functions are provided (see __include/netcdf_filter.h__). - Obtain a list of the filters associated with a variable - Obtain the parameters for a specific filter. * The existing __nc_inq_var_filter__ function now returns info about the first defined filter. * The utilities (ncgen, ncdump, and nccopy) now support an extended format for specifying a sequence of filters. The general form is __<filter>|<filter>..._. * The ncdump **_Filter** attribute now dumps a list of all the filters associated with a variable using the above new format. * Filter specifications can now use a filter name instead of number for filters known to the netcdf library, which in turn is taken from the HDF5 filter registration page. * New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter is returned if an attempt is made to access an unknown filter. * Internally, the dispatch table has been extended to add a function to handle all of the filter functions. * New, filter-related, tests were added to nc_test4. * A new plugin was added to the plugins directory to help with testing. Notes: 1. The shuffle and fletcher32 filters are not part of the multifilter system. Misc. changes: 1. A debug module was added to libhdf5 to help catch error locations.
2020-02-17 03:59:33 +08:00
#endif
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
#endif
moving to trunk subdir
2010-06-03 21:24:43 +08:00
int yylex(void);
#ifndef NO_STDARG
static void yyerror(const char *fmt, ...);
#else
static void yyerror(fmt,va_alist) const char* fmt; va_dcl;
#endif
/* Extern */
extern int lex_init(void);
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 221 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# ifndef YY_CAST
# ifdef __cplusplus
# define YY_CAST(Type, Val) static_cast<Type> (Val)
# define YY_REINTERPRET_CAST(Type, Val) reinterpret_cast<Type> (Val)
# else
# define YY_CAST(Type, Val) ((Type) (Val))
# define YY_REINTERPRET_CAST(Type, Val) ((Type) (Val))
# endif
# endif
More tweaking.
2020-12-08 05:45:14 +08:00
# ifndef YY_NULLPTR
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# if defined __cplusplus
# if 201103L <= __cplusplus
# define YY_NULLPTR nullptr
# else
# define YY_NULLPTR 0
# endif
More tweaking.
2020-12-08 05:45:14 +08:00
# else
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# define YY_NULLPTR ((void*)0)
More tweaking.
2020-12-08 05:45:14 +08:00
# endif
# endif
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#include "ncgeny.h"
/* Symbol kind. */
enum yysymbol_kind_t
Resolves Github issue https://github.com/Unidata/netcdf-c/issues/349. Update utf8proc.[ch] to use the version now maintained by the Julia Language project (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). The license for the previous version was unacceptable for the Debian and Ubuntu release systems. The new version both updates the code and addresses the license issue. It turns out that the utf8proc software we are using was turned over to the Julia Language developers and the license terms changed to allow modification. (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). So the fix here is as follows: 1. Wrap the library with a fixed interface: libdispatch/dutf8.c and include/ncutf8.h. 2. Replace the existing utf8proc code with the new version from https://github.com/JuliaLang/utf8proc. 3. Add a couple more test cases: nc_test/tst_utf8_validate.c and nc_test_utf8_phrases.c. If/when I can find a usable normalization test, I will incorporate that later.
2017-02-17 05:27:54 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YYSYMBOL_YYEMPTY = -2,
YYSYMBOL_YYEOF = 0, /* "end of file" */
YYSYMBOL_YYerror = 1, /* error */
YYSYMBOL_YYUNDEF = 2, /* "invalid token" */
YYSYMBOL_NC_UNLIMITED_K = 3, /* NC_UNLIMITED_K */
YYSYMBOL_CHAR_K = 4, /* CHAR_K */
YYSYMBOL_BYTE_K = 5, /* BYTE_K */
YYSYMBOL_SHORT_K = 6, /* SHORT_K */
YYSYMBOL_INT_K = 7, /* INT_K */
YYSYMBOL_FLOAT_K = 8, /* FLOAT_K */
YYSYMBOL_DOUBLE_K = 9, /* DOUBLE_K */
YYSYMBOL_UBYTE_K = 10, /* UBYTE_K */
YYSYMBOL_USHORT_K = 11, /* USHORT_K */
YYSYMBOL_UINT_K = 12, /* UINT_K */
YYSYMBOL_INT64_K = 13, /* INT64_K */
YYSYMBOL_UINT64_K = 14, /* UINT64_K */
YYSYMBOL_STRING_K = 15, /* STRING_K */
YYSYMBOL_IDENT = 16, /* IDENT */
YYSYMBOL_TERMSTRING = 17, /* TERMSTRING */
YYSYMBOL_CHAR_CONST = 18, /* CHAR_CONST */
YYSYMBOL_BYTE_CONST = 19, /* BYTE_CONST */
YYSYMBOL_SHORT_CONST = 20, /* SHORT_CONST */
YYSYMBOL_INT_CONST = 21, /* INT_CONST */
YYSYMBOL_INT64_CONST = 22, /* INT64_CONST */
YYSYMBOL_UBYTE_CONST = 23, /* UBYTE_CONST */
YYSYMBOL_USHORT_CONST = 24, /* USHORT_CONST */
YYSYMBOL_UINT_CONST = 25, /* UINT_CONST */
YYSYMBOL_UINT64_CONST = 26, /* UINT64_CONST */
YYSYMBOL_FLOAT_CONST = 27, /* FLOAT_CONST */
YYSYMBOL_DOUBLE_CONST = 28, /* DOUBLE_CONST */
YYSYMBOL_DIMENSIONS = 29, /* DIMENSIONS */
YYSYMBOL_VARIABLES = 30, /* VARIABLES */
YYSYMBOL_NETCDF = 31, /* NETCDF */
YYSYMBOL_DATA = 32, /* DATA */
YYSYMBOL_TYPES = 33, /* TYPES */
YYSYMBOL_COMPOUND = 34, /* COMPOUND */
YYSYMBOL_ENUM = 35, /* ENUM */
YYSYMBOL_OPAQUE_ = 36, /* OPAQUE_ */
YYSYMBOL_OPAQUESTRING = 37, /* OPAQUESTRING */
YYSYMBOL_GROUP = 38, /* GROUP */
YYSYMBOL_PATH = 39, /* PATH */
YYSYMBOL_FILLMARKER = 40, /* FILLMARKER */
YYSYMBOL_NIL = 41, /* NIL */
YYSYMBOL__FILLVALUE = 42, /* _FILLVALUE */
YYSYMBOL__FORMAT = 43, /* _FORMAT */
YYSYMBOL__STORAGE = 44, /* _STORAGE */
YYSYMBOL__CHUNKSIZES = 45, /* _CHUNKSIZES */
YYSYMBOL__DEFLATELEVEL = 46, /* _DEFLATELEVEL */
YYSYMBOL__SHUFFLE = 47, /* _SHUFFLE */
YYSYMBOL__ENDIANNESS = 48, /* _ENDIANNESS */
YYSYMBOL__NOFILL = 49, /* _NOFILL */
YYSYMBOL__FLETCHER32 = 50, /* _FLETCHER32 */
YYSYMBOL__NCPROPS = 51, /* _NCPROPS */
YYSYMBOL__ISNETCDF4 = 52, /* _ISNETCDF4 */
YYSYMBOL__SUPERBLOCK = 53, /* _SUPERBLOCK */
YYSYMBOL__FILTER = 54, /* _FILTER */
YYSYMBOL__CODECS = 55, /* _CODECS */
YYSYMBOL__QUANTIZEBG = 56, /* _QUANTIZEBG */
YYSYMBOL__QUANTIZEBR = 57, /* _QUANTIZEBR */
YYSYMBOL_DATASETID = 58, /* DATASETID */
YYSYMBOL_59_ = 59, /* '{' */
YYSYMBOL_60_ = 60, /* '}' */
YYSYMBOL_61_ = 61, /* ';' */
YYSYMBOL_62_ = 62, /* ',' */
YYSYMBOL_63_ = 63, /* '=' */
YYSYMBOL_64_ = 64, /* '(' */
YYSYMBOL_65_ = 65, /* ')' */
YYSYMBOL_66_ = 66, /* '*' */
YYSYMBOL_67_ = 67, /* ':' */
YYSYMBOL_YYACCEPT = 68, /* $accept */
YYSYMBOL_ncdesc = 69, /* ncdesc */
YYSYMBOL_datasetid = 70, /* datasetid */
YYSYMBOL_rootgroup = 71, /* rootgroup */
YYSYMBOL_groupbody = 72, /* groupbody */
YYSYMBOL_subgrouplist = 73, /* subgrouplist */
YYSYMBOL_namedgroup = 74, /* namedgroup */
YYSYMBOL_75_1 = 75, /* $@1 */
YYSYMBOL_76_2 = 76, /* $@2 */
YYSYMBOL_typesection = 77, /* typesection */
YYSYMBOL_typedecls = 78, /* typedecls */
YYSYMBOL_typename = 79, /* typename */
YYSYMBOL_type_or_attr_decl = 80, /* type_or_attr_decl */
YYSYMBOL_typedecl = 81, /* typedecl */
YYSYMBOL_optsemicolon = 82, /* optsemicolon */
YYSYMBOL_enumdecl = 83, /* enumdecl */
YYSYMBOL_enumidlist = 84, /* enumidlist */
YYSYMBOL_enumid = 85, /* enumid */
YYSYMBOL_opaquedecl = 86, /* opaquedecl */
YYSYMBOL_vlendecl = 87, /* vlendecl */
YYSYMBOL_compounddecl = 88, /* compounddecl */
YYSYMBOL_fields = 89, /* fields */
YYSYMBOL_field = 90, /* field */
YYSYMBOL_primtype = 91, /* primtype */
YYSYMBOL_dimsection = 92, /* dimsection */
YYSYMBOL_dimdecls = 93, /* dimdecls */
YYSYMBOL_dim_or_attr_decl = 94, /* dim_or_attr_decl */
YYSYMBOL_dimdeclist = 95, /* dimdeclist */
YYSYMBOL_dimdecl = 96, /* dimdecl */
YYSYMBOL_dimd = 97, /* dimd */
YYSYMBOL_vasection = 98, /* vasection */
YYSYMBOL_vadecls = 99, /* vadecls */
YYSYMBOL_vadecl_or_attr = 100, /* vadecl_or_attr */
YYSYMBOL_vardecl = 101, /* vardecl */
YYSYMBOL_varlist = 102, /* varlist */
YYSYMBOL_varspec = 103, /* varspec */
YYSYMBOL_dimspec = 104, /* dimspec */
YYSYMBOL_dimlist = 105, /* dimlist */
YYSYMBOL_dimref = 106, /* dimref */
YYSYMBOL_fieldlist = 107, /* fieldlist */
YYSYMBOL_fieldspec = 108, /* fieldspec */
YYSYMBOL_fielddimspec = 109, /* fielddimspec */
YYSYMBOL_fielddimlist = 110, /* fielddimlist */
YYSYMBOL_fielddim = 111, /* fielddim */
YYSYMBOL_varref = 112, /* varref */
YYSYMBOL_typeref = 113, /* typeref */
YYSYMBOL_ambiguous_ref = 114, /* ambiguous_ref */
YYSYMBOL_attrdecllist = 115, /* attrdecllist */
YYSYMBOL_attrdecl = 116, /* attrdecl */
YYSYMBOL_path = 117, /* path */
YYSYMBOL_datasection = 118, /* datasection */
YYSYMBOL_datadecls = 119, /* datadecls */
YYSYMBOL_datadecl = 120, /* datadecl */
YYSYMBOL_datalist = 121, /* datalist */
YYSYMBOL_datalist0 = 122, /* datalist0 */
YYSYMBOL_datalist1 = 123, /* datalist1 */
YYSYMBOL_dataitem = 124, /* dataitem */
YYSYMBOL_constdata = 125, /* constdata */
YYSYMBOL_econstref = 126, /* econstref */
YYSYMBOL_function = 127, /* function */
YYSYMBOL_arglist = 128, /* arglist */
YYSYMBOL_simpleconstant = 129, /* simpleconstant */
YYSYMBOL_intlist = 130, /* intlist */
YYSYMBOL_constint = 131, /* constint */
YYSYMBOL_conststring = 132, /* conststring */
YYSYMBOL_constbool = 133, /* constbool */
YYSYMBOL_varident = 134, /* varident */
YYSYMBOL_ident = 135 /* ident */
More tweaking.
2020-12-08 05:45:14 +08:00
};
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
typedef enum yysymbol_kind_t yysymbol_kind_t;
Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute The XArray implementation that uses Zarr for storage provides a mechanism to simulate named dimensions. It does this by adding a per-variable attribute called _ARRAY_DIMENSIONS. This attribute contains a list of names to be matched against the shape values of the variable. In effect a named dimension is created with the name _ARRAY_DIMENSIONS(i) and length shape(i) for all i in range 0..rank(variable). Both read and write support is provided. This XArray support is only invoked if the mode value of "xarray" is defined. So for example, as in this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,xarray,s3 ```` Note that the "xarray" mode flag also implies mode flag "zarr", so the above is equivalent to this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,zarr,xarray,s3 ```` The primary change to implement this was to unify the handling of dimension references in libnczarr/zsync. A test for this and other pure-zarr features was added as nczarr_test/run_purezarr.sh Other changes: * Make sure distcheck leaves no files around. * Change the special attribute flag DIMSCALEFLAG to HIDDENATTRFLAG to support the xarray attribute. * Annotate the zmap implementations with feature flags such as WRITEONCE (for zip files).
2021-02-25 04:46:11 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
[NCF-271] Fix bug introduced by [NCF-267]. The bug was that octal constants that had the highest bit set (e.g. '\200') were not recognized as proper octal constants. Fix was to keep as integer until it was needed as an 8-bit byte.
2013-09-27 00:10:26 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#ifdef short
# undef short
#endif
Merge branch 'ncgenchunks.dmh' of https://github.com/DennisHeimbigner/netcdf-c into master
2020-09-10 00:24:33 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* On compilers that do not define __PTRDIFF_MAX__ etc., make sure
<limits.h> and (if available) <stdint.h> are included
so that the code can choose integer types of a good width. */
Mostly revert the filter code to reduce its complexity of use. re: https://github.com/Unidata/netcdf-c/issues/1836 Revert the internal filter code to simplify it. From the user's point of view, the only visible changes should be: 1. The functions that convert text to filter specs have had their signature reverted and have been moved to netcdf_aux.h 2. Some filter API functions now return NC_ENOFILTER when inquiry is made about some filter. Internally,the dispatch table has been modified to get rid of the filter_actions entry and associated complex structures. It has been replaced with inq_var_filter_ids and inq_var_filter_info entries and the dispatch table version has been bumped to 3. Corresponding NOOP and NOTNC4 functions were added to libdispatch/dnotnc4.c. Also, the filter_action entries in dispatch tables were replaced for all dispatch code bases (HDF5, DAP2, etc). This should only impact UDF users. In the process, it became clear that the form of the filters field in NC_VAR_INFO_T was format dependent, so I converted it to be of type void* and pushed its management into the various dispatch code bases. Specifically libhdf5 and libnczarr now manage the filters field in their own way. The auxilliary functions for parsing textual filter specifications were moved to netcdf_aux.h and were renamed to the following: * ncaux_h5filterspec_parse * ncaux_h5filterspec_parselist * ncaux_h5filterspec_free * ncaux_h5filter_fix8 Misc. Other Changes: 1. Document NUG/filters.md updated to reflect the changes above. 2. All the old data types (structs and enums) used by filter_actions actions were deleted. The exception is the NC_H5_Filterspec because it is needed by ncaux_h5filterspec_parselist. 3. Clientside filters were removed -- another enhancement for which no-one ever asked. 4. The ability to remove filters was itself removed. 5. Some functionality needed by nczarr was moved from libhdf5 to libsrc4 e.g. nc4_find_default_chunksizes 6. All the filterx code was removed 7. ncfilter.h and nc4filter.c no longer used Misc. Unrelated Changes: 1. The nczarr_test makefile clean was leaving some directories; so add clean-local to take care of them.
2020-09-28 02:43:46 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#ifndef __PTRDIFF_MAX__
# include <limits.h> /* INFRINGES ON USER NAME SPACE */
# if defined __STDC_VERSION__ && 199901 <= __STDC_VERSION__
# include <stdint.h> /* INFRINGES ON USER NAME SPACE */
# define YY_STDINT_H
# endif
#endif
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Narrow types that promote to a signed type and that can represent a
signed or unsigned integer of at least N bits. In tables they can
save space and decrease cache pressure. Promoting to a signed type
helps avoid bugs in integer arithmetic. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#ifdef __INT_LEAST8_MAX__
typedef __INT_LEAST8_TYPE__ yytype_int8;
#elif defined YY_STDINT_H
typedef int_least8_t yytype_int8;
#else
typedef signed char yytype_int8;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#ifdef __INT_LEAST16_MAX__
typedef __INT_LEAST16_TYPE__ yytype_int16;
#elif defined YY_STDINT_H
typedef int_least16_t yytype_int16;
More tweaking.
2020-12-08 05:45:14 +08:00
#else
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
typedef short yytype_int16;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Work around bug in HP-UX 11.23, which defines these macros
incorrectly for preprocessor constants. This workaround can likely
be removed in 2023, as HPE has promised support for HP-UX 11.23
(aka HP-UX 11i v2) only through the end of 2022; see Table 2 of
<https://h20195.www2.hpe.com/V2/getpdf.aspx/4AA4-7673ENW.pdf>. */
#ifdef __hpux
# undef UINT_LEAST8_MAX
# undef UINT_LEAST16_MAX
# define UINT_LEAST8_MAX 255
# define UINT_LEAST16_MAX 65535
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#if defined __UINT_LEAST8_MAX__ && __UINT_LEAST8_MAX__ <= __INT_MAX__
typedef __UINT_LEAST8_TYPE__ yytype_uint8;
#elif (!defined __UINT_LEAST8_MAX__ && defined YY_STDINT_H \
&& UINT_LEAST8_MAX <= INT_MAX)
typedef uint_least8_t yytype_uint8;
#elif !defined __UINT_LEAST8_MAX__ && UCHAR_MAX <= INT_MAX
typedef unsigned char yytype_uint8;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#else
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
typedef short yytype_uint8;
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
#endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#if defined __UINT_LEAST16_MAX__ && __UINT_LEAST16_MAX__ <= __INT_MAX__
typedef __UINT_LEAST16_TYPE__ yytype_uint16;
#elif (!defined __UINT_LEAST16_MAX__ && defined YY_STDINT_H \
&& UINT_LEAST16_MAX <= INT_MAX)
typedef uint_least16_t yytype_uint16;
#elif !defined __UINT_LEAST16_MAX__ && USHRT_MAX <= INT_MAX
typedef unsigned short yytype_uint16;
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
#else
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
typedef int yytype_uint16;
#endif
#ifndef YYPTRDIFF_T
# if defined __PTRDIFF_TYPE__ && defined __PTRDIFF_MAX__
# define YYPTRDIFF_T __PTRDIFF_TYPE__
# define YYPTRDIFF_MAXIMUM __PTRDIFF_MAX__
# elif defined PTRDIFF_MAX
# ifndef ptrdiff_t
# include <stddef.h> /* INFRINGES ON USER NAME SPACE */
# endif
# define YYPTRDIFF_T ptrdiff_t
# define YYPTRDIFF_MAXIMUM PTRDIFF_MAX
# else
# define YYPTRDIFF_T long
# define YYPTRDIFF_MAXIMUM LONG_MAX
# endif
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#endif
#ifndef YYSIZE_T
# ifdef __SIZE_TYPE__
# define YYSIZE_T __SIZE_TYPE__
# elif defined size_t
# define YYSIZE_T size_t
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# elif defined __STDC_VERSION__ && 199901 <= __STDC_VERSION__
moving to trunk subdir
2010-06-03 21:24:43 +08:00
# include <stddef.h> /* INFRINGES ON USER NAME SPACE */
# define YYSIZE_T size_t
# else
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# define YYSIZE_T unsigned
moving to trunk subdir
2010-06-03 21:24:43 +08:00
# endif
#endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#define YYSIZE_MAXIMUM \
YY_CAST (YYPTRDIFF_T, \
(YYPTRDIFF_MAXIMUM < YY_CAST (YYSIZE_T, -1) \
? YYPTRDIFF_MAXIMUM \
: YY_CAST (YYSIZE_T, -1)))
#define YYSIZEOF(X) YY_CAST (YYPTRDIFF_T, sizeof (X))
/* Stored state numbers (used for stacks). */
typedef yytype_int16 yy_state_t;
/* State numbers in computations. */
typedef int yy_state_fast_t;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#ifndef YY_
disallow uppercase section keywords
2010-08-26 03:01:07 +08:00
# if defined YYENABLE_NLS && YYENABLE_NLS
moving to trunk subdir
2010-06-03 21:24:43 +08:00
# if ENABLE_NLS
# include <libintl.h> /* INFRINGES ON USER NAME SPACE */
More tweaking.
2020-12-08 05:45:14 +08:00
# define YY_(Msgid) dgettext ("bison-runtime", Msgid)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
# endif
# endif
# ifndef YY_
More tweaking.
2020-12-08 05:45:14 +08:00
# define YY_(Msgid) Msgid
# endif
#endif
Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute The XArray implementation that uses Zarr for storage provides a mechanism to simulate named dimensions. It does this by adding a per-variable attribute called _ARRAY_DIMENSIONS. This attribute contains a list of names to be matched against the shape values of the variable. In effect a named dimension is created with the name _ARRAY_DIMENSIONS(i) and length shape(i) for all i in range 0..rank(variable). Both read and write support is provided. This XArray support is only invoked if the mode value of "xarray" is defined. So for example, as in this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,xarray,s3 ```` Note that the "xarray" mode flag also implies mode flag "zarr", so the above is equivalent to this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,zarr,xarray,s3 ```` The primary change to implement this was to unify the handling of dimension references in libnczarr/zsync. A test for this and other pure-zarr features was added as nczarr_test/run_purezarr.sh Other changes: * Make sure distcheck leaves no files around. * Change the special attribute flag DIMSCALEFLAG to HIDDENATTRFLAG to support the xarray attribute. * Annotate the zmap implementations with feature flags such as WRITEONCE (for zip files).
2021-02-25 04:46:11 +08:00
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
#ifndef YY_ATTRIBUTE_PURE
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# if defined __GNUC__ && 2 < __GNUC__ + (96 <= __GNUC_MINOR__)
# define YY_ATTRIBUTE_PURE __attribute__ ((__pure__))
# else
# define YY_ATTRIBUTE_PURE
# endif
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
#endif
More tweaking.
2020-12-08 05:45:14 +08:00
#ifndef YY_ATTRIBUTE_UNUSED
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# if defined __GNUC__ && 2 < __GNUC__ + (7 <= __GNUC_MINOR__)
# define YY_ATTRIBUTE_UNUSED __attribute__ ((__unused__))
More tweaking.
2020-12-08 05:45:14 +08:00
# else
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# define YY_ATTRIBUTE_UNUSED
[NCF-271] Fix bug introduced by [NCF-267]. The bug was that octal constants that had the highest bit set (e.g. '\200') were not recognized as proper octal constants. Fix was to keep as integer until it was needed as an 8-bit byte.
2013-09-27 00:10:26 +08:00
# endif
#endif
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Suppress unused-variable warnings by "using" E. */
#if ! defined lint || defined __GNUC__
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# define YY_USE(E) ((void) (E))
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#else
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# define YY_USE(E) /* empty */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#if defined __GNUC__ && ! defined __ICC && 407 <= __GNUC__ * 100 + __GNUC_MINOR__
More tweaking.
2020-12-08 05:45:14 +08:00
/* Suppress an incorrect diagnostic about yylval being uninitialized. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# define YY_IGNORE_MAYBE_UNINITIALIZED_BEGIN \
_Pragma ("GCC diagnostic push") \
_Pragma ("GCC diagnostic ignored \"-Wuninitialized\"") \
More tweaking.
2020-12-08 05:45:14 +08:00
_Pragma ("GCC diagnostic ignored \"-Wmaybe-uninitialized\"")
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# define YY_IGNORE_MAYBE_UNINITIALIZED_END \
More tweaking.
2020-12-08 05:45:14 +08:00
_Pragma ("GCC diagnostic pop")
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
#else
More tweaking.
2020-12-08 05:45:14 +08:00
# define YY_INITIAL_VALUE(Value) Value
1. Keep up to date by merging master 2. Fixed plugin building (nc_test4/hdf5plugins) to be done properly by cmake and automake. 4. Duplicated part of the nc_test4 filter test code in examples/C An incomplete and untested set of hooks exist for OS-X in nc_test4/findplugins.in. They need testing.
2018-01-17 02:00:09 +08:00
#endif
More tweaking.
2020-12-08 05:45:14 +08:00
#ifndef YY_IGNORE_MAYBE_UNINITIALIZED_BEGIN
# define YY_IGNORE_MAYBE_UNINITIALIZED_BEGIN
# define YY_IGNORE_MAYBE_UNINITIALIZED_END
#endif
#ifndef YY_INITIAL_VALUE
# define YY_INITIAL_VALUE(Value) /* Nothing. */
#endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#if defined __cplusplus && defined __GNUC__ && ! defined __ICC && 6 <= __GNUC__
# define YY_IGNORE_USELESS_CAST_BEGIN \
_Pragma ("GCC diagnostic push") \
_Pragma ("GCC diagnostic ignored \"-Wuseless-cast\"")
# define YY_IGNORE_USELESS_CAST_END \
_Pragma ("GCC diagnostic pop")
#endif
#ifndef YY_IGNORE_USELESS_CAST_BEGIN
# define YY_IGNORE_USELESS_CAST_BEGIN
# define YY_IGNORE_USELESS_CAST_END
#endif
#define YY_ASSERT(E) ((void) (0 && (E)))
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#if 1
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* The parser invokes alloca or malloc; define the necessary symbols. */
# ifdef YYSTACK_USE_ALLOCA
# if YYSTACK_USE_ALLOCA
# ifdef __GNUC__
# define YYSTACK_ALLOC __builtin_alloca
# elif defined __BUILTIN_VA_ARG_INCR
# include <alloca.h> /* INFRINGES ON USER NAME SPACE */
# elif defined _AIX
# define YYSTACK_ALLOC __alloca
# elif defined _MSC_VER
# include <malloc.h> /* INFRINGES ON USER NAME SPACE */
# define alloca _alloca
# else
# define YYSTACK_ALLOC alloca
More tweaking.
2020-12-08 05:45:14 +08:00
# if ! defined _ALLOCA_H && ! defined EXIT_SUCCESS
moving to trunk subdir
2010-06-03 21:24:43 +08:00
# include <stdlib.h> /* INFRINGES ON USER NAME SPACE */
More tweaking.
2020-12-08 05:45:14 +08:00
/* Use EXIT_SUCCESS as a witness for stdlib.h. */
# ifndef EXIT_SUCCESS
# define EXIT_SUCCESS 0
moving to trunk subdir
2010-06-03 21:24:43 +08:00
# endif
# endif
# endif
# endif
# endif
# ifdef YYSTACK_ALLOC
More tweaking.
2020-12-08 05:45:14 +08:00
/* Pacify GCC's 'empty if-body' warning. */
# define YYSTACK_FREE(Ptr) do { /* empty */; } while (0)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
# ifndef YYSTACK_ALLOC_MAXIMUM
/* The OS might guarantee only one guard page at the bottom of the stack,
and a page size can be as small as 4096 bytes. So we cannot safely
invoke alloca (N) if N exceeds 4096. Use a slightly smaller number
to allow for a few compiler-allocated temporary stack slots. */
# define YYSTACK_ALLOC_MAXIMUM 4032 /* reasonable circa 2006 */
# endif
# else
# define YYSTACK_ALLOC YYMALLOC
# define YYSTACK_FREE YYFREE
# ifndef YYSTACK_ALLOC_MAXIMUM
# define YYSTACK_ALLOC_MAXIMUM YYSIZE_MAXIMUM
# endif
More tweaking.
2020-12-08 05:45:14 +08:00
# if (defined __cplusplus && ! defined EXIT_SUCCESS \
moving to trunk subdir
2010-06-03 21:24:43 +08:00
&& ! ((defined YYMALLOC || defined malloc) \
More tweaking.
2020-12-08 05:45:14 +08:00
&& (defined YYFREE || defined free)))
moving to trunk subdir
2010-06-03 21:24:43 +08:00
# include <stdlib.h> /* INFRINGES ON USER NAME SPACE */
More tweaking.
2020-12-08 05:45:14 +08:00
# ifndef EXIT_SUCCESS
# define EXIT_SUCCESS 0
moving to trunk subdir
2010-06-03 21:24:43 +08:00
# endif
# endif
# ifndef YYMALLOC
# define YYMALLOC malloc
More tweaking.
2020-12-08 05:45:14 +08:00
# if ! defined malloc && ! defined EXIT_SUCCESS
moving to trunk subdir
2010-06-03 21:24:43 +08:00
void *malloc (YYSIZE_T); /* INFRINGES ON USER NAME SPACE */
# endif
# endif
# ifndef YYFREE
# define YYFREE free
More tweaking.
2020-12-08 05:45:14 +08:00
# if ! defined free && ! defined EXIT_SUCCESS
moving to trunk subdir
2010-06-03 21:24:43 +08:00
void free (void *); /* INFRINGES ON USER NAME SPACE */
# endif
# endif
# endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#endif /* 1 */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#if (! defined yyoverflow \
&& (! defined __cplusplus \
More tweaking.
2020-12-08 05:45:14 +08:00
|| (defined YYSTYPE_IS_TRIVIAL && YYSTYPE_IS_TRIVIAL)))
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* A type that is properly aligned for any stack member. */
union yyalloc
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yy_state_t yyss_alloc;
More tweaking.
2020-12-08 05:45:14 +08:00
YYSTYPE yyvs_alloc;
};
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* The size of the maximum gap between one aligned stack and the next. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# define YYSTACK_GAP_MAXIMUM (YYSIZEOF (union yyalloc) - 1)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* The size of an array large to enough to hold all stacks, each with
N elements. */
# define YYSTACK_BYTES(N) \
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
((N) * (YYSIZEOF (yy_state_t) + YYSIZEOF (YYSTYPE)) \
moving to trunk subdir
2010-06-03 21:24:43 +08:00
+ YYSTACK_GAP_MAXIMUM)
More tweaking.
2020-12-08 05:45:14 +08:00
# define YYCOPY_NEEDED 1
Addressed an API-related usage of strncmp.
2015-11-25 07:19:36 +08:00
/* Relocate STACK from its old location to the new one. The
local variables YYSIZE and YYSTACKSIZE give the old and new number of
elements in the stack, and YYPTR gives the new location of the
stack. Advance YYPTR to a properly aligned location for the next
stack. */
More tweaking.
2020-12-08 05:45:14 +08:00
# define YYSTACK_RELOCATE(Stack_alloc, Stack) \
do \
{ \
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YYPTRDIFF_T yynewbytes; \
More tweaking.
2020-12-08 05:45:14 +08:00
YYCOPY (&yyptr->Stack_alloc, Stack, yysize); \
Stack = &yyptr->Stack_alloc; \
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yynewbytes = yystacksize * YYSIZEOF (*Stack) + YYSTACK_GAP_MAXIMUM; \
yyptr += yynewbytes / YYSIZEOF (*yyptr); \
More tweaking.
2020-12-08 05:45:14 +08:00
} \
while (0)
Addressed an API-related usage of strncmp.
2015-11-25 07:19:36 +08:00
#endif
1) fix out of date ncgentab/ncgenyy files 2)fix Jira-ncf213 bug and add test case
2013-01-04 04:45:34 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
#if defined YYCOPY_NEEDED && YYCOPY_NEEDED
/* Copy COUNT objects from SRC to DST. The source and destination do
not overlap. */
# ifndef YYCOPY
# if defined __GNUC__ && 1 < __GNUC__
# define YYCOPY(Dst, Src, Count) \
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
__builtin_memcpy (Dst, Src, YY_CAST (YYSIZE_T, (Count)) * sizeof (*(Src)))
More tweaking.
2020-12-08 05:45:14 +08:00
# else
# define YYCOPY(Dst, Src, Count) \
do \
{ \
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YYPTRDIFF_T yyi; \
More tweaking.
2020-12-08 05:45:14 +08:00
for (yyi = 0; yyi < (Count); yyi++) \
(Dst)[yyi] = (Src)[yyi]; \
} \
while (0)
# endif
# endif
#endif /* !YYCOPY_NEEDED */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* YYFINAL -- State number of the termination state. */
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
#define YYFINAL 5
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* YYLAST -- Last index in YYTABLE. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#define YYLAST 446
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* YYNTOKENS -- Number of terminals. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#define YYNTOKENS 68
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* YYNNTS -- Number of nonterminals. */
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
#define YYNNTS 68
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* YYNRULES -- Number of rules. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#define YYNRULES 158
More tweaking.
2020-12-08 05:45:14 +08:00
/* YYNSTATES -- Number of states. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#define YYNSTATES 273
/* YYMAXUTOK -- Last valid token kind. */
#define YYMAXUTOK 313
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* YYTRANSLATE(TOKEN-NUM) -- Symbol number corresponding to TOKEN-NUM
as returned by yylex, with out-of-bounds checking. */
#define YYTRANSLATE(YYX) \
(0 <= (YYX) && (YYX) <= YYMAXUTOK \
? YY_CAST (yysymbol_kind_t, yytranslate[YYX]) \
: YYSYMBOL_YYUNDEF)
More tweaking.
2020-12-08 05:45:14 +08:00
/* YYTRANSLATE[TOKEN-NUM] -- Symbol number corresponding to TOKEN-NUM
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
as returned by yylex. */
static const yytype_int8 yytranslate[] =
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
0, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
64, 65, 66, 2, 62, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 67, 61,
2, 63, 2, 2, 2, 2, 2, 2, 2, 2,
moving to trunk subdir
2010-06-03 21:24:43 +08:00
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
2, 2, 2, 59, 2, 60, 2, 2, 2, 2,
moving to trunk subdir
2010-06-03 21:24:43 +08:00
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 2, 2, 2, 1, 2, 3, 4,
5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18, 19, 20, 21, 22, 23, 24,
25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
35, 36, 37, 38, 39, 40, 41, 42, 43, 44,
Finalize the compression support. This relies on the HDF5 capability to dynamically load compression filters. Note that a compression filter is just a subcase of filters. The primary user-visible changes are as follows: 1. Add a standard header "netcdf_filter.h" that defines the necessary API extensions 2. Modify ncgen to support two new special attributes "_Filter_ID" and "_Filter_Parameters" so that compression can be turned on when creating a file using ncgen. 4. Add a detailed description of filtering support to the user's guide; see the file filters.md 5. Add a test case directory for this: nc_test4/filter_test. It is fragile and a ./configure flags (-enable-filter-test) is defined (default disabled) to shut this off this test to avoid spurious 'make check' failures. Note that the HDF5 documentation is not up-to-date, so much of what is encoded here comes from examining the actual code in the file H5PL.c in the HDF5 source code.
2017-04-28 03:01:59 +08:00
45, 46, 47, 48, 49, 50, 51, 52, 53, 54,
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
55, 56, 57, 58
moving to trunk subdir
2010-06-03 21:24:43 +08:00
};
#if YYDEBUG
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
/* YYRLINE[YYN] -- Source line where rule number YYN was defined. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
static const yytype_int16 yyrline[] =
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
0, 242, 242, 248, 250, 257, 264, 264, 267, 276,
266, 281, 282, 283, 287, 287, 289, 299, 299, 302,
303, 304, 305, 308, 308, 311, 341, 343, 360, 369,
381, 395, 428, 429, 432, 446, 447, 448, 449, 450,
451, 452, 453, 454, 455, 456, 457, 460, 461, 462,
465, 466, 469, 469, 471, 472, 476, 484, 494, 506,
507, 508, 511, 512, 515, 515, 517, 539, 543, 547,
576, 577, 580, 581, 585, 599, 603, 608, 637, 638,
642, 643, 648, 658, 678, 689, 700, 719, 726, 726,
729, 731, 733, 735, 737, 746, 757, 759, 761, 763,
765, 767, 769, 771, 773, 775, 777, 779, 781, 783,
788, 795, 804, 805, 806, 809, 810, 813, 817, 818,
822, 826, 827, 832, 833, 837, 838, 839, 840, 841,
842, 846, 850, 854, 856, 861, 862, 863, 864, 865,
866, 867, 868, 869, 870, 871, 872, 876, 877, 881,
883, 885, 887, 892, 896, 897, 905, 906, 910
moving to trunk subdir
2010-06-03 21:24:43 +08:00
};
#endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/** Accessing symbol of state STATE. */
#define YY_ACCESSING_SYMBOL(State) YY_CAST (yysymbol_kind_t, yystos[State])
#if 1
/* The user-facing name of the symbol whose (internal) number is
YYSYMBOL. No bounds checking. */
static const char *yysymbol_name (yysymbol_kind_t yysymbol) YY_ATTRIBUTE_UNUSED;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* YYTNAME[SYMBOL-NUM] -- String name of the symbol SYMBOL-NUM.
First, the terminals, then, starting at YYNTOKENS, nonterminals. */
static const char *const yytname[] =
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
"\"end of file\"", "error", "\"invalid token\"", "NC_UNLIMITED_K",
"CHAR_K", "BYTE_K", "SHORT_K", "INT_K", "FLOAT_K", "DOUBLE_K", "UBYTE_K",
"USHORT_K", "UINT_K", "INT64_K", "UINT64_K", "STRING_K", "IDENT",
"TERMSTRING", "CHAR_CONST", "BYTE_CONST", "SHORT_CONST", "INT_CONST",
"INT64_CONST", "UBYTE_CONST", "USHORT_CONST", "UINT_CONST",
"UINT64_CONST", "FLOAT_CONST", "DOUBLE_CONST", "DIMENSIONS", "VARIABLES",
"NETCDF", "DATA", "TYPES", "COMPOUND", "ENUM", "OPAQUE_", "OPAQUESTRING",
"GROUP", "PATH", "FILLMARKER", "NIL", "_FILLVALUE", "_FORMAT",
"_STORAGE", "_CHUNKSIZES", "_DEFLATELEVEL", "_SHUFFLE", "_ENDIANNESS",
"_NOFILL", "_FLETCHER32", "_NCPROPS", "_ISNETCDF4", "_SUPERBLOCK",
"_FILTER", "_CODECS", "_QUANTIZEBG", "_QUANTIZEBR", "DATASETID", "'{'",
"'}'", "';'", "','", "'='", "'('", "')'", "'*'", "':'", "$accept",
"ncdesc", "datasetid", "rootgroup", "groupbody", "subgrouplist",
"namedgroup", "$@1", "$@2", "typesection", "typedecls", "typename",
"type_or_attr_decl", "typedecl", "optsemicolon", "enumdecl",
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
"enumidlist", "enumid", "opaquedecl", "vlendecl", "compounddecl",
"fields", "field", "primtype", "dimsection", "dimdecls",
"dim_or_attr_decl", "dimdeclist", "dimdecl", "dimd", "vasection",
"vadecls", "vadecl_or_attr", "vardecl", "varlist", "varspec", "dimspec",
"dimlist", "dimref", "fieldlist", "fieldspec", "fielddimspec",
"fielddimlist", "fielddim", "varref", "typeref", "ambiguous_ref",
"attrdecllist", "attrdecl", "path", "datasection", "datadecls",
"datadecl", "datalist", "datalist0", "datalist1", "dataitem",
"constdata", "econstref", "function", "arglist", "simpleconstant",
"intlist", "constint", "conststring", "constbool", "varident", "ident", YY_NULLPTR
moving to trunk subdir
2010-06-03 21:24:43 +08:00
};
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
static const char *
yysymbol_name (yysymbol_kind_t yysymbol)
{
return yytname[yysymbol];
}
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
#endif
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#ifdef YYPRINT
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
/* YYTOKNUM[NUM] -- (External) token number corresponding to the
(internal) symbol number NUM (which must be that of a token). */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
static const yytype_int16 yytoknum[] =
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
{
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
0, 256, 257, 258, 259, 260, 261, 262, 263, 264,
265, 266, 267, 268, 269, 270, 271, 272, 273, 274,
275, 276, 277, 278, 279, 280, 281, 282, 283, 284,
285, 286, 287, 288, 289, 290, 291, 292, 293, 294,
295, 296, 297, 298, 299, 300, 301, 302, 303, 304,
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
305, 306, 307, 308, 309, 310, 311, 312, 313, 123,
125, 59, 44, 61, 40, 41, 42, 58
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
};
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#endif
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#define YYPACT_NINF (-149)
1. Keep up to date by merging master 2. Fixed plugin building (nc_test4/hdf5plugins) to be done properly by cmake and automake. 4. Duplicated part of the nc_test4 filter test code in examples/C An incomplete and untested set of hooks exist for OS-X in nc_test4/findplugins.in. They need testing.
2018-01-17 02:00:09 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#define yypact_value_is_default(Yyn) \
((Yyn) == YYPACT_NINF)
Merge branch 'ncgenchunks.dmh' of https://github.com/DennisHeimbigner/netcdf-c into master
2020-09-10 00:24:33 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#define YYTABLE_NINF (-159)
Mostly revert the filter code to reduce its complexity of use. re: https://github.com/Unidata/netcdf-c/issues/1836 Revert the internal filter code to simplify it. From the user's point of view, the only visible changes should be: 1. The functions that convert text to filter specs have had their signature reverted and have been moved to netcdf_aux.h 2. Some filter API functions now return NC_ENOFILTER when inquiry is made about some filter. Internally,the dispatch table has been modified to get rid of the filter_actions entry and associated complex structures. It has been replaced with inq_var_filter_ids and inq_var_filter_info entries and the dispatch table version has been bumped to 3. Corresponding NOOP and NOTNC4 functions were added to libdispatch/dnotnc4.c. Also, the filter_action entries in dispatch tables were replaced for all dispatch code bases (HDF5, DAP2, etc). This should only impact UDF users. In the process, it became clear that the form of the filters field in NC_VAR_INFO_T was format dependent, so I converted it to be of type void* and pushed its management into the various dispatch code bases. Specifically libhdf5 and libnczarr now manage the filters field in their own way. The auxilliary functions for parsing textual filter specifications were moved to netcdf_aux.h and were renamed to the following: * ncaux_h5filterspec_parse * ncaux_h5filterspec_parselist * ncaux_h5filterspec_free * ncaux_h5filter_fix8 Misc. Other Changes: 1. Document NUG/filters.md updated to reflect the changes above. 2. All the old data types (structs and enums) used by filter_actions actions were deleted. The exception is the NC_H5_Filterspec because it is needed by ncaux_h5filterspec_parselist. 3. Clientside filters were removed -- another enhancement for which no-one ever asked. 4. The ability to remove filters was itself removed. 5. Some functionality needed by nczarr was moved from libhdf5 to libsrc4 e.g. nc4_find_default_chunksizes 6. All the filterx code was removed 7. ncfilter.h and nc4filter.c no longer used Misc. Unrelated Changes: 1. The nczarr_test makefile clean was leaving some directories; so add clean-local to take care of them.
2020-09-28 02:43:46 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#define yytable_value_is_error(Yyn) \
More tweaking.
2020-12-08 05:45:14 +08:00
0
Mostly revert the filter code to reduce its complexity of use. re: https://github.com/Unidata/netcdf-c/issues/1836 Revert the internal filter code to simplify it. From the user's point of view, the only visible changes should be: 1. The functions that convert text to filter specs have had their signature reverted and have been moved to netcdf_aux.h 2. Some filter API functions now return NC_ENOFILTER when inquiry is made about some filter. Internally,the dispatch table has been modified to get rid of the filter_actions entry and associated complex structures. It has been replaced with inq_var_filter_ids and inq_var_filter_info entries and the dispatch table version has been bumped to 3. Corresponding NOOP and NOTNC4 functions were added to libdispatch/dnotnc4.c. Also, the filter_action entries in dispatch tables were replaced for all dispatch code bases (HDF5, DAP2, etc). This should only impact UDF users. In the process, it became clear that the form of the filters field in NC_VAR_INFO_T was format dependent, so I converted it to be of type void* and pushed its management into the various dispatch code bases. Specifically libhdf5 and libnczarr now manage the filters field in their own way. The auxilliary functions for parsing textual filter specifications were moved to netcdf_aux.h and were renamed to the following: * ncaux_h5filterspec_parse * ncaux_h5filterspec_parselist * ncaux_h5filterspec_free * ncaux_h5filter_fix8 Misc. Other Changes: 1. Document NUG/filters.md updated to reflect the changes above. 2. All the old data types (structs and enums) used by filter_actions actions were deleted. The exception is the NC_H5_Filterspec because it is needed by ncaux_h5filterspec_parselist. 3. Clientside filters were removed -- another enhancement for which no-one ever asked. 4. The ability to remove filters was itself removed. 5. Some functionality needed by nczarr was moved from libhdf5 to libsrc4 e.g. nc4_find_default_chunksizes 6. All the filterx code was removed 7. ncfilter.h and nc4filter.c no longer used Misc. Unrelated Changes: 1. The nczarr_test makefile clean was leaving some directories; so add clean-local to take care of them.
2020-09-28 02:43:46 +08:00
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
/* YYPACT[STATE-NUM] -- Index in YYTABLE of the portion describing
STATE-NUM. */
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
static const yytype_int16 yypact[] =
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
-10, -13, 46, -149, -3, -149, 237, -149, -149, -149,
-149, -149, -149, -149, -149, -149, -149, -149, -149, -149,
-149, -149, -4, -149, -149, 407, -12, 40, 18, -149,
-149, 29, 32, 33, 38, 39, -9, 24, 251, 224,
56, 237, 86, 86, 43, 1, 333, 88, -149, -149,
-1, 42, 44, 45, 49, 53, 54, 57, 58, 60,
61, 62, 63, 64, 88, 65, 224, -149, -149, 67,
67, 67, 67, 71, 273, 69, 237, 101, -149, -149,
-149, -149, -149, -149, -149, -149, -149, -149, -149, -149,
-149, -149, -149, -149, -149, -149, -149, -149, -149, -149,
-149, -149, -149, -149, -149, 333, -149, 72, -149, -149,
-149, -149, -149, -149, -149, 76, 74, 78, 79, 333,
86, 1, 1, 43, 86, 43, 43, 86, 86, 1,
1, 333, 84, -149, 123, -149, -149, -149, -149, -149,
-149, 88, 80, -149, 237, 89, 85, -149, 90, -149,
92, 237, 117, 16, 333, 408, -149, 333, 333, 72,
-149, 94, -149, -149, -149, -149, -149, -149, -149, -149,
-149, -149, 72, 407, 87, 100, 96, 102, -149, 88,
41, 237, 103, -149, 371, -149, 407, -149, -149, -149,
-45, -149, 237, 72, 72, 1, 309, 104, 88, -149,
88, 88, 88, -149, -149, -149, -149, -149, 105, -149,
95, -149, 106, -149, 108, 107, -149, 407, 112, 408,
-149, -149, -149, -149, 113, -149, 115, -149, 111, -149,
22, -149, 119, -149, -149, 2, -2, -149, 333, 122,
-149, -149, 129, -149, 88, 11, -149, -149, 88, 1,
-149, -149, -27, -149, -149, 72, -149, 91, -149, -149,
-149, 12, -149, -149, -149, -2, -149, 237, 11, -149,
-149, -149, -149
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
};
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
/* YYDEFACT[STATE-NUM] -- Default reduction number in state STATE-NUM.
Performed when YYTABLE does not specify something else to do. Zero
means the default is an error. */
More tweaking.
2020-12-08 05:45:14 +08:00
static const yytype_uint8 yydefact[] =
{
0, 0, 0, 3, 0, 1, 88, 2, 35, 36,
37, 38, 39, 40, 41, 42, 43, 44, 45, 46,
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
158, 111, 0, 6, 87, 0, 85, 11, 0, 86,
110, 0, 0, 0, 0, 0, 0, 0, 0, 12,
47, 88, 0, 0, 0, 0, 120, 0, 4, 7,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 13, 14, 17, 23,
23, 23, 23, 87, 0, 0, 48, 59, 89, 153,
109, 90, 149, 151, 150, 152, 155, 154, 91, 92,
146, 135, 136, 137, 138, 139, 140, 141, 142, 143,
144, 145, 126, 127, 128, 120, 131, 93, 118, 119,
121, 123, 129, 130, 125, 110, 0, 0, 0, 120,
More tweaking.
2020-12-08 05:45:14 +08:00
0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
0, 120, 0, 16, 0, 15, 24, 19, 22, 21,
20, 0, 0, 18, 49, 0, 52, 54, 0, 53,
110, 60, 112, 0, 0, 0, 8, 120, 120, 96,
98, 99, 147, 101, 102, 103, 108, 100, 104, 105,
106, 107, 95, 0, 0, 0, 0, 0, 50, 0,
0, 61, 0, 64, 0, 65, 113, 5, 124, 122,
0, 133, 88, 97, 94, 0, 0, 0, 0, 85,
0, 0, 0, 51, 55, 58, 57, 56, 0, 62,
156, 157, 66, 67, 70, 0, 84, 114, 0, 0,
132, 6, 148, 31, 0, 32, 34, 75, 78, 29,
0, 26, 0, 30, 63, 0, 0, 69, 120, 0,
115, 134, 9, 33, 0, 0, 77, 25, 0, 0,
156, 68, 0, 72, 74, 117, 116, 0, 76, 83,
82, 0, 80, 27, 28, 0, 71, 88, 0, 79,
73, 10, 81
More tweaking.
2020-12-08 05:45:14 +08:00
};
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
/* YYPGOTO[NTERM-NUM]. */
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
static const yytype_int16 yypgoto[] =
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
-149, -149, -149, -149, -8, -35, -149, -149, -149, -149,
-149, -130, 121, -149, 28, -149, -149, -63, -149, -149,
-149, -149, -7, -26, -149, -149, 47, -149, 9, -149,
-149, -149, 14, -149, -149, -42, -149, -149, -75, -149,
-48, -149, -149, -71, -149, -36, -15, -40, -33, -44,
-149, -149, -19, -100, -149, -149, 50, -149, -149, -149,
-149, -148, -149, -41, -34, -73, -149, -22
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
};
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
/* YYDEFGOTO[NTERM-NUM]. */
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
static const yytype_int16 yydefgoto[] =
More tweaking.
2020-12-08 05:45:14 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
0, 2, 4, 7, 23, 36, 49, 192, 257, 40,
66, 132, 67, 68, 137, 69, 230, 231, 70, 71,
72, 196, 197, 24, 77, 144, 145, 146, 147, 148,
152, 181, 182, 183, 212, 213, 237, 252, 253, 226,
227, 246, 261, 262, 215, 25, 26, 27, 28, 29,
187, 217, 218, 107, 108, 109, 110, 111, 112, 113,
190, 114, 161, 86, 87, 88, 214, 30
More tweaking.
2020-12-08 05:45:14 +08:00
};
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
/* YYTABLE[YYPACT[STATE-NUM]] -- What to do in state STATE-NUM. If
positive, shift that token. If negative, reduce the rule whose
number is the opposite. If YYTABLE_NINF, syntax error. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
static const yytype_int16 yytable[] =
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
35, 78, 106, 74, 89, 153, 75, 191, 80, 81,
37, 175, 20, 73, 20, 20, 63, 219, 250, 159,
220, 1, 82, 83, 115, 116, 84, 85, 118, 47,
74, 172, 259, 75, 211, 265, 260, 21, 266, 31,
73, 117, 133, 149, 206, 3, 5, 32, 33, 34,
164, 48, 166, 167, 150, 38, 6, 193, 194, 37,
79, 106, 82, 83, 82, 83, 84, 85, 84, 85,
229, 241, 233, 39, 268, 106, 188, 269, 154, 41,
162, 163, 247, 115, 248, 76, 160, 106, 170, 171,
165, 50, 42, 168, 169, 43, 44, 115, 138, 139,
140, 45, 46, 79, 20, 119, 141, 120, 121, 115,
106, 149, 122, 106, 106, 184, 123, 124, 185, 133,
125, 126, 150, 127, 128, 129, 130, 131, 136, 134,
143, 151, 115, 156, 154, 115, 115, 198, 255, 207,
155, 157, 158, 173, 174, 184, 176, 179, 185, 186,
178, 267, 200, 180, 222, -58, 195, 205, 199, 201,
198, 202, -158, 203, 209, 225, 234, 47, 235, 37,
238, 216, 236, 240, 243, 245, 228, 244, 133, 232,
133, 199, 249, 256, 221, 263, 242, 135, 204, 224,
270, 177, 254, 251, 106, 208, 258, 272, 239, 0,
0, 0, 216, 0, 189, 0, 0, 0, 264, 0,
0, 0, 0, 0, 0, 0, 115, 0, 0, 0,
0, 254, 228, 0, 0, 0, 232, 271, 8, 9,
10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 0, 0, 0, 0, 64, 0,
65, 0, 0, 21, 0, 0, 0, 20, 0, 0,
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
0, 0, 0, 0, 0, 0, 21, 8, 9, 10,
11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
0, 22, 0, 51, 0, 52, 53, 54, 55, 56,
57, 58, 0, 0, 22, 59, 60, 61, 62, 0,
0, 0, 21, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18, 19, 20, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 142, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 21, 20,
90, 91, 92, 93, 94, 95, 96, 97, 98, 99,
100, 101, 0, 0, 0, 0, 0, 0, 0, 223,
102, 0, 21, 103, 104, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18, 19, 210, 0, 0,
0, 0, 105, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 211, 0, 0, 0, 0, 0, 0,
21, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 0, 90, 91, 92, 93, 94,
95, 96, 97, 98, 99, 100, 101, 0, 0, 0,
0, 0, 0, 0, 0, 0, 21
moving to trunk subdir
2010-06-03 21:24:43 +08:00
};
static const yytype_int16 yycheck[] =
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
22, 41, 46, 39, 45, 105, 39, 155, 42, 43,
25, 141, 16, 39, 16, 16, 38, 62, 16, 119,
65, 31, 21, 22, 46, 47, 25, 26, 50, 38,
66, 131, 21, 66, 32, 62, 25, 39, 65, 43,
66, 42, 64, 76, 3, 58, 0, 51, 52, 53,
123, 60, 125, 126, 76, 67, 59, 157, 158, 74,
17, 105, 21, 22, 21, 22, 25, 26, 25, 26,
200, 219, 202, 33, 62, 119, 60, 65, 62, 61,
121, 122, 60, 105, 62, 29, 120, 131, 129, 130,
124, 67, 63, 127, 128, 63, 63, 119, 70, 71,
72, 63, 63, 17, 16, 63, 35, 63, 63, 131,
154, 144, 63, 157, 158, 151, 63, 63, 151, 141,
63, 63, 144, 63, 63, 63, 63, 63, 61, 64,
61, 30, 154, 59, 62, 157, 158, 173, 238, 180,
64, 63, 63, 59, 21, 181, 66, 62, 181, 32,
61, 60, 65, 63, 195, 63, 62, 179, 173, 59,
196, 65, 67, 61, 61, 61, 61, 38, 62, 184,
63, 186, 64, 61, 61, 64, 198, 62, 200, 201,
202, 196, 63, 61, 192, 248, 221, 66, 179, 196,
265, 144, 236, 235, 238, 181, 244, 268, 217, -1,
-1, -1, 217, -1, 154, -1, -1, -1, 249, -1,
-1, -1, -1, -1, -1, -1, 238, -1, -1, -1,
-1, 265, 244, -1, -1, -1, 248, 267, 4, 5,
6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16, -1, -1, -1, -1, 34, -1,
36, -1, -1, 39, -1, -1, -1, 16, -1, -1,
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
-1, -1, -1, -1, -1, -1, 39, 4, 5, 6,
7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
-1, 67, -1, 42, -1, 44, 45, 46, 47, 48,
49, 50, -1, -1, 67, 54, 55, 56, 57, -1,
-1, -1, 39, 4, 5, 6, 7, 8, 9, 10,
11, 12, 13, 14, 15, 16, -1, -1, -1, -1,
-1, -1, -1, -1, -1, -1, -1, 64, -1, -1,
-1, -1, -1, -1, -1, -1, -1, -1, 39, 16,
17, 18, 19, 20, 21, 22, 23, 24, 25, 26,
27, 28, -1, -1, -1, -1, -1, -1, -1, 60,
37, -1, 39, 40, 41, 4, 5, 6, 7, 8,
9, 10, 11, 12, 13, 14, 15, 16, -1, -1,
-1, -1, 59, -1, -1, -1, -1, -1, -1, -1,
-1, -1, -1, 32, -1, -1, -1, -1, -1, -1,
39, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16, -1, 17, 18, 19, 20, 21,
22, 23, 24, 25, 26, 27, 28, -1, -1, -1,
-1, -1, -1, -1, -1, -1, 39
moving to trunk subdir
2010-06-03 21:24:43 +08:00
};
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
/* YYSTOS[STATE-NUM] -- The (internal number of the) accessing
symbol of state STATE-NUM. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
static const yytype_uint8 yystos[] =
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
0, 31, 69, 58, 70, 0, 59, 71, 4, 5,
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
16, 39, 67, 72, 91, 113, 114, 115, 116, 117,
135, 43, 51, 52, 53, 135, 73, 114, 67, 33,
77, 61, 63, 63, 63, 63, 63, 38, 60, 74,
67, 42, 44, 45, 46, 47, 48, 49, 50, 54,
55, 56, 57, 135, 34, 36, 78, 80, 81, 83,
86, 87, 88, 91, 113, 116, 29, 92, 115, 17,
132, 132, 21, 22, 25, 26, 131, 132, 133, 131,
17, 18, 19, 20, 21, 22, 23, 24, 25, 26,
27, 28, 37, 40, 41, 59, 117, 121, 122, 123,
124, 125, 126, 127, 129, 135, 135, 42, 135, 63,
63, 63, 63, 63, 63, 63, 63, 63, 63, 63,
63, 63, 79, 135, 64, 80, 61, 82, 82, 82,
82, 35, 64, 61, 93, 94, 95, 96, 97, 116,
135, 30, 98, 121, 62, 64, 59, 63, 63, 121,
132, 130, 131, 131, 133, 132, 133, 133, 132, 132,
131, 131, 121, 59, 21, 79, 66, 94, 61, 62,
63, 99, 100, 101, 113, 116, 32, 118, 60, 124,
128, 129, 75, 121, 121, 62, 89, 90, 113, 114,
65, 59, 65, 61, 96, 135, 3, 131, 100, 61,
16, 32, 102, 103, 134, 112, 114, 119, 120, 62,
65, 72, 131, 60, 90, 61, 107, 108, 135, 79,
84, 85, 135, 79, 61, 62, 64, 104, 63, 120,
61, 129, 73, 61, 62, 64, 109, 60, 62, 63,
16, 103, 105, 106, 117, 121, 61, 76, 108, 21,
25, 110, 111, 85, 131, 62, 65, 60, 62, 65,
106, 115, 111
moving to trunk subdir
2010-06-03 21:24:43 +08:00
};
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
/* YYR1[YYN] -- Symbol number of symbol that rule YYN derives. */
More tweaking.
2020-12-08 05:45:14 +08:00
static const yytype_uint8 yyr1[] =
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
0, 68, 69, 70, 71, 72, 73, 73, 75, 76,
74, 77, 77, 77, 78, 78, 79, 80, 80, 81,
81, 81, 81, 82, 82, 83, 84, 84, 85, 86,
87, 88, 89, 89, 90, 91, 91, 91, 91, 91,
91, 91, 91, 91, 91, 91, 91, 92, 92, 92,
93, 93, 94, 94, 95, 95, 96, 96, 97, 98,
98, 98, 99, 99, 100, 100, 101, 102, 102, 103,
104, 104, 105, 105, 106, 107, 107, 108, 109, 109,
110, 110, 111, 111, 112, 113, 114, 114, 115, 115,
116, 116, 116, 116, 116, 116, 116, 116, 116, 116,
116, 116, 116, 116, 116, 116, 116, 116, 116, 116,
117, 117, 118, 118, 118, 119, 119, 120, 121, 121,
122, 123, 123, 124, 124, 125, 125, 125, 125, 125,
125, 126, 127, 128, 128, 129, 129, 129, 129, 129,
129, 129, 129, 129, 129, 129, 129, 130, 130, 131,
131, 131, 131, 132, 133, 133, 134, 134, 135
More tweaking.
2020-12-08 05:45:14 +08:00
};
[NCF-271] Fix bug introduced by [NCF-267]. The bug was that octal constants that had the highest bit set (e.g. '\200') were not recognized as proper octal constants. Fix was to keep as integer until it was needed as an 8-bit byte.
2013-09-27 00:10:26 +08:00
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
/* YYR2[YYN] -- Number of symbols on the right hand side of rule YYN. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
static const yytype_int8 yyr2[] =
More tweaking.
2020-12-08 05:45:14 +08:00
{
0, 2, 3, 1, 4, 5, 0, 2, 0, 0,
9, 0, 1, 2, 1, 2, 1, 1, 2, 2,
2, 2, 2, 0, 1, 6, 1, 3, 3, 5,
5, 5, 2, 3, 2, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 0, 1, 2,
2, 3, 1, 1, 1, 3, 3, 3, 1, 0,
1, 2, 2, 3, 1, 1, 2, 1, 3, 2,
0, 3, 1, 3, 1, 1, 3, 2, 0, 3,
1, 3, 1, 1, 1, 1, 1, 1, 0, 3,
4, 4, 4, 4, 6, 5, 5, 6, 5, 5,
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
5, 5, 5, 5, 5, 5, 5, 5, 5, 4,
1, 1, 0, 1, 2, 2, 3, 3, 1, 1,
0, 1, 3, 1, 3, 1, 1, 1, 1, 1,
1, 1, 4, 1, 3, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 3, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1
More tweaking.
2020-12-08 05:45:14 +08:00
};
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Resolves Github issue https://github.com/Unidata/netcdf-c/issues/349. Update utf8proc.[ch] to use the version now maintained by the Julia Language project (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). The license for the previous version was unacceptable for the Debian and Ubuntu release systems. The new version both updates the code and addresses the license issue. It turns out that the utf8proc software we are using was turned over to the Julia Language developers and the license terms changed to allow modification. (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). So the fix here is as follows: 1. Wrap the library with a fixed interface: libdispatch/dutf8.c and include/ncutf8.h. 2. Replace the existing utf8proc code with the new version from https://github.com/JuliaLang/utf8proc. 3. Add a couple more test cases: nc_test/tst_utf8_validate.c and nc_test_utf8_phrases.c. If/when I can find a usable normalization test, I will incorporate that later.
2017-02-17 05:27:54 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
enum { YYENOMEM = -2 };
More tweaking.
2020-12-08 05:45:14 +08:00
#define yyerrok (yyerrstatus = 0)
#define yyclearin (yychar = YYEMPTY)
Merge branch 'master' into filters.dmh
2017-11-14 02:15:02 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
#define YYACCEPT goto yyacceptlab
#define YYABORT goto yyabortlab
#define YYERROR goto yyerrorlab
Resolves Github issue https://github.com/Unidata/netcdf-c/issues/349. Update utf8proc.[ch] to use the version now maintained by the Julia Language project (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). The license for the previous version was unacceptable for the Debian and Ubuntu release systems. The new version both updates the code and addresses the license issue. It turns out that the utf8proc software we are using was turned over to the Julia Language developers and the license terms changed to allow modification. (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). So the fix here is as follows: 1. Wrap the library with a fixed interface: libdispatch/dutf8.c and include/ncutf8.h. 2. Replace the existing utf8proc code with the new version from https://github.com/JuliaLang/utf8proc. 3. Add a couple more test cases: nc_test/tst_utf8_validate.c and nc_test_utf8_phrases.c. If/when I can find a usable normalization test, I will incorporate that later.
2017-02-17 05:27:54 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
#define YYRECOVERING() (!!yyerrstatus)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#define YYBACKUP(Token, Value) \
do \
if (yychar == YYEMPTY) \
{ \
yychar = (Token); \
yylval = (Value); \
YYPOPSTACK (yylen); \
yystate = *yyssp; \
goto yybackup; \
} \
else \
{ \
yyerror (YY_("syntax error: cannot back up")); \
YYERROR; \
} \
while (0)
/* Backward compatibility with an undocumented macro.
Use YYerror or YYUNDEF. */
#define YYERRCODE YYUNDEF
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Enable debugging if requested. */
#if YYDEBUG
# ifndef YYFPRINTF
# include <stdio.h> /* INFRINGES ON USER NAME SPACE */
# define YYFPRINTF fprintf
# endif
More tweaking.
2020-12-08 05:45:14 +08:00
# define YYDPRINTF(Args) \
do { \
if (yydebug) \
YYFPRINTF Args; \
} while (0)
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
/* This macro is provided for backward compatibility. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# ifndef YY_LOCATION_PRINT
# define YY_LOCATION_PRINT(File, Loc) ((void) 0)
# endif
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# define YY_SYMBOL_PRINT(Title, Kind, Value, Location) \
More tweaking.
2020-12-08 05:45:14 +08:00
do { \
if (yydebug) \
{ \
YYFPRINTF (stderr, "%s ", Title); \
yy_symbol_print (stderr, \
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
Kind, Value); \
More tweaking.
2020-12-08 05:45:14 +08:00
YYFPRINTF (stderr, "\n"); \
} \
} while (0)
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/*-----------------------------------.
| Print this symbol's value on YYO. |
`-----------------------------------*/
More tweaking.
2020-12-08 05:45:14 +08:00
static void
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yy_symbol_value_print (FILE *yyo,
yysymbol_kind_t yykind, YYSTYPE const * const yyvaluep)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
FILE *yyoutput = yyo;
YY_USE (yyoutput);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
if (!yyvaluep)
return;
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
# ifdef YYPRINT
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
if (yykind < YYNTOKENS)
YYPRINT (yyo, yytoknum[yykind], *yyvaluep);
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
# endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YY_IGNORE_MAYBE_UNINITIALIZED_BEGIN
YY_USE (yykind);
YY_IGNORE_MAYBE_UNINITIALIZED_END
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/*---------------------------.
| Print this symbol on YYO. |
`---------------------------*/
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
static void
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yy_symbol_print (FILE *yyo,
yysymbol_kind_t yykind, YYSTYPE const * const yyvaluep)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YYFPRINTF (yyo, "%s %s (",
yykind < YYNTOKENS ? "token" : "nterm", yysymbol_name (yykind));
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yy_symbol_value_print (yyo, yykind, yyvaluep);
YYFPRINTF (yyo, ")");
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
/*------------------------------------------------------------------.
| yy_stack_print -- Print the state stack from its BOTTOM up to its |
| TOP (included). |
`------------------------------------------------------------------*/
static void
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yy_stack_print (yy_state_t *yybottom, yy_state_t *yytop)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
YYFPRINTF (stderr, "Stack now");
More tweaking.
2020-12-08 05:45:14 +08:00
for (; yybottom <= yytop; yybottom++)
{
int yybot = *yybottom;
YYFPRINTF (stderr, " %d", yybot);
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
YYFPRINTF (stderr, "\n");
}
More tweaking.
2020-12-08 05:45:14 +08:00
# define YY_STACK_PRINT(Bottom, Top) \
do { \
if (yydebug) \
yy_stack_print ((Bottom), (Top)); \
} while (0)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/*------------------------------------------------.
| Report that the YYRULE is going to be reduced. |
`------------------------------------------------*/
Attempting to fix visual studio errors in support of https://github.com/Unidata/netcdf-c/pull/375
2017-03-14 05:12:47 +08:00
static void
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yy_reduce_print (yy_state_t *yyssp, YYSTYPE *yyvsp,
int yyrule)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
int yylno = yyrline[yyrule];
moving to trunk subdir
2010-06-03 21:24:43 +08:00
int yynrhs = yyr2[yyrule];
int yyi;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YYFPRINTF (stderr, "Reducing stack by rule %d (line %d):\n",
More tweaking.
2020-12-08 05:45:14 +08:00
yyrule - 1, yylno);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* The symbols being reduced. */
for (yyi = 0; yyi < yynrhs; yyi++)
{
More tweaking.
2020-12-08 05:45:14 +08:00
YYFPRINTF (stderr, " $%d = ", yyi + 1);
yy_symbol_print (stderr,
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YY_ACCESSING_SYMBOL (+yyssp[yyi + 1 - yynrhs]),
&yyvsp[(yyi + 1) - (yynrhs)]);
More tweaking.
2020-12-08 05:45:14 +08:00
YYFPRINTF (stderr, "\n");
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
}
More tweaking.
2020-12-08 05:45:14 +08:00
# define YY_REDUCE_PRINT(Rule) \
do { \
if (yydebug) \
yy_reduce_print (yyssp, yyvsp, Rule); \
} while (0)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Nonzero means print parse trace. It is left uninitialized so that
multiple parsers can coexist. */
int yydebug;
#else /* !YYDEBUG */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# define YYDPRINTF(Args) ((void) 0)
# define YY_SYMBOL_PRINT(Title, Kind, Value, Location)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
# define YY_STACK_PRINT(Bottom, Top)
# define YY_REDUCE_PRINT(Rule)
#endif /* !YYDEBUG */
/* YYINITDEPTH -- initial size of the parser's stacks. */
More tweaking.
2020-12-08 05:45:14 +08:00
#ifndef YYINITDEPTH
moving to trunk subdir
2010-06-03 21:24:43 +08:00
# define YYINITDEPTH 200
#endif
/* YYMAXDEPTH -- maximum size the stacks can grow to (effective only
if the built-in stack extension method is used).
Do not make this value too large; the results are undefined if
YYSTACK_ALLOC_MAXIMUM < YYSTACK_BYTES (YYMAXDEPTH)
evaluated with infinite-precision integer arithmetic. */
#ifndef YYMAXDEPTH
# define YYMAXDEPTH 10000
#endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Context of a parse error. */
typedef struct
{
yy_state_t *yyssp;
yysymbol_kind_t yytoken;
} yypcontext_t;
/* Put in YYARG at most YYARGN of the expected tokens given the
current YYCTX, and return the number of tokens stored in YYARG. If
YYARG is null, return the number of expected tokens (guaranteed to
be less than YYNTOKENS). Return YYENOMEM on memory exhaustion.
Return 0 if there are more than YYARGN expected tokens, yet fill
YYARG up to YYARGN. */
static int
yypcontext_expected_tokens (const yypcontext_t *yyctx,
yysymbol_kind_t yyarg[], int yyargn)
{
/* Actual size of YYARG. */
int yycount = 0;
int yyn = yypact[+*yyctx->yyssp];
if (!yypact_value_is_default (yyn))
{
/* Start YYX at -YYN if negative to avoid negative indexes in
YYCHECK. In other words, skip the first -YYN actions for
this state because they are default actions. */
int yyxbegin = yyn < 0 ? -yyn : 0;
/* Stay within bounds of both yycheck and yytname. */
int yychecklim = YYLAST - yyn + 1;
int yyxend = yychecklim < YYNTOKENS ? yychecklim : YYNTOKENS;
int yyx;
for (yyx = yyxbegin; yyx < yyxend; ++yyx)
if (yycheck[yyx + yyn] == yyx && yyx != YYSYMBOL_YYerror
&& !yytable_value_is_error (yytable[yyx + yyn]))
{
if (!yyarg)
++yycount;
else if (yycount == yyargn)
return 0;
else
yyarg[yycount++] = YY_CAST (yysymbol_kind_t, yyx);
}
}
if (yyarg && yycount == 0 && 0 < yyargn)
yyarg[0] = YYSYMBOL_YYEMPTY;
return yycount;
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#ifndef yystrlen
# if defined __GLIBC__ && defined _STRING_H
# define yystrlen(S) (YY_CAST (YYPTRDIFF_T, strlen (S)))
# else
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Return the length of YYSTR. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
static YYPTRDIFF_T
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yystrlen (const char *yystr)
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YYPTRDIFF_T yylen;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
for (yylen = 0; yystr[yylen]; yylen++)
continue;
return yylen;
}
# endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#endif
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#ifndef yystpcpy
# if defined __GLIBC__ && defined _STRING_H && defined _GNU_SOURCE
# define yystpcpy stpcpy
# else
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Copy YYSRC to YYDEST, returning the address of the terminating '\0' in
YYDEST. */
static char *
yystpcpy (char *yydest, const char *yysrc)
{
char *yyd = yydest;
const char *yys = yysrc;
while ((*yyd++ = *yys++) != '\0')
continue;
return yyd - 1;
}
# endif
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#endif
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#ifndef yytnamerr
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Copy to YYRES the contents of YYSTR after stripping away unnecessary
quotes and backslashes, so that it's suitable for yyerror. The
heuristic is that double-quoting is unnecessary unless the string
contains an apostrophe, a comma, or backslash (other than
backslash-backslash). YYSTR is taken from yytname. If YYRES is
null, do not copy; instead, return the length of what the result
would have been. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
static YYPTRDIFF_T
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yytnamerr (char *yyres, const char *yystr)
{
if (*yystr == '"')
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YYPTRDIFF_T yyn = 0;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
char const *yyp = yystr;
for (;;)
More tweaking.
2020-12-08 05:45:14 +08:00
switch (*++yyp)
{
case '\'':
case ',':
goto do_not_strip_quotes;
case '\\':
if (*++yyp != '\\')
goto do_not_strip_quotes;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
else
goto append;
append:
More tweaking.
2020-12-08 05:45:14 +08:00
default:
if (yyres)
yyres[yyn] = *yyp;
yyn++;
break;
case '"':
if (yyres)
yyres[yyn] = '\0';
return yyn;
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
do_not_strip_quotes: ;
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
if (yyres)
return yystpcpy (yyres, yystr) - yyres;
else
moving to trunk subdir
2010-06-03 21:24:43 +08:00
return yystrlen (yystr);
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#endif
moving to trunk subdir
2010-06-03 21:24:43 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
static int
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yy_syntax_error_arguments (const yypcontext_t *yyctx,
yysymbol_kind_t yyarg[], int yyargn)
1. There were several bugs in ncdump/dumplib.c that were not taking the CDF-5 format into account. 2. Had to update ncgen.1 man page to define the new k-flag rules to deal with cdf-5. 3. Had to fix some tests that use 'cmp' for comparison; this really should be deprecated. 3. There was a bug in configure.ac with respect to using the enable-netcdf-4 flag vs using disable-netcdf-4.
2015-11-07 08:03:28 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Actual size of YYARG. */
More tweaking.
2020-12-08 05:45:14 +08:00
int yycount = 0;
/* There are many possibilities here to consider:
- If this state is a consistent state with a default action, then
the only way this function was invoked is if the default action
is an error action. In that case, don't check for expected
tokens because there are none.
- The only way there can be no lookahead present (in yychar) is if
this state is a consistent state with a default action. Thus,
detecting the absence of a lookahead is sufficient to determine
that there is no unexpected or expected token to report. In that
case, just report a simple "syntax error".
- Don't assume there isn't a lookahead just because this state is a
consistent state with a default action. There might have been a
previous inconsistent state, consistent state with a non-default
action, or user semantic action that manipulated yychar.
- Of course, the expected token list depends on states to have
correct lookahead information, and it depends on the parser not
to perform extra reductions after fetching a lookahead from the
scanner and before detecting a syntax error. Thus, state merging
(from LALR or IELR) and default reductions corrupt the expected
token list. However, the list is correct for canonical LR with
one exception: it will still contain any token that will not be
accepted due to an error action in a later state.
*/
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
if (yyctx->yytoken != YYSYMBOL_YYEMPTY)
More tweaking.
2020-12-08 05:45:14 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
int yyn;
if (yyarg)
yyarg[yycount] = yyctx->yytoken;
++yycount;
yyn = yypcontext_expected_tokens (yyctx,
yyarg ? yyarg + 1 : yyarg, yyargn - 1);
if (yyn == YYENOMEM)
return YYENOMEM;
else
yycount += yyn;
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
return yycount;
}
/* Copy into *YYMSG, which is of size *YYMSG_ALLOC, an error message
about the unexpected token YYTOKEN for the state stack whose top is
YYSSP.
Return 0 if *YYMSG was successfully written. Return -1 if *YYMSG is
not large enough to hold the message. In that case, also set
*YYMSG_ALLOC to the required number of bytes. Return YYENOMEM if the
required number of bytes is too large to store. */
static int
yysyntax_error (YYPTRDIFF_T *yymsg_alloc, char **yymsg,
const yypcontext_t *yyctx)
{
enum { YYARGS_MAX = 5 };
/* Internationalized format string. */
const char *yyformat = YY_NULLPTR;
/* Arguments of yyformat: reported tokens (one for the "unexpected",
one per "expected"). */
yysymbol_kind_t yyarg[YYARGS_MAX];
/* Cumulated lengths of YYARG. */
YYPTRDIFF_T yysize = 0;
/* Actual size of YYARG. */
int yycount = yy_syntax_error_arguments (yyctx, yyarg, YYARGS_MAX);
if (yycount == YYENOMEM)
return YYENOMEM;
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
switch (yycount)
Mostly revert the filter code to reduce its complexity of use. re: https://github.com/Unidata/netcdf-c/issues/1836 Revert the internal filter code to simplify it. From the user's point of view, the only visible changes should be: 1. The functions that convert text to filter specs have had their signature reverted and have been moved to netcdf_aux.h 2. Some filter API functions now return NC_ENOFILTER when inquiry is made about some filter. Internally,the dispatch table has been modified to get rid of the filter_actions entry and associated complex structures. It has been replaced with inq_var_filter_ids and inq_var_filter_info entries and the dispatch table version has been bumped to 3. Corresponding NOOP and NOTNC4 functions were added to libdispatch/dnotnc4.c. Also, the filter_action entries in dispatch tables were replaced for all dispatch code bases (HDF5, DAP2, etc). This should only impact UDF users. In the process, it became clear that the form of the filters field in NC_VAR_INFO_T was format dependent, so I converted it to be of type void* and pushed its management into the various dispatch code bases. Specifically libhdf5 and libnczarr now manage the filters field in their own way. The auxilliary functions for parsing textual filter specifications were moved to netcdf_aux.h and were renamed to the following: * ncaux_h5filterspec_parse * ncaux_h5filterspec_parselist * ncaux_h5filterspec_free * ncaux_h5filter_fix8 Misc. Other Changes: 1. Document NUG/filters.md updated to reflect the changes above. 2. All the old data types (structs and enums) used by filter_actions actions were deleted. The exception is the NC_H5_Filterspec because it is needed by ncaux_h5filterspec_parselist. 3. Clientside filters were removed -- another enhancement for which no-one ever asked. 4. The ability to remove filters was itself removed. 5. Some functionality needed by nczarr was moved from libhdf5 to libsrc4 e.g. nc4_find_default_chunksizes 6. All the filterx code was removed 7. ncfilter.h and nc4filter.c no longer used Misc. Unrelated Changes: 1. The nczarr_test makefile clean was leaving some directories; so add clean-local to take care of them.
2020-09-28 02:43:46 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#define YYCASE_(N, S) \
More tweaking.
2020-12-08 05:45:14 +08:00
case N: \
yyformat = S; \
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
break
default: /* Avoid compiler warnings. */
More tweaking.
2020-12-08 05:45:14 +08:00
YYCASE_(0, YY_("syntax error"));
YYCASE_(1, YY_("syntax error, unexpected %s"));
YYCASE_(2, YY_("syntax error, unexpected %s, expecting %s"));
YYCASE_(3, YY_("syntax error, unexpected %s, expecting %s or %s"));
YYCASE_(4, YY_("syntax error, unexpected %s, expecting %s or %s or %s"));
YYCASE_(5, YY_("syntax error, unexpected %s, expecting %s or %s or %s or %s"));
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#undef YYCASE_
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Compute error message size. Don't count the "%s"s, but reserve
room for the terminator. */
yysize = yystrlen (yyformat) - 2 * yycount + 1;
More tweaking.
2020-12-08 05:45:14 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
int yyi;
for (yyi = 0; yyi < yycount; ++yyi)
{
YYPTRDIFF_T yysize1
= yysize + yytnamerr (YY_NULLPTR, yytname[yyarg[yyi]]);
if (yysize <= yysize1 && yysize1 <= YYSTACK_ALLOC_MAXIMUM)
yysize = yysize1;
else
return YYENOMEM;
}
More tweaking.
2020-12-08 05:45:14 +08:00
}
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
if (*yymsg_alloc < yysize)
{
*yymsg_alloc = 2 * yysize;
if (! (yysize <= *yymsg_alloc
&& *yymsg_alloc <= YYSTACK_ALLOC_MAXIMUM))
*yymsg_alloc = YYSTACK_ALLOC_MAXIMUM;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
return -1;
Mostly revert the filter code to reduce its complexity of use. re: https://github.com/Unidata/netcdf-c/issues/1836 Revert the internal filter code to simplify it. From the user's point of view, the only visible changes should be: 1. The functions that convert text to filter specs have had their signature reverted and have been moved to netcdf_aux.h 2. Some filter API functions now return NC_ENOFILTER when inquiry is made about some filter. Internally,the dispatch table has been modified to get rid of the filter_actions entry and associated complex structures. It has been replaced with inq_var_filter_ids and inq_var_filter_info entries and the dispatch table version has been bumped to 3. Corresponding NOOP and NOTNC4 functions were added to libdispatch/dnotnc4.c. Also, the filter_action entries in dispatch tables were replaced for all dispatch code bases (HDF5, DAP2, etc). This should only impact UDF users. In the process, it became clear that the form of the filters field in NC_VAR_INFO_T was format dependent, so I converted it to be of type void* and pushed its management into the various dispatch code bases. Specifically libhdf5 and libnczarr now manage the filters field in their own way. The auxilliary functions for parsing textual filter specifications were moved to netcdf_aux.h and were renamed to the following: * ncaux_h5filterspec_parse * ncaux_h5filterspec_parselist * ncaux_h5filterspec_free * ncaux_h5filter_fix8 Misc. Other Changes: 1. Document NUG/filters.md updated to reflect the changes above. 2. All the old data types (structs and enums) used by filter_actions actions were deleted. The exception is the NC_H5_Filterspec because it is needed by ncaux_h5filterspec_parselist. 3. Clientside filters were removed -- another enhancement for which no-one ever asked. 4. The ability to remove filters was itself removed. 5. Some functionality needed by nczarr was moved from libhdf5 to libsrc4 e.g. nc4_find_default_chunksizes 6. All the filterx code was removed 7. ncfilter.h and nc4filter.c no longer used Misc. Unrelated Changes: 1. The nczarr_test makefile clean was leaving some directories; so add clean-local to take care of them.
2020-09-28 02:43:46 +08:00
}
More tweaking.
2020-12-08 05:45:14 +08:00
/* Avoid sprintf, as that infringes on the user's name space.
Don't have undefined behavior even if the translation
produced a string with the wrong number of "%s"s. */
{
char *yyp = *yymsg;
int yyi = 0;
while ((*yyp = *yyformat) != '\0')
if (*yyp == '%' && yyformat[1] == 's' && yyi < yycount)
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yyp += yytnamerr (yyp, yytname[yyarg[yyi++]]);
More tweaking.
2020-12-08 05:45:14 +08:00
yyformat += 2;
}
else
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
++yyp;
++yyformat;
More tweaking.
2020-12-08 05:45:14 +08:00
}
}
return 0;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/*-----------------------------------------------.
| Release the memory associated to this symbol. |
`-----------------------------------------------*/
static void
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yydestruct (const char *yymsg,
yysymbol_kind_t yykind, YYSTYPE *yyvaluep)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YY_USE (yyvaluep);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
if (!yymsg)
yymsg = "Deleting";
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YY_SYMBOL_PRINT (yymsg, yykind, yyvaluep, yylocationp);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
YY_IGNORE_MAYBE_UNINITIALIZED_BEGIN
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YY_USE (yykind);
More tweaking.
2020-12-08 05:45:14 +08:00
YY_IGNORE_MAYBE_UNINITIALIZED_END
Added a tweak to prevent problems with ncdump and hdf5 trying to correct for a lack of ssize_t.
2016-05-11 05:52:46 +08:00
}
Addressed an API-related usage of strncmp.
2015-11-25 07:19:36 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Lookahead token kind. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
int yychar;
More tweaking.
2020-12-08 05:45:14 +08:00
/* The semantic value of the lookahead symbol. */
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
YYSTYPE yylval;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Number of syntax errors so far. */
int yynerrs;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
Resolves Github issue https://github.com/Unidata/netcdf-c/issues/349. Update utf8proc.[ch] to use the version now maintained by the Julia Language project (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). The license for the previous version was unacceptable for the Debian and Ubuntu release systems. The new version both updates the code and addresses the license issue. It turns out that the utf8proc software we are using was turned over to the Julia Language developers and the license terms changed to allow modification. (https://github.com/JuliaLang/utf8proc/blob/master/LICENSE.md). So the fix here is as follows: 1. Wrap the library with a fixed interface: libdispatch/dutf8.c and include/ncutf8.h. 2. Replace the existing utf8proc code with the new version from https://github.com/JuliaLang/utf8proc. 3. Add a couple more test cases: nc_test/tst_utf8_validate.c and nc_test_utf8_phrases.c. If/when I can find a usable normalization test, I will incorporate that later.
2017-02-17 05:27:54 +08:00
/*----------.
| yyparse. |
`----------*/
moving to trunk subdir
2010-06-03 21:24:43 +08:00
int
yyparse (void)
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yy_state_fast_t yystate = 0;
More tweaking.
2020-12-08 05:45:14 +08:00
/* Number of tokens to shift before error messages enabled. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
int yyerrstatus = 0;
Working out an error with unistd.h in windows.
2017-01-31 05:54:00 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Refer to the stacks through separate pointers, to allow yyoverflow
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
to reallocate them elsewhere. */
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Their size. */
YYPTRDIFF_T yystacksize = YYINITDEPTH;
merge-squash
2015-08-16 06:26:35 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* The state stack: array, bottom, top. */
yy_state_t yyssa[YYINITDEPTH];
yy_state_t *yyss = yyssa;
yy_state_t *yyssp = yyss;
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* The semantic value stack: array, bottom, top. */
YYSTYPE yyvsa[YYINITDEPTH];
YYSTYPE *yyvs = yyvsa;
YYSTYPE *yyvsp = yyvs;
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
int yyn;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* The return value of yyparse. */
More tweaking.
2020-12-08 05:45:14 +08:00
int yyresult;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Lookahead symbol kind. */
yysymbol_kind_t yytoken = YYSYMBOL_YYEMPTY;
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
/* The variables used to return semantic value and location from the
action routines. */
YYSTYPE yyval;
More tweaking.
2020-12-08 05:45:14 +08:00
/* Buffer for error messages, and its allocated size. */
char yymsgbuf[128];
char *yymsg = yymsgbuf;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YYPTRDIFF_T yymsg_alloc = sizeof yymsgbuf;
More tweaking.
2020-12-08 05:45:14 +08:00
#define YYPOPSTACK(N) (yyvsp -= (N), yyssp -= (N))
Merge branch 'ncgenkw.dmh' of https://github.com/DennisHeimbigner/netcdf-c into gh1753.wif
2020-12-08 02:29:12 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* The number of symbols on the RHS of the reduced rule.
Keep to zero when no symbol should be popped. */
int yylen = 0;
YYDPRINTF ((stderr, "Starting parse\n"));
More tweaking.
2020-12-08 05:45:14 +08:00
yychar = YYEMPTY; /* Cause a token to be read. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
goto yysetstate;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/*------------------------------------------------------------.
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
| yynewstate -- push a new state, which is found in yystate. |
moving to trunk subdir
2010-06-03 21:24:43 +08:00
`------------------------------------------------------------*/
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yynewstate:
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* In all cases, when you get here, the value and location stacks
have just been pushed. So pushing a state here evens the stacks. */
yyssp++;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/*--------------------------------------------------------------------.
| yysetstate -- set current state (the top of the stack) to yystate. |
`--------------------------------------------------------------------*/
yysetstate:
YYDPRINTF ((stderr, "Entering state %d\n", yystate));
YY_ASSERT (0 <= yystate && yystate < YYNSTATES);
YY_IGNORE_USELESS_CAST_BEGIN
*yyssp = YY_CAST (yy_state_t, yystate);
YY_IGNORE_USELESS_CAST_END
YY_STACK_PRINT (yyss, yyssp);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
if (yyss + yystacksize - 1 <= yyssp)
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#if !defined yyoverflow && !defined YYSTACK_RELOCATE
goto yyexhaustedlab;
#else
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
/* Get the current used size of the three stacks, in elements. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YYPTRDIFF_T yysize = yyssp - yyss + 1;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# if defined yyoverflow
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
More tweaking.
2020-12-08 05:45:14 +08:00
/* Give user a chance to reallocate the stack. Use copies of
these so that the &'s don't force the real ones into
memory. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yy_state_t *yyss1 = yyss;
More tweaking.
2020-12-08 05:45:14 +08:00
YYSTYPE *yyvs1 = yyvs;
/* Each stack pointer address is followed by the size of the
data in use in that stack, in bytes. This used to be a
conditional around just the two extra args, but that might
be undefined if yyoverflow is a macro. */
yyoverflow (YY_("memory exhausted"),
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
&yyss1, yysize * YYSIZEOF (*yyssp),
&yyvs1, yysize * YYSIZEOF (*yyvsp),
More tweaking.
2020-12-08 05:45:14 +08:00
&yystacksize);
yyss = yyss1;
yyvs = yyvs1;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
# else /* defined YYSTACK_RELOCATE */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Extend the stack our own way. */
if (YYMAXDEPTH <= yystacksize)
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
goto yyexhaustedlab;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yystacksize *= 2;
if (YYMAXDEPTH < yystacksize)
More tweaking.
2020-12-08 05:45:14 +08:00
yystacksize = YYMAXDEPTH;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yy_state_t *yyss1 = yyss;
More tweaking.
2020-12-08 05:45:14 +08:00
union yyalloc *yyptr =
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YY_CAST (union yyalloc *,
YYSTACK_ALLOC (YY_CAST (YYSIZE_T, YYSTACK_BYTES (yystacksize))));
More tweaking.
2020-12-08 05:45:14 +08:00
if (! yyptr)
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
goto yyexhaustedlab;
More tweaking.
2020-12-08 05:45:14 +08:00
YYSTACK_RELOCATE (yyss_alloc, yyss);
YYSTACK_RELOCATE (yyvs_alloc, yyvs);
Add support for the XArray Zarr _ARRAY_DIMENSIONS attribute The XArray implementation that uses Zarr for storage provides a mechanism to simulate named dimensions. It does this by adding a per-variable attribute called _ARRAY_DIMENSIONS. This attribute contains a list of names to be matched against the shape values of the variable. In effect a named dimension is created with the name _ARRAY_DIMENSIONS(i) and length shape(i) for all i in range 0..rank(variable). Both read and write support is provided. This XArray support is only invoked if the mode value of "xarray" is defined. So for example, as in this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,xarray,s3 ```` Note that the "xarray" mode flag also implies mode flag "zarr", so the above is equivalent to this URL. ```` https://s3.us-west-1.amazonaws.com/bucket/dataset#mode=nczarr,zarr,xarray,s3 ```` The primary change to implement this was to unify the handling of dimension references in libnczarr/zsync. A test for this and other pure-zarr features was added as nczarr_test/run_purezarr.sh Other changes: * Make sure distcheck leaves no files around. * Change the special attribute flag DIMSCALEFLAG to HIDDENATTRFLAG to support the xarray attribute. * Annotate the zmap implementations with feature flags such as WRITEONCE (for zip files).
2021-02-25 04:46:11 +08:00
# undef YYSTACK_RELOCATE
More tweaking.
2020-12-08 05:45:14 +08:00
if (yyss1 != yyssa)
YYSTACK_FREE (yyss1);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
# endif
yyssp = yyss + yysize - 1;
yyvsp = yyvs + yysize - 1;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YY_IGNORE_USELESS_CAST_BEGIN
YYDPRINTF ((stderr, "Stack size increased to %ld\n",
YY_CAST (long, yystacksize)));
YY_IGNORE_USELESS_CAST_END
moving to trunk subdir
2010-06-03 21:24:43 +08:00
if (yyss + yystacksize - 1 <= yyssp)
More tweaking.
2020-12-08 05:45:14 +08:00
YYABORT;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#endif /* !defined yyoverflow && !defined YYSTACK_RELOCATE */
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
if (yystate == YYFINAL)
YYACCEPT;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
goto yybackup;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/*-----------.
| yybackup. |
`-----------*/
yybackup:
/* Do appropriate processing given the current state. Read a
More tweaking.
2020-12-08 05:45:14 +08:00
lookahead token if we need one and don't already have one. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
/* First try to decide what to do without reference to lookahead token. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yyn = yypact[yystate];
More tweaking.
2020-12-08 05:45:14 +08:00
if (yypact_value_is_default (yyn))
moving to trunk subdir
2010-06-03 21:24:43 +08:00
goto yydefault;
More tweaking.
2020-12-08 05:45:14 +08:00
/* Not known => get a lookahead token if don't already have one. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* YYCHAR is either empty, or end-of-input, or a valid lookahead. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
if (yychar == YYEMPTY)
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YYDPRINTF ((stderr, "Reading a token\n"));
More tweaking.
2020-12-08 05:45:14 +08:00
yychar = yylex ();
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
if (yychar <= YYEOF)
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yychar = YYEOF;
yytoken = YYSYMBOL_YYEOF;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
YYDPRINTF ((stderr, "Now at end of input.\n"));
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
else if (yychar == YYerror)
{
/* The scanner already issued an error message, process directly
to error recovery. But do not keep the error token as
lookahead, it is too special and may lead us to an endless
loop in error recovery. */
yychar = YYUNDEF;
yytoken = YYSYMBOL_YYerror;
goto yyerrlab1;
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
else
{
yytoken = YYTRANSLATE (yychar);
YY_SYMBOL_PRINT ("Next token is", yytoken, &yylval, &yylloc);
}
/* If the proper action on seeing token YYTOKEN is to reduce or to
detect an error, take that action. */
yyn += yytoken;
if (yyn < 0 || YYLAST < yyn || yycheck[yyn] != yytoken)
goto yydefault;
yyn = yytable[yyn];
if (yyn <= 0)
{
More tweaking.
2020-12-08 05:45:14 +08:00
if (yytable_value_is_error (yyn))
goto yyerrlab;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yyn = -yyn;
goto yyreduce;
}
/* Count tokens shifted since error; after three, turn off error
status. */
if (yyerrstatus)
yyerrstatus--;
More tweaking.
2020-12-08 05:45:14 +08:00
/* Shift the lookahead token. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
YY_SYMBOL_PRINT ("Shifting", yytoken, &yylval, &yylloc);
yystate = yyn;
More tweaking.
2020-12-08 05:45:14 +08:00
YY_IGNORE_MAYBE_UNINITIALIZED_BEGIN
moving to trunk subdir
2010-06-03 21:24:43 +08:00
*++yyvsp = yylval;
More tweaking.
2020-12-08 05:45:14 +08:00
YY_IGNORE_MAYBE_UNINITIALIZED_END
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Discard the shifted token. */
yychar = YYEMPTY;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
goto yynewstate;
/*-----------------------------------------------------------.
| yydefault -- do the default action for the current state. |
`-----------------------------------------------------------*/
yydefault:
yyn = yydefact[yystate];
if (yyn == 0)
goto yyerrlab;
goto yyreduce;
/*-----------------------------.
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
| yyreduce -- do a reduction. |
moving to trunk subdir
2010-06-03 21:24:43 +08:00
`-----------------------------*/
yyreduce:
/* yyn is the number of a rule to reduce with. */
yylen = yyr2[yyn];
/* If YYLEN is nonzero, implement the default value of the action:
More tweaking.
2020-12-08 05:45:14 +08:00
'$$ = $1'.
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Otherwise, the following line sets YYVAL to garbage.
This behavior is undocumented and Bison
users should not rely upon it. Assigning to YYVAL
unconditionally makes the parser a bit smaller, and it avoids a
GCC warning that YYVAL may be used uninitialized. */
yyval = yyvsp[1-yylen];
YY_REDUCE_PRINT (yyn);
switch (yyn)
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 2: /* ncdesc: NETCDF datasetid rootgroup */
#line 245 "ncgen.y"
{if (error_count > 0) YYABORT;}
#line 1867 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 3: /* datasetid: DATASETID */
#line 248 "ncgen.y"
{createrootgroup(datasetname);}
#line 1873 "ncgeny.c"
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 8: /* $@1: %empty */
#line 267 "ncgen.y"
{
More tweaking.
2020-12-08 05:45:14 +08:00
Symbol* id = (yyvsp[-1].sym);
Fixed the following Jira minor bugs: NCF-42: _Format attribute sometimes being ignored NCF-43: Fixed unsigned long long parsing. NCF-47: Make opendap code properly handle illegal names like "x.y" by supressing them NCF-49: check for uint type NCF-50: properly handle username:pwd embedded in urls.
2011-05-13 01:51:32 +08:00
markcdf4("Group specification");
Addressed UNIX API issues reported by clang.
2015-06-19 04:31:10 +08:00
if(creategroup(id) == NULL)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yyerror("duplicate group declaration within parent group for %s",
disallow uppercase section keywords
2010-08-26 03:01:07 +08:00
id->name);
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 1885 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 9: /* $@2: %empty */
#line 276 "ncgen.y"
{listpop(groupstack);}
#line 1891 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 12: /* typesection: TYPES */
#line 282 "ncgen.y"
{}
#line 1897 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 13: /* typesection: TYPES typedecls */
#line 284 "ncgen.y"
{markcdf4("Type specification");}
#line 1903 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 16: /* typename: ident */
#line 290 "ncgen.y"
{ /* Use when defining a type */
More tweaking.
2020-12-08 05:45:14 +08:00
(yyvsp[0].sym)->objectclass = NC_TYPE;
if(dupobjectcheck(NC_TYPE,(yyvsp[0].sym)))
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yyerror("duplicate type declaration for %s",
More tweaking.
2020-12-08 05:45:14 +08:00
(yyvsp[0].sym)->name);
listpush(typdefs,(void*)(yyvsp[0].sym));
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 1915 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 17: /* type_or_attr_decl: typedecl */
#line 299 "ncgen.y"
{}
#line 1921 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 18: /* type_or_attr_decl: attrdecl ';' */
#line 299 "ncgen.y"
{}
#line 1927 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 25: /* enumdecl: primtype ENUM typename '{' enumidlist '}' */
#line 313 "ncgen.y"
{
moving to trunk subdir
2010-06-03 21:24:43 +08:00
int i;
More tweaking.
2020-12-08 05:45:14 +08:00
addtogroup((yyvsp[-3].sym)); /* sets prefix*/
(yyvsp[-3].sym)->objectclass=NC_TYPE;
(yyvsp[-3].sym)->subclass=NC_ENUM;
(yyvsp[-3].sym)->typ.basetype=(yyvsp[-5].sym);
(yyvsp[-3].sym)->typ.size = (yyvsp[-5].sym)->typ.size;
(yyvsp[-3].sym)->typ.alignment = (yyvsp[-5].sym)->typ.alignment;
stackbase=(yyvsp[-1].mark);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
stacklen=listlength(stack);
More tweaking.
2020-12-08 05:45:14 +08:00
(yyvsp[-3].sym)->subnodes = listnew();
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Variety of field fixups*/
/* 1. add in the enum values*/
/* 2. make this type be their container*/
/* 3. make constant names visible in the group*/
/* 4. set field basetype to be same as enum basetype*/
for(i=stackbase;i<stacklen;i++) {
Symbol* eid = (Symbol*)listget(stack,i);
assert(eid->subclass == NC_ECONST);
addtogroup(eid);
More tweaking.
2020-12-08 05:45:14 +08:00
listpush((yyvsp[-3].sym)->subnodes,(void*)eid);
eid->container = (yyvsp[-3].sym);
eid->typ.basetype = (yyvsp[-3].sym)->typ.basetype;
Addressed UNIX API issues reported by clang.
2015-06-19 04:31:10 +08:00
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
listsetlength(stack,stackbase);/* remove stack nodes*/
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 1958 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 26: /* enumidlist: enumid */
#line 342 "ncgen.y"
{(yyval.mark)=listlength(stack); listpush(stack,(void*)(yyvsp[0].sym));}
#line 1964 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 27: /* enumidlist: enumidlist ',' enumid */
#line 344 "ncgen.y"
{
moving to trunk subdir
2010-06-03 21:24:43 +08:00
int i;
More tweaking.
2020-12-08 05:45:14 +08:00
(yyval.mark)=(yyvsp[-2].mark);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* check for duplicates*/
More tweaking.
2020-12-08 05:45:14 +08:00
stackbase=(yyvsp[-2].mark);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
stacklen=listlength(stack);
for(i=stackbase;i<stacklen;i++) {
Symbol* elem = (Symbol*)listget(stack,i);
More tweaking.
2020-12-08 05:45:14 +08:00
if(strcmp((yyvsp[0].sym)->name,elem->name)==0)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yyerror("duplicate enum declaration for %s",
elem->name);
Addressed UNIX API issues reported by clang.
2015-06-19 04:31:10 +08:00
}
More tweaking.
2020-12-08 05:45:14 +08:00
listpush(stack,(void*)(yyvsp[0].sym));
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 1983 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 28: /* enumid: ident '=' constint */
#line 361 "ncgen.y"
{
More tweaking.
2020-12-08 05:45:14 +08:00
(yyvsp[-2].sym)->objectclass=NC_TYPE;
(yyvsp[-2].sym)->subclass=NC_ECONST;
(yyvsp[-2].sym)->typ.econst=(yyvsp[0].constant);
(yyval.sym)=(yyvsp[-2].sym);
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 1994 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 29: /* opaquedecl: OPAQUE_ '(' INT_CONST ')' typename */
#line 370 "ncgen.y"
{
Fixed the following Jira minor bugs: NCF-42: _Format attribute sometimes being ignored NCF-43: Fixed unsigned long long parsing. NCF-47: Make opendap code properly handle illegal names like "x.y" by supressing them NCF-49: check for uint type NCF-50: properly handle username:pwd embedded in urls.
2011-05-13 01:51:32 +08:00
vercheck(NC_OPAQUE);
More tweaking.
2020-12-08 05:45:14 +08:00
addtogroup((yyvsp[0].sym)); /*sets prefix*/
(yyvsp[0].sym)->objectclass=NC_TYPE;
(yyvsp[0].sym)->subclass=NC_OPAQUE;
(yyvsp[0].sym)->typ.typecode=NC_OPAQUE;
(yyvsp[0].sym)->typ.size=int32_val;
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
(void)ncaux_class_alignment(NC_OPAQUE,&(yyvsp[0].sym)->typ.alignment);
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2008 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 30: /* vlendecl: typeref '(' '*' ')' typename */
#line 382 "ncgen.y"
{
More tweaking.
2020-12-08 05:45:14 +08:00
Symbol* basetype = (yyvsp[-4].sym);
Fixed the following Jira minor bugs: NCF-42: _Format attribute sometimes being ignored NCF-43: Fixed unsigned long long parsing. NCF-47: Make opendap code properly handle illegal names like "x.y" by supressing them NCF-49: check for uint type NCF-50: properly handle username:pwd embedded in urls.
2011-05-13 01:51:32 +08:00
vercheck(NC_VLEN);
More tweaking.
2020-12-08 05:45:14 +08:00
addtogroup((yyvsp[0].sym)); /*sets prefix*/
(yyvsp[0].sym)->objectclass=NC_TYPE;
(yyvsp[0].sym)->subclass=NC_VLEN;
(yyvsp[0].sym)->typ.basetype=basetype;
(yyvsp[0].sym)->typ.typecode=NC_VLEN;
(yyvsp[0].sym)->typ.size=VLENSIZE;
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
(void)ncaux_class_alignment(NC_VLEN,&(yyvsp[0].sym)->typ.alignment);
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2024 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 31: /* compounddecl: COMPOUND typename '{' fields '}' */
#line 396 "ncgen.y"
{
moving to trunk subdir
2010-06-03 21:24:43 +08:00
int i,j;
Fixed the following Jira minor bugs: NCF-42: _Format attribute sometimes being ignored NCF-43: Fixed unsigned long long parsing. NCF-47: Make opendap code properly handle illegal names like "x.y" by supressing them NCF-49: check for uint type NCF-50: properly handle username:pwd embedded in urls.
2011-05-13 01:51:32 +08:00
vercheck(NC_COMPOUND);
More tweaking.
2020-12-08 05:45:14 +08:00
addtogroup((yyvsp[-3].sym));
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* check for duplicate field names*/
More tweaking.
2020-12-08 05:45:14 +08:00
stackbase=(yyvsp[-1].mark);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
stacklen=listlength(stack);
for(i=stackbase;i<stacklen;i++) {
Symbol* elem1 = (Symbol*)listget(stack,i);
for(j=i+1;j<stacklen;j++) {
Symbol* elem2 = (Symbol*)listget(stack,j);
if(strcmp(elem1->name,elem2->name)==0) {
yyerror("duplicate field declaration for %s",elem1->name);
}
}
}
More tweaking.
2020-12-08 05:45:14 +08:00
(yyvsp[-3].sym)->objectclass=NC_TYPE;
(yyvsp[-3].sym)->subclass=NC_COMPOUND;
(yyvsp[-3].sym)->typ.basetype=NULL;
(yyvsp[-3].sym)->typ.typecode=NC_COMPOUND;
(yyvsp[-3].sym)->subnodes = listnew();
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Add in the fields*/
for(i=stackbase;i<stacklen;i++) {
Symbol* fsym = (Symbol*)listget(stack,i);
More tweaking.
2020-12-08 05:45:14 +08:00
fsym->container = (yyvsp[-3].sym);
listpush((yyvsp[-3].sym)->subnodes,(void*)fsym);
Addressed UNIX API issues reported by clang.
2015-06-19 04:31:10 +08:00
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
listsetlength(stack,stackbase);/* remove stack nodes*/
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2058 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 32: /* fields: field ';' */
#line 428 "ncgen.y"
{(yyval.mark)=(yyvsp[-1].mark);}
#line 2064 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 33: /* fields: fields field ';' */
#line 429 "ncgen.y"
{(yyval.mark)=(yyvsp[-2].mark);}
#line 2070 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 34: /* field: typeref fieldlist */
#line 433 "ncgen.y"
{
moving to trunk subdir
2010-06-03 21:24:43 +08:00
int i;
More tweaking.
2020-12-08 05:45:14 +08:00
(yyval.mark)=(yyvsp[0].mark);
stackbase=(yyvsp[0].mark);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
stacklen=listlength(stack);
/* process each field in the fieldlist*/
for(i=stackbase;i<stacklen;i++) {
Symbol* f = (Symbol*)listget(stack,i);
More tweaking.
2020-12-08 05:45:14 +08:00
f->typ.basetype = (yyvsp[-1].sym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2086 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 35: /* primtype: CHAR_K */
#line 446 "ncgen.y"
{ (yyval.sym) = primsymbols[NC_CHAR]; }
#line 2092 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 36: /* primtype: BYTE_K */
#line 447 "ncgen.y"
{ (yyval.sym) = primsymbols[NC_BYTE]; }
#line 2098 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 37: /* primtype: SHORT_K */
#line 448 "ncgen.y"
{ (yyval.sym) = primsymbols[NC_SHORT]; }
#line 2104 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 38: /* primtype: INT_K */
#line 449 "ncgen.y"
{ (yyval.sym) = primsymbols[NC_INT]; }
#line 2110 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 39: /* primtype: FLOAT_K */
#line 450 "ncgen.y"
{ (yyval.sym) = primsymbols[NC_FLOAT]; }
#line 2116 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 40: /* primtype: DOUBLE_K */
#line 451 "ncgen.y"
{ (yyval.sym) = primsymbols[NC_DOUBLE]; }
#line 2122 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 41: /* primtype: UBYTE_K */
#line 452 "ncgen.y"
{ vercheck(NC_UBYTE); (yyval.sym) = primsymbols[NC_UBYTE]; }
#line 2128 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 42: /* primtype: USHORT_K */
#line 453 "ncgen.y"
{ vercheck(NC_USHORT); (yyval.sym) = primsymbols[NC_USHORT]; }
#line 2134 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 43: /* primtype: UINT_K */
#line 454 "ncgen.y"
{ vercheck(NC_UINT); (yyval.sym) = primsymbols[NC_UINT]; }
#line 2140 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 44: /* primtype: INT64_K */
#line 455 "ncgen.y"
{ vercheck(NC_INT64); (yyval.sym) = primsymbols[NC_INT64]; }
#line 2146 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 45: /* primtype: UINT64_K */
#line 456 "ncgen.y"
{ vercheck(NC_UINT64); (yyval.sym) = primsymbols[NC_UINT64]; }
#line 2152 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 46: /* primtype: STRING_K */
#line 457 "ncgen.y"
{ vercheck(NC_STRING); (yyval.sym) = primsymbols[NC_STRING]; }
#line 2158 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 48: /* dimsection: DIMENSIONS */
#line 461 "ncgen.y"
{}
#line 2164 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 49: /* dimsection: DIMENSIONS dimdecls */
#line 462 "ncgen.y"
{}
#line 2170 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 52: /* dim_or_attr_decl: dimdeclist */
#line 469 "ncgen.y"
{}
#line 2176 "ncgeny.c"
Finalize the compression support. This relies on the HDF5 capability to dynamically load compression filters. Note that a compression filter is just a subcase of filters. The primary user-visible changes are as follows: 1. Add a standard header "netcdf_filter.h" that defines the necessary API extensions 2. Modify ncgen to support two new special attributes "_Filter_ID" and "_Filter_Parameters" so that compression can be turned on when creating a file using ncgen. 4. Add a detailed description of filtering support to the user's guide; see the file filters.md 5. Add a test case directory for this: nc_test4/filter_test. It is fragile and a ./configure flags (-enable-filter-test) is defined (default disabled) to shut this off this test to avoid spurious 'make check' failures. Note that the HDF5 documentation is not up-to-date, so much of what is encoded here comes from examining the actual code in the file H5PL.c in the HDF5 source code.
2017-04-28 03:01:59 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 53: /* dim_or_attr_decl: attrdecl */
#line 469 "ncgen.y"
{}
#line 2182 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 56: /* dimdecl: dimd '=' constint */
#line 477 "ncgen.y"
{
More tweaking.
2020-12-08 05:45:14 +08:00
(yyvsp[-2].sym)->dim.declsize = (size_t)extractint((yyvsp[0].constant));
1. [NCF-276]/XXI-796914 Columbia server does not serve up proper opendap DDS replies. The Dataset {...} name changes depending on if the request has certain kinds of constraints. Code for a hack was not being used, so restore it. The fix is to effectively ignore differences in Dataset node names if the code is coming from columbia.edu. 2. [NCF-278] The ncgen code is improperly typing int64 integer constants as uint64. 3. [NCF-279] Empty string constants were not being properly filled when their target array is length 1 or more.
2013-11-18 05:26:14 +08:00
#ifdef GENDEBUG1
More tweaking.
2020-12-08 05:45:14 +08:00
fprintf(stderr,"dimension: %s = %llu\n",(yyvsp[-2].sym)->name,(unsigned long long)(yyvsp[-2].sym)->dim.declsize);
Resolve Lynton's vlen problems (partially) re: jira NCF-145
2012-02-14 08:25:32 +08:00
#endif
More tweaking.
2020-12-08 05:45:14 +08:00
reclaimconstant((yyvsp[0].constant));
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2194 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 57: /* dimdecl: dimd '=' NC_UNLIMITED_K */
#line 485 "ncgen.y"
{
More tweaking.
2020-12-08 05:45:14 +08:00
(yyvsp[-2].sym)->dim.declsize = NC_UNLIMITED;
(yyvsp[-2].sym)->dim.isunlimited = 1;
1. [NCF-276]/XXI-796914 Columbia server does not serve up proper opendap DDS replies. The Dataset {...} name changes depending on if the request has certain kinds of constraints. Code for a hack was not being used, so restore it. The fix is to effectively ignore differences in Dataset node names if the code is coming from columbia.edu. 2. [NCF-278] The ncgen code is improperly typing int64 integer constants as uint64. 3. [NCF-279] Empty string constants were not being properly filled when their target array is length 1 or more.
2013-11-18 05:26:14 +08:00
#ifdef GENDEBUG1
More tweaking.
2020-12-08 05:45:14 +08:00
fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name);
Resolve Lynton's vlen problems (partially) re: jira NCF-145
2012-02-14 08:25:32 +08:00
#endif
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2206 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 58: /* dimd: ident */
#line 495 "ncgen.y"
{
More tweaking.
2020-12-08 05:45:14 +08:00
(yyvsp[0].sym)->objectclass=NC_DIM;
if(dupobjectcheck(NC_DIM,(yyvsp[0].sym)))
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yyerror( "Duplicate dimension declaration for %s",
More tweaking.
2020-12-08 05:45:14 +08:00
(yyvsp[0].sym)->name);
addtogroup((yyvsp[0].sym));
(yyval.sym)=(yyvsp[0].sym);
listpush(dimdefs,(void*)(yyvsp[0].sym));
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2220 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 60: /* vasection: VARIABLES */
#line 507 "ncgen.y"
{}
#line 2226 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 61: /* vasection: VARIABLES vadecls */
#line 508 "ncgen.y"
{}
#line 2232 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 64: /* vadecl_or_attr: vardecl */
#line 515 "ncgen.y"
{}
#line 2238 "ncgeny.c"
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 65: /* vadecl_or_attr: attrdecl */
#line 515 "ncgen.y"
{}
#line 2244 "ncgeny.c"
Finalize the compression support. This relies on the HDF5 capability to dynamically load compression filters. Note that a compression filter is just a subcase of filters. The primary user-visible changes are as follows: 1. Add a standard header "netcdf_filter.h" that defines the necessary API extensions 2. Modify ncgen to support two new special attributes "_Filter_ID" and "_Filter_Parameters" so that compression can be turned on when creating a file using ncgen. 4. Add a detailed description of filtering support to the user's guide; see the file filters.md 5. Add a test case directory for this: nc_test4/filter_test. It is fragile and a ./configure flags (-enable-filter-test) is defined (default disabled) to shut this off this test to avoid spurious 'make check' failures. Note that the HDF5 documentation is not up-to-date, so much of what is encoded here comes from examining the actual code in the file H5PL.c in the HDF5 source code.
2017-04-28 03:01:59 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 66: /* vardecl: typeref varlist */
#line 518 "ncgen.y"
{
moving to trunk subdir
2010-06-03 21:24:43 +08:00
int i;
More tweaking.
2020-12-08 05:45:14 +08:00
stackbase=(yyvsp[0].mark);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
stacklen=listlength(stack);
/* process each variable in the varlist*/
for(i=stackbase;i<stacklen;i++) {
Symbol* sym = (Symbol*)listget(stack,i);
sym->objectclass = NC_VAR;
if(dupobjectcheck(NC_VAR,sym)) {
yyerror("Duplicate variable declaration for %s",
sym->name);
} else {
More tweaking.
2020-12-08 05:45:14 +08:00
sym->typ.basetype = (yyvsp[-1].sym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
addtogroup(sym);
update utilities: list,byte,hash,uri,log
2012-08-20 05:54:30 +08:00
listpush(vardefs,(void*)sym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
}
listsetlength(stack,stackbase);/* remove stack nodes*/
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2268 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 67: /* varlist: varspec */
#line 540 "ncgen.y"
{(yyval.mark)=listlength(stack);
More tweaking.
2020-12-08 05:45:14 +08:00
listpush(stack,(void*)(yyvsp[0].sym));
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2276 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 68: /* varlist: varlist ',' varspec */
#line 544 "ncgen.y"
{(yyval.mark)=(yyvsp[-2].mark); listpush(stack,(void*)(yyvsp[0].sym));}
#line 2282 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 69: /* varspec: varident dimspec */
#line 548 "ncgen.y"
{
moving to trunk subdir
2010-06-03 21:24:43 +08:00
int i;
Dimset dimset;
More tweaking.
2020-12-08 05:45:14 +08:00
Symbol* var = (yyvsp[-1].sym); /* for debugging */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
stacklen=listlength(stack);
More tweaking.
2020-12-08 05:45:14 +08:00
stackbase=(yyvsp[0].mark);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
count = stacklen - stackbase;
if(count >= NC_MAX_VAR_DIMS) {
More tweaking.
2020-12-08 05:45:14 +08:00
yyerror("%s has too many dimensions",(yyvsp[-1].sym)->name);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
count = NC_MAX_VAR_DIMS - 1;
stacklen = stackbase + count;
}
dimset.ndims = count;
/* extract the actual dimensions*/
if(dimset.ndims > 0) {
for(i=0;i<count;i++) {
Symbol* dsym = (Symbol*)listget(stack,stackbase+i);
dimset.dimsyms[i] = dsym;
}
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
var->typ.dimset = dimset;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
var->typ.basetype = NULL; /* not yet known*/
var->objectclass=NC_VAR;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
listsetlength(stack,stackbase);/* remove stack nodes*/
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
(yyval.sym) = var;
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2313 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 70: /* dimspec: %empty */
#line 576 "ncgen.y"
{(yyval.mark)=listlength(stack);}
#line 2319 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 71: /* dimspec: '(' dimlist ')' */
#line 577 "ncgen.y"
{(yyval.mark)=(yyvsp[-1].mark);}
#line 2325 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 72: /* dimlist: dimref */
#line 580 "ncgen.y"
{(yyval.mark)=listlength(stack); listpush(stack,(void*)(yyvsp[0].sym));}
#line 2331 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 73: /* dimlist: dimlist ',' dimref */
#line 582 "ncgen.y"
{(yyval.mark)=(yyvsp[-2].mark); listpush(stack,(void*)(yyvsp[0].sym));}
#line 2337 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 74: /* dimref: path */
#line 586 "ncgen.y"
{Symbol* dimsym = (yyvsp[0].sym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
dimsym->objectclass = NC_DIM;
/* Find the actual dimension*/
dimsym = locate(dimsym);
if(dimsym == NULL) {
More tweaking.
2020-12-08 05:45:14 +08:00
derror("Undefined or forward referenced dimension: %s",(yyvsp[0].sym)->name);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
YYABORT;
}
(yyval.sym)=dimsym;
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2352 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 75: /* fieldlist: fieldspec */
#line 600 "ncgen.y"
{(yyval.mark)=listlength(stack);
More tweaking.
2020-12-08 05:45:14 +08:00
listpush(stack,(void*)(yyvsp[0].sym));
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2360 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 76: /* fieldlist: fieldlist ',' fieldspec */
#line 604 "ncgen.y"
{(yyval.mark)=(yyvsp[-2].mark); listpush(stack,(void*)(yyvsp[0].sym));}
#line 2366 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 77: /* fieldspec: ident fielddimspec */
#line 609 "ncgen.y"
{
moving to trunk subdir
2010-06-03 21:24:43 +08:00
int i;
Dimset dimset;
More tweaking.
2020-12-08 05:45:14 +08:00
stackbase=(yyvsp[0].mark);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
stacklen=listlength(stack);
count = stacklen - stackbase;
if(count >= NC_MAX_VAR_DIMS) {
More tweaking.
2020-12-08 05:45:14 +08:00
yyerror("%s has too many dimensions",(yyvsp[-1].sym)->name);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
count = NC_MAX_VAR_DIMS - 1;
stacklen = stackbase + count;
}
dimset.ndims = count;
if(count > 0) {
/* extract the actual dimensions*/
for(i=0;i<count;i++) {
Symbol* dsym = (Symbol*)listget(stack,stackbase+i);
dimset.dimsyms[i] = dsym;
}
More tweaking.
2020-12-08 05:45:14 +08:00
(yyvsp[-1].sym)->typ.dimset = dimset;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
More tweaking.
2020-12-08 05:45:14 +08:00
(yyvsp[-1].sym)->typ.basetype = NULL; /* not yet known*/
(yyvsp[-1].sym)->objectclass=NC_TYPE;
(yyvsp[-1].sym)->subclass=NC_FIELD;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
listsetlength(stack,stackbase);/* remove stack nodes*/
More tweaking.
2020-12-08 05:45:14 +08:00
(yyval.sym) = (yyvsp[-1].sym);
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2397 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 78: /* fielddimspec: %empty */
#line 637 "ncgen.y"
{(yyval.mark)=listlength(stack);}
#line 2403 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 79: /* fielddimspec: '(' fielddimlist ')' */
#line 638 "ncgen.y"
{(yyval.mark)=(yyvsp[-1].mark);}
#line 2409 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 80: /* fielddimlist: fielddim */
#line 642 "ncgen.y"
{(yyval.mark)=listlength(stack); listpush(stack,(void*)(yyvsp[0].sym));}
#line 2415 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 81: /* fielddimlist: fielddimlist ',' fielddim */
#line 644 "ncgen.y"
{(yyval.mark)=(yyvsp[-2].mark); listpush(stack,(void*)(yyvsp[0].sym));}
#line 2421 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 82: /* fielddim: UINT_CONST */
#line 649 "ncgen.y"
{ /* Anonymous integer dimension.
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Can only occur in type definitions*/
char anon[32];
sprintf(anon,"const%u",uint32_val);
(yyval.sym) = install(anon);
(yyval.sym)->objectclass = NC_DIM;
(yyval.sym)->dim.isconstant = 1;
(yyval.sym)->dim.declsize = uint32_val;
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2435 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 83: /* fielddim: INT_CONST */
#line 659 "ncgen.y"
{ /* Anonymous integer dimension.
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Can only occur in type definitions*/
char anon[32];
if(int32_val <= 0) {
derror("field dimension must be positive");
YYABORT;
}
sprintf(anon,"const%d",int32_val);
(yyval.sym) = install(anon);
(yyval.sym)->objectclass = NC_DIM;
(yyval.sym)->dim.isconstant = 1;
(yyval.sym)->dim.declsize = int32_val;
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2453 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 84: /* varref: ambiguous_ref */
#line 679 "ncgen.y"
{Symbol* vsym = (yyvsp[0].sym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
if(vsym->objectclass != NC_VAR) {
derror("Undefined or forward referenced variable: %s",vsym->name);
YYABORT;
}
(yyval.sym)=vsym;
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2465 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 85: /* typeref: ambiguous_ref */
#line 690 "ncgen.y"
{Symbol* tsym = (yyvsp[0].sym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
if(tsym->objectclass != NC_TYPE) {
derror("Undefined or forward referenced type: %s",tsym->name);
YYABORT;
}
(yyval.sym)=tsym;
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2477 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 86: /* ambiguous_ref: path */
#line 701 "ncgen.y"
{Symbol* tvsym = (yyvsp[0].sym); Symbol* sym;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* disambiguate*/
tvsym->objectclass = NC_VAR;
sym = locate(tvsym);
if(sym == NULL) {
tvsym->objectclass = NC_TYPE;
sym = locate(tvsym);
if(tvsym == NULL) {
More tweaking.
2020-12-08 05:45:14 +08:00
derror("Undefined or forward referenced name: %s",(yyvsp[0].sym)->name);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
YYABORT;
} else tvsym = sym;
} else tvsym = sym;
if(tvsym == NULL) {
More tweaking.
2020-12-08 05:45:14 +08:00
derror("Undefined name (line %d): %s",(yyvsp[0].sym)->lineno,(yyvsp[0].sym)->name);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
YYABORT;
}
(yyval.sym)=tvsym;
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2500 "ncgeny.c"
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 87: /* ambiguous_ref: primtype */
#line 719 "ncgen.y"
{(yyval.sym)=(yyvsp[0].sym);}
#line 2506 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 88: /* attrdecllist: %empty */
#line 726 "ncgen.y"
{}
#line 2512 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 89: /* attrdecllist: attrdecl ';' attrdecllist */
#line 726 "ncgen.y"
{}
#line 2518 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 90: /* attrdecl: ':' _NCPROPS '=' conststring */
#line 730 "ncgen.y"
{(yyval.sym) = makespecial(_NCPROPS_FLAG,NULL,NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2524 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 91: /* attrdecl: ':' _ISNETCDF4 '=' constbool */
#line 732 "ncgen.y"
{(yyval.sym) = makespecial(_ISNETCDF4_FLAG,NULL,NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2530 "ncgeny.c"
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 92: /* attrdecl: ':' _SUPERBLOCK '=' constint */
#line 734 "ncgen.y"
{(yyval.sym) = makespecial(_SUPERBLOCK_FLAG,NULL,NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2536 "ncgeny.c"
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 93: /* attrdecl: ':' ident '=' datalist */
#line 736 "ncgen.y"
{ (yyval.sym)=makeattribute((yyvsp[-2].sym),NULL,NULL,(yyvsp[0].datalist),ATTRGLOBAL);}
#line 2542 "ncgeny.c"
Finalize the compression support. This relies on the HDF5 capability to dynamically load compression filters. Note that a compression filter is just a subcase of filters. The primary user-visible changes are as follows: 1. Add a standard header "netcdf_filter.h" that defines the necessary API extensions 2. Modify ncgen to support two new special attributes "_Filter_ID" and "_Filter_Parameters" so that compression can be turned on when creating a file using ncgen. 4. Add a detailed description of filtering support to the user's guide; see the file filters.md 5. Add a test case directory for this: nc_test4/filter_test. It is fragile and a ./configure flags (-enable-filter-test) is defined (default disabled) to shut this off this test to avoid spurious 'make check' failures. Note that the HDF5 documentation is not up-to-date, so much of what is encoded here comes from examining the actual code in the file H5PL.c in the HDF5 source code.
2017-04-28 03:01:59 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 94: /* attrdecl: typeref ambiguous_ref ':' ident '=' datalist */
#line 738 "ncgen.y"
{Symbol* tsym = (yyvsp[-5].sym); Symbol* vsym = (yyvsp[-4].sym); Symbol* asym = (yyvsp[-2].sym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
if(vsym->objectclass == NC_VAR) {
More tweaking.
2020-12-08 05:45:14 +08:00
(yyval.sym)=makeattribute(asym,vsym,tsym,(yyvsp[0].datalist),ATTRVAR);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
} else {
derror("Doubly typed attribute: %s",asym->name);
YYABORT;
}
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2555 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 95: /* attrdecl: ambiguous_ref ':' ident '=' datalist */
#line 747 "ncgen.y"
{Symbol* sym = (yyvsp[-4].sym); Symbol* asym = (yyvsp[-2].sym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
if(sym->objectclass == NC_VAR) {
More tweaking.
2020-12-08 05:45:14 +08:00
(yyval.sym)=makeattribute(asym,sym,NULL,(yyvsp[0].datalist),ATTRVAR);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
} else if(sym->objectclass == NC_TYPE) {
More tweaking.
2020-12-08 05:45:14 +08:00
(yyval.sym)=makeattribute(asym,NULL,sym,(yyvsp[0].datalist),ATTRGLOBAL);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
} else {
derror("Attribute prefix not a variable or type: %s",asym->name);
YYABORT;
}
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2570 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 96: /* attrdecl: ambiguous_ref ':' _FILLVALUE '=' datalist */
#line 758 "ncgen.y"
{(yyval.sym) = makespecial(_FILLVALUE_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].datalist),ISLIST);}
#line 2576 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 97: /* attrdecl: typeref ambiguous_ref ':' _FILLVALUE '=' datalist */
#line 760 "ncgen.y"
{(yyval.sym) = makespecial(_FILLVALUE_FLAG,(yyvsp[-4].sym),(yyvsp[-5].sym),(void*)(yyvsp[0].datalist),ISLIST);}
#line 2582 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 98: /* attrdecl: ambiguous_ref ':' _STORAGE '=' conststring */
#line 762 "ncgen.y"
{(yyval.sym) = makespecial(_STORAGE_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2588 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 99: /* attrdecl: ambiguous_ref ':' _CHUNKSIZES '=' intlist */
#line 764 "ncgen.y"
{(yyval.sym) = makespecial(_CHUNKSIZES_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].datalist),ISLIST);}
#line 2594 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 100: /* attrdecl: ambiguous_ref ':' _FLETCHER32 '=' constbool */
#line 766 "ncgen.y"
{(yyval.sym) = makespecial(_FLETCHER32_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2600 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 101: /* attrdecl: ambiguous_ref ':' _DEFLATELEVEL '=' constint */
#line 768 "ncgen.y"
{(yyval.sym) = makespecial(_DEFLATE_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2606 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 102: /* attrdecl: ambiguous_ref ':' _SHUFFLE '=' constbool */
#line 770 "ncgen.y"
{(yyval.sym) = makespecial(_SHUFFLE_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2612 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 103: /* attrdecl: ambiguous_ref ':' _ENDIANNESS '=' conststring */
#line 772 "ncgen.y"
{(yyval.sym) = makespecial(_ENDIAN_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2618 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 104: /* attrdecl: ambiguous_ref ':' _FILTER '=' conststring */
#line 774 "ncgen.y"
{(yyval.sym) = makespecial(_FILTER_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2624 "ncgeny.c"
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 105: /* attrdecl: ambiguous_ref ':' _CODECS '=' conststring */
#line 776 "ncgen.y"
{(yyval.sym) = makespecial(_CODECS_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2630 "ncgeny.c"
Finalize the compression support. This relies on the HDF5 capability to dynamically load compression filters. Note that a compression filter is just a subcase of filters. The primary user-visible changes are as follows: 1. Add a standard header "netcdf_filter.h" that defines the necessary API extensions 2. Modify ncgen to support two new special attributes "_Filter_ID" and "_Filter_Parameters" so that compression can be turned on when creating a file using ncgen. 4. Add a detailed description of filtering support to the user's guide; see the file filters.md 5. Add a test case directory for this: nc_test4/filter_test. It is fragile and a ./configure flags (-enable-filter-test) is defined (default disabled) to shut this off this test to avoid spurious 'make check' failures. Note that the HDF5 documentation is not up-to-date, so much of what is encoded here comes from examining the actual code in the file H5PL.c in the HDF5 source code.
2017-04-28 03:01:59 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 106: /* attrdecl: ambiguous_ref ':' _QUANTIZEBG '=' constint */
#line 778 "ncgen.y"
{(yyval.sym) = makespecial(_QUANTIZEBG_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2636 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 107: /* attrdecl: ambiguous_ref ':' _QUANTIZEBR '=' constint */
#line 780 "ncgen.y"
{(yyval.sym) = makespecial(_QUANTIZEBR_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2642 "ncgeny.c"
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 108: /* attrdecl: ambiguous_ref ':' _NOFILL '=' constbool */
#line 782 "ncgen.y"
{(yyval.sym) = makespecial(_NOFILL_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2648 "ncgeny.c"
break;
case 109: /* attrdecl: ':' _FORMAT '=' conststring */
#line 784 "ncgen.y"
{(yyval.sym) = makespecial(_FORMAT_FLAG,NULL,NULL,(void*)(yyvsp[0].constant),ISCONST);}
#line 2654 "ncgeny.c"
break;
case 110: /* path: ident */
#line 789 "ncgen.y"
{
More tweaking.
2020-12-08 05:45:14 +08:00
(yyval.sym)=(yyvsp[0].sym);
(yyvsp[0].sym)->ref.is_ref=1;
(yyvsp[0].sym)->is_prefixed=0;
setpathcurrent((yyvsp[0].sym));
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2665 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 111: /* path: PATH */
#line 796 "ncgen.y"
{
More tweaking.
2020-12-08 05:45:14 +08:00
(yyval.sym)=(yyvsp[0].sym);
(yyvsp[0].sym)->ref.is_ref=1;
(yyvsp[0].sym)->is_prefixed=1;
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
/* path is set in ncgen.l*/
More tweaking.
2020-12-08 05:45:14 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2676 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 113: /* datasection: DATA */
#line 805 "ncgen.y"
{}
#line 2682 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 114: /* datasection: DATA datadecls */
#line 806 "ncgen.y"
{}
#line 2688 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 117: /* datadecl: varref '=' datalist */
#line 814 "ncgen.y"
{(yyvsp[-2].sym)->data = (yyvsp[0].datalist);}
#line 2694 "ncgeny.c"
Add new capabilities to filter code: 1. Allow nccopy to apply filters, especially on the output file. This provides a third way to do this other than using ncgen or programatically 2. Make sure that even if the filter code is not available, it is possible to see the filter id and parameters for variables using e.g ncdump -hs. 3. Fix bug in nccopy so that the input file does not necessarily have to be netcdf-4. 4. At last minute decided to change to using a single "_Filter" attribute for ncgen 5. Added a test to tst_filter.sh to generate C code using ncgen.
2017-05-15 08:10:02 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 118: /* datalist: datalist0 */
#line 817 "ncgen.y"
{(yyval.datalist) = (yyvsp[0].datalist);}
#line 2700 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 119: /* datalist: datalist1 */
#line 818 "ncgen.y"
{(yyval.datalist) = (yyvsp[0].datalist);}
#line 2706 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 120: /* datalist0: %empty */
#line 822 "ncgen.y"
{(yyval.datalist) = builddatalist(0);}
#line 2712 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 121: /* datalist1: dataitem */
#line 826 "ncgen.y"
{(yyval.datalist) = const2list((yyvsp[0].constant));}
#line 2718 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 122: /* datalist1: datalist ',' dataitem */
#line 828 "ncgen.y"
{dlappend((yyvsp[-2].datalist),((yyvsp[0].constant))); (yyval.datalist)=(yyvsp[-2].datalist); }
#line 2724 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 123: /* dataitem: constdata */
#line 832 "ncgen.y"
{(yyval.constant)=(yyvsp[0].constant);}
#line 2730 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 124: /* dataitem: '{' datalist '}' */
#line 833 "ncgen.y"
{(yyval.constant)=builddatasublist((yyvsp[-1].datalist));}
#line 2736 "ncgeny.c"
Add new capabilities to filter code: 1. Allow nccopy to apply filters, especially on the output file. This provides a third way to do this other than using ncgen or programatically 2. Make sure that even if the filter code is not available, it is possible to see the filter id and parameters for variables using e.g ncdump -hs. 3. Fix bug in nccopy so that the input file does not necessarily have to be netcdf-4. 4. At last minute decided to change to using a single "_Filter" attribute for ncgen 5. Added a test to tst_filter.sh to generate C code using ncgen.
2017-05-15 08:10:02 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 125: /* constdata: simpleconstant */
#line 837 "ncgen.y"
{(yyval.constant)=(yyvsp[0].constant);}
#line 2742 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 126: /* constdata: OPAQUESTRING */
#line 838 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_OPAQUE);}
#line 2748 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 127: /* constdata: FILLMARKER */
#line 839 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_FILLVALUE);}
#line 2754 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 128: /* constdata: NIL */
#line 840 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_NIL);}
#line 2760 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 129: /* constdata: econstref */
#line 841 "ncgen.y"
{(yyval.constant)=(yyvsp[0].constant);}
#line 2766 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 131: /* econstref: path */
#line 846 "ncgen.y"
{(yyval.constant) = makeenumconstref((yyvsp[0].sym));}
#line 2772 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 132: /* function: ident '(' arglist ')' */
#line 850 "ncgen.y"
{(yyval.constant)=evaluate((yyvsp[-3].sym),(yyvsp[-1].datalist));}
#line 2778 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 133: /* arglist: simpleconstant */
#line 855 "ncgen.y"
{(yyval.datalist) = const2list((yyvsp[0].constant));}
#line 2784 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 134: /* arglist: arglist ',' simpleconstant */
#line 857 "ncgen.y"
{dlappend((yyvsp[-2].datalist),((yyvsp[0].constant))); (yyval.datalist)=(yyvsp[-2].datalist);}
#line 2790 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 135: /* simpleconstant: CHAR_CONST */
#line 861 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_CHAR);}
#line 2796 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 136: /* simpleconstant: BYTE_CONST */
#line 862 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_BYTE);}
#line 2802 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 137: /* simpleconstant: SHORT_CONST */
#line 863 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_SHORT);}
#line 2808 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 138: /* simpleconstant: INT_CONST */
#line 864 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_INT);}
#line 2814 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 139: /* simpleconstant: INT64_CONST */
#line 865 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_INT64);}
#line 2820 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 140: /* simpleconstant: UBYTE_CONST */
#line 866 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_UBYTE);}
#line 2826 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 141: /* simpleconstant: USHORT_CONST */
#line 867 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_USHORT);}
#line 2832 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 142: /* simpleconstant: UINT_CONST */
#line 868 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_UINT);}
#line 2838 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 143: /* simpleconstant: UINT64_CONST */
#line 869 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_UINT64);}
#line 2844 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 144: /* simpleconstant: FLOAT_CONST */
#line 870 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_FLOAT);}
#line 2850 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 145: /* simpleconstant: DOUBLE_CONST */
#line 871 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_DOUBLE);}
#line 2856 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 146: /* simpleconstant: TERMSTRING */
#line 872 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_STRING);}
#line 2862 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 147: /* intlist: constint */
#line 876 "ncgen.y"
{(yyval.datalist) = const2list((yyvsp[0].constant));}
#line 2868 "ncgeny.c"
fix NCF-120
2011-09-16 00:57:16 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 148: /* intlist: intlist ',' constint */
#line 877 "ncgen.y"
{(yyval.datalist)=(yyvsp[-2].datalist); dlappend((yyvsp[-2].datalist),((yyvsp[0].constant)));}
#line 2874 "ncgeny.c"
fix NCF-120
2011-09-16 00:57:16 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 149: /* constint: INT_CONST */
#line 882 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_INT);}
#line 2880 "ncgeny.c"
fix NCF-120
2011-09-16 00:57:16 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 150: /* constint: UINT_CONST */
#line 884 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_UINT);}
#line 2886 "ncgeny.c"
fix NCF-120
2011-09-16 00:57:16 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 151: /* constint: INT64_CONST */
#line 886 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_INT64);}
#line 2892 "ncgeny.c"
fix NCF-120
2011-09-16 00:57:16 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 152: /* constint: UINT64_CONST */
#line 888 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_UINT64);}
#line 2898 "ncgeny.c"
typo in ncgen.y
2012-01-16 09:29:19 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 153: /* conststring: TERMSTRING */
#line 892 "ncgen.y"
{(yyval.constant)=makeconstdata(NC_STRING);}
#line 2904 "ncgeny.c"
Added code to support NIL values for strings in ncgen. Needs test cases.
2013-07-11 04:00:48 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 154: /* constbool: conststring */
#line 896 "ncgen.y"
{(yyval.constant)=(yyvsp[0].constant);}
#line 2910 "ncgeny.c"
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 155: /* constbool: constint */
#line 897 "ncgen.y"
{(yyval.constant)=(yyvsp[0].constant);}
#line 2916 "ncgeny.c"
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 156: /* varident: IDENT */
#line 905 "ncgen.y"
{(yyval.sym)=(yyvsp[0].sym);}
#line 2922 "ncgeny.c"
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 157: /* varident: DATA */
#line 906 "ncgen.y"
{(yyval.sym)=identkeyword((yyvsp[0].sym));}
#line 2928 "ncgeny.c"
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case 158: /* ident: IDENT */
#line 910 "ncgen.y"
{(yyval.sym)=(yyvsp[0].sym);}
#line 2934 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 2938 "ncgeny.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
default: break;
}
More tweaking.
2020-12-08 05:45:14 +08:00
/* User semantic actions sometimes alter yychar, and that requires
that yytoken be updated with the new translation. We take the
approach of translating immediately before every use of yytoken.
One alternative is translating here after every semantic action,
but that translation would be missed if the semantic action invokes
YYABORT, YYACCEPT, or YYERROR immediately after altering yychar or
if it invokes YYBACKUP. In the case of YYABORT or YYACCEPT, an
incorrect destructor might then be invoked immediately. In the
case of YYERROR or YYBACKUP, subsequent parser actions might lead
to an incorrect destructor call or verbose syntax error message
before the lookahead is translated. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YY_SYMBOL_PRINT ("-> $$ =", YY_CAST (yysymbol_kind_t, yyr1[yyn]), &yyval, &yyloc);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
YYPOPSTACK (yylen);
yylen = 0;
*++yyvsp = yyval;
More tweaking.
2020-12-08 05:45:14 +08:00
/* Now 'shift' the result of the reduction. Determine what state
moving to trunk subdir
2010-06-03 21:24:43 +08:00
that goes to, based on the state we popped back to and the rule
number reduced by. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
{
const int yylhs = yyr1[yyn] - YYNTOKENS;
const int yyi = yypgoto[yylhs] + *yyssp;
yystate = (0 <= yyi && yyi <= YYLAST && yycheck[yyi] == *yyssp
? yytable[yyi]
: yydefgoto[yylhs]);
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
goto yynewstate;
More tweaking.
2020-12-08 05:45:14 +08:00
/*--------------------------------------.
| yyerrlab -- here on detecting error. |
`--------------------------------------*/
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yyerrlab:
More tweaking.
2020-12-08 05:45:14 +08:00
/* Make sure we have latest lookahead translation. See comments at
user semantic actions for why this is necessary. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yytoken = yychar == YYEMPTY ? YYSYMBOL_YYEMPTY : YYTRANSLATE (yychar);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* If not already recovering from an error, report this error. */
if (!yyerrstatus)
{
++yynerrs;
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yypcontext_t yyctx
= {yyssp, yytoken};
More tweaking.
2020-12-08 05:45:14 +08:00
char const *yymsgp = YY_("syntax error");
int yysyntax_error_status;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yysyntax_error_status = yysyntax_error (&yymsg_alloc, &yymsg, &yyctx);
More tweaking.
2020-12-08 05:45:14 +08:00
if (yysyntax_error_status == 0)
yymsgp = yymsg;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
else if (yysyntax_error_status == -1)
More tweaking.
2020-12-08 05:45:14 +08:00
{
if (yymsg != yymsgbuf)
YYSTACK_FREE (yymsg);
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yymsg = YY_CAST (char *,
YYSTACK_ALLOC (YY_CAST (YYSIZE_T, yymsg_alloc)));
if (yymsg)
More tweaking.
2020-12-08 05:45:14 +08:00
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yysyntax_error_status
= yysyntax_error (&yymsg_alloc, &yymsg, &yyctx);
yymsgp = yymsg;
More tweaking.
2020-12-08 05:45:14 +08:00
}
else
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yymsg = yymsgbuf;
yymsg_alloc = sizeof yymsgbuf;
yysyntax_error_status = YYENOMEM;
More tweaking.
2020-12-08 05:45:14 +08:00
}
}
yyerror (yymsgp);
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
if (yysyntax_error_status == YYENOMEM)
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
goto yyexhaustedlab;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
}
if (yyerrstatus == 3)
{
More tweaking.
2020-12-08 05:45:14 +08:00
/* If just tried and failed to reuse lookahead token after an
error, discard it. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
if (yychar <= YYEOF)
More tweaking.
2020-12-08 05:45:14 +08:00
{
/* Return failure if at end of input. */
if (yychar == YYEOF)
YYABORT;
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
else
More tweaking.
2020-12-08 05:45:14 +08:00
{
yydestruct ("Error: discarding",
yytoken, &yylval);
yychar = YYEMPTY;
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
More tweaking.
2020-12-08 05:45:14 +08:00
/* Else will try to reuse lookahead token after shifting the error
moving to trunk subdir
2010-06-03 21:24:43 +08:00
token. */
goto yyerrlab1;
/*---------------------------------------------------.
| yyerrorlab -- error raised explicitly by YYERROR. |
`---------------------------------------------------*/
yyerrorlab:
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Pacify compilers when the user code never invokes YYERROR and the
label yyerrorlab therefore never appears in user code. */
if (0)
YYERROR;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
More tweaking.
2020-12-08 05:45:14 +08:00
/* Do not reclaim the symbols of the rule whose action triggered
moving to trunk subdir
2010-06-03 21:24:43 +08:00
this YYERROR. */
YYPOPSTACK (yylen);
yylen = 0;
YY_STACK_PRINT (yyss, yyssp);
yystate = *yyssp;
goto yyerrlab1;
/*-------------------------------------------------------------.
| yyerrlab1 -- common code for both syntax error and YYERROR. |
`-------------------------------------------------------------*/
yyerrlab1:
More tweaking.
2020-12-08 05:45:14 +08:00
yyerrstatus = 3; /* Each real token shifted decrements this. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/* Pop stack until we find a state that shifts the error token. */
moving to trunk subdir
2010-06-03 21:24:43 +08:00
for (;;)
{
yyn = yypact[yystate];
More tweaking.
2020-12-08 05:45:14 +08:00
if (!yypact_value_is_default (yyn))
{
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
yyn += YYSYMBOL_YYerror;
if (0 <= yyn && yyn <= YYLAST && yycheck[yyn] == YYSYMBOL_YYerror)
More tweaking.
2020-12-08 05:45:14 +08:00
{
yyn = yytable[yyn];
if (0 < yyn)
break;
}
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Pop the current state because it cannot handle the error token. */
if (yyssp == yyss)
More tweaking.
2020-12-08 05:45:14 +08:00
YYABORT;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yydestruct ("Error: popping",
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YY_ACCESSING_SYMBOL (yystate), yyvsp);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
YYPOPSTACK (1);
yystate = *yyssp;
YY_STACK_PRINT (yyss, yyssp);
}
More tweaking.
2020-12-08 05:45:14 +08:00
YY_IGNORE_MAYBE_UNINITIALIZED_BEGIN
moving to trunk subdir
2010-06-03 21:24:43 +08:00
*++yyvsp = yylval;
More tweaking.
2020-12-08 05:45:14 +08:00
YY_IGNORE_MAYBE_UNINITIALIZED_END
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Shift the error token. */
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YY_SYMBOL_PRINT ("Shifting", YY_ACCESSING_SYMBOL (yyn), yyvsp, yylsp);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yystate = yyn;
goto yynewstate;
/*-------------------------------------.
| yyacceptlab -- YYACCEPT comes here. |
`-------------------------------------*/
yyacceptlab:
yyresult = 0;
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
goto yyreturn;
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/*-----------------------------------.
| yyabortlab -- YYABORT comes here. |
`-----------------------------------*/
yyabortlab:
yyresult = 1;
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
goto yyreturn;
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#if 1
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
/*-------------------------------------------------.
| yyexhaustedlab -- memory exhaustion comes here. |
`-------------------------------------------------*/
moving to trunk subdir
2010-06-03 21:24:43 +08:00
yyexhaustedlab:
yyerror (YY_("memory exhausted"));
yyresult = 2;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
goto yyreturn;
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
#endif
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
/*-------------------------------------------------------.
| yyreturn -- parsing is finished, clean up and return. |
`-------------------------------------------------------*/
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
yyreturn:
More tweaking.
2020-12-08 05:45:14 +08:00
if (yychar != YYEMPTY)
{
/* Make sure we have latest lookahead translation. See comments at
user semantic actions for why this is necessary. */
yytoken = YYTRANSLATE (yychar);
yydestruct ("Cleanup: discarding lookahead",
yytoken, &yylval);
}
/* Do not reclaim the symbols of the rule whose action triggered
moving to trunk subdir
2010-06-03 21:24:43 +08:00
this YYABORT or YYACCEPT. */
YYPOPSTACK (yylen);
YY_STACK_PRINT (yyss, yyssp);
while (yyssp != yyss)
{
yydestruct ("Cleanup: popping",
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
YY_ACCESSING_SYMBOL (+*yyssp), yyvsp);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
YYPOPSTACK (1);
}
#ifndef yyoverflow
if (yyss != yyssa)
YYSTACK_FREE (yyss);
#endif
if (yymsg != yymsgbuf)
YYSTACK_FREE (yymsg);
More tweaking.
2020-12-08 05:45:14 +08:00
return yyresult;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
#line 913 "ncgen.y"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#ifndef NO_STDARG
static void
yyerror(const char *fmt, ...)
#else
static void
yyerror(fmt,va_alist) const char* fmt; va_dcl
#endif
{
va_list argv;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
va_start(argv,fmt);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
(void)fprintf(stderr,"%s: %s line %d: ", progname, cdlname, lineno);
vderror(fmt,argv);
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
va_end(argv);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
/* undefine yywrap macro, in case we are using bison instead of yacc */
#ifdef yywrap
#undef yywrap
#endif
static int
ncgwrap(void) /* returns 1 on EOF if no more input */
{
return 1;
}
/* get lexical input routine generated by lex */
ckp
2015-11-20 04:44:07 +08:00
#include "ncgenl.c"
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Really should init our data within this file */
void
parse_init(void)
{
int i;
opaqueid = 0;
arrayuid = 0;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
symlist = listnew();
moving to trunk subdir
2010-06-03 21:24:43 +08:00
stack = listnew();
groupstack = listnew();
consttype = NC_NAT;
grpdefs = listnew();
dimdefs = listnew();
attdefs = listnew();
gattdefs = listnew();
xattdefs = listnew();
typdefs = listnew();
vardefs = listnew();
tmp = listnew();
/* Create the primitive types */
for(i=NC_NAT+1;i<=NC_STRING;i++) {
primsymbols[i] = makeprimitivetype(i);
}
lex_init();
}
static Symbol*
makeprimitivetype(nc_type nctype)
{
Symbol* sym = install(primtypenames[nctype]);
sym->objectclass=NC_TYPE;
sym->subclass=NC_PRIM;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
sym->nc_id = nctype;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
sym->typ.typecode = nctype;
sym->typ.size = ncsize(nctype);
sym->typ.nelems = 1;
Fix various problem around VLEN's re: https://github.com/Unidata/netcdf-c/issues/541 re: https://github.com/Unidata/netcdf-c/issues/1208 re: https://github.com/Unidata/netcdf-c/issues/2078 re: https://github.com/Unidata/netcdf-c/issues/2041 re: https://github.com/Unidata/netcdf-c/issues/2143 For a long time, there have been known problems with the management of complex types containing VLENs. This also involves the string type because it is stored as a VLEN of chars. This PR (mostly) fixes this problem. But note that it adds new functions to netcdf.h (see below) and this may require bumping the .so number. These new functions can be removed, if desired, in favor of functions in netcdf_aux.h, but netcdf.h seems the better place for them because they are intended as alternatives to the nc_free_vlen and nc_free_string functions already in netcdf.h. The term complex type refers to any type that directly or transitively references a VLEN type. So an array of VLENS, a compound with a VLEN field, and so on. In order to properly handle instances of these complex types, it is necessary to have function that can recursively walk instances of such types to perform various actions on them. The term "deep" is also used to mean recursive. At the moment, the two operations needed by the netcdf library are: * free'ing an instance of the complex type * copying an instance of the complex type. The current library does only shallow free and shallow copy of complex types. This means that only the top level is properly free'd or copied, but deep internal blocks in the instance are not touched. Note that the term "vector" will be used to mean a contiguous (in memory) sequence of instances of some type. Given an array with, say, dimensions 2 X 3 X 4, this will be stored in memory as a vector of length 2*3*4=24 instances. The use cases are primarily these. ## nc_get_vars Suppose one is reading a vector of instances using nc_get_vars (or nc_get_vara or nc_get_var, etc.). These functions will return the vector in the top-level memory provided. All interior blocks (form nested VLEN or strings) will have been dynamically allocated. After using this vector of instances, it is necessary to free (aka reclaim) the dynamically allocated memory, otherwise a memory leak occurs. So, the recursive reclaim function is used to walk the returned instance vector and do a deep reclaim of the data. Currently functions are defined in netcdf.h that are supposed to handle this: nc_free_vlen(), nc_free_vlens(), and nc_free_string(). Unfortunately, these functions only do a shallow free, so deeply nested instances are not properly handled by them. Note that internally, the provided data is immediately written so there is no need to copy it. But the caller may need to reclaim the data it passed into the function. ## nc_put_att Suppose one is writing a vector of instances as the data of an attribute using, say, nc_put_att. Internally, the incoming attribute data must be copied and stored so that changes/reclamation of the input data will not affect the attribute. Again, the code inside the netcdf library does only shallow copying rather than deep copy. As a result, one sees effects such as described in Github Issue https://github.com/Unidata/netcdf-c/issues/2143. Also, after defining the attribute, it may be necessary for the user to free the data that was provided as input to nc_put_att(). ## nc_get_att Suppose one is reading a vector of instances as the data of an attribute using, say, nc_get_att. Internally, the existing attribute data must be copied and returned to the caller, and the caller is responsible for reclaiming the returned data. Again, the code inside the netcdf library does only shallow copying rather than deep copy. So this can lead to memory leaks and errors because the deep data is shared between the library and the user. # Solution The solution is to build properly recursive reclaim and copy functions and use those as needed. These recursive functions are defined in libdispatch/dinstance.c and their signatures are defined in include/netcdf.h. For back compatibility, corresponding "ncaux_XXX" functions are defined in include/netcdf_aux.h. ```` int nc_reclaim_data(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_reclaim_data_all(int ncid, nc_type xtypeid, void* memory, size_t count); int nc_copy_data(int ncid, nc_type xtypeid, const void* memory, size_t count, void* copy); int nc_copy_data_all(int ncid, nc_type xtypeid, const void* memory, size_t count, void** copyp); ```` There are two variants. The first two, nc_reclaim_data() and nc_copy_data(), assume the top-level vector is managed by the caller. For reclaim, this is so the user can use, for example, a statically allocated vector. For copy, it assumes the user provides the space into which the copy is stored. The second two, nc_reclaim_data_all() and nc_copy_data_all(), allows the functions to manage the top-level. So for nc_reclaim_data_all, the top level is assumed to be dynamically allocated and will be free'd by nc_reclaim_data_all(). The nc_copy_data_all() function will allocate the top level and return a pointer to it to the user. The user can later pass that pointer to nc_reclaim_data_all() to reclaim the instance(s). # Internal Changes The netcdf-c library internals are changed to use the proper reclaim and copy functions. It turns out that the places where these functions are needed is quite pervasive in the netcdf-c library code. Using these functions also allows some simplification of the code since the stdata and vldata fields of NC_ATT_INFO are no longer needed. Currently this is commented out using the SEPDATA \#define macro. When any bugs are largely fixed, all this code will be removed. # Known Bugs 1. There is still one known failure that has not been solved. All the failures revolve around some variant of this .cdl file. The proximate cause of failure is the use of a VLEN FillValue. ```` netcdf x { types: float(*) row_of_floats ; dimensions: m = 5 ; variables: row_of_floats ragged_array(m) ; row_of_floats ragged_array:_FillValue = {-999} ; data: ragged_array = {10, 11, 12, 13, 14}, {20, 21, 22, 23}, {30, 31, 32}, {40, 41}, _ ; } ```` When a solution is found, I will either add it to this PR or post a new PR. # Related Changes * Mark nc_free_vlen(s) as deprecated in favor of ncaux_reclaim_data. * Remove the --enable-unfixed-memory-leaks option. * Remove the NC_VLENS_NOTEST code that suppresses some vlen tests. * Document this change in docs/internal.md * Disable the tst_vlen_data test in ncdump/tst_nccopy4.sh. * Mark types as fixed size or not (transitively) to optimize the reclaim and copy functions. # Misc. Changes * Make Doxygen process libdispatch/daux.c * Make sure the NC_ATT_INFO_T.container field is set.
2022-01-09 09:30:00 +08:00
(void)ncaux_class_alignment(nctype,&sym->typ.alignment);
fix discovered bug in opaque data handling
2012-05-05 03:22:30 +08:00
/* Make the basetype circular so we can always ask for it */
sym->typ.basetype = sym;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
sym->prefix = listnew();
return sym;
}
/* Symbol table operations for ncgen tool */
/* install sname in symbol table even if it is already there */
Symbol*
install(const char *sname)
Fix the handling of certain alias types on CDL files. re: https://github.com/Unidata/netcdf-c/issues/1977 PR https://github.com/Unidata/netcdf-c/pull/1753, changed ncgen to allows certain type names to be used as identifiers in selected situations. An unwanted side effect was that existing type aliases no longer were accepted by ncgen. Specifically, using the "long" type caused an error. I was able to figure out a better solution to the original problem (https://github.com/Unidata/netcdf-c/issues/1750) that also fixes this problem as well. This PR fixes that problem in ncgen/ncgen.l, and adds tests to ncdump/test_keywords.sh
2021-04-14 06:56:43 +08:00
{
return installin(sname,currentgroup());
}
Symbol*
installin(const char *sname, Symbol* grp)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
Symbol* sp;
Cleanup ncgen memory management
2017-10-31 05:52:08 +08:00
sp = (Symbol*) ecalloc (sizeof (struct Symbol));
moving to trunk subdir
2010-06-03 21:24:43 +08:00
sp->name = nulldup(sname);
sp->lineno = lineno;
Fix the handling of certain alias types on CDL files. re: https://github.com/Unidata/netcdf-c/issues/1977 PR https://github.com/Unidata/netcdf-c/pull/1753, changed ncgen to allows certain type names to be used as identifiers in selected situations. An unwanted side effect was that existing type aliases no longer were accepted by ncgen. Specifically, using the "long" type caused an error. I was able to figure out a better solution to the original problem (https://github.com/Unidata/netcdf-c/issues/1750) that also fixes this problem as well. This PR fixes that problem in ncgen/ncgen.l, and adds tests to ncdump/test_keywords.sh
2021-04-14 06:56:43 +08:00
sp->location = grp;
sp->container = grp;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
listpush(symlist,sp);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
return sp;
}
static Symbol*
currentgroup(void)
{
if(listlength(groupstack) == 0) return rootgroup;
return (Symbol*)listtop(groupstack);
}
static Symbol*
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
createrootgroup(const char* dataset)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
Symbol* gsym = install(dataset);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
gsym->objectclass = NC_GRP;
gsym->container = NULL;
gsym->subnodes = listnew();
gsym->grp.is_root = 1;
gsym->prefix = listnew();
update utilities: list,byte,hash,uri,log
2012-08-20 05:54:30 +08:00
listpush(grpdefs,(void*)gsym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
rootgroup = gsym;
return gsym;
}
static Symbol*
creategroup(Symbol * gsym)
{
/* See if this group already exists in currentgroup */
gsym->objectclass = NC_GRP;
if(dupobjectcheck(NC_GRP,gsym)) {
derror("Duplicate group name in same scope: %s",gsym->name);
return NULL;
}
addtogroup(gsym);
gsym->subnodes = listnew();
update utilities: list,byte,hash,uri,log
2012-08-20 05:54:30 +08:00
listpush(groupstack,(void*)gsym);
listpush(grpdefs,(void*)gsym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
return gsym;
}
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
static NCConstant*
moving to trunk subdir
2010-06-03 21:24:43 +08:00
makeconstdata(nc_type nctype)
{
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
NCConstant* con = nullconst();
moving to trunk subdir
2010-06-03 21:24:43 +08:00
consttype = nctype;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
con->nctype = nctype;
con->lineno = lineno;
con->filled = 0;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
switch (nctype) {
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
case NC_CHAR: con->value.charv = char_val; break;
case NC_BYTE: con->value.int8v = byte_val; break;
case NC_SHORT: con->value.int16v = int16_val; break;
case NC_INT: con->value.int32v = int32_val; break;
fixed: ncgen: nan bug made semicolons optional after type decls libncdap{3,4}: revamped the NC surrogate to better match with libsrc libdispatch: Added a new_nc function to the dispatch table; its purpose is to allow hierarchical use of NC compatible data structures. libsrc: cleaned up the NC structure by removing drno field general: removed --with-oc options
2010-11-10 06:53:03 +08:00
case NC_FLOAT:
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
con->value.floatv = float_val;
fixed: ncgen: nan bug made semicolons optional after type decls libncdap{3,4}: revamped the NC surrogate to better match with libsrc libdispatch: Added a new_nc function to the dispatch table; its purpose is to allow hierarchical use of NC compatible data structures. libsrc: cleaned up the NC structure by removing drno field general: removed --with-oc options
2010-11-10 06:53:03 +08:00
break;
case NC_DOUBLE:
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
con->value.doublev = double_val;
fixed: ncgen: nan bug made semicolons optional after type decls libncdap{3,4}: revamped the NC surrogate to better match with libsrc libdispatch: Added a new_nc function to the dispatch table; its purpose is to allow hierarchical use of NC compatible data structures. libsrc: cleaned up the NC structure by removing drno field general: removed --with-oc options
2010-11-10 06:53:03 +08:00
break;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
case NC_STRING: { /* convert to a set of chars*/
close Jira NCF-154
2012-03-08 07:38:51 +08:00
size_t len;
Resolve Lynton's vlen problems (partially) re: jira NCF-145
2012-02-14 08:25:32 +08:00
len = bbLength(lextext);
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
con->value.stringv.len = len;
con->value.stringv.stringv = bbExtract(lextext);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
break;
/* Allow these constants even in netcdf-3 */
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
case NC_UBYTE: con->value.uint8v = ubyte_val; break;
case NC_USHORT: con->value.uint16v = uint16_val; break;
case NC_UINT: con->value.uint32v = uint32_val; break;
case NC_INT64: con->value.int64v = int64_val; break;
case NC_UINT64: con->value.uint64v = uint64_val; break;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#ifdef USE_NETCDF4
case NC_OPAQUE: {
char* s;
fix bug in opaque parsing
2012-05-06 06:31:24 +08:00
int len;
Resolve Lynton's vlen problems (partially) re: jira NCF-145
2012-02-14 08:25:32 +08:00
len = bbLength(lextext);
Cleanup ncgen memory management
2017-10-31 05:52:08 +08:00
s = (char*)ecalloc(len+1);
Resolve Lynton's vlen problems (partially) re: jira NCF-145
2012-02-14 08:25:32 +08:00
strncpy(s,bbContents(lextext),len);
fix bug in opaque parsing
2012-05-06 06:31:24 +08:00
s[len] = '\0';
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
con->value.opaquev.stringv = s;
con->value.opaquev.len = len;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
} break;
Added code to support NIL values for strings in ncgen. Needs test cases.
2013-07-11 04:00:48 +08:00
case NC_NIL:
break; /* no associated value*/
moving to trunk subdir
2010-06-03 21:24:43 +08:00
#endif
Cleanup some ncgen memory problems
2017-11-01 04:03:57 +08:00
case NC_FILLVALUE:
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break; /* no associated value*/
Added code to support NIL values for strings in ncgen. Needs test cases.
2013-07-11 04:00:48 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
default:
yyerror("Data constant: unexpected NC type: %s",
nctypename(nctype));
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
con->value.stringv.stringv = NULL;
con->value.stringv.len = 0;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
return con;
}
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
static NCConstant*
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
makeenumconstref(Symbol* refsym)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
NCConstant* con = nullconst();
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
Resolve Lynton's vlen problems (partially) re: jira NCF-145
2012-02-14 08:25:32 +08:00
markcdf4("Enum type");
moving to trunk subdir
2010-06-03 21:24:43 +08:00
consttype = NC_ENUM;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
con->nctype = NC_ECONST;
con->lineno = lineno;
con->filled = 0;
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
refsym->objectclass = NC_TYPE;
refsym->subclass = NC_ECONST;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
con->value.enumv = refsym;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
return con;
}
static void
addtogroup(Symbol* sym)
{
Symbol* grp = currentgroup();
sym->container = grp;
update utilities: list,byte,hash,uri,log
2012-08-20 05:54:30 +08:00
listpush(grp->subnodes,(void*)sym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
setpathcurrent(sym);
}
/* Check for duplicate name of given type within current group*/
static int
dupobjectcheck(nc_class objectclass, Symbol* pattern)
{
int i;
Symbol* grp;
if(pattern == NULL) return 0;
grp = pattern->container;
if(grp == NULL || grp->subnodes == NULL) return 0;
for(i=0;i<listlength(grp->subnodes);i++) {
Symbol* sym = (Symbol*)listget(grp->subnodes,i);
[NCF-265] Ncgen is unable to resolve ambiguous references to an enum constant when two different enums have same econstant name. Solved by allowing more specific forms for econstant references. 1. /.../enumname.enumconstname 2. enumname.enumconstname 3. enumconstname Case 1 is resolved by using the econstant in the specific enum definition. If none is found, an error is reported. Case 2 is resolved by 1. finding an enclosing group with an enum definition with the specified name and containing the specified econstant. If there are more than one, then an error is reported 2. finding all enum definitions in the dataset that have the specified enum name and contain the specified econstant. If more than one is found, then an error is reported. If the above two methods fail, then report an error. Case 3 is similar to case 2, but all enums, irrespective of name are used if they contains the specified enum constant. The ref_tst_econst.cdl test in ncdump is causing ncdump to fail. So there may be yet some problem.
2013-09-21 10:31:21 +08:00
if(!sym->ref.is_ref && sym->objectclass == objectclass
moving to trunk subdir
2010-06-03 21:24:43 +08:00
&& strcmp(sym->name,pattern->name)==0) return 1;
}
return 0;
}
static void
setpathcurrent(Symbol* sym)
{
sym->is_prefixed = 0;
sym->prefix = prefixdup(groupstack);
}
/* Convert an nc_type code to the corresponding Symbol*/
Symbol*
basetypefor(nc_type nctype)
{
return primsymbols[nctype];
}
static int
Refactored 'Constant' to 'NCConstant' in ncgen/ to avoid reserved word collision on Windows. Added tst_ncgen4_classic.sh to CMake-based builds.
2013-07-17 04:22:48 +08:00
truefalse(NCConstant* con, int tag)
moving to trunk subdir
2010-06-03 21:24:43 +08:00
{
if(con->nctype == NC_STRING) {
char* sdata = con->value.stringv.stringv;
if(strncmp(sdata,"false",NC_MAX_NAME) == 0
|| strncmp(sdata,"0",NC_MAX_NAME) == 0)
return 0;
else if(strncmp(sdata,"true",NC_MAX_NAME) == 0
|| strncmp(sdata,"1",NC_MAX_NAME) == 0)
return 1;
else goto fail;
} else if(con->value.int32v < 0 || con->value.int32v > 1)
goto fail;
return con->value.int32v;
fail:
derror("%s: illegal value",specialname(tag));
return 0;
}
/* Since this may be affected by the _Format attribute, which
may come last, capture all the special info and sort it out
in semantics.
*/
static Symbol*
makespecial(int tag, Symbol* vsym, Symbol* tsym, void* data, int isconst)
{
Symbol* attr = NULL;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
Datalist* list = NULL;
NCConstant* con = NULL;
NCConstant* tmp = NULL;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
int tf = 0;
char* sdata = NULL;
int idata = -1;
Addressed some instances of Coverity issue 711927, uninitialized scalar variable. Modified 'makeparser' target in CMake build system.
2013-08-15 00:12:49 +08:00
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
if((GLOBAL_SPECIAL & tag) != 0) {
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
if(vsym != NULL) {
derror("_Format: must be global attribute");
vsym = NULL;
}
} else {
if(vsym == NULL) {
derror("%s: must have non-NULL vsym", specialname(tag));
return NULL;
}
}
Addressed some instances of Coverity issue 711927, uninitialized scalar variable. Modified 'makeparser' target in CMake build system.
2013-08-15 00:12:49 +08:00
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
if(tag != _FILLVALUE_FLAG && tag != _FORMAT_FLAG)
/*Main.*/specials_flag++;
Resolve Lynton's vlen problems (partially) re: jira NCF-145
2012-02-14 08:25:32 +08:00
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
if(isconst)
Refactored 'Constant' to 'NCConstant' in ncgen/ to avoid reserved word collision on Windows. Added tst_ncgen4_classic.sh to CMake-based builds.
2013-07-17 04:22:48 +08:00
con = (NCConstant*)data;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
else
moving to trunk subdir
2010-06-03 21:24:43 +08:00
list = (Datalist*)data;
switch (tag) {
case _FLETCHER32_FLAG:
case _SHUFFLE_FLAG:
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
case _ISNETCDF4_FLAG:
moving to trunk subdir
2010-06-03 21:24:43 +08:00
case _NOFILL_FLAG:
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
tmp = nullconst();
tmp->nctype = (con->nctype == NC_STRING?NC_STRING:NC_INT);
convert1(con,tmp);
tf = truefalse(tmp,tag);
reclaimconstant(tmp);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
case _FORMAT_FLAG:
case _STORAGE_FLAG:
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
case _NCPROPS_FLAG:
moving to trunk subdir
2010-06-03 21:24:43 +08:00
case _ENDIAN_FLAG:
Add new capabilities to filter code: 1. Allow nccopy to apply filters, especially on the output file. This provides a third way to do this other than using ncgen or programatically 2. Make sure that even if the filter code is not available, it is possible to see the filter id and parameters for variables using e.g ncdump -hs. 3. Fix bug in nccopy so that the input file does not necessarily have to be netcdf-4. 4. At last minute decided to change to using a single "_Filter" attribute for ncgen 5. Added a test to tst_filter.sh to generate C code using ncgen.
2017-05-15 08:10:02 +08:00
case _FILTER_FLAG:
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
case _CODECS_FLAG:
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
tmp = nullconst();
tmp->nctype = NC_STRING;
convert1(con,tmp);
if(tmp->nctype == NC_STRING) {
sdata = tmp->value.stringv.stringv;
tmp->value.stringv.stringv = NULL;
tmp->value.stringv.len = 0;
} else
moving to trunk subdir
2010-06-03 21:24:43 +08:00
derror("%s: illegal value",specialname(tag));
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
reclaimconstant(tmp);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
case _SUPERBLOCK_FLAG:
moving to trunk subdir
2010-06-03 21:24:43 +08:00
case _DEFLATE_FLAG:
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case _QUANTIZEBG_FLAG:
case _QUANTIZEBR_FLAG:
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
tmp = nullconst();
tmp->nctype = NC_INT;
convert1(con,tmp);
if(tmp->nctype == NC_INT)
idata = tmp->value.int32v;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
else
derror("%s: illegal value",specialname(tag));
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
reclaimconstant(tmp);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
case _CHUNKSIZES_FLAG:
case _FILLVALUE_FLAG:
/* Handle below */
break;
default: PANIC1("unexpected special tag: %d",tag);
}
Addressed UNIX API issues reported by clang.
2015-06-19 04:31:10 +08:00
moving to trunk subdir
2010-06-03 21:24:43 +08:00
if(tag == _FORMAT_FLAG) {
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
/* Watch out: this is a global attribute */
Fixed the following Jira minor bugs: NCF-42: _Format attribute sometimes being ignored NCF-43: Fixed unsigned long long parsing. NCF-47: Make opendap code properly handle illegal names like "x.y" by supressing them NCF-49: check for uint type NCF-50: properly handle username:pwd embedded in urls.
2011-05-13 01:51:32 +08:00
struct Kvalues* kvalue;
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
int found = 0;
moving to trunk subdir
2010-06-03 21:24:43 +08:00
/* Use the table in main.c */
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
for(kvalue = legalkinds; kvalue->name; kvalue++) {
Addressed UNIX API issues reported by clang.
2015-06-19 04:31:10 +08:00
if(sdata) {
if(strcmp(sdata, kvalue->name) == 0) {
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
globalspecials._Format = kvalue->k_flag;
/*Main.*/format_attribute = 1;
Addressed UNIX API issues reported by clang.
2015-06-19 04:31:10 +08:00
found = 1;
break;
}
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
if(!found)
derror("_Format: illegal value: %s",sdata);
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
} else if((GLOBAL_SPECIAL & tag) != 0) {
if(tag == _ISNETCDF4_FLAG)
globalspecials._IsNetcdf4 = tf;
else if(tag == _SUPERBLOCK_FLAG)
globalspecials._Superblock = idata;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
else if(tag == _NCPROPS_FLAG) {
globalspecials._NCProperties = sdata;
sdata = NULL;
Fix errors when building on big-endian machine re: issue https://github.com/Unidata/netcdf-c/issues/1278 re: issue https://github.com/Unidata/netcdf-c/issues/876 re: issue https://github.com/Unidata/netcdf-c/issues/806 * Major change to the handling of 8-byte parameters for nc_def_var_filter. The old code was not well thought out. * The new algorithm is documented in docs/filters.md. * Added new utility file plugins/H5Zutil.c to support * Modified plugins/H5Zmisc.c to use new algorithm the new algorithm. * Renamed include/ncfilter.h to include/netcdf_filter.h and made it an installed header so clients can access the new algorithm utility. * Fixed nc_test4/tst_filterparser.c and nc_test4/test_filter_misc.c to use the new algorithm * libdap4/ fixes: * d4swap.c has an error in the endian pre-processing such that record counts were not being swapped correctly. * d4data.c had an error in that checksums were being computed after endian swapping rather than before. * ocinitialize() was never being called, so xxdr bigendian handling was never set correctly. * Required adding debug statements to occompile * Found and fixed memory leak in ncdump.c Not tested: * HDF4 * Pnetcdf * parallel HDF5
2019-02-01 12:13:06 +08:00
}
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
} else {
Specialdata* special;
/* Set up special info */
This PR adds EXPERIMENTAL support for accessing data in the cloud using a variant of the Zarr protocol and storage format. This enhancement is generically referred to as "NCZarr". The data model supported by NCZarr is netcdf-4 minus the user-defined types and the String type. In this sense it is similar to the CDF-5 data model. More detailed information about enabling and using NCZarr is described in the document NUG/nczarr.md and in a [Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in). WARNING: this code has had limited testing, so do use this version for production work. Also, performance improvements are ongoing. Note especially the following platform matrix of successful tests: Platform | Build System | S3 support ------------------------------------ Linux+gcc | Automake | yes Linux+gcc | CMake | yes Visual Studio | CMake | no Additionally, and as a consequence of the addition of NCZarr, major changes have been made to the Filter API. NOTE: NCZarr does not yet support filters, but these changes are enablers for that support in the future. Note that it is possible (probable?) that there will be some accidental reversions if the changes here did not correctly mimic the existing filter testing. In any case, previously filter ids and parameters were of type unsigned int. In order to support the more general zarr filter model, this was all converted to char*. The old HDF5-specific, unsigned int operations are still supported but they are wrappers around the new, char* based nc_filterx_XXX functions. This entailed at least the following changes: 1. Added the files libdispatch/dfilterx.c and include/ncfilter.h 2. Some filterx utilities have been moved to libdispatch/daux.c 3. A new entry, "filter_actions" was added to the NCDispatch table and the version bumped. 4. An overly complex set of structs was created to support funnelling all of the filterx operations thru a single dispatch "filter_actions" entry. 5. Move common code to from libhdf5 to libsrc4 so that it is accessible to nczarr. Changes directly related to Zarr: 1. Modified CMakeList.txt and configure.ac to support both C and C++ -- this is in support of S3 support via the awd-sdk libraries. 2. Define a size64_t type to support nczarr. 3. More reworking of libdispatch/dinfermodel.c to support zarr and to regularize the structure of the fragments section of a URL. Changes not directly related to Zarr: 1. Make client-side filter registration be conditional, with default off. 2. Hack include/nc4internal.h to make some flags added by Ed be unique: e.g. NC_CREAT, NC_INDEF, etc. 3. cleanup include/nchttp.h and libdispatch/dhttp.c. 4. Misc. changes to support compiling under Visual Studio including: * Better testing under windows for dirent.h and opendir and closedir. 5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags and to centralize error reporting. 6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them. 7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible. Changes Left TO-DO: 1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
special = &vsym->var.special;
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
if(tag == _FILLVALUE_FLAG) {
/* fillvalue must be a single value*/
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
if(!isconst && datalistlen(list) != 1)
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
derror("_FillValue: must be a single (possibly compound) value",
vsym->name);
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
if(isconst) {
list = const2list(con);
con = NULL;
}
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
/* check that the attribute value contains no fill values*/
if(containsfills(list)) {
derror("Attribute data may not contain fill values (i.e. _ )");
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
/* _FillValue is also a real attribute*/
if(vsym->objectclass != NC_VAR) {
derror("_FillValue attribute not associated with variable: %s",vsym->name);
}
if(tsym == NULL) tsym = vsym->typ.basetype;
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
#if 0 /* No longer require matching types */
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
else if(vsym->typ.basetype != tsym) {
derror("_FillValue attribute type does not match variable type: %s",vsym->name);
}
Support MSYS2/Mingw platform re: The current netcdf-c release has some problems with the mingw platform on windows. Mostly they are path issues. Changes to support mingw+msys2: ------------------------------- * Enable option of looking into the windows registry to find the mingw root path. In aid of proper path handling. * Add mingw+msys as a specific platform in configure.ac and move testing of the platform to the front so it is available early. * Handle mingw X libncpoco (dynamic loader) properly even though mingw does not yet support it. * Handle mingw X plugins properly even though mingw does not yet support it. * Alias pwd='pwd -W' to better handle paths in shell scripts. * Plus a number of other minor compile irritations. * Disallow the use of multiple nc_open's on the same file for windows (and mingw) because windows does not seem to handle these properly. Not sure why we did not catch this earlier. * Add mountpoint info to dpathmgr.c to help support mingw. * Cleanup dpathmgr conversions. Known problems: --------------- * I have not been able to get shared libraries to work, so plugins/filters must be disabled. * There is some kind of problem with libcurl that I have not solved, so all uses of libcurl (currently DAP+Byterange) must be disabled. Misc. other fixes: ------------------ * Cleanup the relationship between ENABLE_PLUGINS and various other flags in CMakeLists.txt and configure.ac. * Re-arrange the TESTDIRS order in Makefile.am. * Add pseudo-breakpoint to nclog.[ch] for debugging. * Improve the documentation of the path manager code in ncpathmgr.h * Add better support for relative paths in dpathmgr.c * Default the mode args to NCfopen to include "b" (binary) for windows. * Add optional debugging output in various places. * Make sure that everything builds with plugins disabled. * Fix numerous (s)printf inconsistencies betweenb the format spec and the arguments.
2021-12-24 13:18:56 +08:00
#endif
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
special->_Fillvalue = clonedatalist(list);
/* Create the corresponding attribute */
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
attr = makeattribute(install("_FillValue"),vsym,tsym,list,ATTRVAR);
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
list = NULL;
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
} else switch (tag) {
merge-squash
2015-08-16 06:26:35 +08:00
/* These will be output as attributes later */
Moved libsrc5 to libsrcp because pnetcdf is used for more than CDF-5 files.
2015-08-15 10:38:30 +08:00
case _STORAGE_FLAG:
if(!sdata)
derror("_Storage: illegal NULL value");
else if(strcmp(sdata,"contiguous") == 0)
special->_Storage = NC_CONTIGUOUS;
Make utilities support NC_COMPACT re: https://github.com/Unidata/netcdf-c/issues/1642 Modify ncdump, nccopy, and ncgen to support the NC_COMPACT storage option. Added test cases and added description to the man pages for the utilities. 1. ncdump: For compact storage variable, print special attribute __Storage_ as ```` <var>: _Storage = "compact"; ```` 2. ncgen: parse and implement ```` <var>: _Storage = "compact"; ```` in a .cdl file 3. nccopy: Extend the chunk specification (-c flag) to support compact using the forms ```` nccopy ... -c <var>:compact and nccopy ... -c <var>:contiguous ```` Misc. other changes 1. cleanup the copy_chunking function in ncdump/nccopy.c
2020-03-01 03:06:21 +08:00
else if(strcmp(sdata,"compact") == 0)
special->_Storage = NC_COMPACT;
Moved libsrc5 to libsrcp because pnetcdf is used for more than CDF-5 files.
2015-08-15 10:38:30 +08:00
else if(strcmp(sdata,"chunked") == 0)
special->_Storage = NC_CHUNKED;
else
derror("_Storage: illegal value: %s",sdata);
special->flags |= _STORAGE_FLAG;
break;
case _FLETCHER32_FLAG:
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
special->_Fletcher32 = tf;
special->flags |= _FLETCHER32_FLAG;
break;
case _DEFLATE_FLAG:
special->_DeflateLevel = idata;
special->flags |= _DEFLATE_FLAG;
break;
Rename GranularBitGroom to GranularBitRound As per Charlie Zender's request (https://github.com/Unidata/netcdf-c/pull/2197#issuecomment-1022762863), the GranularBitGroom name is changed to GranularBitRound with attendant code changes.
2022-01-29 04:04:16 +08:00
case _QUANTIZEBG_FLAG:
special->_Quantizer = NC_QUANTIZE_BITGROOM;
special->_NSD = idata;
special->flags |= _QUANTIZEBG_FLAG;
break;
case _QUANTIZEBR_FLAG:
special->_Quantizer = NC_QUANTIZE_GRANULARBR;
special->_NSD = idata;
special->flags |= _QUANTIZEBR_FLAG;
break;
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
case _SHUFFLE_FLAG:
special->_Shuffle = tf;
special->flags |= _SHUFFLE_FLAG;
break;
case _ENDIAN_FLAG:
Addressed UNIX API issues reported by clang.
2015-06-19 04:31:10 +08:00
if(!sdata)
Moved libsrc5 to libsrcp because pnetcdf is used for more than CDF-5 files.
2015-08-15 10:38:30 +08:00
derror("_Endianness: illegal NULL value");
Addressed UNIX API issues reported by clang.
2015-06-19 04:31:10 +08:00
else if(strcmp(sdata,"little") == 0)
special->_Endianness = 1;
else if(strcmp(sdata,"big") == 0)
special->_Endianness = 2;
else
derror("_Endianness: illegal value: %s",sdata);
special->flags |= _ENDIAN_FLAG;
break;
case _NOFILL_FLAG:
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
special->_Fill = (1 - tf); /* negate */
special->flags |= _NOFILL_FLAG;
break;
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
case _CHUNKSIZES_FLAG: {
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
int i;
Fix errors when building on big-endian machine re: issue https://github.com/Unidata/netcdf-c/issues/1278 re: issue https://github.com/Unidata/netcdf-c/issues/876 re: issue https://github.com/Unidata/netcdf-c/issues/806 * Major change to the handling of 8-byte parameters for nc_def_var_filter. The old code was not well thought out. * The new algorithm is documented in docs/filters.md. * Added new utility file plugins/H5Zutil.c to support * Modified plugins/H5Zmisc.c to use new algorithm the new algorithm. * Renamed include/ncfilter.h to include/netcdf_filter.h and made it an installed header so clients can access the new algorithm utility. * Fixed nc_test4/tst_filterparser.c and nc_test4/test_filter_misc.c to use the new algorithm * libdap4/ fixes: * d4swap.c has an error in the endian pre-processing such that record counts were not being swapped correctly. * d4data.c had an error in that checksums were being computed after endian swapping rather than before. * ocinitialize() was never being called, so xxdr bigendian handling was never set correctly. * Required adding debug statements to occompile * Found and fixed memory leak in ncdump.c Not tested: * HDF4 * Pnetcdf * parallel HDF5
2019-02-01 12:13:06 +08:00
list = (isconst ? const2list(con) : list);
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
special->nchunks = list->length;
Cleanup ncgen memory management
2017-10-31 05:52:08 +08:00
special->_ChunkSizes = (size_t*)ecalloc(sizeof(size_t)*special->nchunks);
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
for(i=0;i<special->nchunks;i++) {
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
tmp = nullconst();
tmp->nctype = NC_INT;
convert1(list->data[i],tmp);
if(tmp->nctype == NC_INT) {
special->_ChunkSizes[i] = (size_t)tmp->value.int32v;
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
} else {
efree(special->_ChunkSizes);
derror("%s: illegal value",specialname(tag));
}
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
reclaimconstant(tmp);
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
}
special->flags |= _CHUNKSIZES_FLAG;
/* Chunksizes => storage == chunked */
special->flags |= _STORAGE_FLAG;
special->_Storage = NC_CHUNKED;
} break;
Add new capabilities to filter code: 1. Allow nccopy to apply filters, especially on the output file. This provides a third way to do this other than using ncgen or programatically 2. Make sure that even if the filter code is not available, it is possible to see the filter id and parameters for variables using e.g ncdump -hs. 3. Fix bug in nccopy so that the input file does not necessarily have to be netcdf-4. 4. At last minute decided to change to using a single "_Filter" attribute for ncgen 5. Added a test to tst_filter.sh to generate C code using ncgen.
2017-05-15 08:10:02 +08:00
case _FILTER_FLAG:
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
#ifdef USE_NETCDF4
Add new capabilities to filter code: 1. Allow nccopy to apply filters, especially on the output file. This provides a third way to do this other than using ncgen or programatically 2. Make sure that even if the filter code is not available, it is possible to see the filter id and parameters for variables using e.g ncdump -hs. 3. Fix bug in nccopy so that the input file does not necessarily have to be netcdf-4. 4. At last minute decided to change to using a single "_Filter" attribute for ncgen 5. Added a test to tst_filter.sh to generate C code using ncgen.
2017-05-15 08:10:02 +08:00
/* Parse the filter spec */
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
if(parsefilterflag(sdata,special) == NC_NOERR)
Add new capabilities to filter code: 1. Allow nccopy to apply filters, especially on the output file. This provides a third way to do this other than using ncgen or programatically 2. Make sure that even if the filter code is not available, it is possible to see the filter id and parameters for variables using e.g ncdump -hs. 3. Fix bug in nccopy so that the input file does not necessarily have to be netcdf-4. 4. At last minute decided to change to using a single "_Filter" attribute for ncgen 5. Added a test to tst_filter.sh to generate C code using ncgen.
2017-05-15 08:10:02 +08:00
special->flags |= _FILTER_FLAG;
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
else {
Spelling fixes
2019-09-18 10:27:43 +08:00
derror("_Filter: unparsable filter spec: %s",sdata);
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
}
#else
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
derror("%s: the filter attribute requires netcdf-4 to be enabled",specialname(tag));
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
#endif
break;
case _CODECS_FLAG:
#ifdef USE_NETCDF4
/* Parse the codec spec */
if(parsecodecsflag(sdata,special) == NC_NOERR)
special->flags |= _CODECS_FLAG;
else {
derror("_Codecs: unparsable codec spec: %s",sdata);
}
#else
derror("%s: the _Codecs attribute requires netcdf-4 to be enabled",specialname(tag));
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
#endif
Finalize the compression support. This relies on the HDF5 capability to dynamically load compression filters. Note that a compression filter is just a subcase of filters. The primary user-visible changes are as follows: 1. Add a standard header "netcdf_filter.h" that defines the necessary API extensions 2. Modify ncgen to support two new special attributes "_Filter_ID" and "_Filter_Parameters" so that compression can be turned on when creating a file using ncgen. 4. Add a detailed description of filtering support to the user's guide; see the file filters.md 5. Add a test case directory for this: nc_test4/filter_test. It is fragile and a ./configure flags (-enable-filter-test) is defined (default disabled) to shut this off this test to avoid spurious 'make check' failures. Note that the HDF5 documentation is not up-to-date, so much of what is encoded here comes from examining the actual code in the file H5PL.c in the HDF5 source code.
2017-04-28 03:01:59 +08:00
break;
1. Any test that references nctestserver/NC_findtestserver should be under ENABLE_DAP_REMOTE_TESTS. Fixed to make sure that this is so. Also attempted to fix ncdap_test/CMakeLists.txt, but probably got it wrong. HT to Nico Schlomer. 2. Attempted to reduce the number of conversion errors when -Wconversion is set. Fixed oc2, but rest of netcdf remains to be done. HT to Nico Schlomer. 3. When doing #2, I discovered an error in ncgen.y that has remained hidden. This required some other test case fixes.
2014-03-09 11:41:30 +08:00
default: PANIC1("makespecial: illegal token: %d",tag);
}
}
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
if(sdata) efree(sdata);
if(con) reclaimconstant(con);
if(list) reclaimdatalist(list);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
return attr;
}
static Symbol*
makeattribute(Symbol* asym,
Symbol* vsym,
Symbol* tsym,
Datalist* data,
Attrkind kind) /* global var or unknown*/
{
asym->objectclass = NC_ATT;
asym->data = data;
switch (kind) {
case ATTRVAR:
asym->att.var = vsym;
asym->typ.basetype = tsym;
update utilities: list,byte,hash,uri,log
2012-08-20 05:54:30 +08:00
listpush(attdefs,(void*)asym);
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
addtogroup(asym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
case ATTRGLOBAL:
asym->att.var = NULL; /* NULL => NC_GLOBAL*/
asym->typ.basetype = tsym;
Oops, forgot to test nc3 only. Fixed some errors from that case.
2016-05-06 05:59:54 +08:00
listpush(gattdefs,(void*)asym);
addtogroup(asym);
moving to trunk subdir
2010-06-03 21:24:43 +08:00
break;
default: PANIC1("unexpected attribute type: %d",kind);
}
/* finally; check that the attribute value contains no fill values*/
if(containsfills(data)) {
derror("Attribute data may not contain fill values (i.e. _ ): %s",asym->name);
}
return asym;
}
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
static long long
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
extractint(NCConstant* con)
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
{
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
switch (con->nctype) {
case NC_BYTE: return (long long)(con->value.int8v);
case NC_SHORT: return (long long)(con->value.int16v);
case NC_INT: return (long long)(con->value.int32v);
case NC_UBYTE: return (long long)(con->value.uint8v);
case NC_USHORT: return (long long)(con->value.uint16v);
case NC_UINT: return (long long)(con->value.uint32v);
case NC_INT64: return (long long)(con->value.int64v);
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
default:
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
derror("Not a signed integer type: %d",con->nctype);
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
break;
}
return 0;
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
static int
containsfills(Datalist* list)
{
Resolve Lynton's vlen problems (partially) re: jira NCF-145
2012-02-14 08:25:32 +08:00
if(list != NULL) {
int i;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
NCConstant** cons = list->data;
for(i=0;i<list->length;i++) {
if(cons[i]->nctype == NC_COMPOUND) {
if(containsfills(cons[i]->value.compoundv)) return 1;
} else if(cons[i]->nctype == NC_FILLVALUE)
Cleanup some ncgen memory problems
2017-11-01 04:03:57 +08:00
return 1;
Resolve Lynton's vlen problems (partially) re: jira NCF-145
2012-02-14 08:25:32 +08:00
}
moving to trunk subdir
2010-06-03 21:24:43 +08:00
}
return 0;
}
1. Keep up to date by merging master 2. Factored out the parameter string parsing for ncgen and nccopy int libdispatch/dfilter.c + include/ncfilter.h 3. Allow a parameter string to use constant types other than unsigned int. See docs/filters.md for details. 4. Moved the old content of include/netcdf_filter.h into include/netcdf.h and removed include/netcdf_filter.h as no longer needed. 5. Force the test filter (bzip2) in nc_test4/filter_test to be built using BUILT_SOURCES.
2017-10-09 05:56:45 +08:00
/*
Try to infer the file type from the
kinds of constructs used in the cdl file.
*/
Fixed the following Jira minor bugs: NCF-42: _Format attribute sometimes being ignored NCF-43: Fixed unsigned long long parsing. NCF-47: Make opendap code properly handle illegal names like "x.y" by supressing them NCF-49: check for uint type NCF-50: properly handle username:pwd embedded in urls.
2011-05-13 01:51:32 +08:00
static void
1. There were several bugs in ncdump/dumplib.c that were not taking the CDF-5 format into account. 2. Had to update ncgen.1 man page to define the new k-flag rules to deal with cdf-5. 3. Had to fix some tests that use 'cmp' for comparison; this really should be deprecated. 3. There was a bug in configure.ac with respect to using the enable-netcdf-4 flag vs using disable-netcdf-4.
2015-11-07 08:03:28 +08:00
vercheck(int tid)
Fixed the following Jira minor bugs: NCF-42: _Format attribute sometimes being ignored NCF-43: Fixed unsigned long long parsing. NCF-47: Make opendap code properly handle illegal names like "x.y" by supressing them NCF-49: check for uint type NCF-50: properly handle username:pwd embedded in urls.
2011-05-13 01:51:32 +08:00
{
1. There were several bugs in ncdump/dumplib.c that were not taking the CDF-5 format into account. 2. Had to update ncgen.1 man page to define the new k-flag rules to deal with cdf-5. 3. Had to fix some tests that use 'cmp' for comparison; this really should be deprecated. 3. There was a bug in configure.ac with respect to using the enable-netcdf-4 flag vs using disable-netcdf-4.
2015-11-07 08:03:28 +08:00
switch (tid) {
1. Keep up to date by merging master 2. Factored out the parameter string parsing for ncgen and nccopy int libdispatch/dfilter.c + include/ncfilter.h 3. Allow a parameter string to use constant types other than unsigned int. See docs/filters.md for details. 4. Moved the old content of include/netcdf_filter.h into include/netcdf.h and removed include/netcdf_filter.h as no longer needed. 5. Force the test filter (bzip2) in nc_test4/filter_test to be built using BUILT_SOURCES.
2017-10-09 05:56:45 +08:00
case NC_UBYTE: markcdf4("netCDF4/5 type: UBYTE"); break;
case NC_USHORT: markcdf4("netCDF4/5 type: USHORT"); break;
case NC_UINT: markcdf4("netCDF4/5 type: UINT"); break;
case NC_INT64: markcdf4("netCDF4/5 type: INT64"); break;
case NC_UINT64: markcdf4("netCDF4/5 type: UINT64"); break;
1. There were several bugs in ncdump/dumplib.c that were not taking the CDF-5 format into account. 2. Had to update ncgen.1 man page to define the new k-flag rules to deal with cdf-5. 3. Had to fix some tests that use 'cmp' for comparison; this really should be deprecated. 3. There was a bug in configure.ac with respect to using the enable-netcdf-4 flag vs using disable-netcdf-4.
2015-11-07 08:03:28 +08:00
case NC_STRING: markcdf4("netCDF4 type: STRING"); break;
case NC_VLEN: markcdf4("netCDF4 type: VLEN"); break;
case NC_OPAQUE: markcdf4("netCDF4 type: OPAQUE"); break;
case NC_ENUM: markcdf4("netCDF4 type: ENUM"); break;
case NC_COMPOUND: markcdf4("netCDF4 type: COMPOUND"); break;
Fixed the following Jira minor bugs: NCF-42: _Format attribute sometimes being ignored NCF-43: Fixed unsigned long long parsing. NCF-47: Make opendap code properly handle illegal names like "x.y" by supressing them NCF-49: check for uint type NCF-50: properly handle username:pwd embedded in urls.
2011-05-13 01:51:32 +08:00
default: break;
}
}
Add provenance info for netcdf-4 files. This consists of a persistent attribute named _NCProperties plus two computed attributes _IsNetcdf4 and _SuperblockVersion. See the 'Provenance Attributes' section of docs/attribute_conventions.md for details.
2016-05-04 11:17:06 +08:00
const char*
specialname(int tag)
{
struct Specialtoken* spp = specials;
for(;spp->name;spp++) {
if(spp->tag == tag)
return spp->name;
}
return "<unknown>";
}
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
#ifdef USE_NETCDF4
Add new capabilities to filter code: 1. Allow nccopy to apply filters, especially on the output file. This provides a third way to do this other than using ncgen or programatically 2. Make sure that even if the filter code is not available, it is possible to see the filter id and parameters for variables using e.g ncdump -hs. 3. Fix bug in nccopy so that the input file does not necessarily have to be netcdf-4. 4. At last minute decided to change to using a single "_Filter" attribute for ncgen 5. Added a test to tst_filter.sh to generate C code using ncgen.
2017-05-15 08:10:02 +08:00
/*
Parse a filter spec string and store it in special
*/
static int
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
parsefilterflag(const char* sdata, Specialdata* special)
Add new capabilities to filter code: 1. Allow nccopy to apply filters, especially on the output file. This provides a third way to do this other than using ncgen or programatically 2. Make sure that even if the filter code is not available, it is possible to see the filter id and parameters for variables using e.g ncdump -hs. 3. Fix bug in nccopy so that the input file does not necessarily have to be netcdf-4. 4. At last minute decided to change to using a single "_Filter" attribute for ncgen 5. Added a test to tst_filter.sh to generate C code using ncgen.
2017-05-15 08:10:02 +08:00
{
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
int stat = NC_NOERR;
Replaced the overwritten updated parsefilterflag
2017-11-14 04:19:46 +08:00
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
if(sdata == NULL || strlen(sdata) == 0) return NC_EINVAL;
Replaced the overwritten updated parsefilterflag
2017-11-14 04:19:46 +08:00
Mostly revert the filter code to reduce its complexity of use. re: https://github.com/Unidata/netcdf-c/issues/1836 Revert the internal filter code to simplify it. From the user's point of view, the only visible changes should be: 1. The functions that convert text to filter specs have had their signature reverted and have been moved to netcdf_aux.h 2. Some filter API functions now return NC_ENOFILTER when inquiry is made about some filter. Internally,the dispatch table has been modified to get rid of the filter_actions entry and associated complex structures. It has been replaced with inq_var_filter_ids and inq_var_filter_info entries and the dispatch table version has been bumped to 3. Corresponding NOOP and NOTNC4 functions were added to libdispatch/dnotnc4.c. Also, the filter_action entries in dispatch tables were replaced for all dispatch code bases (HDF5, DAP2, etc). This should only impact UDF users. In the process, it became clear that the form of the filters field in NC_VAR_INFO_T was format dependent, so I converted it to be of type void* and pushed its management into the various dispatch code bases. Specifically libhdf5 and libnczarr now manage the filters field in their own way. The auxilliary functions for parsing textual filter specifications were moved to netcdf_aux.h and were renamed to the following: * ncaux_h5filterspec_parse * ncaux_h5filterspec_parselist * ncaux_h5filterspec_free * ncaux_h5filter_fix8 Misc. Other Changes: 1. Document NUG/filters.md updated to reflect the changes above. 2. All the old data types (structs and enums) used by filter_actions actions were deleted. The exception is the NC_H5_Filterspec because it is needed by ncaux_h5filterspec_parselist. 3. Clientside filters were removed -- another enhancement for which no-one ever asked. 4. The ability to remove filters was itself removed. 5. Some functionality needed by nczarr was moved from libhdf5 to libsrc4 e.g. nc4_find_default_chunksizes 6. All the filterx code was removed 7. ncfilter.h and nc4filter.c no longer used Misc. Unrelated Changes: 1. The nczarr_test makefile clean was leaving some directories; so add clean-local to take care of them.
2020-09-28 02:43:46 +08:00
stat = ncaux_h5filterspec_parselist(sdata, NULL, &special->nfilters, &special->_Filters);
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
if(stat)
derror("Malformed filter spec: %s",sdata);
Add support for multiple filters per variable. re: https://github.com/Unidata/netcdf-c/issues/1584 Support has been added for multiple filters per variable. This affects a number of components in netcdf. The new APIs are documented in NUG/filters.md. The primary changes are: * A set of new functions are provided (see __include/netcdf_filter.h__). - Obtain a list of the filters associated with a variable - Obtain the parameters for a specific filter. * The existing __nc_inq_var_filter__ function now returns info about the first defined filter. * The utilities (ncgen, ncdump, and nccopy) now support an extended format for specifying a sequence of filters. The general form is __<filter>|<filter>..._. * The ncdump **_Filter** attribute now dumps a list of all the filters associated with a variable using the above new format. * Filter specifications can now use a filter name instead of number for filters known to the netcdf library, which in turn is taken from the HDF5 filter registration page. * New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter is returned if an attempt is made to access an unknown filter. * Internally, the dispatch table has been extended to add a function to handle all of the filter functions. * New, filter-related, tests were added to nc_test4. * A new plugin was added to the plugins directory to help with testing. Notes: 1. The shuffle and fletcher32 filters are not part of the multifilter system. Misc. changes: 1. A debug module was added to libhdf5 to help catch error locations.
2020-02-17 03:59:33 +08:00
#ifdef GENDEBUG1
printfilters(special->nfilters,special->_Filters);
#endif
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
return stat;
Add new capabilities to filter code: 1. Allow nccopy to apply filters, especially on the output file. This provides a third way to do this other than using ncgen or programatically 2. Make sure that even if the filter code is not available, it is possible to see the filter id and parameters for variables using e.g ncdump -hs. 3. Fix bug in nccopy so that the input file does not necessarily have to be netcdf-4. 4. At last minute decided to change to using a single "_Filter" attribute for ncgen 5. Added a test to tst_filter.sh to generate C code using ncgen.
2017-05-15 08:10:02 +08:00
}
Add filter support to NCZarr Filter support has three goals: 1. Use the existing HDF5 filter implementations, 2. Allow filter metadata to be stored in the NumCodecs metadata format used by Zarr, 3. Allow filters to be used even when HDF5 is disabled Detailed usage directions are define in docs/filters.md. For now, the existing filter API is left in place. So filters are defined using ''nc_def_var_filter'' using the HDF5 style where the id and parameters are unsigned integers. This is a big change since filters affect many parts of the code. In the following, the terms "compressor" and "filter" and "codec" are generally used synonomously. ### Filter-Related Changes: * In order to support dynamic loading of shared filter libraries, a new library was added in the libncpoco directory; it helps to isolate dynamic loading across multiple platforms. * Provide a json parsing library for use by plugins; this is created by merging libdispatch/ncjson.c with include/ncjson.h. * Add a new _Codecs attribute to allow clients to see what codecs are being used; let ncdump -s print it out. * Provide special headers to help support compilation of HDF5 filters when HDF5 is not enabled: netcdf_filter_hdf5_build.h and netcdf_filter_build.h. * Add a number of new test to test the new nczarr filters. * Let ncgen parse _Codecs attribute, although it is ignored. ### Plugin directory changes: * Add support for the Blosc compressor; this is essential because it is the most common compressor used in Zarr datasets. This also necessitated adding a CMake FindBlosc.cmake file * Add NCZarr support for the big-four filters provided by HDF5: shuffle, fletcher32, deflate (zlib), and szip * Add a Codec defaulter (see docs/filters.md) for the big four filters. * Make plugins work with windows by properly adding __declspec declaration. ### Misc. Non-Filter Changes * Replace most uses of USE_NETCDF4 (deprecated) with USE_HDF5. * Improve support for caching * More fixes for path conversion code * Fix misc. memory leaks * Add new utility -- ncdump/ncpathcvt -- that does more or less the same thing as cygpath. * Add a number of new test to test the non-filter fixes. * Update the parsers * Convert most instances of '#ifdef _MSC_VER' to '#ifdef _WIN32'
2021-09-03 07:04:26 +08:00
/*
Store a Codecs spec string in special
*/
static int
parsecodecsflag(const char* sdata, Specialdata* special)
{
int stat = NC_NOERR;
if(sdata == NULL || strlen(sdata) == 0) return NC_EINVAL;
if((special->_Codecs = strdup(sdata))==NULL)
return NC_ENOMEM;
return stat;
}
This is the initial step in moving to the new higher performance (I hope) metadata mechanism. This mostly just adds new pieces of code (e.g. nclistmap) and does some minor fixes. It should be transparent to everything else. The next set of changes will be the big step.
2018-02-09 10:53:40 +08:00
#endif
Add new capabilities to filter code: 1. Allow nccopy to apply filters, especially on the output file. This provides a third way to do this other than using ncgen or programatically 2. Make sure that even if the filter code is not available, it is possible to see the filter id and parameters for variables using e.g ncdump -hs. 3. Fix bug in nccopy so that the input file does not necessarily have to be netcdf-4. 4. At last minute decided to change to using a single "_Filter" attribute for ncgen 5. Added a test to tst_filter.sh to generate C code using ncgen.
2017-05-15 08:10:02 +08:00
fix NCF-120
2011-09-16 00:57:16 +08:00
/*
2011-09-17 02:36:08 +08:00
Since the arguments are all simple constants,
fix NCF-120
2011-09-16 00:57:16 +08:00
we can evaluate the function immediately
and return its value.
2011-09-17 02:36:08 +08:00
Note that currently, only a single value can
be returned.
fix NCF-120
2011-09-16 00:57:16 +08:00
*/
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
static NCConstant*
fix NCF-120
2011-09-16 00:57:16 +08:00
evaluate(Symbol* fcn, Datalist* arglist)
{
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
NCConstant* result = nullconst();
fix NCF-120
2011-09-16 00:57:16 +08:00
/* prepare the result */
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
result->lineno = fcn->lineno;
fix NCF-120
2011-09-16 00:57:16 +08:00
if(strcasecmp(fcn->name,"time") == 0) {
2011-09-17 02:36:08 +08:00
char* timekind = NULL;
char* timevalue = NULL;
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
result->nctype = NC_DOUBLE;
result->value.doublev = 0;
2011-09-17 02:36:08 +08:00
/* int time([string],string) */
switch (arglist->length) {
case 2:
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
if(arglist->data[1]->nctype != NC_STRING) {
2011-09-17 02:36:08 +08:00
derror("Expected function signature: time([string,]string)");
fix NCF-120
2011-09-16 00:57:16 +08:00
goto done;
}
2011-09-17 02:36:08 +08:00
/* fall thru */
case 1:
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
if(arglist->data[0]->nctype != NC_STRING) {
2011-09-17 02:36:08 +08:00
derror("Expected function signature: time([string,]string)");
fix NCF-120
2011-09-16 00:57:16 +08:00
goto done;
}
2011-09-17 02:36:08 +08:00
break;
case 0:
Addressed UNIX API issues reported by clang.
2015-06-19 04:31:10 +08:00
default:
2011-09-17 02:36:08 +08:00
derror("Expected function signature: time([string,]string)");
goto done;
fix NCF-120
2011-09-16 00:57:16 +08:00
}
2011-09-17 02:36:08 +08:00
if(arglist->length == 2) {
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
timekind = arglist->data[0]->value.stringv.stringv;
timevalue = arglist->data[1]->value.stringv.stringv;
2011-09-17 02:36:08 +08:00
} else
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
timevalue = arglist->data[0]->value.stringv.stringv;
2011-09-17 02:36:08 +08:00
if(timekind == NULL) { /* use cd time as the default */
cdCompTime comptime;
CdTime cdtime;
cdCalenType timetype = cdStandard;
cdChar2Comp(timetype,timevalue,&comptime);
/* convert comptime to cdTime */
Addressed UNIX API issues reported by clang.
2015-06-19 04:31:10 +08:00
cdtime.year = comptime.year;
2011-09-17 02:36:08 +08:00
cdtime.month = comptime.month;
Addressed UNIX API issues reported by clang.
2015-06-19 04:31:10 +08:00
cdtime.day = comptime.day;
2011-09-17 02:36:08 +08:00
cdtime.hour = comptime.hour;
cdtime.baseYear = 1970;
cdtime.timeType = CdChron;
/* convert to double value */
Fix more memory leaks in netcdf-c library This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173 Sorry that it is so big, but leak suppression can be complex. This PR fixes all remaining memory leaks -- as determined by -fsanitize=address, and with the exceptions noted below. Unfortunately. there remains a significant leak that I cannot solve. It involves vlens, and it is unclear if the leak is occurring in the netcdf-c library or the HDF5 library. I have added a check_PROGRAM to the ncdump directory to show the problem. The program is called tst_vlen_demo.c To exercise it, build the netcdf library with -fsanitize=address enabled. Then go into ncdump and do a "make clean check". This should build tst_vlen_demo without actually executing it. Then do the command "./tst_vlen_demo" to see the output of the memory checker. Note the the lost malloc is deep in the HDF5 library (in H5Tvlen.c). I am temporarily working around this error in the following way. 1. I modified several test scripts to not execute known vlen tests that fail as described above. 2. Added an environment variable called NC_VLEN_NOTEST. If set, then those specific tests are suppressed. This should mean that the --disable-utilities option to ./configure should not need to be set to get a memory leak clean build. This should allow for detection of any new leaks. Note: I used an environment variable rather than a ./configure option to control the vlen tests. This is because it is temporary (I hope) and because it is a bit tricky for shell scripts to access ./configure options. Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-16 01:00:38 +08:00
Cdh2e(&cdtime,&result->value.doublev);
2011-09-17 02:36:08 +08:00
} else {
derror("Time conversion '%s' not supported",timekind);
goto done;
}
} else { /* Unknown function */
fix NCF-120
2011-09-16 00:57:16 +08:00
derror("Unknown function name: %s",fcn->name);
goto done;
}
done:
return result;
}
Add support for multiple filters per variable. re: https://github.com/Unidata/netcdf-c/issues/1584 Support has been added for multiple filters per variable. This affects a number of components in netcdf. The new APIs are documented in NUG/filters.md. The primary changes are: * A set of new functions are provided (see __include/netcdf_filter.h__). - Obtain a list of the filters associated with a variable - Obtain the parameters for a specific filter. * The existing __nc_inq_var_filter__ function now returns info about the first defined filter. * The utilities (ncgen, ncdump, and nccopy) now support an extended format for specifying a sequence of filters. The general form is __<filter>|<filter>..._. * The ncdump **_Filter** attribute now dumps a list of all the filters associated with a variable using the above new format. * Filter specifications can now use a filter name instead of number for filters known to the netcdf library, which in turn is taken from the HDF5 filter registration page. * New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter is returned if an attempt is made to access an unknown filter. * Internally, the dispatch table has been extended to add a function to handle all of the filter functions. * New, filter-related, tests were added to nc_test4. * A new plugin was added to the plugins directory to help with testing. Notes: 1. The shuffle and fletcher32 filters are not part of the multifilter system. Misc. changes: 1. A debug module was added to libhdf5 to help catch error locations.
2020-02-17 03:59:33 +08:00
#ifdef GENDEBUG1
static void
Fix the handling of certain alias types on CDL files. re: https://github.com/Unidata/netcdf-c/issues/1977 PR https://github.com/Unidata/netcdf-c/pull/1753, changed ncgen to allows certain type names to be used as identifiers in selected situations. An unwanted side effect was that existing type aliases no longer were accepted by ncgen. Specifically, using the "long" type caused an error. I was able to figure out a better solution to the original problem (https://github.com/Unidata/netcdf-c/issues/1750) that also fixes this problem as well. This PR fixes that problem in ncgen/ncgen.l, and adds tests to ncdump/test_keywords.sh
2021-04-14 06:56:43 +08:00
printfilters(int nfilters, NC_FilterSpec** filters)
Add support for multiple filters per variable. re: https://github.com/Unidata/netcdf-c/issues/1584 Support has been added for multiple filters per variable. This affects a number of components in netcdf. The new APIs are documented in NUG/filters.md. The primary changes are: * A set of new functions are provided (see __include/netcdf_filter.h__). - Obtain a list of the filters associated with a variable - Obtain the parameters for a specific filter. * The existing __nc_inq_var_filter__ function now returns info about the first defined filter. * The utilities (ncgen, ncdump, and nccopy) now support an extended format for specifying a sequence of filters. The general form is __<filter>|<filter>..._. * The ncdump **_Filter** attribute now dumps a list of all the filters associated with a variable using the above new format. * Filter specifications can now use a filter name instead of number for filters known to the netcdf library, which in turn is taken from the HDF5 filter registration page. * New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter is returned if an attempt is made to access an unknown filter. * Internally, the dispatch table has been extended to add a function to handle all of the filter functions. * New, filter-related, tests were added to nc_test4. * A new plugin was added to the plugins directory to help with testing. Notes: 1. The shuffle and fletcher32 filters are not part of the multifilter system. Misc. changes: 1. A debug module was added to libhdf5 to help catch error locations.
2020-02-17 03:59:33 +08:00
{
int i;
fprintf(stderr,"xxx: nfilters=%lu: ",(unsigned long)nfilters);
for(i=0;i<nfilters;i++) {
int k;
This PR adds EXPERIMENTAL support for accessing data in the cloud using a variant of the Zarr protocol and storage format. This enhancement is generically referred to as "NCZarr". The data model supported by NCZarr is netcdf-4 minus the user-defined types and the String type. In this sense it is similar to the CDF-5 data model. More detailed information about enabling and using NCZarr is described in the document NUG/nczarr.md and in a [Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in). WARNING: this code has had limited testing, so do use this version for production work. Also, performance improvements are ongoing. Note especially the following platform matrix of successful tests: Platform | Build System | S3 support ------------------------------------ Linux+gcc | Automake | yes Linux+gcc | CMake | yes Visual Studio | CMake | no Additionally, and as a consequence of the addition of NCZarr, major changes have been made to the Filter API. NOTE: NCZarr does not yet support filters, but these changes are enablers for that support in the future. Note that it is possible (probable?) that there will be some accidental reversions if the changes here did not correctly mimic the existing filter testing. In any case, previously filter ids and parameters were of type unsigned int. In order to support the more general zarr filter model, this was all converted to char*. The old HDF5-specific, unsigned int operations are still supported but they are wrappers around the new, char* based nc_filterx_XXX functions. This entailed at least the following changes: 1. Added the files libdispatch/dfilterx.c and include/ncfilter.h 2. Some filterx utilities have been moved to libdispatch/daux.c 3. A new entry, "filter_actions" was added to the NCDispatch table and the version bumped. 4. An overly complex set of structs was created to support funnelling all of the filterx operations thru a single dispatch "filter_actions" entry. 5. Move common code to from libhdf5 to libsrc4 so that it is accessible to nczarr. Changes directly related to Zarr: 1. Modified CMakeList.txt and configure.ac to support both C and C++ -- this is in support of S3 support via the awd-sdk libraries. 2. Define a size64_t type to support nczarr. 3. More reworking of libdispatch/dinfermodel.c to support zarr and to regularize the structure of the fragments section of a URL. Changes not directly related to Zarr: 1. Make client-side filter registration be conditional, with default off. 2. Hack include/nc4internal.h to make some flags added by Ed be unique: e.g. NC_CREAT, NC_INDEF, etc. 3. cleanup include/nchttp.h and libdispatch/dhttp.c. 4. Misc. changes to support compiling under Visual Studio including: * Better testing under windows for dirent.h and opendir and closedir. 5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags and to centralize error reporting. 6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them. 7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible. Changes Left TO-DO: 1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
NC_Filterspec* sp = filters[i];
Add support for multiple filters per variable. re: https://github.com/Unidata/netcdf-c/issues/1584 Support has been added for multiple filters per variable. This affects a number of components in netcdf. The new APIs are documented in NUG/filters.md. The primary changes are: * A set of new functions are provided (see __include/netcdf_filter.h__). - Obtain a list of the filters associated with a variable - Obtain the parameters for a specific filter. * The existing __nc_inq_var_filter__ function now returns info about the first defined filter. * The utilities (ncgen, ncdump, and nccopy) now support an extended format for specifying a sequence of filters. The general form is __<filter>|<filter>..._. * The ncdump **_Filter** attribute now dumps a list of all the filters associated with a variable using the above new format. * Filter specifications can now use a filter name instead of number for filters known to the netcdf library, which in turn is taken from the HDF5 filter registration page. * New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter is returned if an attempt is made to access an unknown filter. * Internally, the dispatch table has been extended to add a function to handle all of the filter functions. * New, filter-related, tests were added to nc_test4. * A new plugin was added to the plugins directory to help with testing. Notes: 1. The shuffle and fletcher32 filters are not part of the multifilter system. Misc. changes: 1. A debug module was added to libhdf5 to help catch error locations.
2020-02-17 03:59:33 +08:00
fprintf(stderr,"{");
Fix the handling of certain alias types on CDL files. re: https://github.com/Unidata/netcdf-c/issues/1977 PR https://github.com/Unidata/netcdf-c/pull/1753, changed ncgen to allows certain type names to be used as identifiers in selected situations. An unwanted side effect was that existing type aliases no longer were accepted by ncgen. Specifically, using the "long" type caused an error. I was able to figure out a better solution to the original problem (https://github.com/Unidata/netcdf-c/issues/1750) that also fixes this problem as well. This PR fixes that problem in ncgen/ncgen.l, and adds tests to ncdump/test_keywords.sh
2021-04-14 06:56:43 +08:00
fprintf(stderr,"filterid=%llu format=%d nparams=%lu params=%p",
Add support for multiple filters per variable. re: https://github.com/Unidata/netcdf-c/issues/1584 Support has been added for multiple filters per variable. This affects a number of components in netcdf. The new APIs are documented in NUG/filters.md. The primary changes are: * A set of new functions are provided (see __include/netcdf_filter.h__). - Obtain a list of the filters associated with a variable - Obtain the parameters for a specific filter. * The existing __nc_inq_var_filter__ function now returns info about the first defined filter. * The utilities (ncgen, ncdump, and nccopy) now support an extended format for specifying a sequence of filters. The general form is __<filter>|<filter>..._. * The ncdump **_Filter** attribute now dumps a list of all the filters associated with a variable using the above new format. * Filter specifications can now use a filter name instead of number for filters known to the netcdf library, which in turn is taken from the HDF5 filter registration page. * New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter is returned if an attempt is made to access an unknown filter. * Internally, the dispatch table has been extended to add a function to handle all of the filter functions. * New, filter-related, tests were added to nc_test4. * A new plugin was added to the plugins directory to help with testing. Notes: 1. The shuffle and fletcher32 filters are not part of the multifilter system. Misc. changes: 1. A debug module was added to libhdf5 to help catch error locations.
2020-02-17 03:59:33 +08:00
sp->filterid,sp->format,(unsigned long)sp->nparams,sp->params);
This PR adds EXPERIMENTAL support for accessing data in the cloud using a variant of the Zarr protocol and storage format. This enhancement is generically referred to as "NCZarr". The data model supported by NCZarr is netcdf-4 minus the user-defined types and the String type. In this sense it is similar to the CDF-5 data model. More detailed information about enabling and using NCZarr is described in the document NUG/nczarr.md and in a [Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in). WARNING: this code has had limited testing, so do use this version for production work. Also, performance improvements are ongoing. Note especially the following platform matrix of successful tests: Platform | Build System | S3 support ------------------------------------ Linux+gcc | Automake | yes Linux+gcc | CMake | yes Visual Studio | CMake | no Additionally, and as a consequence of the addition of NCZarr, major changes have been made to the Filter API. NOTE: NCZarr does not yet support filters, but these changes are enablers for that support in the future. Note that it is possible (probable?) that there will be some accidental reversions if the changes here did not correctly mimic the existing filter testing. In any case, previously filter ids and parameters were of type unsigned int. In order to support the more general zarr filter model, this was all converted to char*. The old HDF5-specific, unsigned int operations are still supported but they are wrappers around the new, char* based nc_filterx_XXX functions. This entailed at least the following changes: 1. Added the files libdispatch/dfilterx.c and include/ncfilter.h 2. Some filterx utilities have been moved to libdispatch/daux.c 3. A new entry, "filter_actions" was added to the NCDispatch table and the version bumped. 4. An overly complex set of structs was created to support funnelling all of the filterx operations thru a single dispatch "filter_actions" entry. 5. Move common code to from libhdf5 to libsrc4 so that it is accessible to nczarr. Changes directly related to Zarr: 1. Modified CMakeList.txt and configure.ac to support both C and C++ -- this is in support of S3 support via the awd-sdk libraries. 2. Define a size64_t type to support nczarr. 3. More reworking of libdispatch/dinfermodel.c to support zarr and to regularize the structure of the fragments section of a URL. Changes not directly related to Zarr: 1. Make client-side filter registration be conditional, with default off. 2. Hack include/nc4internal.h to make some flags added by Ed be unique: e.g. NC_CREAT, NC_INDEF, etc. 3. cleanup include/nchttp.h and libdispatch/dhttp.c. 4. Misc. changes to support compiling under Visual Studio including: * Better testing under windows for dirent.h and opendir and closedir. 5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags and to centralize error reporting. 6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them. 7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible. Changes Left TO-DO: 1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
if(sp->nparams > 0 && sp->params != NULL) {
Add support for multiple filters per variable. re: https://github.com/Unidata/netcdf-c/issues/1584 Support has been added for multiple filters per variable. This affects a number of components in netcdf. The new APIs are documented in NUG/filters.md. The primary changes are: * A set of new functions are provided (see __include/netcdf_filter.h__). - Obtain a list of the filters associated with a variable - Obtain the parameters for a specific filter. * The existing __nc_inq_var_filter__ function now returns info about the first defined filter. * The utilities (ncgen, ncdump, and nccopy) now support an extended format for specifying a sequence of filters. The general form is __<filter>|<filter>..._. * The ncdump **_Filter** attribute now dumps a list of all the filters associated with a variable using the above new format. * Filter specifications can now use a filter name instead of number for filters known to the netcdf library, which in turn is taken from the HDF5 filter registration page. * New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter is returned if an attempt is made to access an unknown filter. * Internally, the dispatch table has been extended to add a function to handle all of the filter functions. * New, filter-related, tests were added to nc_test4. * A new plugin was added to the plugins directory to help with testing. Notes: 1. The shuffle and fletcher32 filters are not part of the multifilter system. Misc. changes: 1. A debug module was added to libhdf5 to help catch error locations.
2020-02-17 03:59:33 +08:00
fprintf(stderr," params={");
for(k=0;k<sp->nparams;k++) {
if(i==0) fprintf(stderr,",");
Fix the handling of certain alias types on CDL files. re: https://github.com/Unidata/netcdf-c/issues/1977 PR https://github.com/Unidata/netcdf-c/pull/1753, changed ncgen to allows certain type names to be used as identifiers in selected situations. An unwanted side effect was that existing type aliases no longer were accepted by ncgen. Specifically, using the "long" type caused an error. I was able to figure out a better solution to the original problem (https://github.com/Unidata/netcdf-c/issues/1750) that also fixes this problem as well. This PR fixes that problem in ncgen/ncgen.l, and adds tests to ncdump/test_keywords.sh
2021-04-14 06:56:43 +08:00
fprintf(stderr,"%u",sp->params[i]);
Add support for multiple filters per variable. re: https://github.com/Unidata/netcdf-c/issues/1584 Support has been added for multiple filters per variable. This affects a number of components in netcdf. The new APIs are documented in NUG/filters.md. The primary changes are: * A set of new functions are provided (see __include/netcdf_filter.h__). - Obtain a list of the filters associated with a variable - Obtain the parameters for a specific filter. * The existing __nc_inq_var_filter__ function now returns info about the first defined filter. * The utilities (ncgen, ncdump, and nccopy) now support an extended format for specifying a sequence of filters. The general form is __<filter>|<filter>..._. * The ncdump **_Filter** attribute now dumps a list of all the filters associated with a variable using the above new format. * Filter specifications can now use a filter name instead of number for filters known to the netcdf library, which in turn is taken from the HDF5 filter registration page. * New errors are defined: NC_EFILTER and NC_ENOFILTER. The latter is returned if an attempt is made to access an unknown filter. * Internally, the dispatch table has been extended to add a function to handle all of the filter functions. * New, filter-related, tests were added to nc_test4. * A new plugin was added to the plugins directory to help with testing. Notes: 1. The shuffle and fletcher32 filters are not part of the multifilter system. Misc. changes: 1. A debug module was added to libhdf5 to help catch error locations.
2020-02-17 03:59:33 +08:00
}
fprintf(stderr,"}");
} else
fprintf(stderr,"params=NULL");
fprintf(stderr,"}");
}
fprintf(stderr,"\n");
fflush(stderr);
}
#endif
Reference in New Issue Copy Permalink