Commit Graph

128 Commits

Author SHA1 Message Date
Greg Sjaardema
c84b475ccf Remove var linked list 2016-11-16 08:45:10 -07:00
Greg Sjaardema
dee1baca8e Store vars in array instead of linked list (linked list still active) 2016-11-16 08:45:06 -07:00
Dennis Heimbigner
7e0db68dce Finally get around to removing all that
obsolete pnetcdf related code in libsrc4.
2016-05-14 22:31:41 -06:00
Dennis Heimbigner
11a259ad86 Add provenance info for netcdf-4 files.
This consists of a persistent attribute named
_NCProperties plus two computed attributes
_IsNetcdf4 and _SuperblockVersion.
See the 'Provenance Attributes' section
of docs/attribute_conventions.md for details.
2016-05-07 14:32:07 -06:00
Ward Fisher
0bb9856880 Merge branch 'upstream' of https://github.com/gsjaardema/netcdf-c into gs-pulls 2016-03-04 14:58:10 -07:00
Greg Sjaardema
4ccebf25b5 Use dim field of var instead of finding dim from var->dimids.
The var struct has a 'dim' field which was not being used
Instead, the dimids field would always search for the dim
with the matching dimid.  For db with large numbers of dims,
this could be a significant time sync.

Modified code to always set var-dim[i] when var->dimids[i] was
set (if the dim existed at that point).  Then use the var->dim
field instead of var->dimids and search whenever requested.

All var->dim accesses are protected by asserts that verify
non-null and that the var->dim[]->dimid == var->dimids[].
2016-03-04 10:45:36 -07:00
Greg Sjaardema
1a84a6a99e Add hash field to dim and var to facilitate fast name compare
In non-classic netcdf-4 models, it is allowable to have
large numbers of dims and vars.  In many operations, the
entire list of dims or vars is searched for a dim/var matching
a specific name which results in *lots* of strncmp or strcmp
calls.

If we add a hash field to the var and dim structs similar to what
has already been done for the netcdf-3 formats, then we can hash the
name being searched for and numerically compare that value with
the var/dim hash value.  If they match, then do a more expensive
strncmp call to ensure that the names truly match.
2016-03-03 13:18:31 -07:00
Dennis Heimbigner
45572f5971 Fix github issue: https://github.com/Unidata/netcdf-c/issues/208
Return an error when specifying deflation (compression) or fletcher32 on
a file created for parallel IO in netcdf-4.
2016-02-01 16:15:58 -07:00
Ward Fisher
bb00562779 Addressed a static-analysis issue. 2015-12-31 11:47:39 -07:00
Ward Fisher
c1210f4020 Merge branch 'master' into cdf5-sync-master 2015-11-09 13:45:11 -07:00
dmh
5ad26bb68f Fix github issues #140
1. Added check to libsrc4/nc4var.nc_def_var_extra to
   check that the no specified chunks size is greater than
   the dimension size.

2. Added test to nc_test4/tst_chunks.c
2015-11-07 20:29:16 -07:00
dmh
49597a64af merge-squash 2015-10-09 10:12:11 -06:00
Russ Rew
d3d442537d Fix 1D variables with an unlimited dimension taking DEFAULT_CHUNK_SIZE (4MiB), by default, in netCDF-4 files 2015-09-29 13:58:51 -06:00
Quincey Koziol
8769d58b1d Initial fix for further rename issue. 2014-11-24 09:36:58 -06:00
dmh
ed068ef643 Fix misc. typos from previous commit 2014-06-02 14:02:34 -06:00
dmh
909940f9d3 #endfi -> #endif 2014-06-02 13:25:53 -06:00
dmh
83ec39f5db [NCF-272]
re: e support ZCL-340681 and CPW-270700

HDF4 supports compression (and chunking)
but the chunking was not being recorded
for HDF4 files. So, I modified the necessary
files to support HDF4 chunking.
2014-06-02 13:04:28 -06:00
Ward Fisher
76e5e0f1fa Addressed several minor issues revealed by static analysis (coverity, clang). Mostly to do with extreme corner cases in which a null pointer would be dereferenced, or dereferencing a non-null (but non-initialized) pointer would result in garbage. 2014-03-28 17:11:26 -06:00
Russ Rew
7d36a6c7f3 Fix a couple of warnings 2014-03-27 11:11:06 -06:00
Russ Rew
b95ab08f8f Fix bug in setting default chunksize for 1D record variables 2014-03-27 09:58:53 -06:00
Russ Rew
93878e8816 Fix possibility of divide by zero in rechunking. 2014-02-27 12:19:14 -07:00
Russ Rew
9212968f54 Fix default chunksize for 1-dimensional record variables 2014-02-26 13:30:12 -07:00
Quincey Koziol
d033f9f6ff Clean up Coverity warnings from last checkin. 2014-02-20 08:24:55 -06:00
Quincey Koziol
36c960c0ca Add a few cases for NC_CHAR type class that were missed earlier. 2014-02-12 08:48:13 -06:00
Quincey Koziol
b2dfacbcfa Big clean up to type handling in libsrc4, which makes fill-values work
correctly for variables with string datatype, plus a few other minor changes.
2014-02-11 17:12:08 -06:00
Quincey Koziol
cd71eb525c Corrected "BAIL" macros to avoid infinite loop when logging is disabled and an
error occurs after an "exit:" label.

Corrected a dozen Coverity errors (mainly allocation issues, along with a few
    other things):
        711711, 711802, 711803, 711905, 970825, 996123, 996124, 1025787,
        1047274, 1130013, 1130014, 1139538

Refactored internal fill-value code to correctly handle string types, and
    especially to allow NULL pointers and null strings (ie. "") to be
    distinguished.  The code now avoids partially aliasing the two together
    (which only happened on the 'write' side of things and wasn't reflected on
    the 'read' side, adding to the previous confusion).

    Probably still weak on handling fill-values of variable-length and compound
    datatypes.

Refactored the recursive metadata reads a bit more, to process HDF5 named
    datatypes and datasets immediately, avoiding chewing up memory for those
    types of objects, etc.

Finished uncommenting and updating the nc_test4/tst_fills2.c code (as I'm
    proceeding alphabetically through the nc_test4 code files).
2013-12-29 01:12:43 -06:00
Quincey Koziol
b3044de434 Refactored read_scale(), memio_new(), var_create_dataset() and makespecial()
to clean up resources properly on failure.

Refactored doubly-linked list code for objects in the libsrc4 directory,
    cleaning up the add/del routines, breaking out the common next/prev
    pointers into a struct and extracting the add/del operations on them,
    changed the list of dims to add new dims in the same order as the other
    types, made all add routines able to optionally return a pointer to the
    newly created object.

Removed some dead code (pg_var(), nc4_pg_var1(), nc4_pg_varm(), misc. small
    routines, etc)

Fixed fill value handling for string types in nc4_get_vara().

Changed many malloc()+strcpy() pairs into calls to strdup().

Cleaned up misc. other minor Coverity issues.
2013-12-08 03:29:26 -06:00
Quincey Koziol
5ccb8a24b5 Merge branch 'master' into ncf-177 2013-11-30 23:40:26 -06:00
Quincey Koziol
e1fc13b215 Many changes to address NCF-177 (renaming dimensions and variables). Also
many cleanups to fix compiler warnings, streamline iteration over objects
in HDF5 file when opening the file, and generally straightening out the code
to be cleaner and simpler.

Tested on Mac OS/X with gcc 4.8 and OpenMPI (which uses clang).
2013-11-30 23:20:28 -06:00
dmh
2bc308432d sync with oc2.0 2013-11-14 15:13:20 -07:00
Russ Rew
751ac82ef7 For chunk computations that might overflow ints, use double instead of
float.
2013-10-17 14:25:39 -06:00
Russ Rew
e4f9350c23 Quincey's fixes for NCF-56, "netcdf-4 can't read HDF5 scalar atts".
Also added ability capability for netCDF-4 to write and read NIL
values for string type attributes and variables, so these can be read
if used in HDF5 files.

Include are additions to CMakeLists files to reflect new tests.
2013-07-10 20:09:31 +00:00
Russ Rew
3152893da0 Updated documentation on default chunk size for unlimited dimensions.
Removed old heuristic for dims several orders of magnitude smaller
than the max dimension.
2013-05-28 15:34:55 +00:00
Dennis Heimbigner
c659633d2e merge from trunk 2013-03-26 16:45:02 +00:00
Russ Rew
2071ca9bf3 Fixed NCF-244, a netCDF-4 bug resulting in two dimensions with the same dimension ID 2013-03-25 18:06:19 +00:00
Dennis Heimbigner
dea3c726c8 merge trunk into this branch 2013-03-15 20:31:07 +00:00
Ward Fisher
5703c0979e Corrected an issue I introduced which broke a particular parallel test. 2013-02-12 21:27:52 +00:00
Ward Fisher
c20baac22b Removed dependency on NC_MAX_DIMS from libsrc4 files.
Improved HDF4 suport in CMakeLists.txt
Remaining files were not changed, but svn picks up
'property changes' on them.
2013-01-28 19:58:06 +00:00
Ward Fisher
19dcece866 Merged latest trunk, addressed NC_MAX_DIMS dependence in libsrc4 2013-01-28 18:31:02 +00:00
Dennis Heimbigner
0d8c0b9276 more pnetcdf fixes 2013-01-18 02:25:12 +00:00
Russ Rew
22b411a17c Merged in pnetcdf fixes from Greg Sjardema, to make set_fill_mode work
with parallel-netcdf, to fully implement parallel-netcdf support for
other functions, and to prevent a hang in hdf5 from an eary return in
an nc4_put_vara() call.  Also fixed an nccopy bug when
nc_inq_var_deflate() returns defalate_level of 0, but says the variable
is deflated.
2013-01-15 03:46:46 +00:00
Dennis Heimbigner
635a03f2f7 It turns out that when I converted the code
to keep the NC structure pure and independent
of any file format, I neglected to check with
--enable-hdf4. These changes appear to fix that
problem.
2012-12-13 18:22:11 +00:00
Dennis Heimbigner
5f2eb8afbf Fix a number of potential problems by changing calls to nc_XXX to NC3/4_XXX 2012-12-12 20:05:06 +00:00
Dennis Heimbigner
5ca78309cc The effect of this change is to make the struct NC structure
contain as little file-type specific info as possible.  It
modifies especially libsrc so that all of the netcdf-3 data
that used to be in struct NC is now kept in a separate chunk
of data pointed to by the struct NC. This makes all of
current protocols consistent: netcdf-3, netcdf-4, and dap.
2012-09-06 19:44:03 +00:00
Russ Rew
8d53da4826 Fixed bug "adding a bad _FillValue" bug (NCF-190). Fixed minor bugs
reported by static analysis, including memory leak in ncdump, missing
size_t cast for chunk cache.  Fixed various doc problems, including
byte vs. char issues, missing NC_UBYTE in type list, needed link to
"Building with Windows" page.
2012-08-16 18:31:48 +00:00
Russ Rew
c6f399c731 Fix non-portable test that depends on nonstandardized floating-point format using e+08, not e+008. Fixes for some problems reported in scan-build static analysis. 2012-04-23 23:59:24 +00:00
Dennis Heimbigner
aebd11348a 1)Integrate the oc with improved performance 2012-01-29 18:56:29 +00:00
Russ Rew
e7b2f2b4c6 Revert last change after discovering problem 2012-01-17 22:49:01 +00:00
Russ Rew
67304dc7e2 Fix Jira issue NCF-138, bad default chunksize for variables with only unlimited dimensions 2012-01-17 21:58:09 +00:00
Ed Hartnett
8945e92708 fixed some missing config.h problems 2011-09-26 10:14:55 +00:00
Russ Rew
b3b7861756 Fix ncdump/tst_compress bug that shows up on Solaris systems. Fix default variable chunksize algorithm. 2011-06-23 17:15:01 +00:00
Ed Hartnett
965a3aac70 minor refactor of the build system to work better for cross-compiling 2011-03-15 10:19:08 +00:00
Ed Hartnett
ca6b34a92c changes to only run tst_many_atts when benchmarks are turned on, and to fix minor bug: handling of a scalar dimension and scalar var of same name 2011-02-23 17:33:11 +00:00
Ed Hartnett
2f9339e3af fixed integer overflow for ginormous data sizes on 32-bit build 2011-02-15 15:47:10 +00:00
Ed Hartnett
8d91b49ad2 added overhang (not hangover) calculations to default chunksizes 2011-02-14 19:15:36 +00:00
Ed Hartnett
3340b641fd fixed bug that allowed for zero chunksize when one small, and two giant dimension sizes are used 2011-02-14 16:49:05 +00:00
Ed Hartnett
6b0feecb05 some improvements to chunking algorithm 2011-02-09 12:23:09 +00:00
Ed Hartnett
d6a3fb724c fix bug in checking magnitudes of dims when determining default chunk sizes, in the case of unlimited dimensions. 2011-02-08 19:35:33 +00:00
Ed Hartnett
5da7f43a3f more changes for extreme dimension sizes and default chunking 2011-02-04 20:00:22 +00:00
Ed Hartnett
b47678006e changed default chunksize calculations so they will not have integer overflows, even on 32-bit machines with huge datasets 2011-02-04 17:17:08 +00:00
Ed Hartnett
c03aabe083 fixed default chunking for some really big dimensions 2011-02-03 16:50:51 +00:00
Ed Hartnett
668ed2e0a5 fixed chunking bug: default chunks must always be under 4GB in size 2011-02-02 14:09:15 +00:00
Ed Hartnett
bb07a21f1f added chunksize test code from Russ to help develop better default chunks 2011-01-31 22:15:37 +00:00
Ed Hartnett
1d86c471a6 fixed def_var_chunking bug when chunksizes are not provided, but chunking is turned on 2011-01-31 21:30:39 +00:00
Ed Hartnett
2ebb1cbafc fixed problems with parallel I/O of classic files with pnetcdf library 2011-01-28 12:45:52 +00:00
Ed Hartnett
03f63a5f1c many changes for memory fixes 2010-11-29 22:23:16 +00:00
Ed Hartnett
d53c6bdcfc fixed bug with nf90_inquire_variable returning range error because of chunksizes for a contiguous variable 2010-09-22 14:25:42 +00:00
Russ Rew
4ce85da310 nccopy changes to support compression 2010-08-29 15:08:12 +00:00
Dennis Heimbigner
c5cb1104b7 removed USE_DISPATCH and most uses of nc3 from libsrc4 2010-07-30 19:20:22 +00:00
Ed Hartnett
45a6f02d7c some changes to get build working on bluefire 2010-07-09 20:57:09 +00:00
Ed Hartnett
bc6c5a071b took out more per-variable memory for netCDF-4 files 2010-07-01 12:52:44 +00:00
Ed Hartnett
0e855e0dc5 more changes to make netcdf-4 use less memory per variable 2010-07-01 11:39:34 +00:00
Ed Hartnett
2ca9240913 removed more memory hogging arrays from NC_VAR_INFO_T to improve netcdf-4 memory performance 2010-06-30 21:16:50 +00:00
Ed Hartnett
6db57b7f09 made var->chunksizes array allocated intead of fixed size, to improve memory performance of netCDF-4 2010-06-30 21:05:11 +00:00
Ed Hartnett
f0a72cbefb fixed broken classic-only build 2010-06-28 17:17:43 +00:00
Ed Hartnett
a2593be245 some changes 2010-06-22 17:12:29 +00:00
Ed Hartnett
e5512f4e56 removed v2 test in nc_test4 when --enable-c-only is used 2010-06-22 10:18:15 +00:00
Ed Hartnett
18f4bca367 moving to trunk subdir 2010-06-03 13:24:43 +00:00