Commit Graph

363 Commits

Author SHA1 Message Date
Dennis Heimbigner
c9d16d82d6 Fix cmake X mmap
supercede PR: https://github.com/Unidata/netcdf-c/pull/1384

Since we have an mmap user, undeprecate it and make sure
it works. Other changes:

* fix test cases to work with make -j
* fix exposed ncgen error.
2019-04-19 20:32:26 -06:00
Ward Fisher
0c0b95da67 Merge branch 'master' into filemacroncgen.dmh 2019-03-08 14:04:57 -07:00
Dennis Heimbigner
98caf87116 Fix ncgen handling of octal constants (with leading 0).
re: https://github.com/Unidata/netcdf-c/issues/1330

The ncgen utility is documented to accept octal integer constants
if the leading digit is zero. This was not implemented. Fix ncgen.l
to properly handle such constants. Also add a test to c0.cdl.
2019-02-25 20:57:23 -07:00
Dennis Heimbigner
ad3e3bd48e Remove all references to __FILE__ in ncgen
re: issue https://github.com/Unidata/netcdf-c/issues/1323
    and PR https://github.com/Unidata/netcdf-c/pull/1326

Replace the uses of the __FILE__ macro in ncgen with the
constant string "ncgen" in order to keep external package
builders (like Debian) happy.

Note that __FILE__ will still appear in ncgen generated C code
files, but this should be ok since it is not part of the distro
per-se.
2019-02-17 14:22:30 -07:00
Dennis Heimbigner
8714066b18 Fix errors when building on big-endian machine
re: issue https://github.com/Unidata/netcdf-c/issues/1278
re: issue https://github.com/Unidata/netcdf-c/issues/876
re: issue https://github.com/Unidata/netcdf-c/issues/806

* Major change to the handling of 8-byte parameters for nc_def_var_filter.
  The old code was not well thought out.
  * The new algorithm is documented in docs/filters.md.
  * Added new utility file plugins/H5Zutil.c to support
  * Modified plugins/H5Zmisc.c to use new algorithm
  the new algorithm.
  * Renamed include/ncfilter.h to include/netcdf_filter.h
    and made it an installed header so clients can access the
    new algorithm utility.
  * Fixed nc_test4/tst_filterparser.c and nc_test4/test_filter_misc.c
    to use the new algorithm
* libdap4/ fixes:
  * d4swap.c has an error in the endian pre-processing such
    that record counts were not being swapped correctly.
  * d4data.c had an error in that checksums were being computed
    after endian swapping rather than before.
* ocinitialize() was never being called, so xxdr bigendian handling
  was never set correctly.
  * Required adding debug statements to occompile
* Found and fixed memory leak in ncdump.c

Not tested:
* HDF4
* Pnetcdf
* parallel HDF5
2019-01-31 21:13:06 -07:00
Ward Fisher
d66d642e24 Corrected an issue observed on OSX 2019-01-15 14:36:07 -07:00
Ward Fisher
c21820a1ae Merge branch 'master' into ansifix.dmh 2019-01-02 22:26:59 -08:00
Ward Fisher
94af1d49c6
Merge branch 'master' into ansicomment.dmh 2019-01-02 20:51:28 -08:00
Ward Fisher
c62d7eb541 Merge remote-tracking branch 'origin/gh1233.dmh' into pr-aggregation.wif 2018-12-30 23:55:30 -08:00
Dennis Heimbigner
75759ca957 Separate out the --ansi comment fixes.
re: pull request https://github.com/Unidata/netcdf-c/pull/1242

This pr should be applied before https://github.com/Unidata/netcdf-c/pull/1242.
It fixes only the -ansi '//' comment problems. There may be some
slight conflicts with that other pr when it is applied, since in some
cases I converted #if 0...#endif to /*...*/
2018-12-12 13:23:09 -07:00
Dennis Heimbigner
735ae80928 merge master and fix conflicts 2018-12-12 11:47:54 -07:00
Ward Fisher
30ea33435c Merge remote-tracking branch 'origin/license_update.wif' into pr-aggregation.wif 2018-12-11 17:08:21 -05:00
Ward Fisher
d6b480cec1 Merge branch 'ejh_next_23' of https://github.com/NetCDF-World-Domination-Council/netcdf-c into pr-aggregation.wif 2018-12-11 17:06:56 -05:00
Ward Fisher
7fd7696940 ncgen directory updated 2018-12-06 15:40:43 -07:00
Dennis Heimbigner
1a7531392f Make the netcdf-c library compile with gcc -ansi.
Primary fixes to get -ansi to work.
1. Convert all '//' C++ style comments to /*...*/ or to use #if 0...#endif
2. It turns out that when -ansi is specified, then a number of
   functions no longer are defined in the header -- but they are still
   in the .so file.<br>
   The big example is strdup(). So, added code to include/ncconfig.h to define
   externs for those missing functions that occur in more than one place.
   These are enabled if !_WIN32 && __STDC__ == 1 (__STDC__ is supposed to
   be the equivalent compile time flag to -ansi). Note that this requires
   config.h (which references ncconfig.h) to be included in files where it is
   currently not included. Single uses will be only in the file that uses them.
3. Added mmap test for the MAP_ANONYMOUS flag to configure.ac. Apparently
   this is not always defined with -ansi.
4. fix some large integer constants in nc_test4/tst_atts3.c and nc_test4/tst_filterparser.c
   to avoid compiler complaints.
5. fix a double constant in nc_test4/tst_filterparser.c to avoid compiler complaints.

[Note I suspect #4 and #5 will be a problem on big-endian machines, but we have no way to test]

Misc. Changes:
1. convert more instances of _MSC_VER to _WIN32.
2. added some debugging code to include/nctestserver.h
3. added comment about libdispatch/drc.c always being compiled.
4. modify parser generation in ncgen to remove unneeded files.
2018-12-05 19:20:43 -07:00
Dennis Heimbigner
30d63bbaa5 Fix failings of nc_test/run_inmemory.sh
re: issue https://github.com/Unidata/netcdf-c/issues/1233

Changes:

1. remove exit that was there for testing.
2. the program tst_open_mem must be netcdf-4 only.
3. fix some diff problems
   - Change dataset name for tst_inmemory4_create to tst_inmemory4
   - Modify tst_inmemory.c to reorder the variables (somewhat major rewrite)

Minor Unrelated Fixes:
1. fix comment problem in nc_provenance.h
2. Fix memory leak in tst_open_mem.c
3. fix ncdump/bindata.c to properly compile if netcdf4 is disabled.
4. minor changes to ncgen.l
2018-12-04 19:22:02 -07:00
Ed Hartnett
bbfbd98f6e test and util changes to split libsrc4 and libhdf5 2018-12-01 08:24:56 -07:00
Dennis Heimbigner
0db7b873ef Remove // style comments from source code.
We need to do this periodically because they sneek in.
2018-11-16 11:56:32 -07:00
Ward Fisher
9dd9a19142 Merge remote-tracking branch 'origin/ncgenmem.dmh' into v4.6.2-release-branch.wif 2018-11-15 10:51:59 -07:00
Dennis Heimbigner
b89e3f50d0 Fix introduced alert https://lgtm.com/projects/g/Unidata/netcdf-c/rev/pr-5fdf9cb537c1a981abb3838fdfb061b1088636f0
by removing offending, unused global variable.
2018-11-15 10:31:36 -07:00
Ward Fisher
53dc9022be Merge remote-tracking branch 'origin/ncgenmem.dmh' into tmptmp 2018-11-15 10:20:08 -07:00
Dennis Heimbigner
751300ec59 Fix more memory leaks in netcdf-c library
This is a follow up to PR https://github.com/Unidata/netcdf-c/pull/1173

Sorry that it is so big, but leak suppression can be complex.

This PR fixes all remaining memory leaks -- as determined by
-fsanitize=address, and with the exceptions noted below.

Unfortunately. there remains a significant leak that I cannot
solve. It involves vlens, and it is unclear if the leak is
occurring in the netcdf-c library or the HDF5 library.

I have added a check_PROGRAM to the ncdump directory to show the
problem.  The program is called tst_vlen_demo.c To exercise it,
build the netcdf library with -fsanitize=address enabled. Then
go into ncdump and do a "make clean check".  This should build
tst_vlen_demo without actually executing it.  Then do the
command "./tst_vlen_demo" to see the output of the memory
checker.  Note the the lost malloc is deep in the HDF5 library
(in H5Tvlen.c).

I am temporarily working around this error in the following way.
1. I modified several test scripts to not execute known vlen tests
   that fail as described above.
2. Added an environment variable called NC_VLEN_NOTEST.
   If set, then those specific tests are suppressed.

This should mean that the --disable-utilities option to
./configure should not need to be set to get a memory leak clean
build.  This should allow for detection of any new leaks.

Note: I used an environment variable rather than a ./configure
option to control the vlen tests. This is because it is
temporary (I hope) and because it is a bit tricky for shell
scripts to access ./configure options.

Finally, as before, this only been tested with netcdf-4 and hdf5 support.
2018-11-15 10:00:38 -07:00
Greg Sjaardema
cd011cd1bb Eliminate compiler warnings (and code error) 2018-11-14 11:00:47 -07:00
Dennis Heimbigner
245961de00 re: github issues
https://github.com/Unidata/netcdf-c/issues/1168
    https://github.com/Unidata/netcdf-c/issues/1163
    https://github.com/Unidata/netcdf-c/issues/1162

This PR partially fixes memory leaks in the netcdf-c library,
in the ncdump utility, and in some test cases.

The netcdf-c library now runs memory clean with the assumption
that the --disable-utilities option is used. The primary remaining
problem is ncgen. Once that is fixed, I believe the netcdf-c library
will run memory clean with no limitations.

Notes
-----------
1. Memory checking was performed using gcc -fsanitize=address.
   Valgrind-based testing has yet to be performed.
2. The pnetcdf, hdf4, and examples code has not been tested.

Misc. Non-leak changes
1. Make tst_diskless2 only run when netcdf4 is enabled (issue 1162)
2. Fix CmakeLists.txt to turn off logging if ENABLE_NETCDF_4 is OFF
3. Isolated all my debug scripts into a single top-level directory
   called debug
4. Fix some USE_NETCDF4 dependencies in nc_test and nc_test4 Makefile.am
2018-10-30 20:48:12 -06:00
Ward Fisher
784d777bff Merge branch 'master' into provenance.dmh 2018-09-06 15:13:09 -06:00
Ward Fisher
5e69b0a52d
Merge branch 'master' into patch-17 2018-09-04 11:21:47 -06:00
Dennis Heimbigner
2ea1cf5f1b There was a request to extend the provenance information
stored in the _NCProperties attribute to allow two things:
1. capture of additional library dependencies (over and above
   hdf5)
2. Recognition of non-netcdf libraries that create netcdf-4 format
   files.

To this end, the _NCProperties format has been extended to be
and arbitrary set of key=value pairs separated by commas.
This new format has version = 2, and uses commas as the pair separator.
Thus the general form is:
    _NCProperties = "version=2,key1=value,key2=value2..." ;

This new version is accompanied by a new ./configure option of the form
    --with-ncproperties="key1=value1,key2=value2..."
that specifies pairs to add to the _NCProperties attribute for all
files created with that netcdf library.

At this point, what is missing is some programmatic way to
specify either all the pairs or additional pairs
to the _NCProperties attribute. Not sure of the best way
to do this.

Builders using non-netcdf libraries can specify
whatever they want in the key value pairs (as long
as the version=2 is specified first).

By convention, the primary library is expected to be the
the first pair after the leading version=2 pair, but this
is convention only and is neither required nor enforced.

Related changes:
1. Fixed the tests that check _NCProperties to properly operate with version=2.
2. When reading a version 1 _NCProperties attribute, convert it to look
   like a version 2 attribute.
2. Added some version 2 tests to ncdump/tst_fileinfo.c and
   ncdump/tst_fileinfo.sh

Misc Changes:
1. Fix minor problem in ncdap_test/testurl.sh where a parameter to
   buildurl needed to be quoted.
2. Minor fix to ncgen to swap switches -H and -h to be consistent
   with other utilities.
3. Document the -M flag in nccopy usage() and the nccopy man page.
4. Modify a test case to use the nccopy -M flag.
2018-08-25 21:44:41 -06:00
Ward Fisher
a996ed554e Swapped /bin/bash for /bin/sh to test on osx. 2018-08-12 23:01:08 -06:00
Greg Sjaardema
0d8cee251b
Merge branch 'master' into patch-17 2018-07-09 09:15:58 -04:00
Wei-keng Liao
f95d3e3325 replace USE_CDF5 with ENABLE_CDF5 2018-06-29 21:17:07 -05:00
Ed Hartnett
7984eee043
Merge branch 'master' into ejh_streq 2018-06-11 15:50:22 -06:00
Ward Fisher
df4942d280 Merge branch 'master' into vars.dmh 2018-06-08 15:50:28 -06:00
Ward Fisher
335f25e947 Corrected a couple of potential memory related issues and an uninitialized variable issue. 2018-06-08 11:38:11 -06:00
Dennis Heimbigner
ee509ff4f3 Re-Implement the nc_get/put_vars operations for netcdf-4 using the
corresponding HDF5 operations.

re: github issue https://github.com/Unidata/netcdf-c/issues/908
also in reference to https://github.com/pydata/xarray/issues/2004

The netcdf-c library has implemented the nc_get_vars and nc_put_vars
operations as element at a time. This has resulted in very slow
operation.

This pr attempts to improve the situation for netcdf-4/hdf5 files
by using the slab operations provided by the hdf5 library. The new
implementation passes the get/put vars stride information down to
the hdf5 slab operations.

The result appears to improve performance significantly. Some simple
tests on large 2-D arrays shows speedups in excess of 150.

Misc. other changes:
1. fix bug in ncgen/semantics.c; using a list's allocated length
   instead of actual length.
2. Added a temporary hook in the netcdf library plus a performance
   test case (tst_varsperf.c) to estimate the speedup. After users
   have had some experience with this, I will remove it, probably
   after the 4.7 release.
2018-05-22 16:50:52 -06:00
Ward Fisher
ec1437dd09
Merge branch 'master' into patch-17 2018-05-21 13:36:55 -06:00
Ed Hartnett
b03277caea
Merge branch 'master' into ejh_streq 2018-05-17 04:26:59 -06:00
Ward Fisher
a8c4c0aa68 Merge remote-tracking branch 'origin/cleanncgen.dmh' into small-pr-aggregate.wif 2018-05-16 14:32:10 -06:00
Ward Fisher
c9d1589c39
Merge branch 'master' into cleanncgen.dmh 2018-05-16 14:31:11 -06:00
Ed Hartnett
0c0d066927 changed macro STREQ to NCSTREQ to avoid name collusion with HDF4 library 2018-05-12 08:55:51 -06:00
Greg Sjaardema
7069ee7232
Merge branch 'master' into patch-17 2018-05-07 09:52:53 -06:00
Greg Sjaardema
33a88b5de6
Fix compression logic
Level is unsigned, so test "level >= 0 ? 1:0" is the exact same as "1"
2018-05-07 08:34:41 -06:00
luz.paz
74fbacdb82 Misc. source comment typos
Some are user-facing. Found via `codespell` and through the downstream FreeCAD.
2018-04-26 23:04:01 -04:00
Ward Fisher
7faa84b34c
Merge branch 'master' into patch-17 2018-04-23 13:38:03 -06:00
Dennis Heimbigner
867e51145b
Fix missing return statement identified by lgtm 2018-04-23 10:54:44 -06:00
Ward Fisher
ee7bbb2320 Debugging pull request for OSX. 2018-04-20 14:40:28 -06:00
Ward Fisher
0f36b279be Merge remote-tracking branch 'origin/cleanncgen.dmh' into pr-aggregate.wif 2018-04-20 13:06:26 -06:00
Ward Fisher
480b644658
Merge branch 'master' into patch-17 2018-03-07 12:25:20 -07:00
Ward Fisher
bee7413989 Corrected a buffer overrun error in genlib.c 2018-02-27 15:00:49 -07:00
Dennis Heimbigner
727b613459 This is the initial step in moving to the new higher performance
(I hope) metadata mechanism. This mostly just adds new pieces of
code (e.g. nclistmap) and does some minor fixes.

It should be transparent to everything else.
The next set of changes will be the big step.
2018-02-08 19:53:40 -07:00
Greg Sjaardema
7933e456f8
Fix compression level logic
* level is unsigned, can never be < 0
* if level == 0, do not enable deflate filter.
2018-01-31 10:04:00 -07:00