Commit Graph

72 Commits

Author SHA1 Message Date
Ward Fisher
05818ac990 Misc. files updated with copyright stanza. 2018-12-06 15:51:35 -07:00
Wei-keng Liao
0ed70756cc Ignore flags NC_MPIIO and NC_MPIPOSIX. 2018-09-22 20:22:34 -05:00
Ward Fisher
784d777bff Merge branch 'master' into provenance.dmh 2018-09-06 15:13:09 -06:00
Dennis Heimbigner
79e38de840 Add the ability to set some additional curlopt values
Add the ability to set some additional curlopt values via .daprc (aka .dodsrc).
This effects both DAP2 and DAP4 protocols.

Related issues:
[1] re: esupport: KOZ-821332
[2] re: github issue https://github.com/Unidata/netcdf4-python/issues/836
[3] re: github issue https://github.com/Unidata/netcdf-c/issues/1074

1. CURLOPT_BUFFERSIZE: Relevant to [1]. Allow user to set the read/write
buffersizes used by curl.
This is done by adding the following to .daprc (aka .dodsrc):
	HTTP.READ.BUFFERSIZE=n
where n is the buffersize in bytes. There is a built-in (to curl)
limit of 512k for this value.

2. CURLOPT_TCP_KEEPALIVE (and CURLOPT_TCP_KEEPIDLE and CURLOPT_TCP_KEEPINTVL):
Relevant (maybe) to [2] and [3]. Allow the user to turn on KEEPALIVE
This is done by adding the following to .daprc (aka .dodsrc):
	HTTP.KEEPALIVE=on|n/m
If the value is "on", then simply enable default KEEPALIVE. If the value
is n/m, then enable KEEPALIVE and set KEEPIDLE to n and KEEPINTVL to m.
2018-08-26 17:04:46 -06:00
Dennis Heimbigner
2ea1cf5f1b There was a request to extend the provenance information
stored in the _NCProperties attribute to allow two things:
1. capture of additional library dependencies (over and above
   hdf5)
2. Recognition of non-netcdf libraries that create netcdf-4 format
   files.

To this end, the _NCProperties format has been extended to be
and arbitrary set of key=value pairs separated by commas.
This new format has version = 2, and uses commas as the pair separator.
Thus the general form is:
    _NCProperties = "version=2,key1=value,key2=value2..." ;

This new version is accompanied by a new ./configure option of the form
    --with-ncproperties="key1=value1,key2=value2..."
that specifies pairs to add to the _NCProperties attribute for all
files created with that netcdf library.

At this point, what is missing is some programmatic way to
specify either all the pairs or additional pairs
to the _NCProperties attribute. Not sure of the best way
to do this.

Builders using non-netcdf libraries can specify
whatever they want in the key value pairs (as long
as the version=2 is specified first).

By convention, the primary library is expected to be the
the first pair after the leading version=2 pair, but this
is convention only and is neither required nor enforced.

Related changes:
1. Fixed the tests that check _NCProperties to properly operate with version=2.
2. When reading a version 1 _NCProperties attribute, convert it to look
   like a version 2 attribute.
2. Added some version 2 tests to ncdump/tst_fileinfo.c and
   ncdump/tst_fileinfo.sh

Misc Changes:
1. Fix minor problem in ncdap_test/testurl.sh where a parameter to
   buildurl needed to be quoted.
2. Minor fix to ncgen to swap switches -H and -h to be consistent
   with other utilities.
3. Document the -M flag in nccopy usage() and the nccopy man page.
4. Modify a test case to use the nccopy -M flag.
2018-08-25 21:44:41 -06:00
Ed Hartnett
ef7b525ce4 merged master 2018-07-23 09:45:39 -06:00
Ed Hartnett
03de993ce5 attempting to fix ENABLE_SET_LOG_LEVEL problem with cmake build 2018-07-16 06:31:44 -06:00
Wei-keng Liao
f95d3e3325 replace USE_CDF5 with ENABLE_CDF5 2018-06-29 21:17:07 -05:00
Dennis Heimbigner
8cb1fc4cfe This is the second step in refactoring the libsrc4 code.
The first was branch newhash0.dmh.

As with newhash0.dmh, these changes should be transparent.
2018-02-24 20:36:24 -07:00
Ward Fisher
f9f27c5ab7
Merge branch 'master' into newhash0.dmh 2018-02-21 14:19:43 -07:00
Ben Boeckel
b432a527c4 c: remove __CHAR_UNSIGNED__
In C, `char`, `signed char`, and `unsigned char` are three separate,
distinct types, so just because `char` happens to be signed does not
mean it is interchangeable with `signed char`.
2018-02-14 17:24:49 -05:00
Ben Boeckel
a7057925d6 configure: remove unused configure checks
These checks all control variables which are unused within the codebase.
2018-02-14 17:24:45 -05:00
Dennis Heimbigner
727b613459 This is the initial step in moving to the new higher performance
(I hope) metadata mechanism. This mostly just adds new pieces of
code (e.g. nclistmap) and does some minor fixes.

It should be transparent to everything else.
The next set of changes will be the big step.
2018-02-08 19:53:40 -07:00
Ward Fisher
d02a905aa9 Updated typo. 2018-02-02 20:27:06 -07:00
Ward Fisher
0fee3b9404 Added a missing line in config.h.cmake.in 2018-02-02 20:22:49 -07:00
Ward Fisher
c1d54b0213 Added check for genlib.h 2018-02-02 20:57:55 -06:00
Dennis Heimbigner
4db4393e69 Begin changing over to use strlcat instead of strncat because
strlcat provides better protection against buffer overflows.

Code is taken from the FreeBSD project source code. Specifically:
https://github.com/freebsd/freebsd/blob/master/lib/libc/string/strlcat.c
License appears to be acceptable, but needs to be checked by e.g. Debian.

Step 1:
1. Add to netcdf-c/include/ncconfigure.h to use our version
   if not already available as determined by HAVE_STRLCAT in config.h.
2. Add the strlcat code to libdispatch/dstring.c
3. Turns out that strlcat was already defined in several places.
   So remove it from:
	ncgen3/genlib.c
	ncdump/dumplib.c
3. Define strlcat extern definition in ncconfigure.h.
4. Modify following directories to use strlcat:
	libdap2 libdap4 ncdap_test dap4_test
   Will do others in subsequent steps.
2017-11-23 10:55:24 -07:00
Ward Fisher
e8af76c2f4 Wiring in a quick test. 2017-11-20 13:52:06 -07:00
Dennis Heimbigner
9935d54fdf Merge master and resolve conflicts 2017-10-28 13:57:23 -06:00
Ward Fisher
4299653319 Updated an issue with libcurl and dap4 on Windows 2017-09-25 17:38:48 -06:00
Ward Fisher
8b824a3dd0 Added support for probing libcurl for CURLINFO_HTTP_CODE in cmake. 2017-09-25 13:29:04 -06:00
Ward Fisher
1a56d3fdc8 Making cdf5 tests conditional on cdf5 support setting at configure time. 2017-09-14 14:18:56 -06:00
Ward Fisher
035ec80fb2 Wiring in CDF5 configure-time option. 2017-09-13 15:25:40 -06:00
Dennis Heimbigner
ad32350355 Oops. Forgot to convert over libdap4 use of NC_mktmp
and NC_readfile and NC_combinehostport.
2017-09-03 15:09:10 -06:00
Dennis Heimbigner
a2e0f069ec This pr should probably be delayed until after Version 4.5.
Primary change is to cleanup code and remove duplicated code.

1. Unify the rc file reading into libdispatch/drc.c. Eventually extend
   if we need rc file for netcdf itself as opposed to the dap code.
2. Unify the extraction from the rc file of DAP authorization info.
3. Misc. other small unifications: make temp file, read file.
4. Avoid use of libcurl when reading file:// because
   there is some kind of problem with the Visual Studio version.
   Might be related to the winpath problem.
   In any case, do direct read instead.
5. Add new error code NC_ERCFILE for errors in reading RC file.
6. Complete documentation cleanup as indicated in this comment
   https://github.com/Unidata/netcdf-c/pull/472#issuecomment-325926426
7. Convert some occurrences of #ifdef _WIN32 to #ifdef _MSC_VER
2017-09-02 18:09:36 -06:00
Ward Fisher
9e7a902dcf Merge branch 'issue435.dmh' into multi-pull 2017-07-27 12:20:11 -06:00
Ward Fisher
3e166fd26a Accomodating Windows winsock issue. 2017-07-26 13:40:03 -06:00
Dennis Heimbigner
88b3d20e4e turn debug on 2017-07-16 13:13:10 -06:00
Dennis Heimbigner
715a6fe5eb The files libdispatch/dwinpath.c and include/ncwinpath.h
were added to provide a path name converter from e.g. cygwin
paths to e.g. windows paths. This is necessary because
the shell scripts may produce cygwin paths, but the code
may have been compiled with Visual Studio. Similar issues
arise with Mingw.

At appropriate places, and if using Visual Studio or Mingw,
I added calls to the path conversion code.
Apparently I forgot to find all the places where this
conversion was needed. So this pr does the following:
1. Push the calls to the converter to the various libXXX
   directories and out of libdispatch/dfile.c.
2. Add conversion calls to other parts of the code like oc2.

I also turns out that conversion code in dapcvt.c
had a bug when handling DAP Byte type under visual studio.

Notes:
1. there may still be places I missed that need to do path conversion.
2. need to make sure that calls to e.g. H5open also use converted path.
2017-07-13 10:40:07 -06:00
Dennis Heimbigner
9719fbfbad re: hithub issue https://github.com/Unidata/netcdf-c/issues/435
Some temporary files are being left in a tempdir (e.g. /tmp
under *nix*).

The situation is described tersely in
netcdf-c/docs/auth.html#REDIR Basically, when a url is used that
requires redirection, a physical cookiejar file is required
to exist in the file system in order for this to work.

Since it was difficult to figure out when redirection was
being used (it was internal to libcurl) I needed to be prepared for that
eventuality. The result was that I always created a cookiejar file if one
was not specified in the rc file. This actually occurs in two places:
one inside oc2 and one inside libdap4.

The solution was two-fold:
1. do not use a cookiejar directory -- create cookiejar file directly
2. ensure that all cookiejar related files are reclaimed by nc_close().
Note that if nc_close (or nc_abort) is not called for whatever reason,
then reclamation will not occur.
2017-07-05 10:03:48 -06:00
Ward Fisher
89ebc2b778 Fixed an issue on visual studio which can result in a race state (more or less) when dap4 AND hdf4 support is enabled. 2017-06-01 14:48:23 -06:00
Ward Fisher
f29ae34572 Corrected another typo. 2017-05-23 16:41:23 -06:00
Ward Fisher
b717a22284 Corrected a typo. 2017-05-23 16:36:46 -06:00
Ward Fisher
b88507e7de More rigorous sync between old config.h cmake template and the current one. 2017-05-23 16:32:36 -06:00
Ward Fisher
ac56c10b0d Updated config.h template for CMake with some large file info. 2017-05-23 16:12:26 -06:00
Ward Fisher
cfdc3ea605 Restored the old non-working config.h template so that the comparisons can be easily made. 2017-05-23 16:07:18 -06:00
Ward Fisher
9408607d81 Added a version of config.h.cmake.in, based on the version from 4.4.1.1, that fixes at least one of the tests broken under the more recent re-written config.h.cmake.in. 2017-05-23 16:03:58 -06:00
Ward Fisher
9c78b4686b Restored some file checks. 2017-05-12 15:33:02 -06:00
Ward Fisher
8dddd222a3 Merged master, DAP4 support into branch. 2017-04-19 09:29:35 -06:00
Dennis Heimbigner
6d8809100f Fix pull request https://github.com/Unidata/netcdf-c/pull/374 (dap4.dmh)
1. When running under windows (as opposed to cygwin)
   we need to make sure to not user /cygdrive/ file paths.
   This was ocurring in libdap4/d4read.c, but may occur
   elsewhere.
2. Shell scripts in the git repo are not being checked-out
   with the executable mode set. Had core.filemode set to false.
   Was a major hassle to fix.
2017-04-03 21:39:44 -06:00
Ward Fisher
8e3790f7ce Merged master. 2017-03-09 12:53:28 -07:00
Dennis Heimbigner
3db4f013bf Primary change: add dap4 support
Specific changes:
1. Add dap4 code: libdap4 and dap4_test.
   Note that until the d4ts server problem is solved, dap4 is turned off.
2. Modify various files to support dap4 flags:
	configure.ac, Makefile.am, CMakeLists.txt, etc.
3. Add nc_test/test_common.sh. This centralizes
   the handling of the locations of various
   things in the build tree: e.g. where is
   ncgen.exe located. See nc_test/test_common.sh
   for details.
4. Modify .sh files to use test_common.sh
5. Obsolete separate oc2 by moving it to be part of
   netcdf-c. This means replacing code with netcdf-c
   equivalents.
5. Add --with-testserver to configure.ac to allow
   override of the servers to be used for --enable-dap-remote-tests.
6. There were multiple versions of nctypealignment code. Try to
   centralize in libdispatch/doffset.c and include/ncoffsets.h
7. Add a unit test for the ncuri code because of its complexity.
8. Move the findserver code out of libdispatch and into
   a separate, self contained program in ncdap_test and dap4_test.
9. Move the dispatch header files (nc{3,4}dispatch.h) to
   .../include because they are now shared by modules.
10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts.
11. Make use of MREMAP if available
12. Misc. minor changes e.g.
	- #include <config.h> -> #include "config.h"
	- Add some no-install headers to /include
	- extern -> EXTERNL and vice versa as needed
	- misc header cleanup
	- clean up checking for misc. unix vs microsoft functions
13. Change copyright decls in some files to point to LICENSE file.
14. Add notes to RELEASENOTES.md
2017-03-08 17:01:10 -07:00
Ward Fisher
fa1c96a67f Attempting to fix an issue. 2017-02-22 09:55:13 -07:00
Ward Fisher
581737b04e Integrating a big endian test. 2017-02-08 14:51:41 -07:00
Ward Fisher
f87ef5d929 Removed duplicate entries. 2017-02-01 13:49:25 -07:00
Ward Fisher
3e5124996e Merged https://github.com/Unidata/netcdf-c/pull/319 to resolve some recently arisen conflicts, in preparation for larger merge. 2017-02-01 13:27:29 -07:00
Ward Fisher
7c68a255c9 Merge branch 'master' into ghpull-335 2017-01-31 12:25:46 -07:00
Ward Fisher
27f81325a6 Added some missing pieces. 2017-01-31 10:59:03 -07:00
Ward Fisher
840fd113d5 Compatibility define for Windows. 2017-01-30 15:03:22 -07:00
Ward Fisher
870b6c534f Working out an error with unistd.h in windows. 2017-01-30 14:54:00 -07:00