Commit Graph

40 Commits

Author SHA1 Message Date
Peter Hill
d07dac918c
Silence conversion warnings from malloc arguments
Mostly just add an explicit cast when calling `malloc` and its
variants. Sometimes instead change the type of a local variable if
this would silence multiple warnings.
2023-11-24 18:20:52 +00:00
Dennis Heimbigner
6901206927 Regularize the semantics of mkstemp.
re: https://github.com/Unidata/netcdf-c/issues/1827

The issue is partly resolved by this PR. The proximate problem appears to be that the semantics of mkstemp in **nix is different than the semantics of _mktemp_s in Windows. I had thought they were the same but that is incorrect. The _mktemp_s function will only produce 26 different files and so the netcdf temp file code will fail after about that many iterations.

So, to solve this, I created my own version of mkstemp for windows that uses a random number generator. This appears to solve the reported issue.  I also added the testcase ncdap_test/test_manyurls but made it conditional on --enable-dap-long-tests because it is very slow.

I did note that the provided test program now fails after some 800 iterations with a libcurl error claiming it cannot resolve the host name. My belief is that the library is just running out of resources at this point: too many open curl handles or some such. I doubt if this failure is fixable.

So bottom line is that it is really important to do nc_close when you are finished with a file.

Misc. Other Changes:

1. I took the opportunity to clean up some bad string hacks in the code. Specifically
    * change all uses of strncat to strlcat
    * remove old string hacks: occoncat and occopycat
2. Add heck to see if test.opendap.org is running and if not, then skip test
3. Make CYGWIN use TEMP environment variable
2021-05-14 11:33:03 -06:00
Dennis Heimbigner
59e04ae071 This PR adds EXPERIMENTAL support for accessing data in the
cloud using a variant of the Zarr protocol and storage
format. This enhancement is generically referred to as "NCZarr".

The data model supported by NCZarr is netcdf-4 minus the user-defined
types and the String type. In this sense it is similar to the CDF-5
data model.

More detailed information about enabling and using NCZarr is
described in the document NUG/nczarr.md and in a
[Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in).

WARNING: this code has had limited testing, so do use this version
for production work. Also, performance improvements are ongoing.
Note especially the following platform matrix of successful tests:

Platform | Build System | S3 support
------------------------------------
Linux+gcc      | Automake     | yes
Linux+gcc      | CMake        | yes
Visual Studio  | CMake        | no

Additionally, and as a consequence of the addition of NCZarr,
major changes have been made to the Filter API. NOTE: NCZarr
does not yet support filters, but these changes are enablers for
that support in the future.  Note that it is possible
(probable?) that there will be some accidental reversions if the
changes here did not correctly mimic the existing filter testing.

In any case, previously filter ids and parameters were of type
unsigned int. In order to support the more general zarr filter
model, this was all converted to char*.  The old HDF5-specific,
unsigned int operations are still supported but they are
wrappers around the new, char* based nc_filterx_XXX functions.
This entailed at least the following changes:
1. Added the files libdispatch/dfilterx.c and include/ncfilter.h
2. Some filterx utilities have been moved to libdispatch/daux.c
3. A new entry, "filter_actions" was added to the NCDispatch table
   and the version bumped.
4. An overly complex set of structs was created to support funnelling
   all of the filterx operations thru a single dispatch
   "filter_actions" entry.
5. Move common code to from libhdf5 to libsrc4 so that it is accessible
   to nczarr.

Changes directly related to Zarr:
1. Modified CMakeList.txt and configure.ac to support both C and C++
   -- this is in support of S3 support via the awd-sdk libraries.
2. Define a size64_t type to support nczarr.
3. More reworking of libdispatch/dinfermodel.c to
   support zarr and to regularize the structure of the fragments
   section of a URL.

Changes not directly related to Zarr:
1. Make client-side filter registration be conditional, with default off.
2. Hack include/nc4internal.h to make some flags added by Ed be unique:
   e.g. NC_CREAT, NC_INDEF, etc.
3. cleanup include/nchttp.h and libdispatch/dhttp.c.
4. Misc. changes to support compiling under Visual Studio including:
   * Better testing under windows for dirent.h and opendir and closedir.
5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags
   and to centralize error reporting.
6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them.
7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible.

Changes Left TO-DO:
1. fix provenance code, it is too HDF5 specific.
2020-06-28 18:02:47 -06:00
Greg Sjaardema
56c0d5cf8a Spelling fixes 2019-09-18 08:03:01 -06:00
Dennis Heimbigner
6934aa2e8b Thread safety: step 1: cleanup
re: https://github.com/Unidata/netcdf-c/issues/1373 (partial)

* Mark some global constants be const to indicate to make them easier to track.
* Hide direct access to the ncrc_globalstate behind a function call.
* Convert dispatch tables to constants (except the user defined ones)
  This has some consequences in terms of function arguments needing to be marked
  as const also.
* Remove some no longer needed global fields
* Aggregate all the globals in nclog.c
* Uniformly replace nc_sizevector{0,1} with NC_coord_{zero,one}
* Uniformly replace nc_ptrdffvector1 with NC_stride_one
* Remove some obsolete code
2019-03-30 14:06:20 -06:00
Ward Fisher
921a217621 nctest, oc2 files updated. 2018-12-06 15:47:47 -07:00
Dennis Heimbigner
a2e0f069ec This pr should probably be delayed until after Version 4.5.
Primary change is to cleanup code and remove duplicated code.

1. Unify the rc file reading into libdispatch/drc.c. Eventually extend
   if we need rc file for netcdf itself as opposed to the dap code.
2. Unify the extraction from the rc file of DAP authorization info.
3. Misc. other small unifications: make temp file, read file.
4. Avoid use of libcurl when reading file:// because
   there is some kind of problem with the Visual Studio version.
   Might be related to the winpath problem.
   In any case, do direct read instead.
5. Add new error code NC_ERCFILE for errors in reading RC file.
6. Complete documentation cleanup as indicated in this comment
   https://github.com/Unidata/netcdf-c/pull/472#issuecomment-325926426
7. Convert some occurrences of #ifdef _WIN32 to #ifdef _MSC_VER
2017-09-02 18:09:36 -06:00
Dennis Heimbigner
715a6fe5eb The files libdispatch/dwinpath.c and include/ncwinpath.h
were added to provide a path name converter from e.g. cygwin
paths to e.g. windows paths. This is necessary because
the shell scripts may produce cygwin paths, but the code
may have been compiled with Visual Studio. Similar issues
arise with Mingw.

At appropriate places, and if using Visual Studio or Mingw,
I added calls to the path conversion code.
Apparently I forgot to find all the places where this
conversion was needed. So this pr does the following:
1. Push the calls to the converter to the various libXXX
   directories and out of libdispatch/dfile.c.
2. Add conversion calls to other parts of the code like oc2.

I also turns out that conversion code in dapcvt.c
had a bug when handling DAP Byte type under visual studio.

Notes:
1. there may still be places I missed that need to do path conversion.
2. need to make sure that calls to e.g. H5open also use converted path.
2017-07-13 10:40:07 -06:00
Dennis Heimbigner
3db4f013bf Primary change: add dap4 support
Specific changes:
1. Add dap4 code: libdap4 and dap4_test.
   Note that until the d4ts server problem is solved, dap4 is turned off.
2. Modify various files to support dap4 flags:
	configure.ac, Makefile.am, CMakeLists.txt, etc.
3. Add nc_test/test_common.sh. This centralizes
   the handling of the locations of various
   things in the build tree: e.g. where is
   ncgen.exe located. See nc_test/test_common.sh
   for details.
4. Modify .sh files to use test_common.sh
5. Obsolete separate oc2 by moving it to be part of
   netcdf-c. This means replacing code with netcdf-c
   equivalents.
5. Add --with-testserver to configure.ac to allow
   override of the servers to be used for --enable-dap-remote-tests.
6. There were multiple versions of nctypealignment code. Try to
   centralize in libdispatch/doffset.c and include/ncoffsets.h
7. Add a unit test for the ncuri code because of its complexity.
8. Move the findserver code out of libdispatch and into
   a separate, self contained program in ncdap_test and dap4_test.
9. Move the dispatch header files (nc{3,4}dispatch.h) to
   .../include because they are now shared by modules.
10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts.
11. Make use of MREMAP if available
12. Misc. minor changes e.g.
	- #include <config.h> -> #include "config.h"
	- Add some no-install headers to /include
	- extern -> EXTERNL and vice versa as needed
	- misc header cleanup
	- clean up checking for misc. unix vs microsoft functions
13. Change copyright decls in some files to point to LICENSE file.
14. Add notes to RELEASENOTES.md
2017-03-08 17:01:10 -07:00
Ward Fisher
393db0018e Addressed an issue causing MSVC-based builds to fail. See https://github.com/Unidata/netcdf-c/issues/96 2015-01-13 12:44:28 -07:00
dmh
f423f27693 Sync with oc project.
This supports better authorization
handling for DAP requests, especially redirection
based authorization. I also added a new test case
ncdap_tests/testauth.sh.

Specifically, suppose I have a netrc file /tmp/netrc
containing this.
    machine uat.urs.earthdata.nasa.gov login xxxxxx password yyyyyy
Also suppose I have a .ocrc file containing these lines
    HTTP.COOKIEJAR=/tmp/cookies
    HTTP.NETRC=/tmp/netrc
Assume that .ocrc is in the local directory or HOME.

Then this command should work (assuming a valid login and password).
    ncdump -h "https://54.86.135.31/opendap/data/nc/fnoc1.nc"
2014-12-24 10:22:47 -07:00
dmh
a2e9fb854b synch with base oc library 2014-11-21 16:20:44 -07:00
Ward Fisher
b43813d821 When using the '-ansi' flag with gcc, netcdf would fail to build because of the use of '//' for one-line comments. I've addressed this by replacing instances of // with enclosing comment blocks. 2014-08-26 11:19:32 -06:00
Ward Fisher
6c071be031 Corrected issues where functions were not available in Windows. Specifically strndup. Also accomodated an included needed for getcwd. 2014-08-07 17:03:27 -06:00
dmh
b524e151ef Remove last vestiges of thredds-test 2014-03-14 14:07:35 -06:00
dmh
a189b98b0b 1. Any test that references nctestserver/NC_findtestserver
should be under ENABLE_DAP_REMOTE_TESTS.
   Fixed to make sure that this is so.
   Also attempted to fix ncdap_test/CMakeLists.txt,
   but probably got it wrong.
   HT to Nico Schlomer.
2. Attempted to reduce the number of conversion errors
   when -Wconversion is set. Fixed oc2, but
   rest of netcdf remains to be done.
   HT to Nico Schlomer.
3. When doing #2, I discovered an error in ncgen.y
   that has remained hidden. This required some other
   test case fixes.
2014-03-08 20:41:30 -07:00
Ward Fisher
fcb0eb2df6 Corrected an error preventing netcdf from compiling. 2014-03-03 15:36:19 -07:00
dmh
ba726d55ce synch with oc2.0 2014-03-03 13:43:47 -07:00
Quincey Koziol
d033f9f6ff Clean up Coverity warnings from last checkin. 2014-02-20 08:24:55 -06:00
Quincey Koziol
4bfd9b27a0 Merge remote-tracking branch 'upstream/master'
Conflicts:
	libdap2/common34.c
	oc2/ocutil.c
2014-02-11 17:26:09 -06:00
dmh
0fa9dc98eb synch oc 2014-01-21 12:50:15 -07:00
Quincey Koziol
cd71eb525c Corrected "BAIL" macros to avoid infinite loop when logging is disabled and an
error occurs after an "exit:" label.

Corrected a dozen Coverity errors (mainly allocation issues, along with a few
    other things):
        711711, 711802, 711803, 711905, 970825, 996123, 996124, 1025787,
        1047274, 1130013, 1130014, 1139538

Refactored internal fill-value code to correctly handle string types, and
    especially to allow NULL pointers and null strings (ie. "") to be
    distinguished.  The code now avoids partially aliasing the two together
    (which only happened on the 'write' side of things and wasn't reflected on
    the 'read' side, adding to the previous confusion).

    Probably still weak on handling fill-values of variable-length and compound
    datatypes.

Refactored the recursive metadata reads a bit more, to process HDF5 named
    datatypes and datasets immediately, avoiding chewing up memory for those
    types of objects, etc.

Finished uncommenting and updating the nc_test4/tst_fills2.c code (as I'm
    proceeding alphabetically through the nc_test4 code files).
2013-12-29 01:12:43 -06:00
Ward Fisher
0af7499eb4 Added a couple items to .gitignore.
Corrected an issue reported by Coverity static analysis where memory was allocated but was not freed.
2013-12-04 16:02:43 -07:00
dmh
d6061db87d define mode_t when under windows 2013-11-15 15:41:17 -07:00
dmh
2bc308432d sync with oc2.0 2013-11-14 15:13:20 -07:00
Dennis Heimbigner
4e41843740 1) fix some minor bugs when pnetcdf is enabled 2)Comment out some apparently unused procedures 2013-03-25 17:12:50 +00:00
Dennis Heimbigner
9b4db7a039 Make some things static that do not need to be externally visible 2013-03-24 17:33:17 +00:00
Dennis Heimbigner
4109174d00 Add quick check for Error {...} at start of the datadds data part 2013-02-07 22:14:57 +00:00
Dennis Heimbigner
c11e78314e periodic sync of oc2 dir with the official oc2.0 repository 2012-12-18 21:08:23 +00:00
Dennis Heimbigner
c114861e68 fix dfile.c error reporting 2012-12-06 21:17:36 +00:00
Dennis Heimbigner
97344d8551 make consistent with svn/oc2.0 2012-12-04 03:32:41 +00:00
Dennis Heimbigner
0803ba20ba fix Jira NCF-208; but note that this causes test.06 to fail because thredds mishandles grids 2012-11-21 23:57:42 +00:00
Ward Fisher
afa67452f6 Took some time to address a handful of errors identified by Coverity.
Primarily focused on memory errors falling into a couple different types:

1) Static overrun errors.
2) Dereference uninitialized memory errors.

make distcheck works after applying these fixes, and coverity no longer sees an issue, so hopefully they are properly resolved.
2012-11-14 18:24:45 +00:00
Dennis Heimbigner
29d3fe7d51 fix bug in reading a file url (needs to signal binary if possible) 2012-08-21 23:34:25 +00:00
Dennis Heimbigner
3c843c9578 add more debug output to try to find Ward's bug 2012-08-21 16:46:10 +00:00
Dennis Heimbigner
2a0d68c530 update nchashmap; move some old stuff in libdap2; fix new debug code 2012-08-19 21:12:35 +00:00
Dennis Heimbigner
685e1bf66a add extra debug about findbod 2012-08-17 20:51:03 +00:00
Dennis Heimbigner
499f8d3a1c complete update of utilities: ncbyte, nclist, ncuri, nclog 2012-08-08 23:15:18 +00:00
Dennis Heimbigner
bc3a732e25 fix e-support YOP-841363 plus a number of coverity found bugs 2012-08-02 18:43:21 +00:00
Dennis Heimbigner
42999f4c7c move from oc1.0 to oc2.0; create new dir oc2 2012-07-31 20:34:13 +00:00