Commit Graph

8275 Commits

Author SHA1 Message Date
Dennis Heimbigner
9380790ea8 Support MSYS2/Mingw platform
re:

The current netcdf-c release has some problems with the mingw platform
on windows. Mostly they are path issues.

Changes to support mingw+msys2:
-------------------------------
* Enable option of looking into the windows registry to find
  the mingw root path. In aid of proper path handling.
* Add mingw+msys as a specific platform in configure.ac and move testing
  of the platform to the front so it is available early.
* Handle mingw X libncpoco (dynamic loader) properly even though
  mingw does not yet support it.
* Handle mingw X plugins properly even though mingw does not yet support it.
* Alias pwd='pwd -W' to better handle paths in shell scripts.
* Plus a number of other minor compile irritations.
* Disallow the use of multiple nc_open's on the same file for windows
  (and mingw) because windows does not seem to handle these properly.
  Not sure why we did not catch this earlier.
* Add mountpoint info to dpathmgr.c to help support mingw.
* Cleanup dpathmgr conversions.

Known problems:
---------------
* I have not been able to get shared libraries to work, so
  plugins/filters must be disabled.
* There is some kind of problem with libcurl that I have not solved,
  so all uses of libcurl (currently DAP+Byterange) must be disabled.

Misc. other fixes:
------------------
* Cleanup the relationship between ENABLE_PLUGINS and various other flags
  in CMakeLists.txt and configure.ac.
* Re-arrange the TESTDIRS order in Makefile.am.
* Add pseudo-breakpoint to nclog.[ch] for debugging.
* Improve the documentation of the path manager code in ncpathmgr.h
* Add better support for relative paths in dpathmgr.c
* Default the mode args to NCfopen to include "b" (binary) for windows.
* Add optional debugging output in various places.
* Make sure that everything builds with plugins disabled.
* Fix numerous (s)printf inconsistencies betweenb the format spec
  and the arguments.
2021-12-23 22:18:56 -07:00
Ward Fisher
dfb10a3f21
Merge pull request #2168 from DennisHeimbigner/ncdumptests.dmh
Cleanup the CMake inter-test dependencies
2021-12-21 14:29:46 -07:00
Dennis Heimbigner
5fb64d59af Update RELEASE Notes 2021-12-20 18:10:56 -07:00
Dennis Heimbigner
73caeb674d Cleanup the CMake inter-test dependencies
The ncdump test set has a number of inter-test dependencies
that are not properly established in ncdump/CMakeLists.txt.

So this PR attempts to:
1. reorder the tests
2. change tests in CMakeLists.txt from build_bin_test_no_prefix to add_bin_test_no_prefix so they get executed

Plus a couple of minor bug fixes.
1. Change ENABLE_NC4 => ENABLE_HDF5 in github action.
2. fix a memory error in findtestserver.c.in
3. fix bug in ncdap_tests/tst_urls.sh
4. fix netcdf file name bug in tst_netcdf4_4.sh
2021-12-20 15:13:08 -07:00
Ward Fisher
1ad0a53595
Merge pull request #2167 from DennisHeimbigner/addremotetest.dmh
Restore default enable for enable-dap-remote-testsc
2021-12-17 12:02:46 -06:00
Dennis Heimbigner
f00d03284a Restore default on for enable-dap-remote-testsc
The remote test server is up again. So re-enable.
Also do some test cleanup.
2021-12-16 15:43:22 -07:00
Ward Fisher
ebbb742819
Merge pull request #2164 from DennisHeimbigner/cmakelog4j.dmh
Disable by default dap-remote-tests for CMake
2021-12-13 10:16:18 -06:00
Dennis Heimbigner
ed1b82cdd3 The remotetest server is down for a while
because of the log4j security flaw.
So we default to disabling dap-remote-tests.
This PR adds disable to CMake
2021-12-11 19:19:27 -07:00
Ward Fisher
72a7926fdb
Merge pull request #2163 from DennisHeimbigner/log4j.dmh 2021-12-10 21:51:45 -07:00
Dennis Heimbigner
b768d5d2a2 The remotetest server is down for a while
because of the log4j security flaw.
So we default to disabling dap-remote-tests.
2021-12-10 18:53:00 -07:00
Edward Hartnett
3ecf6a452e more test development 2021-12-03 06:41:35 -07:00
Edward Hartnett
34a0cfc46a more test development 2021-12-03 06:41:20 -07:00
Edward Hartnett
5508a7aba4 added sequential benchmark program for compression 2021-11-27 07:40:44 -07:00
Edward Hartnett
b7b4e53687 now setting deflate_level to zero does not turn on deflate 2021-11-27 07:40:02 -07:00
Edward Hartnett
0a00163b05 better comments 2021-11-26 06:35:18 -07:00
Edward Hartnett
2d504e02d8 adding quantize test 2021-11-26 06:29:35 -07:00
Jennifer Oxelson
45bce8d395 docs link updates 2021-11-15 11:19:34 -07:00
Jennifer Oxelson
f7e3d3e44f docs cleanup 2021-11-11 11:40:45 -07:00
Jennifer Oxelson
5b7ddcca06 docs migration, part 2 2021-11-11 10:55:52 -07:00
Jennifer Oxelson
a11349482c Docs migration 2021-11-11 10:47:49 -07:00
Jennifer Oxelson
4fcd4f2c3f intellij 2021-11-11 10:43:04 -07:00
Charlie Zender
48560bfad9 Change test to verify that using quantize mode one greater than NC_GRANULARBG (instead of NC_QUANTIZE_BITGROOM) fails. 2021-11-04 16:13:12 -07:00
Ward Fisher
26ac2a6cba
Merge pull request #2121 from gsjaardema/patch-48
Refactor Z library detection
2021-11-04 15:42:02 -06:00
Ward Fisher
7dc355aab8
Merge pull request #2139 from WardF/gh2135.wif
libxml2 capability
2021-11-04 14:47:14 -06:00
Ward Fisher
b3fe92e304 Fencepost unistd.h inclusion. 2021-11-04 14:21:55 -06:00
Ward Fisher
c0a6acaf5a
Merge branch 'main' into gh2135.wif 2021-11-04 13:56:05 -06:00
Ward Fisher
b69163a387 Preprocessing for generated ezxml files. 2021-11-04 13:46:07 -06:00
Ward Fisher
33e4ec4675
Merge pull request #2138 from DennisHeimbigner/fixosbugs.dmh
Fix a number of OS specific bugs
2021-11-04 13:36:49 -06:00
Ward Fisher
0e336208da Addressed an issue in ncbytes where long unsigned ~= size_t 2021-11-04 10:10:28 -06:00
Ward Fisher
25d3d008f6 Added missing include for read() function. 2021-11-04 10:01:27 -06:00
Ward Fisher
0471869f58 Merge remote-tracking branch 'upstream/main' into gh2135.wif 2021-11-03 13:47:35 -06:00
Ward Fisher
8f648252ee Adding control block, --enable and disable option for libxml2 to configure.ac 2021-11-03 13:40:47 -06:00
Ward Fisher
0e205f9513
Merge pull request #2137 from DennisHeimbigner/typo.dmh
Typo in previous PR
2021-11-03 13:34:47 -06:00
Ward Fisher
516a9c35d3 Updated .gitignore to ignore .vscode directory. 2021-11-03 13:14:02 -06:00
Ward Fisher
73c4cc3883 Updated release notes. 2021-11-03 13:13:25 -06:00
Dennis Heimbigner
6bd289d655 Update release notes 2021-11-03 12:52:42 -06:00
Dennis Heimbigner
55a2643cac Fix a number of OS specific bugs
1. Issue https://github.com/Unidata/netcdf-c/issues/2043
   * FreeBSD build fails because of conflicts in defining the fileno() function. So removed all extern declarations of fileno.

2. Issue https://github.com/Unidata/netcdf-c/issues/2124
   * There were a couple of problems here.
     * I was conflating msys with mingw and they need separate handling of paths. So treat mingw like windows.
     * memio.c was not always writing the full content of the memory to file. Untested fix by properly accounting for zero size writes.
     * Fix bug when skipping white space in tst_xcache.c

3. Issue https://github.com/Unidata/netcdf-c/pull/2105
   * On MINGW, bash and other POSIX utilities use a mounted root directory,
     but executables compiled for Windows do not recognise the mount point.
     Ensure that Windows paths are used in tests of Windows executables.

4. Issue https://github.com/Unidata/netcdf-c/issues/2132
   * Apparently the Intel C compiler on OSX defines isnan etc.
     So disable declaration in dutil.c under that condition.

5. Fix and re-enable test_rcmerge.sh by allowing override of where to
   look for .rc files

6. CMakeLists.txt suppresses certain ncdump directory tests because of differences in printing floats/doubles.
   * Extend the list to include those that also fail under mingw.
   * Suppress the mingw tests in ncdump/Makefile.am
2021-11-03 12:49:54 -06:00
Dennis Heimbigner
e14c17fd08 Typo in previous PR
re: https://github.com/Unidata/netcdf-c/issues/2136

Patch for: https://github.com/Unidata/netcdf-c/pull/2134/

Inadvertent error changing ->info to ->s3.
2021-11-03 12:06:50 -06:00
Ward Fisher
eead7fa6d2 Begin wiring in option to toggle automatic search for libxml2 in support of https://github.com/Unidata/netcdf-c/pulls/2135 2021-11-02 15:35:04 -06:00
Dennis Heimbigner
cf903a678a Update release notes 2021-11-01 22:40:08 -06:00
Dennis Heimbigner
53464e8963 Allow optional use of libxml2
re: https://github.com/Unidata/netcdf-c/issues/2119

H/T to [Egbert Eich](https://github.com/e4t) and [Bas Couwenberg](https://github.com/sebastic) for this PR.

It is undesirable to make netcdf be dependent on the availability
of libxml2, but it is desirable to allow its use if available.

In order to do this, a wrapper API (include/ncxml.h) was constructed
that supports either ezxml or libxml2 as the implementation.
Additionally, the xml support code was moved to a new directory
netcdf-c/libncxml.

Primary changes:
* Create a new sub-directory named netcdf-c/libncxml to hold all the xml implementation code.
* Move ezxml.c and ezxml.h to libncxml
* Create a wrapper API -- include/ncxml.h
* Create an implementation, ncxml_ezxml.c to support use of ezxml.
* Create an implementation, ncxml_xml2.c to support use of libxml2.
* Add a check for libxml2 in configure.ac and CMakeLists.txt
* Modify libdap to use the wrapper API instead of ezxml directly.

Misc. Other Changes:
* Change include/netcdf_json.h from built source to be part of the distribution.
2021-11-01 22:37:05 -06:00
Ward Fisher
fcf0fa7dbd
Merge pull request #2134 from DennisHeimbigner/zarrs3part2.dmh 2021-11-01 16:00:39 -06:00
Dennis Heimbigner
6f3bfde222 Update release notes 2021-10-29 20:35:13 -06:00
Dennis Heimbigner
f6e25b695e Fix additional S3 support issues
re: https://github.com/Unidata/netcdf-c/issues/2117
re: https://github.com/Unidata/netcdf-c/issues/2119

* Modify libsrc to allow byte-range reading of netcdf-3 files in private S3 buckets; this required using the aws sdk. Also add a test case.
* The aws sdk can sometimes cause problems if the Awd::ShutdownAPI function is not called. So at optional atexit() support to ensure it is called. This is disabled for Windows.
* Add documentation to nczarr.md on how to build and use the aws sdk under windows. Currently it builds, but testing fails.
* Switch testing from stratus to the Unidata bucket on S3.
* Improve support for the s3: url protocol.
* Add a s3 specific utility code file: ds3util.c
* Modify NC_infermodel to attempt to read the magic number of byte-ranged files in S3.

## Misc.

* Move and rename the core S3 SDK wrapper code (libnczarr/zs3sdk.cpp) to libdispatch since it now used in libsrc as well as libnczarr.
* Add calls to nc_finalize in the utilities in case atexit is disabled.
* Add header only json parser to the distribution rather than as a built source.
2021-10-29 20:06:37 -06:00
Ward Fisher
228e7f5b5c
Merge pull request #2114 from DennisHeimbigner/zarrs3.dmh
Significantly Improve Amazon S3 Cloud Storage Support
2021-10-28 14:22:56 -06:00
Dennis Heimbigner
2da684fc37 ckp 2021-10-26 22:52:23 -06:00
Dennis Heimbigner
58ba84de0b update merge 2021-10-26 20:53:05 -06:00
Ward Fisher
783dc54d12
Merge pull request #2131 from gsjaardema/patch-49
Remove duplicate line
2021-10-26 09:48:28 -06:00
Greg Sjaardema
9c6181da09
Remove duplicate line
Remove a duplicate line...
2021-10-25 17:09:17 -06:00
Charlie Zender
e7394af9be Change NC_QUANTIZE_ATT_NAME to NC_QUANTIZE_BITGROOM_ATT_NAME 2021-10-21 11:39:39 -07:00