Commit Graph

12 Commits

Author SHA1 Message Date
Dennis Heimbigner
e632d02041 Re-enable DAP2 authorization tests
The thredds-test server now has some password protected datasets
that can be used to test DAP2 authorization support.
The general location is
````
https://thredds.ucar.edu/thredds/tdscapabilities/authTest.html
````
and specifically:
````
https://thredds.ucar.edu/thredds/dodsC/test3/testData.nc.html
````

This PR replaces old testcases with ncdap_test/testauth.sh.
This testcase allows us to test use of the .dodsrc file and .netrc file
and embedded user+pwd.

As part of this, I had to create a program (ncdap_test/pathcvt.c)
that is essentially the equivalent to cygpath. Given a path in
windows, unix, msys or cygwin format, it converts it to the
equivalent format in one of those four cases.  So it can be used
to convert a cygwin path to a windows path, for example. This is
needed in testpathcvt and testauth to make sure that the paths
in .daprc (e.g. the reference to .netrc) are of the proper
format.

Misc. Other Changes:
1. Fix some memory leaks in libdap2
2. Setting the env variable CURLOPT_VERBOSE allows tracking of curl
   operations.
3. Make tst_charvlenbug be conditional on NC_VLEN_NOTEST.
2021-05-29 21:30:33 -06:00
Dennis Heimbigner
1243c3d866 Allow .rc tests to work in parallel by isolation 2021-04-25 22:02:29 -06:00
Dennis Heimbigner
74b40fd788 Upgrade the nczarr code to match Zarr V2
Re: https://github.com/zarr-developers/zarr-python/pull/716

The Zarr version 2 spec has been extended to include the ability
to choose the dimension separator in chunk name keys. The legal
separators has been extended from {'.'} to {'.' '/'}.  So now it
is possible to use a key like "0/1/2/0" for chunk names.

This PR implements this for NCZarr. The V2 spec now says that
this separator can be set on a per-variable basis. For now, I
have chosen to allow this be set only globally by adding a key
named "ZARR.DIMENSION_SEPARATOR=<char>" in the
.daprc/.dodsrc/ncrc file. Currently, the only legal separator
characters are '.' (the default) and '/'. On writing, this key
will only be written if its value is different than the default.
This change caused problems because supporting a separator of '/'
is difficult to parse when keys/paths use '/' as the path separator.
A test case was added for this.

Additionally, make nczarr be enabled default by default. This required
some additional changes so that if zip and/or AWS S3 sdk are unavailable,
then they are disabled for NCZarr.

In addition the following unrelated changes were made.

1. Tested that pure-zarr mode could read an nczarr formatted store.
1. The .rc file handling now merges all known .rc files (.ncrc,.daprc, and .dodsrc) in that order and using those in HOME first, then in current directory. For duplicate entries, the later ones override the earlier ones. This change is to remove some of the conflicts inherent in the current .rc file load process. A set of test cases was also added.
1. Re-order tests in configure.ac and CMakeLists.txt so that if libcurl
   is not found then the other options that depend upon it properly
   are disabled.
1. I decided that xarray support should be enabled by default for pure
   zarr. In order to allow disabling, I added a new mode flag "noxarray".
1. Certain test in nczarr_test depend on use of .dodsrc. In order for these
   to work when testing in parallel, some inter-test dependencies needed to
   be added.
1. Improved authorization testing to use changes in thredds.ucar.edu
2021-04-24 19:48:15 -06:00
Dennis Heimbigner
d1544b8ad9 Error in ncuri in handling user:pwd@ in url 2017-07-05 14:39:23 -06:00
Dennis Heimbigner
9719fbfbad re: hithub issue https://github.com/Unidata/netcdf-c/issues/435
Some temporary files are being left in a tempdir (e.g. /tmp
under *nix*).

The situation is described tersely in
netcdf-c/docs/auth.html#REDIR Basically, when a url is used that
requires redirection, a physical cookiejar file is required
to exist in the file system in order for this to work.

Since it was difficult to figure out when redirection was
being used (it was internal to libcurl) I needed to be prepared for that
eventuality. The result was that I always created a cookiejar file if one
was not specified in the rc file. This actually occurs in two places:
one inside oc2 and one inside libdap4.

The solution was two-fold:
1. do not use a cookiejar directory -- create cookiejar file directly
2. ensure that all cookiejar related files are reclaimed by nc_close().
Note that if nc_close (or nc_abort) is not called for whatever reason,
then reclamation will not occur.
2017-07-05 10:03:48 -06:00
Dennis Heimbigner
6d8809100f Fix pull request https://github.com/Unidata/netcdf-c/pull/374 (dap4.dmh)
1. When running under windows (as opposed to cygwin)
   we need to make sure to not user /cygdrive/ file paths.
   This was ocurring in libdap4/d4read.c, but may occur
   elsewhere.
2. Shell scripts in the git repo are not being checked-out
   with the executable mode set. Had core.filemode set to false.
   Was a major hassle to fix.
2017-04-03 21:39:44 -06:00
Dennis Heimbigner
3db4f013bf Primary change: add dap4 support
Specific changes:
1. Add dap4 code: libdap4 and dap4_test.
   Note that until the d4ts server problem is solved, dap4 is turned off.
2. Modify various files to support dap4 flags:
	configure.ac, Makefile.am, CMakeLists.txt, etc.
3. Add nc_test/test_common.sh. This centralizes
   the handling of the locations of various
   things in the build tree: e.g. where is
   ncgen.exe located. See nc_test/test_common.sh
   for details.
4. Modify .sh files to use test_common.sh
5. Obsolete separate oc2 by moving it to be part of
   netcdf-c. This means replacing code with netcdf-c
   equivalents.
5. Add --with-testserver to configure.ac to allow
   override of the servers to be used for --enable-dap-remote-tests.
6. There were multiple versions of nctypealignment code. Try to
   centralize in libdispatch/doffset.c and include/ncoffsets.h
7. Add a unit test for the ncuri code because of its complexity.
8. Move the findserver code out of libdispatch and into
   a separate, self contained program in ncdap_test and dap4_test.
9. Move the dispatch header files (nc{3,4}dispatch.h) to
   .../include because they are now shared by modules.
10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts.
11. Make use of MREMAP if available
12. Misc. minor changes e.g.
	- #include <config.h> -> #include "config.h"
	- Add some no-install headers to /include
	- extern -> EXTERNL and vice versa as needed
	- misc header cleanup
	- clean up checking for misc. unix vs microsoft functions
13. Change copyright decls in some files to point to LICENSE file.
14. Add notes to RELEASENOTES.md
2017-03-08 17:01:10 -07:00
dmh
3dd807a155 The original mem branch somehow got
hosed, so I rebuilt it as a new mem2 branch.
2015-05-28 15:10:10 -06:00
dmh
be5e3cd606 1. Allow for the user specified rc file via the env variable
DAPRCFILE.  Note that the value of this environment
   variable should be the absolute path of the rc file, not
   the path to its containing directory.
2. fixup testauth.sh and add some new tests
3. synch oc
2015-05-24 17:31:39 -06:00
dmh
f9edce92ed make testauth be under the enable-auth-tests 2015-03-03 21:01:43 -07:00
dmh
e35a14a85d 1. synch with oc
2. modify oc2/ocrc.c rcfilenames to look for .ocrc before .dodsrc.
3. Modify testauth.sh to avoid using names that might already
   exist for cookies file and netrc file. Still must use .ocrc
   to test for local/home search.
4. Modify testauth.sh to save and restore any file it creates
   that already exists.
2014-12-31 22:29:36 -07:00
dmh
f423f27693 Sync with oc project.
This supports better authorization
handling for DAP requests, especially redirection
based authorization. I also added a new test case
ncdap_tests/testauth.sh.

Specifically, suppose I have a netrc file /tmp/netrc
containing this.
    machine uat.urs.earthdata.nasa.gov login xxxxxx password yyyyyy
Also suppose I have a .ocrc file containing these lines
    HTTP.COOKIEJAR=/tmp/cookies
    HTTP.NETRC=/tmp/netrc
Assume that .ocrc is in the local directory or HOME.

Then this command should work (assuming a valid login and password).
    ncdump -h "https://54.86.135.31/opendap/data/nc/fnoc1.nc"
2014-12-24 10:22:47 -07:00