netcdf-c/oc2/ocinternal.h

245 lines
6.0 KiB
C
Raw Normal View History

2018-12-07 06:47:47 +08:00
/* Copyright 2018, UCAR/Unidata and OPeNDAP, Inc.
See the COPYRIGHT file for more information. */
#ifndef OCINTERNAL_H
#define OCINTERNAL_H
#include "config.h"
#ifdef _MSC_VER
#include <malloc.h>
#endif
/* Required for getcwd, other functions. */
#ifdef _MSC_VER
#include <direct.h>
#define getcwd _getcwd
#endif
#ifdef _AIX
#include <netinet/in.h>
#endif
#include <stdlib.h>
#include <assert.h>
#include <string.h>
2012-12-07 05:17:36 +08:00
#ifdef HAVE_STRINGS_H
#include <strings.h>
#endif
2012-12-07 05:17:36 +08:00
#ifdef HAVE_STDARG_H
#include <stdarg.h>
2012-12-07 05:17:36 +08:00
#endif
#ifdef HAVE_UNISTD_H
#include <unistd.h>
#endif
#ifdef HAVE_SYS_TYPES_H
#include <sys/types.h>
#endif
#define CURL_DISABLE_TYPECHECK 1
#include <curl/curl.h>
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
#include "netcdf.h"
2017-09-01 04:19:56 +08:00
#include "ncauth.h"
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
#include "nclist.h"
#include "ncbytes.h"
#include "ncuri.h"
2014-11-22 07:20:44 +08:00
#ifndef HAVE_STRNDUP
/* Not all systems have strndup, so provide one*/
#define strndup ocstrndup
#endif
#define OCCACHEPOS
#ifndef HAVE_STRNDUP
/* Not all systems have strndup, so provide one*/
#define strndup ocstrndup
#endif
#define OCPATHMAX 8192
#ifndef nullfree
#define nullfree(x) {if((x)!=NULL) free(x);}
#endif
/* Forwards */
typedef struct OCstate OCstate;
typedef struct OCnode OCnode;
typedef struct OCdata OCdata;
struct OCTriplestore;
/* Define the internal node classification values */
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
#define OC_None ((unsigned int)0)
#define OC_State ((unsigned int)1)
#define OC_Node ((unsigned int)2)
#define OC_Data ((unsigned int)3)
/* Define a magic number to mark externally visible oc objects */
#define OCMAGIC ((unsigned int)0x0c0c0c0c) /*clever, huh*/
/* Max rc file line size */
#define MAXRCLINESIZE 4096
/* Max number of triples from an rc file */
#define MAXRCLINES 4096
/* Define a struct that all oc objects must start with */
/* OCheader must be defined here to make it available in other headers */
typedef struct OCheader {
unsigned int magic;
unsigned int occlass;
} OCheader;
#include "oc.h"
#include "ocx.h"
#include "ocnode.h"
#include "ocdata.h"
#include "occonstraints.h"
#include "ocutil.h"
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
#include "nclog.h"
#include "xxdr.h"
#include "ocdatatypes.h"
#include "occompile.h"
#ifndef nulldup
#define nulldup(s) (s==NULL?NULL:strdup(s))
#endif
/*
* Macros for dealing with flag bits.
*/
#define fset(t,f) ((t) |= (f))
#define fclr(t,f) ((t) &= ~(f))
#define fisset(t,f) ((t) & (f))
#define nullstring(s) (s==NULL?"(null)":s)
#define PATHSEPARATOR "."
#define OCCOOKIEDIR "occookies"
2013-11-15 06:13:20 +08:00
/* Define infinity for memory size */
#if SIZEOF_SIZE_T == 4
#define OCINFINITY ((size_t)0xffffffff)
#else
#define OCINFINITY ((size_t)0xffffffffffffffff)
#endif
/* Extend the OCdxd type for internal use */
#define OCVER 3
/* Default initial memory packet size */
#define DFALTPACKETSIZE 0x20000 /*approximately 100k bytes*/
/* Default maximum memory packet size */
#define DFALTMAXPACKETSIZE 0x3000000 /*approximately 50M bytes*/
2012-12-04 11:32:41 +08:00
/* Default user agent string (will have version appended)*/
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
#ifndef DFALTUSERAGENT
2012-12-04 11:32:41 +08:00
#define DFALTUSERAGENT "oc"
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
#endif
2012-12-04 11:32:41 +08:00
#if 0
/* Hold known curl flags */
enum OCCURLFLAGTYPE {CF_UNKNOWN=0,CF_OTHER=1,CF_STRING=2,CF_LONG=3};
struct OCCURLFLAG {
const char* name;
int flag;
int value;
enum OCCURLFLAGTYPE type;
};
#endif
struct OCTriplestore {
int ntriples;
struct OCTriple {
char host[MAXRCLINESIZE]; /* includes port if specified */
char key[MAXRCLINESIZE];
char value[MAXRCLINESIZE];
} triples[MAXRCLINES];
};
/*! Specifies the OCstate = non-opaque version of OClink */
struct OCstate {
OCheader header; /* class=OC_State */
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
NClist* trees; /* list<OCNODE*> ; all root objects */
NCURI* uri; /* parsed base URI*/
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
NCbytes* packet; /* shared by all trees during construction */
struct OCerrdata {/* Hold info for an error return from server */
char* code;
char* message;
long httpcode;
char curlerrorbuf[CURL_ERROR_SIZE]; /* CURLOPT_ERRORBUFFER*/
} error;
CURL* curl; /* curl handle*/
char curlerror[CURL_ERROR_SIZE];
void* usercurldata;
NCauth* auth; /* curl auth data */
long ddslastmodified;
long datalastmodified;
long curlbuffersize;
struct {
int active; /* Activate keepalive? */
long idle; /* KEEPIDLE value */
long interval; /* KEEPINTVL value */
} curlkeepalive; /* keepalive info */
};
/*! OCtree holds extra state info about trees */
typedef struct OCtree
{
OCdxd dxdclass;
char* constraint;
char* text;
struct OCnode* root; /* cross link */
struct OCstate* state; /* cross link */
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
NClist* nodes; /* all nodes in tree*/
/* when dxdclass == OCDATADDS */
struct {
char* memory; /* allocated memory if OC_ONDISK is not set */
char* filename; /* If OC_ONDISK is set */
FILE* file;
off_t datasize; /* xdr size on disk or in memory */
off_t bod; /* offset of the beginning of packet data */
off_t ddslen; /* length of ddslen (assert(ddslen <= bod)) */
XXDR* xdrs; /* access either memory or file */
OCdata* data;
} data;
} OCtree;
/* (Almost) All shared procedure definitions are kept here
except for: ocdebug.h ocutil.h
The true external interface is defined in oc.h
*/
#if 0
/* Location: ceparselex.c*/
extern int cedebug;
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
extern NClist* CEparse(OCstate*,char* input);
#endif
extern int ocinitialized;
extern OCerror ocopen(OCstate** statep, const char* url);
extern void occlose(OCstate* state);
extern OCerror ocfetch(OCstate*, const char*, OCdxd, OCflags, OCnode**);
extern int oc_network_order;
extern int oc_invert_xdr_double;
extern OCerror ocinternalinitialize(void);
Yet another fix for DAP2 double URL encoding. re: https://github.com/Unidata/netcdf-c/issues/1876 and: https://github.com/Unidata/netcdf-c/pull/1835 and: https://github.com/Unidata/netcdf4-python/issues/1041 The change in PR 1835 was correct with respect to using %20 instead of '+' for encoding blanks. However, it was a mistake to assume everything was unencoded and then to do encoding ourselves. The problem is that different servers do different things, with Columbia being an outlier. So, I have added a set of client controls that can at least give the caller some control over this. The caller can append the following fragment to his URL to control what gets encoded before sending it to the server. The syntax is as follows: ```` https://<host>/<path>/<query>#encode=path|query|all|none ```` The possible values: * path -- URL encode (i.e. %xx encode) as needed in the path part of the URL. * query -- URL encode as needed in the query part of the URL. * all -- equivalent to ````#encode=path,query````. * none -- do not url encode any part of the URL sent to the server; not strictly necessary, so mostly for completeness. Note that if "encode=" is used, then before it is processed, all encoding is turned of so that ````#encode=path```` will only encode the path and not the query. The default is ````#encode=query````, so the path is left untouched, but the query is always encoded. Internally, this required changes to pass the encode flags down into the OC2 library. Misc. Unrelated Changes: * Shut up those irritating warning from putget.m4
2020-11-06 02:04:56 +08:00
extern OCerror ocupdatelastmodifieddata(OCstate* state, OCflags);
2014-01-22 03:50:15 +08:00
extern OCerror ocset_useragent(OCstate* state, const char* agent);
extern OCerror ocset_netrc(OCstate* state, const char* path);
/* From ocrc.c */
extern OCerror ocrc_load(void); /* find, read, and compile */
extern OCerror ocrc_process(OCstate* state); /* extract relevant triples */
extern char* ocrc_lookup(char* key, char* url);
extern struct OCTriple* ocrc_triple_iterate(char* key, char* url, struct OCTriple* prevp);
extern int ocparseproxy(OCstate* state, char* v);
2012-12-04 11:32:41 +08:00
#endif /*COMMON_H*/