netcdf-c/include/ncdispatch.h

258 lines
7.3 KiB
C
Raw Normal View History

/* Copyright 2018-2018 University Corporation for Atmospheric
2018-11-27 02:22:32 +08:00
Research/Unidata. */
/**
* @file
* @internal Includes prototypes for core dispatch functionality.
*
* @author Dennis Heimbigner
*/
2010-06-03 21:24:43 +08:00
Provide byte-range reading of remote datasets re: issue https://github.com/Unidata/netcdf-c/issues/1251 Assume that you have the URL to a remote dataset which is a normal netcdf-3 or netcdf-4 file. This PR allows the netcdf-c to read that dataset's contents as a netcdf file using HTTP byte ranges if the remote server supports byte-range access. Originally, this PR was set up to access Amazon S3 objects, but it can also access other remote datasets such as those provided by a Thredds server via the HTTPServer access protocol. It may also work for other kinds of servers. Note that this is not intended as a true production capability because, as is known, this kind of access to can be quite slow. In addition, the byte-range IO drivers do not currently do any sort of optimization or caching. An additional goal here is to gain some experience with the Amazon S3 REST protocol. This architecture and its use documented in the file docs/byterange.dox. There are currently two test cases: 1. nc_test/tst_s3raw.c - this does a simple open, check format, close cycle for a remote netcdf-3 file and a remote netcdf-4 file. 2. nc_test/test_s3raw.sh - this uses ncdump to investigate some remote datasets. This PR also incorporates significantly changed model inference code (see the superceded PR https://github.com/Unidata/netcdf-c/pull/1259). 1. It centralizes the code that infers the dispatcher. 2. It adds support for byte-range URLs Other changes: 1. NC_HDF5_finalize was not being properly called by nc_finalize(). 2. Fix minor bug in ncgen3.l 3. fix memory leak in nc4info.c 4. add code to walk the .daprc triples and to replace protocol= fragment tag with a more general mode= tag. Final Note: Th inference code is still way too complicated. We need to move to the validfile() model used by netcdf Java, where each dispatcher is asked if it can process the file. This decentralizes the inference code. This will be done after all the major new dispatchers (PIO, Zarr, etc) have been implemented.
2019-01-02 09:27:36 +08:00
#ifndef NC_DISPATCH_H
#define NC_DISPATCH_H
2010-06-03 21:24:43 +08:00
2017-12-21 10:53:30 +08:00
#if HAVE_CONFIG_H
2010-06-03 21:24:43 +08:00
#include "config.h"
2017-12-21 10:53:30 +08:00
#endif
2010-06-03 21:24:43 +08:00
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <assert.h>
#if defined(HDF5_PARALLEL) || defined(USE_PNETCDF)
2015-08-16 06:26:35 +08:00
#include <mpi.h>
#endif
2010-06-03 21:24:43 +08:00
#include "netcdf.h"
Provide byte-range reading of remote datasets re: issue https://github.com/Unidata/netcdf-c/issues/1251 Assume that you have the URL to a remote dataset which is a normal netcdf-3 or netcdf-4 file. This PR allows the netcdf-c to read that dataset's contents as a netcdf file using HTTP byte ranges if the remote server supports byte-range access. Originally, this PR was set up to access Amazon S3 objects, but it can also access other remote datasets such as those provided by a Thredds server via the HTTPServer access protocol. It may also work for other kinds of servers. Note that this is not intended as a true production capability because, as is known, this kind of access to can be quite slow. In addition, the byte-range IO drivers do not currently do any sort of optimization or caching. An additional goal here is to gain some experience with the Amazon S3 REST protocol. This architecture and its use documented in the file docs/byterange.dox. There are currently two test cases: 1. nc_test/tst_s3raw.c - this does a simple open, check format, close cycle for a remote netcdf-3 file and a remote netcdf-4 file. 2. nc_test/test_s3raw.sh - this uses ncdump to investigate some remote datasets. This PR also incorporates significantly changed model inference code (see the superceded PR https://github.com/Unidata/netcdf-c/pull/1259). 1. It centralizes the code that infers the dispatcher. 2. It adds support for byte-range URLs Other changes: 1. NC_HDF5_finalize was not being properly called by nc_finalize(). 2. Fix minor bug in ncgen3.l 3. fix memory leak in nc4info.c 4. add code to walk the .daprc triples and to replace protocol= fragment tag with a more general mode= tag. Final Note: Th inference code is still way too complicated. We need to move to the validfile() model used by netcdf Java, where each dispatcher is asked if it can process the file. This decentralizes the inference code. This will be done after all the major new dispatchers (PIO, Zarr, etc) have been implemented.
2019-01-02 09:27:36 +08:00
#include "ncmodel.h"
2010-06-03 21:24:43 +08:00
#include "nc.h"
#include "ncuri.h"
#ifdef USE_PARALLEL
#include "netcdf_par.h"
#endif
#include "netcdf_dispatch.h"
2010-06-03 21:24:43 +08:00
2011-07-15 06:24:02 +08:00
#define longtype ((sizeof(long) == sizeof(int) ? NC_INT : NC_INT64))
2011-04-17 04:56:36 +08:00
#define X_INT_MAX 2147483647
2010-06-03 21:24:43 +08:00
/* Given a filename, check its magic number */
re e-support UBS-599337 re pull request https://github.com/Unidata/netcdf-c/pull/405 re pull request https://github.com/Unidata/netcdf-c/pull/446 Notes: 1. This branch is a cleanup of the magic.dmh branch. 2. magic.dmh was originally merged, but caused problems with parallel IO. It was re-issued as pull request https://github.com/Unidata/netcdf-c/pull/446. 3. This branch + pull request replace any previous pull requests and magic.dmh branch. Given an otherwise valid netCDF file that has a corrupted header, the netcdf library currently crashes. Instead, it should return NC_ENOTNC. Additionally, the NC_check_file_type code does not do the forward search required by hdf5 files. It currently only looks at file position 0 instead of 512, 1024, 2048,... Also, it turns out that the HDF4 magic number is assumed to always be at the beginning of the file (unlike HDF5). The change is localized to libdispatch/dfile.c See https://support.hdfgroup.org/release4/doc/DSpec_html/DS.pdf Also, it turns out that the code in NC_check_file_type is duplicated (mostly) in the function libsrc4/nc4file.c#nc_check_for_hdf. This branch does the following. 1. Make NC_check_file_type return NC_ENOTNC instead of crashing. 2. Remove nc_check_for_hdf and centralize all file format checking NC_check_file_type. 3. Add proper forward search for HDF5 files (but not HDF4 files) to look for the magic number at offsets of 0, 512, 1024... 4. Add test tst_hdf5_offset.sh. This tests that hdf5 files with an offset are properly recognized. It does so by prefixing a legal file with some number of zero bytes: 512, 1024, etc. 5. Off-topic: Added -N flag to ncdump to force a specific output dataset name.
2017-10-25 06:25:09 +08:00
/* Change magic number size from 4 to 8 to be more precise for HDF5 */
#define MAGIC_NUMBER_LEN ((size_t)8)
2010-06-03 21:24:43 +08:00
#define MAGIC_HDF5_FILE 1
#define MAGIC_HDF4_FILE 2
#define MAGIC_CDF1_FILE 1 /* std classic format */
#define MAGIC_CDF2_FILE 2 /* classic 64 bit */
/* Define the mappings from fcn name types
to corresponding NC types. */
#define T_text NC_CHAR
#define T_schar NC_BYTE
#define T_char NC_CHAR
#define T_short NC_SHORT
#define T_int NC_INT
#define T_float NC_FLOAT
#define T_double NC_DOUBLE
#define T_ubyte NC_UBYTE
#define T_ushort NC_USHORT
#define T_uint NC_UINT
#define T_longlong NC_INT64
#define T_ulonglong NC_UINT64
#ifdef USE_NETCDF4
#define T_string NC_STRING
#endif
/* Synthetic type to handle special memtypes */
#define T_uchar NC_UBYTE
#define T_long longtype
#define T_ulong ulongtype
/**************************************************/
2013-03-16 04:31:07 +08:00
2010-06-03 21:24:43 +08:00
/* Define a type for use when doing e.g. nc_get_vara_long, etc. */
/* Should matche values in libsrc4/netcdf.h */
#ifndef NC_UINT64
#define NC_UBYTE 7 /* unsigned 1 byte int */
#define NC_USHORT 8 /* unsigned 2-byte int */
#define NC_UINT 9 /* unsigned 4-byte int */
#define NC_INT64 10 /* signed 8-byte int */
#define NC_UINT64 11 /* unsigned 8-byte int */
#define NC_STRING 12 /* char* */
#endif
/* Define the range of Atomic types */
2015-08-16 06:26:35 +08:00
#define ATOMICTYPEMAX4 NC_STRING
#define ATOMICTYPEMAX3 NC_DOUBLE
#define ATOMICTYPEMAX5 NC_UINT64
2010-06-03 21:24:43 +08:00
2010-12-16 05:45:05 +08:00
/* Define an alias for int to indicate an error return */
typedef int NCerror;
2015-08-16 06:26:35 +08:00
#if !defined HDF5_PARALLEL && !defined USE_PNETCDF
typedef int MPI_Comm;
typedef int MPI_Info;
2013-03-16 04:31:07 +08:00
#define MPI_COMM_WORLD 0
#define MPI_INFO_NULL 0
2015-08-16 06:26:35 +08:00
#endif
2013-03-16 04:31:07 +08:00
2010-06-03 21:24:43 +08:00
/* Define a struct to hold the MPI info so it can be passed down the
* call stack. This is used internally by the netCDF library. It
* should not be used by netcdf users. */
typedef struct NC_MPI_INFO {
MPI_Comm comm;
MPI_Info info;
} NC_MPI_INFO;
2011-09-19 04:57:51 +08:00
/* Define known dispatch tables and initializers */
extern int NCDISPATCH_initialize(void);
2015-08-16 06:26:35 +08:00
extern int NCDISPATCH_finalize(void);
extern const NC_Dispatch* NC3_dispatch_table;
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
extern int NC3_initialize(void);
extern int NC3_finalize(void);
2011-09-19 04:57:51 +08:00
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
#ifdef ENABLE_DAP
extern const NC_Dispatch* NCD2_dispatch_table;
extern int NCD2_initialize(void);
2015-08-16 06:26:35 +08:00
extern int NCD2_finalize(void);
2010-06-03 21:24:43 +08:00
#endif
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
#ifdef ENABLE_DAP4
extern const NC_Dispatch* NCD4_dispatch_table;
Primary change: add dap4 support Specific changes: 1. Add dap4 code: libdap4 and dap4_test. Note that until the d4ts server problem is solved, dap4 is turned off. 2. Modify various files to support dap4 flags: configure.ac, Makefile.am, CMakeLists.txt, etc. 3. Add nc_test/test_common.sh. This centralizes the handling of the locations of various things in the build tree: e.g. where is ncgen.exe located. See nc_test/test_common.sh for details. 4. Modify .sh files to use test_common.sh 5. Obsolete separate oc2 by moving it to be part of netcdf-c. This means replacing code with netcdf-c equivalents. 5. Add --with-testserver to configure.ac to allow override of the servers to be used for --enable-dap-remote-tests. 6. There were multiple versions of nctypealignment code. Try to centralize in libdispatch/doffset.c and include/ncoffsets.h 7. Add a unit test for the ncuri code because of its complexity. 8. Move the findserver code out of libdispatch and into a separate, self contained program in ncdap_test and dap4_test. 9. Move the dispatch header files (nc{3,4}dispatch.h) to .../include because they are now shared by modules. 10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts. 11. Make use of MREMAP if available 12. Misc. minor changes e.g. - #include <config.h> -> #include "config.h" - Add some no-install headers to /include - extern -> EXTERNL and vice versa as needed - misc header cleanup - clean up checking for misc. unix vs microsoft functions 13. Change copyright decls in some files to point to LICENSE file. 14. Add notes to RELEASENOTES.md
2017-03-09 08:01:10 +08:00
extern int NCD4_initialize(void);
extern int NCD4_finalize(void);
#endif
2010-06-03 21:24:43 +08:00
2013-03-16 04:31:07 +08:00
#ifdef USE_PNETCDF
extern const NC_Dispatch* NCP_dispatch_table;
extern int NCP_initialize(void);
2015-08-16 06:26:35 +08:00
extern int NCP_finalize(void);
2013-03-16 04:31:07 +08:00
#endif
2011-09-21 01:30:02 +08:00
#ifdef USE_NETCDF4
2011-09-21 04:39:04 +08:00
extern int NC4_initialize(void);
2015-08-16 06:26:35 +08:00
extern int NC4_finalize(void);
2015-08-17 11:44:18 +08:00
#endif
2011-09-21 04:39:04 +08:00
#ifdef USE_HDF5
extern const NC_Dispatch* HDF5_dispatch_table;
extern int NC_HDF5_initialize(void);
extern int NC_HDF5_finalize(void);
#endif
2018-02-08 21:20:58 +08:00
#ifdef USE_HDF4
extern const NC_Dispatch* HDF4_dispatch_table;
2018-02-08 21:20:58 +08:00
extern int HDF4_initialize(void);
extern int HDF4_finalize(void);
#endif
/* User-defined formats.*/
Provide byte-range reading of remote datasets re: issue https://github.com/Unidata/netcdf-c/issues/1251 Assume that you have the URL to a remote dataset which is a normal netcdf-3 or netcdf-4 file. This PR allows the netcdf-c to read that dataset's contents as a netcdf file using HTTP byte ranges if the remote server supports byte-range access. Originally, this PR was set up to access Amazon S3 objects, but it can also access other remote datasets such as those provided by a Thredds server via the HTTPServer access protocol. It may also work for other kinds of servers. Note that this is not intended as a true production capability because, as is known, this kind of access to can be quite slow. In addition, the byte-range IO drivers do not currently do any sort of optimization or caching. An additional goal here is to gain some experience with the Amazon S3 REST protocol. This architecture and its use documented in the file docs/byterange.dox. There are currently two test cases: 1. nc_test/tst_s3raw.c - this does a simple open, check format, close cycle for a remote netcdf-3 file and a remote netcdf-4 file. 2. nc_test/test_s3raw.sh - this uses ncdump to investigate some remote datasets. This PR also incorporates significantly changed model inference code (see the superceded PR https://github.com/Unidata/netcdf-c/pull/1259). 1. It centralizes the code that infers the dispatcher. 2. It adds support for byte-range URLs Other changes: 1. NC_HDF5_finalize was not being properly called by nc_finalize(). 2. Fix minor bug in ncgen3.l 3. fix memory leak in nc4info.c 4. add code to walk the .daprc triples and to replace protocol= fragment tag with a more general mode= tag. Final Note: Th inference code is still way too complicated. We need to move to the validfile() model used by netcdf Java, where each dispatcher is asked if it can process the file. This decentralizes the inference code. This will be done after all the major new dispatchers (PIO, Zarr, etc) have been implemented.
2019-01-02 09:27:36 +08:00
extern NC_Dispatch* UDF0_dispatch_table;
extern char UDF0_magic_number[NC_MAX_MAGIC_NUMBER_LEN + 1];
extern NC_Dispatch* UDF1_dispatch_table;
extern char UDF1_magic_number[NC_MAX_MAGIC_NUMBER_LEN + 1];
2018-08-14 22:56:58 +08:00
/* Prototypes. */
int NC_check_nulls(int ncid, int varid, const size_t *start, size_t **count,
ptrdiff_t **stride);
2010-06-03 21:24:43 +08:00
/**************************************************/
/* Forward */
#ifndef USE_NETCDF4
/* Taken from libsrc4/netcdf.h */
struct nc_vlen_t;
#define NC_NETCDF4 0x1000
#define NC_CLASSIC_MODEL 0x0100
#define NC_ENOPAR (-114)
2012-08-02 01:18:58 +08:00
#endif /*!USE_NETCDF4*/
2010-06-03 21:24:43 +08:00
struct NC;
int NC_create(const char *path, int cmode,
2015-08-16 06:26:35 +08:00
size_t initialsz, int basepe, size_t *chunksizehintp,
2018-09-23 09:22:34 +08:00
int useparallel, void *parameters, int *ncidp);
2010-06-03 21:24:43 +08:00
int NC_open(const char *path, int cmode,
int basepe, size_t *chunksizehintp,
2018-09-23 09:22:34 +08:00
int useparallel, void *parameters, int *ncidp);
2010-06-03 21:24:43 +08:00
/* Expose the default vars and varm dispatch entries */
2018-05-30 16:50:09 +08:00
EXTERNL int NCDEFAULT_get_vars(int, int, const size_t*,
const size_t*, const ptrdiff_t*, void*, nc_type);
2018-05-30 16:50:09 +08:00
EXTERNL int NCDEFAULT_put_vars(int, int, const size_t*,
const size_t*, const ptrdiff_t*, const void*, nc_type);
2018-05-30 16:50:09 +08:00
EXTERNL int NCDEFAULT_get_varm(int, int, const size_t*,
const size_t*, const ptrdiff_t*, const ptrdiff_t*,
void*, nc_type);
2018-05-30 16:50:09 +08:00
EXTERNL int NCDEFAULT_put_varm(int, int, const size_t*,
const size_t*, const ptrdiff_t*, const ptrdiff_t*,
const void*, nc_type);
2010-06-03 21:24:43 +08:00
/**************************************************/
/* Forward */
struct NCHDR;
/* Following functions must be handled as non-dispatch */
#ifdef NONDISPATCH
2015-08-16 06:26:35 +08:00
void (*nc_advise)(const char*cdf_routine_name,interr,const char*fmt,...);
void (*nc_set_log_level)(int);
2010-06-03 21:24:43 +08:00
const char* (*nc_inq_libvers)(void);
const char* (*nc_strerror)(int);
2015-08-16 06:26:35 +08:00
int (*nc_delete)(const char*path);
int (*nc_delete_mp)(const char*path,intbasepe);
int (*nc_initialize)();
int (*nc_finalize)();
2010-06-03 21:24:43 +08:00
#endif /*NONDISPATCH*/
/* Define the common fields for NC and NC_FILE_INFO_T etc */
typedef struct NCcommon {
int ext_ncid; /* uid << 16 */
int int_ncid; /* unspecified other id */
const struct NC_Dispatch* dispatch;
2011-09-17 02:36:08 +08:00
void* dispatchdata; /* per-protocol instance data */
char* path; /* as specified at open or create */
2010-06-03 21:24:43 +08:00
} NCcommon;
2018-05-30 16:50:09 +08:00
EXTERNL size_t NC_atomictypelen(nc_type xtype);
EXTERNL char* NC_atomictypename(nc_type xtype);
2010-06-03 21:24:43 +08:00
2010-12-16 05:45:05 +08:00
/* Misc */
extern int NC_getshape(int ncid, int varid, int ndims, size_t* shape);
2011-07-15 06:24:02 +08:00
extern int NC_is_recvar(int ncid, int varid, size_t* nrecs);
extern int NC_inq_recvar(int ncid, int varid, int* nrecdims, int* is_recdim);
2010-12-16 05:45:05 +08:00
#define nullstring(s) (s==NULL?"(null)":s)
2016-04-07 09:51:40 +08:00
#undef TRACECALLS
#ifdef TRACECALLS
#include <stdio.h>
#define TRACE(fname) fprintf(stderr,"call: %s\n",#fname)
#else
#define TRACE(fname)
#endif
/* Vectors of ones and zeros */
extern size_t NC_coord_zero[NC_MAX_VAR_DIMS];
extern size_t NC_coord_one[NC_MAX_VAR_DIMS];
extern ptrdiff_t NC_stride_one[NC_MAX_VAR_DIMS];
2015-08-16 06:26:35 +08:00
extern int NC_initialized;
2016-04-07 09:51:40 +08:00
/**
Certain functions are in the dispatch table,
but not in the netcdf.h API. These need to
be exposed for use in delegation such as
in libdap2.
*/
2018-05-30 16:50:09 +08:00
EXTERNL int
2016-04-07 09:51:40 +08:00
NCDISPATCH_inq_var_all(int ncid, int varid, char *name, nc_type *xtypep,
2016-04-07 04:05:58 +08:00
int *ndimsp, int *dimidsp, int *nattsp,
int *shufflep, int *deflatep, int *deflate_levelp,
int *fletcher32p, int *contiguousp, size_t *chunksizesp,
int *no_fill, void *fill_valuep, int *endiannessp,
unsigned int* idp, size_t* nparamsp, unsigned int* paramsp
);
2018-05-30 16:50:09 +08:00
EXTERNL int
2016-04-07 09:51:40 +08:00
NCDISPATCH_get_att(int ncid, int varid, const char* name, void* value, nc_type t);
Provide byte-range reading of remote datasets re: issue https://github.com/Unidata/netcdf-c/issues/1251 Assume that you have the URL to a remote dataset which is a normal netcdf-3 or netcdf-4 file. This PR allows the netcdf-c to read that dataset's contents as a netcdf file using HTTP byte ranges if the remote server supports byte-range access. Originally, this PR was set up to access Amazon S3 objects, but it can also access other remote datasets such as those provided by a Thredds server via the HTTPServer access protocol. It may also work for other kinds of servers. Note that this is not intended as a true production capability because, as is known, this kind of access to can be quite slow. In addition, the byte-range IO drivers do not currently do any sort of optimization or caching. An additional goal here is to gain some experience with the Amazon S3 REST protocol. This architecture and its use documented in the file docs/byterange.dox. There are currently two test cases: 1. nc_test/tst_s3raw.c - this does a simple open, check format, close cycle for a remote netcdf-3 file and a remote netcdf-4 file. 2. nc_test/test_s3raw.sh - this uses ncdump to investigate some remote datasets. This PR also incorporates significantly changed model inference code (see the superceded PR https://github.com/Unidata/netcdf-c/pull/1259). 1. It centralizes the code that infers the dispatcher. 2. It adds support for byte-range URLs Other changes: 1. NC_HDF5_finalize was not being properly called by nc_finalize(). 2. Fix minor bug in ncgen3.l 3. fix memory leak in nc4info.c 4. add code to walk the .daprc triples and to replace protocol= fragment tag with a more general mode= tag. Final Note: Th inference code is still way too complicated. We need to move to the validfile() model used by netcdf Java, where each dispatcher is asked if it can process the file. This decentralizes the inference code. This will be done after all the major new dispatchers (PIO, Zarr, etc) have been implemented.
2019-01-02 09:27:36 +08:00
#endif /* NC_DISPATCH_H */