2018-12-07 05:13:56 +08:00
|
|
|
/* Copyright 2018-2018 University Corporation for Atmospheric
|
2018-11-27 02:22:32 +08:00
|
|
|
Research/Unidata. */
|
|
|
|
|
/**
|
|
|
|
|
* @file
|
|
|
|
|
* @internal Includes prototypes for core dispatch functionality.
|
|
|
|
|
*
|
|
|
|
|
* @author Dennis Heimbigner
|
|
|
|
|
*/
|
2010-06-03 21:24:43 +08:00
|
|
|
|
Provide byte-range reading of remote datasets
re: issue https://github.com/Unidata/netcdf-c/issues/1251
Assume that you have the URL to a remote dataset
which is a normal netcdf-3 or netcdf-4 file.
This PR allows the netcdf-c to read that dataset's
contents as a netcdf file using HTTP byte ranges
if the remote server supports byte-range access.
Originally, this PR was set up to access Amazon S3 objects,
but it can also access other remote datasets such as those
provided by a Thredds server via the HTTPServer access protocol.
It may also work for other kinds of servers.
Note that this is not intended as a true production
capability because, as is known, this kind of access to
can be quite slow. In addition, the byte-range IO drivers
do not currently do any sort of optimization or caching.
An additional goal here is to gain some experience with
the Amazon S3 REST protocol.
This architecture and its use documented in
the file docs/byterange.dox.
There are currently two test cases:
1. nc_test/tst_s3raw.c - this does a simple open, check format, close cycle
for a remote netcdf-3 file and a remote netcdf-4 file.
2. nc_test/test_s3raw.sh - this uses ncdump to investigate some remote
datasets.
This PR also incorporates significantly changed model inference code
(see the superceded PR https://github.com/Unidata/netcdf-c/pull/1259).
1. It centralizes the code that infers the dispatcher.
2. It adds support for byte-range URLs
Other changes:
1. NC_HDF5_finalize was not being properly called by nc_finalize().
2. Fix minor bug in ncgen3.l
3. fix memory leak in nc4info.c
4. add code to walk the .daprc triples and to replace protocol=
fragment tag with a more general mode= tag.
Final Note:
Th inference code is still way too complicated. We need to move
to the validfile() model used by netcdf Java, where each
dispatcher is asked if it can process the file. This decentralizes
the inference code. This will be done after all the major new
dispatchers (PIO, Zarr, etc) have been implemented.
2019-01-02 09:27:36 +08:00
|
|
|
#ifndef NC_DISPATCH_H
|
|
|
|
|
#define NC_DISPATCH_H
|
2010-06-03 21:24:43 +08:00
|
|
|
|
2017-12-21 10:53:30 +08:00
|
|
|
#if HAVE_CONFIG_H
|
2010-06-03 21:24:43 +08:00
|
|
|
#include "config.h"
|
2017-12-21 10:53:30 +08:00
|
|
|
#endif
|
2010-06-03 21:24:43 +08:00
|
|
|
#include <stdlib.h>
|
|
|
|
|
#include <stdio.h>
|
|
|
|
|
#include <string.h>
|
|
|
|
|
#include <assert.h>
|
2015-11-10 02:21:04 +08:00
|
|
|
#if defined(HDF5_PARALLEL) || defined(USE_PNETCDF)
|
2015-08-16 06:26:35 +08:00
|
|
|
#include <mpi.h>
|
|
|
|
|
#endif
|
2010-06-03 21:24:43 +08:00
|
|
|
#include "netcdf.h"
|
Provide byte-range reading of remote datasets
re: issue https://github.com/Unidata/netcdf-c/issues/1251
Assume that you have the URL to a remote dataset
which is a normal netcdf-3 or netcdf-4 file.
This PR allows the netcdf-c to read that dataset's
contents as a netcdf file using HTTP byte ranges
if the remote server supports byte-range access.
Originally, this PR was set up to access Amazon S3 objects,
but it can also access other remote datasets such as those
provided by a Thredds server via the HTTPServer access protocol.
It may also work for other kinds of servers.
Note that this is not intended as a true production
capability because, as is known, this kind of access to
can be quite slow. In addition, the byte-range IO drivers
do not currently do any sort of optimization or caching.
An additional goal here is to gain some experience with
the Amazon S3 REST protocol.
This architecture and its use documented in
the file docs/byterange.dox.
There are currently two test cases:
1. nc_test/tst_s3raw.c - this does a simple open, check format, close cycle
for a remote netcdf-3 file and a remote netcdf-4 file.
2. nc_test/test_s3raw.sh - this uses ncdump to investigate some remote
datasets.
This PR also incorporates significantly changed model inference code
(see the superceded PR https://github.com/Unidata/netcdf-c/pull/1259).
1. It centralizes the code that infers the dispatcher.
2. It adds support for byte-range URLs
Other changes:
1. NC_HDF5_finalize was not being properly called by nc_finalize().
2. Fix minor bug in ncgen3.l
3. fix memory leak in nc4info.c
4. add code to walk the .daprc triples and to replace protocol=
fragment tag with a more general mode= tag.
Final Note:
Th inference code is still way too complicated. We need to move
to the validfile() model used by netcdf Java, where each
dispatcher is asked if it can process the file. This decentralizes
the inference code. This will be done after all the major new
dispatchers (PIO, Zarr, etc) have been implemented.
2019-01-02 09:27:36 +08:00
|
|
|
#include "ncmodel.h"
|
2010-06-03 21:24:43 +08:00
|
|
|
#include "nc.h"
|
2012-08-09 07:15:18 +08:00
|
|
|
#include "ncuri.h"
|
2018-10-08 02:05:26 +08:00
|
|
|
#ifdef USE_PARALLEL
|
|
|
|
|
#include "netcdf_par.h"
|
|
|
|
|
#endif
|
2019-07-05 18:53:53 +08:00
|
|
|
#include "netcdf_dispatch.h"
|
2010-06-03 21:24:43 +08:00
|
|
|
|
2011-07-15 06:24:02 +08:00
|
|
|
#define longtype ((sizeof(long) == sizeof(int) ? NC_INT : NC_INT64))
|
2011-04-17 04:56:36 +08:00
|
|
|
|
2010-06-18 22:01:51 +08:00
|
|
|
#define X_INT_MAX 2147483647
|
2010-06-03 21:24:43 +08:00
|
|
|
|
|
|
|
|
/* Given a filename, check its magic number */
|
re e-support UBS-599337
re pull request https://github.com/Unidata/netcdf-c/pull/405
re pull request https://github.com/Unidata/netcdf-c/pull/446
Notes:
1. This branch is a cleanup of the magic.dmh branch.
2. magic.dmh was originally merged, but caused problems with parallel IO.
It was re-issued as pull request https://github.com/Unidata/netcdf-c/pull/446.
3. This branch + pull request replace any previous pull requests and magic.dmh branch.
Given an otherwise valid netCDF file that has a corrupted header,
the netcdf library currently crashes. Instead, it should return
NC_ENOTNC.
Additionally, the NC_check_file_type code does not do the
forward search required by hdf5 files. It currently only looks
at file position 0 instead of 512, 1024, 2048,... Also, it turns
out that the HDF4 magic number is assumed to always be at the
beginning of the file (unlike HDF5).
The change is localized to libdispatch/dfile.c See
https://support.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Also, it turns out that the code in NC_check_file_type is duplicated
(mostly) in the function libsrc4/nc4file.c#nc_check_for_hdf.
This branch does the following.
1. Make NC_check_file_type return NC_ENOTNC instead of crashing.
2. Remove nc_check_for_hdf and centralize all file format checking
NC_check_file_type.
3. Add proper forward search for HDF5 files (but not HDF4 files)
to look for the magic number at offsets of 0, 512, 1024...
4. Add test tst_hdf5_offset.sh. This tests that hdf5 files with
an offset are properly recognized. It does so by prefixing
a legal file with some number of zero bytes: 512, 1024, etc.
5. Off-topic: Added -N flag to ncdump to force a specific output dataset name.
2017-10-25 06:25:09 +08:00
|
|
|
/* Change magic number size from 4 to 8 to be more precise for HDF5 */
|
2018-02-25 11:36:24 +08:00
|
|
|
#define MAGIC_NUMBER_LEN ((size_t)8)
|
2010-06-03 21:24:43 +08:00
|
|
|
#define MAGIC_HDF5_FILE 1
|
|
|
|
|
#define MAGIC_HDF4_FILE 2
|
|
|
|
|
#define MAGIC_CDF1_FILE 1 /* std classic format */
|
|
|
|
|
#define MAGIC_CDF2_FILE 2 /* classic 64 bit */
|
|
|
|
|
|
|
|
|
|
/* Define the mappings from fcn name types
|
|
|
|
|
to corresponding NC types. */
|
|
|
|
|
#define T_text NC_CHAR
|
|
|
|
|
#define T_schar NC_BYTE
|
|
|
|
|
#define T_char NC_CHAR
|
|
|
|
|
#define T_short NC_SHORT
|
|
|
|
|
#define T_int NC_INT
|
|
|
|
|
#define T_float NC_FLOAT
|
|
|
|
|
#define T_double NC_DOUBLE
|
|
|
|
|
#define T_ubyte NC_UBYTE
|
|
|
|
|
#define T_ushort NC_USHORT
|
|
|
|
|
#define T_uint NC_UINT
|
|
|
|
|
#define T_longlong NC_INT64
|
|
|
|
|
#define T_ulonglong NC_UINT64
|
|
|
|
|
#define T_string NC_STRING
|
|
|
|
|
|
|
|
|
|
/* Synthetic type to handle special memtypes */
|
|
|
|
|
#define T_uchar NC_UBYTE
|
|
|
|
|
#define T_long longtype
|
|
|
|
|
#define T_ulong ulongtype
|
|
|
|
|
|
|
|
|
|
/**************************************************/
|
2013-03-16 04:31:07 +08:00
|
|
|
|
2010-06-03 21:24:43 +08:00
|
|
|
/* Define a type for use when doing e.g. nc_get_vara_long, etc. */
|
|
|
|
|
/* Should matche values in libsrc4/netcdf.h */
|
|
|
|
|
#ifndef NC_UINT64
|
|
|
|
|
#define NC_UBYTE 7 /* unsigned 1 byte int */
|
|
|
|
|
#define NC_USHORT 8 /* unsigned 2-byte int */
|
|
|
|
|
#define NC_UINT 9 /* unsigned 4-byte int */
|
|
|
|
|
#define NC_INT64 10 /* signed 8-byte int */
|
|
|
|
|
#define NC_UINT64 11 /* unsigned 8-byte int */
|
|
|
|
|
#define NC_STRING 12 /* char* */
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/* Define the range of Atomic types */
|
2015-08-16 06:26:35 +08:00
|
|
|
#define ATOMICTYPEMAX4 NC_STRING
|
|
|
|
|
#define ATOMICTYPEMAX3 NC_DOUBLE
|
|
|
|
|
#define ATOMICTYPEMAX5 NC_UINT64
|
2010-06-03 21:24:43 +08:00
|
|
|
|
2015-08-16 06:26:35 +08:00
|
|
|
#if !defined HDF5_PARALLEL && !defined USE_PNETCDF
|
2013-07-09 05:31:13 +08:00
|
|
|
typedef int MPI_Comm;
|
|
|
|
|
typedef int MPI_Info;
|
2013-03-16 04:31:07 +08:00
|
|
|
#define MPI_COMM_WORLD 0
|
|
|
|
|
#define MPI_INFO_NULL 0
|
2015-08-16 06:26:35 +08:00
|
|
|
#endif
|
2013-03-16 04:31:07 +08:00
|
|
|
|
2010-06-03 21:24:43 +08:00
|
|
|
/* Define a struct to hold the MPI info so it can be passed down the
|
|
|
|
|
* call stack. This is used internally by the netCDF library. It
|
|
|
|
|
* should not be used by netcdf users. */
|
|
|
|
|
typedef struct NC_MPI_INFO {
|
|
|
|
|
MPI_Comm comm;
|
|
|
|
|
MPI_Info info;
|
|
|
|
|
} NC_MPI_INFO;
|
|
|
|
|
|
2011-09-19 04:57:51 +08:00
|
|
|
/* Define known dispatch tables and initializers */
|
|
|
|
|
|
|
|
|
|
extern int NCDISPATCH_initialize(void);
|
2015-08-16 06:26:35 +08:00
|
|
|
extern int NCDISPATCH_finalize(void);
|
|
|
|
|
|
2019-03-31 04:06:20 +08:00
|
|
|
extern const NC_Dispatch* NC3_dispatch_table;
|
2017-03-09 08:01:10 +08:00
|
|
|
extern int NC3_initialize(void);
|
|
|
|
|
extern int NC3_finalize(void);
|
2011-09-19 04:57:51 +08:00
|
|
|
|
2017-03-09 08:01:10 +08:00
|
|
|
#ifdef ENABLE_DAP
|
2019-03-31 04:06:20 +08:00
|
|
|
extern const NC_Dispatch* NCD2_dispatch_table;
|
2014-03-25 04:02:52 +08:00
|
|
|
extern int NCD2_initialize(void);
|
2015-08-16 06:26:35 +08:00
|
|
|
extern int NCD2_finalize(void);
|
2010-06-03 21:24:43 +08:00
|
|
|
#endif
|
2017-03-09 08:01:10 +08:00
|
|
|
#ifdef ENABLE_DAP4
|
2019-03-31 04:06:20 +08:00
|
|
|
extern const NC_Dispatch* NCD4_dispatch_table;
|
2017-03-09 08:01:10 +08:00
|
|
|
extern int NCD4_initialize(void);
|
|
|
|
|
extern int NCD4_finalize(void);
|
|
|
|
|
#endif
|
2010-06-03 21:24:43 +08:00
|
|
|
|
2013-03-16 04:31:07 +08:00
|
|
|
#ifdef USE_PNETCDF
|
2019-03-31 04:06:20 +08:00
|
|
|
extern const NC_Dispatch* NCP_dispatch_table;
|
2015-08-15 10:38:30 +08:00
|
|
|
extern int NCP_initialize(void);
|
2015-08-16 06:26:35 +08:00
|
|
|
extern int NCP_finalize(void);
|
2013-03-16 04:31:07 +08:00
|
|
|
#endif
|
|
|
|
|
|
2011-09-21 01:30:02 +08:00
|
|
|
#ifdef USE_NETCDF4
|
2011-09-21 04:39:04 +08:00
|
|
|
extern int NC4_initialize(void);
|
2015-08-16 06:26:35 +08:00
|
|
|
extern int NC4_finalize(void);
|
2015-08-17 11:44:18 +08:00
|
|
|
#endif
|
2011-09-21 04:39:04 +08:00
|
|
|
|
2018-11-26 20:44:59 +08:00
|
|
|
#ifdef USE_HDF5
|
2019-03-31 04:06:20 +08:00
|
|
|
extern const NC_Dispatch* HDF5_dispatch_table;
|
2018-11-26 20:44:59 +08:00
|
|
|
extern int NC_HDF5_initialize(void);
|
|
|
|
|
extern int NC_HDF5_finalize(void);
|
|
|
|
|
#endif
|
|
|
|
|
|
2018-02-08 21:20:58 +08:00
|
|
|
#ifdef USE_HDF4
|
2019-03-31 04:06:20 +08:00
|
|
|
extern const NC_Dispatch* HDF4_dispatch_table;
|
2018-02-08 21:20:58 +08:00
|
|
|
extern int HDF4_initialize(void);
|
|
|
|
|
extern int HDF4_finalize(void);
|
|
|
|
|
#endif
|
|
|
|
|
|
This PR adds EXPERIMENTAL support for accessing data in the
cloud using a variant of the Zarr protocol and storage
format. This enhancement is generically referred to as "NCZarr".
The data model supported by NCZarr is netcdf-4 minus the user-defined
types and the String type. In this sense it is similar to the CDF-5
data model.
More detailed information about enabling and using NCZarr is
described in the document NUG/nczarr.md and in a
[Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in).
WARNING: this code has had limited testing, so do use this version
for production work. Also, performance improvements are ongoing.
Note especially the following platform matrix of successful tests:
Platform | Build System | S3 support
------------------------------------
Linux+gcc | Automake | yes
Linux+gcc | CMake | yes
Visual Studio | CMake | no
Additionally, and as a consequence of the addition of NCZarr,
major changes have been made to the Filter API. NOTE: NCZarr
does not yet support filters, but these changes are enablers for
that support in the future. Note that it is possible
(probable?) that there will be some accidental reversions if the
changes here did not correctly mimic the existing filter testing.
In any case, previously filter ids and parameters were of type
unsigned int. In order to support the more general zarr filter
model, this was all converted to char*. The old HDF5-specific,
unsigned int operations are still supported but they are
wrappers around the new, char* based nc_filterx_XXX functions.
This entailed at least the following changes:
1. Added the files libdispatch/dfilterx.c and include/ncfilter.h
2. Some filterx utilities have been moved to libdispatch/daux.c
3. A new entry, "filter_actions" was added to the NCDispatch table
and the version bumped.
4. An overly complex set of structs was created to support funnelling
all of the filterx operations thru a single dispatch
"filter_actions" entry.
5. Move common code to from libhdf5 to libsrc4 so that it is accessible
to nczarr.
Changes directly related to Zarr:
1. Modified CMakeList.txt and configure.ac to support both C and C++
-- this is in support of S3 support via the awd-sdk libraries.
2. Define a size64_t type to support nczarr.
3. More reworking of libdispatch/dinfermodel.c to
support zarr and to regularize the structure of the fragments
section of a URL.
Changes not directly related to Zarr:
1. Make client-side filter registration be conditional, with default off.
2. Hack include/nc4internal.h to make some flags added by Ed be unique:
e.g. NC_CREAT, NC_INDEF, etc.
3. cleanup include/nchttp.h and libdispatch/dhttp.c.
4. Misc. changes to support compiling under Visual Studio including:
* Better testing under windows for dirent.h and opendir and closedir.
5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags
and to centralize error reporting.
6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them.
7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible.
Changes Left TO-DO:
1. fix provenance code, it is too HDF5 specific.
2020-06-29 08:02:47 +08:00
|
|
|
#ifdef ENABLE_NCZARR
|
|
|
|
|
extern const NC_Dispatch* NCZ_dispatch_table;
|
|
|
|
|
extern int NCZ_initialize(void);
|
|
|
|
|
extern int NCZ_finalize(void);
|
|
|
|
|
#endif
|
|
|
|
|
|
2019-03-31 04:06:20 +08:00
|
|
|
/* User-defined formats.*/
|
Provide byte-range reading of remote datasets
re: issue https://github.com/Unidata/netcdf-c/issues/1251
Assume that you have the URL to a remote dataset
which is a normal netcdf-3 or netcdf-4 file.
This PR allows the netcdf-c to read that dataset's
contents as a netcdf file using HTTP byte ranges
if the remote server supports byte-range access.
Originally, this PR was set up to access Amazon S3 objects,
but it can also access other remote datasets such as those
provided by a Thredds server via the HTTPServer access protocol.
It may also work for other kinds of servers.
Note that this is not intended as a true production
capability because, as is known, this kind of access to
can be quite slow. In addition, the byte-range IO drivers
do not currently do any sort of optimization or caching.
An additional goal here is to gain some experience with
the Amazon S3 REST protocol.
This architecture and its use documented in
the file docs/byterange.dox.
There are currently two test cases:
1. nc_test/tst_s3raw.c - this does a simple open, check format, close cycle
for a remote netcdf-3 file and a remote netcdf-4 file.
2. nc_test/test_s3raw.sh - this uses ncdump to investigate some remote
datasets.
This PR also incorporates significantly changed model inference code
(see the superceded PR https://github.com/Unidata/netcdf-c/pull/1259).
1. It centralizes the code that infers the dispatcher.
2. It adds support for byte-range URLs
Other changes:
1. NC_HDF5_finalize was not being properly called by nc_finalize().
2. Fix minor bug in ncgen3.l
3. fix memory leak in nc4info.c
4. add code to walk the .daprc triples and to replace protocol=
fragment tag with a more general mode= tag.
Final Note:
Th inference code is still way too complicated. We need to move
to the validfile() model used by netcdf Java, where each
dispatcher is asked if it can process the file. This decentralizes
the inference code. This will be done after all the major new
dispatchers (PIO, Zarr, etc) have been implemented.
2019-01-02 09:27:36 +08:00
|
|
|
extern NC_Dispatch* UDF0_dispatch_table;
|
|
|
|
|
extern char UDF0_magic_number[NC_MAX_MAGIC_NUMBER_LEN + 1];
|
|
|
|
|
extern NC_Dispatch* UDF1_dispatch_table;
|
|
|
|
|
extern char UDF1_magic_number[NC_MAX_MAGIC_NUMBER_LEN + 1];
|
|
|
|
|
|
2018-08-14 22:56:58 +08:00
|
|
|
/* Prototypes. */
|
|
|
|
|
int NC_check_nulls(int ncid, int varid, const size_t *start, size_t **count,
|
|
|
|
|
ptrdiff_t **stride);
|
|
|
|
|
|
2010-06-03 21:24:43 +08:00
|
|
|
/**************************************************/
|
|
|
|
|
/* Forward */
|
|
|
|
|
#ifndef USE_NETCDF4
|
|
|
|
|
/* Taken from libsrc4/netcdf.h */
|
|
|
|
|
struct nc_vlen_t;
|
|
|
|
|
#define NC_NETCDF4 0x1000
|
|
|
|
|
#define NC_CLASSIC_MODEL 0x0100
|
|
|
|
|
#define NC_ENOPAR (-114)
|
2012-08-02 01:18:58 +08:00
|
|
|
#endif /*!USE_NETCDF4*/
|
2010-06-03 21:24:43 +08:00
|
|
|
|
|
|
|
|
struct NC;
|
|
|
|
|
|
|
|
|
|
int NC_create(const char *path, int cmode,
|
2015-08-16 06:26:35 +08:00
|
|
|
size_t initialsz, int basepe, size_t *chunksizehintp,
|
2018-09-23 09:22:34 +08:00
|
|
|
int useparallel, void *parameters, int *ncidp);
|
2010-06-03 21:24:43 +08:00
|
|
|
int NC_open(const char *path, int cmode,
|
|
|
|
|
int basepe, size_t *chunksizehintp,
|
2018-09-23 09:22:34 +08:00
|
|
|
int useparallel, void *parameters, int *ncidp);
|
2010-06-03 21:24:43 +08:00
|
|
|
|
2011-02-24 01:37:11 +08:00
|
|
|
/* Expose the default vars and varm dispatch entries */
|
2018-05-30 16:50:09 +08:00
|
|
|
EXTERNL int NCDEFAULT_get_vars(int, int, const size_t*,
|
2011-02-24 01:37:11 +08:00
|
|
|
const size_t*, const ptrdiff_t*, void*, nc_type);
|
2018-05-30 16:50:09 +08:00
|
|
|
EXTERNL int NCDEFAULT_put_vars(int, int, const size_t*,
|
2011-02-24 01:37:11 +08:00
|
|
|
const size_t*, const ptrdiff_t*, const void*, nc_type);
|
2018-05-30 16:50:09 +08:00
|
|
|
EXTERNL int NCDEFAULT_get_varm(int, int, const size_t*,
|
2011-02-24 01:37:11 +08:00
|
|
|
const size_t*, const ptrdiff_t*, const ptrdiff_t*,
|
|
|
|
|
void*, nc_type);
|
2018-05-30 16:50:09 +08:00
|
|
|
EXTERNL int NCDEFAULT_put_varm(int, int, const size_t*,
|
2011-02-24 01:37:11 +08:00
|
|
|
const size_t*, const ptrdiff_t*, const ptrdiff_t*,
|
|
|
|
|
const void*, nc_type);
|
|
|
|
|
|
2010-06-03 21:24:43 +08:00
|
|
|
/**************************************************/
|
|
|
|
|
/* Forward */
|
|
|
|
|
struct NCHDR;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
/* Following functions must be handled as non-dispatch */
|
|
|
|
|
#ifdef NONDISPATCH
|
2015-08-16 06:26:35 +08:00
|
|
|
void (*nc_advise)(const char*cdf_routine_name,interr,const char*fmt,...);
|
|
|
|
|
void (*nc_set_log_level)(int);
|
2010-06-03 21:24:43 +08:00
|
|
|
const char* (*nc_inq_libvers)(void);
|
|
|
|
|
const char* (*nc_strerror)(int);
|
2015-08-16 06:26:35 +08:00
|
|
|
int (*nc_delete)(const char*path);
|
|
|
|
|
int (*nc_delete_mp)(const char*path,intbasepe);
|
|
|
|
|
int (*nc_initialize)();
|
|
|
|
|
int (*nc_finalize)();
|
2010-06-03 21:24:43 +08:00
|
|
|
#endif /*NONDISPATCH*/
|
|
|
|
|
|
|
|
|
|
/* Define the common fields for NC and NC_FILE_INFO_T etc */
|
|
|
|
|
typedef struct NCcommon {
|
|
|
|
|
int ext_ncid; /* uid << 16 */
|
|
|
|
|
int int_ncid; /* unspecified other id */
|
2019-03-31 04:06:20 +08:00
|
|
|
const struct NC_Dispatch* dispatch;
|
2011-09-17 02:36:08 +08:00
|
|
|
void* dispatchdata; /* per-protocol instance data */
|
|
|
|
|
char* path; /* as specified at open or create */
|
2010-06-03 21:24:43 +08:00
|
|
|
} NCcommon;
|
|
|
|
|
|
2018-05-30 16:50:09 +08:00
|
|
|
EXTERNL size_t NC_atomictypelen(nc_type xtype);
|
|
|
|
|
EXTERNL char* NC_atomictypename(nc_type xtype);
|
2010-06-03 21:24:43 +08:00
|
|
|
|
2010-12-16 05:45:05 +08:00
|
|
|
/* Misc */
|
2011-05-13 01:51:32 +08:00
|
|
|
|
2014-03-11 02:09:36 +08:00
|
|
|
extern int NC_getshape(int ncid, int varid, int ndims, size_t* shape);
|
2011-07-15 06:24:02 +08:00
|
|
|
extern int NC_is_recvar(int ncid, int varid, size_t* nrecs);
|
2015-03-10 20:10:18 +08:00
|
|
|
extern int NC_inq_recvar(int ncid, int varid, int* nrecdims, int* is_recdim);
|
2010-12-16 05:45:05 +08:00
|
|
|
|
|
|
|
|
#define nullstring(s) (s==NULL?"(null)":s)
|
|
|
|
|
|
2016-04-07 09:51:40 +08:00
|
|
|
#undef TRACECALLS
|
|
|
|
|
#ifdef TRACECALLS
|
|
|
|
|
#include <stdio.h>
|
|
|
|
|
#define TRACE(fname) fprintf(stderr,"call: %s\n",#fname)
|
|
|
|
|
#else
|
|
|
|
|
#define TRACE(fname)
|
|
|
|
|
#endif
|
|
|
|
|
|
2019-03-31 04:06:20 +08:00
|
|
|
/* Vectors of ones and zeros */
|
2019-10-10 07:18:48 +08:00
|
|
|
extern size_t NC_coord_zero[NC_MAX_VAR_DIMS];
|
|
|
|
|
extern size_t NC_coord_one[NC_MAX_VAR_DIMS];
|
|
|
|
|
extern ptrdiff_t NC_stride_one[NC_MAX_VAR_DIMS];
|
2011-07-12 20:30:16 +08:00
|
|
|
|
2015-08-16 06:26:35 +08:00
|
|
|
extern int NC_initialized;
|
2015-11-06 03:57:49 +08:00
|
|
|
|
2016-04-07 09:51:40 +08:00
|
|
|
/**
|
|
|
|
|
Certain functions are in the dispatch table,
|
|
|
|
|
but not in the netcdf.h API. These need to
|
|
|
|
|
be exposed for use in delegation such as
|
|
|
|
|
in libdap2.
|
|
|
|
|
*/
|
2018-05-30 16:50:09 +08:00
|
|
|
EXTERNL int
|
2016-04-07 09:51:40 +08:00
|
|
|
NCDISPATCH_inq_var_all(int ncid, int varid, char *name, nc_type *xtypep,
|
2016-04-07 04:05:58 +08:00
|
|
|
int *ndimsp, int *dimidsp, int *nattsp,
|
|
|
|
|
int *shufflep, int *deflatep, int *deflate_levelp,
|
|
|
|
|
int *fletcher32p, int *contiguousp, size_t *chunksizesp,
|
|
|
|
|
int *no_fill, void *fill_valuep, int *endiannessp,
|
2017-04-28 03:01:59 +08:00
|
|
|
unsigned int* idp, size_t* nparamsp, unsigned int* paramsp
|
|
|
|
|
);
|
2018-05-30 16:50:09 +08:00
|
|
|
EXTERNL int
|
2016-04-07 09:51:40 +08:00
|
|
|
NCDISPATCH_get_att(int ncid, int varid, const char* name, void* value, nc_type t);
|
|
|
|
|
|
Provide byte-range reading of remote datasets
re: issue https://github.com/Unidata/netcdf-c/issues/1251
Assume that you have the URL to a remote dataset
which is a normal netcdf-3 or netcdf-4 file.
This PR allows the netcdf-c to read that dataset's
contents as a netcdf file using HTTP byte ranges
if the remote server supports byte-range access.
Originally, this PR was set up to access Amazon S3 objects,
but it can also access other remote datasets such as those
provided by a Thredds server via the HTTPServer access protocol.
It may also work for other kinds of servers.
Note that this is not intended as a true production
capability because, as is known, this kind of access to
can be quite slow. In addition, the byte-range IO drivers
do not currently do any sort of optimization or caching.
An additional goal here is to gain some experience with
the Amazon S3 REST protocol.
This architecture and its use documented in
the file docs/byterange.dox.
There are currently two test cases:
1. nc_test/tst_s3raw.c - this does a simple open, check format, close cycle
for a remote netcdf-3 file and a remote netcdf-4 file.
2. nc_test/test_s3raw.sh - this uses ncdump to investigate some remote
datasets.
This PR also incorporates significantly changed model inference code
(see the superceded PR https://github.com/Unidata/netcdf-c/pull/1259).
1. It centralizes the code that infers the dispatcher.
2. It adds support for byte-range URLs
Other changes:
1. NC_HDF5_finalize was not being properly called by nc_finalize().
2. Fix minor bug in ncgen3.l
3. fix memory leak in nc4info.c
4. add code to walk the .daprc triples and to replace protocol=
fragment tag with a more general mode= tag.
Final Note:
Th inference code is still way too complicated. We need to move
to the validfile() model used by netcdf Java, where each
dispatcher is asked if it can process the file. This decentralizes
the inference code. This will be done after all the major new
dispatchers (PIO, Zarr, etc) have been implemented.
2019-01-02 09:27:36 +08:00
|
|
|
#endif /* NC_DISPATCH_H */
|
