netcdf-c/libdispatch/dnotnc4.c
Dennis Heimbigner 59e04ae071 This PR adds EXPERIMENTAL support for accessing data in the
cloud using a variant of the Zarr protocol and storage
format. This enhancement is generically referred to as "NCZarr".

The data model supported by NCZarr is netcdf-4 minus the user-defined
types and the String type. In this sense it is similar to the CDF-5
data model.

More detailed information about enabling and using NCZarr is
described in the document NUG/nczarr.md and in a
[Unidata Developer's blog entry](https://www.unidata.ucar.edu/blogs/developer/en/entry/overview-of-zarr-support-in).

WARNING: this code has had limited testing, so do use this version
for production work. Also, performance improvements are ongoing.
Note especially the following platform matrix of successful tests:

Platform | Build System | S3 support
------------------------------------
Linux+gcc      | Automake     | yes
Linux+gcc      | CMake        | yes
Visual Studio  | CMake        | no

Additionally, and as a consequence of the addition of NCZarr,
major changes have been made to the Filter API. NOTE: NCZarr
does not yet support filters, but these changes are enablers for
that support in the future.  Note that it is possible
(probable?) that there will be some accidental reversions if the
changes here did not correctly mimic the existing filter testing.

In any case, previously filter ids and parameters were of type
unsigned int. In order to support the more general zarr filter
model, this was all converted to char*.  The old HDF5-specific,
unsigned int operations are still supported but they are
wrappers around the new, char* based nc_filterx_XXX functions.
This entailed at least the following changes:
1. Added the files libdispatch/dfilterx.c and include/ncfilter.h
2. Some filterx utilities have been moved to libdispatch/daux.c
3. A new entry, "filter_actions" was added to the NCDispatch table
   and the version bumped.
4. An overly complex set of structs was created to support funnelling
   all of the filterx operations thru a single dispatch
   "filter_actions" entry.
5. Move common code to from libhdf5 to libsrc4 so that it is accessible
   to nczarr.

Changes directly related to Zarr:
1. Modified CMakeList.txt and configure.ac to support both C and C++
   -- this is in support of S3 support via the awd-sdk libraries.
2. Define a size64_t type to support nczarr.
3. More reworking of libdispatch/dinfermodel.c to
   support zarr and to regularize the structure of the fragments
   section of a URL.

Changes not directly related to Zarr:
1. Make client-side filter registration be conditional, with default off.
2. Hack include/nc4internal.h to make some flags added by Ed be unique:
   e.g. NC_CREAT, NC_INDEF, etc.
3. cleanup include/nchttp.h and libdispatch/dhttp.c.
4. Misc. changes to support compiling under Visual Studio including:
   * Better testing under windows for dirent.h and opendir and closedir.
5. Misc. changes to the oc2 code to support various libcurl CURLOPT flags
   and to centralize error reporting.
6. By default, suppress the vlen tests that have unfixed memory leaks; add option to enable them.
7. Make part of the nc_test/test_byterange.sh test be contingent on remotetest.unidata.ucar.edu being accessible.

Changes Left TO-DO:
1. fix provenance code, it is too HDF5 specific.
2020-06-28 18:02:47 -06:00

670 lines
14 KiB
C

/* Copyright 2018, UCAR/Unidata See netcdf/COPYRIGHT file for copying
* and redistribution conditions.*/
/**
* @file
* @internal This file contains functions that
* can be used in dispatch tables to handle
* such things as unimplemented functions.
* As a rule, these functions will return NC_ENOTNC4.
* In some cases, the function may succeed and set
* appropriate output parameter values. These are indicated
* by the NC_NOOP prefix to the function name.
*
* @author Ed Hartnett, Dennis Heimbigner
*/
#include "netcdf.h"
#include "netcdf_filter.h"
#include "netcdf_aux.h"
#include "ncdispatch.h"
#include "nc4internal.h"
#include "ncfilter.h"
/**
* @internal Not implemented in some dispatch tables
*
* @param ncid Ignored.
* @param varid Ignored.
* @param id Ignored.
* @param nparams Ignored.
* @param parms Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table
* @author Ed Hartnett
*/
int
NC_NOTNC4_def_var_filter(int ncid, int varid, unsigned int id, size_t nparams,
const unsigned int* parms)
{
return NC_ENOTNC4;
}
/**
* @internal Not allowed for classic model.
*
* @param parent_ncid Ignored.
* @param name Ignored.
* @param new_ncid Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_def_grp(int parent_ncid, const char *name, int *new_ncid)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param grpid Ignored.
* @param name Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_rename_grp(int grpid, const char *name)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param size Ignored.
* @param name Ignored.
* @param typeidp Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_def_compound(int ncid, size_t size, const char *name, nc_type *typeidp)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param typeid1 Ignored.
* @param name Ignored.
* @param offset Ignored.
* @param field Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_insert_compound(int ncid, nc_type typeid1, const char *name, size_t offset,
nc_type field_typeid)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param typeid1 Ignored.
* @param name Ignored.
* @param offset Ignored.
* @param field Ignored.
* @param ndims Ignored.
* @param dim Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
extern int
NC_NOTNC4_insert_array_compound(int ncid, int typeid1, const char *name,
size_t offset, nc_type field_typeid,
int ndims, const int *dim_sizesp)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param typeid1 Ignored.
* @param fieldid Ignored.
* @param name Ignored.
* @param offsetp Ignored.
* @param field Ignored.
* @param ndimsp Ignored.
* @param dim Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_compound_field(int ncid, nc_type typeid1, int fieldid, char *name,
size_t *offsetp, nc_type *field_typeidp, int *ndimsp,
int *dim_sizesp)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param typeid1 Ignored.
* @param name Ignored.
* @param fieldidp Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_compound_fieldindex(int ncid, nc_type typeid1, const char *name, int *fieldidp)
{
return NC_ENOTNC4;
}
/* Opaque type. */
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param datum Ignored.
* @param name Ignored.
* @param typeidp Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_def_opaque(int ncid, size_t datum_size, const char *name,
nc_type *typeidp)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param name Ignored.
* @param base_typeid Ignored.
* @param typeidp Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_def_vlen(int ncid, const char *name, nc_type base_typeid,
nc_type *typeidp)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param base_typeid Ignored.
* @param name Ignored.
* @param typeidp Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_def_enum(int ncid, nc_type base_typeid, const char *name,
nc_type *typeidp)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param xtype Ignored.
* @param value Ignored.
* @param identifier Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_enum_ident(int ncid, nc_type xtype, long long value, char *identifier)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param typeid1 Ignored.
* @param idx Ignored.
* @param identifier Ignored.
* @param value Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_enum_member(int ncid, nc_type typeid1, int idx, char *identifier,
void *value)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param typeid1 Ignored.
* @param identifier Ignored.
* @param value Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_insert_enum(int ncid, nc_type typeid1, const char *identifier,
const void *value)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param typeid1 Ignored.
* @param vlen_element Ignored.
* @param len Ignored.
* @param data Ignored.
*
* @return ::NC_NOERR No error.
* @author Ed Hartnett
*/
int
NC_NOTNC4_put_vlen_element(int ncid, int typeid1, void *vlen_element,
size_t len, const void *data)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param typeid1 Ignored.
* @param vlen_element Ignored.
* @param len Ignored.
* @param data Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_get_vlen_element(int ncid, int typeid1, const void *vlen_element,
size_t *len, void *data)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param varid Ignored.
* @param size Ignored.
* @param nelems Ignored.
* @param preemption Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_set_var_chunk_cache(int ncid, int varid, size_t size, size_t nelems,
float preemption)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param varid Ignored.
* @param sizep Ignored.
* @param nelemsp Ignored.
* @param preemptionp Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_get_var_chunk_cache(int ncid, int varid, size_t *sizep,
size_t *nelemsp, float *preemptionp)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param varid Ignored.
* @param shuffle Ignored.
* @param deflate Ignored.
* @param deflate_level Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett, Dennis Heimbigner
*/
int
NC_NOTNC4_def_var_deflate(int ncid, int varid, int shuffle, int deflate,
int deflate_level)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param varid Ignored.
* @param fletcher32 Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett, Dennis Heimbigner
*/
int
NC_NOTNC4_def_var_fletcher32(int ncid, int varid, int fletcher32)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param varid Ignored.
* @param contiguous Ignored.
* @param chunksizesp Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett, Dennis Heimbigner
*/
int
NC_NOTNC4_def_var_chunking(int ncid, int varid, int contiguous, const size_t *chunksizesp)
{
return NC_EPERM;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param varid Ignored.
* @param endianness Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_def_var_endian(int ncid, int varid, int endianness)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param varid Ignored.
* @param par_access Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett, Dennis Heimbigner
*/
int
NC_NOTNC4_var_par_access(int ncid, int varid, int par_access)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param name Ignored.
* @param grp_ncid Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_ncid(int ncid, const char *name, int *grp_ncid)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param numgrps Ignored.
* @param ncids Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_grps(int ncid, int *numgrps, int *ncids)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param name Ignored.
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_grpname(int ncid, char *name)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param lenp Ignored.
* @param full_name Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_grpname_full(int ncid, size_t *lenp, char *full_name)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param parent_ncid Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_grp_parent(int ncid, int *parent_ncid)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param full_name Ignored.
* @param grp_ncid Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_grp_full_ncid(int ncid, const char *full_name, int *grp_ncid)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param nvars Ignored.
* @param varids Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_varids(int ncid, int *nvars, int *varids)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param ndims Ignored.
* @param dimids Ignored.
* @param include_parents Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_dimids(int ncid, int *ndims, int *dimids, int include_parents)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param ntypes Ignored.
* @param typeids Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_typeids(int ncid, int *ntypes, int *typeids)
{
return NC_ENOTNC4;
}
/**
* @internal Not implemented for a dispatch table.
*
* @param ncid Ignored.
* @param typeid1 Ignored.
* @param name Ignored.
* @param size Ignored.
* @param base_nc_typep Ignored.
* @param nfieldsp Ignored.
* @param classp Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_user_type(int ncid, nc_type typeid1, char *name, size_t *size,
nc_type *base_nc_typep, size_t *nfieldsp, int *classp)
{
return NC_ENOTNC4;
}
/**
* @internal Get the id of a type from the name.
*
* @param ncid Ignored.
* @param name Ignored.
* @param typeidp Ignored.
*
* @return ::NC_ENOTNC4 Not implemented for a dispatch table.
* @author Ed Hartnett
*/
int
NC_NOTNC4_inq_typeid(int ncid, const char *name, nc_type *typeidp)
{
/* Note that this should actually work for atomic types */
return NC_ENOTNC4;
}
/**
* @internal Carry out one of several filter actions
*
* @param ncid Containing group id
* @param varid Containing variable id
* @param action Action to perform
*
* @return ::NC_NOERR Implemented as a no-op.
* @return ::NC_ENOTNC4 Not implemented
* @return ::NC_ENOFILTER No filter defined
* @author D. Heimbigner
*/
int
NC_NOTNC4_filter_actions(int ncid, int varid, int action, void* args)
{
return NC_ENOTNC4;
}
/**
* @internal Carry out one of several filter actions
*
* @param ncid Containing group id
* @param varid Containing variable id
* @param action Action to perform
*
* @return ::NC_NOERR Implemented as a no-op.
* @return ::NC_ENOTNC4 Not implemented
* @return ::NC_ENOFILTER No filter defined
* @author D. Heimbigner
*/
int
NC_NOOP_filter_actions(int ncid, int varid, int action, void* args)
{
NC_FILTERX_OBJ* obj = (NC_FILTERX_OBJ*)args;
switch (action) {
case NCFILTER_FILTERIDS:
obj->u.ids.nfilters = 0;
return NC_NOERR;
case NCFILTER_INFO:
return NC_ENOFILTER;
default:
return NC_EFILTER;
}
}