mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2024-11-27 07:30:33 +08:00
Code Issues Packages Projects Releases Wiki Activity

Revert/Improve nc_create + NC_DISKLESS behavior

Browse Source 
re: https://github.com/Unidata/netcdf-c/issues/1154

Inadvertently, the behavior of NC_DISKLESS with nc_create() was
changed in release 4.6.1. Previously, the NC_WRITE flag needed
to be explicitly used with NC_DISKLESS in order to cause the
created file to be persisted to disk.

Additional analyis indicated that the current NC_DISKLESS
implementation was seriously flawed.

This PR attempts to clean up and regularize the situation with
respect to NC_DISKLESS control. One important aspect of diskless
operation is that there are two different notions of write.

1. The file is read-write vs read-only when using the netcdf API.
2. The file is persisted or not to disk at nc_close().

Previously, these two were conflated. The rules now are
as follows.

1. NC_DISKLESS + NC_WRITE means that the file is read/write using the netcdf API
2. NC_DISKLESS + NC_PERSIST means that the file is persisted to a disk file at nc_close.
3. NC_DISKLESS + NC_PERSIST + NC_WRITE means both 1 and 2.

The NC_PERSIST flag is new and takes over the obsolete NC_MPIPOSIX flag.
NC_MPIPOSIX is still defined, but is now an alias for the NC_MPIIO flag.

It is also now the case that for netcdf-4, NC_DISKLESS is independent
of NC_INMEMORY and in fact it is an error to specify both flags
simultaneously.

Finally, the MMAP code was fixed to use NC_PERSIST as well.
Also marked MMAP as deprecated.

Also added a test case to test various combinations of NC_DISKLESS,
NC_PERSIST, and NC_WRITE.

This PR affects a number of files and especially test cases
that used NC_DISKLESS.

Misc. Unrelated fixes
1. fixed some warnings in ncdump/dumplib.c
This commit is contained in:
Dennis Heimbigner 2018-10-10 13:32:17 -06:00
parent 4a5b15bcbb
commit 4636584d5b
28 changed files with 614 additions and 240 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
CMakeLists.txt
View File

@ -1402,16 +1402,21 @@ CHECK_FUNCTION_EXISTS(getpagesize HAVE_GETPAGESIZE)
CHECK_FUNCTION_EXISTS(sysconf HAVE_SYSCONF)
CHECK_FUNCTION_EXISTS(getrlimit HAVE_GETRLIMIT)
CHECK_FUNCTION_EXISTS(_filelengthi64 HAVE_FILE_LENGTH_I64)
CHECK_FUNCTION_EXISTS(mmmap HAVE_MMAP)
CHECK_FUNCTION_EXISTS(mremap HAVE_MREMAP)
IF(NOT HAVE_MMMAP)
MESSAGE(WARNING "mmap not found: disabling MMAP support.")
SET(ENABLE_MMAP OFF)
ENDIF()
IF(NOT HAVE_MREMAP)
MESSAGE(WARNING "mremap not found: disabling MMAP support.")
SET(ENABLE_MMAP OFF)
ENDIF()
IF(ENABLE_MMAP)
IF(NOT HAVE_MREMAP)
MESSAGE(WARNING "mremap not found: disabling MMAP support.")
SET(ENABLE_MMAP OFF)
ELSE(NOT HAVE_MREMAP)
SET(BUILD_MMAP ON)
SET(USE_MMAP ON)
ENDIF(NOT HAVE_MREMAP)
SET(BUILD_MMAP ON)
SET(USE_MMAP ON)
ENDIF(ENABLE_MMAP)
#CHECK_FUNCTION_EXISTS(alloca HAVE_ALLOCA)

6
RELEASE_NOTES.md
View File

@ -7,6 +7,12 @@ This file contains a high-level description of this package's evolution. Release
## 4.6.2 - TBD
* [Bug Fix] The use of NC_DISKLESS has been modified to make it cleaner. This
adds a new flag called NC_PERSIST that takes over the now obsolete NC_MPIPOSIX.
* [Obsolete] Obsolete the MPIPOSIX flag.
* [Bug Fix] When using filters with HDF5 1.10.x or later, it is necessary
to utilize the HDF5 replacements for malloc, realloc, and free in the filter
code.
* [Enhancement] Create a new version of _NCProperties provenance attribute. This version (version 2) supports arbitrary key-value pairs. It is the default when new files are created. Version 1 continues to be accepted.
* [Enhancement] Allow user to set http read buffersize for DAP2 and DAP4 using the tag HTTP.READ.BUFFERSIZE in the .daprc file.
* [Enhancement] Allow user to set http keepalive for DAP2 and DAP4 using the tag HTTP.KEEPALIVE in the .daprc file (see the OPeNDAP documentation for details).

5
cf
View File

@ -108,11 +108,10 @@ FLAGS="$FLAGS --enable-extreme-numbers"
#FLAGS="$FLAGS --disable-testsets"
#FLAGS="$FLAGS --disable-dap-remote-tests"
#FLAGS="$FLAGS --enable-dap-auth-tests" -- requires a new remotetest server
#FLAGS="$FLAGS --enable-doxygen"
#FLAGS="$FLAGS --enable-internal-docs"
#FLAGS="$FLAGS --enable-doxygen --enable-internal-docs"
FLAGS="$FLAGS --enable-logging"
#FLAGS="$FLAGS --disable-diskless"
#FLAGS="$FLAGS --enable-mmap"
FLAGS="$FLAGS --enable-mmap"
#FLAGS="$FLAGS --with-udunits"
#FLAGS="$FLAGS --with-libcf"
#FLAGS="$FLAGS --enable-jna"

11
configure.ac
View File

@ -887,10 +887,15 @@ test "x$enable_mmap" = xyes || enable_mmap=no
AC_MSG_RESULT($enable_mmap)
# check for mmap and mremap availability before committing to use mmap
AC_CHECK_FUNCS([mremap])
AC_CHECK_FUNCS([mmap],[have_mmap=yes],[have_mmap=no])
AC_CHECK_FUNCS([mremap],[have_mremap=yes],[have_mremap=no])
if test "x$ac_cv_func_mmap_fixed_mapped" != xyes -o "x$ac_cv_func_mremap" != xyes ; then
echo "mmap function or mremap function is not available: disabling mmap"
if test "x$have_mmap" != xyes ; then
echo "mmap function is not available: disabling mmap"
enable_mmap=no
fi
if test "x$have_mremap" != xyes ; then
echo "mremap function is not available: disabling mmap"
enable_mmap=no
fi

58
docs/inmemory.md
View File

@ -24,15 +24,19 @@ write it back out to disk when nc_close() is called.
of memory as if it were a netcdf file. At close, it is possible to ask
for the final contents of the memory chunk. Be warned that there is
some complexity to this as described below.
4. MMAP -- Tell the netcdf-c library to use the *mmap()* operating
4. MMAP -- (deprecated) Tell the netcdf-c library to use the *mmap()* operating
system functionality to access a file.
The first two capabilities are intertwined in the sense that the *diskless*
capability makes use internally of the *inmemory* capability. But, the
*inmemory* capability can be used independently of the *diskless* capability.
The first two capabilities are intertwined in the sense that the
*diskless* capability makes use internally of the *inmemory*
capability (for netcdf classic only). But, the *inmemory*
capability can be used independently of the *diskless*
capability.
The *mmap()* capability provides a capability similar to *diskless* but
using special capabilities of the underlying operating system.
using special capabilities of the underlying operating system. It turns out
that the mmap capability has seen no significant use, so its use is deprecated
and will be removed at some point in the future.
Note also that *diskless* and *inmemory* can be used for both
*netcdf-3* (classic) and *netcdf-4* (enhanced) data. The *mmap*
@ -47,22 +51,40 @@ Note that since the file is stored in memory, size limitations apply.
If you are on using a 32-bit pointer then the file size must be less than 2^32
bytes in length. On a 64-bit machine, the size must be less than 2^64 bytes.
Also note that for a diskless file, there are two notions of
*write* with respect to the file. The first notion is that the
file is read-only through the netCDF API. For example, if the file
is read-only, then a call to, for example, _nc_def_dim()_ will fail.
The second notion of *write* refers to the file on disk to which
the contents of memory might be persisted.
WARNING: control of the two kinds of *write* has changed since
release 4.6.1.
The mode flag NC_WRITE determines the first kind of *write*.
If set, then NC_WRITE means that the file can be modified through
the netCDF API, otherwise it is read-only. This is a change since
release 4.6.1.
The new mode flag NC_PERSIST now determines the second kind of
*write*. If set, then NC_PERSIST means that the memory contents
will be persisted to disk, possibly overwriting the previous
file contents. Otherwise, the default is to throw away the
in-memory contents.
### Diskless File Open
Calling *nc_open()* using the mode flag *NC_DISKLESS* will cause
the file being opened to be read into memory. When calling *nc_close()*,
the file will optionally be re-written (aka "persisted") to disk. This
persist capability will be invoked if and only if *NC_WRITE* is specified
persist capability will be invoked if and only if *NC_PERSIST* is specified
in the mode flags at the call to *nc_open()*.
### Diskless File Create
Calling *nc_create()* using the mode flag *NC_DISKLESS* will cause
the file to initially be created and kept in memory.
When calling *nc_close()*, the file will be written
to disk.
Note that if it is desired to create the file in memory,
but not write to a disk file, then one can either set
the NC_NOCLOBBER mode flag or one can call *nc_abort()*
instead of *nc_close()*.
to disk if and only if *NC_PERSIST* is specified
in the mode flags at the call to *nc_create()*.
Enabling Inmemory File Access {#Enable_Inmemory}
--------------
@ -180,9 +202,11 @@ In this way, it is possible to avoid memory reallocation while still
allowing modifications to the file. You will still need to call
*nc_close_memio()* to obtain the size of the final, modified, file.
Enabling MMAP File Access {#Enable_MMAP}
Enabling MMAP File Access (Deprecated) {#Enable_MMAP}
--------------
The MMAP functionality is deprecated.
Some operating systems provide a capability called MMAP.
This allows disk files to automatically be mapped to chunks of memory.
It operates in a fashion somewhat similar to operating system virtual
@ -202,6 +226,16 @@ Known Bugs {#Inmemory_Bugs}
you overrun the available space, then the HDF5 library will
fail with a segmentation fault.
2. You will get an HDF5 error under the following conditions.
1. You call nc_open on a file with the flags NC_DISKLESS|NC_WRITE
but without NC_PERSIST.
2. The file to be read is read-only (i.e. mode 0444).
Note that this should be ok because the modifications to the file
are not intended to pushed back into the disk file. However, the
HDF5 core driver does not allow this.
References {#Inmemory_References}
--------------

1
include/ncrc.h
View File

@ -55,6 +55,7 @@ extern char* NC_backslashEscape(const char* s);
extern char* NC_backslashUnescape(const char* esc);
extern char* NC_entityescape(const char* s);
extern int NC_readfile(const char* filename, NCbytes* content);
extern int NC_writefile(const char* filename, size_t size, void* content);
extern char* NC_mktmp(const char* base);
#endif /*NCRC_H*/

18
include/netcdf.h
View File

@ -116,22 +116,24 @@ extern "C" {
/* Define the ioflags bits for nc_create and nc_open.
currently unused:
0x0002
0x0040
0x0080
and the whole upper 16 bits
*/
#define NC_NOWRITE 0x0000 /**< Set read-only access for nc_open(). */
#define NC_WRITE 0x0001 /**< Set read-write access for nc_open(). */
#define NC_CLOBBER 0x0000 /**< Destroy existing file. Mode flag for nc_create(). */
#define NC_NOCLOBBER 0x0004 /**< Don't destroy existing file. Mode flag for nc_create(). */
#define NC_DISKLESS 0x0008 /**< Use diskless file. Mode flag for nc_open() or nc_create(). */
#define NC_MMAP 0x0010 /**< Use diskless file with mmap. Mode flag for nc_open() or nc_create(). */
#define NC_MMAP 0x0010 /**< \deprecated Use diskless file with mmap. Mode flag for nc_open() or nc_create()*/
#define NC_64BIT_DATA 0x0020 /**< CDF-5 format: classic model but 64 bit dimensions and sizes */
#define NC_CDF5 NC_64BIT_DATA /**< Alias NC_CDF5 to NC_64BIT_DATA */
#define NC_UDF0 0x0040 /**< User-defined format 0. */
#define NC_UDF1 0x0080 /**< User-defined format 1. */
#define NC_CLASSIC_MODEL 0x0100 /**< Enforce classic model on netCDF-4. Mode flag for nc_create(). */
#define NC_64BIT_OFFSET 0x0200 /**< Use large (64-bit) file offsets. Mode flag for nc_create(). */
@ -152,16 +154,14 @@ Use this in mode flags for both nc_create() and nc_open(). */
#define NC_MPIIO 0x2000 /**< \deprecated */
/** Turn on MPI POSIX I/O.
Use this in mode flags for both nc_create() and nc_open(). */
#define NC_MPIPOSIX 0x4000 /**< \deprecated As of libhdf5 1.8.13. */
#define NC_MPIPOSIX NC_MPIIO /**< \deprecated As of libhdf5 1.8.13. Now an alias */
#define NC_INMEMORY 0x8000 /**< Read from memory. Mode flag for nc_open() or nc_create() => NC_DISKLESS */
#define NC_PERSIST 0x4000 /**< Save diskless contents to disk. Mode flag for nc_open() or nc_create() */
#define NC_INMEMORY 0x8000 /**< Read from memory. Mode flag for nc_open() or nc_create() */
#define NC_PNETCDF (NC_MPIIO) /**< \deprecated Use PnetCDF library; alias for NC_MPIIO. */
#define NC_UDF0 0x0080 /**< User-defined format 0. */
#define NC_UDF1 0x0002 /**< User-defined format 1. */
#define NC_MAX_MAGIC_NUMBER_LEN 8 /**< Max len of ser-defined format magic number. */
#define NC_MAX_MAGIC_NUMBER_LEN 8 /**< Max len of user-defined format magic number. */
/** Format specifier for nc_set_default_format() and returned
* by nc_inq_format. This returns the format as provided by

114
libdispatch/dfile.c
View File

@ -302,8 +302,13 @@ NC_check_file_type(const char *path, int flags, int use_parallel,
*model = 0;
*version = 0;
assert(inmemory ? !mmap : 1); /* inmemory => !mmap */
assert((diskless && inmemory) ? !mmap : 1);/*diskless & inmemory => !mmap*/
/* NC_INMEMORY and NC_DISKLESS and NC_MMAP are all mutually exclusive */
if(diskless && inmemory) {status = NC_EDISKLESS; goto done;}
if(diskless && mmap) {status = NC_EDISKLESS; goto done;}
if(inmemory && mmap) {status = NC_EINMEMORY; goto done;}
/* mmap is not allowed for netcdf-4 */
if(mmap && (flags & NC_NETCDF4)) {status = NC_EINVAL; goto done;}
memset((void*)&file,0,sizeof(file));
file.path = path; /* do not free */
@ -372,7 +377,8 @@ and attributes.
NC_NETCDF4 (create netCDF-4/HDF5 file),
NC_CLASSIC_MODEL (enforce netCDF classic mode on netCDF-4/HDF5 files),
NC_DISKLESS (store data in memory), and
NC_MMAP (use MMAP for NC_DISKLESS instead of NC_INMEMORY).
NC_PERSIST (force the NC_DISKLESS data from memory to a file),
NC_MMAP (use MMAP for NC_DISKLESS instead of NC_INMEMORY -- deprecated).
See discussion below.
\param ncidp Pointer to location where returned netCDF ID is to be
@ -385,10 +391,7 @@ aspects of how it may be used.
Setting NC_NOCLOBBER means you do not want to clobber (overwrite) an
existing dataset; an error (NC_EEXIST) is returned if the specified
dataset already exists. As a slight variation on this, if you
specify NC_DISKLESS and NC_NOCLOBBER, the file will be created
in-memory, but no attempt will be made to persiste the in-memory
data to a disk file.
dataset already exists.
The NC_SHARE flag is appropriate when one process may be writing the
dataset and one or more other processes reading the dataset
@ -424,33 +427,23 @@ types, multiple unlimited dimensions, or new atomic types. The
advantage of this restriction is that such files are guaranteed to
work with existing netCDF software.
Setting NC_DISKLESS causes netCDF to create the file only in memory.
This allows for the use of files that have no long term purpose. Note that
with one exception, the in-memory file is destroyed upon calling
nc_close. If, however, the flag combination (NC_DISKLESS|NC_WRITE)
is used, then at close, the contents of the memory file will be
made persistent in the file path that was specified in the nc_create
call. If NC_DISKLESS is going to be used for creating a large classic file,
it behooves one to use either nc__create or nc_create_mp and specify
an appropriately large value of the initialsz parameter to avoid
to many extensions to the in-memory space for the file.
This flag applies to files in classic format and to file in extended
Setting NC_DISKLESS causes netCDF to create the file only in
memory and to optionally write the final contents to the
correspondingly named disk file. This allows for the use of
files that have no long term purpose. Operating on an existing file
in memory may also be faster. The decision on whether
or not to "persist" the memory contents to a disk file is
described in detail in the file docs/inmemory.md, which is
definitive. By default, closing a diskless fill will cause it's
contents to be lost.
If NC_DISKLESS is going to be used for creating a large classic
file, it behooves one to use nc__create and specify an
appropriately large value of the initialsz parameter to avoid to
many extensions to the in-memory space for the file. This flag
applies to files in classic format and to file in extended
format (netcdf-4).
Normally, NC_DISKLESS allocates space in the heap for
storing the in-memory file. If, however, the ./configure
flags --enable-mmap is used, and the additional mode flag
NC_MMAP is specified, then the file will be created using
the operating system MMAP facility.
This flag only applies to files in classic format. Extended
format (netcdf-4) files will ignore the NC_MMAP flag.
Using NC_MMAP for nc_create is
only included for completeness vis-a-vis nc_open. The
ability to use MMAP is of limited use for nc_create because
nc_create is going to create the file in memory anyway.
Closing a MMAP'd file will be slightly faster, but not significantly.
Note that nc_create(path,cmode,ncidp) is equivalent to the invocation of
nc__create(path,cmode,NC_SIZEHINT_DEFAULT,NULL,ncidp).
@ -527,7 +520,7 @@ the classic netCDF-3 data model.
if (status != NC_NOERR) handle_error(status);
@endcode
In this example we create a in-memory netCDF classic dataset named
In this example we create an in-memory netCDF classic dataset named
diskless.nc whose content will be lost when nc_close() is called.
@code
@ -550,7 +543,7 @@ in a file named diskless.nc when nc_close() is called.
int status = NC_NOERR;
int ncid;
...
status = nc_create("diskless.nc", NC_DISKLESS|NC_WRITE, &ncid);
status = nc_create("diskless.nc", NC_DISKLESS|NC_PERSIST, &ncid);
if (status != NC_NOERR) handle_error(status);
@endcode
@ -679,7 +672,7 @@ int
nc_create_mem(const char* path, int mode, size_t initialsize, int* ncidp)
{
if(mode & NC_MMAP) return NC_EINVAL;
mode |= (NC_INMEMORY|NC_NOCLOBBER); /* Specifically, do not set NC_DISKLESS */
mode |= NC_INMEMORY; /* Specifically, do not set NC_DISKLESS */
return NC_create(path, mode, initialsize, 0, NULL, 0, NULL, ncidp);
}
@ -754,7 +747,7 @@ nc__create_mp(const char *path, int cmode, size_t initialsz,
*
* If NC_DISKLESS is specified, then the whole file is read completely
* into memory. In effect this creates an in-memory cache of the file.
* If the omode flag also specifies NC_WRITE, then the in-memory cache
* If the omode flag also specifies NC_PERSIST, then the in-memory cache
* will be re-written to the disk file when nc_close() is called. For
* some kinds of manipulations, having the in-memory cache can speed
* up file processing. But in simple cases, non-cached processing may
@ -945,7 +938,7 @@ nc_open_mem(const char* path, int omode, size_t size, void* memory, int* ncidp)
return NC_EINVAL;
if(omode & (NC_WRITE|NC_MMAP))
return NC_EINVAL;
omode |= (NC_INMEMORY); /* DO not set NC_DISKLESS */
omode |= (NC_INMEMORY); /* Note: NC_INMEMORY and NC_DISKLESS are mutually exclusive*/
meminfo.size = size;
meminfo.memory = memory;
meminfo.flags = NC_MEMIO_LOCKED;
@ -1951,6 +1944,7 @@ check_create_mode(int mode)
int mode_format;
int mmap = 0;
int inmemory = 0;
int diskless = 0;
/* This is a clever check to see if more than one format bit is
* set. */
@ -1961,8 +1955,19 @@ check_create_mode(int mode)
mmap = ((mode & NC_MMAP) == NC_MMAP);
inmemory = ((mode & NC_INMEMORY) == NC_INMEMORY);
if(mmap && inmemory) /* cannot have both */
return NC_EINMEMORY;
diskless = ((mode & NC_DISKLESS) == NC_DISKLESS);
/* NC_INMEMORY and NC_DISKLESS and NC_MMAP are all mutually exclusive */
if(diskless && inmemory) return NC_EDISKLESS;
if(diskless && mmap) return NC_EDISKLESS;
if(inmemory && mmap) return NC_EINMEMORY;
/* mmap is not allowed for netcdf-4 */
if(mmap && (mode & NC_NETCDF4)) return NC_EINVAL;
/* Can't use both parallel and diskless|inmemory|mmap. */
if (mode & NC_MPIIO && mode & (NC_DISKLESS|NC_INMEMORY|NC_MMAP))
return NC_EINVAL;
#ifndef USE_NETCDF4
/* If the user asks for a netCDF-4 file, and the library was built
@ -2014,23 +2019,11 @@ NC_create(const char *path0, int cmode, size_t initialsz,
int model = NC_FORMATX_UNDEFINED; /* one of the NC_FORMATX values */
int isurl = 0; /* dap or cdmremote or neither */
char* path = NULL;
int mmap = 0;
int diskless = 0;
TRACE(nc_create);
if(path0 == NULL)
return NC_EINVAL;
/* Fix the inmemory related flags */
mmap = ((cmode & NC_MMAP) == NC_MMAP);
diskless = ((cmode & NC_DISKLESS) == NC_DISKLESS);
/* diskless && !mmap => inmemory */
if(diskless && !mmap) cmode |= NC_INMEMORY;
/* Can't use both parallel and diskless|inmemory. */
if (useparallel && (cmode & (NC_DISKLESS|NC_INMEMORY)))
return NC_EINVAL;
/* Check mode flag for sanity. */
if ((stat = check_create_mode(cmode)))
return stat;
@ -2209,7 +2202,6 @@ NC_open(const char *path0, int omode, int basepe, size_t *chunksizehintp,
int model = 0;
int isurl = 0;
int version = 0;
int flags = 0;
char* path = NULL;
TRACE(nc_open);
@ -2221,14 +2213,12 @@ NC_open(const char *path0, int omode, int basepe, size_t *chunksizehintp,
/* Fix the inmemory related flags */
mmap = ((omode & NC_MMAP) == NC_MMAP);
diskless = ((omode & NC_DISKLESS) == NC_DISKLESS);
/* diskless && !mmap => inmemory */
if(diskless && !mmap) omode |= NC_INMEMORY;
inmemory = ((omode & NC_INMEMORY) == NC_INMEMORY);
if(mmap && inmemory) /* cannot have both */
return NC_EINMEMORY;
if(mmap && diskless) /* cannot have both */
return NC_EDISKLESS;
/* Attempt to do file path conversion: note that this will do
nothing if path is a 'file:...' url, so it will need to be
@ -2286,9 +2276,7 @@ NC_open(const char *path0, int omode, int basepe, size_t *chunksizehintp,
if(model == 0) {
version = 0;
/* Try to find dataset type */
if(inmemory) flags |= NC_INMEMORY;
if(diskless) flags |= NC_DISKLESS;
if(mmap) flags |= NC_MMAP;
int flags = omode;
stat = NC_check_file_type(path,flags,useparallel,parameters,&model,&version);
if(stat == NC_NOERR) {
if(model == 0) {
@ -2470,8 +2458,8 @@ static int
openmagic(struct MagicFile* file)
{
int status = NC_NOERR;
assert((!file->diskless && file->inmemory) ? file->parameters != NULL : 1);
if(file->inmemory && !file->diskless) {
assert((file->inmemory) ? file->parameters != NULL : 1);
if(file->inmemory) {
/* Get its length */
NC_memio* meminfo = (NC_memio*)file->parameters;
file->filelen = (long long)meminfo->size;
@ -2546,11 +2534,11 @@ readmagic(struct MagicFile* file, long pos, char* magic)
{
int status = NC_NOERR;
memset(magic,0,MAGIC_NUMBER_LEN);
if(file->inmemory && !file->diskless) {
if(file->inmemory) {
char* mempos;
NC_memio* meminfo = (NC_memio*)file->parameters;
if((pos + MAGIC_NUMBER_LEN) > meminfo->size)
{status = NC_EDISKLESS; goto done;}
{status = NC_EINMEMORY; goto done;}
mempos = ((char*)meminfo->memory) + pos;
memcpy((void*)magic,mempos,MAGIC_NUMBER_LEN);
#ifdef DEBUG

79
libdispatch/dutil.c
View File

@ -161,32 +161,6 @@ NC_entityescape(const char* s)
return escaped;
}
int
NC_readfile(const char* filename, NCbytes* content)
{
int ret = NC_NOERR;
FILE* stream = NULL;
char part[1024];
#ifdef _MSC_VER
stream = NCfopen(filename,"rb");
#else
stream = NCfopen(filename,"r");
#endif
if(stream == NULL) {ret=errno; goto done;}
for(;;) {
size_t count = fread(part, 1, sizeof(part), stream);
if(count <= 0) break;
ncbytesappendn(content,part,count);
if(ferror(stream)) {ret = NC_EIO; goto done;}
if(feof(stream)) break;
}
ncbytesnull(content);
done:
if(stream) fclose(stream);
return ret;
}
/**
Wrap mktmp and return the generated path,
or null if failed.
@ -252,3 +226,56 @@ NC_mktmp(const char* base)
return strdup(tmp);
}
int
NC_readfile(const char* filename, NCbytes* content)
{
int ret = NC_NOERR;
FILE* stream = NULL;
char part[1024];
#ifdef _WIN32
stream = NCfopen(filename,"rb");
#else
stream = NCfopen(filename,"r");
#endif
if(stream == NULL) {ret=errno; goto done;}
for(;;) {
size_t count = fread(part, 1, sizeof(part), stream);
if(count <= 0) break;
ncbytesappendn(content,part,count);
if(ferror(stream)) {ret = NC_EIO; goto done;}
if(feof(stream)) break;
}
ncbytesnull(content);
done:
if(stream) fclose(stream);
return ret;
}
int
NC_writefile(const char* filename, size_t size, void* content)
{
int ret = NC_NOERR;
FILE* stream = NULL;
void* p;
size_t remain;
#ifdef _WIN32
stream = NCfopen(filename,"wb");
#else
stream = NCfopen(filename,"w");
#endif
if(stream == NULL) {ret=errno; goto done;}
p = content;
remain = size;
while(remain > 0) {
size_t written = fwrite(p, 1, remain, stream);
if(ferror(stream)) {ret = NC_EIO; goto done;}
if(feof(stream)) break;
remain -= written;
}
done:
if(stream) fclose(stream);
return ret;
}

36
libhdf5/hdf5create.c
View File

@ -73,9 +73,14 @@ nc4_create_file(const char *path, int cmode, size_t initialsz,
nc4_info->mem.inmemory = (cmode & NC_INMEMORY) == NC_INMEMORY;
nc4_info->mem.diskless = (cmode & NC_DISKLESS) == NC_DISKLESS;
nc4_info->mem.persist = (cmode & NC_PERSIST) == NC_PERSIST;
nc4_info->mem.created = 1;
nc4_info->mem.initialsize = initialsz;
/* diskless => !inmemory */
if(nc4_info->mem.inmemory && nc4_info->mem.diskless)
BAIL(NC_EINTERNAL);
if(nc4_info->mem.inmemory && parameters)
nc4_info->mem.memio = *(NC_memio*)parameters;
#ifdef USE_PARALLEL4
@ -94,14 +99,11 @@ nc4_create_file(const char *path, int cmode, size_t initialsz,
/* If this file already exists, and NC_NOCLOBBER is specified,
return an error (unless diskless|inmemory) */
if (nc4_info->mem.diskless) {
if((cmode & NC_WRITE) && (cmode & NC_NOCLOBBER) == 0)
nc4_info->mem.persist = 1;
} else if (nc4_info->mem.inmemory) {
/* ok */
} else if ((cmode & NC_NOCLOBBER) && (fp = fopen(path, "r"))) {
fclose(fp);
BAIL(NC_EEXIST);
if (!nc4_info->mem.diskless && !nc4_info->mem.inmemory) {
if ((cmode & NC_NOCLOBBER) && (fp = fopen(path, "r"))) {
fclose(fp);
BAIL(NC_EEXIST);
}
}
/* Need this access plist to control how HDF5 handles open objects
@ -189,6 +191,24 @@ nc4_create_file(const char *path, int cmode, size_t initialsz,
BAIL(retval);
}
else
if(nc4_info->mem.diskless) {
size_t alloc_incr; /* Buffer allocation increment */
size_t min_incr = 65536; /* Minimum buffer increment */
double buf_prcnt = 0.1f; /* Percentage of buffer size to set as increment */
/* set allocation increment to a percentage of the supplied buffer size, or
* a pre-defined minimum increment value, whichever is larger
*/
if ((buf_prcnt * initialsz) > min_incr)
alloc_incr = (size_t)(buf_prcnt * initialsz);
else
alloc_incr = min_incr;
/* Configure FAPL to use the core file driver */
if (H5Pset_fapl_core(fapl_id, alloc_incr, (nc4_info->mem.persist?1:0)) < 0)
BAIL(NC_EHDFERR);
if ((hdf5_info->hdfid = H5Fcreate(path, flags, fcpl_id, fapl_id)) < 0)
BAIL(EACCES);
}
else /* Normal file */
{
/* Create the HDF5 file. */
if ((hdf5_info->hdfid = H5Fcreate(path, flags, fcpl_id, fapl_id)) < 0)

11
libhdf5/hdf5file.c
View File

@ -13,6 +13,7 @@
#include "config.h"
#include "hdf5internal.h"
#include "ncrc.h"
static void dumpopenobjects(NC_FILE_INFO_T* h5);
@ -225,15 +226,15 @@ nc4_close_netcdf4_file(NC_FILE_INFO_T *h5, int abort, NC_memio* memio)
}
/* If needed, reclaim extraneous memory */
if(h5->mem.memio.memory != NULL) {
/* If the original block of memory is not resizeable, then
it belongs to the caller and we should not free it. */
if(!h5->mem.locked)
free(h5->mem.memio.memory);
/* If the original block of memory is not resizeable, then
it belongs to the caller and we should not free it. */
if(!h5->mem.locked)
free(h5->mem.memio.memory);
}
h5->mem.memio.memory = NULL;
h5->mem.memio.size = 0;
NC4_image_finalize(h5->mem.udata);
}
}
/* Free the HDF5-specific info. */
if (h5->format_file_info)

37
libhdf5/hdf5open.c
View File

@ -11,6 +11,7 @@
#include "config.h"
#include "hdf5internal.h"
#include "ncrc.h"
#define NUM_TYPES 12 /**< Number of netCDF atomic types. */
#define CD_NELEMS_ZLIB 1 /**< Number of parameters needed for ZLIB filter. */
@ -389,6 +390,13 @@ nc4_open_file(const char *path, int mode, void* parameters, NC *nc)
nc4_info->mem.inmemory = ((mode & NC_INMEMORY) == NC_INMEMORY);
nc4_info->mem.diskless = ((mode & NC_DISKLESS) == NC_DISKLESS);
nc4_info->mem.persist = ((mode & NC_PERSIST) == NC_PERSIST);
/* Does the mode specify that this file is read-only? */
if ((mode & NC_WRITE) == 0)
nc4_info->no_write = NC_TRUE;
if(nc4_info->mem.inmemory && nc4_info->mem.diskless)
BAIL(NC_EINTERNAL);
#ifdef USE_PARALLEL4
mpiinfo = (NC_MPI_INFO*)parameters; /* assume, may be changed if inmemory is true */
@ -444,38 +452,42 @@ nc4_open_file(const char *path, int mode, void* parameters, NC *nc)
nc4_chunk_cache_preemption));
#endif /* USE_PARALLEL4 */
/* Does the mode specify that this file is read-only? */
if ((mode & NC_WRITE) == 0)
nc4_info->no_write = NC_TRUE;
/* Now process if NC_INMEMORY is set (recall NC_DISKLESS => NC_INMEMORY) */
/* Process NC_INMEMORY */
if(nc4_info->mem.inmemory) {
NC_memio* memio;
/* validate */
if(parameters == NULL)
BAIL(NC_EINMEMORY);
memio = (NC_memio*)parameters;
if(memio->memory == NULL || memio->size == 0)
BAIL(NC_EINMEMORY);
/* initialize h5->mem */
nc4_info->mem.memio = *memio;
/* Is the incoming memory locked? */
nc4_info->mem.locked = (nc4_info->mem.memio.flags & NC_MEMIO_LOCKED) == NC_MEMIO_LOCKED;
/* As a safeguard, if !locked and not read-only,
then we must take control of the incoming memory */
if(!nc4_info->mem.locked && !nc4_info->no_write) {
memio->memory = NULL; /* take control */
memio->size = 0;
memio->memory = NULL; /* take control */
memio->size = 0;
}
retval = NC4_open_image_file(nc4_info);
if(retval)
BAIL(NC_EHDFERR);
}
else
if(nc4_info->mem.diskless) { /* Process NC_DISKLESS */
NC_HDF5_FILE_INFO_T *hdf5_info;
size_t min_incr = 65536; /* Minimum buffer increment */
/* Configure FAPL to use the core file driver */
if (H5Pset_fapl_core(fapl_id, min_incr, (nc4_info->mem.persist?1:0)) < 0)
BAIL(NC_EHDFERR);
hdf5_info = (NC_HDF5_FILE_INFO_T *)nc4_info->format_file_info;
/* Open the HDF5 file. */
if ((hdf5_info->hdfid = H5Fopen(path, flags, fapl_id)) < 0)
BAIL(NC_EHDFERR);
}
else
{
NC_HDF5_FILE_INFO_T *hdf5_info;
hdf5_info = (NC_HDF5_FILE_INFO_T *)nc4_info->format_file_info;
@ -574,6 +586,9 @@ NC4_open(const char *path, int mode, int basepe, size_t *chunksizehintp,
if (mode & ILLEGAL_OPEN_FLAGS)
return NC_EINVAL;
if((mode & NC_DISKLESS) && (mode & NC_INMEMORY))
return NC_EINVAL;
/* If this is our first file, initialize HDF5. */
if (!nc4_hdf5_initialized)
nc4_hdf5_initialize();

4
libhdf5/nc4info.c
View File

@ -137,8 +137,8 @@ locate(char* p, char tag)
/**
* @internal Parse file properties.
*
* @param properties list of parsed (key,value) pairs
* @param text Text properties.
* @param text0 Text properties.
* @param pairs list of parsed (key,value) pairs
*
* @return ::NC_NOERR No error.
* @author Dennis Heimbigner

18
libsrc/memio.c
View File

@ -94,7 +94,7 @@ reallocx(void* mem, size_t newsize, size_t oldsize)
typedef struct NCMEMIO {
int locked; /* => we cannot realloc or free*/
int modified; /* => we realloc'd memory at least once */
int persist; /* => save to a file; triggered by NC_WRITE */
int persist; /* => save to a file; triggered by NC_PERSIST*/
char* memory;
size_t alloc;
size_t size;
@ -131,6 +131,10 @@ memio_new(const char* path, int ioflags, off_t initialsize, ncio** nciopp, NCMEM
NCMEMIO* memio = NULL;
size_t minsize = (size_t)initialsize;
/* Unlike netcdf-4, INMEMORY and DISKLESS share code */
if(fIsSet(ioflags,NC_DISKLESS))
fSet(ioflags,NC_INMEMORY);
/* use asserts because this is an internal function */
assert(fIsSet(ioflags,NC_INMEMORY));
assert(memiop != NULL && nciopp != NULL);
@ -199,7 +203,7 @@ memio_new(const char* path, int ioflags, off_t initialsize, ncio** nciopp, NCMEM
memio->diskless = 1;
if(fIsSet(ioflags,NC_INMEMORY))
memio->inmemory = 1;
if(fIsSet(ioflags,NC_WRITE) && !fIsSet(ioflags,NC_NOCLOBBER) && memio->diskless)
if(fIsSet(ioflags,NC_PERSIST))
memio->persist = 1;
done:
@ -265,7 +269,7 @@ fprintf(stderr,"memio_create: initial memory: %lu/%lu\n",(unsigned long)memio->m
fd = nc__pseudofd();
*((int* )&nciop->fd) = fd;
fSet(nciop->ioflags, NC_WRITE);
fSet(nciop->ioflags, NC_WRITE); /* Always writeable */
if(igetsz != 0)
{
@ -318,9 +322,11 @@ memio_open(const char* path,
size_t initialsize;
/* Should be the case that diskless => inmemory but not converse */
int diskless = (fIsSet(ioflags,NC_DISKLESS));
int inmemory = (fIsSet(ioflags,NC_INMEMORY) && !diskless);
int inmemory = fIsSet(ioflags,NC_INMEMORY);
int locked = 0;
assert(inmemory ? !diskless : 1);
if(path == NULL || strlen(path) == 0)
return NC_EINVAL;
@ -440,10 +446,10 @@ memio_pad_length(ncio* nciop, off_t length)
if(nciop == NULL || nciop->pvt == NULL) return NC_EINVAL;
memio = (NCMEMIO*)nciop->pvt;
if(!memio->persist)
if(fIsSet(nciop->ioflags,NC_WRITE))
return EPERM; /* attempt to write readonly file*/
if(memio->locked)
return NC_EDISKLESS;
return NC_EINMEMORY;
if(len > memio->alloc) {
/* Realloc the allocated memory to a multiple of the pagesize*/

16
libsrc/mmapio.c
View File

@ -90,7 +90,7 @@
typedef struct NCMMAPIO {
int locked; /* => we cannot realloc */
int persist; /* => save to a file; triggered by NC_WRITE */
int persist; /* => save to a file; triggered by NC_PERSIST */
char* memory;
off_t alloc;
off_t size;
@ -164,7 +164,7 @@ mmapio_new(const char* path, int ioflags, off_t initialsize, ncio** nciopp, NCMM
mmapio->memory = NULL;
mmapio->size = 0;
mmapio->pos = 0;
mmapio->persist = fIsSet(ioflags,NC_WRITE);
mmapio->persist = fIsSet(ioflags,NC_PERSIST);
/* See if ok to use mmap */
if(sizeof(void*) < 8 &&
@ -211,7 +211,7 @@ mmapio_create(const char* path, int ioflags,
int fd;
int status;
NCMMAPIO* mmapio = NULL;
int persist = (ioflags & NC_WRITE?1:0);
int persist = (ioflags & NC_PERSIST?1:0);
int oflags;
if(path == NULL ||* path == 0)
@ -234,8 +234,8 @@ mmapio_create(const char* path, int ioflags,
mmapio->mapfd,0);
{mmapio->memory[0] = 0;} /* test writing of the mmap'd memory */
} else { /*persist */
/* Open the file, but make sure we can write it if needed */
oflags = (persist ? O_RDWR : O_RDONLY);
/* Open the file to get fd, but make sure we can write it if needed */
oflags = O_RDWR;
#ifdef O_BINARY
fSet(oflags, O_BINARY);
#endif
@ -319,11 +319,11 @@ mmapio_open(const char* path,
ncio* nciop;
int fd;
int status;
int persist = (fIsSet(ioflags,NC_WRITE)?1:0);
int oflags;
NCMMAPIO* mmapio = NULL;
size_t sizehint;
off_t filesize;
int readwrite = (fIsSet(ioflags,NC_WRITE)?1:0);
if(path == NULL ||* path == 0)
return EINVAL;
@ -332,7 +332,7 @@ mmapio_open(const char* path,
sizehint = *sizehintp;
/* Open the file, but make sure we can write it if needed */
oflags = (persist ? O_RDWR : O_RDONLY);
oflags = (readwrite ? O_RDWR : O_RDONLY);
#ifdef O_BINARY
fSet(oflags, O_BINARY);
#endif
@ -359,7 +359,7 @@ mmapio_open(const char* path,
mmapio->mapfd = fd;
mmapio->memory = (char*)mmap(NULL,mmapio->alloc,
persist?(PROT_READ|PROT_WRITE):(PROT_READ),
readwrite?(PROT_READ|PROT_WRITE):(PROT_READ),
MAP_SHARED,
mmapio->mapfd,0);
#ifdef DEBUG

29
libsrc/ncio.c
View File

@ -43,14 +43,16 @@ ncio_create(const char *path, int ioflags, size_t initialsz,
void* parameters,
ncio** iopp, void** const mempp)
{
if(fIsSet(ioflags,NC_INMEMORY)) {
# ifdef USE_MMAP
if(fIsSet(ioflags,NC_MMAP) && fIsSet(ioflags, NC_DISKLESS))
return mmapio_create(path,ioflags,initialsz,igeto,igetsz,sizehintp,parameters,iopp,mempp);
else
# endif /*USE_MMAP*/
if(fIsSet(ioflags,NC_DISKLESS)) {
return memio_create(path,ioflags,initialsz,igeto,igetsz,sizehintp,parameters,iopp,mempp);
} else if(fIsSet(ioflags,NC_INMEMORY)) {
return memio_create(path,ioflags,initialsz,igeto,igetsz,sizehintp,parameters,iopp,mempp);
}
# ifdef USE_MMAP
else if(fIsSet(ioflags,NC_MMAP)) {
return mmapio_create(path,ioflags,initialsz,igeto,igetsz,sizehintp,parameters,iopp,mempp);
}
# endif /*USE_MMAP*/
#ifdef USE_STDIO
return stdio_create(path,ioflags,initialsz,igeto,igetsz,sizehintp,parameters,iopp,mempp);
@ -70,14 +72,17 @@ ncio_open(const char *path, int ioflags,
/* Diskless open has the following constraints:
1. file must be classic version 1 or 2 or 5
*/
if(fIsSet(ioflags,NC_INMEMORY)) {
# ifdef USE_MMAP
if(fIsSet(ioflags,NC_MMAP) && fIsSet(ioflags, NC_DISKLESS))
return mmapio_open(path,ioflags,igeto,igetsz,sizehintp,parameters,iopp,mempp);
else
# endif /*USE_MMAP*/
if(fIsSet(ioflags,NC_DISKLESS)) {
return memio_open(path,ioflags,igeto,igetsz,sizehintp,parameters,iopp,mempp);
}
if(fIsSet(ioflags,NC_INMEMORY)) {
return memio_open(path,ioflags,igeto,igetsz,sizehintp,parameters,iopp,mempp);
}
# ifdef USE_MMAP
if(fIsSet(ioflags,NC_MMAP)) {
return mmapio_open(path,ioflags,igeto,igetsz,sizehintp,parameters,iopp,mempp);
}
# endif /*USE_MMAP*/
#ifdef USE_STDIO
return stdio_open(path,ioflags,igeto,igetsz,sizehintp,parameters,iopp,mempp);
#elif defined(USE_FFIO)

1
nc_test/CMakeLists.txt
View File

@ -56,6 +56,7 @@ ENDIF()
SET(TESTFILES ${TESTFILES} tst_diskless tst_diskless3 tst_diskless4 tst_diskless5 tst_inmemory tst_open_mem)
IF(USE_NETCDF4)
SET(TESTFILES ${TESTFILES} tst_diskless2)
SET(TESTFILES ${TESTFILES} tst_diskless6)
ENDIF()
# Build executables required for the shell scripts.

6
nc_test/Make0
View File

@ -1,12 +1,12 @@
# Test c output
T=nc_test
T=tst_diskless3
#H58=8
H510=10
ARGS=persist
ARGS=diskless persist
#SRC=test_put.c test_get.c test_read.c test_write.c util.c error.c
#SRC=
#CMD=env HDF5_DEBUG=trace
#CMD=export NETCDF_LOG_LEVEL=5 ;gdb --args

5
nc_test/Makefile.am
View File

@ -59,16 +59,13 @@ check_PROGRAMS += tst_diskless tst_diskless3 tst_diskless4 \
tst_diskless5 tst_inmemory tst_open_mem
if USE_NETCDF4
check_PROGRAMS += tst_diskless2
TESTPROGRAMS += tst_diskless6
endif
TESTS = $(TESTPROGRAMS)
if BUILD_UTILITIES
TESTS += run_diskless.sh run_diskless5.sh run_inmemory.sh
if BUILD_MMAP
TESTS += run_mmap.sh
run_mmap.log: run_diskless.log
endif
if LARGE_FILE_TESTS
TESTS += run_diskless2.sh
endif

9
nc_test/run_diskless.sh
View File

@ -1,11 +1,10 @@
#!/bin/sh
if test "x$srcdir" = x ; then srcdir=`pwd`; fi
. ../test_common.sh
set -e
set -x
# Get the target OS and CPU
CPU=`uname -p`
OS=`uname`
@ -100,15 +99,15 @@ echo "**** Testing nc_open in-memory (diskless) files"
rm -f tst_diskless3_file.cdl tst_diskless3_memory.cdl
echo ""
echo "**** Create and modify file without using diskless"
echo "**** Create baseline cdl"
rm -f $FILE3
${execdir}/tst_diskless3
${execdir}/tst_diskless3 file
${NCDUMP} $FILE3 >tst_diskless3_file.cdl
echo ""
echo "**** Create and modify file using diskless"
rm -f $FILE3
${execdir}/tst_diskless3 diskless
${execdir}/tst_diskless3 diskless persist
${NCDUMP} $FILE3 >tst_diskless3_memory.cdl
# compare

22
nc_test/run_mmap.sh
View File

@ -16,41 +16,35 @@ FILE2=tst_diskless2.nc
FILE3=tst_diskless3.nc
echo ""
echo "*** Testing in-memory (diskless) files with mmap"
echo "*** Testing create files with mmap"
echo "**** Test diskless+mmap netCDF classic file without persistence"
echo "**** Test create mmap netCDF classic file without persistence"
${execdir}/tst_diskless mmap
echo "PASS: diskless+mmap netCDF classic file without persistence"
echo "PASS: create mmap netCDF classic file without persistence"
echo ""
echo "**** Test diskless+mmap netCDF classic file with persistence"
echo "**** Test create mmap netCDF classic file with persistence"
rm -f $FILE1
${execdir}/tst_diskless mmap persist
if test -f $FILE1 ; then
echo "**** $FILE1 created"
# ${NCDUMP} $FILE1
echo "PASS: diskless+mmap netCDF classic file with persistence"
echo "PASS: create mmap netCDF classic file with persistence"
else
echo "#### $FILE1 not created"
echo "FAIL: diskless+mmap netCDF classic file with persistence"
echo "FAIL: create mmap netCDF classic file with persistence"
fi
rm -f tmp1.cdl tmp2.cdl tmp1.nc tmp2.nc
echo ""
echo "**** Testing nc_open in-memory (diskless+mmap) files"
echo "**** Testing open files with mmap"
# clear old files
rm -f tst_diskless3_file.cdl tst_diskless3_memory.cdl
echo ""
echo "**** Create and modify file without using diskless+mmap"
rm -f $FILE3
${execdir}/tst_diskless3
${NCDUMP} $FILE3 >tst_diskless3_file.cdl
echo ""
echo "**** Create and modify file using diskless+mmap"
echo "**** Open and modify file using mmap"
rm -f $FILE3
${execdir}/tst_diskless3 diskless mmap
${NCDUMP} $FILE3 >tst_diskless3_memory.cdl

54
nc_test/tst_diskless.c
View File

@ -5,7 +5,7 @@ Copyright 2011, UCAR/Unidata. See COPYRIGHT file for copying and
redistribution conditions.
*/
#undef DDBG
#define DDBG
#include "config.h"
#include <stdio.h>
@ -23,11 +23,12 @@ variables:
}
*/
#ifndef NC_NETCDF3
#define NC_NETCDF3 0
#endif
#define FLAGS4 (NC_DISKLESS|NC_NETCDF4|NC_CLASSIC_MODEL)
#define FLAGS3 (NC_DISKLESS)
#define PERSIST (NC_WRITE)
#define FLAGS4 (NC_NETCDF4|NC_CLASSIC_MODEL)
#define FLAGS3 (NC_NETCDF3)
#define RESISTOR "resistor_value"
#define CAPACITOR "capacitor_value"
@ -37,7 +38,7 @@ variables:
#undef ERR
void fail(int line) {
fflush(stdout);
fprintf(stderr,"\nline=%d\n",line);
fprintf(stderr,"\nfail: line=%d\n",line);
fflush(stderr);
exit(1);
}
@ -45,7 +46,31 @@ void fail(int line) {
#endif
/* Control flags */
static int flags, persist, usenetcdf4, mmap;
static int flags, persist, usenetcdf4, mmap, diskless;
char*
smode(int mode)
{
static char ms[8192];
ms[0] = '\0';
if(mode & NC_NETCDF4)
strcat(ms,"NC_NETCDF4");
else
strcat(ms,"NC_NETCDF3");
if(mode & NC_DISKLESS)
strcat(ms,"|NC_DISKLESS");
if(mode & NC_WRITE)
strcat(ms,"|NC_WRITE");
if(mode & NC_NOCLOBBER)
strcat(ms,"|NC_NOCLOBBER");
if(mode & NC_INMEMORY)
strcat(ms,"|NC_INMEMORY");
if(mode & NC_PERSIST)
strcat(ms,"|NC_PERSIST");
if(mode & NC_MMAP)
strcat(ms,"|NC_MMAP");
return ms;
}
/* Remove a file; do not care if it does not exist */
static void
@ -66,26 +91,31 @@ main(int argc, char **argv)
persist = 0;
usenetcdf4 = 0;
mmap = 0;
diskless = 0;
for(i=1;i<argc;i++) {
if(strcmp(argv[i],"netcdf4")==0) usenetcdf4=1;
else if(strcmp(argv[i],"persist")==0) persist=1;
else if(strcmp(argv[i],"mmap")==0) mmap=1;
else if(strcmp(argv[i],"diskless")==0) diskless=1;
/* ignore anything not recognized */
}
#ifndef USE_NETCDF4
fprintf(stderr,"netcdf-4 not supported; ignored\n");
usenetcdf4 = 0;
#endif
if(mmap)
usenetcdf4 = 0;
/* Invalid combinations */
if(mmap && diskless) {fprintf(stderr,"Illegal: mmap+diskless\n"); exit(1);};
if(mmap && usenetcdf4) {fprintf(stderr,"Illegal: mmap+netcdf4\n"); exit(1);};
flags = usenetcdf4?FLAGS4:FLAGS3;
if(persist) flags |= PERSIST;
if(persist) flags |= NC_PERSIST;
if(mmap) flags |= NC_MMAP;
if(diskless) flags |= NC_DISKLESS;
printf("\n*** Testing the diskless API.\n");
printf("\n*** Testing the diskless|mmap API.\n");
printf("*** testing diskless file with scalar vars...");
{
int ncid, varid0, varid1, varid2;
@ -98,7 +128,7 @@ printf("*** testing diskless file with scalar vars...");
removefile(persist,filename);
/* Create a netCDF file (which exists only in memory). */
/* Create a netCDF file (which exists in memory). */
if (nc_create(filename, flags, &ncid)) ERR;
/* Create some variables. */

65
nc_test/tst_diskless3.c
View File

@ -38,10 +38,34 @@
static int status = NC_NOERR;
/* Control flags */
static int persist, usenetcdf4, mmap, diskless;
static int persist, usenetcdf4, mmap, diskless, file;
static int diskmode;
char*
smode(int mode)
{
static char ms[8192];
ms[0] = '\0';
if(mode & NC_NETCDF4)
strcat(ms,"NC_NETCDF4");
else
strcat(ms,"NC_NETCDF3");
if(mode & NC_DISKLESS)
strcat(ms,"|NC_DISKLESS");
if(mode & NC_WRITE)
strcat(ms,"|NC_WRITE");
if(mode & NC_NOCLOBBER)
strcat(ms,"|NC_NOCLOBBER");
if(mode & NC_INMEMORY)
strcat(ms,"|NC_INMEMORY");
if(mode & NC_PERSIST)
strcat(ms,"|NC_PERSIST");
if(mode & NC_MMAP)
strcat(ms,"|NC_MMAP");
return ms;
}
/* Test a diskless file with two record vars, which grow, and has
* attributes added. */
static int
@ -54,23 +78,18 @@ test_two_growing_with_att(const char *testfile)
int v, r;
/* Create a file with one ulimited dimensions, and one var. */
if((status=nc_create(testfile, NC_CLOBBER, &ncid))) ERRSTAT(status);
if((status=nc_create(testfile, diskmode|NC_CLOBBER, &ncid))) ERRSTAT(status);
if((status=nc_def_dim(ncid, DIM1_NAME, NC_UNLIMITED, &dimid))) ERRSTAT(status);
if((status=nc_def_var(ncid, VAR_NAME, NC_CHAR, 1, &dimid, &varid[0]))) ERRSTAT(status);
if((status=nc_def_var(ncid, VAR_NAME2, NC_CHAR, 1, &dimid, &varid[1]))) ERRSTAT(status);
if((status=nc_close(ncid))) ERRSTAT(status);
if((status=nc_enddef(ncid))) ERRSTAT(status);
/* Create some phoney data. */
for (data[0] = 'a', r = 1; r < MAX_RECS; r++)
data[r] = data[r - 1] + 1;
/* Normally one would not close and reopen the file for each
* record, nor add an attribute each time I add a record, but I am
* giving the library a little work-out here... */
for (r = 0; r < MAX_RECS; r++)
{
/* Write one record of var data, a single character. */
if((status=nc_open(testfile, NC_WRITE, &ncid))) ERRSTAT(status);
count[0] = 1;
start[0] = r;
sprintf(att_name, "a_%d", data[r]);
@ -81,10 +100,8 @@ test_two_growing_with_att(const char *testfile)
if((status=nc_put_att_text(ncid, varid[v], att_name, 1, &data[r]))) ERRSTAT(status);
if((status=nc_enddef(ncid))) ERRSTAT(status);
}
if((status=nc_close(ncid))) ERRSTAT(status);
/* Reopen the file and check it. */
if((status=nc_open(testfile, diskmode|NC_WRITE, &ncid))) ERRSTAT(status);
/* verify */
if((status=nc_inq_dimlen(ncid, 0, &len_in))) ERRSTAT(status);
if (len_in != r + 1) ERR;
index[0] = r;
@ -93,8 +110,8 @@ test_two_growing_with_att(const char *testfile)
if((status=nc_get_var1_text(ncid, varid[v], index, &data_in))) ERRSTAT(status);
if (data_in != data[r]) ERR;
}
if((status=nc_close(ncid))) ERRSTAT(status);
} /* Next record. */
if((status=nc_close(ncid))) ERRSTAT(status);
return 0;
}
@ -146,24 +163,36 @@ main(int argc, char **argv)
usenetcdf4 = 0;
mmap = 0;
diskless = 0;
file = 0;
diskmode = 0;
for(i=1;i<argc;i++) {
if(strcmp(argv[i],"diskless")==0) diskless=1;
else if(strcmp(argv[i],"mmap")==0) mmap=1;
else if(strcmp(argv[i],"file")==0) file=1;
else if(strcmp(argv[i],"persist")==0) persist=1;
/* ignore anything not recognized */
}
if(diskless && mmap) {
fprintf(stderr,"NC_DISKLESS and NC_MMAP are mutually exclusive\n");
exit(1);
}
if(!diskless && !mmap && !file) {
fprintf(stderr,"file or diskless or mmap must be specified\n");
exit(1);
}
if(diskless)
diskmode |= NC_DISKLESS;
if(diskless && mmap)
if(mmap)
diskmode |= NC_MMAP;
if(persist)
diskmode |= NC_PERSIST;
printf("\n*** Testing diskless file: create/modify %s",
diskless?"in-memory":"in-file");
if(diskless && mmap)
printf("+mmap");
printf(" %s\n",NCFILENAME);
printf("\n*** Testing create/modify file=%s mode=%s\n", NCFILENAME,
diskless?"diskless":"mmap");
/* case NC_FORMAT_CLASSIC: only test this format */
nc_set_default_format(NC_FORMAT_CLASSIC, NULL);

6
nc_test/tst_diskless5.c
View File

@ -1,5 +1,3 @@
/* Diskless test in support of https://github.com/Unidata/netcdf-c/issues/400 */
#include <stdlib.h>
#include <stdio.h>
#include <assert.h>
@ -27,10 +25,10 @@ static int mem = 0;
#endif
#ifdef DISKLESS
#define MODE NC_NOCLOBBER|NC_DISKLESS
#define MODE NC_DISKLESS
static int diskless = 1;
#else
#define MODE NC_NOCLOBBER
#define MODE 0
static int diskless = 0;
#endif

212
nc_test/tst_diskless6.c Normal file
View File

@ -0,0 +1,212 @@
#include "config.h"
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#ifdef HAVE_TIME_H
#include <time.h>
#endif
#ifdef HAVE_SYS_STAT_H
#include <sys/stat.h>
#endif
#ifdef HAVE_SYS_TYPES_H
#include <sys/types.h>
#endif
#ifdef HAVE_UNISTD_H
#include "unistd.h"
#endif
#ifdef _WIN32
#include <io.h>
#endif
#include "netcdf.h"
#define CLEANUP
#ifndef NC_NETCDF3
#define NC_NETCDF3 0
#endif
#define RDONLY (S_IRUSR|S_IRGRP|S_IROTH)
#define RDWRITE (RDONLY | S_IWUSR)
#define FILE3D "file3d.nc"
#define FILE3DP "file3dp.nc"
#define FILE4D "file4d.nc"
#define FILE4DP "file4dp.nc"
typedef enum OC { OPEN, CLOSE} OC;
static int lineno = 0;
static char*
smode(int mode)
{
static char ms[8192];
ms[0] = '\0';
if(mode & NC_NETCDF4)
strcat(ms,"NC_NETCDF4");
else
strcat(ms,"NC_NETCDF3");
if(mode & NC_DISKLESS)
strcat(ms,"|NC_DISKLESS");
if(mode & NC_WRITE)
strcat(ms,"|NC_WRITE");
if(mode & NC_NOCLOBBER)
strcat(ms,"|NC_NOCLOBBER");
if(mode & NC_INMEMORY)
strcat(ms,"|NC_INMEMORY");
if(mode & NC_PERSIST)
strcat(ms,"|NC_PERSIST");
if(mode & NC_MMAP)
strcat(ms,"|NC_MMAP");
return ms;
}
/* Return 1 if file was changed else 0 */
static time_t
getmodified(const char* path)
{
struct stat attr;
stat(path, &attr);
return attr.st_mtime;
}
static void
changeaccess(mode_t mode)
{
(void)chmod(FILE3DP,mode);
(void)chmod(FILE4DP,mode);
}
static void
cleanup()
{
changeaccess(RDWRITE);
/* cleanup */
(void)unlink(FILE3DP);
(void)unlink(FILE4DP);
}
static void
fail(int ret)
{
if(ret != NC_NOERR) {
fprintf(stderr,"*** Fail: line: %d: (%d) %s\n", lineno, ret, nc_strerror(ret));
fflush(stderr);
#ifdef CLEANUP
cleanup();
#endif
exit(1);
}
}
void
exists(const char* file)
{
FILE* f = fopen(file,"r");
if(f == NULL) fail(NC_EPERM);
fclose(f);
}
void
notexists(const char* file)
{
FILE* f = fopen(file,"r");
if(f != NULL) {fclose(f); fail(NC_EEXIST);}
}
#define TESTCREATE(file,mode,exist) {lineno=__LINE__; testcreate(file,mode,exist);}
#define TESTOPEN(file,mode,rw) {lineno=__LINE__; testopen(file,mode,rw);}
static void
testcreate(const char* file, int mode, int mustexist)
{
int ret = NC_NOERR;
int ncid, dimid;
printf("test: file=%s mode=%s\n",file,smode(mode)); fflush(stdout);
if((ret = nc_create(file,mode,&ncid))) fail(ret);
if((ret = nc_def_dim(ncid,"dim",5,&dimid))) fail(ret);
if((ret = nc_close(ncid))) fail(ret);
if(mustexist)
exists(file);
else
notexists(file);
}
static void
testopen(const char* file, int mode, int rdwrite)
{
int ret = NC_NOERR;
int ncid, dimid;
size_t len;
time_t time1, time2;
printf("test: file=%s mode=%s\n",file,smode(mode)); fflush(stdout);
time1 = getmodified(file);
sleep(1); /* Ensure that if modified, it will be reported */
if((ret = nc_open(file,mode,&ncid))) fail(ret);
if(rdwrite) {
/* Modify */
if((ret=nc_redef(ncid))) fail(ret);
/* See if dim2 is already defined */
ret = nc_inq_dimid(ncid,"dim2",&dimid);
if(ret == NC_NOERR) {/* defined */
if((ret = nc_def_dim(ncid,"dim3",3,&dimid))) fail(ret);
} else if(ret == NC_EBADDIM) { /* not defined */
if((ret = nc_def_dim(ncid,"dim2",2,&dimid))) fail(ret);
} else
fail(ret);
if((ret=nc_enddef(ncid))) fail(ret);
}
if((ret = nc_inq_dimid(ncid,"dim",&dimid))) fail(ret);
if((ret = nc_inq_dimlen(ncid,dimid,&len))) fail(ret);
ret = nc_close(ncid);
if(ret != NC_NOERR && rdwrite) fail(ret);
else if(ret == NC_NOERR && !rdwrite) fail(NC_EINVAL);
time2 = getmodified(file);
if(!rdwrite) {
if(time2 != time1) {
fprintf(stderr,"file modified: %s\n",file);
fail(NC_EACCESS);
}
}
}
int
main()
{
changeaccess(RDWRITE);
cleanup();
/* Test various mode combinations */
/* Create and persist some files using diskless */
printf("*** Test create\n"); fflush(stdout);
TESTCREATE(FILE3D,NC_NETCDF3|NC_DISKLESS,0); /* Not persisted */
TESTCREATE(FILE3DP,NC_NETCDF3|NC_DISKLESS|NC_PERSIST,1); /* persisted */
TESTCREATE(FILE4D,NC_NETCDF4|NC_DISKLESS,0); /* Not persisted */
TESTCREATE(FILE4DP,NC_NETCDF4|NC_DISKLESS|NC_PERSIST,1); /* persisted */
/* Of the above persisted files, re-read and modify and re-persist */
printf("*** Test open + modify + rdwrite\n"); fflush(stdout);
TESTOPEN(FILE3DP,NC_NETCDF3|NC_DISKLESS|NC_PERSIST|NC_WRITE,1); /* re-persist */
TESTOPEN(FILE4DP,NC_NETCDF4|NC_DISKLESS|NC_PERSIST|NC_WRITE,1); /* re-persist */
/* Of the above modified files, re-read and modify but do not re-persist */
/* Test open + modify + rdonly; requires NC_DISKLESS*/
#if 0
/* Fails with hdf5, file must be writeable even if not persisted */
changeaccess(RDONLY); /* prevent re-persist */
#endif
printf("*** Testopen modify + rdonly\n"); fflush(stdout);
TESTOPEN(FILE3DP,NC_NETCDF3|NC_DISKLESS|NC_WRITE,1);
TESTOPEN(FILE4DP,NC_NETCDF4|NC_DISKLESS|NC_WRITE,1);
#ifdef CLEANUP
cleanup();
#endif
return 0;
}

8
ncdump/dumplib.c
View File

@ -841,7 +841,8 @@ ncuint64_typ_tostring(const nctype_t *typ, safebuf_t *sfbf, const void *valp) {
return sbuf_len(sfbf);
}
int ncstring_typ_tostring(const nctype_t *typ, safebuf_t *sfbf, const void *valp) {
int ncstring_typ_tostring(const nctype_t *typ, safebuf_t *sfbf, const void *valp)
{
const char *cp;
cp = ((char **)valp)[0];
@ -851,7 +852,8 @@ int ncstring_typ_tostring(const nctype_t *typ, safebuf_t *sfbf, const void *valp
char *sp;
unsigned char uc;
slen = 3 + 5 * strlen(cp); /* need "'s around string, and extra space to escape control characters */
slen = 4 + 5 * strlen(cp); /* need "'s around string, and extra space to escape control characters */
slen++; /* nul term */
sout = emalloc(slen);
sp = sout;
*sp++ = '"' ;
@ -895,7 +897,7 @@ int ncstring_typ_tostring(const nctype_t *typ, safebuf_t *sfbf, const void *valp
break;
default:
if (iscntrl(uc)) {
snprintf(sp,3,"\\%03o",uc);
snprintf(sp,4+1,"\\%03o",uc); /* +1 for nul */
sp += 4;
}
else

2
ncdump/ncdump.c
View File

@ -2314,7 +2314,7 @@ main(int argc, char *argv[])
void* mem = NULL;
nc_status = fileopen(path,&mem,&size);
if(nc_status == NC_NOERR)
nc_status = nc_open_mem(path,NC_DISKLESS|NC_INMEMORY,size,mem,&ncid);
nc_status = nc_open_mem(path,NC_INMEMORY,size,mem,&ncid);
} else
nc_status = nc_open(path, NC_NOWRITE, &ncid);
if (nc_status != NC_NOERR) {

2
ncdump/utils.c
View File

@ -84,7 +84,7 @@ escaped_name(const char* cp) {
for (; *cp; cp++) {
if (isascii((int)*cp)) {
if(iscntrl((int)*cp)) { /* render control chars as two hex digits, \%xx */
snprintf(sp, 4,"\\%%%.2x", *cp);
snprintf(sp, 4+1,"\\%%%.2x", *cp);
sp += 4;
} else {
switch (*cp) {