Commit Graph

44 Commits

Author SHA1 Message Date
Ed Hartnett
9b0192fe94 moved memfile code to libhdf5 2018-07-19 07:26:27 -06:00
Ed Hartnett
c402b79ed8 merged changes from master 2018-05-08 12:13:03 -06:00
Ed Hartnett
5526ca65d1 created libhdf5, moved some files 2018-05-08 11:58:01 -06:00
Dennis Heimbigner
4739cd3225 Master merge and conflict resolution 2018-04-12 21:51:17 -06:00
Dennis Heimbigner
25f062528b This completes (for now) the refactoring of libsrc4.
The file docs/indexing.dox tries to provide design
information for the refactoring.

The primary change is to replace all walking of linked
lists with the use of the NCindex data structure.
Ncindex is a combination of a hash table (for name-based
lookup) and a vector (for walking the elements in the index).
Additionally, global vectors are added to NC_HDF5_FILE_INFO_T
to support direct mapping of an e.g. dimid to the NC_DIM_INFO_T
object. These global vectors exist for dimensions, types, and groups
because they have globally unique id numbers.

WARNING:
1. since libsrc4 and libsrchdf4 share code, there are also
   changes in libsrchdf4.
2. Any outstanding pull requests that change libsrc4 or libhdf4
   are likely to cause conflicts with this code.
3. The original reason for doing this was for performance improvements,
   but as noted elsewhere, this may not be significant because
   the meta-data read performance apparently is being dominated
   by the hdf5 library because we do bulk meta-data reading rather
   than lazy reading.
2018-03-16 11:46:18 -06:00
Dennis Heimbigner
ccc70d640b re: esupport MQO-415619
and https://github.com/Unidata/netcdf-c/issues/708

Expand the NC_INMEMORY capabilities to support writing and accessing
the final modified memory.

Three new functions have been added:
nc_open_memio, nc_create_mem, and nc_close_memio.

The following new capabilities were added.
1. nc_open_memio() allows the NC_WRITE mode flag
   so a chunk of memory can be passed in and be modified
2. nc_create_mem() allows the NC_INMEMORY flag to be set
   to cause the created file to be kept in memory.
3. nc_close_mem() allows the final in-memory contents to be
   retrieved at the time the file is closed.
4. A special flag, NC_MEMIO_LOCK, is provided to ensure that
   the provided memory will not be freed or reallocated.

Note the following.
1. If nc_open_memio() is called with NC_WRITE, and NC_MEMIO_LOCK is not set,
   then the netcdf-c library will take control of the incoming memory.
   This means that the original memory block should not be freed
   but the block returned by nc_close_mem() must be freed.
2. If nc_open_memio() is called with NC_WRITE, and NC_MEMIO_LOCK is set,
   then modifications to the original memory may fail if the space available
   is insufficient.

Documentation is provided in the file docs/inmemory.md.
A test case is provided: nc_test/tst_inmemory.c driven by
nc_test/run_inmemory.sh

WARNING: changes were made to the dispatch table for
the close entry. From int (*close)(int) to int (*close)(int,void*).
2018-02-25 21:45:31 -07:00
Ed Hartnett
4de61e21f2 more docs, more cleaning 2017-12-04 12:21:14 -07:00
Dennis Heimbigner
3db4f013bf Primary change: add dap4 support
Specific changes:
1. Add dap4 code: libdap4 and dap4_test.
   Note that until the d4ts server problem is solved, dap4 is turned off.
2. Modify various files to support dap4 flags:
	configure.ac, Makefile.am, CMakeLists.txt, etc.
3. Add nc_test/test_common.sh. This centralizes
   the handling of the locations of various
   things in the build tree: e.g. where is
   ncgen.exe located. See nc_test/test_common.sh
   for details.
4. Modify .sh files to use test_common.sh
5. Obsolete separate oc2 by moving it to be part of
   netcdf-c. This means replacing code with netcdf-c
   equivalents.
5. Add --with-testserver to configure.ac to allow
   override of the servers to be used for --enable-dap-remote-tests.
6. There were multiple versions of nctypealignment code. Try to
   centralize in libdispatch/doffset.c and include/ncoffsets.h
7. Add a unit test for the ncuri code because of its complexity.
8. Move the findserver code out of libdispatch and into
   a separate, self contained program in ncdap_test and dap4_test.
9. Move the dispatch header files (nc{3,4}dispatch.h) to
   .../include because they are now shared by modules.
10. Revamp the handling of TOPSRCDIR and TOPBUILDDIR for shell scripts.
11. Make use of MREMAP if available
12. Misc. minor changes e.g.
	- #include <config.h> -> #include "config.h"
	- Add some no-install headers to /include
	- extern -> EXTERNL and vice versa as needed
	- misc header cleanup
	- clean up checking for misc. unix vs microsoft functions
13. Change copyright decls in some files to point to LICENSE file.
14. Add notes to RELEASENOTES.md
2017-03-08 17:01:10 -07:00
Dennis Heimbigner
0cf1e2c49f re: Github issue netcdf-c 300
Modified provenance code to allocate the minimal space
needed for _NCProperties attribute in file.  Basically
required using malloc in the provenance code and in ncdump.
Otherwise should cause no externally visible effects.
Also removed the ENABLE_FILEINFO from configure.ac since
the provenance code is no longer optional.
2016-08-08 09:24:19 -06:00
Dennis Heimbigner
11a259ad86 Add provenance info for netcdf-4 files.
This consists of a persistent attribute named
_NCProperties plus two computed attributes
_IsNetcdf4 and _SuperblockVersion.
See the 'Provenance Attributes' section
of docs/attribute_conventions.md for details.
2016-05-07 14:32:07 -06:00
Quincey Koziol
b2dfacbcfa Big clean up to type handling in libsrc4, which makes fill-values work
correctly for variables with string datatype, plus a few other minor changes.
2014-02-11 17:12:08 -06:00
Ward Fisher
ddf3c31bb0 Corrected a handful of syntax issues in CMake config files,
probably introduced more.  

Added CMake-related files to Makefile.am files for inclusion
when creating a distribution package.
2013-02-20 23:28:28 +00:00
Ed Hartnett
d58c18c623 added diskless files API, subsetting not working, classic model only 2011-08-16 21:04:33 +00:00
Ed Hartnett
e7b3fa19c1 removed mention of pnetcdf and hdf4 link options as they are handled by configure 2011-05-31 15:03:31 +00:00
Ed Hartnett
8d99e0b43a more changes for mingw/DLL builds 2011-05-11 19:51:11 +00:00
Ed Hartnett
2f4cf5382e took out ldflags in Makefile.am files 2011-05-11 17:09:12 +00:00
Dennis Heimbigner
0b477e29cc rebuilt dap constraints 2011-04-16 20:56:36 +00:00
Ed Hartnett
0259975e26 changes in response to user feedback to rc1, plus cleaned up H5 test dependancy 2011-03-21 15:45:17 +00:00
Ed Hartnett
d334706ffe added math library 2011-03-18 12:20:13 +00:00
Ed Hartnett
965a3aac70 minor refactor of the build system to work better for cross-compiling 2011-03-15 10:19:08 +00:00
Ed Hartnett
b9358f5890 updated (and fixed) shared library version numbers for all netcdf libraries 2011-02-17 12:58:08 +00:00
Ed Hartnett
668ed2e0a5 fixed chunking bug: default chunks must always be under 4GB in size 2011-02-02 14:09:15 +00:00
Ed Hartnett
cdd254c983 added -lm to please fedore netcdf package maintainer 2011-01-31 11:40:56 +00:00
Ed Hartnett
03f63a5f1c many changes for memory fixes 2010-11-29 22:23:16 +00:00
Ed Hartnett
a658960034 added more valgrind testing 2010-10-13 22:53:25 +00:00
Ed Hartnett
c9033810e1 added many valgrind tests in libsrc4, fixed some memory problems 2010-10-12 09:12:08 +00:00
Ed Hartnett
d8cdc2085d more testing of HDF5 vlen type 2010-10-11 14:52:29 +00:00
Ed Hartnett
54297ac014 changed way that netcdf-4 scans HDF5 file on open for big performance boost 2010-09-22 13:38:15 +00:00
Dennis Heimbigner
6edff3f964 cleanup pre-dispatch stuff 2010-07-30 22:16:15 +00:00
Ed Hartnett
f1bea72f66 fixed position of test in list, which matters for extra-test builds, and fixed its error handling 2010-06-29 13:03:46 +00:00
Ed Hartnett
f0a72cbefb fixed broken classic-only build 2010-06-28 17:17:43 +00:00
Ed Hartnett
4220dca07a broke out test into libsrc/tst_interops6.c to help find memory problem 2010-06-25 14:51:16 +00:00
Ed Hartnett
663e344099 changes 2010-06-23 18:43:18 +00:00
Ed Hartnett
9da0b67627 fixed some automake issues to get distcheck working 2010-06-23 17:48:10 +00:00
Ed Hartnett
f6447ee94c fixed extra-test libsrc4/tst_h_vl2 problem and valgrind test problems 2010-06-23 13:57:33 +00:00
Ed Hartnett
bb57cf2e4b moved functions out of m4 into c files in libdispatch/netcdf.m4 2010-06-18 14:01:51 +00:00
Ed Hartnett
0b54b49991 moved some tests which used v2 API from libsrc4 to nc_test4, to allow the library to be built before the tests 2010-06-15 15:24:28 +00:00
Ed Hartnett
552151d61a fix libnetcdf2 problems with enable-c-only build in libsrc4 and libncdap4 2010-06-14 18:41:52 +00:00
Ed Hartnett
57d743dddb fixed build problems for libcf/gridspec, pnetcdf, and hdf4 2010-06-10 17:18:48 +00:00
Ed Hartnett
bb0035c95d fixed Makefile.am includes to find new netcdf.h location 2010-06-07 15:40:31 +00:00
Ed Hartnett
1ba136e4cd fixed makefile problems to allow make dist to work 2010-06-04 12:23:10 +00:00
Ed Hartnett
cdabf7bbf4 build system clean-up 2010-06-03 20:33:02 +00:00
Ed Hartnett
92ccf1c5fa moved headers to include directory 2010-06-03 20:22:55 +00:00
Ed Hartnett
18f4bca367 moving to trunk subdir 2010-06-03 13:24:43 +00:00