Commit Graph

5807 Commits

Author SHA1 Message Date
Ward Fisher
b4c24b1a23 Merge remote-tracking branch 'origin/newchunkdoc.dmh' into pr-aggregation.wif 2018-10-18 12:56:52 -06:00
Ward Fisher
ddd3722b41
Merge branch 'master' into newchunkdoc.dmh 2018-10-15 10:53:51 -06:00
Ward Fisher
76ff749bf2
Merge pull request #1153 from NetCDF-World-Domination-Council/ejh_hdf4_config_issues
Fix HDF4 config issues
2018-10-15 10:53:37 -06:00
Dennis Heimbigner
7e2a9a9bdf Restore the lost documentation in nccopy.1 of
the new per-variable chunking specifications.
2018-10-13 16:24:37 -06:00
Ward Fisher
16ee523a45
Merge branch 'master' into ejh_hdf4_config_issues 2018-10-12 17:20:43 -06:00
Ward Fisher
1554ecdaa9
Merge pull request #1158 from Unidata/disklesswrite.dmh
Revert/Improve nc_create + NC_DISKLESS behavior
2018-10-12 17:18:33 -06:00
Dennis Heimbigner
553de966d6 Add some visual studio bug fixes 2018-10-11 12:09:42 -06:00
Dennis Heimbigner
4636584d5b Revert/Improve nc_create + NC_DISKLESS behavior
re: https://github.com/Unidata/netcdf-c/issues/1154

Inadvertently, the behavior of NC_DISKLESS with nc_create() was
changed in release 4.6.1. Previously, the NC_WRITE flag needed
to be explicitly used with NC_DISKLESS in order to cause the
created file to be persisted to disk.

Additional analyis indicated that the current NC_DISKLESS
implementation was seriously flawed.

This PR attempts to clean up and regularize the situation with
respect to NC_DISKLESS control. One important aspect of diskless
operation is that there are two different notions of write.

1. The file is read-write vs read-only when using the netcdf API.
2. The file is persisted or not to disk at nc_close().

Previously, these two were conflated. The rules now are
as follows.

1. NC_DISKLESS + NC_WRITE means that the file is read/write using the netcdf API
2. NC_DISKLESS + NC_PERSIST means that the file is persisted to a disk file at nc_close.
3. NC_DISKLESS + NC_PERSIST + NC_WRITE means both 1 and 2.

The NC_PERSIST flag is new and takes over the obsolete NC_MPIPOSIX flag.
NC_MPIPOSIX is still defined, but is now an alias for the NC_MPIIO flag.

It is also now the case that for netcdf-4, NC_DISKLESS is independent
of NC_INMEMORY and in fact it is an error to specify both flags
simultaneously.

Finally, the MMAP code was fixed to use NC_PERSIST as well.
Also marked MMAP as deprecated.

Also added a test case to test various combinations of NC_DISKLESS,
NC_PERSIST, and NC_WRITE.

This PR affects a number of files and especially test cases
that used NC_DISKLESS.

Misc. Unrelated fixes
1. fixed some warnings in ncdump/dumplib.c
2018-10-10 13:32:17 -06:00
Ed Hartnett
8259455582
Merge branch 'master' into ejh_hdf4_config_issues 2018-10-10 06:17:27 +03:00
Ward Fisher
4a5b15bcbb
Merge pull request #1147 from wkliao/nc_mpiio_nc_mpiposix
Ignore NC_MPIIO and NC_MPIPOSIX
2018-10-09 16:21:00 -06:00
Ward Fisher
ea66842bea
Merge branch 'master' into nc_mpiio_nc_mpiposix 2018-10-09 15:21:50 -06:00
Ward Fisher
6b0217f38c
Merge pull request #1149 from Unidata/netcdf_mem_fix.dmh
Remove unneeded include of netcdf.h from netcdf_mem.h
2018-10-09 13:50:57 -06:00
Dennis Heimbigner
534364b069 This Pre removed #include <netcdf.h> from netcdf_mem.h.
In turn that caused some problems elsewhere because some files
did not include netcdf.h preceding netcdf_mem.h. Specifically,
building under visual studio fails.
So fix those cases that I can find. May need to
add more later.
2018-10-07 12:05:26 -06:00
Dennis Heimbigner
a5a34f6aba
Merge branch 'master' into nc_mpiio_nc_mpiposix 2018-10-06 13:33:55 -06:00
Dennis Heimbigner
77886deff2
Merge branch 'master' into netcdf_mem_fix.dmh 2018-10-06 13:32:57 -06:00
Ward Fisher
266d3c5997
Merge pull request #1157 from Unidata/filtermem.dmh
Fix memory problems when using HDF5 version 1.10.x and later.
2018-10-04 16:26:29 -06:00
Ward Fisher
cbcd6c616d Added new generated files to .gitignore file. 2018-10-04 15:02:02 -06:00
Ward Fisher
35ea9da7f6
Merge branch 'master' into filtermem.dmh 2018-10-04 15:01:29 -06:00
Ward Fisher
debb149a4f
Merge pull request #1141 from NetCDF-World-Domination-Council/ejh_cache_sep
Separate the HDF5 cache code
2018-10-04 12:05:39 -06:00
Dennis Heimbigner
d07c05b58f Fix memory problems when using HDF5 version 1.10.x and later.
re: issue https://github.com/Unidata/netcdf-c/issues/1156

Starting with HDF5 version 1.10.x, the plugin code MUST be
careful when using the standard *malloc()*, *realloc()*, and
*free()* function.

In the event that the code is allocating, reallocating, or
free'ing memory that either came from -- or will be exported to --
the calling HDF5 library, then one MUST use the corresponding
HDF5 functions *H5allocate_memory()*, *H5resize_memory()*,
*H5free_memory()* [5] to avoid memory failures.

Additionally, if your filter code leaks memory, then the HDF5 library
generates a failure something like this.
````
H5MM.c:232: H5MM_final_sanity_check: Assertion `0 == H5MM_curr_alloc_bytes_s' failed.
````

This PR modifies the code in the plugins directory to
conform to these new requirements.

This raises a question about the libhdf5 code where this
same problem may occur. We need to scan especially nc4hdf.c
to look for this problem.
2018-10-04 11:37:21 -06:00
Wei-keng Liao
65ca7d09b9 add NC_WRITE to LEGAL_CREATE_FLAGS for PnetCDF dispatcher 2018-10-04 11:53:39 -05:00
Ward Fisher
de3b0902fc
Merge branch 'master' into ejh_cache_sep 2018-10-03 14:14:06 -06:00
Ward Fisher
0381e1a4e6
Merge pull request #1155 from Unidata/fillmismatch.dmh
Modify DAP2 and DAP4 to optionally allow Fillvalue/Variable mismatch
2018-10-03 14:08:52 -06:00
Ward Fisher
9b02df9472 Updated travis.yml to enable filter testing as part of the travis CI tests. 2018-10-02 17:41:52 -06:00
Dennis Heimbigner
26da2fae01 Remove debug output 2018-10-02 11:43:11 -06:00
Wei-keng Liao
cfcf93eda3 add missing option --has_parallel4 2018-10-02 01:58:21 -05:00
Wei-keng Liao
82d7ae2a93 revise comments about collective I/O mode 2018-10-02 01:19:44 -05:00
Wei-keng Liao
08629267e4 add default setting in configure help strings 2018-10-02 01:18:51 -05:00
Dennis Heimbigner
dd654a272e Remove debug from tst_filter.sh 2018-10-01 15:54:25 -06:00
Dennis Heimbigner
8072d1f6bb Modify DAP2 and DAP4 to optionally allow Fillvalue/Variable mismatch
re: issue https://github.com/Unidata/netcdf-c/issues/1151

Modify DAP2 and DAP4 code to handle case when _FillValue type is not
same as the parent variable type.

Specifically:
1. Define a parameter [fillmismatch] to allow this mismatch;
   default is to disallow.
2. If allowed, forcibly change the type of the _FillValue to match
   the parent variable.
3. If allowed Convert the values to match new type
4. Generate a log message
5. if not allowed, then fail

Implementing this required some changes to ncdap_test/dapcvt.c
Also added test cases.

Minor Unrelated Changes:
1. There were a number of warnings about e.g.
   assigning a const char* to a char*. Fix these
2. In nccopy.1, replace .NP with .IP "n"
   (re PR https://github.com/Unidata/netcdf-c/pull/1144)
3. fix minor error in ncdump/ocprint
2018-10-01 15:51:43 -06:00
Ed
8cd4f89aa4 fixing curl.h problem a bit 2018-09-25 04:17:56 -06:00
Ed
faac874399 now ensure that --disable-netcdf was used when building hdf4, also check for jpeg library with HDF4 builds 2018-09-24 23:53:05 -06:00
Dennis Heimbigner
bba2a676a7
Remove unneeded include of netcdf.h from netcdf_mem.h 2018-09-23 14:16:45 -06:00
Wei-keng Liao
a7c08c3a49 add a missed update to NC_HDF4_open 2018-09-23 01:36:50 -05:00
Wei-keng Liao
0ed70756cc Ignore flags NC_MPIIO and NC_MPIPOSIX. 2018-09-22 20:22:34 -05:00
Wei-keng Liao
50e5b4fe8b remove unused files, test_get_p5.m4 is covered by test_get.m4 2018-09-22 18:21:55 -05:00
Wei-keng Liao
01da14da6f if cmode does not contain NC_NETCDF4, then it is for PnetCDF 2018-09-22 18:21:45 -05:00
Wei-keng Liao
bc63c08f41 add a check for MPI_Info_f2c 2018-09-21 17:32:36 -05:00
Wei-keng Liao
740a50d1f2 add more descriptions for parallel APIs 2018-09-20 12:59:19 -05:00
Wei-keng Liao
0853b3c9ee Merge branch 'pnetcdf_driver' of github.com:wkliao/netcdf-c into pnetcdf_driver 2018-09-20 11:50:27 -05:00
Wei-keng Liao
5ed6c239e1 More replacement of "parallel-netcdf" with "PnetCDF". 2018-09-20 11:45:25 -05:00
Ed Hartnett
cc62140f1d
Merge branch 'master' into ejh_cache_sep 2018-09-18 16:59:49 -06:00
Ward Fisher
5b5311ef80
Merge branch 'master' into pnetcdf_driver 2018-09-18 12:21:11 -06:00
Ward Fisher
18a05d0ed6
Merge pull request #1129 from NetCDF-World-Domination-Council/ejh_hdf5_sep_next_2
Fix benchmark programs
2018-09-18 12:20:59 -06:00
Wei-keng Liao
28512f77f0 remove unused variables 2018-09-18 01:27:57 -05:00
Wei-keng Liao
83608143ee more checks of NC_ENOTBUILT 2018-09-17 17:18:48 -05:00
Wei-keng Liao
5d297238d5 fix nc_test/tst_addvar.c. This program reads output file
tst_pnetcdf.nc generated from tst_pnetcdf.c.
2018-09-17 17:18:48 -05:00
Wei-keng Liao
48da78e133 Use PnetCDF instead of parallel-netcdf to avoid confusion with
parallel netcdf4. Also, update PnetCDF web page.
2018-09-17 17:18:48 -05:00
Wei-keng Liao
64e2982d0f adjust test logics for NC_ENOTBUILT 2018-09-17 17:18:48 -05:00
Wei-keng Liao
d24f292077 return NC_ENOTBUILT when CDF5 is not enabled but NC_64BIT_DATA is used 2018-09-17 17:18:48 -05:00