mirror/netcdf-c
2
0
Fork 0
mirror of https://github.com/Unidata/netcdf-c.git synced 2025-01-06 15:34:44 +08:00
Code Issues Packages Projects Releases Wiki Activity
6,866 Commits 90 Branches 75 Tags 88 MiB
1a665b6d80
Commit Graph

415 Commits

Author SHA1 Message Date
Ed Hartnett
1eb7e7a8e8 added comment describing netcdf-4 behavior in data mode dim renames with longer names 2019-02-25 06:36:39 -07:00
Dennis Heimbigner
0c59e13bf7 Master merge, conflict resolution, cleanup 2019-02-24 16:54:13 -07:00
Dennis Heimbigner
959c213c18 conflict resolution 2019-02-23 22:17:53 -07:00
Dennis Heimbigner
45a8a265b8 master merge 2019-02-23 17:14:12 -07:00
Ed Hartnett
cfa8e3808f cleanup of whitespace in HDF5 directory 2019-02-19 05:19:37 -07:00
Ed Hartnett
b1d30a0f67 cleanup of whitespace in HDF5 directory 2019-02-19 05:18:53 -07:00
Ed Hartnett
c771443e43 cleanup of whitespace in HDF5 directory 2019-02-19 05:18:25 -07:00
Ed Hartnett
8b1f5a8fad cleanup of whitespace in HDF5 directory 2019-02-19 05:18:02 -07:00
Ed Hartnett
384a6f1303 cleanup of whitespace in HDF5 directory 2019-02-19 05:17:47 -07:00
Ward Fisher
1fde39c8d7
Merge branch 'master' into byterange.dmh 2019-02-07 14:28:23 -07:00
Ed Hartnett
70201adb51 comment cleanup 2019-02-03 07:43:56 -07:00
Ed Hartnett
5d908a0bbb now preserve order of varids after a var rename 2019-02-03 06:56:03 -07:00
Ed Hartnett
8e6f38b099 detecting conditions for mandatory rename of vars with varid > renamed var 2019-02-03 06:35:29 -07:00
Ed Hartnett
e30a2bf208 added come comments 2019-02-02 07:32:31 -07:00
Ed Hartnett
8acde75e3c converted hdf5gtp.c to use H5Lmove instead of deprecated H5Gmove 2019-02-02 07:24:02 -07:00
Ed Hartnett
f25f050be8 converted hdf5var to use H5Lmove instead of deprecated H5Gmove 2019-02-02 07:20:14 -07:00
Ed Hartnett
1dd76c996e added tst_rename3.c for more rename testing 2019-02-02 05:53:45 -07:00
Ed Hartnett
828304ed41 now using secret hdf5 var name during renames if needed 2019-01-27 11:33:06 -07:00
Ed Hartnett
1b3f397c4c added name parameter to give_var_secret_name to base secret name on 2019-01-27 11:29:49 -07:00
Ed Hartnett
784dc0e0ad now creating secret hdf5 name on var rename, if needed 2019-01-27 11:17:57 -07:00
Ed Hartnett
660bda1be3 made function give_var_secret_name() static again 2019-01-27 11:14:29 -07:00
Ed Hartnett
e74ec6f2a0 made function give_var_secret_name() not static, fixed warning 2019-01-27 11:10:41 -07:00
Ed Hartnett
c95887cc53 removed name param from function give_var_secret_name() 2019-01-27 11:07:57 -07:00
Ed Hartnett
627a55cf78 added function give_var_secret_name() 2019-01-27 11:06:02 -07:00
Ed Hartnett
42c64598dc created function create_dim_wo_var() 2019-01-27 10:51:25 -07:00
Ward Fisher
b27c7d899d Merge branch 'master' into byterange.dmh 2019-01-25 14:50:23 -07:00
Ed Hartnett
a89f9ddeb9
Merge branch 'master' into ejh_rename_bug 2019-01-25 07:05:13 -07:00
Ed Hartnett
66cc9c5020 fixed rename bug 2019-01-24 10:20:46 -07:00
Ward Fisher
237f0c6e65 Merge branch 'ejh_tidy' of https://github.com/NetCDF-World-Domination-Council/netcdf-c into pr-aggregation.wif 2019-01-23 15:11:59 -07:00
Ward Fisher
cb2affbef5 Merge branch 'patch-32' of https://github.com/gsjaardema/netcdf-c into pr-aggregation.wif 2019-01-23 15:09:35 -07:00
Ed Hartnett
840d51d035 changed NC_GRP_INFO_T to use atts_read instead of atts_not_read 2019-01-22 08:11:52 -07:00
Ed Hartnett
243cef8fa5 changed var atts_not_read to atts_read 2019-01-21 08:40:04 -07:00
Ed Hartnett
60132a0ed7 made function static, removed unneeded if statement 2019-01-20 09:53:00 -07:00
Ed Hartnett
9d40e0a2af made function static, removed unneeded if statement 2019-01-20 09:52:42 -07:00
Ed Hartnett
281f67da6e removed unneeded vars, fixed and added comments 2019-01-20 09:46:15 -07:00
Ed Hartnett
adb3356aff merged ejh_test_pnetcdf, fixes broken logging statement, added comments 2019-01-20 09:42:18 -07:00
Ed Hartnett
c6a9948a8e removed unneeded var, fixed broken log statements that cause segfaults 2019-01-20 09:37:13 -07:00
Ed Hartnett
15e6a782db removed unneeded variable, shortened function name 2019-01-20 09:25:04 -07:00
Ed Hartnett
e1cd4018c5 removed unneeded variable 2019-01-20 09:18:14 -07:00
Greg Sjaardema
1ab53924cb
Tests on equalp are always true 
If `equalp` is NULL, then the function returns early, so all subsequent tests on `equalp` are not needed.
 2019-01-04 17:39:48 -07:00
Dennis Heimbigner
ac2e6f9a10 typo5 2019-01-02 21:37:31 -07:00
Dennis Heimbigner
8cd450206a typeo4 2019-01-02 20:53:44 -07:00
Dennis Heimbigner
dc491f9bf9 typos3 2019-01-02 20:31:06 -07:00
Dennis Heimbigner
fe3ba0904c Another typo (sigh\!) 2019-01-02 16:48:11 -07:00
Dennis Heimbigner
6e76972c99 Fix typo 2019-01-02 16:35:22 -07:00
Dennis Heimbigner
a7fa2d8d95 It turns out the the type H5FD_class_t was changed 
between HDF5 versions 1.8 and 1.10.
So modify H5FDhttp.c to be conditional on the
HDF5 major+minor version from H5public.h
 2019-01-02 14:37:23 -07:00
Dennis Heimbigner
84c2bc0d78 Merge branch 'master' into byterange.dmh 2019-01-02 13:18:45 -07:00
Dennis Heimbigner
bf2746b8ea Provide byte-range reading of remote datasets 
re: issue https://github.com/Unidata/netcdf-c/issues/1251

Assume that you have the URL to a remote dataset
which is a normal netcdf-3 or netcdf-4 file.

This PR allows the netcdf-c to read that dataset's
contents as a netcdf file using HTTP byte ranges
if the remote server supports byte-range access.

Originally, this PR was set up to access Amazon S3 objects,
but it can also access other remote datasets such as those
provided by a Thredds server via the HTTPServer access protocol.
It may also work for other kinds of servers.

Note that this is not intended as a true production
capability because, as is known, this kind of access to
can be quite slow. In addition, the byte-range IO drivers
do not currently do any sort of optimization or caching.

An additional goal here is to gain some experience with
the Amazon S3 REST protocol.

This architecture and its use documented in
the file docs/byterange.dox.

There are currently two test cases:

1. nc_test/tst_s3raw.c - this does a simple open, check format, close cycle
   for a remote netcdf-3 file and a remote netcdf-4 file.
2. nc_test/test_s3raw.sh - this uses ncdump to investigate some remote
   datasets.

This PR also incorporates significantly changed model inference code
(see the superceded PR https://github.com/Unidata/netcdf-c/pull/1259).

1. It centralizes the code that infers the dispatcher.
2. It adds support for byte-range URLs

Other changes:

1. NC_HDF5_finalize was not being properly called by nc_finalize().
2. Fix minor bug in ncgen3.l
3. fix memory leak in nc4info.c
4. add code to walk the .daprc triples and to replace protocol=
   fragment tag with a more general mode= tag.

Final Note:
Th inference code is still way too complicated. We need to move
to the validfile() model used by netcdf Java, where each
dispatcher is asked if it can process the file. This decentralizes
the inference code. This will be done after all the major new
dispatchers (PIO, Zarr, etc) have been implemented.
 2019-01-01 18:27:36 -07:00
Ed Hartnett
029aa5f626 now using hidden coordinates att to speed file opens 2018-12-20 05:59:31 -07:00
Ed Hartnett
7249350c3f now remember whether coords att has been read for a var 2018-12-19 09:43:32 -07:00
Ed Hartnett
6e3d284fcc write coordinates hidden attribute for all variables 2018-12-19 09:16:21 -07:00
Ed Hartnett
f5c7209838 comment and code cleanup 2018-12-19 09:10:15 -07:00
Ed Hartnett
070214f81a more comments, code cleanup 2018-12-19 06:52:26 -07:00
Ed Hartnett
25184f3843 moved code to get_attached_info() 2018-12-18 09:16:03 -07:00
Ed Hartnett
1b38d9aef8 lazy read of some var metadata 2018-12-18 07:48:22 -07:00
Ed Hartnett
e9bd94821d split out some var meta reading, fixed setting of var->container 2018-12-17 08:41:43 -07:00
Ed Hartnett
77b3a81d86 starting to deal with var metadata separately 2018-12-17 08:25:19 -07:00
Ed Hartnett
8ca5a1ac17
Merge branch 'master' into ejh_fast_var_prep 2018-12-12 07:05:45 -07:00
Ward Fisher
30ea33435c Merge remote-tracking branch 'origin/license_update.wif' into pr-aggregation.wif 2018-12-11 17:08:21 -05:00
Ward Fisher
50fa6b4f32 Merge branch 'ejh_next_22' of https://github.com/NetCDF-World-Domination-Council/netcdf-c into pr-aggregation.wif 2018-12-11 17:06:43 -05:00
Ward Fisher
dedc8e7dde Merge branch 'ejh_next_20' of https://github.com/NetCDF-World-Domination-Council/netcdf-c into pr-aggregation.wif 2018-12-11 17:06:32 -05:00
Ward Fisher
6deb77bade Merge branch 'master' into gh1207.dmh 2018-12-11 16:44:04 -05:00
Ed Hartnett
aaa5a50ca9 cleanup of hdf5open.c, now check for COORDINATES hidden att and use it to find dimids if available 2018-12-11 09:57:08 -07:00
Ed Hartnett
dc1115ae76 moved rec_match_dimscales() to hdf5open.c, made it faster by skipping already-identified dims 2018-12-11 09:40:59 -07:00
Ed Hartnett
28aa22f8cf cleanup and some comments for rec_write_metadata() 2018-12-11 09:33:46 -07:00
Ed Hartnett
8e1d781ea3 fixed error handling in write_dim(), also converted dimscale creation to H5Dcreate2() 2018-12-11 09:28:23 -07:00
Ed Hartnett
1a0cfb729e fixed error handling in write_var() 2018-12-11 08:10:08 -07:00
Ed Hartnett
26239a0897 cleaned up loop reattaching dimscales 2018-12-11 07:21:09 -07:00
Ed Hartnett
7b72d0b832 fixed error handling in attach_dimscales() 2018-12-11 07:15:46 -07:00
Ed Hartnett
39a0f822fa changed loop to use ncindexlookup() instead of looping through the names 2018-12-11 07:07:00 -07:00
Ed Hartnett
d7b657af4e cleaned up comments, changed loop to use ncindexlookup() instead of looping through the names 2018-12-11 06:43:27 -07:00
Ed Hartnett
53bdb74869 fixed error handling of write_netcdf4_dimid() 2018-12-11 06:38:23 -07:00
Ed Hartnett
ef5a39e19a cleaned up hdf5internal.c 2018-12-11 06:22:48 -07:00
Ed Hartnett
3c9a141ee3 moved function detect_preserve_dimids and made it static 2018-12-11 06:15:47 -07:00
Ward Fisher
87110139f6 More synchronization of copyright stanzas. 2018-12-06 14:34:40 -07:00
Ed Hartnett
f6443bce8f rest of separation of libhdf5 and libsrc4 2018-11-30 14:05:11 -07:00
Ed Hartnett
b5ed407e9f better handling of normalizing names in HDF5 atts 2018-11-30 13:28:18 -07:00
Ed Hartnett
d0587c9536 more name normalization 2018-11-30 09:14:53 -07:00
Ed Hartnett
77d0922d49 starting to deal with normalized name in HDF5 attribute code 2018-11-30 08:59:58 -07:00
Ed Hartnett
433499771b moved special att reading function to libhdf5 2018-11-30 07:50:15 -07:00
Ed Hartnett
104b4b50fe clean up 2018-11-29 06:25:34 -07:00
Ed Hartnett
d51b221c62 move setting of var chunk cache to libhdf5 2018-11-29 06:10:39 -07:00
Ed Hartnett
c824ad9ec9 further removal of libhdf5 code from libsrc4 2018-11-28 15:48:08 -07:00
Ed Hartnett
f9d3fff2c3 rest of removal of lazy att code from libsrc4 2018-11-26 11:11:56 -07:00
Ed Hartnett
016f675fad more separation of lazy att reads from libsrc4 2018-11-26 10:44:48 -07:00
Ed Hartnett
c56b1cecd4 moving lazy att code to libhdf5 2018-11-26 10:18:24 -07:00
Ed Hartnett
c710b42707 moving lazy att code to libhdf5 2018-11-26 10:14:03 -07:00
Ed Hartnett
cc18944fa7 moved lazy atts handling for nc_inq_attid() 2018-11-26 09:58:31 -07:00
Ed Hartnett
d97824d3e6 moved lazy atts handling for nc_inq_att() 2018-11-26 09:51:28 -07:00
Ed Hartnett
1f64c66cdf rename HDF5 dispatch functions to start with NC4_HDF5 2018-11-26 08:13:57 -07:00
Ed Hartnett
aade08ee22 moving checking for lazy att reads to libhdf5 2018-11-26 07:49:58 -07:00
Ed Hartnett
8bb644204e added hdf5dispatch.c to cmake build 2018-11-26 06:00:38 -07:00
Ed Hartnett
0e9784a867 more changes to separate HDF5 from libsrc4 initialization 2018-11-26 05:44:59 -07:00
Ed Hartnett
6f969b0117 separate HDF5 initialization 2018-11-26 05:35:44 -07:00
Ed Hartnett
64b45b5e75 changes in support of separating HDF5 and libsrc4 2018-11-26 04:40:33 -07:00
Ed Hartnett
f6c3093fb3 moved convert function to libsrc4 2018-11-21 14:49:52 -07:00
Ed Hartnett
26db90e4e3 changes 2018-11-21 07:39:05 -07:00
Ed Hartnett
c1ffb92a7f checking return value for HDF5 function call 2018-11-20 15:43:34 -07:00
Ed Hartnett
ab963e3d41 removing HDF5 type info from libsrc4 2018-11-20 14:24:40 -07:00
Ed Hartnett
9aedbd0c41 changing over native_hdf_typeid 2018-11-20 10:55:45 -07:00