The reference manual states that the offset parameter of H5Soffset_simple()
can be set to NULL to reset the offset of a simple dataspace to 0. This
has never been true, and passing NULL was regarded as an error.
The library will now accept NULL for the offset parameter and will
correctly set the offset to zero.
Fixes HDFFV-9299
The datatype conversion code previously used IDs for the source and
destination datatypes rather than pointers to the internal structures
for those datatypes. This was mostly due to the need for an ID for these
datatypes that can be passed to an application-registered datatype
conversion function or datatype conversion exception function. However,
using IDs internally caused a lot of unnecessary ID lookups and hurt
performance of datatype conversions in general. This was especially
problematic for compound datatype conversions, where the ID lookups were
occuring on every member of every compound element of a dataset. The
code has now been refactored to use pointers internally and only create
IDs for datatypes when necessary.
Fixed a test issue in dt_arith where a library datatype conversion
function was being cast to an application conversion function. Since the
two have different prototypes, this started failing after the parameters
for a library conversion function changed from hid_t to H5T_t * and an
extra parameter was added. This appears to have worked coincidentally in
the past since the only different between a library conversion function
and application conversion function was an extra DXPL parameter at the
end of an application conversion function
Fixed an issue where memory wasn't being freed in the h5fc_chk_idx test
program. Even though the program exits quickly after allocating the
memory, it still causes failures when testing with -fsanitize=address
When clearing out datatype conversion paths involving variable-length or reference datatypes
on file close, also check for these datatypes inside compound or array datatypes
* Fixed asserts due to H5Pset_est_link_info() values
If large values for est_num_entries and/or est_name_len were passed
to H5Pset_est_link_info(), the library would attempt to create an
object header NIL message to reserve enough space to hold the links in
compact form (i.e., concatenated), which could exceed allowable object
header message size limits and trip asserts in the library.
This bug only occurred when using the HDF5 1.8 file format or later and
required the product of the two values to be ~64k more than the size
of any links written to the group, which would cause the library to
write out a too-large NIL spacer message to reserve the space for the
unwritten links.
The library now inspects the phase change values to see if the dataset
is likely to be compact and checks the size to ensure any NIL spacer
messages won't be larger than the library allows.
Fixes GitHub #1632
* Fix copy-paste comments
- Close/reopen file and file objects to prevent cache from being used instead of actual I/O.
- Moved vlen io test datasets under the dset container group instead of the root group
- Moved the PASSED() invocation to after individual test cleanup in case an error occurs during H5Treclaim
Compound fill values were set to the integer -1, causing valgrind
to flag 'uninitialized bytes' errors.
This is just a problem with the cmpd_dset test and not a core
library problem.
H5Tset_fields did not account for any offset in a floating-point datatype,
causing it to fail when a datatype's precision is correctly set such that
it doesn't include the offset bits.
Fixes an issue where the HDF5_USE_FILE_LOCKING environment variable being
set can interfere with the file locking setting that the test expects to
be returned.
H5F_get_access_plist previously did not copy over the file locking settings
from a file into the new File Access Property List that it creates. This would
make it difficult to match the file locking settings between an external file
and its parent file.
Remove cached datatype conversion path table entries on file close
When performing datatype conversions during I/O, the library
checks to see whether it can re-use a cached datatype conversion
pathway by performing comparisons between the source and destination
datatypes of the current operation and the source and destination
datatypes associated with each cached datatype conversion pathway.
For variable-length and reference datatypes, a comparison is made
between the VOL object for the file associated with these datatypes,
which may change as a file is closed and reopened. In workflows
involving a loop that opens a file, performs I/O on an object with a
variable-length or reference datatype and then closes the file, this
can lead to constant memory usage growth as the library compares the
file VOL objects between the datatypes as different and adds a new
cached conversion pathway entry on each iteration during I/O. This is
now fixed by clearing out any cached conversion pathway entries for
variable-length or reference datatypes associated with a particular
file when that file is closed.
off_t is a 32-bit signed value on Windows, so we should use HDoff_t
(which is __int64 on Windows) internally instead.
Also defines HDftell on Windows to be _ftelli64().
* Add 'warning density' computation to the warnhist script, along with several
cleanups to it. Add "--enable-show-all-warnings" configure (and CMake)
option to disable compiler diagnostic suppression (and therefore show all the
otherwise suppressed compiler diagnostics), disabled by default. Clean up
a buncn of misc. warnings.
Signed-off-by: Quincey Koziol <qkoziol@amazon.com>
* Fix issue with HDF5_VOL_ALLOW_EXTERNAL CMake variable
* Add initial API test workflow
* Initialize parallel testing with MPI_THREAD_MULTIPLE when testing API
* Add CMake variable to allow specifying a VOL connector's package name
* Remove call to MPI_Init in serial API tests
While previously necessary, it now interferes with VOL connectors that
may need to be initialized with MPI_THREAD_MULTIPLE
* HDF5 API test updates
Removed test duplication from bringing API tests
back into the library from external VOL tests
repo
Synced changes between API tests and library's
tests
Updated API tests CMake code to directly use and
install testhdf5, testphdf5, etc. instead of
creating duplicate binaries
Added new h5_using_native_vol() test function to
determine whether the VOL connector being used
is (or the VOL connector stack being used resolves
to) the native VOL connector
* Remove duplicate variable
Allow H5Pset_evict_on_close to be called regardless of whether a parallel build of HDF5 is being used
Fail during file opens if H5Pset_evict_on_close has been set to true on the given File Access Property List and the size of the MPI communicator being used is greater than 1
Some tests in tfile.c use h5_fileaccess to get a VFD-dependent file
name but use the scheme from testhdf5, reusing the FILE1 and FILE8
names. This leads to files like test1.h5.h5 which are unintended
and not cleaned up.
This changes the filename scheme for a few tests to work with h5test,
resulting in more informative names and allowing the files to
be cleaned up at the end of the test. The test files have also
been added to the `make clean` target for the Autotools.
This function allows the user to determine if the library performed selection I/O, vector I/O, or scalar (legacy) I/O during the last HDF5 operation performed with the provided DXPL. Expanded existing tests to check this functionality.
When the Subfiling VFD is enabled, the links test may
try to initialize the Subfiling VFD and call MPI_Init_thread.
On Theta, this appears to have an issue that will cause
the links test to fail. Reworked the test to check for
the same conditions in a more roundabout way that doesn't
involved initializing the Subfiling VFD
lseek on Windows uses __int64 for both the offset and return type
instead of off_t like most POSIX systems. This changes ensures we
use HDoff_t (which is typdef'd correctly on Windows) w/ lseek.
* Strip HD prefix from string/char C API calls
* HD(f)(put|get)(s|c)
* HDstr*
* HDv*printf
* HD(s)(print|scan)f
* HDperror
But NOT:
* HDstrcase*
* HDvasprintf
* HDstrtok_r
* HDstrndup
As those are not C99 and have portability work-around
implementations. They will be handled later.
* Fix th5_system.c screwup
* Fix for the bug exposed from running test/set_extent.c when selection I/O is enabled.
This is a fix from Neil.
The test/set_extent.c is modified to test for selection I/O enabled.
* Fixed extra semi warning by adjusting alternative macro definitions
* Find-replace H5E_END_TRY; -> H5E_END_TRY
* Made H5Epush_goto a do-while loop, fixed indentation
* Made GOTOERROR and ERRMSG do-while loops
* Made Hgoto_error and Hgoto_done do-while loops
* Made vrfy_cint_type and vrfy_ctype do-while loops
* Made TEST_TYPE_CONTIG and others do-while loops
* Removed extraneous semi-colons
* Committing clang-format changes
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Update files to skip list and ignore_words_list for codespell to not
check files generated by autotools. Autotools generate misspellings
that can't be fixed in HDF5 code.
- Implemented support for AWS temporary security credentials.
For this kind of credentials also a session/security token
should be included in the request by adding the
x-amz-security-token header.
Co-authored-by: Larry Knox <lrknox@hdfgroup.org>
Co-authored-by: Jordan Henderson <jhenderson@hdfgroup.org>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Replace support.hdfgroup.org URLs for alternative COPYING file
locations in copyright headers with https://www.hdfgroup.org/licenses.
Replace support.hdfgroup.org URL for alternative COPYING_LBNL_HDF5
with github URL.
Tweak chkcopyright script for change from UICOPYRIGHTSTR to
THGCOPYRIGHTSTR.
These are meaningless noise.
Removes the "Programmer" lines on comment start lines:
/* Programmer: John Smith
These complicate my sed script that will rip out the rest of
the comments.
These calls are non-C99 but protected by ifdefs and have no Windows
equivalents:
* HDalarm
* HDasprintf
* HDclock_gettime
* HDfcntl
* HDgethostname
* HDgetrusage
* HDsymlink
* Fixed various -Wmissing-variable-declarations by adding static keyword
* In a few cases, renamed the variable suffix from _g to _s.
* Fixed some -Wmissing-variable-declarations by using different declaration macros
* Fixed various -Wconditional-uninitialized warnings by just initializing variable to zero
* Fixed various -Wcomma warnings
* Fixed clang -Wstrict-prototypes warnings
* Fixed various -Wunused-variable warnings
* Updated some casts to fix the only 3 -Wcast-qual warnings
* Fixed the only -Wsometimes-uninitialized warning
* Adds semicolons to function-like macros
* Adds a do..while(0) loop to some macros
* Removes semicolons when inappropriate, especially H5E_TRY_BEGIN/END
Several options and public symbols that were provided to paper
over non-standard long double conversions between signed/unsigned
long and long long values were removed from the Autotools and
CMake. These were added twenty years ago, when C99 and 64-bit
platforms were less common and are no longer needed.
Autotools:
--enable-dconv-accuracy
CMake:
HDF5_WANT_DATA_ACCURACY
H5pubconf.h symbols:
H5_WANT_DATA_ACCURACY
H5_LDOUBLE_TO_LONG_SPECIAL
H5_LONG_TO_LDOUBLE_SPECIAL
H5_LDOUBLE_TO_LLONG_ACCURATE
H5_LLONG_TO_LDOUBLE_CORRECT
H5_DISABLE_SOME_LDOUBLE_CONV
The C99 standard says that a long long is 8+ bytes. This change
removes a few parts of the code where we assume that a long long
could be less than that.
When copying an empty enum type (including implicitly, as when an enum
is contained in a compound type), the library would allocate 0-size
blocks of memory and attempt to memcpy 0 bytes from NULL pointers, which
are undefined behavior. In debug mode, the library would raise an
assert in H5MM.
The library now avoid undefined memory operations when copying empty
enum types and a test that copies empty enums has been added.
Several macros for handling haddr_t values exist in H5Fprivate.h
and have H5F prefixes, even though they have nothing to do with
a particular file (e.g., H5F_addr_lt()).
These macros have been moved to H5private.h and renamed to have
an H5 prefix.
Affected macros:
H5F_addr_overflow
H5F_addr_defined
H5F_addr_eq
H5F_addr_ne
H5F_addr_lt
H5F_addr_le
H5F_addr_gt
H5F_addr_ge
H5F_addr_cmp
H5F_addr_overlap
H5F_addr_pow2 was unused and removed instead of converted.
* Remove macros from api tests (#2929)
* Remove macros and undefined callbacks (#2959)
* Remove remaining macros from H5_api_tests_disabled.h (#2968)
* Put some vol capability checks in testpar tests and remove remaining warnings (#2995)
* API tests datatype generation cleanup
* Clean up API tests' random datatype generation and fix bug with enum
datatype generation
* Init parallel API tests with MPI_THREAD_MULTIPLE
* HDF5 API tests - Check VOL connector registration
* Determine whether a VOL connector failed to load before running API
tests
* Cleanup some usages of H5VL_CAP_FLAG_CREATION_ORDER in API tests
* Remove some now-unused macros from H5_api_tests_disabled.h
* Enable HDF5 API tests by default
* Implement CMake option to install HDF5 API tests
* Check for invalid AAPL from H5Acreate
* Enable building of VOL connectors alongside HDF5 in CMake
* Prepend CMake VOL URL option indices with 0s so they come in order
* Don't turn on API tests by default yet
* Document VOL connector FetchContent functionality
* Add release note for API test updates
* Only install testing library if API tests are installed
* Fix grammar
* Move functions into more focused source code modules, along with a small #
of directly secondary effects. No actual changes to the contents of any
moved routines.
Signed-off-by: Quincey Koziol <quincey@koziol.cc>
* Committing clang-format changes
---------
Signed-off-by: Quincey Koziol <quincey@koziol.cc>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
* Add failure value where it's missing from 1+ macros. Clean up
whitespace / continuation characters ('\'). Made hash-table macros generic
for use in both the package header and test header. Remove duplicated
copy & pasted macros (by hoisting difference into #ifdef'd macro).
Updated and re-flowed comments to read better.
Also clean up a few compiler warnings in production builds.
Signed-off-by: Quincey Koziol <quincey@koziol.cc>
* Committing clang-format changes
* Remove unused variable warning in H5C.c (#2844)
* Remove trailing /* NDEBUG */ comment from #endif's
* Committing clang-format changes
---------
Signed-off-by: Quincey Koziol <quincey@koziol.cc>
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Initial implementation of selection I/O with type conversion. Allows
Parallel collective I/O with type conversion, as long as selection I/O
is enabled.
* Jira issue OESS-337: Adding a test case for H5VLconnector_info_to_str, H5VLconnector_str_to_info, and H5VLconnector_free_info. The test may need to change after the possible changes of the parameters of the API functions.
* Correct concurrency bugs when running tests, along with a bugfix & small
warning cleanup.
* Committing clang-format changes
* Allow spaces (and tabs) in VOL connector info string from environment variable.
* Parse connector name from HDF5_PLUGIN_PATH environment variable better
* Correct H5VLquery_optional to use H5VL routine instead of H5I. Also add an
error message to the failure return value from not finding a plugin.
* Play nice with existing plugin paths
* Use API routine to determine if native connector is terminal.
* Committing clang-format changes
* Make string size larger, to allow for connectors with longer names.
* Be more flexible about testing external pass through connectors, especially if
they have registered new optional operations.
* Dead code removal
* Committing clang-format changes
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: AWS ParallelCluster user <ec2-user@ip-10-0-0-65.us-east-2.compute.internal>
Co-authored-by: Koziol <qkoziol@88665a374c70.ant.amazon.com>
* HDFFV-10368 (OESS-319): Move two useful macros (H5L_EXT_FLAGS_ALL and H5L_EXT_VERSION) from H5Lpkg.h to H5Lpublic.h for public usage. The test in vol.c only envokes H5Lcreate_external using the passthru VOL. To test if the macros are public, one can use them in the link creation of the passthru VOL, such as printing out their values.
* Committing clang-format changes
* Moved the test for the external link public macros (H5L_EXT_FLAGS_ALL and H5L_EXT_VERSION) from vol.c to links.c.
* Committing clang-format changes
* Minor change: changed a pointer variable to an integer.
* Committing clang-format changes
* Moved two macros (H5L_EXT_VERSION and H5L_EXT_FLAGS_ALL) to H5Ldevelop.h for only developers to use.
---------
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
szip (or libaec) is currently not tested in CI. This adds szip to the
the Autotools GitHub CI actions on Linux when building with the
Autotools.
This PR also cleans up a few warnings that remained in the szip-
related code so the -Werror check will pass.
