mirror of
https://github.com/HDFGroup/hdf5.git
synced 2025-01-30 15:32:37 +08:00
9a433b99a5
Lots of performance improvements & a couple new internal API interfaces.
Description:
Performance Improvements:
- Cached file offset & length sizes in shared file struct, to avoid
constantly looking them up in the FCPL.
- Generic property improvements:
- Added "revision" number to generic property classes to speed
up comparisons.
- Changed method of storing properties from using a hash-table
to the TBBT routines in the library.
- Share the propery names between classes and the lists derived
from them.
- Removed redundant 'def_value' buffer from each property.
- Switching code to use a "copy on write" strategy for
properties in each list, where the properties in each list
are shared with the properties in the class, until a
property's value is changed in a list.
- Fixed error in layout code which was allocating too many buffers.
- Redefined public macros of the form (H5open()/H5check, <variable>)
internally to only be (<variable>), avoiding innumerable useless
calls to H5open() and H5check_version().
- Reuse already zeroed buffers in H5F_contig_fill instead of
constantly re-zeroing them.
- Don't write fill values if writing entire dataset.
- Use gettimeofday() system call instead of time() system when
checking the modification time of a dataset.
- Added reference counted string API and use it for tracking the
names of objects opening in a file (for the ID->name code).
- Removed redundant H5P_get() calls in B-tree routines.
- Redefine H5T datatype macros internally to the library, to avoid
calling H5check redundantly.
- Keep dataspace information for dataset locally instead of reading
from disk each time. Added new module to track open objects
in a file, to allow this (which will be useful eventually for
some FPH5 metadata caching issues).
- Remove H5AC_find macro which was inlining metadata cache lookups,
and call function instead.
- Remove redundant memset() calls from H5G_namei() routine.
- Remove redundant checking of object type when locating objects
in metadata cache and rely on the address only.
- Create default dataset object to use when default dataset creation
property list is used to create datasets, bypassing querying
for all the property list values.
- Use default I/O vector size when performing raw data with the
default dataset transfer property list, instead of querying for
I/O vector size.
- Remove H5P_DEFAULT internally to the library, replacing it with
more specific default property list based on the type of
property list needed.
- Remove redundant memset() calls in object header message (H5O*)
routines.
- Remove redunant memset() calls in data I/O routines.
- Split free-list allocation routines into malloc() and calloc()-
like routines, instead of one combined routine.
- Remove lots of indirection in H5O*() routines.
- Simplify metadata cache entry comparison routine (used when
flushing entire cache out).
- Only enable metadata cache statistics when H5AC_DEBUG is turned
on, instead of always tracking them.
- Simplify address comparison macro (H5F_addr_eq).
- Remove redundant metadata cache entry protections during dataset
creation by protecting the object header once and making all
the modifications necessary for the dataset creation before
unprotecting it.
- Reduce # of "number of element in extent" computations performed
by computing and storing the value during dataspace creation.
- Simplify checking for group location's file information, when file
has not been involving in file-mounting operations.
- Use binary encoding for modification time, instead of ASCII.
- Hoist H5HL_peek calls (to get information in a local heap)
out of loops in many group routine.
- Use static variable for iterators of selections, instead of
dynamically allocation them each time.
- Lookup & insert new entries in one step, avoiding traversing
group's B-tree twice.
- Fixed memory leak in H5Gget_objname_idx() routine (tangential to
performance improvements, but fixed along the way).
- Use free-list for reference counted strings.
- Don't bother copying object names into cached group entries,
since they are re-created when an object is opened.
The benchmark I used to measure these results created several thousand
small (2K) datasets in a file and wrote out the data for them. This is
Elena's "regular.c" benchmark.
These changes resulted in approximately ~4.3x speedup of the
development branch when compared to the previous code in the
development branch and ~1.4x speedup compared to the release
branch.
Additionally, these changes reduce the total memory used (code and
data) by the development branch by ~800KB, bringing the development
branch back into the same ballpark as the release branch.
I'll send out a more detailed description of the benchmark results
as a followup note.
New internal API routines:
Added "reference counted strings" API for tracking strings that get
used by multiple owners without duplicating the strings.
Added "ternary search tree" API for text->object mappings.
Platforms tested:
Tested h5committest {arabica (fortran), eirene (fortran, C++)
modi4 (parallel, fortran)}
Other platforms/configurations tested?
FreeBSD 4.7 (sleipnir) serial & parallel
Solaris 2.6 (baldric) serial
159 lines
6.1 KiB
C
159 lines
6.1 KiB
C
/****************************************************************************
|
|
* NCSA HDF *
|
|
* Software Development Group *
|
|
* National Center for Supercomputing Applications *
|
|
* University of Illinois at Urbana-Champaign *
|
|
* 605 E. Springfield, Champaign IL 61820 *
|
|
* *
|
|
* For conditions of distribution and use, see the accompanying *
|
|
* hdf/COPYING file. *
|
|
* *
|
|
****************************************************************************/
|
|
|
|
/* $Id$ */
|
|
|
|
/*
|
|
* This header file contains information required for testing the HDF5 library.
|
|
*/
|
|
|
|
#ifndef HDF5TEST_H
|
|
#define HDF5TEST_H
|
|
|
|
/*
|
|
* Include required headers. This file tests internal library functions,
|
|
* so we include the private headers here.
|
|
*/
|
|
#include "H5private.h"
|
|
#include "H5Eprivate.h"
|
|
|
|
#ifndef HDF5_TEST_MASTER
|
|
extern int num_errs;
|
|
extern int Verbosity;
|
|
#endif /* HDF5_TEST_MASTER */
|
|
|
|
/* Use %ld to print the value because long should cover most cases. */
|
|
/* Used to make certain a return value _is_not_ a value */
|
|
#define CHECK(ret, val, where) do { \
|
|
if (Verbosity>9) print_func(" Call to routine: %15s at line %4d " \
|
|
"in %s returned %ld \n", \
|
|
where, (int)__LINE__, __FILE__, \
|
|
(long)ret); \
|
|
if (ret == val) { \
|
|
print_func("*** UNEXPECTED RETURN from %s is %ld at line %4d " \
|
|
"in %s\n", where, (long)ret, (int)__LINE__, __FILE__); \
|
|
num_errs++; \
|
|
H5Eprint (stdout); \
|
|
} \
|
|
H5Eclear(); \
|
|
} while(0)
|
|
|
|
#define CHECK_I(ret,where) { \
|
|
if (Verbosity>9) { \
|
|
print_func(" Call to routine: %15s at line %4d in %s returned %ld\n", \
|
|
(where), (int)__LINE__, __FILE__, (long)(ret)); \
|
|
} \
|
|
if ((ret)<0) { \
|
|
print_func ("*** UNEXPECTED RETURN from %s is %ld line %4d in %s\n", \
|
|
(where), (long)(ret), (int)__LINE__, __FILE__); \
|
|
H5Eprint (stdout); \
|
|
num_errs++; \
|
|
} \
|
|
H5Eclear (); \
|
|
}
|
|
|
|
#define CHECK_PTR(ret,where) { \
|
|
if (Verbosity>9) { \
|
|
print_func(" Call to routine: %15s at line %4d in %s returned %p\n", \
|
|
(where), (int)__LINE__, __FILE__, (ret)); \
|
|
} \
|
|
if (!(ret)) { \
|
|
print_func ("*** UNEXPECTED RETURN from %s is NULL line %4d in %s\n", \
|
|
(where), (int)__LINE__, __FILE__); \
|
|
H5Eprint (stdout); \
|
|
num_errs++; \
|
|
} \
|
|
H5Eclear (); \
|
|
}
|
|
|
|
/* Used to make certain a return value _is_ a value */
|
|
#define VERIFY(x, val, where) do { \
|
|
if (Verbosity>9) { \
|
|
print_func(" Call to routine: %15s at line %4d in %s had value " \
|
|
"%ld \n", where, (int)__LINE__, __FILE__, (long)x); \
|
|
} \
|
|
if (x != val) { \
|
|
print_func("*** UNEXPECTED VALUE from %s is %ld at line %4d " \
|
|
"in %s\n", where, (long)x, (int)__LINE__, __FILE__); \
|
|
H5Eprint (stdout); \
|
|
num_errs++; \
|
|
} \
|
|
H5Eclear(); \
|
|
} while(0)
|
|
|
|
/* Used to document process through a test and to check for errors */
|
|
#define RESULT(ret,func) do { \
|
|
if (Verbosity>8) { \
|
|
print_func(" Call to routine: %15s at line %4d in %s returned " \
|
|
"%ld\n", func, (int)__LINE__, __FILE__, (long)ret); \
|
|
} \
|
|
if (Verbosity>9) HEprint(stdout, 0); \
|
|
if (ret == FAIL) { \
|
|
print_func("*** UNEXPECTED RETURN from %s is %ld at line %4d " \
|
|
"in %s\n", func, (long)ret, (int)__LINE__, __FILE__); \
|
|
H5Eprint (stdout); \
|
|
num_errs++; \
|
|
} \
|
|
H5Eclear(); \
|
|
} while(0)
|
|
|
|
/* Used to document process through a test */
|
|
#define MESSAGE(V,A) {if (Verbosity>(V)) print_func A;}
|
|
|
|
/* definitions for command strings */
|
|
#define VERBOSITY_STR "Verbosity"
|
|
#define SKIP_STR "Skip"
|
|
#define TEST_STR "Test"
|
|
#define CLEAN_STR "Cleanup"
|
|
|
|
/* Prototypes for the support routines */
|
|
int print_func(const char *,...);
|
|
|
|
/* Prototypes for the test routines */
|
|
void test_metadata(void);
|
|
void test_tbbt(void);
|
|
void test_tst(void);
|
|
void test_refstr(void);
|
|
void test_file(void);
|
|
void test_h5t(void);
|
|
void test_h5s(void);
|
|
void test_h5d(void);
|
|
void test_attr(void);
|
|
void test_select(void);
|
|
void test_time(void);
|
|
void test_reference(void);
|
|
void test_vltypes(void);
|
|
void test_vlstrings(void);
|
|
void test_iterate(void);
|
|
void test_array(void);
|
|
void test_genprop(void);
|
|
void test_configure(void);
|
|
void test_misc(void);
|
|
|
|
/* Prototypes for the cleanup routines */
|
|
void cleanup_metadata(void);
|
|
void cleanup_file(void);
|
|
void cleanup_h5s(void);
|
|
void cleanup_attr(void);
|
|
void cleanup_select(void);
|
|
void cleanup_time(void);
|
|
void cleanup_reference(void);
|
|
void cleanup_vltypes(void);
|
|
void cleanup_vlstrings(void);
|
|
void cleanup_iterate(void);
|
|
void cleanup_array(void);
|
|
void cleanup_genprop(void);
|
|
void cleanup_configure(void);
|
|
void cleanup_misc(void);
|
|
|
|
#endif /* HDF5cleanup_H */
|