# The Data Type Interface (H5T)

## 1. Introduction

The data type interface provides a mechanism to describe the storage format of individual data points of a data set and is hopefully designed in such a way as to allow new features to be easily added without disrupting applications that use the data type interface. A dataset (the H5D interface) is composed of a collection or raw data points of homogeneous type organized according to the data space (the H5S interface).

A data type is a collection of data type properties, all of which can be stored on disk, and which when taken as a whole, provide complete information for data conversion to or from that data type. The interface provides functions to set and query properties of a data type.

A data point is an instance of a data type, which is an instance of a type class. We have defined a set of type classes and properties which can be extended at a later time. The atomic type classes are those which describe types which cannot be decomposed at the data type interface level; all other classes are compound.

## 2. General Data Type Operations

The functions defined in this section operate on data types as a whole. New data types can be created from scratch or copied from existing data types. When a data type is no longer needed its resources should be released by calling `H5Tclose()`.

Data types come in two flavors: named data types and transient data types. A named data type is stored in a file while the transient flavor is independent of any file. Named data types are always read-only, but transient types come in three varieties: modifiable, read-only, and immutable. The difference between read-only and immutable types is that immutable types cannot be closed except when the entire library is closed (the predefined types like `H5T_NATIVE_INT` are immutable transient types).

`hid_t H5Tcreate (H5T_class_t class, size_t size)`
  Data types can be created by calling this function, where class is a data type class identifier. However, the only class currently allowed is `H5T_COMPOUND` to create a new empty compound data type where size is the total size in bytes of an instance of this data type. Other data types are created with `H5Tcopy()`. All functions that return data type identifiers return a negative value for failure.
`hid_t H5Topen (hid_t location, const char *name)`
  A named data type can be opened by calling this function, which returns a handle to the data type. The handle should eventually be closed by calling `H5Tclose()` to release resources. The named data type returned by this function is read-only or a negative value is returned for failure. The location is either a file or group handle.
`herr_t H5Tcommit (hid_t location, const char *name, hid_t type)`
  A transient data type (not immutable) can be committed to a file and turned into a named data type by calling this function. The location is either a file or group handle and when combined with name refers to a new named data type.
`hbool_t H5Tcommitted (hid_t type)`
  A type can be queried to determine if it is a named type or a transient type. If this function returns a positive value then the type is named (that is, it has been committed perhaps by some other application). Datasets which return committed data types with `H5Dget_type()` are able to share the data type with other datasets in the same file.
`hid_t H5Tcopy (hid_t type)`
  This function returns a modifiable transient data type which is a copy of type or a negative value for failure. If type is a dataset handle then the type returned is a modifiable transient copy of the data type of the specified dataset.
`herr_t H5Tclose (hid_t type)`
  Releases resources associated with a data type. The data type identifier should not be subsequently used since the results would be unpredictable. It is illegal to close an immutable transient data type.
`hbool_t H5Tequal (hid_t type1, hid_t type2)`
  Determines if two types are equal. If type1 and type2 are the same then this function returns `TRUE`, otherwise it returns `FALSE` (an error results in a negative return value).
`herr_t H5Tlock (hid_t type)`
  A transient data type can be locked, making it immutable (read-only and not closable). The library does this to all predefined types to prevent the application from inadvertently modifying or deleting (closing) them, but the application is also allowed to do this for its own data types. Immutable data types are closed when the library closes (either by `H5close()` or by normal program termination).

## 3. Properties of Atomic Types

An atomic type is a type which cannot be decomposed into smaller units at the API level. All atomic types have a common set of properties which are augmented by properties specific to a particular type class. Some of these properties also apply to compound data types, but we discuss them only as they apply to atomic data types here. The properties and the functions that query and set their values are:

`H5T_class_t H5Tget_class (hid_t type)`
  This property holds one of the class names: `H5T_INTEGER, H5T_FLOAT, H5T_TIME, H5T_STRING, H5T_BITFIELD`, or `H5T_OPAQUE`. This property is read-only and is set when the datatype is created or copied (see `H5Tcreate()`, `H5Tcopy()`). If this function fails it returns `H5T_NO_CLASS` which has a negative value (all other class constants are non-negative).
`size_t H5Tget_size (hid_t type)`
`herr_t H5Tset_size (hid_t type, size_t size)`
  This property is total size of the datum in bytes, including padding which may appear on either side of the actual value. If this property is reset to a smaller value which would cause the significant part of the data to extend beyond the edge of the data type then the `offset` property is decremented a bit at a time. If the offset reaches zero and the significant part of the data still extends beyond the edge of the data type then the `precision` property is decremented a bit at a time. Decreasing the size of a data type may fail if the precesion must be decremented and the data type is of the `H5T_OPAQUE` class or the `H5T_FLOAT` bit fields would extend beyond the significant part of the type. Adjusting the size of an `H5T_STRING` automatically adjusts the precision as well. On error, `H5Tget_size()` returns zero which is never a valid size.
`H5T_order_t H5Tget_order (hid_t type)`
`herr_t H5Tset_order (hid_t type, H5T_order_t order)`
  All atomic data types have a byte order which describes how the bytes of the data type are layed out in memory. If the lowest memory address contains the least significant byte of the datum then it is said to be little-endian or `H5T_ORDER_LE`. If the bytes are in the oposite order then they are said to be big-endian or `H5T_ORDER_BE`. Some data types have the same byte order on all machines and are `H5T_ORDER_NONE` (like character strings). If `H5Tget_order()` fails then it returns `H5T_ORDER_ERROR` which is a negative value (all successful return values are non-negative).
`size_t H5Tget_precision (hid_t type)`
`herr_t H5Tset_precision (hid_t type, size_t precision)`
  Some data types occupy more bytes than what is needed to store the value. For instance, a `short` on a Cray is 32 significant bits in an eight-byte field. The `precision` property identifies the number of significant bits of a datatype and the `offset` property (defined below) identifies its location. The `size` property defined above represents the entire size (in bytes) of the data type. If the precision is decreased then padding bits are inserted on the MSB side of the significant bits (this will fail for `H5T_FLOAT` types if it results in the sign, mantissa, or exponent bit field extending beyond the edge of the significant bit field). On the other hand, if the precision is increased so that it "hangs over" the edge of the total size then the `offset` property is decremented a bit at a time. If the `offset` reaches zero and the significant bits still hang over the edge, then the total size is increased a byte at a time. The precision of an `H5T_STRING` is read-only and is always eight times the value returned by `H5Tget_size()`. `H5Tget_precision()` returns zero on failure since zero is never a valid precision.
`size_t H5Tget_offset (hid_t type)`
`herr_t H5Tset_offset (hid_t type, size_t offset)`
  While the `precision` property defines the number of significant bits, the `offset` property defines the location of those bits within the entire datum. The bits of the entire data are numbered beginning at zero at the least significant bit of the least significant byte (the byte at the lowest memory address for a little-endian type or the byte at the highest address for a big-endian type). The `offset` property defines the bit location of the least signficant bit of a bit field whose length is `precision`. If the offset is increased so the significant bits "hang over" the edge of the datum, then the `size` property is automatically incremented. The offset is a read-only property of an `H5T_STRING` and is always zero. `H5Tget_offset()` returns zero on failure which is also a valid offset, but is guaranteed to succeed if a call to `H5Tget_precision()` succeeds with the same arguments.
`herr_t H5Tget_pad (hid_t type, H5T_pad_t *lsb, H5T_pad_t *msb)`
`herr_t H5Tset_pad (hid_t type, H5T_pad_t lsb, H5T_pad_t msb)`
  The bits of a datum which are not significant as defined by the `precision` and `offset` properties are called padding. Padding falls into two categories: padding in the low-numbered bits is lsb padding and padding in the high-numbered bits is msb padding (bits are numbered according to the description for the `offset` property). Padding bits can always be set to zero (`H5T_PAD_ZERO`) or always set to one (`H5T_PAD_ONE`). The current pad types are returned through arguments of `H5Tget_pad()` either of which may be null pointers.

### 3.1. Properties of Integer Atomic Types

Integer atomic types (`class=H5T_INTEGER`) describe integer number formats. Such types include the following information which describes the type completely and allows conversion between various integer atomic types.

`H5T_sign_t H5Tget_sign (hid_t type)`
`herr_t H5Tset_sign (hid_t type, H5T_sign_t sign)`
  Integer data can be signed two's complement (`H5T_SGN_2`) or unsigned (`H5T_SGN_NONE`). Whether data is signed or not becomes important when converting between two integer data types of differing sizes as it determines how values are truncated and sign extended.

### 3.2. Properties of Floating-point Atomic Types

The library supports floating-point atomic types (`class=H5T_FLOAT`) as long as the bits of the exponent are contiguous and stored as a biased positive number, the bits of the mantissa are contiguous and stored as a positive magnitude, and a sign bit exists which is set for negative values. Properties specific to floating-point types are:

`herr_t H5Tget_fields (hid_t type, size_t *spos, size_t *epos, size_t *esize, size_t *mpos, size_t *msize)`
`herr_t H5Tset_fields (hid_t type, size_t spos, size_t epos, size_t esize, size_t mpos, size_t msize)`
  A floating-point datum has bit fields which are the exponent and mantissa as well as a mantissa sign bit. These properties define the location (bit position of least significant bit of the field) and size (in bits) of each field. The bit positions are numbered beginning at zero at the beginning of the significant part of the datum (see the descriptions of the `precision` and `offset` properties). The sign bit is always of length one and none of the fields are allowed to overlap. When expanding a floating-point type one should set the precision first; when decreasing the size one should set the field positions and sizes first.
`size_t H5Tget_ebias (hid_t type)`
`herr_t H5Tset_ebias (hid_t type, size_t ebias)`
  The exponent is stored as a non-negative value which is `ebias` larger than the true exponent. `H5Tget_ebias()` returns zero on failure which is also a valid exponent bias, but the function is guaranteed to succeed if `H5Tget_precision()` succeeds when called with the same arguments.
`H5T_norm_t H5Tget_norm (hid_t type)`
`herr_t H5Tset_norm (hid_t type, H5T_norm_t norm)`
  This property determines the normalization method of the mantissa.
  - If the value is `H5T_NORM_MSBSET` then the mantissa is shifted left (if non-zero) until the first bit after the radix point is set and the exponent is adjusted accordingly. All bits of the mantissa after the radix point are stored.
  - If its value is `H5T_NORM_IMPLIED` then the mantissa is shifted left (if non-zero) until the first bit after the radix point is set and the exponent is adjusted accordingly. The first bit after the radix point is not stored since it's always set.
  - If its value is `H5T_NORM_NONE` then the fractional part of the mantissa is stored without normalizing it.
`H5T_pad_t H5Tget_inpad (hid_t type)`
`herr_t H5Tset_inpad (hid_t type, H5T_pad_t inpad)`
  If any internal bits (that is, bits between the sign bit, the mantissa field, and the exponent field but within the precision field) are unused, then they will be filled according to the value of this property. The inpad argument can be `H5T_PAD_ZERO` if the internal padding should always be set to zero, or `H5T_PAD_ONE` if it should always be set to one. `H5Tget_inpad()` returns `H5T_PAD_ERROR` on failure which is a negative value (successful return is always non-negative).

### 3.3. Properties of Date and Time Atomic Types

Dates and times (`class=H5T_TIME`) are stored as character strings in one of the ISO-8601 formats like "1997-12-05 16:25:30"; as character strings using the Unix asctime(3) format like "Thu Dec 05 16:25:30 1997"; as an integer value by juxtaposition of the year, month, and day-of-month, hour, minute and second in decimal like 19971205162530; as an integer value in Unix time(2) format; or other variations.

I'm deferring definition until later since they're probably not as important as the other data types.

### 3.4. Properties of Character String Atomic Types

Fixed-length character string types are used to store textual information. The `offset` property of a string is always zero and the `precision` property is eight times as large as the value returned by `H5Tget_size()` (since precision is measured in bits while size is measured in bytes). Both properties are read-only.

`H5T_cset_t H5Tget_cset (hid_t type)`
`herr_t H5Tset_cset (hid_t type, H5T_cset_t cset)`
  HDF5 is able to distinguish between character sets of different nationalities and to convert between them to the extent possible. The only character set currently supported is `H5T_CSET_ASCII`.
`H5T_str_t H5Tget_strpad (hid_t type)`
`herr_t H5Tset_strpad (hid_t type, H5T_str_t strpad)`
  The method used to store character strings differs with the programming language: C usually null terminates strings while Fortran left-justifies and space-pads strings. This property defines the storage mechanism and can be

  `H5T_STR_NULLTERM`
    A C-style string which is guaranteed to be null terminated. When converting from a longer string the value will be truncated and then a null character appended.
  `H5T_STR_NULLPAD`
    A C-style string which is padded with null characters but not necessarily null terminated. Conversion from a long string to a shorter `H5T_STR_NULLPAD` string will truncate but not null terminate. Conversion from a short value to a longer value will append null characters as with `H5T_STR_NULLTERM`.
  `H5T_STR_SPACEPAD`
    A Fortran-style string which is padded with space characters. This is the same as `H5T_STR_NULLPAD` except the padding character is a space instead of a null.

  `H5Tget_strpad()` returns `H5T_STR_ERROR` on failure, a negative value (all successful return values are non-negative).

### 3.5. Properties of Bit Field Atomic Types

Converting a bit field (`class=H5T_BITFIELD`) from one type to another simply copies the significant bits. If the destination is smaller than the source then bits are truncated. Otherwise new bits are filled according to the `msb` padding type.

### 3.6. Properties of Opaque Atomic Types

Opaque atomic types (`class=H5T_OPAQUE`) act like bit fields except conversions which change the precision are not allowed. However, padding can be added or removed from either end and the bytes can be reordered. Opaque types can be used to create novel data types not directly supported by the library, but the application is responsible for data conversion of these types.

## 4. Properties of Compound Types

A compound data type is similar to a `struct` in C or a common block in Fortran: it is a collection of one or more atomic types or small arrays of such types. Each member of a compound type has a name which is unique within that type, and a byte offset that determines the first byte (smallest byte address) of that member in a compound datum. A compound data type has the following properties:

`H5T_class_t H5Tget_class (hid_t type)`
  All compound data types belong to the type class `H5T_COMPOUND`. This property is read-only and is defined when a data type is created or copied (see `H5Tcreate()` or `H5Tcopy()`).
`size_t H5Tget_size (hid_t type)`
  Compound data types have a total size in bytes which is returned by this function. All members of a compound data type must exist within this size. A value of zero is returned for failure; all successful return values are positive.
`int H5Tget_nmembers (hid_t type)`
  A compound data type consists of zero or more members (defined in any order) with unique names and which occupy non-overlapping regions within the datum. In the functions that follow, individual members are referenced by an index number between zero and N-1, inclusive, where N is the value returned by this function. `H5Tget_nmembers()` returns -1 on failure.
`char *H5Tget_member_name (hid_t type, int membno)`
  Each member has a name which is unique among its siblings in a compound data type. This function returns a pointer to a null-terminated copy of the name allocated with `malloc()` or the null pointer on failure. The caller is responsible for freeing the memory returned by this function.
`size_t H5Tget_member_offset (hid_t type, int membno)`
  The byte offset of member number membno with respect to the beginning of the containing compound datum is returned by this function. A zero is returned on failure which is also a valid offset, but this function is guaranteed to succeed if a call to `H5Tget_member_dims()` succeeds when called with the same type and membno arguments.
`int H5Tget_member_dims (hid_t type, int membno, int dims[4], int perm[4])`
  Each member can be a small array of up to four dimensions, making it convenient to describe things like transposition matrices. The dimensionality of the member is returned (or negative for failure) and the size in each dimension is returned through the dims argument. The perm argument describes how the array's elements are mapped to the linear address space of memory with respect to some reference order (the reference order is specified in natural language documentation which describes the compound data type). The application which "invented" the type will often use the identity permutation and other applications will use a permutation that causes the elements to be rearranged to the desired order. Only the first few elements of dims and perm are initialized according to the dimensionality of the member. Scalar members have dimensionality zero. The only permutations supported at this time are the identity permutation and the transpose permutation (in the 4d case, {0,1,2,3} and {3,2,1,0}).
`hid_t H5Tget_member_type (hid_t type, int membno)`
  Each member has its own data type, a copy of which is returned by this function. The returned data type identifier should be released by eventually calling `H5Tclose()` on that type.

Properties of members of a compound data type are defined when the member is added to the compound type (see `H5Tinsert()`) and cannot be subsequently modified. This makes it imposible to define recursive data structures.

## 5. Predefined Atomic Data Types

The library predefines a modest number of data types having names like `H5T_arch_base` where arch is an architecture name and base is a programming type name. New types can be derived from the predifined types by copying the predefined type (see `H5Tcopy()`) and then modifying the result.

Architecture Name | Description
--- | ---
`IEEE` | This architecture defines standard floating point types in various byte orders.
`STD` | This is an architecture that contains semi-standard datatypes like signed two's complement integers, unsigned integers, and bitfields in various byte orders.
`UNIX` | Types which are specific to Unix operating systems are defined in this architecture. The only type currently defined is the Unix date and time types (`time_t`).
`C FORTRAN` | Types which are specific to the C or Fortran programming languages are defined in these architectures. For instance, `H5T_C_STRING` defines a base string type with null termination which can be used to derive string types of other lengths.
`NATIVE` | This architecture contains C-like data types for the machine on which the library was compiled. The types were actually defined by running the `H5detect` program when the library was compiled. In order to be portable, applications should almost always use this architecture to describe things in memory.
`CRAY` | Cray architectures. These are word-addressable, big-endian systems with non-IEEE floating point.
`INTEL` | All Intel and compatible CPU's including 80286, 80386, 80486, Pentium, Pentium-Pro, and Pentium-II. These are little-endian systems with IEEE floating-point.
`MIPS` | All MIPS CPU's commonly used in SGI systems. These are big-endian systems with IEEE floating-point.
`ALPHA` | All DEC Alpha CPU's, little-endian systems with IEEE floating-point.

The base name of most types consists of a letter, a precision in bits, and an indication of the byte order. The letters are:

B | Bitfield
D | Date and time
F | Floating point
I | Signed integer
R | References
S | Character string
U | Unsigned integer

The byte order is a two-letter sequence:

BE | Big endian
LE | Little endian
VX | Vax order

Example | Description
--- | ---
`H5T_IEEE_F64LE` | Eight-byte, little-endian, IEEE floating-point
`H5T_IEEE_F32BE` | Four-byte, big-endian, IEEE floating point
`H5T_STD_I32LE` | Four-byte, little-endian, signed two's complement integer
`H5T_STD_U16BE` | Two-byte, big-endian, unsigned integer
`H5T_UNIX_D32LE` | Four-byte, little-endian, time_t
`H5T_C_S1` | One-byte, null-terminated string of eight-bit characters
`H5T_INTEL_B64` | Eight-byte bit field on an Intel CPU
`H5T_CRAY_F64` | Eight-byte Cray floating point
`H5T_STD_ROBJ` | Reference to an entire object in a file

The `NATIVE` architecture has base names which don't follow the same rules as the others. Instead, native type names are similar to the C type names. Here are some examples:

Example | Corresponding C Type
--- | ---
`H5T_NATIVE_CHAR` | `signed char`
`H5T_NATIVE_UCHAR` | `unsigned char`
`H5T_NATIVE_SHORT` | `short`
`H5T_NATIVE_USHORT` | `unsigned short`
`H5T_NATIVE_INT` | `int`
`H5T_NATIVE_UINT` | `unsigned`
`H5T_NATIVE_LONG` | `long`
`H5T_NATIVE_ULONG` | `unsigned long`
`H5T_NATIVE_LLONG` | `long long`
`H5T_NATIVE_ULLONG` | `unsigned long long`
`H5T_NATIVE_FLOAT` | `float`
`H5T_NATIVE_DOUBLE` | `double`
`H5T_NATIVE_LDOUBLE` | `long double`
`H5T_NATIVE_HSIZE` | `hsize_t`
`H5T_NATIVE_HSSIZE` | `hssize_t`
`H5T_NATIVE_HERR` | `herr_t`
`H5T_NATIVE_HBOOL` | `hbool_t`

#### Example: A 128-bit integer

To create a 128-bit, little-endian signed integer type one could use the following (increasing the precision of a type automatically increases the total size):

``

    hid_t new_type = H5Tcopy (H5T_NATIVE_INT);
    H5Tset_precision (new_type, 128);
    H5Tset_order (new_type, H5T_ORDER_LE);

#### Example: An 80-character string

To create an 80-byte null terminated string type one might do this (the offset of a character string is always zero and the precision is adjusted automatically to match the size):

``

    hid_t str80 = H5Tcopy (H5T_C_S1);
    H5Tset_size (str80, 80);

## 6. Defining Compound Data Types

Unlike atomic data types which are derived from other atomic data types, compound data types are created from scratch. First, one creates an empty compound data type and specifies it's total size. Then members are added to the compound data type in any order.

Usually a C struct will be defined to hold a data point in memory, and the offsets of the members in memory will be the offsets of the struct members from the beginning of an instance of the struct.

`HOFFSET(s,m)`
  This macro computes the offset of member m within a struct s.
`offsetof(s,m)`
  This macro defined in `stddef.h` does exactly the same thing as the `HOFFSET()` macro.

Each member must have a descriptive name which is the key used to uniquely identify the member within the compound data type. A member name in an HDF5 data type does not necessarily have to be the same as the name of the member in the C struct, although this is often the case. Nor does one need to define all members of the C struct in the HDF5 compound data type (or vice versa).

#### Example: A simple struct

An HDF5 data type is created to describe complex numbers whose type is defined by the `complex_t` struct.

``

    typedef struct {
       double re;   /*real part*/
       double im;   /*imaginary part*/
    } complex_t;

    hid_t complex_id = H5Tcreate (H5T_COMPOUND, sizeof tmp);
    H5Tinsert (complex_id, "real", HOFFSET(complex_t,re),
               H5T_NATIVE_DOUBLE);
    H5Tinsert (complex_id, "imaginary", HOFFSET(complex_t,im),
               H5T_NATIVE_DOUBLE);

Member alignment is handled by the `HOFFSET` macro. However, data stored on disk does not require alignment, so unaligned versions of compound data structures can be created to improve space efficiency on disk. These unaligned compound data types can be created by computing offsets by hand to eliminate inter-member padding, or the members can be packed by calling `H5Tpack()` (which modifies a data type directly, so it is usually preceded by a call to `H5Tcopy()`):

#### Example: A packed struct

This example shows how to create a disk version of a compound data type in order to store data on disk in as compact a form as possible. Packed compound data types should generally not be used to describe memory as they may violate alignment constraints for the architecture being used. Note also that using a packed data type for disk storage may involve a higher data conversion cost.

``

    hid_t complex_disk_id = H5Tcopy (complex_id);
    H5Tpack (complex_disk_id);

#### Example: A flattened struct

Compound data types that have a compound data type member can be handled two ways. This example shows that the compound data type can be flattened, resulting in a compound type with only atomic members.

``

    typedef struct {
       complex_t x;
       complex_t y;
    } surf_t;

    hid_t surf_id = H5Tcreate (H5T_COMPOUND, sizeof tmp);
    H5Tinsert (surf_id, "x-re", HOFFSET(surf_t,x.re),
               H5T_NATIVE_DOUBLE);
    H5Tinsert (surf_id, "x-im", HOFFSET(surf_t,x.im),
               H5T_NATIVE_DOUBLE);
    H5Tinsert (surf_id, "y-re", HOFFSET(surf_t,y.re),
               H5T_NATIVE_DOUBLE);
    H5Tinsert (surf_id, "y-im", HOFFSET(surf_t,y.im),
               H5T_NATIVE_DOUBLE);

#### Example: A nested struct

However, when the `complex_t` is used often it becomes inconvenient to list its members over and over again. So the alternative approach to flattening is to define a compound data type and then use it as the type of the compound members, as is done here (the typedefs are defined in the previous examples).

``

    hid_t complex_id, surf_id; /*hdf5 data types*/

    complex_id = H5Tcreate (H5T_COMPOUND, sizeof c);
    H5Tinsert (complex_id, "re", HOFFSET(complex_t,re),
               H5T_NATIVE_DOUBLE);
    H5Tinsert (complex_id, "im", HOFFSET(complex_t,im),
               H5T_NATIVE_DOUBLE);

    surf_id = H5Tcreate (H5T_COMPOUND, sizeof s);
    H5Tinsert (surf_id, "x", HOFFSET(surf_t,x), complex_id);
    H5Tinsert (surf_id, "y", HOFFSET(surf_t,y), complex_id);

## 7. Sharing Data Types among Datasets

If a file has lots of datasets which have a common data type then the file could be made smaller by having all the datasets share a single data type. Instead of storing a copy of the data type in each dataset object header, a single data type is stored and the object headers point to it. The space savings is probably only significant for datasets with a compound data type since the simple data types can be described with just a few bytes anyway.

To create a bunch of datasets that share a single data type just create the datasets with a committed (named) data type.

#### Example: Shared Types

To create two datasets that share a common data type one just commits the data type, giving it a name, and then uses that data type to create the datasets.

``

    hid_t t1 = ...some transient type...;
    H5Tcommit (file, "shared_type", t1);
    hid_t dset1 = H5Dcreate (file, "dset1", t1, space, H5P_DEFAULT);
    hid_t dset2 = H5Dcreate (file, "dset2", t1, space, H5P_DEFAULT);

And to create two additional datasets later which share the same type as the first two datasets:

``

    hid_t dset1 = H5Dopen (file, "dset1");
    hid_t t2 = H5Dget_type (dset1);
    hid_t dset3 = H5Dcreate (file, "dset3", t2, space, H5P_DEFAULT);
    hid_t dset4 = H5Dcreate (file, "dset4", t2, space, H5P_DEFAULT);

## 8. Data Conversion

The library is capable of converting data from one type to another and does so automatically when reading or writing the raw data of a dataset. The data type interface does not provide functions to the application for changing data types directly, but the user is allowed a certain amount of control over the conversion process.

In order to insure that data conversion exceeds disk I/O rates, common data conversion paths can be hand-tuned and optimized for performance. If a hand-tuned conversion function is not available, then the library falls back to a slower but more general conversion function. Although conversion paths include data space conversion, only data type conversions are described here. Most applications will not be concerned with data type conversions since the library will contain hand-tuned conversion functions for many common conversion paths. In fact, if an application does define a conversion function which would be of general interest, we request that the function be submitted to the HDF5 development team for inclusion in the library (there might be less overhead involved with calling an internal conversion functions than calling an application-defined conversion function).

Note: The HDF5 library contains a deliberately limited set of conversion routines. It can convert from one integer format to another, from one floating point format to another, and from one struct to another. It can also perform byte swapping when the source and destination types are otherwise the same. The library does not contain any functions for converting data between integer and floating point formats. It is anticipated that some users will find it necessary to develop float to integer or integer to float conversion functions at the application level; if they wish, users are invited to submit those functions to be considered for inclusion in future versions of the library.

A conversion path contains a source and destination data type and each path contains a hard conversion function and/or a soft conversion function. The only difference between hard and soft functions is the way in which the library chooses which function applies: A hard function applies to a specific conversion path while a soft function may apply to multiple paths. When both hard and soft functions apply to a conversion path, then the hard function is favored and when multiple soft functions apply, the one defined last is favored.

A data conversion function is of type `H5T_conv_t which is defined as:`

``

    typedef herr_t (*H5T_conv_t)(hid_t src_type,
                                 hid_t dest_type,
    			     H5T_cdata_t *cdata,
    			     size_t nelmts,
    			     void *buffer,
                                 void *background);

The conversion function is called with the source and destination data types (src_type and dst_type), path-constant data (cdata), the number of instances of the data type to convert (nelmts), a buffer which initially contains an array of data having the source type and on return will contain an array of data having the destination type (buffer), and a temporary or background buffer (background). Functions return a negative value on failure and some other value on success.

The `command` field of the cdata argument determines what happens within the conversion function. It's values can be:

`H5T_CONV_INIT`
  This command is to hard conversion functions when they're registered or soft conversion functions when the library is determining if a conversion can be used for a particular path. The src_type and dst_type are the end-points of the path being queried and cdata is all zero. The library should examine the source and destination types and return zero if the conversion is possible and negative otherwise (hard conversions need not do this since they've presumably been registered only on paths they support). If the conversion is possible the library may allocate and initialize private data and assign the pointer to the `priv` field of cdata (or private data can be initialized later). It should also initialize the `need_bkg` field described below. The buf and background pointers will be null pointers.
`H5T_CONV_CONV`
  This is the usually command which indicates that data points should be converted. The conversion function should initialize the `priv` field of cdata if it wasn't initialize during the `H5T_CONV_INIT` command and then convert nelmts instances of the src_type to the dst_type. The buffer serves as both input and output. The background buffer is supplied according to the value of the `need_bkg` field of cdata (the values are described below).
`H5T_CONV_FREE`
  The conversion function is about to be removed from some path and the private data (the `cdata->priv` pointer) should be freed and set to null. All other pointer arguments are null and the nelmts argument is zero.
Others...
  Other commands might be implemented later and conversion functions that don't support those commands should return a negative value.

Whether a background buffer is supplied to a conversion function, and whether the background buffer is initialized depends on the value of `cdata->need_bkg` which the conversion function should have initialized during the H5T_CONV_INIT command. It can have one of these values:

`H5T_BKG_NONE`
  No background buffer will be supplied to the conversion function. This is the default.
`H5T_BKG_TEMP`
  A background buffer will be supplied but it will not be initialized. This is useful for those functions requiring some extra buffer space as the buffer can probably be allocated more efficiently by the library (the application can supply the buffer as part of the dataset transfer property list).
`H5T_BKG_YES`
  An initialized background buffer is passed to the conversion function. The buffer is initialized with the current values of the destination for the data which is passed in through the buffer argument. It can be used to "fill in between the cracks". For instance, if the destination type is a compound data type and we are initializing only part of the compound data type from the source type then the background buffer can be used to initialize the other part of the destination.

Other fields of cdata can be read or written by the conversion functions. Many of these contain performance-measuring fields which can be printed by the conversion function during the `H5T_CONV_FREE` command which is issued whenever the function is removed from a conversion path.

`hbool_t recalc`
  This field is set by the library when any other data type conversion function is registered or unregistered. It allows conversion functions to cache pointers to other conversion functions and be notified when the cache should be recalculated.
`unsigned long ncalls`
  This field contains the number of times the conversion function was called with the command `H5T_CONV_CONV`. It is updated automatically by the library.
`unsigned long nelmts`
  This is the total number of data points converted by this function and is updated automatically by the library.

Once a conversion function is written it can be registered and unregistered with these functions:

`herr_t H5Tregister_hard (const char *name, hid_t src_type, hid_t dest_type, H5T_conv_t func)`
  Once a conversion function is written, the library must be notified so it can be used. The function can be registered as a hard conversion for one or more conversion paths by calling `H5Tregister_hard()`, displacing any previous hard conversion for those paths. The name is used only for debugging but must be supplied.
`herr_t H5Tregister_soft (const char *name, H5T_class_t src_class, H5T_class_t dest_class, H5T_conv_t func)`
  The function can be registered as a generic function which will be automatically added to any conversion path for which it returns an indication that it applies. The name is used only for debugging but must be supplied.
`herr_t H5Tunregister (H5T_conv_t func)`
  A function can be removed from the set of known conversion functions by calling `H5Tunregister()`. The function is removed from all conversion paths.

#### Example: A conversion function

Here's an example application-level function that converts Cray `unsigned short` to any other 16-bit unsigned big-endian integer. A cray `short` is a big-endian value which has 32 bits of precision in the high-order bits of a 64-bit word.

``

     1 typedef struct {
     2     size_t dst_size;
     3     int direction;
     4 } cray_ushort2be_t;
     5
     6 herr_t
     7 cray_ushort2be (hid_t src, hid_t dst,
     8                 H5T_cdata_t *cdata,
     9                 size_t nelmts, void *buf,
    10                 const void *background)
    11 {
    12     unsigned char *src = (unsigned char *)buf;
    13     unsigned char *dst = src;
    14     cray_ushort2be_t *priv = NULL;
    15
    16     switch (cdata->command) {
    17     case H5T_CONV_INIT:
    18         /*
    19          * We are being queried to see if we handle this
    20          * conversion.  We can handle conversion from
    21          * Cray unsigned short to any other big-endian
    22          * unsigned integer that doesn't have padding.
    23          */
    24         if (!H5Tequal (src, H5T_CRAY_USHORT) ||
    25             H5T_ORDER_BE != H5Tget_order (dst) ||
    26             H5T_SGN_NONE != H5Tget_signed (dst) ||
    27             8*H5Tget_size (dst) != H5Tget_precision (dst)) {
    28             return -1;
    29         }
    30
    31         /*
    32          * Initialize private data.  If the destination size
    33          * is larger than the source size, then we must
    34          * process the elements from right to left.
    35          */
    36         cdata->priv = priv = malloc (sizeof(cray_ushort2be_t));
    37         priv->dst_size = H5Tget_size (dst);
    38         if (priv->dst_size>8) {
    39             priv->direction = -1;
    40         } else {
    41             priv->direction = 1;
    42         }
    43         break;
    44
    45     case H5T_CONV_FREE:
    46         /*
    47          * Free private data.
    48          */
    49         free (cdata->priv);
    50         cdata->priv = NULL;
    51         break;
    52
    53     case H5T_CONV_CONV:
    54         /*
    55          * Convert each element, watch out for overlap src
    56          * with dst on the left-most element of the buffer.
    57          */
    58         priv = (cray_ushort2be_t *)(cdata->priv);
    59         if (priv->direction<0) {
    60             src += (nelmts - 1) * 8;
    61             dst += (nelmts - 1) * dst_size;
    62         }
    63         for (i=0; i<n; i++) {
    64             if (src==dst && dst_size<4) {
    65                 for (j=0; j<dst_size; j++) {
    66                     dst[j] = src[j+4-dst_size];
    67                 }
    68             } else {
    69                 for (j=0; j<4 && j<dst_size; j++) {
    70                     dst[dst_size-(j+1)] = src[3-j];
    71                 }
    72                 for (j=4; j<dst_size; j++) {
    73                     dst[dst_size-(j+1)] = 0;
    74                 }
    75             }
    76             src += 8 * direction;
    77             dst += dst_size * direction;
    78         }
    79         break;
    80
    81     default:
    82         /*
    83          * Unknown command.
    84          */
    85         return -1;
    86     }
    87     return 0;
    88 }

The background argument is ignored since it's generally not applicable to atomic data types.

#### Example: Soft Registration

The convesion function described in the previous example applies to more than one conversion path. Instead of enumerating all possible paths, we register it as a soft function and allow it to decide which paths it can handle.

``

    H5Tregister_soft ("cus2be", H5T_INTEGER, H5T_INTEGER, cray_ushort2be);

This causes it to be consulted for any conversion from an integer type to another integer type. The first argument is just a short identifier which will be printed with the data type conversion statistics.

NOTE: The idea of a master soft list and being able to query conversion functions for their abilities tries to overcome problems we saw with AIO. Namely, that there was a dichotomy between generic conversions and specific conversions that made it very difficult to write a conversion function that operated on, say, integers of any size and order as long as they don't have zero padding. The AIO mechanism required such a function to be explicitly registered (like `H5Tregister_hard()`) for each an every possible conversion path whether that conversion path was actually used or not.

Last modified: 27 October 1998