hdf5/doc/file-locking.md

# File Locking in HDF5

This document describes the file locking scheme that was added to HDF5 in
version 1.10.0 and how you can work around it, if you choose to do so. I'll
try to keep it understandable for everyone, though diving into technical
details is unavoidable, given the complexity of the material. We're in the
process of converting the HDF5 user guide (UG) to Doxygen and this document
will eventually be rolled up into those files as we update things.

**Parallel HDF5 Note**

Everything written here is from the perspective of serial HDF5. When we say
that you can't access a file for write access from more than one process, we
mean "from more than one independent, serial process". Parallel HDF5 can
obviously write to a file from more than one process, but that involves
IPC and multiple processes working together, not independent processes with
no knowledge of each other, which is what the file locks are for.


## Why file locks?

The short answer is: "To prevent you from corrupting your HDF5 files and/or
crashing your reader processes."

The long answer is more complicated.

An HDF5 file's state exists in two places when it is open for writing:

1. The HDF5 file itself
2. The HDF5 library's various caches

One of those caches is the metadata cache, which stores things like B-tree
nodes that we use to locate data in the file. Problems arise when parent
objects are flushed to storage before child objects. If a reader tries to
load unflushed children, the object's file offset could point at garbage
and it will encounter library failures as it tries to access the non-existent
objects.

Keep in mind that the HDF5 library is not analogous to a database server. The
HDF5 library is just a simple shared library, like libjpeg. Library state is
maintained per-library-instance and there is no IPC between HDF5 libraries
loaded by different processes (exception: collective operations in parallel
HDF5, but that's not what were talking about here).

Prior to HDF5 1.10.0, concurrent access to an HDF5 file by multiple processes,
when one or more processes is a writer, was not supported. There was no
enforcement mechanism for this. We simply told people not to do it.

In HDF5 1.10.0, we updated the library to allow the single-writer / multiple-readers
(SWMR - pronounced "swimmer") access pattern. This setup allows one writer and
multiple readers to access the same file, as long as a certain protocol is
followed concerning file opening order and setting the right flags. Since
synchronization might be tricky to pull off and the consequences of getting
it wrong could result in corrupt files or crashed readers, we decided to add
a file locking scheme to help users get it right. Since this would also help
prevent harmful accesses when SWMR is not in use, we decided to switch the
file locking scheme on by default. This scheme has been carried forward into
HDF5 1.12 and 1.13 (soon to be 1.14).

Note that the current implementation of SWMR is only useful for appending to chunked
datasets. Creating file objects like groups and datasets is not supported
in the current SWMR implementation.

Unfortunately, this file locking scheme has caused problems for some users.
This is usually people who are working on network file systems like NFS or
on parallel file systems, especially when file locks have been disabled, which
often causes lock calls to fail. As a result of this, we've added work-arounds
to disable the file locking scheme over the years.

## The existing scheme

There are two parts to the file locking scheme. One is the file lock itself.
The second is a mark we make in the HDF5 file's superblock. The superblock
mark isn't really that important for understanding the file locking, but since
it's entwined with the file locking scheme, we'll cover it in the
algorithm below. The lower-level details of file lock implementations are
described in the appendix, but the semantics are straightforward: Locks are
mandatory unless disabled, always for the entire file, and non-blocking. They
are also not required for SWMR operations and simply exist to help you set up
SWMR and prevent dangerous file access.

Here's how it all works:

1. The first thing we do is check if we're using file locks

    - We first check the file locking property in the file access property list
      (fapl). The default value of this property is set at configure time when
      the library is built.
    - Next we check the value of the `HDF5_USE_FILE_LOCKING` environment variable,
      which was previously parsed at library startup. If this is set,
      we use the value to override the property list setting.

    The particulars of the ways you can disable file locks are described in a
    separate section below.

    If we are not using file locking, no further file locking operations will
    take place.

2. We also check for ignoring file locks when they are disabled on the file system.

    - The environment variable setting for this is checked at VFD initialization
      time for all library VFDs.
    - We check the value in the fapl in the `open` callback. The default value for
      this property was set at configure time when the library was built.

3. When we open a file, we lock it based on the file access flags:

    - If the `H5F_ACC_RDWR` flag is set, use an exclusive lock
    - Otherwise use a shared lock

    If we are ignoring disabled file locks (see below), we will silently swallow
    lock API call failure when locks are not implemented on the file system.

4. If the VFD supports locking and the file is open for writing, we mark the
   file consistency flags in the file's superblock to indicate this.

    **NOTE!**

    - The VFD has to have a lock callback for this to happen. It doesn't matter if
      the locking was disabled - the check is simply for the callback.
    - We mark the superblock in **ANY** write case - both SWMR and non-SWMR.
    - Only the latest version of the superblock is marked in this way. If you
      open up a file that wasn't created with the 1.10.0 or later file format,
      it won't get the superblock mark, even if it's been opened for writing.

    According to the file format document and H5Fpkg.h:

    - Bit 0 is set if the file is open for writing (`H5F_SUPER_WRITE_ACCESS`)
    - Bit 2 is set if the file is open for SWMR writing (`H5F_SUPER_SWMR_WRITE_ACCESS`)

    We check these superblock flags on file open and error out if they are
    unsuitable.

    - If the file is already opened for non-SWMR writing, no other process can open
      it.
    - If the file is open for SWMR writing, only SWMR readers can open the file.
    - If you try to open a file for reading with `H5F_ACC_SWMR_READ` set and the
      file does not have the SWMR writer bits set in the superblock, the open
      call will fail.

    This scheme is often confused with the file locking, so it's included here,
    even though it's a bit tangential to the locks themselves.

5. If the file is open for SWMR writing (`H5F_ACC_SWMR_WRITE` is set), we
   remove the file lock just before the open call completes.

6. We normally don't explicitly unlock the file on file close. We let the OS
   handle it when the file descriptors are closed since file locks don't
   normally surivive closing the underlying file descriptor.

**TL;DR**

When locks are available, HDF5 files will be exclusively locked while they are
in use. The exception to this are files that are opened for SWMR writing, which
are unlocked. Files that are open for any kind of writing get a "writing"
superblock mark that HDF5 1.10.0+ will respect and refuse to open outside of SWMR.

## `H5Fstart_swmr_write()`

This API call can be used to switch an open file to "SWMR writing" mode as
if it had been opened with the `H5F_ACC_SWMR_WRITE` flag set. This is used
when code needs to perform SWMR-forbidden operations like creating groups
and datasets before appending data to datasets using SWMR.

Most of the work of this API call involves flushing out the library caches
in preparation for SWMR access, but there are a few locking operations that
take place under the hood:

- The file's superblock is marked as in the SWMR writer case, above.
- For a brief period of time in the call, we convert the exclusive lock to
  a shared lock. It's unclear why this was done and we'll look into removing
  this.
- At the end of the call, the lock is removed, as in the SWMR write open
  case described above.

## Disabling the locks

There are several ways to disable the locks, depending on which version of the
HDF5 library you are working with. This section will describe the file lock
disable schemes as they exist in late 2022. The current library versions at
this time are 1.10.9, 1.12.3, and 1.13.2. File locks are not present in HDF5
1.8. The lock feature matrix later in this document will describe the
limitations of earlier versions.

### Configure option

You can set the file locking defaults at configure time. This sets the defaults
for the associated properties in the fapl. Users can override the configure
defaults using `H5Pset_file_locking()` or the `HDF5_USE_FILE_LOCKING`
environment variable.

- Autotools

    `--enable-file-locking=(yes|no|best-effort)` sets the file locking behavior.
    `on` and `off` should be self-explanatory. `best-effort` turns file locking
    on but ignores file locks when they are disabled (default: `best-effort`).

- CMake

    - set `IGNORE_DISABLED_FILE_LOCK` to `ON` to ignore file locks when they
      are disabled on the file system (default: `ON`).
    - set `HDF5_USE_FILE_LOCKING` to `OFF` to disable file locks (default: `ON`)

### `H5Pset_file_locking()`

This API call can be used to override the configure defaults. It takes
`hbool_t` parameters for both the file locking and "ignore file locks when
disabled on the file system" parameters. The values set here can be
overridden by the file locking environment variable.

There is a corresponding `H5Pget_file_locking()` call that can be used to check
the currently set values of both properties in the fapl. **NOTE** that this
call just checks the property list values. It does **NOT** check the
environment variables!

### Environment variables

The `HDF5_USE_FILE_LOCKING` environment variable overrides all other file
locking settings.

HDF5 1.10.0
- No file locking environment variable

HDF5 1.10.1 - 1.10.6, 1.12.0:
- `FALSE` turns file locking off
- Anything else turns file locking on
- Neither of these values ignores disabled file locks
- Environment variable parsed at file create/open time

HDF5 1.10.7+, 1.12.1+, 1.13.x:
- `FALSE` or `0` disables file locking
- `TRUE` or `1` enables file locking
- `BEST_EFFORT` enables file locking and ignores disabled file locks
- Anything else gives you the defaults
- Environment variable parsed at library startup

### Lock disable scheme interactions

As mentioned above and reiterated here:
- Configure-time settings set fapl defaults
- `H5Pset_file_locking()` overrides configure-time defaults
- The environment variable setting overrides all

If you want to check that file locking is on, you'll need to check the fapl
setting AND check the environment variable, which can override the fapl.

**!!! WARNING !!!**

Disabling the file locks is at your own risk. If more than one writer process
modifies an HDF5 file at the same time, the file could be corrupted. If a
reader process reads a file that is being modified by a writer, the writer
process might attempt to read garbage and encounter errors or even crash.

In the case of:

- A single process accessing a file with write access
- Any number of processes accessing a file read-only

You can safely disable the file locking scheme.

If you are trying to set up SWMR without the benefit of the file locks, you'll
just need to be extra careful that you hold to rules for SWMR access.

## Feature Matrix

The following table indicates which versions of the library support which file
lock features. 1.13.0 and 1.13.1 are experimental releases (basically glorified
release candidates) so they are not included here.

**Locks**

- P = POSIX locks only, Windows was a no-op that always succeeded
- WP = POSIX and Windows locks
- (-) = POSIX no-op lock fails
- (+) = POSIX no-op lock passes

**Configure Option and Environment Variable**

- on/off = sets file locks on/off
- try = can also set "best effort", where locks are on but ignored if disabled

|Version|Has locks|Configure option|`H5Pset_file_locking()`|`HDF5_USE_FILE_LOCKING`|
|-------|---------|----------------|-----------------------|-----------------------|
|1.8.x|No|-|-|-|
|1.10.0|P(-)|-|-|-|
|1.10.1|P(-)|-|-|on/off|
|1.10.2|P(-)|-|-|on/off|
|1.10.3|P(-)|-|-|on/off|
|1.10.4|P(-)|-|-|on/off|
|1.10.5|P(-)|-|-|on/off|
|1.10.6|P(-)|-|-|on/off|
|1.10.7|P(+)|try|Y|try|
|1.10.8|WP(+)|try|Y|try|
|1.10.9|WP(+)|try|Y|try|
|1.12.0|P(-)|-|-|on/off|
|1.12.1|WP(+)|try|Y|try|
|1.12.2|WP(+)|try|Y|try|
|1.13.2|WP(+)|try|Y|try|


## Appendix: File lock implementation

The file lock system is implemented with `flock(2)` as the archetype since it
has simple semantics and we don't need range locking. Locks are advisory on many
systems, but this shouldn't be a problem for most users since the HDF5 library
always respects them. If you have a program that parses or modifies HDF5 files
independently of the HDF5 library, you'll want to be mindful of any potential
for concurrent access across processes.

On Unix systems, we call `flock()` directly when it's available and pass
`LOCK_SH` (shared lock), `LOCK_EX` (exclusive lock), and `LOCK_UN` (unlock) as
described in the algorithm section. All locks are non-blocking, so we set the
`LOCK_NB` flag. Sadly, `flock(2)` is not POSIX and it doesn't lock files over
NFS. We didn't consider a lack of NFS support a problem since SWMR isn't
supported on networked file systems like NFS (write order preservation isn't
guaranteed) and `flock(2)` usually doesn't fail when you attempt to lock NFS
files.

On Unix systems without `flock(2)`, we implement a scheme based on `fcntl(2)`
(`Pflock()` in `H5system.c`). On these systems we use `F_SETLK` (non-blocking)
as the operation and set `l_type` in `struct flock` to be:

- `F_UNLOCK` for `LOCK_UN`
- `F_WRLOCK` for `LOCK_EX`
- `F_RDLOCK` for `LOCK_SH`

We set the range to be the entire file. Most Unix-like systems have `flock()`
these days, so this system probably isn't very well tested.

We don't use `fcntl`-based open file locks or mandatory locking anywhere. The
former scheme is non-POSIX and the latter is deprecated.

On Windows, we use `LockFileEx()` and `UnlockFileEx()` to lock the entire file
(`Wflock()` in `H5system.c`). We set `LOCKFILE_FAIL_IMMEDIATELY` to get
non-blocking locks and set `LOCKFILE_EXCLUSIVE_LOCK` when we want an exclusive
lock. SWMR isn't well-tested on Windows, so this scheme hasn't been as
thoroughly vetted as the `flock`-based scheme.

On non-Windows systems where neither `flock(2)` nor `fcntl(2)` is available,
we substitute a no-op stub that always succeeds (`Nflock()` in `H5system.c`).
In the past, the stub always failed (see the matrix for when we made the switch).
We currently know of no non-Windows systems where neither call is available
so this scheme is not well-tested.

One thing that should be immediately apparent to anyone familiar with file
locking, is that all of these schemes have subtly different semantics. We're
using file locking in a fairly crude manner, though, and lock use has always
been optional, so we consider this a lower-order concern.

Locks are implemented at the VFD level via `lock` and `unlock` callbacks. The
VFDs that implement file locks are: core (w/ backing store), direct, log, sec2,
and stdio (`flock(2)` locks only). The family, multi, and splitter VFDs invoke
the lock callback of their underlying sub-files. The onion and MPI-IO VFDs do NOT
use locks, even though they create normal, on-disk native HDF5 files. The
read-only S3 VFD and HDFS VFDs do not use file locking since they use
alternative storage schemes.

Lock failures are detected by checking to see if `errno` is set to `ENOSYS`.
This is not particularly sophisticated and was implemented as a way of working
around disabled locks on popular parallel file systems.

One other thing to note here is that, in all of the locking schemes we use, the
file locks do not survive process termination, so you don't have to worry
about files being locked forever if a process exits abnormally. If a writer
crashed and the library didn't clear the superblock mark, you can remove it with
the h5clear command-line tool, which is built with the library.