openssl/ssl/statem
Richard Levitte e077455e9e Stop raising ERR_R_MALLOC_FAILURE in most places
Since OPENSSL_malloc() and friends report ERR_R_MALLOC_FAILURE, and
at least handle the file name and line number they are called from,
there's no need to report ERR_R_MALLOC_FAILURE where they are called
directly, or when SSLfatal() and RLAYERfatal() is used, the reason
`ERR_R_MALLOC_FAILURE` is changed to `ERR_R_CRYPTO_LIB`.

There were a number of places where `ERR_R_MALLOC_FAILURE` was reported
even though it was a function from a different sub-system that was
called.  Those places are changed to report ERR_R_{lib}_LIB, where
{lib} is the name of that sub-system.
Some of them are tricky to get right, as we have a lot of functions
that belong in the ASN1 sub-system, and all the `sk_` calls or from
the CRYPTO sub-system.

Some extra adaptation was necessary where there were custom OPENSSL_malloc()
wrappers, and some bugs are fixed alongside these changes.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Hugo Landau <hlandau@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19301)
2022-10-05 14:02:03 +02:00
..
extensions_clnt.c Stop raising ERR_R_MALLOC_FAILURE in most places 2022-10-05 14:02:03 +02:00
extensions_cust.c SSL object refactoring using SSL_CONNECTION object 2022-07-28 10:04:28 +01:00
extensions_srvr.c Stop raising ERR_R_MALLOC_FAILURE in most places 2022-10-05 14:02:03 +02:00
extensions.c Stop raising ERR_R_MALLOC_FAILURE in most places 2022-10-05 14:02:03 +02:00
README.md ssl: Correct filename in README 2021-09-27 09:39:17 +10:00
statem_clnt.c Stop raising ERR_R_MALLOC_FAILURE in most places 2022-10-05 14:02:03 +02:00
statem_dtls.c Stop raising ERR_R_MALLOC_FAILURE in most places 2022-10-05 14:02:03 +02:00
statem_lib.c Stop raising ERR_R_MALLOC_FAILURE in most places 2022-10-05 14:02:03 +02:00
statem_local.h Test TLS extension ordering 2022-09-28 09:54:25 -04:00
statem_srvr.c Stop raising ERR_R_MALLOC_FAILURE in most places 2022-10-05 14:02:03 +02:00
statem.c Use an enum for the return value from a construction function 2022-09-20 08:16:57 +01:00
statem.h Use an enum for the return value from a construction function 2022-09-20 08:16:57 +01:00

State Machine Design

This file provides some guidance on the thinking behind the design of the state machine code to aid future maintenance.

The state machine code replaces an older state machine present in OpenSSL versions 1.0.2 and below. The new state machine has the following objectives:

  • Remove duplication of state code between client and server
  • Remove duplication of state code between TLS and DTLS
  • Simplify transitions and bring the logic together in a single location so that it is easier to validate
  • Remove duplication of code between each of the message handling functions
  • Receive a message first and then work out whether that is a valid transition - not the other way around (the other way causes lots of issues where we are expecting one type of message next but actually get something else)
  • Separate message flow state from handshake state (in order to better understand each)
    • message flow state = when to flush buffers; handling restarts in the event of NBIO events; handling the common flow of steps for reading a message and the common flow of steps for writing a message etc
    • handshake state = what handshake message are we working on now
  • Control complexity: only the state machine can change state: keep all the state changes local to the state machine component

The message flow state machine is divided into a reading sub-state machine and a writing sub-state machine. See the source comments in statem.c for a more detailed description of the various states and transitions possible.

Conceptually the state machine component is designed as follows:

                      libssl
                         |
-------------------------|-----statem.h------------------------------------
                         |
                  _______V____________________
                 |                            |
                 |    statem.c                |
                 |                            |
                 |    Core state machine code |
                 |____________________________|
      statem_local.h     ^          ^
               _________|          |_______
              |                            |
 _____________|____________   _____________|____________
|                          | |                          |
| statem_clnt.c            | | statem_srvr.c            |
|                          | |                          |
| TLS/DTLS client specific | | TLS/DTLS server specific |
| state machine code       | | state machine code       |
|__________________________| |__________________________|
             |        |_______________|__       |
             |        ________________|  |      |
             |       |                   |      |
 ____________V_______V________   ________V______V_______________
|                             | |                               |
| statem_lib.c                | | statem_dtls.c                 |
|                             | |                               |
| Non core functions common   | | Non core functions common to  |
| to both servers and clients | | both DTLS servers and clients |
|_____________________________| |_______________________________|