openssl/ssl/statem
Bernd Edlinger e915c3f538 Fix an assertion in the DTLS server code
This fixes an internal error alert from the server and
an unexpected connection failure in the release version,
but a failed assertion and a server crash in the
debug version.

Reproduce this issue with a DTLS server/client like that:

./openssl s_server -dtls -mtu 1500
./openssl s_client -dtls -maxfraglen 512

In the debug version a crash happens in the Server now:

./openssl s_server -dtls -mtu 1500
Using default temp DH parameters
ACCEPT
ssl/statem/statem_dtls.c:269: OpenSSL internal error: Assertion failed: len == written
Aborted (core dumped)

While in the release version the handshake exceeds the
negotiated max fragment size, and fails because of this:

$ ./openssl s_server -dtls -mtu 1500
Using default temp DH parameters
ACCEPT
ERROR
4057152ADA7F0000:error:0A0000C2:SSL routines:do_dtls1_write:exceeds max fragment size:ssl/record/rec_layer_d1.c:826:
shutting down SSL
CONNECTION CLOSED

From the client's point of view the connection fails
with an Internal Error Alert:

$ ./openssl s_client -dtls -maxfraglen 512
Connecting to ::1
CONNECTED(00000003)
40B76343377F0000:error:0A000438:SSL routines:dtls1_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_d1.c:613:SSL alert number 80

and now the connection attempt fails unexpectedly.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18093)
2022-04-14 16:16:47 +02:00
..
extensions_clnt.c Add back check for the DH public key size 2022-02-17 13:21:01 +11:00
extensions_cust.c New extensions can be sent in a certificate request 2021-10-11 11:04:53 +01:00
extensions_srvr.c Add back check for the DH public key size 2022-02-17 13:21:01 +11:00
extensions.c ssl: better support TSAN operations 2022-01-19 21:51:47 +11:00
README.md
statem_clnt.c tls_process_server_hello: Disallow repeated HRR 2022-03-30 11:34:20 +02:00
statem_dtls.c Fix an assertion in the DTLS server code 2022-04-14 16:16:47 +02:00
statem_lib.c ssl: better support TSAN operations 2022-01-19 21:51:47 +11:00
statem_local.h Delete unused param about get_construct_message_f 2022-01-09 13:16:48 +11:00
statem_srvr.c ticket_lifetime_hint may exceed 1 week in TLSv1.3 2022-03-25 12:46:15 -04:00
statem.c Delete unused param about get_construct_message_f 2022-01-09 13:16:48 +11:00
statem.h

State Machine Design

This file provides some guidance on the thinking behind the design of the state machine code to aid future maintenance.

The state machine code replaces an older state machine present in OpenSSL versions 1.0.2 and below. The new state machine has the following objectives:

  • Remove duplication of state code between client and server
  • Remove duplication of state code between TLS and DTLS
  • Simplify transitions and bring the logic together in a single location so that it is easier to validate
  • Remove duplication of code between each of the message handling functions
  • Receive a message first and then work out whether that is a valid transition - not the other way around (the other way causes lots of issues where we are expecting one type of message next but actually get something else)
  • Separate message flow state from handshake state (in order to better understand each)
    • message flow state = when to flush buffers; handling restarts in the event of NBIO events; handling the common flow of steps for reading a message and the common flow of steps for writing a message etc
    • handshake state = what handshake message are we working on now
  • Control complexity: only the state machine can change state: keep all the state changes local to the state machine component

The message flow state machine is divided into a reading sub-state machine and a writing sub-state machine. See the source comments in statem.c for a more detailed description of the various states and transitions possible.

Conceptually the state machine component is designed as follows:

                      libssl
                         |
-------------------------|-----statem.h------------------------------------
                         |
                  _______V____________________
                 |                            |
                 |    statem.c                |
                 |                            |
                 |    Core state machine code |
                 |____________________________|
      statem_local.h     ^          ^
               _________|          |_______
              |                            |
 _____________|____________   _____________|____________
|                          | |                          |
| statem_clnt.c            | | statem_srvr.c            |
|                          | |                          |
| TLS/DTLS client specific | | TLS/DTLS server specific |
| state machine code       | | state machine code       |
|__________________________| |__________________________|
             |        |_______________|__       |
             |        ________________|  |      |
             |       |                   |      |
 ____________V_______V________   ________V______V_______________
|                             | |                               |
| statem_lib.c                | | statem_dtls.c                 |
|                             | |                               |
| Non core functions common   | | Non core functions common to  |
| to both servers and clients | | both DTLS servers and clients |
|_____________________________| |_______________________________|