Provide better documentation for SSL_get_servername()

The behaviour of SSL_get_servername() is quite complicated and depends on
numerous factors such as whether it is called on the client or the server,
whether it is called before or after the handshake, what protocol version
was negotiated, and whether a resumption was attempted or was successful.

We attempt to document the behavior more clearly.

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/10018)

doc/man3/SSL_CTX_set_tlsext_servername_callback.pod

@@ -11,7 +11,7 @@ SSL_set_tlsext_host_name - handle server name indication (SNI)
  #include <openssl/ssl.h>
 
  long SSL_CTX_set_tlsext_servername_callback(SSL_CTX *ctx,
-                                   int (*cb)(SSL *, int *, void *));
+                                   int (*cb)(SSL *s, int *al, void *arg));
  long SSL_CTX_set_tlsext_servername_arg(SSL_CTX *ctx, void *arg);
 
  const char *SSL_get_servername(const SSL *s, const int type);
@@ -21,21 +21,106 @@ SSL_set_tlsext_host_name - handle server name indication (SNI)
 
 =head1 DESCRIPTION
 
-The functionality provided by the servername callback is superseded by the
-ClientHello callback, which can be set using SSL_CTX_set_client_hello_cb().
-The servername callback is retained for historical compatibility.
+The functionality provided by the servername callback is mostly superseded by
+the ClientHello callback, which can be set using SSL_CTX_set_client_hello_cb().
+However, even where the ClientHello callback is used, the servername callback is
+still necessary in order to acknowledge the servername requested by the client.
 
 SSL_CTX_set_tlsext_servername_callback() sets the application callback B<cb>
 used by a server to perform any actions or configuration required based on
 the servername extension received in the incoming connection. When B<cb>
-is NULL, SNI is not used. The B<arg> value is a pointer which is passed to
-the application callback.
+is NULL, SNI is not used.
+
+The servername callback should return one of the following values:
+
+=over 4
+
+=item SSL_TLSEXT_ERR_OK
+
+This is used to indicate that the servername requested by the client has been
+accepted. Typically a server will call SSL_set_SSL_CTX() in the callback to set
+up a different configuration for the selected servername in this case.
+
+=item SSL_TLSEXT_ERR_ALERT_FATAL
+
+In this case the servername requested by the client is not accepted and the
+handshake will be aborted. The value of the alert to be used should be stored in
+the location pointed to by the B<al> parameter to the callback. By default this
+value is initialised to SSL_AD_UNRECOGNIZED_NAME.
+
+=item SSL_TLSEXT_ERR_ALERT_WARNING
+
+If this value is returned then the servername is not accepted by the server.
+However the handshake will continue and send a warning alert instead. The value
+of the alert should be stored in the location pointed to by the B<al> parameter
+as for SSL_TLSEXT_ERR_ALERT_FATAL above. Note that TLSv1.3 does not support
+warning alerts, so if TLSv1.3 has been negotiated then this return value is
+treated the same way as SSL_TLSEXT_ERR_NOACK.
+
+=item SSL_TLSEXT_ERR_NOACK
+
+This return value indicates that the servername is not accepted by the server.
+No alerts are sent and the server will not acknowledge the requested servername.
+
+=back
 
 SSL_CTX_set_tlsext_servername_arg() sets a context-specific argument to be
-passed into the callback for this B<SSL_CTX>.
+passed into the callback (via the B<arg> parameter) for this B<SSL_CTX>.
 
-SSL_get_servername() returns a servername extension value of the specified
-type if provided in the Client Hello or NULL.
+The behaviour of SSL_get_servername() depends on a number of different factors.
+In particular note that in TLSv1.3 the servername is negotiated in every
+handshake. In TLSv1.2 the servername is only negotiated on initial handshakes
+and not on resumption handshakes.
+
+=over 4
+
+=item On the client, before the handshake
+
+If a servername has been set via a call to SSL_set_tlsext_host_name() then it
+will return that servername.
+
+If one has not been set, but a TLSv1.2 resumption is being attempted and the
+session from the original handshake had a servername accepted by the server then
+it will return that servername.
+
+Otherwise it returns NULL.
+
+=item On the client, during or after the handshake and a TLSv1.2 (or below)
+resumption occurred
+
+If the session from the orignal handshake had a servername accepted by the
+server then it will return that servername.
+
+Otherwise it returns the servername set via SSL_set_tlsext_host_name() or NULL
+if it was not called.
+
+=item On the client, during or after the handshake and a TLSv1.2 (or below)
+resumption did not occur
+
+It will return the servername set via SSL_set_tlsext_host_name() or NULL if it
+was not called.
+
+=item On the server, before the handshake
+
+The function will always return NULL before the handshake
+
+=item On the server, after the servername extension has been processed and a
+TLSv1.2 (or below) resumption occurred
+
+If a servername was accepted by the server in the original handshake then it
+will return that servername, or NULL otherwise.
+
+=item On the server, after the servername extension has been processed and a
+TLSv1.2 (or below) resumption did not occur
+
+The function will return the servername requested by the client in this
+handshake or NULL if none was requested.
+
+=back
+
+Note that the ClientHello callback occurs before a servername extension from the
+client is processed. The servername, certificate and ALPN callbacks occur after
+a servername extension from the client is processed.
 
 SSL_get_servername_type() returns the servername type or -1 if no servername
 is present. Currently the only supported type (defined in RFC3546) is
@@ -65,6 +150,23 @@ SSL_set_tlsext_host_name() returns 1 on success, 0 in case of error.
 L<ssl(7)>, L<SSL_CTX_set_alpn_select_cb(3)>,
 L<SSL_get0_alpn_selected(3)>, L<SSL_CTX_set_client_hello_cb(3)>
 
+=head1 HISTORY
+
+SSL_get_servername() historically provided some unexpected results in certain
+corner cases. This has been fixed from OpenSSL 1.1.1e.
+
+Prior to 1.1.1e, when the client requested a servername in an initial TLSv1.2
+handshake, the server accepted it, and then the client successfully resumed but
+set a different explict servername in the second handshake then when called by
+the client it returned the servername from the second handshake. This has now
+been changed to return the servername requested in the original handshake.
+
+Also prior to 1.1.1e, if the client sent a servername in the first handshake but
+the server did not accept it, and then a second handshake occured where TLSv1.2
+resumption was successful then when called by the server it returned the
+servername requested in the original handshake. This has now been changed to
+NULL.
+
 =head1 COPYRIGHT
 
 Copyright 2017 The OpenSSL Project Authors. All Rights Reserved.

doc/man3/SSL_SESSION_get0_hostname.pod

@@ -24,7 +24,10 @@ SSL_SESSION_set1_alpn_selected
 =head1 DESCRIPTION
 
 SSL_SESSION_get0_hostname() retrieves the SNI value that was sent by the
-client when the session was created, or NULL if no value was sent.
+client when the session was created if it was accepted by the server and TLSv1.2
+or below was negotiated. Otherwise NULL is returned. Note that in TLSv1.3 the
+SNI hostname is negotiated with each handshake including resumption handshakes
+and is therefore never associated with the session.
 
 The value returned is a pointer to memory maintained within B<s> and
 should not be free'd.