gnutls: pass in SNI name, not hostname when checking cert

The function we use is called 'gnutls_x509_crt_check_hostname()' but if
we pass in the hostname with a trailing dot, the check fails. If we pass
in the SNI name, which cannot have a trailing dot, it succeeds for
https://pyropus.ca./

I consider this as a flaw in GnuTLS and have submitted this issue
upstream:

  https://gitlab.com/gnutls/gnutls/-/issues/1548

In order to work with old and existing GnuTLS versions, we still need
this change no matter how they view the issue or might change it in the
future.

Fixes #13428
Reported-by: Ryan Carsten Schmidt
Closes #13949

lib/vtls/gtls.c

@@ -1466,7 +1466,13 @@ Curl_gtls_verifyserver(struct Curl_easy *data,
      in RFC2818 (HTTPS), which takes into account wildcards, and the subject
      alternative name PKIX extension. Returns non zero on success, and zero on
      failure. */
-  rc = (int)gnutls_x509_crt_check_hostname(x509_cert, peer->hostname);
+
+  /* This function does not handle trailing dots, so if we have an SNI name
+     use that and fallback to the hostname only if there is no SNI (like for
+     IP addresses) */
+  rc = (int)gnutls_x509_crt_check_hostname(x509_cert,
+                                           peer->sni ? peer->sni :
+                                           peer->hostname);
 #if GNUTLS_VERSION_NUMBER < 0x030306
   /* Before 3.3.6, gnutls_x509_crt_check_hostname() didn't check IP
      addresses. */

tests/http/test_17_ssl_use.py

@@ -104,8 +104,6 @@ class TestSSLUse:
     # use host name with trailing dot, verify handshake
     @pytest.mark.parametrize("proto", ['http/1.1', 'h2', 'h3'])
     def test_17_03_trailing_dot(self, env: Env, httpd, nghttpx, repeat, proto):
-        if env.curl_uses_lib('gnutls'):
-            pytest.skip("gnutls does not match hostnames with trailing dot")
         if proto == 'h3' and not env.have_h3():
             pytest.skip("h3 not supported")
         curl = CurlClient(env=env)