Before, setting CURLOPT_SSLVERSION with wolfSSL restricted the the tls
proto to just the specified version. Now it properly supports a range.
So it can set the min and max tls proto (max requires wolfSSL 4.2.0).
Bump the absolute minimum required version of wolfSSL to 3.4.6 (released
2015) because it is needed for the wolfSSL_CTX_SetMinVersion() function.
Closes#14480
Channel Bindings are used to tie the session context to a specific TLS
channel. This is to provide additional proof of valid identity,
mitigating authentication relay attacks.
Major web servers have the ability to require (None/Accept/Require)
GSSAPI channel binding, rendering Curl unable to connect to such
websites unless support for channel bindings is implemented.
IIS calls this feature Extended Protection (EPA), which is used in
Enterprise environments using Kerberos for authentication.
This change require krb5 >= 1.19, otherwise channel bindings won't be
forwarded through SPNEGO.
Co-Authored-By: Steffen Kieß <947515+steffen-kiess@users.noreply.github.com>
Closes#13098
Bring setting ciphers with WolfSSL in line with other SSL backends,
to make the curl interface more consistent across the backends.
Now the tls1.3 ciphers are set with the --tls13-ciphers option, when
not set the default tls1.3 ciphers are used. The tls1.2 (1.1, 1.0)
ciphers are set with the --ciphers option, when not set the default
tls1.2 ciphers are used. The ciphers available for the connection
are now a union of the tls1.3 and tls1.2 ciphers.
This changes the behaviour for WolfSSL when --ciphers is set, but
--tls13-ciphers is not set. Now the ciphers set with --ciphers
are combined with the default tls1.3 ciphers, whereas before solely
the ciphers of --ciphers were used.
Thus before when no tls1.3 ciphers were specified in --ciphers,
tls1.3 was completely disabled. This might not be what the user
expected, especially as this does not happen with OpenSSL.
Closes#14385
Bring setting ciphers with mbedTLS in line with other SSL backends,
to make the curl interface more consistent across the backends.
Now the tls1.3 ciphers are set with the --tls13-ciphers option, when
not set the default tls1.3 ciphers are used. The tls1.2 (1.1, 1.0)
ciphers are set with the --ciphers option, when not set the default
tls1.2 ciphers are used. The ciphers available for the connection
are now a union of the tls1.3 and tls1.2 ciphers.
This changes the behaviour for mbedTLS when --ciphers is set, but
--tls13-ciphers is not set. Now the ciphers set with --ciphers
are combined with the default tls1.3 ciphers, whereas before solely
the ciphers of --ciphers were used.
Thus before when no tls1.3 ciphers were specified in --ciphers,
tls1.3 was completely disabled. This might not be what the user
expected, especially as this does not happen with OpenSSL.
Closes#14384
- replace the counting of upload lengths with the new eos send flag
- improve frequency of stream draining to happen less on events where it
is not needed
- this PR is based on #14220
http2, cf-h2-proxy: fix EAGAINed out buffer
- in adjust pollset and shutdown handling, a non-empty `ctx->outbufq`
must trigger send polling, irregardless of http/2 flow control
- in http2, fix retry handling of blocked GOAWAY frame
test case improvement:
- let client 'upload-pausing' handle http versions
Closes#14253
Improve handling of shutdown when sending gets blocked.
Add workaround for <https://github.com/wolfSSL/wolfssl/issues/7784>
where wolfSSL keeps on adding close notify messages to its outgoing
buffer on ever attempt.
Closes#14376
Make sure that `io_need` is cleared and set at the filter operations.
Add some more tracing for shutdown situations.
Improve shutdown handling for blocked sends. OpenSSL is a bit tricksy
here that it only reports WANT_WRITE on SSL_shutdown(), but never on
SSL_read() on blocked sends. So we need to use both.
At last, set SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER when available since we
are not always retrying sends from the very same address, as testing
showed.
Closes#14375
wolfSSL supports setting certificates/private keys from memory blobs
which allow us to implement both CURLOPT_SSLCERT_BLOB and
CURLOPT_SSLKEY_BLOB options.
Closes#14018
- For HTTPS if http2-prior-knowledge is set then only offer h2 (HTTP/2)
alpn to the server for protocol negotiation.
Prior to this change both HTTP/2 ("h2") and HTTP/1.1 ("http/1.1") were
offered for ALPN when http2-prior-knowledge was set.
CURL_HTTP_VERSION_2_PRIOR_KNOWLEDGE (tool: --http2-prior-knowledge) is
meant to send non-TLS HTTP requests HTTP/2 when it is known the server
supports them. However when HTTPS is used then it attempts to first
negotiate the connection with ALPN. In that case the user likely does
not want to offer http/1.1 to the server as an acceptable protocol.
Reported-by: kit-ty-kate@users.noreply.github.com
Fixes https://github.com/curl/curl/issues/9963
Closes https://github.com/curl/curl/pull/14266
Use these words and casing more consistently across text, comments and
one curl tool output:
AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,
macOS, MS-DOS, MSYS, MinGW, NTLM, POSIX, Solaris, UNIX, Unix, Unicode,
WINE, WebDAV, Win32, winbind, WinIDN, Windows, Windows CE, Winsock.
Mostly OS names and a few more.
Also a couple of other minor text fixups.
Closes#14360
Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.
Closes#14220
- check that `struct ssl_peer` is only initialized once
- fix vtls peer init to run only once
- check in peer init that hostname is not empty, fail otherwise
Closes#14152
Replace Curl_resolv_unlock() with Curl_resolv_unlink():
-replace inuse member with refcount in Curl_dns_entry
- pass Curl_dns_entry ** to unlink, so it gets always cleared
- solve potential (but unlikley) UAF in FTP's handling of looked up
Curl_dns_entry. Esp. do not use addr information after unlinking an entry.
In reality, the unlink will not free memory, as the dns entry is still
referenced by the hostcache. But this is not safe and relying on no other
code pruning the cache in the meantime.
- pass permanent flag when adding a dns entry instead of fixing timestamp
afterwards.
url.c: fold several static *resolve_* functions into one.
Closes#14195
Fixes:
```
curl\lib\vtls\cipher_suite.c(193,3): error C2220: the following warning is treated as an error
curl\lib\vtls\cipher_suite.c(193,3): warning C4310: cast truncates constant value
```
Closes#14341
Raise the limit for certification information from 10 thousand to 100
thousand bytes. Certificates can be larger than 10k.
Change the infof() debug output to add '...' at the end when the max
limit it can handle is exceeded.
Reported-by: Sergio Durigan Junior
Fixes#14352Closes#14354
The MSVC compiler cannot have forward declaration with const and static
variable, causing this error:
```
curl\lib\vtls\vtls.c(417,44): warning C4132: 'Curl_ssl_multi': const object should be initialized
```
Ref: #14276Closes#14305
When sharing the x509 store in wolfSSL, always use an explicitly
constructed one, as the SSLCTX might have "only" an internal one which
is not obeying reference count lifetimes.
Fixes#14278
Reported-by: Alex Snast
Closes#14279
Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.
Closes#14220
Fix the file of wolfssl.c because of this warning/error:
```
curl\lib\vtls\wolfssl.c(1017,42): error C2220: the following warning is treated as an error [curl\bld\lib\libcurl_object.vcxproj]
curl\lib\vtls\wolfssl.c(1017,42): warning C4267: 'function': conversion from 'size_t' to 'unsigned long', possible loss of data [curl\bld\lib\libcurl_object.vcxproj]
```
`size_t` in MSVC is different. Change it to `unsigned long` because
`wolfSSL_ERR_error_string_n` last argument is defined as
`unsigned long`.
Closes#14193
Fix `-Wpointer-bool-conversion` warnings with the method suggested by
both Apple clang and mainline llvm. This was already tried and dropped
in #1705 (in year 2017), but the issue reported there no longer
replicates.
Verified with Apple clang 14, llvm 15, llvm 18 and gcc 11, 14 that the
generated objects are bit by bit identical before and after this patch.
Also:
- stop silencing `-Wtautological-pointer-compare`. This warning don't
seem to be appearing anymore (with or without this patch), at least
with the tested compilers and SDKs (clang 13.1.6-16.0.0beta, llvm 15,
18, gcc 11, 14) and minimum macOS target of 10.8. Older targets fail
to build curl with SecureTransport.
- silence `-Wunreachable-code` for clang only. Previously I applied it
also to GCC, by mistake.
Ref: 8d7172d20a
Apple clang `-Wpointer-bool-conversion`:
```
curl/lib/vtls/sectransp.c:1103:6: error: address of function 'SSLCreateContext' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
if(SSLCreateContext) { /* use the newer API if available */
~~ ^~~~~~~~~~~~~~~~
curl/lib/vtls/sectransp.c:1103:6: note: prefix with the address-of operator to silence this warning
if(SSLCreateContext) { /* use the newer API if available */
^
&
```
Ref: https://github.com/curl/curl/actions/runs/9819538439/job/27113201384#step:8:382
llvm `-Wpointer-bool-conversion`:
```
curl/lib/vtls/sectransp.c:2663:8: error: address of function 'SSLCreateContext' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
if(SSLCreateContext)
~~ ^~~~~~~~~~~~~~~~
curl/lib/vtls/sectransp.c:2663:8: note: prefix with the address-of operator to silence this warning
if(SSLCreateContext)
^
&
```
Ref: https://github.com/curl/curl/actions/runs/9819538439/job/27113200291#step:8:417
gcc still needs `-Waddress` suppressed to avoid these:
```
curl/lib/vtls/n/sectransp.c: In function 'getsubject':
curl/lib/vtls/n/sectransp.c:379:6: warning: the address of 'SecCertificateCopyLongDescription' will always evaluate as 'true' [-Waddress]
379 | if(&SecCertificateCopyLongDescription)
| ^
[...]
```
Follow-up to 59cadacfcc#14128
Follow-up to af271ce9b9#1722
Follow-up to 2b7ce3f56d#1706
Cherry-picked from #14097Closes#14162
- cmake: enable Apple-specific `-Werror=partial-availability` to match
autotools.
- autotools: enable `-pedantic-errors` with llvm/clang to match gcc and
CMake.
- autotools: enable `-Werror-implicit-function-declaration` for
llvm/clang to match gcc.
- cmake: enable `-Werror-implicit-function-declaration` to match
autotools.
- move `-Wpointer-bool-conversion` from autotools to the local file
(`sectransp.c`) it was meant to apply. This way it applies to all
build methods.
- autotoos: show `CURL_CFLAG_EXTRAS` in the `./configure` summary.
(it may contain `-Werror` and/or `-pedentic-errors`.)
Cherry-picked from #14097Closes#14128
Currently we're using WOLFSSL_MAX_ERROR_SZ to define the error buffer
size, this value is user defined which means it can be overwritten with
-DWOLFSSL_MAX_ERROR_SZ=512 when building wolfssl and this overwrite is
not exported to the users of wolfssl.
Instead of relying on WOLFSSL_MAX_ERROR_SZ we'll just use a 256 bytes
error buffer and use wolfSSL_ERR_error_string_n to fill it thus dropping
the dependency on WOLFSSL_MAX_ERROR_SZ altogether.
Closes#14114
- deduplicate the code in many tls backends that check
for an existing id and delete it before adding the new one
- rename ssl_primary_config's `sessionid` bool to `cache_session`
Closes#14121
`HAVE_BUILTIN_AVAILABLE` is a curl macro set via autotools and cmake.
Like other `HAVE_`s it signals availability if defined.
SecureTransport code was specifically looking for the value 1, which
triggered compiler warnings when the feature was not present.
Replace the existing workaround of locally suppressing the compiler
warning with using `defined()`.
autotools:
```
767 | #if (CURL_BUILD_MAC_10_13 || CURL_BUILD_IOS_11) && HAVE_BUILTIN_AVAILABLE == 1
| ^~~~~~~~~~~~~~~~~~~~~~
../../lib/vtls/sectransp.c: In function 'sectransp_connect_step1':
../../lib/vtls/sectransp.c:1140:52: error: "HAVE_BUILTIN_AVAILABLE" is not defined, evaluates to 0 [-Werror=undef]
1140 | #if (CURL_BUILD_MAC_10_13 || CURL_BUILD_IOS_11) && HAVE_BUILTIN_AVAILABLE == 1
| ^~~~~~~~~~~~~~~~~~~~~~
../../lib/vtls/sectransp.c:1240:52: error: "HAVE_BUILTIN_AVAILABLE" is not defined, evaluates to 0 [-Werror=undef]
1240 | #if (CURL_BUILD_MAC_10_13 || CURL_BUILD_IOS_11) && HAVE_BUILTIN_AVAILABLE == 1
| ^~~~~~~~~~~~~~~~~~~~~~
../../lib/vtls/sectransp.c: In function 'sectransp_connect_step2':
```
Ref: https://github.com/curl/curl/actions/runs/9815428701/job/27104448045#step:6:499
cmake gcc:
```
1140 | #if (CURL_BUILD_MAC_10_13 || CURL_BUILD_IOS_11) && HAVE_BUILTIN_AVAILABLE == 1
| ^~~~~~~~~~~~~~~~~~~~~~
/Users/runner/work/curl/curl/lib/vtls/sectransp.c:1240:52: error: "HAVE_BUILTIN_AVAILABLE" is not defined, evaluates to 0 [-Werror=undef]
1240 | #if (CURL_BUILD_MAC_10_13 || CURL_BUILD_IOS_11) && HAVE_BUILTIN_AVAILABLE == 1
| ^~~~~~~~~~~~~~~~~~~~~~
/Users/runner/work/curl/curl/lib/vtls/sectransp.c: In function 'sectransp_connect_step2':
/Users/runner/work/curl/curl/lib/vtls/sectransp.c:2231:51: error: "HAVE_BUILTIN_AVAILABLE" is not defined, evaluates to 0 [-Werror=undef]
2231 | #if(CURL_BUILD_MAC_10_13 || CURL_BUILD_IOS_11) && HAVE_BUILTIN_AVAILABLE == 1
| ^~~~~~~~~~~~~~~~~~~~~~
```
Ref: https://github.com/curl/curl/actions/runs/9815428701/job/27104445425#step:8:355
Cherry-picked from #14097Closes#14122
Based on the standards and guidelines we use for our documentation.
- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period
Closes#14073
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.
The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.
Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.
As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.
Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.
When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.
The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.
- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.
- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.
- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.
TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.
A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.
- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.
Closes#13976
When user sets CURLOPT_SSLCERT but leaves CURLOPT_SSLKEY unset assume
the path passed in CURLOPT_SSLCERT holds the ssl key which is what we do
in openssl implementation.
Fixes#14007Closes#14008
The function we use is called 'gnutls_x509_crt_check_hostname()' but if
we pass in the hostname with a trailing dot, the check fails. If we pass
in the SNI name, which cannot have a trailing dot, it succeeds for
https://pyropus.ca./
I consider this as a flaw in GnuTLS and have submitted this issue
upstream:
https://gitlab.com/gnutls/gnutls/-/issues/1548
In order to work with old and existing GnuTLS versions, we still need
this change no matter how they view the issue or might change it in the
future.
Fixes#13428
Reported-by: Ryan Carsten Schmidt
Closes#13949
