Bring setting ciphers with WolfSSL in line with other SSL backends,
to make the curl interface more consistent across the backends.
Now the tls1.3 ciphers are set with the --tls13-ciphers option, when
not set the default tls1.3 ciphers are used. The tls1.2 (1.1, 1.0)
ciphers are set with the --ciphers option, when not set the default
tls1.2 ciphers are used. The ciphers available for the connection
are now a union of the tls1.3 and tls1.2 ciphers.
This changes the behaviour for WolfSSL when --ciphers is set, but
--tls13-ciphers is not set. Now the ciphers set with --ciphers
are combined with the default tls1.3 ciphers, whereas before solely
the ciphers of --ciphers were used.
Thus before when no tls1.3 ciphers were specified in --ciphers,
tls1.3 was completely disabled. This might not be what the user
expected, especially as this does not happen with OpenSSL.
Closes#14385
Bring setting ciphers with mbedTLS in line with other SSL backends,
to make the curl interface more consistent across the backends.
Now the tls1.3 ciphers are set with the --tls13-ciphers option, when
not set the default tls1.3 ciphers are used. The tls1.2 (1.1, 1.0)
ciphers are set with the --ciphers option, when not set the default
tls1.2 ciphers are used. The ciphers available for the connection
are now a union of the tls1.3 and tls1.2 ciphers.
This changes the behaviour for mbedTLS when --ciphers is set, but
--tls13-ciphers is not set. Now the ciphers set with --ciphers
are combined with the default tls1.3 ciphers, whereas before solely
the ciphers of --ciphers were used.
Thus before when no tls1.3 ciphers were specified in --ciphers,
tls1.3 was completely disabled. This might not be what the user
expected, especially as this does not happen with OpenSSL.
Closes#14384
add --with-libuv to configure to (optionally) use it in debug-builds to
drive the event-based API
Use curl_multi_socket_action() and friends to drive parallel transfers.
tests/README has brief documentation for this
Closes#14298
- replace the counting of upload lengths with the new eos send flag
- improve frequency of stream draining to happen less on events where it
is not needed
- this PR is based on #14220
http2, cf-h2-proxy: fix EAGAINed out buffer
- in adjust pollset and shutdown handling, a non-empty `ctx->outbufq`
must trigger send polling, irregardless of http/2 flow control
- in http2, fix retry handling of blocked GOAWAY frame
test case improvement:
- let client 'upload-pausing' handle http versions
Closes#14253
Improve handling of shutdown when sending gets blocked.
Add workaround for <https://github.com/wolfSSL/wolfssl/issues/7784>
where wolfSSL keeps on adding close notify messages to its outgoing
buffer on ever attempt.
Closes#14376
Make sure that `io_need` is cleared and set at the filter operations.
Add some more tracing for shutdown situations.
Improve shutdown handling for blocked sends. OpenSSL is a bit tricksy
here that it only reports WANT_WRITE on SSL_shutdown(), but never on
SSL_read() on blocked sends. So we need to use both.
At last, set SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER when available since we
are not always retrying sends from the very same address, as testing
showed.
Closes#14375
Initializes 'no_signal' to TRUE, so that a call to sigpipe_apply() after
init ignores the signal (unless CURLOPT_NOSIGNAL) is set.
I have read the existing code multiple times now and I think it gets the
initial state reversed this missing to ignore.
Regression from 17e6f06ea3
Reported-by: Rasmus Thomsen
Fixes#14344Closes#14390
wolfSSL supports setting certificates/private keys from memory blobs
which allow us to implement both CURLOPT_SSLCERT_BLOB and
CURLOPT_SSLKEY_BLOB options.
Closes#14018
Implement the `--enable-versioned-symbols` feature available in
`./configure` for CMake.
Enable with `-DCURL_LIBCURL_VERSIONED_SYMBOLS=ON`. Customize the version
prefix with `-DCURL_LIBCURL_VERSIONED_SYMBOLS_PREFIX=MYPREFIX_`. By
default the prefix matches what autotools uses.
When enabled, the feature is detected and a warning shown if
unavailable. (E.g. on Apple and Windows, it isn't.)
Included `HIDDEN {};` to match autotools, though I don't know if it's
necessary, useful or making any difference.
Differences from the autotools implementation:
- soversion is dynamic instead of hard-coded.
- omits referencing non-curl symbols.
- allows prefix/flavour override.
- more universal feature detection.
- doesn't rely on the in-repo `lib/libcurl.vers.in` file.
Also:
- add mbedTLS and BearSSL versioned symbol prefix support to autotools.
- enable this option in an old-linux job.
Follow-up to 7cc2e8b349Fixes#14349Closes#14378
- Determine which connection cache is or will be used by the easy handle
and perform connection upkeep on that cache.
This change allows curl_easy_upkeep to be effective on easy handles that
are using a shared connection cache, either from a user created shared
connection cache or a user created multi which has its own shared
connection cache.
Prior to this change curl_easy_upkeep would upkeep the connection cache
for the easy handle only if that cache was from the multi owned by the
easy handle (ie curl_easy_perform was previously called and there's a
connection cache exclusive to the easy handle in
data->multi_easy->conn_cache).
Ref: https://curl.se/mail/lib-2024-01/0016.html
Closes https://github.com/curl/curl/pull/12677
- For HTTPS if http2-prior-knowledge is set then only offer h2 (HTTP/2)
alpn to the server for protocol negotiation.
Prior to this change both HTTP/2 ("h2") and HTTP/1.1 ("http/1.1") were
offered for ALPN when http2-prior-knowledge was set.
CURL_HTTP_VERSION_2_PRIOR_KNOWLEDGE (tool: --http2-prior-knowledge) is
meant to send non-TLS HTTP requests HTTP/2 when it is known the server
supports them. However when HTTPS is used then it attempts to first
negotiate the connection with ALPN. In that case the user likely does
not want to offer http/1.1 to the server as an acceptable protocol.
Reported-by: kit-ty-kate@users.noreply.github.com
Fixes https://github.com/curl/curl/issues/9963
Closes https://github.com/curl/curl/pull/14266
With this option, the entire download is skipped if the selected target
filename already exists when the opertion is about to begin.
Test 994, 995 and 996 verify.
Ref: #11012Closes#13993
revert f6cb3c63#14338
Setting SSLHonorCipherOrder to on means it honors the server cipher
order. From the documentation: "When choosing a cipher during an SSLv3
or TLSv1 handshake, normally the client's preference is used. If this
directive is enabled, the server's preference will be used instead."
Also the commit inhibits test_17_07_ssl_ciphers. The test tries to
tests if all the ciphers specified, and only those, are properly set
in curl. For that to work we need have cases where some or all ciphers
do no intersect with the cipher-set of the server. We need to be able
to assert a failed connection based on a cipher set mismatch.
That is why a restricted set of ciphers is used on the server. This
set is so chosen that it contains the well known most secure ciphers.
Except with the slower aes256 variant intentionally left out, to be
able to test above described.
As test_17_07_ssl_ciphers is currently the only test that tests the
functioning of the --ciphers and --tls13-ciphers options, it is
important that its coverage is as good as possible.
Closes#14381
Use these words and casing more consistently across text, comments and
one curl tool output:
AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,
macOS, MS-DOS, MSYS, MinGW, NTLM, POSIX, Solaris, UNIX, Unix, Unicode,
WINE, WebDAV, Win32, winbind, WinIDN, Windows, Windows CE, Winsock.
Mostly OS names and a few more.
Also a couple of other minor text fixups.
Closes#14360
If the second argument to the script is "commit", then this will
generate a file named `docs/tarball-commit.txt` that contains the latest
commit hash (git rev-parse HEAD) at the time the script runs.
Doing this breaks the reproducibility so it will not be used for "real"
releases but is meant for automated daily snapshots and similar.
Reported-by: Dan Fandrich
Fixes#14363Closes#14369
Since the documentation text blob might be gzipped, it needs to search
for what to output in a streaming manner. It then first searches for
"\nALL OPTIONS".
Then, it looks for the start to display at "\n -[option]" and stops
again at "\n -". Except for the last option in the man page, which
ends at "\nFILES" - the subtitle for the section following all options
in the manpage.
Test 1707 to 1710 verify
Closes#13997
... or pick the last directory part from the path if available.
Instead of returning error.
Add test 690 and 691 to verify. Test 76 and 2036 no longer apply.
Closes#13988
- tidy-up comments.
- use lowercase, underscore prefixed names for internal variables.
- use `IN LISTS` and `IN ITEMS` in `foreach()` loops.
- rename variable name `OUTPUT` to a more distinctive one.
- tidy-up `STREQUAL` syntax.
- delete commented code.
- indent/whitespace.
Closes#14197
Do no more than 5 transfers per 15 seconds with "5/15s" or limit it to 3
transfers per 4 hours with "3/4h" etc.
Previously it would always only work with a single time unit.
Ref: #14242Closes#14245
Cygwin runs stable but slow, while native Windows in the same workflow
runs fast but unreliable (hangs). GHA requires all jobs to finish before
allowing to re-run failed ones. Before this patch this meant waiting for
the slow Cygwin jobs to re-run hung native jobs. After this patch Cygwin
jobs run in their own workflow, allowing to re-run broken Windows jobs
earlier.
Follow-up to 0d12528729#14236Closes#14366
Since data can be held in connection filter buffers when sending gives
EAGAIN, add methods to query this and perform flushing of those buffers.
The transfer loop will continue sending until all upload data is
processed and the connection is flushed.
- add `CF_QUERY_SEND_PENDING` to query filters
- add `CF_CTRL_DATA_SEND_FLUSH` to flush filters
- change `Curl_req_want_send()` to query the connection
if it needs flushing
- use `Curl_req_want_send()` to determine the POLLOUT
in the PERFORMING multi state
- implement flush handling in the HTTP/2 connection filter
Closes#14271
Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.
Closes#14220
- check that `struct ssl_peer` is only initialized once
- fix vtls peer init to run only once
- check in peer init that hostname is not empty, fail otherwise
Closes#14152
Replace Curl_resolv_unlock() with Curl_resolv_unlink():
-replace inuse member with refcount in Curl_dns_entry
- pass Curl_dns_entry ** to unlink, so it gets always cleared
- solve potential (but unlikley) UAF in FTP's handling of looked up
Curl_dns_entry. Esp. do not use addr information after unlinking an entry.
In reality, the unlink will not free memory, as the dns entry is still
referenced by the hostcache. But this is not safe and relying on no other
code pruning the cache in the meantime.
- pass permanent flag when adding a dns entry instead of fixing timestamp
afterwards.
url.c: fold several static *resolve_* functions into one.
Closes#14195
- in DEBUGBUILD, all specifying if true random numbers
are desired or simulated ones via CURL_ENTROPY
- allows to use randoms in other DEBUG checks to not
interfere with the CURL_ENTROPY
- without this change, any Curl_rand() use will alter
results of some AUTHENTICATION methods like DIGEST
Closes#14264
The test-ci target now uses 2 processes by default, but the amount of
parallelism is tuned for each CI service and build environment based on
results of a number of test runs. Some CI services use super-
oversubscribed build machines that can barely run the curl tests
already with no parallelism without frequently failing with
timing-induced failures. These continue to be run without parallelism.
Other services provide two fast, unloaded cores and these run with 14
processes, which is a good default for this kind of environment.
Here's a summary of the number of test processes by CI service:
Appveyor - 2 (Windows MSVC), 1 (others)
Azure - 2
Circle CI - 14
Cirrus - 28 (macOS), 14 (Linux), 7 (FreeBSD), 5 (macOS torture), 2 (Windows)
GitHub Actions - 3 (macOS), 2 (Linux)
Some of these are a bit conservative to keep timing-induced flakiness down.
The net result is that the first test results should arrive only
3 minutes after a commit submission.
Changes merged via separate commits:
- 2a7c8b27fd#14171
- 72341068a2
- efce544418#14244
- c6cf411bac
Ref: #10818Closes#11510
- nghttp2 is now on by default, adjust build configuration accordingly.
Follow-up to 87aa4ebd82#14136
- GHA/macos: disable libidn2 in combination builds, syncing with
autotools. Otherwise it's now auto-detected via pkg-config by default.
Follow-up to f43adc2c49#14137Closes#14364
Add the ability to embed a CA bundle into the curl binary. It is used
when no other runtime or build-time option set one.
This helps curl-for-win macOS and Linux builds to run standalone, and
also helps Windows builds to avoid picking up the CA bundle from an
arbitrary (possibly world-writable) location (though this behaviour is
not currently disablable).
Usage:
- cmake: `-DCURL_CA_EMBED=/path/to/curl-ca-bundle.crt`
- autotools: `--with-ca-embed=/path/to/curl-ca-bundle.crt`
- Makefile.mk: `CURL_CA_EMBED=/path/to/curl-ca-bundle.crt`
Also add new command-line option `--dump-ca-embed` to dump the embedded
CA bundle to standard output.
Closes#14059
