Give warning for an empty pollset only when the connection has at least
IP connectivity. There are cases where the connect in QUIC makes another
attempt on a timeout and no socket will be available during that.
Closes#14108
Improve download performance, minimal effort.
Do not poll the socket for pending data every transfer loop iteration.
This gives 10-20% performance gains on large HTTP/1.1 downloads (on my
machine).
Closes#14098
- add a DEBUGASSERT for when a transfer's pollset should not be empty.
- move write unpausing from transfer loop into curl_easy_pause. This
make sure that the url_updatesocket() finds the correct state when
updating socket events.
- fix HTTP/2 proxy during connect phase to set sockets correctly
- fix test2600 to simulate a socket set
- move write unpausing from transfer loop into curl_easy_pause. This
make sure that the url_updatesocket() finds the correct state when
updating socket events.
- waiting for the resolver to deliver might not involve any sockets to
wait for. Do not generate a warning.
Fixes#14047Closes#14074
Based on the standards and guidelines we use for our documentation.
- expand contractions (they're => they are etc)
- host name = > hostname
- file name => filename
- user name = username
- man page => manpage
- run-time => runtime
- set-up => setup
- back-end => backend
- a HTTP => an HTTP
- Two spaces after a period => one space after period
Closes#14073
- support detecting wolfSSL via pkg-config (like autotools.)
- detect wolfSSL version.
- detect `HAVE_WOLFSSL_DES_ECB_ENCRYPT`.
(needs e.g. `--enable-curl` when building wolfSSL)
- detect `HAVE_WOLFSSL_FULL_BIO` and enable HTTPS-proxy feature.
(needs e.g. `--enable-opensslall` when building wolfSSL)
- fix to show `HTTPS-proxy` in cmake feature list.
Ref: 55807e6c05#9962
- fix to show `NTLM` in cmake feature list.
- fix to show `smb` and `smbs` in cmake protocol list.
- add wolfSSL CMake job to GHA (for macOS).
- fix mqtt and wolfSSL symbol clash.
```
./curl/lib/mqtt.c: In function 'mqtt_doing':
./curl/lib/mqtt.c:746:17: error: declaration of 'byte' shadows a global declaration [-Werror=shadow]
746 | unsigned char byte;
| ^~~~
/opt/homebrew/Cellar/wolfssl/5.7.0_1/include/wolfssl/wolfcrypt/types.h:85:36: note: shadowed declaration is here
85 | typedef unsigned char byte;
| ^~~~
```
- format `FindWolfSSL.cmake` closer to neighbours.
Closes#14064
- when checking for QUIC support in OpenSSL, also check
for it being at least 3.3.0
- remove workarounds for features buggy or missing in 3.2
Closes#14026
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.
The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.
Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.
As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.
Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.
When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.
The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.
- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.
- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.
- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.
TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.
A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.
- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.
Closes#13976
- in phase CONNECTING/TUNNELING/PROTOCONNECT, retrieve
the socket from the connection filters and do not rely
on `conn->sockfd` being already set by the transfer.
- this applies to the default behaviour, a protocol handler
may override this via its callbacks.
- add a warning message in multi_getsock() when the transfer
is expected to have something in its pollset, but instead
it is empty.
Reported-by: saurabhsingh-dev on github
Fixes#13998Closes#14011
When user sets CURLOPT_SSLCERT but leaves CURLOPT_SSLKEY unset assume
the path passed in CURLOPT_SSLCERT holds the ssl key which is what we do
in openssl implementation.
Fixes#14007Closes#14008
The function we use is called 'gnutls_x509_crt_check_hostname()' but if
we pass in the hostname with a trailing dot, the check fails. If we pass
in the SNI name, which cannot have a trailing dot, it succeeds for
https://pyropus.ca./
I consider this as a flaw in GnuTLS and have submitted this issue
upstream:
https://gitlab.com/gnutls/gnutls/-/issues/1548
In order to work with old and existing GnuTLS versions, we still need
this change no matter how they view the issue or might change it in the
future.
Fixes#13428
Reported-by: Ryan Carsten Schmidt
Closes#13949
When aborting the transfer loop early, like when there is rate limiting
in effect, there might be buffered data already read off the socket so
the socket might not signal reability. Therefore we must set the
CSELECT_IN manually if data_pending_() suggests there might be more data
to get. This is particularly noticeable with SSH when the underlying
library has drained the socket and holds pending data in its buffer.
Reported-by: alervd on github
Fixes#13695Closes#13943
Allow overriding SOVERSION with the new CMake option:
`CURL_LIBCURL_SOVERSION=ON/OFF`
For certain target platforms the shared libcurl library filename
contains the SOVERSION. This new option allows to enable/disable
this behavior manually. If set, it takes precedence over the default
setting.
Ref: #13898Closes#13944
- When a transfer sets `data->state.select_bits`, it is
scheduled for rerun with EXPIRE_NOW. If such a transfer
is blocked (due to PAUSE, for example), this will lead to
a busy loop.
- multi.c: check for transfer block
- sendf.*: add Curl_xfer_is_blocked()
- sendf.*: add client reader `is_paused()` callback
- implement is_paused()` callback where needed
Closes#13908
A newly introduced use of getsockname() in the cli tool makes it require
the ascii wrapper module, which is not available outside of the library:
as the tool only uses the address family field (binary), disable
wrappers outside of libcurl.
Fix setsockopt() parameter type mismatch using a (void *) cast.
Sync ILE/RPG binding.
Closes#13930
Instead of bolting on the extra CRLF to the final header - as that makes
the behavior inconsistent and not as documented. The final CRLF is now
also made unconditional, just like it is for HTTP.
Reported-by: dogma
Bug: https://curl.se/mail/lib-2024-06/0033.htmlCloses#13925
- clarify Curl_xfer_setup() with RECV/SEND flags and different calls for
which socket they operate on. Add a shutdown flag for secondary
sockets
- change Curl_xfer_setup() calls to new functions
- implement non-blocking connection shutdown at the end of receiving or
sending a transfer
Closes#13913
- new struct curl_pollfds and struct curl_waitfds
- add structs and methods to init/add/cleanup an array of pollfd and
struct curl_waitfd. Use in multi_wait() and multi_waitfds() to
populate the sets for polling.
- place USE_WINSOCK WSAEventSelect() setting into a separate loop over
all collected pfds
Closes#13900
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.
- add cfilter `Curl_cft_shutdown` callback
- keep a shutdown start timestamp and timeout at connectdata
- provide shutdown timeout default and member in
`data->set.shutdowntimeout`.
- provide methods for starting, interrogating and clearing
shutdown timers
- provide `Curl_conn_shutdown_blocking()` to shutdown the
`sockindex` filter chain in a blocking way. Use that in FTP.
- add `Curl_conn_cf_poll()` to wait for socket events during
shutdown of a connection filter chain.
This gets the monitoring sockets and events via the filters
"adjust_pollset()" methods. This gives correct behaviour when
shutting down a TLS connection through a HTTP/2 proxy.
- Implement shutdown for all socket filters
- for HTTP/2 and h2 proxying to send GOAWAY
- for TLS backends to the best of their capabilities
- for tcp socket filter to make a final, nonblocking
receive to avoid unwanted RST states
- add shutdown forwarding to happy eyeballers and
https connect ballers when applicable.
Closes#13904
- decouple need to recv/send from negotiation state, we need
this later in shutdown handling as well
- move ssl enums from urldata.h to vtls_int.h
- implement use of `connssl->io_need` in vtls.c. and all backends
Closes#13879
When reaching the set maximum limit of allowed connections, allow a new
connection anyway if the transfer is created for the (internal) purpose
of doing a DoH name resolve. Otherwise, unrelated "normal" transfers can
starve out new DoH requests making it impossible to name resolve for new
transfers.
Bug: https://curl.se/mail/lib-2024-06/0001.html
Reported-by: kartatz
Closes#13880
Add new job to test building for UWP (aka `CURL_WINDOWS_APP`).
Fix fallouts when building for UWP:
- rand: do not use `BCryptGenRandom()`.
- cmake: disable using win32 LDAP.
- cmake: disable telnet.
- version_win32: fix code before declaration.
- schannel: disable `HAS_MANUAL_VERIFY_API`.
- schannel: disable `SSLSUPP_PINNEDPUBKEY`
and make `schannel_checksum()` a stub.
Ref: e178fbd40a#1429
- schannel: make `cert_get_name_string()` a failing stub.
- system_win32: make `Curl_win32_impersonating()` a failing stub.
- system_win32: try to fix `Curl_win32_init()` (untested).
- threads: fix to use `CreateThread()`.
- src: disable searching `PATH` for the CA bundle.
- src: disable bold text support and capability detection.
- src: disable `getfiletime()`/`setfiletime()`.
- tests: make `win32_load_system_library()` a failing stub.
- tests/server/util: make it compile.
- tests/server/sockfilt: make it compile.
- tests/lib3026: fix to use `CreateThread()`.
See individual commits for build error details.
Some of these fixes may have better solutions, and some may not work
as expected. The goal of this patch is to make curl build for UWP.
Closes#13870
Introduce new notation for CURLOPT_INTERFACE / --interface:
ifhost!<interface>!<host>
Binding to an interface doesn't set the address, and an interface can
have multiple addresses.
When binding to an address (without interface), the kernel is free to
choose the route, and it can route through any device that can access
the target address, not necessarily the one with the chosen address.
Moreover, it is possible for different interfaces to have the same IP
address, on which case we need to provide a way to be more specific.
Factor out the parsing part of interface option, and add unit tests:
1663.
Closes#13719
