Fix issues detected.
Also:
- One of the `.vc` files used LF EOLs, while the other didn't.
Make that one also use LF EOLs, as this is apparently supported by
`nmake`.
- Drop `.dsw` and `.btn` types from `.gitattributes`.
The repository doesn't use them.
- Sync section order with the rest of files in
`tests/certs/EdelCurlRoot-ca.prm`.
- Indent/align `.prm` and `.pem` files.
- Delete dummy `[something]` section from `.prm` and `.pem` files.
Mental note:
MSVC `.sln` files seem to accept spaces for indentation and also support
LF line-endings. I cannot test this and I don't know what's more
convenient when updating them, so left them as-is, with specific
exclusions.
Closes#14031
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.
The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.
Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.
As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.
Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.
When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.
The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.
- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.
- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.
- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.
TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.
A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.
- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.
Closes#13976
The function we use is called 'gnutls_x509_crt_check_hostname()' but if
we pass in the hostname with a trailing dot, the check fails. If we pass
in the SNI name, which cannot have a trailing dot, it succeeds for
https://pyropus.ca./
I consider this as a flaw in GnuTLS and have submitted this issue
upstream:
https://gitlab.com/gnutls/gnutls/-/issues/1548
In order to work with old and existing GnuTLS versions, we still need
this change no matter how they view the issue or might change it in the
future.
Fixes#13428
Reported-by: Ryan Carsten Schmidt
Closes#13949
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.
- add cfilter `Curl_cft_shutdown` callback
- keep a shutdown start timestamp and timeout at connectdata
- provide shutdown timeout default and member in
`data->set.shutdowntimeout`.
- provide methods for starting, interrogating and clearing
shutdown timers
- provide `Curl_conn_shutdown_blocking()` to shutdown the
`sockindex` filter chain in a blocking way. Use that in FTP.
- add `Curl_conn_cf_poll()` to wait for socket events during
shutdown of a connection filter chain.
This gets the monitoring sockets and events via the filters
"adjust_pollset()" methods. This gives correct behaviour when
shutting down a TLS connection through a HTTP/2 proxy.
- Implement shutdown for all socket filters
- for HTTP/2 and h2 proxying to send GOAWAY
- for TLS backends to the best of their capabilities
- for tcp socket filter to make a final, nonblocking
receive to avoid unwanted RST states
- add shutdown forwarding to happy eyeballers and
https connect ballers when applicable.
Closes#13904
- add special sauce to disable unwanted peer verification by mbedtls
when negotiating TLS v1.3
- add special sauce for MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET
return code on *writing* TLS data. We assume the data had not been
written and EAGAIN.
- return correct Curl error code when peer verification failed.
- disable test_08_05 with 50 HTTP/1.1 connections, as mbedtls reports a
memory allocation failed during handshake.
- bump CI mbedtls version to 3.6.0
Fixes#13653Closes#13838
- add 2 variations on test_07_42 which PAUSEs uploads
and response connections terminating either right away
or after the 100-continue response
- when detecting the connection being closed in transfer.c
readwrite_data(), clear ALL send bits in data->req.keepon.
It no longer makes send to wait for a KEEP_SEND_PAUSE or HOLD.
- in the protocol client writer add the check for incomplete
response bodies. When an EOS is seen and the length is known,
check that and fail if bytes are missing.
Reported-by: Sergey Bronnikov
Fixes#13740Closes#13750
- refs #13556
- allow anon uploads on vsftpd test server
- add test_30_05 for plain upload of 1k, 100k, 1m
- add test_31_05 for SSL upload of 1k, 100k, 1m
- verify file size and contents
Closes#13734
Fixes:
- in uds tests, abort also silently on os errors
- be conservative on the h3 goaway duration
- detect curl debug build and use in checks
- fix caddy version check for slight difference under linux
- set caddy default path fitting for linux
- fix deprecation warnings in valid time checks
FTP tests:
- add '--with-test-vsftpd=path' to configure
- use vsftpd default path suitable for linux
- add test_30 with plain FTP tests
- add test_31 with --ssl-reqd FTP tests
- add vsftpd to linux GHA for pytest workflows
Closes#13661
- identify ngtcp2 and nghttp3 error codes that are fatal
- close quic connection on fatal errors
- refuse further filter operations once connection is closed
- confusion about the nghttp3 API. We should close the QUIC stream on
cancel and not use the nghttp3 calls intended to be invoked when the
QUIC stream was closed by the peer.
Closes#13562
- connect to DNS names with trailing dot
- connect to DNS names with double trailing dot
- rustls, always give `peer->hostname` and let it
figure out SNI itself
- add SNI tests for ip address and localhost
- document in code and TODO that QUIC with ngtcp2+wolfssl
does not do proper peer verification of the certificate
- mbedtls, skip tests with ip address verification as not
supported by the library
Closes#13486
- quiche: error transfers that try to receive on a closed
or draining connection
- ngtcp2: use callback for extending max bidi streams. This
allows more precise calculation of MAX_CONCURRENT as we
only can start a new stream when the server acknowledges
the close - not when we locally have closed it.
- remove a fprintf() from h2-download client to avoid excess
log files on tests timing out.
Closes#13475
- add session with destructor callback
- remove vtls `session_free` method
- let `Curl_ssl_addsessionid()` take ownership
of session object, freeing it also on failures
- change tls backend use
- test_17, add tests for SSL session resumption
Closes#13386
- errors returned by Curl_xfer_write_resp() and the header variant are
not errors in the protocol. The result needs to be returned on the
next recv() from the protocol filter.
- make xfer write errors for response data cause the stream to be
cancelled
- added pytest test_02_14 and test_02_15 to verify that also for
parallel processing
Reported-by: Laramie Leavitt
Fixes#13411Closes#13424
- fix flow handling in ngtcp2 to ACK data on streams
we abort ourself.
- extend test_02_23* cases to also run for h3
- skip test_02_23* for OpenSSL QUIC as it gets stalled
on progressing the connection
Closes#13374
- remember error encountered in invoking write callback and always fail
afterwards without further invokes
- check behaviour in test_02_17 with h2-pausing client
Reported-by: Pavel Kropachev
Fixes#13337Closes#13340
- h2-download now always opens the output file on first write callback
invocation, if it will pause the transfer or not.
- Checks on output files then does not depend on the amount of data curl
has collected for the first write.
Closes#13323
- When the writing of response data fails, reset the stream
and do not return a callback error to nghttp2. That would
be a fatal error for the connection and harm other requests.
- add test cases for various abort scenarios
Reported-by: Konstantin Kuzov
Fixes#13292Closes#13298
A transfer with a completed download that is still uploading needs to
check the connection state when it is PAUSEd, since connection
close/errors would otherwise go unnoticed.
Reported-by: Sergey Bronnikov
Fixes#13260Closes#13271
- curl's transfer handling may write 0-length chunks at the end of the
download with an EOS flag. (HTTP/2 does this commonly)
- content encoders need to pass-through such a write and not count this
as error in case they are finished decoding
Fixes#13209Fixes#13212Closes#13219
Move all handling of HTTP's `Expect: 100-continue` feature into a client
reader. Add sending flag `KEEP_SEND_TIMED` that triggers transfer
sending on general events like a timer.
HTTP installs a `CURL_CR_PROTOCOL` reader when announcing `Expect:
100-continue`. That reader works as follows:
- on first invocation, records time, starts the `EXPIRE_100_TIMEOUT`
timer, disables `KEEP_SEND`, enables `KEEP_SEND_TIMER` and returns 0,
eos=FALSE like a paused upload.
- on subsequent invocation it checks if the timer has expired. If so, it
enables `KEEP_SEND` and switches to passing through reads to the
underlying readers.
Transfer handling's `readwrite()` will be invoked when a timer expires
(like `EXPIRE_100_TIMEOUT`) or when data from the server arrives. Seeing
`KEEP_SEND_TIMER`, it will try to upload more data, which triggers
reading from the client readers again. Which then may lead to a new
pausing or cause the upload to start.
Flags and timestamps connected to this have been moved from
`SingleRequest` into the reader's context.
Closes#13110
This fixes miscellaneous typos and duplicated words in the docs, lib
and test comments and a few user facing errorstrings.
Author: RainRat on Github
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Dan Fandrich <dan@coneharvesters.com>
Closes: #13019
- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
clarify when and at what level they operate
- send/recv of transfer related data is now done via
`Curl_xfer_send()/Curl_xfer_recv()` which no longer has
socket/socketindex as parameter. It decides on the transfer
setup of `conn->sockfd` and `conn->writesockfd` on which
connection filter chain to operate.
- send/recv on a specific connection filter chain is done via
`Curl_conn_send()/Curl_conn_recv()` which get the socket index
as parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for
naming consistency
- clarify that the special CURLE_AGAIN hangling to return
`CURLE_OK` with length 0 only applies to `Curl_xfer_send()`
and CURLE_AGAIN is returned by all other send() variants.
- fix a bug in websocket `curl_ws_recv()` that mixed up data
when it arrived in more than a single chunk (to be made
into a sperate PR, also)
Added as documented [in
CLIENT-READER.md](5b1f31dfba/docs/CLIENT-READERS.md).
- old `Curl_buffer_send()` completely replaced by new `Curl_req_send()`
- old `Curl_fillreadbuffer()` replaced with `Curl_client_read()`
- HTTP chunked uploads are now formatted in a client reader added when
needed.
- FTP line-end conversions are done in a client reader added when
needed.
- when sending requests headers, remaining buffer space is filled with
body data for sending in "one go". This is independent of the request
body size. Resolves#12938 as now small and large requests have the
same code path.
Changes done to test cases:
- test513: now fails before sending request headers as this initial
"client read" triggers the setup fault. Behaves now the same as in
hyper build
- test547, test555, test1620: fix the length check in the lib code to
only fail for reads *smaller* than expected. This was a bug in the
test code that never triggered in the old implementation.
Closes#12969
- when data arrived in several chunks, the collection into
the passed buffer always started at offset 0, overwriting
the data already there.
adding test_20_07 to verify fix
- debug environment var CURL_WS_CHUNK_SIZE can be used to
influence the buffer chunk size used for en-/decoding.
Closes#12945
- add test_05_04 for requests using http/1.0, http/1.1 and h2 against an
Apache resource that does an unclean TLS shutdown.
- revert special workarund in openssl.c for suppressing shutdown errors
on multiplexed connections
- vlts.c restore to its state before 9a90c9dd64Fixes#12885Fixes#12844Closes#12848
- improve info logging when peer verification fails to indicate
if DNS name or ip address has been tried to match
- add test case for contacting https proxy with ip address
- add pytest env check on loaded credentials and re-issue
when they are no longer valid
- disable proxy ip address test for bearssl, since not supported there
Ref: #12831Closes#12838
