Commit Graph

32527 Commits

Author SHA1 Message Date
Stefan Eissing
8d067e61b0
quic: openssl quic, cmake and doc version update to 3.3.0
Closes #14028
2024-06-26 23:10:50 +02:00
Stefan Eissing
bb09a304bb
http/3: add shutdown support
- openssl-quic shutdown handling
- ngtcp2 shutdown handling
- quiche shutdown handling
- add test_19_06 for verfication

Reported-by: Dexter Gerig
Closes #14027
Fixes #14022
2024-06-26 23:09:32 +02:00
Daniel Stenberg
868ae0673c
tests: verify managen
1705: verifies the manpage output

1706: verifies the ascii output

Closes #14025
2024-06-26 15:49:26 +02:00
Daniel Stenberg
0e73b69b3d
runtests: support %DATE for YYYY-MM-DD of right now 2024-06-26 15:49:26 +02:00
Daniel Stenberg
348882ee79
runtests: support %VERNUM
For the plain version number of the built curl without -DEV etc. Only
digits and dots.
2024-06-26 15:49:26 +02:00
Daniel Stenberg
c4187ea5fa
managen: only output .RE for manpage output
For ascii they are just rubbish.

Closes #14025
2024-06-26 15:49:26 +02:00
Tatsuhiro Tsujikawa
a571afc02e
quic: enable UDP GRO
Closes #14012
2024-06-26 15:15:23 +02:00
Stefan Eissing
411af83010
quic: require at least OpenSSL 3.3 for QUIC
- when checking for QUIC support in OpenSSL, also check
  for it being at least 3.3.0
- remove workarounds for features buggy or missing in 3.2

Closes #14026
2024-06-26 13:21:52 +02:00
Daniel Stenberg
4d58ed2d2c
FILEFORMAT.md: mentioned <file[num]> for "client"
They can be used to create more files.

Closes #14024
2024-06-26 10:55:40 +02:00
Marcel Raad
4a512bd3f2
system_win32: add missing curl.h include
It's required for `CURLcode`.

Closes https://github.com/curl/curl/pull/14019
2024-06-26 10:05:32 +02:00
Daniel Stenberg
10cc302090
TODO: specify which response codes that make -f/--fail return error
Suggestion from the user survey 2024

Closes #14020
2024-06-26 08:41:41 +02:00
Stefan Eissing
c9b95c0bb3
lib: graceful connection shutdown
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.

The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.

Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.

As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.

Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.

When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.

The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.

- In multi_wait() and multi_waitfds(), collect all connection caches
  involved (each transfer might carry its own) into a temporary list.
  Let each connection cache on the list contribute sockets and
  POLLIN/OUT events it's connections are waiting for.

- in multi_perform() collect the connection caches the same way and let
  them peform their maintenance. This will make another non-blocking
  attempt to shutdown all connections on its shutdown list.

- for event based multis (multi->socket_cb set), add the sockets and
  their poll events via the callback. When `multi_socket()` is invoked
  for a socket not known by an active transfer, forward this to the
  multi's cache for processing. On closing a connection, remove its
  socket(s) via the callback.

TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.

A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.

- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
  which happen without a graceful shutdown and should
  no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
  on FTP up- and downloads.

Closes #13976
2024-06-26 08:33:17 +02:00
Daniel Stenberg
c1845dc0e2
managen: fix blank line detection
Follow-up to d14a53eea7 which ruined the output somewhat.

Closes #14017
2024-06-25 16:41:54 +02:00
Daniel Stenberg
473ec8a05d
managen: output tabs for each 8 leading spaces
This replacing of eight leading spaces into tabs was already done for
the embedded uncompressed version in tool_hugehelp.c so it does not save
anything there. But the gzip compressed version ends up almost 2K
smaller.

The output in a terminal should be identical.

Before using TABs:

curl.txt 282492 bytes
curl.txt.gz 73261 bytes

With this change applied:

curl.txt 249382 bytes
curl.txt.gz 71470 bytes

Closes #14016
2024-06-25 16:13:30 +02:00
Daniel Stenberg
d14a53eea7
managen: error on trailing blank lines in input files
Ref: #14014
Closes #14015
2024-06-25 16:07:07 +02:00
Viktor Szakats
2d5aea9c93
tidy-up: more whitespace
Closes #14014
2024-06-25 14:40:44 +02:00
Stefan Eissing
def99d8507
multi: multi_getsock(), check correct socket
- in phase CONNECTING/TUNNELING/PROTOCONNECT, retrieve
   the socket from the connection filters and do not rely
   on `conn->sockfd` being already set by the transfer.
 - this applies to the default behaviour, a protocol handler
   may override this via its callbacks.
 - add a warning message in multi_getsock() when the transfer
   is expected to have something in its pollset, but instead
   it is empty.

Reported-by: saurabhsingh-dev on github
Fixes #13998
Closes #14011
2024-06-25 13:15:01 +02:00
Daniel Stenberg
8e3e3921e6
managen: fix each options footer to end with newline
A previous change sometimes made a command line option's description not
end with a newline immediately before the next command line.

Also widened the lines to wrap on column 79 instead of 78.

Closes #14010
2024-06-25 13:11:06 +02:00
Alex Snast
ad3e476275
wolfssl: assume key_file equal to clientcert in the absence of key_file
When user sets CURLOPT_SSLCERT but leaves CURLOPT_SSLKEY unset assume
the path passed in CURLOPT_SSLCERT holds the ssl key which is what we do
in openssl implementation.

Fixes #14007
Closes #14008
2024-06-25 10:01:37 +02:00
Viktor Szakats
b7b41b1ea9
autotools: fix pkg-config names (zstd, ngtcp2*)
Also verified that all names now match up with CMake.

Follow-up to f057de5a1a #13911
Follow-up to eeab0ea7aa #13994
Reported-by: 李四
Fixes #14005
Closes #14006
2024-06-24 23:25:18 +02:00
Viktor Szakats
ff92e3b13c
tidy-up: whitespace [ci skip] 2024-06-24 20:26:34 +02:00
Daniel Stenberg
1f1975b84c
cmdline-docs: "added in" cleanups
- markup fixes
- remove some mentions of < 7.60.0 changes

Closes #14003
2024-06-24 16:56:52 +02:00
Daniel Stenberg
d28ecf9439
RELEASE-NOTES: synced 2024-06-24 16:34:14 +02:00
Daniel Stenberg
7628502dff
managen: "added in" fixes
- up the limit: remove all mentions of 7.60 or earlier from manpage
   7.60 is 6 years old now.
 - warn on "broken" added in lines, as they avoid detection
 - fixup added in markup in a few curldown files

Closes #14002
2024-06-24 16:20:03 +02:00
Matt Jolly
eeab0ea7aa
configure: fix pkg-config library name 'libnghttp3'
Closes #13994
2024-06-24 13:18:57 +02:00
Daniel Stenberg
1c9250c46d
managen: cleanups to generate nicer-looking output
- output "see also" last
- when there are multiple mutex items, use commas between all of them
  except the last.
- call them mututally exclusive WITH not TO other options.
- remove trailing space from added in, add newline prefix
- smoother language for requires

Closes #14001
2024-06-24 13:17:19 +02:00
Daniel Stenberg
76e0196b1e
configure: require a QUIC library if nghttp3 is used
Instead of just silently disabling HTTP/3.

Reported-by: Matt Jolly
Fixes #13995
Closes #13999
2024-06-24 13:14:11 +02:00
Daniel Stenberg
134d2834fd
docs/cmdline-opts: remove two superfluous "Added in" mentions
The key "added in" phrase for the option itself is added automatically.

Closes #14000
2024-06-24 11:18:53 +02:00
Daniel Stenberg
179c524040
cookie-jar.md: see also --junk-session-cookies
Closes #13996
2024-06-24 11:17:41 +02:00
Daniel Stenberg
a7ec6a76ab
runtests: support crlf="yes" for the <stderr> section 2024-06-24 08:43:40 +02:00
Daniel Stenberg
fa0b1d9209
TODO: -h option
Support "curl -h --insecure" etc to output the manpage section for the
--insecure command line option in the terminal. Should be possible to
work with either long or short versions of command line options.

Closes #13990
2024-06-23 23:07:13 +02:00
Daniel Stenberg
15e8cebd64
trace-ascii.md: mention "%" for stderr
Closes #13991
2024-06-23 23:06:21 +02:00
Daniel Stenberg
857e334a45
connect-to.md: expand with examples
- add referer from the resolve section to connect-to if user wants
  wildcard for the port number

Closes #13989
2024-06-23 15:52:06 +02:00
Daniel Stenberg
d5777afac6
TODO: connect to multiple IPs in parallel
Closes #13986
2024-06-23 14:31:46 +02:00
Daniel Stenberg
674cbfd613
dump-header.md: mention minus for stdout
Closes #13985
2024-06-23 12:08:58 +02:00
Daniel Stenberg
b84699bf4f
CURLOPT_RESOLVE.md: mention hostname can be wildcard ('*')
Closes #13983
2024-06-23 10:29:22 +02:00
Andy Pan
bd8e45c6c5
cf-socket: optimize curlx_nonblock() and check its return error
Reviewed-by: Stefan Eissing
Closes #13942
2024-06-22 11:56:37 +02:00
z2_
36e9222c42
x509asn1: prevent NULL dereference
Closes #13978
2024-06-22 11:51:27 +02:00
Daniel Stenberg
df2fbc2777
unit2604: use 'unitfail' instead of 'error' variable
Since the framework is already returning that variable by default.
Avoids a warning for unreachable code.

Reported-by: Tal Regev
Fixes #13967
Closes #13973
2024-06-19 13:37:36 +02:00
Daniel Stenberg
01159b36c2
KNOWN_BUGS: TFTP tests fail on OpenBSD
Closes #13623
Closes #13975
2024-06-19 12:54:19 +02:00
Daniel Stenberg
b715bb371c
VULN-DISCLOSURE-POLICY: NULL dereferences and crashes
If a malicious server can trigger a NULL dereference in curl or
otherwise cause curl to crash (and nothing worse), chances are big that
we do not consider that a security problem.

Closes #13974
2024-06-19 12:53:35 +02:00
Daniel Stenberg
b4ccf3fcbe
RELEASE-NOTES: synced 2024-06-19 11:36:39 +02:00
Sergey Markelov
35c0117f47
mbedtls: support CURLOPT_CERTINFO
Closes #13113
2024-06-19 11:33:12 +02:00
Daniel Stenberg
cb96ca1b64
x509asn1: ASN1tostr() should fail when 'constructed' is set
This is a regression from my refactor in 623c3a8fa0 (#12808)

Follow-up to 623c3a8fa0

Closes #13972
2024-06-19 11:23:53 +02:00
Daniel Stenberg
dc497d7b8f
x509asn1: remove two static variables
cnOID and sanOID were not used outside of the OID table anyway

Closes #13971
2024-06-19 11:22:52 +02:00
brian m. carlson
04739054cd
TODO: TLS channel binding
Closes #13483
2024-06-18 10:30:24 +02:00
Tal Regev
66bf995d1c
cmake: add CURL_USE_GSASL option with detection + CI test
Reviewed-by: Viktor Szakats
Closes #13948
2024-06-17 23:10:02 +02:00
Daniel Stenberg
8dc4493d54
x509asn1: make Curl_extract_certinfo store error message
To help us all better understand where the error actually comes from.

Ref: #13958
Closes #13959
2024-06-16 23:48:11 +02:00
Viktor Szakats
92f42761d2
appveyor: dump build logs on failure in VS2008 jobs
This seems to be the only way to see what actual toolchain commands were
run, and with what arguments.

Without `dos2unix`, `cat` output comes out empty.

Closes #13957
2024-06-15 21:04:55 +02:00
Viktor Szakats
b37040dd60
cmake: fix quotes when appending multiple options (SecureTransport)
Copied from a vcpkg distro patch:
02745e0f47/ports/curl/dependencies.patch (L43C38-L44)

Ref: https://github.com/microsoft/vcpkg/pull/38847
Ref: 795f2f137e
Ref: 36f0c917de (diff-ab5c23e5dc5df412539cc93e24b37abbc588e1918236f8abc019d676b270c85fR39) (sub-commit)

Authored-by: Kai Pastor
Closes #13953
2024-06-15 17:45:37 +02:00