Go to file
Stefan Eissing c9b95c0bb3
lib: graceful connection shutdown
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.

The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.

Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.

As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.

Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.

When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.

The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.

- In multi_wait() and multi_waitfds(), collect all connection caches
  involved (each transfer might carry its own) into a temporary list.
  Let each connection cache on the list contribute sockets and
  POLLIN/OUT events it's connections are waiting for.

- in multi_perform() collect the connection caches the same way and let
  them peform their maintenance. This will make another non-blocking
  attempt to shutdown all connections on its shutdown list.

- for event based multis (multi->socket_cb set), add the sockets and
  their poll events via the callback. When `multi_socket()` is invoked
  for a socket not known by an active transfer, forward this to the
  multi's cache for processing. On closing a connection, remove its
  socket(s) via the callback.

TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.

A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.

- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
  which happen without a graceful shutdown and should
  no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
  on FTP up- and downloads.

Closes #13976
2024-06-26 08:33:17 +02:00
.circleci Add some basic versioning for some workflows to check whether this is detected properly 2024-05-13 22:34:46 +01:00
.github VULN-DISCLOSURE-POLICY: NULL dereferences and crashes 2024-06-19 12:53:35 +02:00
.reuse reuse: migrate standalone license file to dep5 2024-05-15 14:15:25 +02:00
CMake cmake: bring curl-config.cmake closer to FindCURL 2024-06-13 11:17:32 +02:00
docs lib: graceful connection shutdown 2024-06-26 08:33:17 +02:00
include urlapi: use a correct value for CURLU_NO_GUESS_SCHEME 2024-06-12 13:14:03 +02:00
lib lib: graceful connection shutdown 2024-06-26 08:33:17 +02:00
LICENSES
m4 configure: use AC_MSG_WARN for TLS/experimental warning texts 2024-06-13 17:17:15 +02:00
packages tidy-up: more whitespace 2024-06-25 14:40:44 +02:00
plan9
projects tidy-up: use consistent casing for Windows directories 2024-05-30 14:40:12 +02:00
scripts managen: fix blank line detection 2024-06-25 16:41:54 +02:00
src lib: graceful connection shutdown 2024-06-26 08:33:17 +02:00
tests lib: graceful connection shutdown 2024-06-26 08:33:17 +02:00
winbuild winbuild: remove outdated WIN32 defines 2024-05-24 03:23:14 -04:00
.azure-pipelines.yml CI: disable dependency tracking in most autotools builds 2024-05-27 22:25:14 +02:00
.cirrus.yml CI: reduce memory request for FreeBSD builds 2024-06-03 17:00:42 -07:00
.dcignore
.dir-locals.el
.git-blame-ignore-revs copyright: update all copyright lines and remove year ranges 2023-01-03 09:19:21 +01:00
.gitattributes
.gitignore TLS: add support for ECH (Encrypted Client Hello) 2024-04-16 08:10:53 +02:00
.mailmap .mailmap: update Gisle's preferred email 2024-04-09 08:50:07 +02:00
acinclude.m4
appveyor.sh appveyor: dump build logs on failure in VS2008 jobs 2024-06-15 21:04:55 +02:00
appveyor.yml build: untangle CURLDEBUG and DEBUGBUILD macros 2024-05-28 08:12:00 +02:00
buildconf
buildconf.bat buildconf.bat: remove outdated groff/nroff use 2024-03-07 22:38:16 +01:00
CHANGES
CMakeLists.txt tidy-up: more whitespace 2024-06-25 14:40:44 +02:00
configure.ac autotools: fix pkg-config names (zstd, ngtcp2*) 2024-06-24 23:25:18 +02:00
COPYING
curl-config.in curl-config: revert to backticks to support old target envs 2024-06-04 10:13:21 +02:00
Dockerfile Dockerfile: update debian:bookworm-slim to 84d83b2 2024-06-13 17:20:00 +02:00
GIT-INFO.md GIT-INFO: convert to markdown 2024-03-07 09:43:33 +01:00
libcurl.def lib: add curl_multi_waitfds 2024-04-09 16:53:40 +02:00
libcurl.pc.in libcurl.pc: add Requires.private, Requires for static linking 2024-06-13 11:17:33 +02:00
Makefile.am Dockerfile: for release automation and reproducibility 2024-04-16 16:53:25 +02:00
Makefile.dist
maketgz projects: drop MSVC project files for recent versions 2024-04-10 07:55:24 +02:00
README
README.md
RELEASE-NOTES RELEASE-NOTES: synced 2024-06-24 16:34:14 +02:00
renovate.json GHA: unify http3 workflows into one 2024-06-01 10:57:23 +02:00
SECURITY.md

curl logo

Curl is a command-line tool for transferring data specified with URL syntax. Find out how to use curl by reading the curl.1 man page or the MANUAL document. Find out how to install Curl by reading the INSTALL document.

libcurl is the library curl is using to do its job. It is readily available to be used by your software. Read the libcurl.3 man page to learn how.

You can find answers to the most frequent questions we get in the FAQ document.

Study the COPYING file for distribution terms.

Contact

If you have problems, questions, ideas or suggestions, please contact us by posting to a suitable mailing list.

All contributors to the project are listed in the THANKS document.

Commercial support

For commercial support, maybe private and dedicated help with your problems or applications using (lib)curl visit the support page.

Website

Visit the curl website for the latest news and downloads.

Git

To download the latest source from the Git server, do this:

git clone https://github.com/curl/curl.git

(you will get a directory named curl created, filled with the source code)

Security problems

Report suspected security problems via our HackerOne page and not in public.

Notice

Curl contains pieces of source code that is Copyright (c) 1998, 1999 Kungliga Tekniska Högskolan. This notice is included here to comply with the distribution terms.

Backers

Thank you to all our backers! 🙏 Become a backer.

Sponsors

Support this project by becoming a sponsor.