Commit Graph

6045 Commits

Author SHA1 Message Date
Austin Moore
cf3e3d93d1
aws_sigv4: fix canon order for headers with same prefix
If a request containing two headers that have equivalent prefixes (ex.
"x-amz-meta-test:test" and "x-amz-meta-test-two:test2") AWS expects the
header with the shorter name to come first. The previous implementation
used `strcmp` on the full header. Using the example, this would result
in a comparison between the ':' and '-' chars and sort
"x-amz-meta-test-two" before "x-amz-meta-test", which produces a
different "StringToSign" than the one calculated by AWS.

Test 1976 verifies

Closes #14370
2024-08-05 23:32:10 +02:00
Jan Venekamp
4c12892411
wolfssl: add CURLOPT_TLS13_CIPHERS support
Bring setting ciphers with WolfSSL in line with other SSL backends,
to make the curl interface more consistent across the backends.

Now the tls1.3 ciphers are set with the --tls13-ciphers option, when
not set the default tls1.3 ciphers are used. The tls1.2 (1.1, 1.0)
ciphers are set with the --ciphers option, when not set the default
tls1.2 ciphers are used. The ciphers available for the connection
are now a union of the tls1.3 and tls1.2 ciphers.

This changes the behaviour for WolfSSL when --ciphers is set, but
--tls13-ciphers is not set. Now the ciphers set with --ciphers
are combined with the default tls1.3 ciphers, whereas before solely
the ciphers of --ciphers were used.

Thus before when no tls1.3 ciphers were specified in --ciphers,
tls1.3 was completely disabled. This might not be what the user
expected, especially as this does not happen with OpenSSL.

Closes #14385
2024-08-05 23:28:04 +02:00
Jan Venekamp
3f7dc8a404
mbedtls: add CURLOPT_TLS13_CIPHERS support
Bring setting ciphers with mbedTLS in line with other SSL backends,
to make the curl interface more consistent across the backends.

Now the tls1.3 ciphers are set with the --tls13-ciphers option, when
not set the default tls1.3 ciphers are used. The tls1.2 (1.1, 1.0)
ciphers are set with the --ciphers option, when not set the default
tls1.2 ciphers are used. The ciphers available for the connection
are now a union of the tls1.3 and tls1.2 ciphers.

This changes the behaviour for mbedTLS when --ciphers is set, but
--tls13-ciphers is not set. Now the ciphers set with --ciphers
are combined with the default tls1.3 ciphers, whereas before solely
the ciphers of --ciphers were used.

Thus before when no tls1.3 ciphers were specified in --ciphers,
tls1.3 was completely disabled. This might not be what the user
expected, especially as this does not happen with OpenSSL.

Closes #14384
2024-08-05 16:01:20 +02:00
Daniel Stenberg
38d334e3e1
curl: use libuv for parallel transfers with --test-event
add --with-libuv to configure to (optionally) use it in debug-builds to
drive the event-based API

Use curl_multi_socket_action() and friends to drive parallel transfers.

tests/README has brief documentation for this

Closes #14298
2024-08-05 09:12:09 +02:00
Stefan Eissing
35bf766280
http2: improved upload eos handling
- replace the counting of upload lengths with the new eos send flag
- improve frequency of stream draining to happen less on events where it
  is not needed
- this PR is based on #14220

http2, cf-h2-proxy: fix EAGAINed out buffer
- in adjust pollset and shutdown handling, a non-empty `ctx->outbufq`
  must trigger send polling, irregardless of http/2 flow control
- in http2, fix retry handling of blocked GOAWAY frame

test case improvement:
- let client 'upload-pausing' handle http versions

Closes #14253
2024-08-05 08:53:55 +02:00
Daniel Stenberg
7d45b52166
KNOWN_BUGS: mention AppleIDN and WinIDN test problems
Closes #14176
Closes #14387
2024-08-05 08:23:37 +02:00
Daniel Stenberg
732cb15b97
curl: add --skip-existing
With this option, the entire download is skipped if the selected target
filename already exists when the opertion is about to begin.

Test 994, 995 and 996 verify.

Ref: #11012
Closes #13993
2024-08-04 23:28:09 +02:00
Jan Venekamp
eec908bb6e
revert "tests/http: configure test httpd to honor client cipher order"
revert f6cb3c63 #14338

Setting SSLHonorCipherOrder to on means it honors the server cipher
order. From the documentation: "When choosing a cipher during an SSLv3
or TLSv1 handshake, normally the client's preference is used. If this
directive is enabled, the server's preference will be used instead."

Also the commit inhibits test_17_07_ssl_ciphers. The test tries to
tests if all the ciphers specified, and only those, are properly set
in curl. For that to work we need have cases where some or all ciphers
do no intersect with the cipher-set of the server. We need to be able
to assert a failed connection based on a cipher set mismatch.

That is why a restricted set of ciphers is used on the server. This
set is so chosen that it contains the well known most secure ciphers.
Except with the slower aes256 variant intentionally left out, to be
able to test above described.

As test_17_07_ssl_ciphers is currently the only test that tests the
functioning of the --ciphers and --tls13-ciphers options, it is
important that its coverage is as good as possible.

Closes #14381
2024-08-04 23:03:46 +02:00
Viktor Szakats
f81f351b9a
tidy-up: OS names
Use these words and casing more consistently across text, comments and
one curl tool output:
AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,
macOS, MS-DOS, MSYS, MinGW, NTLM, POSIX, Solaris, UNIX, Unix, Unicode,
WINE, WebDAV, Win32, winbind, WinIDN, Windows, Windows CE, Winsock.

Mostly OS names and a few more.

Also a couple of other minor text fixups.

Closes #14360
2024-08-04 19:17:45 +02:00
Daniel Stenberg
9a0cf56471
curl: --help [option] displays documentation for given cmdline option
Since the documentation text blob might be gzipped, it needs to search
for what to output in a streaming manner. It then first searches for
"\nALL OPTIONS".

Then, it looks for the start to display at "\n    -[option]" and stops
again at "\n    -". Except for the last option in the man page, which
ends at "\nFILES" - the subtitle for the section following all options
in the manpage.

Test 1707 to 1710 verify

Closes #13997
2024-08-04 16:06:17 +02:00
Daniel Stenberg
9b1e4b4632
tool_operate: support --dump-header % to direct to stderr
Similar to how --trace and --trace-ascii already do it.

Added test 1489 to verify

Closes #13992
2024-08-04 16:02:24 +02:00
Daniel Stenberg
e26eefd9ce
tool_operate: for -O, use "default" as filename when the URL has none
... or pick the last directory part from the path if available.

Instead of returning error.

Add test 690 and 691 to verify. Test 76 and 2036 no longer apply.

Closes #13988
2024-08-04 15:59:51 +02:00
Viktor Szakats
4f198c8525
tests: fixup tests/data/Makefile.am references
Follow-up to f5b826532f #14357
Reported-by: Stefan Eissing
Fixes #14371
Closes #14372
2024-08-04 15:28:29 +02:00
Stefan Eissing
93d1af4011
pytests: add tests for HEAD requests in all HTTP versions
Closes #14367
2024-08-04 00:20:14 +02:00
Viktor Szakats
acbc6b703f
cmake: tidy-ups
- tidy-up comments.
- use lowercase, underscore prefixed names for internal variables.
- use `IN LISTS` and `IN ITEMS` in `foreach()` loops.
- rename variable name `OUTPUT` to a more distinctive one.
- tidy-up `STREQUAL` syntax.
- delete commented code.
- indent/whitespace.

Closes #14197
2024-08-03 20:49:15 +02:00
Daniel Stenberg
82c53f821f
tool_getparam: make --show-headers the same as --include
Simply a name alias that better explains what the option does.

Closes #13987
2024-08-03 19:56:57 +02:00
Stefan Eissing
5a9262a333
url: dns_entry related improvements
Replace Curl_resolv_unlock() with Curl_resolv_unlink():

-replace inuse member with refcount in Curl_dns_entry

- pass Curl_dns_entry ** to unlink, so it gets always cleared

- solve potential (but unlikley) UAF in FTP's handling of looked up
  Curl_dns_entry. Esp. do not use addr information after unlinking an entry.
  In reality, the unlink will not free memory, as the dns entry is still
  referenced by the hostcache. But this is not safe and relying on no other
  code pruning the cache in the meantime.

- pass permanent flag when adding a dns entry instead of fixing timestamp
  afterwards.

url.c: fold several static *resolve_* functions into one.

Closes #14195
2024-08-03 19:51:02 +02:00
Dan Fandrich
0324d557e4
CI: enable parallel testing in CI builds
The test-ci target now uses 2 processes by default, but the amount of
parallelism is tuned for each CI service and build environment based on
results of a number of test runs.  Some CI services use super-
oversubscribed build machines that can barely run the curl tests
already with no parallelism without frequently failing with
timing-induced failures. These continue to be run without parallelism.
Other services provide two fast, unloaded cores and these run with 14
processes, which is a good default for this kind of environment.

Here's a summary of the number of test processes by CI service:

  Appveyor - 2 (Windows MSVC), 1 (others)
  Azure - 2
  Circle CI - 14
  Cirrus - 28 (macOS), 14 (Linux), 7 (FreeBSD), 5 (macOS torture), 2 (Windows)
  GitHub Actions - 3 (macOS), 2 (Linux)

Some of these are a bit conservative to keep timing-induced flakiness down.

The net result is that the first test results should arrive only
3 minutes after a commit submission.

Changes merged via separate commits:
- 2a7c8b27fd #14171
- 72341068a2
- efce544418 #14244
- c6cf411bac

Ref: #10818
Closes #11510
2024-08-03 16:12:44 +02:00
Viktor Szakats
f518c73a87
cmake: drop unused internal variable
Closes #14361
2024-08-03 09:22:26 +02:00
Viktor Szakats
f5b826532f
cmake: drop no-op tests/data/CMakeLists.txt
It's a no-op since
d162fca69a #9333 (2022-08-18).

Also revert 476499c75c that is
no longer necessary: move `Makefile.inc` back into `Makefile.am`.

Closes #14357
2024-08-03 01:15:45 +02:00
Stefan Eissing
39b9ccea8d
x509asn1: raise size limit for x509 certification information
Raise the limit for certification information from 10 thousand to 100
thousand bytes. Certificates can be larger than 10k.

Change the infof() debug output to add '...' at the end when the max
limit it can handle is exceeded.

Reported-by: Sergio Durigan Junior
Fixes #14352
Closes #14354
2024-08-02 23:20:57 +02:00
Viktor Szakats
4d34fd26db
autotools: fix typo in tests/data target
Ref: c1b369fd4c (2002-12-12)
Closes #14355
2024-08-02 14:36:16 +02:00
Stefan Eissing
45246ebca1
tests: gitignore newly generated files
Closes #14351
2024-08-02 11:19:10 +02:00
Stefan Eissing
eb0a366b70
http2: improve rate limiting of downloads
Set the initial stream window size to 64KB and increase that to the 10MB
we used to start with on the first server reply, unless a rate limit is
in effect.

Continously monitory changes to the transfers rate limit and adjust the
stream window size accordingly. `max_recv_speed` is a transfer propert
that can be changed during processing by a callback.

Closes #14326
2024-08-02 08:45:01 +02:00
Stefan Eissing
f6cb3c6308
tests/http: configure test httpd to honor client cipher order
Let the client, e.g. curl, influence the cipher selected in a TLS
handshake. TLS backends have different preferences and honor that
in httpd the same as Caddy does.

Also makes for a more fair compare of different TLS backends.

Closes #14338
2024-08-02 08:32:34 +02:00
Daniel Stenberg
8f89218b19
tests: provide docs a as curldown, not nroff
As runtests.md and testcurl.md. Very few people actually need these as
manpages anyway.

With this, we have no more nroff formatted documents in git.

Closes #14324
2024-07-31 14:51:12 +02:00
Joe Birr-Pixton
dd95a49d49
rustls: make all tests pass
- supports IP addresses in certs since 0.21
- Remove workaround, and re-enable tests

Closes #14317
2024-07-31 10:44:10 +02:00
Stefan Eissing
27959ecce7
x509asn1: unittests and fixes for gtime2str
Fix issues in GTime2str() and add unit test cases to verify correct
behaviour.

Follow-up to 3c914bc680

Closes #14316
2024-07-30 23:08:59 +02:00
Ivan Kuchin
b2696ac37a
misc: cleanup after removing years from copyright
- remove leftover copyright years from few test files
- fix email in copyright lines
- consistent format of copyright lines

Closes #14312
2024-07-30 14:11:48 +02:00
Viktor Szakats
efce544418
GHA/non-native: improve, migrate x86_64 FreeBSD with tests from Cirrus CI
- run tests via `make test-ci` instead of `make check` with autotools.
- add `x86_64` job for FreeBSD, with tests.
  It matches the existing Cirrus CI job, with these differences:
  - finishes 3x faster (thanks to parallel tests enabled).
  - librtmp is not enabled because it's slated for removal by FreeBSD.
    (already past the removal deadline, thought the package still
    installs.)
  - DICT and TELNET servers fail to start. Couldn't figure out why.
    It means skipping test 1450 and 1452.
  - it runs more tests, e.g. websockets and ip6-localhost.
  - no `pkg update -f`.
  - it misses the `CRYPTOGRAPHY_DONT_BUILD_RUST=1`, `pkg delete curl`,
    `chmod 777`, `sudo -u nobody` and `sysctl net.inet.tcp.blackhole`
    tricks. The latter is the default in these runners, the others did
    not affect results.
- set `-j0` for tests in the NetBSD job. Flaky otherwise.

Closes #14244
2024-07-29 20:46:40 +02:00
Daniel Stenberg
33826994e7
test993: list 1000 messages over POP3
Attempt to verify issue #14201

Closes #14297
2024-07-29 14:55:06 +02:00
Daniel Stenberg
14f630ecf6
tests: provide FTP directory contents in the test file
Instead of providing a fixed single synthetic response in the test
server itself. To allow us to better use *different* directory listings
in different test cases. In this change, most listings remain the same
as before.

The wildcard match tests still use synthetic responses but we should fix
that as well.

Updated numerous test cases to use this.

Closes #14295
2024-07-29 13:04:24 +02:00
Daniel Stenberg
ae0b08ae6e
ftpserver.pl: make POP3 LIST serve content from the test file
instead of a fixed list in the test server.

Adjust test 853 accordingly.

Closes #14293
2024-07-29 10:41:26 +02:00
Viktor Szakats
1565c02ab4
runtests: fold timing details with GHA, sync -r tflags
- move timing details into a foldable group when run in GitHub Actions.
  Spec:
  https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#grouping-log-lines

- enable `-r` (run time stats) option in autotools' `test-ci` target,
  syncing it with cmake.

Closes #14284
2024-07-27 19:17:31 +02:00
Daniel Stenberg
8c1d9378ac
curldown: make 'added-in:' a mandatory header field
- generate AVAILABILITY manpage sections automatically - for consistent
  wording

- allows us to double-check against other documumentation (symbols-in-versions
  etc)

- enables proper automation/scripting based on this data

- lots of them were wrong or missing in the manpages

- several of them repeated (sometimes mismatching) backend support info

Add test 1488 to verify "added-in" version numbers against
symbols-in-versions.

Closes #14217
2024-07-18 18:04:09 +02:00
Daniel Stenberg
5eba0a4b37
tests/scripts: call it 'manpage' (single word)
Mostly in comments

Closes #14216
2024-07-18 15:12:09 +02:00
Stefan Eissing
35ed3f7a89
pytests: scorecard upload tests
- add upload tests to scorecard, invoke with
  > python3 tests/http/scorecard.py -u h1|h2|h3
- add a reverse proxy setup from Caddy to httpd for
  upload tests since Caddy does not have other PUT/POST handling
- add caddy tests in test_08 for POST/PUT
- increase read buffer in mod_curltest for larger reads

Closes #14208
2024-07-18 14:45:40 +02:00
Viktor Szakats
1dd71312bf
runtests: fix newline glitch in FAIL details
Follow-up to bae5553599 #14174
2024-07-18 01:10:29 +02:00
Viktor Szakats
bae5553599
runtests: show name and keywords for failed tests in summary
Useful to see what the numbers listed in the `TESTFAIL:` and `IGNORED:`
lines mean. Also list test keywords to help catching failure patterns.

Example:
```
FAIL 1034: 'HTTP over proxy with malformatted IDN host name' HTTP, HTTP GET, HTTP proxy, IDN, FAILURE, config file
FAIL 1035: 'HTTP over proxy with too long IDN host name' HTTP, HTTP GET, HTTP proxy, IDN, FAILURE

TESTFAIL: These test cases failed: 1034 1035
```

Closes #14174
2024-07-16 11:56:34 +02:00
Viktor Szakats
c09db8b51b
cmake: create configurehelp.pm like autotools does
Required by tests 1119 and 1167 to run a C preprocessor.

Tested OK: https://github.com/curl/curl/actions/runs/9915343826

Besides Apple, it also supports any gcc and clang builds, and MSVC.
For other platforms, it defaults to `cpp` (like autotools).

Follow-up to efc2c5184d #14124
Cherry-picked from #14097
Closes #14129
2024-07-16 11:41:52 +02:00
Viktor Szakats
58772b0e08
runtests: set SOURCE_DATE_EPOCH to fix failing around midnight
To make sure that `managen` called by test 1706 uses the same date as
the test expects in the `%DATE` macro.

Before this patch when tests started running before UTC midnight and
reached test 1706 after, these dates were different and the test failed.

Follow-up to 0e73b69b3d
Fixes #14173
Closes #14187
2024-07-16 02:13:48 +02:00
Viktor Szakats
18f1cd7a77
tests: sync feature names with curl -V
Some feature names used in tests had minor differences compared to
the well-known ones from `curl -V`. This patch syncs them to make test
results easier to grok.

Closes #14183
2024-07-16 00:44:55 +02:00
Stefan Eissing
9e5b11c659
sendf: fix CRLF conversion of input
When CRLF line end conversion was enabled (--crlf), input after the last
newline in the upload buffer was not sent, if the buffer contained a
newline.

Reported-by: vuonganh1993 on github
Fixes #14165
Closes #14169
2024-07-15 23:47:33 +02:00
Stefan Eissing
6f0a8bd43c
test2600: disable on win32
- disbable this test on WIN32 platforms. It uses the file describtor '1'
  as valid socket without events. Not portable.
- reduce trace output somewhat on other runs

Fixes #14177
Reported-by: Viktor Szakats
Closes #14191
2024-07-15 23:24:11 +02:00
Daniel Stenberg
8dab7465a5
libcurl-docs: make option lists alpha-sorted
The man pages for curl_easy_getinfo, curl_easy_setopt and
curl_multi_setopt now feature the lists of options alphabetically
sorted. Test 1139 verify that they are.

The curl_multi_setopt page also got brief explanations of the listed
options.

Closes #14156
2024-07-14 13:44:24 +02:00
Viktor Szakats
6343034dd1
tidy-up: adjust casing of project names
Mostly TLS/SSH project name.

Closes #14160
2024-07-12 13:56:16 +02:00
Stephen Farrell
6a5bb68556
doh: fix leak and zero-length HTTPS RR crash
This PR fixes a leak and a crash that can happen when curl encounters
bad HTTPS RR values in DNS. We're starting to do better testing of that
kind of thing and e.g. have published bad HTTPS RR values at
dodgy.test.defo.ie.

Closes #14151
2024-07-12 12:16:56 +02:00
Daniel Stenberg
fc84583ae3
test1175: scan libcurl-errors.md, not the generated .3 version
Closes #14133
2024-07-09 23:13:40 +02:00
Daniel Stenberg
3f8066bf7c
test1139: scan .md files instead of .3 ones
As they are the canonical sources.

It still uses the curl.1 for command line option info.

Closes #14132
2024-07-09 23:11:08 +02:00
Viktor Szakats
67cc1e3400
test1119: adapt for .md input
Replace logic dealing with `.3` files to handle the Markdown syntax.

Follow-up to eefcc1bda4 #12730
Cherry-picked from #14097
Closes #14125
2024-07-09 04:13:32 +02:00
Viktor Szakats
efc2c5184d
tests: include current directory when running test Perl commands
Necessary to find generated files in the out-of-tree build directory.
E.g. `tests/configurehelp.pm`, for tests 1119 and 1167.

Before this patch macOS autotools builds were failing these two tests
due to falling back to the default preprocessor (`cpp`) instead of
the actual one configured. Then `cpp` failing to compile Apple SDK
headers referenced by curl headers.

Cherry-picked from #14097
Closes #14124
2024-07-09 04:13:04 +02:00
Dan Fandrich
d70ec5c650 tests: fix sshd UserKnownHostsFile path for MinGW/Cygwin
This is the same thing as the previous commit fd194f46 but on the next
line.

Follow-up to 70d2fca2

Ref: #10818
2024-07-06 22:07:25 -07:00
Dan Fandrich
b7b413e75c tests: fix sshd IdentityFile path for MinGW/Cygwin
This was missed during some refactoring more than a year ago and is
causing a warning "Use of uninitialized value $path in pattern match".

Follow-up to 70d2fca2

Ref: #10818
Closes #14113
2024-07-06 20:32:11 -07:00
Viktor Szakats
96a1a05f66
build: add Debug, TrackMemory, ECH to feature list
Also:

- remove stray `ECH` and `HTTPSRR` from cmake protocol list.

- stop excluding `Debug` and `TrackMemory` in `test1013.pl`.

- configure: delete `CURL_CHECK_CURLDEBUG` check.
  Ref: 065047dc62
  This check was effectively doing nothing, except disabling
 `--enable-curldebug` in `curl-config` for
 Cygwin/MSYS/cegcc/OS2/AIX targets with c-ares enabled.

Closes #14096
2024-07-07 00:33:21 +02:00
Dan Fandrich
127aa61f06 curl: follow-up to fix categories in --help
The commit 6483813b was missing changes necessitated by 2abfc75 that
causes a crash. Also, use ARRAYSIZE() for cleaner code.

Follow-up to 6483813b

Ref #14055
2024-07-05 14:44:43 -07:00
Dan Fandrich
6483813be2
curl: list categories in --help
This eliminates the need to run an extra help subcommand to get the
possible categories, reducing the friction in getting relevant help. The
help wording was also slightly tweaked for grammatical accuracy.

Closes #14055
2024-07-05 23:09:47 +02:00
Daniel Stenberg
2abfc759b9
cmdline-opts: category cleanup
Option cleanups:

 --get is not upload
 --form* are post
 - added several options into ldap, smtp, imap and pop3
 - shortened the category descriptions in the list

category curl fixes:

 --create-dirs removed from 'curl'
 --ftp-create-dirs removed from 'curl'
 --netrc moved to 'auth' from 'curl'
 --netrc-file moved to 'auth' from 'curl'
 --netrc-optional moved to 'auth' from 'curl'
 --no-buffer moved to 'output' from 'curl'
 --no-clobber removed from 'curl'
 --output removed from 'curl'
 --output-dir removed from 'curl'
 --remove-on-error removed from 'curl'

Add a "global" category:

- Made all "global" options set this category

Add a "deprecated" category:

- Moved the deprecated options to it (maybe they should not be in any
 category long term)

Add a 'timeout' category

- Put a number of appropriate options in it

Add an 'ldap' category

- Put the LDAP related option in there

Remove categories "ECH" and "ipfs"

- They should not be categories. Had only one single option each.

Remove category "misc"

- It should not be a category as it is impossible to know when to browse
  it.

--use-ascii moved to ftp and output
--xattr moved to output
--service-name moved to auth

Managen fixes:

- errors if an option is given a category name that is not already setup
  for in code

- verifies that options set `scope: global` also is put in category
  `global´

Closes #14101
2024-07-05 11:05:50 +02:00
Viktor Szakats
2583d11f22
tests: delete CharConv remains
Closes #14100
2024-07-04 18:12:40 +02:00
Stefan Eissing
480883cf27
multi: fix pollset during RESOLVING phase
- add a DEBUGASSERT for when a transfer's pollset should not be empty.
- move write unpausing from transfer loop into curl_easy_pause. This
  make sure that the url_updatesocket() finds the correct state when
  updating socket events.
- fix HTTP/2 proxy during connect phase to set sockets correctly
- fix test2600 to simulate a socket set
- move write unpausing from transfer loop into curl_easy_pause. This
  make sure that the url_updatesocket() finds the correct state when
  updating socket events.
- waiting for the resolver to deliver might not involve any sockets to
  wait for. Do not generate a warning.

Fixes #14047
Closes #14074
2024-07-02 11:17:38 +02:00
Daniel Stenberg
c074ba64a8
code: language cleanup in comments
Based on the standards and guidelines we use for our documentation.

 - expand contractions (they're => they are etc)
 - host name = > hostname
 - file name => filename
 - user name = username
 - man page => manpage
 - run-time => runtime
 - set-up => setup
 - back-end => backend
 - a HTTP => an HTTP
 - Two spaces after a period => one space after period

Closes #14073
2024-07-01 22:58:55 +02:00
Stefan Eissing
185a05e943
ngtcp2+quictls: fix cert-status use
- add test for --cert-status on all http versions

Reported-by: Dexter Gerig
Fixes #14049
Closes #14050
2024-06-28 14:34:51 +02:00
Daniel Stenberg
e63e0d60b5
managen: insert final .fi for files ending with a quote
When an individual file ended with a quote (typically an example), the
render function would return without ending the quote correctly with a
".fi" (fill in) in the manpage output.

This made the additional text provided below to render wrongly.

Closes #14048
2024-06-28 12:30:55 +02:00
Viktor Szakats
1ccdad64ef
CI: add whitespace checker
Fix issues detected.

Also:

- One of the `.vc` files used LF EOLs, while the other didn't.
  Make that one also use LF EOLs, as this is apparently supported by
  `nmake`.

- Drop `.dsw` and `.btn` types from `.gitattributes`.
  The repository doesn't use them.

- Sync section order with the rest of files in
  `tests/certs/EdelCurlRoot-ca.prm`.

- Indent/align `.prm` and `.pem` files.

- Delete dummy `[something]` section from `.prm` and `.pem` files.

Mental note:
MSVC `.sln` files seem to accept spaces for indentation and also support
LF line-endings. I cannot test this and I don't know what's more
convenient when updating them, so left them as-is, with specific
exclusions.

Closes #14031
2024-06-27 13:33:30 +02:00
Stefan Eissing
5b614224e4
tests/httpd: adjust ReadBufferSize for better performance
- list httpd and caddy versions in scorecard run

Closes #14039
2024-06-27 12:49:39 +02:00
Daniel Stenberg
cdbfad96ff
runtests: fix %VERNUM
It needs to be set to the leading digits and dots only, so that the
`-[date]` suffix strings are not included, as those used in the daily
snapshots.

Fixes #14035
Reported-by: Marcel Raad
Closes #14036
2024-06-27 12:48:35 +02:00
Stefan Eissing
bb09a304bb
http/3: add shutdown support
- openssl-quic shutdown handling
- ngtcp2 shutdown handling
- quiche shutdown handling
- add test_19_06 for verfication

Reported-by: Dexter Gerig
Closes #14027
Fixes #14022
2024-06-26 23:09:32 +02:00
Daniel Stenberg
868ae0673c
tests: verify managen
1705: verifies the manpage output

1706: verifies the ascii output

Closes #14025
2024-06-26 15:49:26 +02:00
Daniel Stenberg
0e73b69b3d
runtests: support %DATE for YYYY-MM-DD of right now 2024-06-26 15:49:26 +02:00
Daniel Stenberg
348882ee79
runtests: support %VERNUM
For the plain version number of the built curl without -DEV etc. Only
digits and dots.
2024-06-26 15:49:26 +02:00
Daniel Stenberg
4d58ed2d2c
FILEFORMAT.md: mentioned <file[num]> for "client"
They can be used to create more files.

Closes #14024
2024-06-26 10:55:40 +02:00
Stefan Eissing
c9b95c0bb3
lib: graceful connection shutdown
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.

The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.

Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.

As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.

Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.

When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.

The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.

- In multi_wait() and multi_waitfds(), collect all connection caches
  involved (each transfer might carry its own) into a temporary list.
  Let each connection cache on the list contribute sockets and
  POLLIN/OUT events it's connections are waiting for.

- in multi_perform() collect the connection caches the same way and let
  them peform their maintenance. This will make another non-blocking
  attempt to shutdown all connections on its shutdown list.

- for event based multis (multi->socket_cb set), add the sockets and
  their poll events via the callback. When `multi_socket()` is invoked
  for a socket not known by an active transfer, forward this to the
  multi's cache for processing. On closing a connection, remove its
  socket(s) via the callback.

TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.

A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.

- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
  which happen without a graceful shutdown and should
  no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
  on FTP up- and downloads.

Closes #13976
2024-06-26 08:33:17 +02:00
Viktor Szakats
ff92e3b13c
tidy-up: whitespace [ci skip] 2024-06-24 20:26:34 +02:00
Daniel Stenberg
a7ec6a76ab
runtests: support crlf="yes" for the <stderr> section 2024-06-24 08:43:40 +02:00
Daniel Stenberg
df2fbc2777
unit2604: use 'unitfail' instead of 'error' variable
Since the framework is already returning that variable by default.
Avoids a warning for unreachable code.

Reported-by: Tal Regev
Fixes #13967
Closes #13973
2024-06-19 13:37:36 +02:00
Sergey Markelov
35c0117f47
mbedtls: support CURLOPT_CERTINFO
Closes #13113
2024-06-19 11:33:12 +02:00
Daniel Stenberg
6b93190fae
gnutls: pass in SNI name, not hostname when checking cert
The function we use is called 'gnutls_x509_crt_check_hostname()' but if
we pass in the hostname with a trailing dot, the check fails. If we pass
in the SNI name, which cannot have a trailing dot, it succeeds for
https://pyropus.ca./

I consider this as a flaw in GnuTLS and have submitted this issue
upstream:

  https://gitlab.com/gnutls/gnutls/-/issues/1548

In order to work with old and existing GnuTLS versions, we still need
this change no matter how they view the issue or might change it in the
future.

Fixes #13428
Reported-by: Ryan Carsten Schmidt
Closes #13949
2024-06-14 13:19:20 +02:00
Jay Satiro
85efbb92b8
tool_cb_hdr: allow etag and content-disposition for 3xx reply
- Parse etag and content-disposition headers for 3xx replies.

For example, a server may send a content-disposition filename header
with a redirect reply (3xx) but not with the final response (2xx).
Without this change curl would ignore the server's specified filename
and continue to use the filename extracted from the user-specified URL.

Prior to this change, 75d79a4 had limited etag and content-disposition
to 2xx replies only.

Tests-by: Daniel Stenberg

Reported-by: Morgan Willcock
Fixes https://github.com/curl/curl/issues/13302
Closes #13484
2024-06-14 09:28:42 +02:00
Andy Pan
479858e15e
test: fix CURLOPT_TCP_KEEPCNT typo
Follow up to b77d627d24

Closes #13931
2024-06-12 16:08:05 +02:00
Andy Pan
b77d627d24
tcpkeepalive: add CURLOPT_TCP_KEEPCNT and --keepalive-cnt
Closes #13885
2024-06-12 09:31:17 +02:00
Daniel Stenberg
61b465208f
test1486: verify that write-out.md and tool_writeout.c are in sync
- also verify alphabetialal order in the source
- add two missing variables to write-out.md

Closes #13920
2024-06-11 13:28:21 +02:00
Daniel Stenberg
d69ee3ef83
curl: support -w '%{num_retries}
Suggested-by: Jay Guerette
Ref: https://github.com/curl/curl/discussions/13901
Closes #13910
2024-06-11 09:07:43 +02:00
Guilherme Puida
b55c56a745
pytest: include testenv/vsftpd.py in dist tarball
Closes #13918
2024-06-11 08:06:19 +02:00
Daniel Stenberg
4464c9f2c6
tool_writeout: bsearch the variable name
As the list of variable names grows, doing a simple loop to find the
name get increasingly worse. This switches to a bsearch.

Also: do a case sensitive check for the variable name. The names have
not been documented to be case insensitive and there is no point in
having them so.

Closes #13914
2024-06-10 23:12:05 +02:00
Stefan Eissing
c31041b17e
connection: shutdown TLS (for FTP) better
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.

    - add cfilter `Curl_cft_shutdown` callback
    - keep a shutdown start timestamp and timeout at connectdata
    - provide shutdown timeout default and member in
      `data->set.shutdowntimeout`.
    - provide methods for starting, interrogating and clearing
      shutdown timers
    - provide `Curl_conn_shutdown_blocking()` to shutdown the
      `sockindex` filter chain in a blocking way. Use that in FTP.
    - add `Curl_conn_cf_poll()` to wait for socket events during
      shutdown of a connection filter chain.
      This gets the monitoring sockets and events via the filters
      "adjust_pollset()" methods. This gives correct behaviour when
      shutting down a TLS connection through a HTTP/2 proxy.
    - Implement shutdown for all socket filters
      - for HTTP/2 and h2 proxying to send GOAWAY
      - for TLS backends to the best of their capabilities
      - for tcp socket filter to make a final, nonblocking
        receive to avoid unwanted RST states
    - add shutdown forwarding to happy eyeballers and
      https connect ballers when applicable.

Closes #13904
2024-06-10 13:08:12 +02:00
Daniel Stenberg
4e71f134e5
noproxy: test bad ipv6 net size first
No need to parse anything if the size is out of range.

Added some tests to this effect to test 1614.

Closes #13902
2024-06-07 00:22:59 +02:00
Viktor Szakats
998b17ea7f
windows: fix UWP builds, add GHA job
Add new job to test building for UWP (aka `CURL_WINDOWS_APP`).

Fix fallouts when building for UWP:
- rand: do not use `BCryptGenRandom()`.
- cmake: disable using win32 LDAP.
- cmake: disable telnet.
- version_win32: fix code before declaration.
- schannel: disable `HAS_MANUAL_VERIFY_API`.
- schannel: disable `SSLSUPP_PINNEDPUBKEY`
  and make `schannel_checksum()` a stub.
  Ref: e178fbd40a #1429
- schannel: make `cert_get_name_string()` a failing stub.
- system_win32: make `Curl_win32_impersonating()` a failing stub.
- system_win32: try to fix `Curl_win32_init()` (untested).
- threads: fix to use `CreateThread()`.
- src: disable searching `PATH` for the CA bundle.
- src: disable bold text support and capability detection.
- src: disable `getfiletime()`/`setfiletime()`.
- tests: make `win32_load_system_library()` a failing stub.
- tests/server/util: make it compile.
- tests/server/sockfilt: make it compile.
- tests/lib3026: fix to use `CreateThread()`.

See individual commits for build error details.

Some of these fixes may have better solutions, and some may not work
as expected. The goal of this patch is to make curl build for UWP.

Closes #13870
2024-06-05 00:52:24 +02:00
Orgad Shaneh
3060557af7
socket: support binding to interface *AND* IP
Introduce new notation for CURLOPT_INTERFACE / --interface:
ifhost!<interface>!<host>

Binding to an interface doesn't set the address, and an interface can
have multiple addresses.

When binding to an address (without interface), the kernel is free to
choose the route, and it can route through any device that can access
the target address, not necessarily the one with the chosen address.

Moreover, it is possible for different interfaces to have the same IP
address, on which case we need to provide a way to be more specific.

Factor out the parsing part of interface option, and add unit tests:
1663.

Closes #13719
2024-06-04 23:47:54 +02:00
Viktor Szakats
75f182ffb6
tests: delete redundant !MSDOS guard
This fix was supposed to be committed earlier, but ended up missing from
the final commit.

Follow-up to e9a7d4a1c8 #12376
Closes #13878
2024-06-04 18:47:18 +02:00
Stefan Eissing
5f9017d4e2
mbedtls: v3.6.0 workarounds
- add special sauce to disable unwanted peer verification by mbedtls
  when negotiating TLS v1.3
- add special sauce for MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET
  return code on *writing* TLS data. We assume the data had not been
  written and EAGAIN.
- return correct Curl error code when peer verification failed.
- disable test_08_05 with 50 HTTP/1.1 connections, as mbedtls reports a
  memory allocation failed during handshake.
- bump CI mbedtls version to 3.6.0

Fixes #13653
Closes #13838
2024-06-04 09:02:37 +02:00
Dan Fandrich
4a7bb40dea tests: don't log buffer length in throwing away message
It's not available at that point, and it will be written in the
non-error case right afterward.
2024-06-03 22:21:09 -07:00
Dan Fandrich
bc21c505e4 tests: log "Throwing away" messages before throwing away
In case the read that follows hangs we'll get a clue as to what it was
doing.
2024-06-03 17:00:42 -07:00
Daniel Stenberg
655d44d139
urlapi: add CURLU_NO_GUESS_SCHEME
Used for extracting:

- when used asking for a scheme, it will return CURLUE_NO_SCHEME if the
  stored information was a guess

- when used asking for a URL, the URL is returned without a scheme, like
  when previously given to the URL parser when it was asked to guess

- as soon as the scheme is set explicitly, it is no longer internally
  marked as guessed

The idea being:

1. allow a user to figure out if a URL's scheme was set as a result of
  guessing

2. extract the URL without a guessed scheme

3. this makes it work similar to how we already deal with port numbers

Extend test 1560 to verify.

Closes #13616
2024-06-01 23:51:42 +02:00
Daniel Stenberg
e78913e6b5
noproxy: patterns need to be comma separated
or they will not parse correctly.

Mentioned in DEPRECATED since Janurary 2023 (in 7ad8a7ba9e).

Closes #13789
2024-06-01 12:25:13 +02:00
Stefan Eissing
c8096668ae
multi: fix multi_wait() timeout handling
- determine the actual poll timeout *after* all sockets
  have been collected. Protocols and connection filters may
  install new timeouts during collection.
- add debug logging to test1533 where the mistake was noticed

Reported-by: Matt Jolly
Fixes #13782
Closes #13825
2024-05-30 08:29:00 +02:00
Jan Venekamp
2b52fe4115
sectransp: use common code for cipher suite lookup
Take advantage of the Curl_cipher_suite_walk_str() and
Curl_cipher_suite_get_str() functions introduced in commit fba9afe.

Closes #13521
2024-05-29 13:08:14 +02:00
Matthias Gatto
768909d89c
aws-sigv4: url encode the canonical path
Refactors canon_query, so it could use the encoding part of the function
to use it in the path.

As the path doesn't encode '/', but encode '=', I had to add some
conditions to know If I was doing the query or path encoding.

Also, instead of adding a `bool in_path` variable, I use `bool
*found_equals` to know if the function was called for the query or path,
as found_equals is used only in query_encoding.

Test 472 verifies.

Reported-by: Alexander Shtuchkin
Fixes #13754
Closes #13814

Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com>
2024-05-29 13:02:52 +02:00
Daniel Stenberg
4f558d288c
tests: run with "--trace-config all" to provide even more info
in case of problems.

Closes #13791
2024-05-28 09:55:36 +02:00
Viktor Szakats
59dc9f7e69
build: untangle CURLDEBUG and DEBUGBUILD macros
`CURLDEBUG` is meant to enable memory tracking, but in a bunch of cases,
it was protecting debug features that were supposed to be guarded with
`DEBUGBUILD`.

Replace these uses with `DEBUGBUILD`.

This leaves `CURLDEBUG` uses solely for its intended  purpose: to enable
the memory tracking debug feature.

Also:
- autotools: rely on `DEBUGBUILD` to enable `checksrc`.
  Instead of `CURLDEBUG`, which worked in most cases because debug
  builds enable `CURLDEBUG` by default, but it's not accurate.
- include `lib/easyif.h` instead of keeping a copy of a declaration.
- add CI test jobs for the build issues discovered.

Ref: https://github.com/curl/curl/pull/13694#issuecomment-2120311894
Closes #13718
2024-05-28 08:12:00 +02:00
Viktor Szakats
1054c1cc20
cmake: always build unit tests with the testdeps target
Before this patch, the `testdeps` build target required `-DCURLDEBUG`
be set either via `ENABLE_DEBUG=ON` or `ENABLE_CURLDEBUG=ON` to build
the curl unit tests.

After fixing build issues in #13694, we can drop this requirement and
build unit tests unconditionally.

Depends-on: #13694
Depends-on: #13697 (fix unit test issue revealed by Old Linux CI job)
Follow-up to 39e7c22bb4 #11446
Closes #13698
2024-05-27 22:26:41 +02:00
Viktor Szakats
fc8e0dee30
build: untangle UNITTESTS and DEBUGBUILD macros
- fix `DEBUGBUILD` guards that should be `UNITTESTS`, in libcurl code
  used by unit tests.
- fix guards for libcurl functions used in unit tests only.
- sync `UNITTEST` attribute between declarations and definitions.
- drop `DEBUGBUILD` guard from test `unit2600`.
- fix guards for libcurl HSTS code used by both a unit test (`unit1660`)
  and `test0446`.
- update an existing AppVeyor CI job to test the issues fixed.

This fixes building tests with `CURLDEBUG` enabled but `DEBUGBUILD`
disabled. This can happen when building tests with CMake with
`ENABLE_DEBUG=ON` in Release config, or with `ENABLE_CURLDEBUG=ON`
and _without_ `ENABLE_DEBUG=ON`. Possibly also with autotools
when using `--enable-curldebug` without `--enable-debug`.

Test results:
- before:
  https://ci.appveyor.com/project/curlorg/curl/builds/49835609
  https://ci.appveyor.com/project/curlorg/curl/builds/49898529/job/k8qpbs8idby70smw
  https://github.com/curl/curl/actions/runs/9259078835/job/25470318167?pr=13798#step:13:821
- after: https://ci.appveyor.com/project/curlorg/curl/builds/49839255
  (the two failures are unrelated, subject to PR #13705)

Ref: #13592 (issue discovery)
Ref: #13689 (CI testing this PR with `DEBUGBUILD`/`CURLDEBUG` combinations)
Closes #13694
2024-05-27 21:15:50 +02:00