Commit Graph

31295 Commits

Author SHA1 Message Date
Stefan Eissing
ad051e1cbe
lib: client writer, part 2, accounting + logging
This PR has these changes:

Renaming of unencode_* to cwriter, e.g. client writers
- documentation of sendf.h functions
- move max decode stack checks back to content_encoding.c
- define writer phase which was used as order before
- introduce phases for monitoring inbetween decode phases
- offering default implementations for init/write/close

Add type paramter to client writer's do_write()
- always pass all writes through the writer stack
- writers who only care about BODY data will pass other writes unchanged

add RAW and PROTOCOL client writers
- RAW used for Curl_debug() logging of CURLINFO_DATA_IN
- PROTOCOL used for updates to data->req.bytecount, max_filesize checks and
  Curl_pgrsSetDownloadCounter()
- remove all updates of data->req.bytecount and calls to
  Curl_pgrsSetDownloadCounter() and Curl_debug() from other code
- adjust test457 expected output to no longer see the excess write

Closes #12184
2023-11-06 13:14:06 +01:00
Daniel Stenberg
2b16b86bb6
VULN-DISCLOSURE-POLICY: escape sequences are not a security flaw
Closes #12278
2023-11-06 12:51:00 +01:00
Viktor Szakats
7925ba431b
rand: fix build error with autotools + LibreSSL
autotools unexpectedly detects `arc4random` because it is also looking
into dependency libs. One dependency, LibreSSL, happens to publish an
`arc4random` function (via its shared lib before v3.7, also via static
lib as of v3.8.2). When trying to use this function in `lib/rand.c`,
its protoype is missing. To fix that, curl included a prototype, but
that used a C99 type without including `stdint.h`, causing:

```
../../lib/rand.c:37:1: error: unknown type name 'uint32_t'
   37 | uint32_t arc4random(void);
      | ^
1 error generated.
```

This patch improves this by dropping the local prototype and instead
limiting `arc4random` use for non-OpenSSL builds. OpenSSL builds provide
their own random source anyway.

The better fix would be to teach autotools to not link dependency libs
while detecting `arc4random`.

LibreSSL publishing a non-namespaced `arc4random` tracked here:
https://github.com/libressl/portable/issues/928

Regression from 755ddbe901 #10672

Reviewed-by: Daniel Stenberg
Fixes #12257
Closes #12274
2023-11-06 10:11:56 +00:00
Daniel Stenberg
1e4f5a4a74
RELEASE-NOTES: synced 2023-11-05 23:28:45 +01:00
Daniel Stenberg
ac57e69b58
strdup: do Curl_strndup without strncpy
To avoid (false positive) gcc-13 compiler warnings.

Follow-up to 4855debd8a

Assisted-by: Jay Satiro
Reported-by: Viktor Szakats
Fixes #12258
2023-11-05 23:21:18 +01:00
Enno Boland
46878b9e3f
HTTP: fix empty-body warning
This change fixes a compiler warning with gcc-12.2.0 when
`-DCURL_DISABLE_BEARER_AUTH=ON` is used.

    /home/tox/src/curl/lib/http.c: In function 'Curl_http_input_auth':
    /home/tox/src/curl/lib/http.c:1147:12: warning: suggest braces around empty body in an 'else' statement [-Wempty-body]
     1147 |            ;
          |            ^

Closes #12262
2023-11-05 17:38:27 +01:00
Daniel Stenberg
86d4a4124e
openssl: identify the "quictls" backend correctly
Since vanilla OpenSSL does not support the QUIC API I think it helps
users to identify the correct OpenSSL fork in version output. The best
(crude) way to do that right now seems to be to check if ngtcp2 support
is enabled.

Closes #12270
2023-11-05 17:34:49 +01:00
Mark Gaiser
859e88f653
curl: improved IPFS and IPNS URL support
Previously just ipfs://<cid> and ipns://<cid> was supported, which is
too strict for some usecases.

This patch allows paths and query arguments to be used too.
Making this work according to normal http semantics:

 ipfs://<cid>/foo/bar?key=val
 ipns://<cid>/foo/bar?key=val

The gateway url support is changed.
It now only supports gateways in the form of:

 http://<gateway>/foo/bar
 http://<gateway>

Query arguments here are explicitly not allowed and trigger an intended
malformed url error.

There also was a crash when IPFS_PATH was set with a non trailing
forward slash. This has been fixed.

Lastly, a load of test cases have been added to verify the above.

Reported-by: Steven Allen
Fixes #12148
Closes #12152
2023-11-05 10:59:20 +01:00
Harry Mallon
d112c24234
docs: KNOWN_BUGS cleanup
* Remove other mention of hyper memory-leaks from `KNOWN_BUGS`.
  Should have been removed in 629723ecf2

* Remove mention of aws-sigv4 sort query string from `KNOWN_BUGS`.
  Fixed in #11806

* Remove mention of aws-sigv4 query empty value problems

* Remove mention of aws-sigv4 missing amz-content-sha256
  Fixed in #9995
2023-11-05 10:39:32 +01:00
Harry Mallon
bbba69dada
http_aws_sigv4: canonicalise valueless query params
Fixes #8107
Closes #12244
2023-11-05 10:35:45 +01:00
Michael Kaufmann
2c8f4c87e1 docs: preserve the modification date when copying the prebuilt man page
The previously built man page "curl.1" must be copied with the original
modification date, otherwise the man page is never updated.

This fixes a bug that has been introduced with commit 2568441cab.

Reviewed-by: Dan Fandrich
Reviewed-by: Daniel Stenberg

Closes #12199
2023-11-04 19:39:48 +01:00
Daniel Stenberg
0f3e9d5723
docs: remove bold from some man page SYNOPSIS sections
In the name of consistency

Closes #12267
2023-11-04 12:29:41 +01:00
Daniel Stenberg
b3c4bbfcaa
openssl: two multi pointer checks should probably rather be asserts
... so add the asserts now and consider removing the dynamic checks in a
future.

Ref: #12261
Closes #12264
2023-11-04 12:23:46 +01:00
boilingoden
559918d80c
docs: add supported version for the json write-out
xref: https://curl.se/changes.html#7_70_0

Closes #12266
2023-11-04 12:21:56 +01:00
Viktor Szakats
026122efd3
appveyor: make VS2008-built curl tool runnable
By linking the CRT statically. This avoids the error about missing
runtime DLL `MSVCR90.dll` when running the freshly built `curl.exe`.

Closes #12263
2023-11-03 22:53:45 +00:00
Stefan Eissing
3e6254f819
url: proxy ssl connection reuse fix
- tunnel https proxy used for http: transfers does
  no check if proxy-ssl configuration matches
- test cases added, test_10_12 fails on 8.4.0

Closes #12255
2023-11-03 16:43:45 +01:00
Jay Satiro
7e828fe503 curl_sspi: support more revocation error names in error messages
- Add these revocation errors to sspi error list:
  CRYPT_E_NO_REVOCATION_DLL, CRYPT_E_NO_REVOCATION_CHECK,
  CRYPT_E_REVOCATION_OFFLINE and CRYPT_E_NOT_IN_REVOCATION_DATABASE.

Prior to this change those error codes were not matched to their macro
name and instead shown as "unknown error".

Before:

schannel: next InitializeSecurityContext failed:
Unknown error (0x80092013) - The revocation function was
unable to check revocation because the revocation server was offline.

After:

schannel: next InitializeSecurityContext failed:
CRYPT_E_REVOCATION_OFFLINE (0x80092013) - The revocation function was
unable to check revocation because the revocation server was offline.

Bug: https://github.com/curl/curl/issues/12239
Reported-by: Niracler Li

Closes https://github.com/curl/curl/pull/12241
2023-11-03 03:50:59 -04:00
Jay Satiro
4855debd8a strdup: don't allow Curl_strndup to read past a null terminator
- Use malloc + strncpy instead of Curl_memdup to dupe the string before
  null terminating it.

Prior to this change if Curl_strndup was passed a length longer than
the allocated string then it could copy out of bounds.

This change is for posterity. Curl_strndup was added in the parent
commit and currently none of the calls to it pass a length that would
cause it to read past the allocated length of the input.

Follow-up to d3b3ba35.

Closes https://github.com/curl/curl/pull/12254
2023-11-02 23:44:46 -04:00
Daniel Stenberg
d3b3ba35a5
lib: add and use Curl_strndup()
The Curl_strndup() function is similar to memdup(), but copies 'n' bytes
then adds a terminating null byte ('\0').

Closes #12251
2023-11-02 20:35:20 +01:00
Daniel Stenberg
fdaed85b02
CURPOST_POSTFIELDS.3: add CURLOPT_COPYPOSTFIELDS in SEE ALSO 2023-11-02 20:34:14 +01:00
Stefan Eissing
7e1c0851bf
pytest: use lower count in repeat tests
- lower large iteration counts in some tests somewhat for
  the same coverage with less duration

Closes #12248
2023-11-02 16:35:00 +01:00
Daniel Stenberg
30e04beafe
RELEASE-NOTES: synced 2023-11-02 16:30:15 +01:00
Daniel Stenberg
63e1a9a1e7
docs: clarify that curl passes on input unfiltered
... for several options.

Reported-by: Ophir Lojkine

Closes #12249
2023-11-02 16:27:39 +01:00
Daniel Stenberg
5c846a12a3
urlapi: when URL encoding the fragment, pass in the right length
A benign bug because it would only add an extra null terminator.

Made lib1560 get a test that runs this code.

Closes #12250
2023-11-02 16:23:17 +01:00
Stefan Eissing
bbdbd4b025
vtls: late clone of connection ssl config
- perform connection cache matching against `data->set.ssl.primary`
  and proxy counterpart
- fully clone connection ssl config only when connection is used

Closes #12237
2023-11-02 14:30:42 +01:00
Stefan Eissing
5d0b3eda43
msh3: error when built with CURL_DISABLE_SOCKETPAIR set
Reported-by: Gisle Vanem
Closes #12252
Fixes #12213
2023-11-02 14:29:21 +01:00
Daniel Stenberg
8c058a820a
hsts: skip single-dot hostname
Reported-by: Maksymilian Arciemowicz

Closes #12247
2023-11-02 13:17:45 +01:00
Daniel Stenberg
ddb95c155e
vtls: fix build without proxy
Follow-up to bf0e278a3c

closes #12243
2023-11-01 14:55:04 +01:00
Daniel Stenberg
41b9fd5384
docs/example/keepalive.c: show TCP keep-alive options
Closes #12242
2023-11-01 10:57:15 +01:00
Daniel Stenberg
8c8a03f252
lib1560: verify appending blank URL encoded query string 2023-11-01 10:55:58 +01:00
Daniel Stenberg
ffbc9981c4
urlapi: skip appending NULL pointer query
Reported-by: kirbyn17 on hackerone

Closes #12240
2023-11-01 10:55:55 +01:00
Daniel Stenberg
21c5d5971e
lib1560: verify setting host to "" with and without URL encode 2023-11-01 10:55:55 +01:00
Daniel Stenberg
c64d0d67fd
urlapi: avoid null deref if setting blank host to url encode
Reported-by: kirbyn17 on hackerone

Closes #12240
2023-11-01 10:55:46 +01:00
Daniel Stenberg
27f2352d49
dynbuf: assert for NULL pointer inputs
Help us catch more mistakes.

Closes #12238
2023-11-01 10:53:49 +01:00
Daniel Stenberg
5f78cf503c
HTTP3: ngtcp2 builds are no longer experimental
The other HTTP/3 backends are still experimental.

Closes #12235
2023-10-31 14:31:46 +01:00
Stefan Eissing
bf0e278a3c
vtls: cleanup SSL config management
- remove `Curl_ssl_get_config()`, no longer needed

Closes #12204
2023-10-31 14:29:36 +01:00
Daniel Stenberg
faa45a637f
libcurl-thread.3: simplify the TLS section
All TLS libraries curl can use are threadsafe since OpenSSL 1.1.x, August
2016.

Closes #12233
2023-10-31 14:27:25 +01:00
Daniel Stenberg
d2d48f21f3
configure: better --disable-http
- disable HTTPS-proxy as well, since it can't work without HTTP

- curl_setup: when HTTP is disabled, also disable all features that are
  HTTP-only

- version: HTTPS-proxy only exists if HTTP support exists

Closes #12223
2023-10-30 17:02:13 +01:00
Daniel Stenberg
225db9196a
http: consider resume with CURLOPT_FAILONERRROR and 416 to be fine
Finding a 'Content-Range:' in the response changed the handling.

Add test case 1475 to verify -C - with 416 and Content-Range: header,
which is almost exactly like test 194 which instead uses a fixed -C
offset. Adjusted test 194 to also be considered fine.

Fixes #10521
Reported-by: Smackd0wn
Fixes #12174
Reported-by: Anubhav Rai
Closes #12176
2023-10-30 17:00:34 +01:00
Stefan Eissing
5a3e277290
GHA: fix checkout of quictls repository to use correct branch name
Follow-up to c868b0e30f

Closes #12232
2023-10-30 16:58:45 +01:00
Daniel Stenberg
92a315b1a9
docs/example/localport.c: show off CURLOPT_LOCALPORT
Closes #12230
2023-10-30 12:49:00 +01:00
Daniel Stenberg
22130826b7
docs/examples/interface.c: show CURLOPT_INTERFACE use
Although super simple.

Closes #12229
2023-10-30 12:45:34 +01:00
Viktor Szakats
aca7d808c3
build: fix compiler warning with auths disabled
```
./curl/lib/http.c:979:12: warning: unused function 'is_valid_auth_separator' [-Wunused-function]
static int is_valid_auth_separator(char ch)
           ^
5 warnings generated.
```

Follow-up to e92edfbef6 #11490

Closes #12227
2023-10-30 11:00:19 +00:00
Viktor Szakats
960d601481
build: require Windows XP or newer
After this patch we assume availability of `getaddrinfo` and
`freeaddrinfo`, first introduced in Windows XP. Meaning curl
now requires building for Windows XP as a minimum.

TODO: assume these also in autotools.

Ref: https://github.com/curl/curl/pull/12221#issuecomment-1783761806
Closes #12225
2023-10-30 10:46:40 +00:00
Viktor Szakats
4f591db44c
appveyor: bump one job to OpenSSL 3.1 (was 1.1.1)
Use 3.1 with the modern runner image.

We still use 1.1.1 in 8 jobs.

1.1.1 is EOL since 2023-09-11:
https://www.openssl.org/blog/blog/2023/03/28/1.1.1-EOL/

Also:
- add missing SSL-backend to job descriptions.
- tidy up CPU in job descriptions.

Closes #12226
2023-10-30 10:46:40 +00:00
Daniel Stenberg
ab885eccd6
RELEASE-NOTES: synced 2023-10-30 08:48:26 +01:00
Daniel Stenberg
c868b0e30f
GHA: bump ngtcp2, nghttp3, nghttp2 and quictls versions
ngtcp2 1.0.1
nghttp3 1.0.0
nghttp2 1.58.0
quictls 3.1.4+quic

also sync HTTP3.md with these changes

Closes #12132
2023-10-29 17:39:23 +01:00
Kareem
a51fff1850
wolfssl: add default case for wolfssl_connect_step1 switch
Closes #12218
2023-10-29 14:52:19 +01:00
Jay Satiro
904ae12238 curl_setup: disallow Windows IPv6 builds missing getaddrinfo
- On Windows if IPv6 is enabled but getaddrinfo is missing then #error
  the build.

curl can be built with IPv6 support (ENABLE_IPV6) but without the
ability to resolve hosts to IPv6 addresses (HAVE_GETADDRINFO). On
Windows this is highly unlikely and should be considered a bad build
configuration.

Such a bad configuration has already given us a bug that was hard to
diagnose. See #12134 and #12136 for discussion.

Ref: https://github.com/curl/curl/issues/12134
Ref: https://github.com/curl/curl/pull/12136

Closes https://github.com/curl/curl/pull/12221
2023-10-29 03:41:33 -04:00
Nico Rieck
2d6333101a openssl: make CURLSSLOPT_NATIVE_CA import Windows intermediate CAs
- If CURLSSLOPT_NATIVE_CA on Windows then import from intermediate CA
  "CA" store after importing from root CA "ROOT" store.

This change allows curl to work in situations where a server does not
send all intermediate certs and they are present in the "CA" store (the
store with intermediate CAs). This is already allowed by the Schannel
backend.

Also this change makes partial chain verification possible for those
certs since we allow partial chain verification by default for OpenSSL
(unless CURLSSLOPT_NO_PARTIALCHAIN). This is not allowed by the Schannel
backend.

Prior to this change CURLSSLOPT_NATIVE_CA only imported "ROOT" certs.

Fixes https://github.com/curl/curl/issues/12155
Closes https://github.com/curl/curl/pull/12185
2023-10-29 03:39:43 -04:00