Warning: this will make existing curl command lines that use metalink to
stop working.
Reasons for removal:
1. We've found several security problems and issues involving the
metalink support in curl. The issues are not detailed here. When
working on those, it become apparent to the team that several of the
problems are due to the system design, metalink library API and what
the metalink RFC says. They are very hard to fix on the curl side
only.
2. The metalink usage with curl was only very briefly documented and was
not following the "normal" curl usage pattern in several ways, making
it surprising and non-intuitive which could lead to further security
issues.
3. The metalink library was last updated 6 years ago and wasn't so
active the years before that either. An unmaintained library means
there's a security problem waiting to happen. This is probably reason
enough.
4. Metalink requires an XML parsing library, which is complex code (even
the smaller alternatives) and to this day often gets security
updates.
5. Metalink is not a widely used curl feature. In the 2020 curl user
survey, only 1.4% of the responders said that they'd are using it. In
2021 that number was 1.2%. Searching the web also show very few
traces of it being used, even with other tools.
6. The torrent format and associated technology clearly won for
downloading large files from multiple sources in parallel.
Cloes #7176
The warning about missing entries in that file then doesn't require that
the Makefile has been regenerated which was confusing.
The scan for the test num is a little more error prone than before
(since now it doesn't actually verify that it is legitimate Makefile
syntax), but I think it is good enough.
Closes#7177
For options that pass in lists or strings that are subsequently parsed
and must be correct. This broadens the scope for the option previously
known as CURLE_TELNET_OPTION_SYNTAX but the old name is of course still
provided as a #define for existing applications.
Closes#7175
The previous strip also removed the CR which turned problematic.
valgrind.supp: add zstd suppression using hyper
Reported-and-analyzed-by: Kevin Burke
Fixes#7169Closes#7171
Hyper returns the same error for wrong HTTP version as for negative
content-length. Test 178 verifies that negative content-length is
rejected but the hyper backend will return a different error for it (and
without any helpful message telling why the message was bad). It will
also not return any headers at all for the response, not even the ones
that arrived before the error.
Closes#7147
In some situations, it was possible that a transfer was setup to
use an specific IP version, but due do DNS caching or connection
reuse, it ended up using a different IP version from requested.
This commit changes the effect of CURLOPT_IPRESOLVE from simply
restricting address resolution to preventing the wrong connection
type being used, when choosing a connection from the pool, and
to restricting what addresses could be used when establishing
a new connection.
It is important that all addresses versions are resolved, even if
not used in that transfer in particular, because the result is
cached, and could be useful for a different transfer with a
different CURLOPT_IPRESOLVE setting.
Closes#6853
Also added 'CURL_SMALLSENDS' to make Curl_write() send short packets,
which helped verifying this even more.
Add test 363 to verify.
Reported-by: ustcqidi on github
Fixes#6950Closes#7024
When hyper is used, it emits uppercase hexadecimal numbers for chunked
encoding lengths. Without hyper, lowercase hexadecimal numbers are used.
This change adds preprocessor statements to tests where this is an
issue, and adapts the fixtures to match.
Closes#6987
When a TLS server requests a client certificate during handshake and
none can be provided, libcurl now returns this new error code
CURLE_SSL_CLIENTCERT
Only supported by Secure Transport and OpenSSL for TLS 1.3 so far.
Closes#6721
A struct bufref holds a buffer pointer, a data size and a destructor.
When freed or its contents are changed, the previous buffer is implicitly
released by the associated destructor. The data size, although not used
internally, allows binary data support.
A unit test checks its handling methods: test 1661
Closes#6654
Add test 676 to verify that setting CURLOPT_COOKIEFILE to NULL again clears
the cookiejar from memory.
Reported-by: Stefan Karpinski
Fixes#6889Closes#6891
According to Microsoft document MS-NLMP, current flags usage is not
accurate: flag NTLMFLAG_NEGOTIATE_NTLM2_KEY controls the use of
extended security in an NTLM authentication message and NTLM version 2
cannot be negotiated within the protocol.
The solution implemented here is: if the extended security flag is set,
prefer using NTLM version 2 (as a server featuring extended security
should also support version 2). If version 2 has been disabled at
compile time, use extended security.
Tests involving NTLM are adjusted to this new behavior.
Fixes#6813Closes#6849
After 957bc1881e, we no longer compute an
expected_size for directories. This has the upshot that when we compare
even an empty Range with the available size, we fail.
This brings back the previous behaviour, which was to succeed, but with
empty content. This also removes the "Accept-ranges: bytes" header,
which is nonsensical on directories.
Adds test 3016
Fixes#6845Closes#6846
To make sure the Host: header and the URL provide the same authority
portion when sent to the proxy, strip the default port number from the
URL if one was provided.
Reported-by: Michael Brown
Fixes#6769Closes#6778
When asked to resume a download, libcurl will convert that to HTTP logic
and if then the entire file is already transferred it will result in a
416 response from the HTTP server. With CURLOPT_FAILONERRROR set in that
scenario, it should *not* lead to an error return.
Updated test 1156, added test 1273
Reported-by: Jonathan Watt
Fixes#6740Closes#6753
- Increase the minimum number of spaces between the option and the
description from 1 to 2.
Before:
~~~
-u, --user <user:password> Server user and password
-A, --user-agent <name> Send User-Agent <name> to server
-v, --verbose Make the operation more talkative
-V, --version Show version number and quit
-w, --write-out <format> Use output FORMAT after completion
--xattr Store metadata in extended file attributes
~~~
After:
~~~
-u, --user <user:password> Server user and password
-A, --user-agent <name> Send User-Agent <name> to server
-v, --verbose Make the operation more talkative
-V, --version Show version number and quit
-w, --write-out <format> Use output FORMAT after completion
--xattr Store metadata in extended file attributes
~~~
Closes https://github.com/curl/curl/pull/6674
Previously only a single -b cookie parameter was supported with the last
one winning. This adds support for supplying multiple -b params to have
them serialized semicolon separated. Both cookiefiles and cookies can be
entered multiple times.
Closes#6649
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
- add CURLINFO_REFERER libcurl option
- add --write-out '%{referer}' command-line option
- extend --xattr command-line option to fill user.xdg.referrer.url extended
attribute with the referrer (if there was any)
Closes#6591
Using the %NOLISTENPORT to trigger a connection failure is somewhat
"risky" (since it isn't guaranteed to not be listened to) and caused
occasional CI problems. This fix changes the infused error to be a more
reliable one but still verifies the --write-out functionality properly -
which is the purpose of this test.
Reported-by: Jay Satiro
Fixes#6621Closes#6623
HTTP auth "accidentally" worked before this cleanup since the code would
always overwrite the connection credentials with the credentials from
the most recent transfer and since HTTP auth is typically done first
thing, this has not been an issue. It was still wrong and subject to
possible race conditions or future breakage if the sequence of functions
would change.
The data.set.str[] strings MUST remain unmodified exactly as set by the
user, and the credentials to use internally are instead set/updated in
state.aptr.*
Added test 675 to verify different credentials used in two requests done
over a reused HTTP connection, which previously behaved wrongly.
Fixes#6542Closes#6545
Rename it to 'httpwant' and make a cloned field in the state struct as
well for run-time updates.
Also: refuse non-supported HTTP versions. Verified with test 129.
Closes#6585
and rename it from 'ftp_list_only' since it is also used for SSH and
POP3. The state is updated internally for 'type=D' FTP URLs.
Added test case 1570 to verify.
Closes#6578
... and make sure the code never updates 'set.prefer_ascii' as it breaks
handle reuse which should use the setting as the user specified it.
Added test 1569 to verify: it first makes an FTP transfer with ';type=A'
and then another without type on the same handle and the second should
then use binary. Previously, curl failed this.
Closes#6578
- Deduplicate the logic used by write-out and write-out json.
Rather than have separate writeLong, writeString, etc, logic for
each of write-out and write-out json instead have respective shared
functions that can output either format and a 'use_json' parameter to
indicate whether it is json that is output.
This will make it easier to maintain. Rather than have to go through
two sets of logic now we only have to go through one.
- Support write-out %{errormsg} and %{exitcode} in json.
- Clarify in the doc that %{exitcode} is the exit code of the transfer.
Prior to this change it just said "The numerical exitcode" which
implies it's the exit code of the tool, and it's not necessarily that.
Closes https://github.com/curl/curl/pull/6544
It can contain additional restraints for test numbers,
keywords and tools.
The idea is to let third parties like the Privoxy project
distribute an exclude file with their tarballs that specifies
which curl tests are not expected to work when using Privoxy
as a proxy, without having to fork the whole curl test suite.
The syntax could be changed to be extendable and maybe
more closely reflect the "curl test" syntax. Currently
it's a bunch of lines like these:
test:$TESTNUMBER:Reason why this test with number $TESTNUMBER should be skipped
keyword:$KEYWORD:Reason why tests whose keywords contain the $KEYWORD should be skipped
tool:$TOOL:Reason why tests with tools that contain $TOOL should be skipped
To specify multiple $TESTNUMBERs, $KEYWORDs and $TOOLs
on a single line, split them with commas.
This is useful to change the behaviour of the script without
having to modify the file itself, for example to use a custom
compareparts() function that ignores header differences that
are expected to occur when an external proxy is being used.
Such differences are proxy-specific and thus the modifications
should be maintained together with the proxy.
missing CRs and modified %hostip
lib556/test556: use a real HTTP version to make test reuse more convenient
make sure the weekday in Date headers matches the date
test61: replace stray "^M" (5e 4d) at the end of a cookie with a '^M' (0d)
Gets the test working with external proxies like Privoxy again.
Closes#6463
... as we ignore it anyway because servers don't report the correct size
and proftpd even blatantly returns a 550.
Updates a set of tests accordingly.
Reported-by: awesomenode on github
Fixes#6564Closes#6565
- Add support services without region and service prefixes in
the URL endpoint (ex. Min.IO, GCP, Yandex Cloud, Mail.Ru Cloud Solutions, etc)
by providing region and service parameters via aws-sigv4 option.
- Add [:region[:service]] suffix to aws-sigv4 option;
- Fix memory allocation errors.
- Refactor memory management.
- Use Curl_http_method instead() STRING_CUSTOMREQUEST.
- Refactor canonical headers generating.
- Remove repeated sha256_to_hex() usage.
- Add some docs fixes.
- Add some codestyle fixes.
- Add overloaded strndup() for debug - curl_dbg_strndup().
- Update tests.
Closes#6524
- Pass the very long request header via file instead of command line.
Prior to this change the 49k very long request header string was passed
via command line and on Windows that is too long so it was truncated and
the test would fail (specifically msys CI).
Closes https://github.com/curl/curl/pull/6516
... in most cases instead of 'struct connectdata *' but in some cases in
addition to.
- We mostly operate on transfers and not connections.
- We need the transfer handle to log, store data and more. Everything in
libcurl is driven by a transfer (the CURL * in the public API).
- This work clarifies and separates the transfers from the connections
better.
- We should avoid "conn->data". Since individual connections can be used
by many transfers when multiplexing, making sure that conn->data
points to the current and correct transfer at all times is difficult
and has been notoriously error-prone over the years. The goal is to
ultimately remove the conn->data pointer for this reason.
Closes#6425
... and as a practical side-effect, make sure that the
Curl_easyopts_check() function is asserted in debug builds, which we
want to detect mismatches between the options list in easyoptions.c and
the options in curl.h
Found-by: Gisle Vanem
Bug: 08e8455ddd (commitcomment-45991815)Closes#6461
... so that a function can first use MIMEPOST and then set it to NULL to
reset it back to a blank POST.
Added test 584 to verify the fix.
Reported-by: Christoph M. Becker
Fixes#6455Closes#6456
... so that Retry-After and other meta-content can still be used.
Added 1634 to verify. Adjusted test 194 and 281 since --fail now also
includes the header-terminating CRLF in the output before it exits.
Fixes#6408Closes#6409
When doing HTTP authentication and a port number set with CURLOPT_PORT,
the code would previously have the URL's port number override as if it
had been a redirect to an absolute URL.
Added test 1568 to verify.
Reported-by: UrsusArctos on github
Fixes#6397Closes#6400
This option sets the (octal) mode to use for the remote file when one is
created, using the SFTP, SCP or FILE protocols. When not set, the
default is 0644.
Closes#6244
When the initial request isn't possible to send in its entirety, the
remainder of request would be delivered to the debug callback as data
and would wrongly be counted internally as body-bytes sent.
Extended test 1295 to verify.
Closes#6328
