Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.
Closes#14220
- generate AVAILABILITY manpage sections automatically - for consistent
wording
- allows us to double-check against other documumentation (symbols-in-versions
etc)
- enables proper automation/scripting based on this data
- lots of them were wrong or missing in the manpages
- several of them repeated (sometimes mismatching) backend support info
Add test 1488 to verify "added-in" version numbers against
symbols-in-versions.
Closes#14217
When removing an easy handle that had DoH sub-easy handles going, those
were not removed from the multi handle. Their memory was reclaimed on
curl_easy_cleanup() of the owning handle, but multi still had them in
their list.
Add `Curl_doh_close()` and `Curl_doh_cleanup()` as common point for
handling the DoH resource management. Use the `multi` present in the doh
handles (if so), for removal, as the `data->multi` might already have
been NULLed at this time.
Reported-by: 罗朝辉
Fixes#14207Closes#14212
Currently we're waiting for sendbuf_len_in_flight to hit zero before
resuming upload which means we're blocking and waiting for _all_ acks to
arrive before sending more data. This causes significant delays especially
when ack delay is used on the server side.
The fix addresses several issues in h3 over ngtcp2:
- On ack we now call nghttp3_conn_resume_stream() when we have more
data to send.
- upload_left was incorrectly computed on CF_CTRL_DATA_DONE_SEND as
we need to subtract the ammount of data we have in flight.
- Remove upload_blocked_len as we Curl_bufq_write call will do the
right thing when called from cf_ngtcp2_send.
Fixes#14198Closes#14209
- add upload tests to scorecard, invoke with
> python3 tests/http/scorecard.py -u h1|h2|h3
- add a reverse proxy setup from Caddy to httpd for
upload tests since Caddy does not have other PUT/POST handling
- add caddy tests in test_08 for POST/PUT
- increase read buffer in mod_curltest for larger reads
Closes#14208
Useful to see what the numbers listed in the `TESTFAIL:` and `IGNORED:`
lines mean. Also list test keywords to help catching failure patterns.
Example:
```
FAIL 1034: 'HTTP over proxy with malformatted IDN host name' HTTP, HTTP GET, HTTP proxy, IDN, FAILURE, config file
FAIL 1035: 'HTTP over proxy with too long IDN host name' HTTP, HTTP GET, HTTP proxy, IDN, FAILURE
TESTFAIL: These test cases failed: 1034 1035
```
Closes#14174
Fix the file of wolfssl.c because of this warning/error:
```
curl\lib\vtls\wolfssl.c(1017,42): error C2220: the following warning is treated as an error [curl\bld\lib\libcurl_object.vcxproj]
curl\lib\vtls\wolfssl.c(1017,42): warning C4267: 'function': conversion from 'size_t' to 'unsigned long', possible loss of data [curl\bld\lib\libcurl_object.vcxproj]
```
`size_t` in MSVC is different. Change it to `unsigned long` because
`wolfSSL_ERR_error_string_n` last argument is defined as
`unsigned long`.
Closes#14193
- `HAVE_LIBSSH2`: unused in source. Not defined in CMake.
- `HAVE_LIBSOCKET`: unused in source. Used internally in CMake.
autotools sets them implicitly, so add them to the flag comparison
ignore-list.
Closes#14178
- skip the entire CA logic if no selected TLS backend support CA
certs/bundles.
Follow-up to 082bb41311#2545
- sync bundle path detection logic with `./configure`.
- fix to not auto-detect CA bundle/path on Windows.
- fix to reflect that BearSSL has CA bundle support.
- show the detected bundle path (as with the cert bundle).
- tidy up CMake syntax, fix typos in comments.
Closes#14182
- fix to not auto-detect CA bundle/path on Windows.
- two checks missed BearSSL, but they were only run for supported
TLS backends anyway. Delete these redundant checks.
- fix typos in a comment nearby.
Follow-up to 082bb41311#2545Closes#14186
To make sure that `managen` called by test 1706 uses the same date as
the test expects in the `%DATE` macro.
Before this patch when tests started running before UTC midnight and
reached test 1706 after, these dates were different and the test failed.
Follow-up to 0e73b69b3dFixes#14173Closes#14187
Some feature names used in tests had minor differences compared to
the well-known ones from `curl -V`. This patch syncs them to make test
results easier to grok.
Closes#14183
When CRLF line end conversion was enabled (--crlf), input after the last
newline in the upload buffer was not sent, if the buffer contained a
newline.
Reported-by: vuonganh1993 on github
Fixes#14165Closes#14169
- disbable this test on WIN32 platforms. It uses the file describtor '1'
as valid socket without events. Not portable.
- reduce trace output somewhat on other runs
Fixes#14177
Reported-by: Viktor Szakats
Closes#14191
- make sure the TLS handshake after a successful STARTTLS command is
fully done before further sending/receiving on the connection.
Reported-by: tomy2105 on github
Fixes#14166Closes#14190
The man pages for curl_easy_getinfo, curl_easy_setopt and
curl_multi_setopt now feature the lists of options alphabetically
sorted. Test 1139 verify that they are.
The curl_multi_setopt page also got brief explanations of the listed
options.
Closes#14156
Add flags UIDNA_NONTRANSITIONAL_TO_ASCII and
UIDNA_NONTRANSITIONAL_TO_UNICODE to encode ß correctly.
It fixes test 165.
Reported-by: Viktor Szakats
Bug: #14176Closes#14179
It caused IDN to appear in `curl-config`, `libidn2` referenced from
`libcurl.pc`, fail to fallback to `pkg-config` detection. But libidn2
not actually used.
It came up in macOS CI builds after enabling cmake build tests. It
remained hidden for a while due to setting `-DUSE_APPLE_IDN=ON`.
(The half-detection of Homebrew libidn2 was the result of configuring
with `-DCMAKE_EXE_LINKER_FLAGS=-L$(brew --prefix)/lib`, to fix
linking GnuTLS that needs the `nettle` lib from the brew prefix.)
```
FAIL 1014: [Compare curl --version with curl-config --features] curl-config
```
Ref: https://github.com/curl/curl/actions/runs/9919357748/job/27405080722
Cherry-picked from #14097Closes#14175
In specific builds configs, cmake failed to build test `unit1600`,
due missing an OpenSSL (or wolfSSL) header.
The test code relies on `lib/curl_ntlm_core.h`, which in turn included
TLS library headers. But, dependency header directories are not setup
in cmake for tests, because they should not normally be needed.
The issue was hidden in most builds because TLS headers are usually
found under the system prefix. One counterexample is macOS + Homebrew
LibreSSL builds, where OpenSSL is purposefully unlinked from there to
avoid a mixup with LibreSSL that resides under its own prefix. It was
also hidden in autotools, possibly because it sets up header directories
globally, tests included.
The actual bug however is that `lib/curl_ntlm_core.h` should not include
TLS headers. None of its internal users need it, and `curl_ntlm_core.c`
included them already directly.
Fix it by deleting the TLS header includes from this internal header.
Fixes:
```
In file included from curl/tests/unit/unit1600.c:27:
curl/lib/curl_ntlm_core.h:32:12: fatal error: 'openssl/ssl.h' file not found
# include <openssl/ssl.h>
^~~~~~~~~~~~~~~
```
Ref: https://github.com/curl/curl/actions/runs/9912684737/job/27388041520#step:12:1694
Follow-up to 48eb71ade4#10322
Cherry-picked from #14097Closes#14172
Fix `-Wpointer-bool-conversion` warnings with the method suggested by
both Apple clang and mainline llvm. This was already tried and dropped
in #1705 (in year 2017), but the issue reported there no longer
replicates.
Verified with Apple clang 14, llvm 15, llvm 18 and gcc 11, 14 that the
generated objects are bit by bit identical before and after this patch.
Also:
- stop silencing `-Wtautological-pointer-compare`. This warning don't
seem to be appearing anymore (with or without this patch), at least
with the tested compilers and SDKs (clang 13.1.6-16.0.0beta, llvm 15,
18, gcc 11, 14) and minimum macOS target of 10.8. Older targets fail
to build curl with SecureTransport.
- silence `-Wunreachable-code` for clang only. Previously I applied it
also to GCC, by mistake.
Ref: 8d7172d20a
Apple clang `-Wpointer-bool-conversion`:
```
curl/lib/vtls/sectransp.c:1103:6: error: address of function 'SSLCreateContext' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
if(SSLCreateContext) { /* use the newer API if available */
~~ ^~~~~~~~~~~~~~~~
curl/lib/vtls/sectransp.c:1103:6: note: prefix with the address-of operator to silence this warning
if(SSLCreateContext) { /* use the newer API if available */
^
&
```
Ref: https://github.com/curl/curl/actions/runs/9819538439/job/27113201384#step:8:382
llvm `-Wpointer-bool-conversion`:
```
curl/lib/vtls/sectransp.c:2663:8: error: address of function 'SSLCreateContext' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
if(SSLCreateContext)
~~ ^~~~~~~~~~~~~~~~
curl/lib/vtls/sectransp.c:2663:8: note: prefix with the address-of operator to silence this warning
if(SSLCreateContext)
^
&
```
Ref: https://github.com/curl/curl/actions/runs/9819538439/job/27113200291#step:8:417
gcc still needs `-Waddress` suppressed to avoid these:
```
curl/lib/vtls/n/sectransp.c: In function 'getsubject':
curl/lib/vtls/n/sectransp.c:379:6: warning: the address of 'SecCertificateCopyLongDescription' will always evaluate as 'true' [-Waddress]
379 | if(&SecCertificateCopyLongDescription)
| ^
[...]
```
Follow-up to 59cadacfcc#14128
Follow-up to af271ce9b9#1722
Follow-up to 2b7ce3f56d#1706
Cherry-picked from #14097Closes#14162
- bump parallel test for Linux jobs.
Credit-to: Dan Fandrich
Cherry-picked from #11510
- bump parallel test for macOS jobs.
- drop no longer necessary `-Wno-vla` option.
- fold long lines.
- drop `--enable-maintainer-mode` `./configure` option.
- replace a hard-coded prefix with `brew --prefix`.
- update documentation link.
- move `--enable-debug` in front.
- tidy up quotes.
Closes#14171
This PR fixes a leak and a crash that can happen when curl encounters
bad HTTPS RR values in DNS. We're starting to do better testing of that
kind of thing and e.g. have published bad HTTPS RR values at
dodgy.test.defo.ie.
Closes#14151
Fixup faulty target macro initialization in macOS SDK since v14.4 (as of
15.0 beta). The SDK target detection in `TargetConditionals.h` correctly
detects macOS, but fails to set the macro's old name `TARGET_OS_OSX`,
then continues to set it to a default value of 0. Other parts of the SDK
still rely on the old name, and with this inconsistency our builds fail
due to missing declarations. It happens when using mainline llvm older
than v18. Later versions fixed it by predefining these target macros,
avoiding the faulty dynamic detection. gcc is not affected (for now)
because it lacks the necessary dynamic detection features, so the SDK
falls back to a codepath that sets both the old and new macro to 1.
Also move the `TargetConditionals.h` include to the top of to make sure
including it also for c-ares builds, combined with SecureTransport or
other curl features that may call use an Apple SDK.
Before this patch, affected build combinations (e.g. in GHA runners,
llvm@15 + Xcode 15.3, 15.4, 16.0 with their default SDKs +
SecureTransport) fail with:
```
error: use of undeclared identifier 'noErr'
or 'SecCertificateCopyLongDescription'
or 'SecItemImportExportKeyParameters'
or 'SecExternalFormat'
or 'SecExternalItemType'
or 'SEC_KEY_IMPORT_EXPORT_PARAMS_VERSION'
```
Example:
```
curl/lib/vtls/sectransp.c:311:18: error: use of undeclared identifier 'noErr'
OSStatus rtn = noErr;
^
curl/lib/vtls/sectransp.c:379:7: error: use of undeclared identifier 'SecCertificateCopyLongDescription'
if(&SecCertificateCopyLongDescription)
^
curl/lib/vtls/sectransp.c:381:7: error: call to undeclared function 'SecCertificateCopyLongDescription'; ISO C99 and later do not support implicit function declarations [-Werror,-Wimplicit-function-declaration]
SecCertificateCopyLongDescription(NULL, cert, NULL);
^
curl/lib/vtls/sectransp.c:380:25: error: incompatible integer to pointer conversion assigning to 'CFStringRef' (aka 'const struct __CFString *') from 'int' [-Wint-conversion]
server_cert_summary =
^
[...]
```
Ref: https://github.com/curl/curl/actions/runs/9893867519/job/27330135969#step:10:22
llvm v18 patches implementing the predefined macros:
https://github.com/llvm/llvm-project/pull/746766e1f19168bhttps://github.com/llvm/llvm-project/pull/82833e5ed7b6e2f
Cherry-picked from #14097Closes#14159
Homebrew gcc builds starting with 12.4.0, 13.3.0 and 14.1.0 enabled
the `availability` attribute.
This broke builds because the way the Apple SDK uses attributes (when
available) are incompatible with how gcc accepts them. Causing these
errors:
```
error: attributes should be specified before the declarator in a function definition
error: expected ',' or '}' before
```
Upstream commits implementing the `availability` macro:
gcc-12: fd5530b7cb
gcc-13: cb7e4eca68
gcc-14: ff62a10886
The project above is a Darwin gcc compatibility pack, that is applied
to Homebrew gcc builds.
This patch works by redefining the `availability` macro to an invalid
value, making `__has_attribute(availability)` checks fail, stopping
Apple SDK from inserting the incompatible attributes.
It also replaces the previous, local workaround for `lib/macos.c`.
Example with gcc 12.4.0 with macOS SDK 14.0 (Xcode 15.0.1):
```
In file included from <path-to-sdk>/MacOSX14.0.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CoreFoundation.h:54,
from <path-to-sdk>/MacOSX14.0.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCDynamicStoreCopySpecific.h:30,
from /Users/runner/work/curl/curl/lib/macos.c:33,
from /Users/runner/work/curl/curl/build/lib/CMakeFiles/libcurl_shared.dir/Unity/unity_0_c.c:244:
<path-to-sdk>/MacOSX14.0.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CFUserNotification.h:126:1: error: attributes should be specified before the declarator in a function definition
126 | CF_INLINE CFOptionFlags CFUserNotificationCheckBoxChecked(CFIndex i) API_AVAILABLE(macos(10.0)) API_UNAVAILABLE(ios, watchos, tvos) {return ((CFOptionFlags)(1UL << (8 + i)));}
| ^~~~~~~~~
```
Ref: https://github.com/curl/curl/actions/runs/9787982387/job/27025351601?pr=14096#step:7:18
The gcc vs. llvm/clang incompatibility possibly tracked here upstream:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108796
More info:
https://github.com/llvm/llvm-project/issues/817678433baadechttps://discourse.llvm.org/t/changing-attribute-ast-printing-location-for-gcc-compatibility/73215https://reviews.llvm.org/D159362
Follow-up to db135f8d72#14119
Ref: https://github.com/curl/curl/pull/14091#issuecomment-2222703468Fixes#13700
Cherry-picked from #14097Closes#14155
Also:
- GHA/non-native: install `pkg-config` to detect libidn2 with cmake
on NetBSD and FreeBSD.
- GHA/non-native: tidy-up `curl --version` command if here.
Cherry-picked from #14097Closes#14137
Xcode v15 (2023) or newer requires the built-in macro
`__ENVIRONMENT_OS_VERSION_MIN_REQUIRED__`. This macro is missing from
mainline llvm versions released earlier. llvm v17 introduced it here:
c8e2dd8c6f
This patch defines the missing macro when the necessary conditions
align, by using the value via the macro's old name.
The issue affected SecureTransport builds: The SecureTransport code,
`lib/md4.c` and `lib/md5.c`.
Existing gcc versions (as of v14) also don't define this macro, so apply
the patch to it as well. Even though gcc is incompatible in other ways,
so this isn't fixing an actual curl build case that I could find yet.
GHA macOS runner images have llvm v15 pre-installed, which broke builds
when building with an affected Xcode:
```
curl/lib/md4.c:80:14: error: '__ENVIRONMENT_OS_VERSION_MIN_REQUIRED__' is not defined, evaluates to 0 [-Werror,-Wundef]
(__MAC_OS_X_VERSION_MIN_REQUIRED < 101500)) || \
^
/Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/AvailabilityInternal.h:40:53: note: expanded from macro '__MAC_OS_X_VERSION_MIN_REQUIRED'
#define __MAC_OS_X_VERSION_MIN_REQUIRED __ENVIRONMENT_OS_VERSION_MIN_REQUIRED__
^
In file included from curl/build/lib/CMakeFiles/libcurl_shared.dir/Unity/unity_0_c.c:250:
curl/lib/md5.c:75:14: error: '__ENVIRONMENT_OS_VERSION_MIN_REQUIRED__' is not defined, evaluates to 0 [-Werror,-Wundef]
(__MAC_OS_X_VERSION_MIN_REQUIRED < 101500)) || \
^
/Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/AvailabilityInternal.h:40:53: note: expanded from macro '__MAC_OS_X_VERSION_MIN_REQUIRED'
#define __MAC_OS_X_VERSION_MIN_REQUIRED __ENVIRONMENT_OS_VERSION_MIN_REQUIRED__
^
2 errors generated.
```
Ref: https://github.com/curl/curl/actions/runs/9811974634/job/27095218578#step:4:20
Cherry-picked from #14097Closes#14134
