Commit Graph

30579 Commits

Author SHA1 Message Date
Daniel Stenberg
67e9e90f96
examples/ftpuploadresume.c: add use of CURLOPT_ACCEPTTIMEOUT_MS
For show

Closes #11277
2023-06-08 23:37:43 +02:00
Daniel Stenberg
4894ca6813
examples/unixsocket.c: example using CURLOPT_UNIX_SOCKET_PATH
and alternatively CURLOPT_ABSTRACT_UNIX_SOCKET

Closes #11276
2023-06-08 23:36:55 +02:00
Anssi Kolehmainen
c72edfa8db
docs: fix missing parameter names in examples
Closes #11278
2023-06-08 19:57:08 +02:00
Daniel Stenberg
3c9256c8a0
urlapi: have *set(PATH) prepend a slash if one is missing
Previously the code would just do that for the path when extracting the
full URL, which made a subsequent curl_url_get() of the path to
(unexpectedly) still return it without the leading path.

Amend lib1560 to verify this. Clarify the curl_url_set() docs about it.

Bug: https://curl.se/mail/lib-2023-06/0015.html
Closes #11272
Reported-by: Pedro Henrique
2023-06-08 16:08:45 +02:00
Dan Fandrich
7d62f0d9b8 runtests; give each server a unique log lock file
Logs are written by several servers and all of them must be finished
writing before the test results can be determined. This means each
server must have its own lock file rather than sharing a single one,
which is how it was done up to now. Previously, the first server to
complete a test would clear the lock before the other server was done,
which caused flaky tests.

Lock files are now all found in their own directory, so counting locks
equals counting the files in that directory.  The result is that the
proxy logs are now reliably written which actually changes the expected
output for two tests.

Fixes #11231
Closes #11259
2023-06-07 16:02:58 -07:00
Dan Fandrich
7af151ded0 runtests: make test file directories in log/N
Test files in subdirectories were not created after parallel test log
directories were moved down a level due to a now-bad comparison.

Follow-up to 92d7dd39

Ref #11264
Closes #11267
2023-06-07 15:09:35 -07:00
Daniel Stenberg
78886afb50
ws: make the curl_ws_meta() return pointer a const
The returned info is read-only for the user.

Closes #11261
2023-06-07 23:37:21 +02:00
Daniel Stenberg
44296dc6ec
RELEASE-NOTES: synced 2023-06-07 23:35:51 +02:00
Daniel Stenberg
92d7dd3955
runtests: move parallel log dirs from logN to log/N
Having several hundreds of them in there gets annoying.

Closes #11264
2023-06-07 23:32:41 +02:00
Dan Fandrich
c6d97bcea6 test447: move the test file into %LOGDIR 2023-06-07 13:53:08 -07:00
Viktor Szakats
3f8fc25720
cmake: add support for "unity" builds
Aka "jumbo" or "amalgamation" builds. It means to compile all sources
per target as a single C source. This is experimental.

You can enable it by passing `-DCMAKE_UNITY_BUILD=ON` to cmake.
It requires CMake 3.16 or newer.

It makes builds (much) faster, allows for better optimizations and tends
to promote less ambiguous code.

Also add a new AppVeyor CI job and convert an existing one to use
"unity" mode (one MSVC, one MinGW), and enable it for one macOS CI job.

Fix related issues:
- add missing include guard to `easy_lock.h`.
- rename static variables and functions (and a macro) with names reused
  across sources, or shadowed by local variables.
- add an `#undef` after use.
- add a missing `#undef` before use.
- move internal definitions from `ftp.h` to `ftp.c`.
- `curl_memory.h` fixes to make it work when included repeatedly.
- stop building/linking curlx bits twice for a static-mode curl tool.
  These caused doubly defined symbols in unity builds.
- silence missing extern declarations compiler warning for ` _CRT_glob`.
- fix extern declarations for `tool_freq` and `tool_isVistaOrGreater`.
- fix colliding static symbols in debug mode: `debugtime()` and
  `statename`.
- rename `ssl_backend_data` structure to unique names for each
  TLS-backend, along with the `ssl_connect_data` struct member
  referencing them. This required adding casts for each access.
- add workaround for missing `[P]UNICODE_STRING` types in certain Windows
  builds when compiling `lib/ldap.c`. To support "unity" builds, we had
  to enable `SCHANNEL_USE_BLACKLISTS` for Schannel (a Windows
  `schannel.h` option) _globally_. This caused an indirect inclusion of
  Windows `schannel.h` from `ldap.c` via `winldap.h` to have it enabled
  as well. This requires `[P]UNICODE_STRING` types, which is apperantly
  not defined automatically (as seen with both MSVS and mingw-w64).
  This patch includes `<subauth.h>` to fix it.
  Ref: https://github.com/curl/curl/runs/13987772013
  Ref: https://dev.azure.com/daniel0244/curl/_build/results?buildId=15827&view=logs&jobId=2c9f582d-e278-56b6-4354-f38a4d851906&j=2c9f582d-e278-56b6-4354-f38a4d851906&t=90509b00-34fa-5a81-35d7-5ed9569d331c
- tweak unity builds to compile `lib/memdebug.c` separately in memory
  trace builds to avoid PP confusion.
- force-disable unity for test programs.
- do not compile and link libcurl sources to libtests _twice_ when libcurl
  is built in static mode.

KNOWN ISSUES:
- running tests with unity builds may fail in cases.
- some build configurations/env may not compile in unity mode. E.g.:
  https://ci.appveyor.com/project/curlorg/curl/builds/47230972/job/51wfesgnfuauwl8q#L250

Ref: https://github.com/libssh2/libssh2/issues/1034
Ref: https://cmake.org/cmake/help/latest/prop_tgt/UNITY_BUILD.html
Ref: https://en.wikipedia.org/wiki/Unity_build

Closes #11095
2023-06-07 13:06:08 +00:00
Daniel Stenberg
e812473d1e
examples/websocket.c: websocket example using CONNECT_ONLY
Closes #11262
2023-06-07 14:56:02 +02:00
Daniel Stenberg
0e339b9e72
websocket-cb: example doing WebSocket download using callback
Very basic

Closes #11260
2023-06-07 14:52:00 +02:00
Daniel Stenberg
cd18e5c464
test/.gitignore: ignore log* 2023-06-07 14:42:48 +02:00
Dan Fandrich
6e4fedeef7 runtests: document the -j parallel testing option
Reported-by: Daniel Stenberg
Ref: #10818
Closes #11255
2023-06-05 13:39:44 -07:00
Dan Fandrich
f24b4b980d runtests: create multiple test runners when requested
Parallel testing is enabled by using a nonzero value for the -j option
to runtests.pl. Performant values seem to be about 7*num CPU cores, or
1.3*num CPU cores if Valgrind is in use.

Flaky tests due to improper log locking (bug #11231) are exacerbated
while parallel testing, so it is not enabled by default yet.

Fixes #10818
Closes #11246
2023-06-05 11:16:06 -07:00
Dan Fandrich
d454af4737 runtests: handle repeating tests in multiprocess mode
Such as what happens with the --repeat option.  Some functions are
changed to pass the runner ID instead of relying on the non-unique test
number.

Ref: #10818
2023-06-05 11:16:06 -07:00
Dan Fandrich
51c22af890 runtests: buffer logmsg while running singletest()
This allows all messages relating to a single test case to be displayed
together at the end of the test.

Ref: #10818
2023-06-05 11:16:06 -07:00
Dan Fandrich
296baf45a0 runtests: call initserverconfig() in the runner
This must be done so variables pick up the runner's unique $LOGDIR.

Ref: #10818
2023-06-05 11:16:06 -07:00
Dan Fandrich
78d8bc4c63 runtests: use a per-runner random seed
Each runner needs a unique random seed to reduce the chance of port
number collisions. The new scheme uses a consistent per-runner source of
randomness which results in deterministic behaviour, as it did before.

Ref: #10818
2023-06-05 11:16:06 -07:00
Dan Fandrich
4317c5549b runtests: complete main test loop refactor for multiple runners
The main test loop is now able to handle multiple runners, or no
additional runner processes at all. At most one process is still
created, however.

Ref: #10818
2023-06-05 11:16:06 -07:00
Dan Fandrich
acc0a92897 runtests: prepare main test loop for multiple runners
Some variables are expanded to arrays and hashes so that multiple
runners can be used for running tests.

Ref: #10818
2023-06-05 11:16:06 -07:00
Stefan Eissing
73022b52c1
bufq: make write/pass methods more robust
- related to #11242 where curl enters busy loop when
  sending http2 data to the server

Closes #11247
2023-06-05 20:11:41 +02:00
Boris Verkhovskiy
b832cab112
tool_getparam: fix comment
Closes #11253
2023-06-05 20:10:36 +02:00
Raito Bezarius
0a75964d0d
haproxy: add --haproxy-clientip flag to spoof client IPs
CURLOPT_HAPROXY_CLIENT_IP in the library

Closes #10779
2023-06-05 20:08:37 +02:00
Daniel Stenberg
9ad23c38e5
curl: add --ca-native and --proxy-ca-native
These are two boolean options to ask curl to use the native OS's CA
store when verifying TLS servers. For peers and for proxies
respectively.

They currently only have an effect for curl on Windows when built to use
OpenSSL for TLS.

Closes #11049
2023-06-05 20:07:11 +02:00
Viktor Szakats
c78a185df7
build: drop unused/redundant HAVE_WINLDAP_H
Sources did not use it. Autotools used it when checking for the
`winldap` library, which is redundant.

With CMake, detection was broken:
```
Run Build Command(s):/usr/local/Cellar/cmake/3.26.3/bin/cmake -E env VERBOSE=1 /usr/bin/make -f Makefile cmTC_2d8fe/fast && /Library/Developer/CommandLineTools/usr/bin/make  -f CMakeFiles/cmTC_2d8fe.dir/build.make CMakeFiles/cmTC_2d8fe.dir/build
Building C object CMakeFiles/cmTC_2d8fe.dir/HAVE_WINLDAP_H.c.obj
/usr/local/opt/llvm/bin/clang --target=x86_64-w64-mingw32 --sysroot=/usr/local/opt/mingw-w64/toolchain-x86_64 -D_WINSOCKAPI_="" -I/my/quictls/x64-ucrt/usr/include -I/my/zlib/x64-ucrt/usr/include -I/my/brotli/x64-ucrt/usr/include -Wno-unused-command-line-argument   -D_UCRT -DCURL_HIDDEN_SYMBOLS -DHAVE_SSL_SET0_WBIO -DHAS_ALPN -DNGHTTP2_STATICLIB -DNGHTTP3_STATICLIB -DNGTCP2_STATICLIB -DUSE_MANUAL=1  -fuse-ld=lld -Wl,-s -static-libgcc  -lucrt  -Wextra -Wall -pedantic -Wbad-function-cast -Wconversion -Winline -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wno-long-long -Wno-multichar -Wpointer-arith -Wshadow -Wsign-compare -Wundef -Wunused -Wwrite-strings -Wcast-align -Wdeclaration-after-statement -Wempty-body -Wendif-labels -Wfloat-equal -Wignored-qualifiers -Wno-format-nonliteral -Wno-sign-conversion -Wno-system-headers -Wstrict-prototypes -Wtype-limits -Wvla -Wshift-sign-overflow -Wshorten-64-to-32 -Wdouble-promotion -Wenum-conversion -Wunused-const-variable -Wcomma -Wmissing-variable-declarations -Wassign-enum -Wextra-semi-stmt  -MD -MT CMakeFiles/cmTC_2d8fe.dir/HAVE_WINLDAP_H.c.obj -MF CMakeFiles/cmTC_2d8fe.dir/HAVE_WINLDAP_H.c.obj.d -o CMakeFiles/cmTC_2d8fe.dir/HAVE_WINLDAP_H.c.obj -c /my/curl/bld-cmake-llvm-x64-shared/CMakeFiles/CMakeScratch/TryCompile-3JP6dR/HAVE_WINLDAP_H.c
In file included from /my/curl/bld-cmake-llvm-x64-shared/CMakeFiles/CMakeScratch/TryCompile-3JP6dR/HAVE_WINLDAP_H.c:2:
In file included from /usr/local/opt/mingw-w64/toolchain-x86_64/x86_64-w64-mingw32/include/winldap.h:17:
In file included from /usr/local/opt/mingw-w64/toolchain-x86_64/x86_64-w64-mingw32/include/schnlsp.h:9:
In file included from /usr/local/opt/mingw-w64/toolchain-x86_64/x86_64-w64-mingw32/include/schannel.h:10:
/usr/local/opt/mingw-w64/toolchain-x86_64/x86_64-w64-mingw32/include/wincrypt.h:5041:254: error: unknown type name 'PSYSTEMTIME'
  WINIMPM PCCERT_CONTEXT WINAPI CertCreateSelfSignCertificate (HCRYPTPROV_OR_NCRYPT_KEY_HANDLE hCryptProvOrNCryptKey, PCERT_NAME_BLOB pSubjectIssuerBlob, DWORD dwFlags, PCRYPT_KEY_PROV_INFO pKeyProvInfo, PCRYPT_ALGORITHM_IDENTIFIER pSignatureAlgorithm, PSYSTEMTIME pStartTime, PSYSTEMTIME pEndTime, PCERT_EXTENSIONS pExtensions);
                                                                                                                                                                                                                                                             ^
/usr/local/opt/mingw-w64/toolchain-x86_64/x86_64-w64-mingw32/include/wincrypt.h:5041:278: error: unknown type name 'PSYSTEMTIME'
  WINIMPM PCCERT_CONTEXT WINAPI CertCreateSelfSignCertificate (HCRYPTPROV_OR_NCRYPT_KEY_HANDLE hCryptProvOrNCryptKey, PCERT_NAME_BLOB pSubjectIssuerBlob, DWORD dwFlags, PCRYPT_KEY_PROV_INFO pKeyProvInfo, PCRYPT_ALGORITHM_IDENTIFIER pSignatureAlgorithm, PSYSTEMTIME pStartTime, PSYSTEMTIME pEndTime, PCERT_EXTENSIONS pExtensions);
                                                                                                                                                                                                                                                                                     ^
2 errors generated.
make[1]: *** [CMakeFiles/cmTC_2d8fe.dir/HAVE_WINLDAP_H.c.obj] Error 1
make: *** [cmTC_2d8fe/fast] Error 2
exitCode: 2
```

Cherry-picked from #11095 88e4a21ff70ccef391cf99c8165281ff81374503
Reviewed-by: Daniel Stenberg
Closes #11245
2023-06-05 16:31:38 +00:00
Daniel Stenberg
ba669d072d
urlapi: scheme starts with alpha
Add multiple tests to lib1560 to verify

Fixes #11249
Reported-by: ad0p on github
Closes #11250
2023-06-05 16:28:27 +02:00
Daniel Stenberg
6854b6c58e
RELEASE-NOTES: synced 2023-06-03 23:52:39 +02:00
Daniel Stenberg
fff65550bc
CURLOPT_MAIL_RCPT_ALLOWFAILS: replace CURLOPT_MAIL_RCPT_ALLLOWFAILS
Deprecate the name using three Ls and prefer the name with two.

Replaces #10047
Closes #11218
2023-06-03 23:29:04 +02:00
Daniel Stenberg
8cf4189529
tests/servers: generate temp names in /tmp for unix domain sockets
... instead of putting them in the regular pid directories because
systems generally have strict length requirements for the path name to
be shorter than 107 bytes and we easily hit that boundary otherwise.

The new concept generates two random names: one for the socks daemon and
one for http.

Reported-by: Andy Fiddaman
Fixes #11152
Closes #11166
2023-06-02 23:10:46 +02:00
Stefan Eissing
f4b5c88ab7
http2: better support for --limit-rate
- leave transfer loop when --limit-rate is in effect and has
  been received
- adjust stream window size to --limit-rate plus some slack
  to make the server observe the pacing we want
- add test case to confirm behaviour

Closes #11115
2023-06-02 23:06:12 +02:00
Stefan Eissing
e054a16830
curl_log: evaluate log statement only when transfer is verbose
Closes #11238
2023-06-02 23:03:45 +02:00
Daniel Stenberg
93df7134b5
libssh2: provide error message when setting host key type fails
Ref: https://curl.se/mail/archive-2023-06/0001.html

Closes #11240
2023-06-02 23:00:27 +02:00
Igor Todorovski
310eb47168
system.h: remove __IBMC__/__IBMCPP__ guards and apply to all z/OS compiles
Closes #11241
2023-06-02 22:55:58 +02:00
Daniel Stenberg
d45b76e0d2
docs/SECURITY-PROCESS.md: link to example of previous critical flaw 2023-06-02 18:05:22 +02:00
Mark Seuffert
ec70d14690
README.md: updated link to opencollective
Closes #11232
2023-06-02 11:29:15 +02:00
Daniel Stenberg
10d8404301
libssh2: use custom memory functions
Because of how libssh2_userauth_keyboard_interactive_ex() works: the
libcurl callback allocates memory that is later free()d by libssh2, we
must set the custom memory functions.

Reverts 8b5f100db3

Ref: https://github.com/libssh2/libssh2/issues/1078
Closes #11235
2023-06-01 16:06:39 +02:00
Daniel Stenberg
bfa7006424
test447: test PUTting a file that grows
... and have curl trim the end when it reaches the expected total amount
of bytes instead of over-sending.

Reported-by: JustAnotherArchivist on github
Closes #11223
2023-06-01 13:43:31 +02:00
Daniel Stenberg
1f85420a28
curl: count uploaded data to stop at the originally given size
Closes #11223
Fixes #11222
Reported-by: JustAnotherArchivist on github
2023-06-01 13:43:28 +02:00
Daniel Stenberg
6661bd588d
tool: remove exclamation marks from error/warning messages 2023-06-01 08:19:21 +02:00
Daniel Stenberg
741f7ed4bc
tool: use errorf() for error output
Convert a number of fprintf() calls.
2023-06-01 08:19:11 +02:00
Daniel Stenberg
6d45b9ca9c
tool: remove newlines from all helpf/notef/warnf/errorf calls
Make voutf() always add one.

Closes #11226
2023-06-01 08:18:21 +02:00
Daniel Stenberg
4efa0b5749
tests/servers.pm: pick unused port number with a server socket
This change replaces the previous method of picking a port number at
random to try to start servers on, then retrying up to ten times with
new random numbers each time, with a function that creates a server
socket on port zero, thereby getting a suitable random port set by the
kernel. That server socket is then closed and that port number is used
to setup the actual test server on.

There is a risk that *another* server can be started on the machine in
the time gap, but the server verification feature will detect that.

Closes #11220
2023-05-31 14:00:47 +02:00
Daniel Stenberg
10413994d6
RELEASE-NOTES: synced
bump to 8.2.0
2023-05-31 08:59:48 +02:00
Alejandro R. Sedeño
5a023938fa
configure: fix run-compiler for old /bin/sh
If you try to assign and export on the same line on some older /bin/sh
implementations, it complains:

```
$ export "NAME=value"
NAME=value: is not an identifier
```

This commit rewrites run-compiler's assignments and exports to work with
old /bin/sh, splitting assignment and export into two separate
statements, and only quote the value. So now we have:

```
NAME="value"
export NAME
```

While we're here, make the same change to the two supporting
assign+export lines preceeding the script to be consistent with how
exports work throughout the rest of configure.ac.

Closes #11228
2023-05-31 08:56:23 +02:00
Philip H
9496d32802
circleci: install impacket & wolfssl 5.6.0
Closes #11221
2023-05-31 08:34:22 +02:00
Daniel Stenberg
0807fd72f9
tool_urlglob: use curl_off_t instead of longs
To handle more globs better (especially on Windows)

Closes #11224
2023-05-31 08:14:41 +02:00
Dan Fandrich
a1730b6106 scripts: Fix GHA matrix job detection in cijobs.pl
The parsing is pretty brittle and it broke detecting some jobs at some
point. Also, detect if Windows is used in GHA.
2023-05-30 16:54:13 -07:00
Dan Fandrich
8ac18ee2bb runtests: abort test run after failure without -a
This was broken in a recent refactor and test runs would not stop.

Follow-up to d4a1b5b6

Reported-by: Daniel Stenberg
Fixes #11225
Closes #11227
2023-05-30 13:49:48 -07:00