357 Commits

Author SHA1 Message Date
Stefan Eissing
1ebd92d0fd
async: DoH improvements
Adds a "meta_hash" to each easy handle for keeping special data during
operations. All meta data set needs to add its destructor callback, so
that meta data gets destroyed properly when the easy handle is cleaned
up or reset.

Add data->master_mid for "sub" transfers that belong to a "master" easy
handle. When a "sub" transfer is done, the corresponding "master" can
add a callback to be invoked. Used in DoH name resolution.

DoH: use easy meta hash to add internal structs for DoH name resolution.
One in each in each probe easy handle. When probes are done, response
data is copied from the probe to the initiating easy.

This allows DoH using transfers and their probes to be cleaned up in any
sequence correctly.

Fold DoH cleanup into the Curl_async_shutdown() and Curl_async_destroy()
functions.

Closes #16384
2025-04-16 16:06:03 +02:00
Daniel Stenberg
625f2c1644
lib: include files using known path
by including headers using "../[header]" when done from C files in
subdirectories, we do not need to specify the lib source dir as an
include path and we reduce the risk of header name collisions with
headers in the SDK using the same file names.

Idea-by: Kai Pastor

Ref: #16949
Closes #16991
2025-04-08 17:00:00 +02:00
Daniel Stenberg
eeed87f056
mk-ca-bundle.pl: follow redirects
The Mozilla hosted files have started to redirect. Follow them to restore
script functionality.

Reported-by: Harry Sintonen
Closes #16995
2025-04-08 11:47:05 +02:00
Carlos Henrique Lima Melara
2f5e4e0db4
scripts: completion.pl: sort the completion file for all shells
The reproducible builds effort in Debian has caught a regression in curl
8.13.0-rc1 but we were a bit slow to realize it. The ordering of the
completion file for fish is not deterministic so it can differ between
builds. Since there is no restriction about the order of the completion
file for fish, let's just sort it too.

Closes #16985
2025-04-06 11:35:38 +02:00
Daniel Stenberg
f3527ed443
randdisable: build randomizer
This script makes a "random" build using configure and verifies that it
builds curl correctly. It randomly adds a number of the available
--disable-* flags to configure. When it detects a problem the script
stops, otherwise it continues trying more combinations.

Closes #16962
2025-04-05 14:59:17 +02:00
Viktor Szakats
b809629f8c
cmake: use absolute paths for completion targets
Fixing potential:
```
CMake Error at scripts/CMakeLists.txt:72 (install):
  install FILES given directory "/usr/ports/ftp/curl/work/.build/scripts/" to
  install.
```

Reported-by: Daniel Engberg
Fixes #16946
Follow-up to c8b0f0c9ad78eafc6c8f0005113de346ee797c21 #16833
Closes #16954
2025-04-04 02:34:52 +02:00
Dan Fandrich
eb0c622a5b scripts: fix --opts-dir help in completion.pl
The help text gave the wrong option name.

Reported-by: Daniel Engberg
Ref: #16946
2025-04-03 16:14:34 -07:00
Daniel Stenberg
8c2392e564
contrithanks.sh: drop set -e
Makes the script work again

Closes #16914
2025-04-02 11:44:26 +02:00
Viktor Szakats
c8b0f0c9ad
cmake: add shell completion support
Add the last (*) missing bit for feature parity with autotools.

Also test in CI. Add a new `cmake install` step to GHA/macos.

(*) AFAIK. Let us know if there's something else missing.

Closes #16833
2025-03-28 13:51:38 +01:00
Viktor Szakats
795425f765
configure: tidy up shell completion rules
- allow generating completions in cross-builds.
  Follow-up to d055a01ce90510a6f8ff44bb7fceace9b2dbcf97 #16789

- fix warning messages.

- language, quotes, whitespace.

Closes #16836
2025-03-26 15:31:54 +01:00
tiymat
d055a01ce9
scripts: update completion.pl to parse options from docs
Reported-by: kpcyrd on github
Fixes #16072
Closes #16789
2025-03-24 23:48:26 +01:00
Viktor Szakats
51d8213579
core: stop redefining E* macros on Windows, map EACCES, related fixes
Before this patch, standard `E*` errno codes were redefined on Windows,
onto matching winsock2 `WSA*` error codes, which have different values.
This broke uses where using the `E*` value in non-socket context, or
other places expecting a POSIX `errno`, e.g. file I/O, threads, IDN or
interfacing with dependencies.

Fix it by introducing a curl-specific `SOCKE*` set of macros that map to
`WSA*` on Windows and standard POSIX codes on other platforms. Then
verify and update the code to use `SOCKE*` or `E*` macro depending on
context.

- Add `SOCKE*` macros that map to either winsock2 or POSIX error codes.
  And use them with `SOCKERRNO` or in contexts requiring
  platform-dependent socket error codes.

  This fixes `E*` uses which were supposed be POSIX values, not `WSA*`
  socket errors, on Windows:
  - lib/curl_multibyte.c
  - lib/curl_threads.c
  - lib/idn.c
  - lib/vtls/gtls.c
  - lib/vtls/rustls.c
  - src/tool_cb_wrt.c
  - src/tool_dirhie.c

- Ban `E*` codes having a `SOCKE*` mapping, via checksrc.
  Authored-by: Daniel Stenberg

- Add exceptions for `E*` codes used in file I/O, or other contexts
  requiring POSIX error codes.

Also:
- ftp: fix missing `SOCKEACCES` mapping for Windows.
- add `SOCKENOMEM` for `Curl_getaddrinfo()` via `asyn-thread.c`.
- tests/server/sockfilt: fix to set `SOCKERRNO` in local `select()`
  override on Windows.
- lib/inet_ntop: fix to return `WSAEINVAL` on Windows, where `ENOSPC` is
  used on other platforms. To simulate Windows' built-in `inet_ntop()`,
  as tested on a Win10 machine.
  Note:
  - WINE returns `STATUS_INVALID_PARAMETER` = `0xC000000D`.
  - Microsoft documentation says it returns `WSA_INVALID_PARAMETER`
    (= `ERROR_INVALID_PARAMETER`) 87:
    https://learn.microsoft.com/windows/win32/api/ws2tcpip/nf-ws2tcpip-inet_ntop#return-value
- lib/inet_ntop: drop redundant `CURL_SETERRNO(ENOSPC)`.
  `inet_ntop4()` already sets it before returning `NULL`.
- replace stray `WSAEWOULDBLOCK` with `USE_WINSOCK` macro to detect
  winsock2.
- move existing `SOCKE*` mappings from `tests/server` to
  `curl_setup_once.h`.
- add missing `EINTR`, `EINVAL` constants for WinCE.

Follow-up to abf80aae384319ef9b19ffbd0d69a1fbe7421f1f #16612
Follow-up to d69425ed7d0918aceddd96048b146a9df85638ec #16615
Bug: https://github.com/curl/curl/pull/16553#issuecomment-2704679377

Closes #16621
2025-03-13 00:03:25 +01:00
Daniel Stenberg
67a1cfa5ef
managen: accept more markdown-quote-markers
Lines starting with ``` or ~~~ indicate start and end of a quoted
section.

Closes #16685
2025-03-12 14:36:11 +01:00
Daniel Stenberg
db2fd7c3de
delta: get contributors without changing THANKS
Give contrithanks.sh the option to send the list of names to stdout so
that delta can use it appropriately for counting.

Closes #16449
2025-02-24 13:26:48 +01:00
Daniel Stenberg
8f79b3e696
contrithanks.sh: update docs/THANKS in place
Now using 'sort' for sorting the names. This has the small side-effect
that it sorts slightly different than the previously used sort function
(emacs).

I think this is a better sort and over all it makes it more convenient
to use the script as it removes a manual step.

Closes #16448
2025-02-24 13:04:11 +01:00
Daniel Stenberg
efec626ebb
contributors.sh: lowercase 'github' for consistency
also fix contrithanks and THANKS-filter

Ref: #16438
Closes #16443
2025-02-24 08:05:06 +01:00
Stefan Eissing
1b710381ca
https-rr: implementation improvements
- fold DoH and async HTTPS-RR handling into common code.
  have common cleanups, etc. Have a CURLcode result in async
  handling to allow HTTPS RR parsing to fail.
- keep target, ipv4hints, ipv6hints, port and echconfig also
  when resolving via cares. We need to know `target` and `port`
  when evaluating possible ALPN candidates to not go astray.
- add CURL_TRC_DNS for tracing DNS operations
- replace DoH specific tracing with DNS, use doh as alias
  for dns in curl_global_tracea()

Closes #16132
2025-02-18 16:12:26 +01:00
Jay Satiro
28d3c5dced scripts/managen: fix parsing of markdown code sections
- Terminate a code section before parsing a heading line.

Prior to this change when a code line (eg "    code") was followed
by a heading line (eg "## heading") the code section in the output
was terminated after converting the header instead of before. That led
to some weird formatting outputs depending on the nroff or roffit etc.

With this change:

.nf
curl \--expand\-url https.//example.com/{{url:trim}}
.fi
.IP json

Without this change:

.nf
curl \--expand\-url https.//example.com/{{url:trim}}
.IP json
.fi

Closes https://github.com/curl/curl/pull/16345
2025-02-16 18:52:17 -05:00
Jay Satiro
760bbb2110 scripts/managen: fix option 'single'
- Fix option 'single' to generate single manpages.

As far as I can tell the option did not work prior to this change.

Example: scripts/managen -d docs/cmdline-opts single variable.md

Closes https://github.com/curl/curl/pull/16344
2025-02-16 18:51:30 -05:00
Daniel Stenberg
c561c94ec3
managen: correct the warning for un-escaped '<' and '>'
1. make sure the check is done before the backticks are replaced

2. ignore less-than and greater-than used within backticks

(adjust proxy.md that now showed a two-space warning)

Closes #16315
2025-02-13 11:17:40 +01:00
Daniel Stenberg
71ace9f3c1
checksrc.pl: warn on FIXME/TODO comments 2025-02-10 14:44:12 +01:00
Daniel Stenberg
bb65a52a05
release-notes.pl: manage more "closes" typos 2025-02-08 10:51:27 +01:00
Viktor Szakats
fb70812437
cmake: add integration tests, run them in CI
Add CMake test project consuming curl via these methods:
`FetchContent`, `add_subdirectory()`, `find_package()`.

Also:
- GHA/distcheck: run these tests in CI.
- cmakelint: exclude a warning for calling "wonky-cased" built-in
  CMake functions, such as `FetchContent_Declare()`.

Closes #16126
2025-02-07 00:15:48 +01:00
Viktor Szakats
b13e9066b3
GHA: tidy up apt commands
- drop `--quiet 2` option where used, to have uniform output.
- replace `apt` with `apt-get` in one job. sync options with rest.
- replace deprecated `apt-key` command with the alternative recommended
  by `apt-key(8)`.
- drop stray `cd /tmp`, no longer needed after migrating to GHA.
- shorten `--option Dpkg::Use-Pty=0` to `-o Dpkg::Use-Pty=0`.
- add `-o Dpkg::Use-Pty=0` to hide `apt-get` progress bars taking
  vertical log space, where missing.
- drop `-y --no-install-suggests --no-install-recommends` `apt-get`
  options. They are the default in the ubuntu-24.04 image.
- GHA/distcheck: move `name:` to top in steps where not there.
- scripts/cijobs.pl: catch `apt-get` lines with the `-o` option.

Closes #16127
2025-01-30 02:36:43 +01:00
Daniel Stenberg
b0009d0216
cd2nroff: do not insist on quoted <> within backticks
Ref: #16118
Closes #16121
2025-01-28 17:36:55 +01:00
Viktor Szakats
108b2153ac
Makefile.dist: delete
It had shorthand aliases to launch `./configure` and
`./configure --with-openssl`. The former hasn't worked for a long while
because of missing TLS.

Its `ca-bundle` and `ca-firefox` targets have been broken for 2.5 years
till recently. These targets also exist in `./configure` and have been
working all along.

Also:
- cmake: add support `curl-ca-bundle` and `curl-ca-firefox` targets.
- tests/testcurl.pl: drop obsolete build logic.

Closes #16094
2025-01-27 20:59:47 +01:00
Daniel Stenberg
3833e5fea1
checksrc: check for return with parens around a value/name
Ref: #15979
Closes #15983
2025-01-13 09:10:50 +01:00
Asger Hautop Drewsen
cacceef5a6
completion.pl: add completion for paths after @ for fish
Closes #15928
2025-01-10 08:37:55 +01:00
Daniel Stenberg
616b66e68f
scripts/delta: add number of days since first httpget release 2025-01-08 23:48:48 +01:00
Stefan Eissing
515a21f350
vtls: feature ssls-export for SSL session im-/export
Adds the experimental feature `ssls-export` to libcurl and curl for
importing and exporting SSL sessions from/to a file.

* add functions to libcurl API
* add command line option `--ssl-sessions <filename>` to curl
* add documenation
* add support in configure
* add support in cmake
+ add pytest case

Closes #15924
2025-01-08 23:32:07 +01:00
Daniel Stenberg
c445b7426a
checksrc: introduce 'banfunc' to ban specific functions
Use 'banfunc' and 'allowfunc' in .checksrc to specify which functions to
ban or allow to be used. This saves us from having to edit the script
going forward when we want to ban or allow specific functions.

This replaces a set of previous rules and all banned functions are now
checked with the BANNEDFUNC rule.

There is a set of default banned functions, shown by invoking
./checksrc.

Also, -a and -b options are added to specify allowed or banned functions
on the command line.

Closes #15835
2024-12-30 08:25:56 +01:00
Daniel Stenberg
3f8452dde7
tool_formparse.c: make curlx_uztoso a static in here
And drop the prefix. This function was not use elsewhere and it should
certainly not be present in libcurl code when not used in the library.

Closes #15796
2024-12-21 13:59:05 +01:00
Daniel Stenberg
28dd14aafe
scripts/mdlinkcheck: fix the ../ handling in file links
Follow-up to 62515e8e9d750f

Closes #15797
2024-12-21 13:57:38 +01:00
Daniel Stenberg
a787442e5d
cd2nroff: support "none" as a TLS backend
When we remove support for a specific TLS backend, it might be the only
one that supports a specific feature and then we need to be able to go
"none".

Closes #15769
2024-12-18 08:13:47 +01:00
Daniel Stenberg
54c5cb8b7f
checksrc: fix the return() checker
It would previously wrongly also catch function calls to function names
ending with 'return'

Amended test1185.

Reported-by: Stefan Eissing
Closes #15764
2024-12-17 11:27:55 +01:00
Daniel Stenberg
62515e8e9d
scripts/mdlinkcheck: markdown link checker
This script parses all markdown files in the repository, extracts all
links and verifies that they work.

It makes sure to only check the URLs once, even if used in multiple
links. There is a whitelist for URLs we deem unnecessary to check.

It uses curl to do the checks.

As a bonus, this makes it easy to run this check locally.

Closes #15742
2024-12-14 14:22:30 +01:00
Daniel Stenberg
c2ac9ea1ee
checksrc: ban use of sscanf()
Using sscanf() is not a (security) problem in itself, but we strongly
discorage using it for parsing input since it is hard to use right, easy
to mess up and often makes for sloppy error checking.

Allow it in examples and tests

Closes #15687
2024-12-13 09:43:05 +01:00
andrewkirillov-ibm
4bba14c35d
dmaketgz: use --no-cache when building docker image
Fixes #15689
Closes #15690
2024-12-05 15:41:53 +01:00
Viktor Szakats
2ccd1c5889
cmakelint: fix to check root CMakeLists.txt
Closes #15565
2024-11-13 12:45:06 +01:00
Daniel Stenberg
448df98d92
mk-ca-bundle: remove CKA_NSS_SERVER_DISTRUST_AFTER conditions
The script's previous treatment of this meta-data was a
misunderstanding. (Added in 1ebc53df25181908) The mistrust is not for
the root cert at this date (it would simply be removed from the bundle
then instead) but for created server certificates:

    If a builtin certificate has a CKA_NSS_SERVER_DISTRUST_AFTER
    timestamp before the SCT or NotBefore date of a certificate that
    builtin issued, then clients can elect not to trust it.

That is however information that cannot be provided in the generated PEM
output.

Fixes #15547
Reported-by: Andrew Ayer
Closes #15552
2024-11-12 08:33:57 +01:00
Gabriel Marin
a58584a881
checksrc: add check for spaces around logical AND operators
Closes #15144
2024-10-21 09:35:00 +02:00
Daniel Stenberg
d78e129d50
WebSockets: make support official (non-experimental)
Inverts the configure/cmake options to instead provide options that
disable WebSockets and have them (ws + wss) enabled by default.

Closes #14936
2024-09-27 13:20:25 +02:00
Viktor Szakats
cfae354a9a
codespell: extend checks to more subdirs
- fix issues found.
- fix a few more found locally.

Closes #15072
2024-09-27 10:27:08 +02:00
Viktor Szakats
fcc89619d9
singleuse: make git grep faster, add Apple nm support
- avoid regexp in grep to make it run faster.
- add support for parsing Apple `nm` output:
  - skip leading underscore from function names.
  - pick object name from output.

Closes #15070
2024-09-27 10:27:08 +02:00
Gabriel Marin
cff75acfec
checksrc: Added checks for colon operator in ternary expressions
Closes #14990
2024-09-25 08:01:49 +02:00
Gabriel Marin
a4703dac13
checksrc: fixed typo
Closes #14968
2024-09-22 21:23:08 +02:00
Viktor Szakats
0aece8f66d
tidy-up: indent, whitespace, #error in make files
Replace invalid C with `#error`.

Cherry-picked from #14692
Closes #14997
2024-09-22 09:51:15 +02:00
Viktor Szakats
60c3d04465
autotools: add support for 'unity' builds, enable in CI
Implement the "unity" builds as known from CMake, but for autotools.
It's limited to `lib` and `src` (CMake also supports it in `tests`).

Enable with: `--enable-unity` (disabled by default)

Unity builds speed up builds significantly. Cygwin and Windows builds in
particular, but the effect is noticeable on most systems. It also allows
discovering unity issues with autotools, benefitting also CMake when
building the same combination. In CI it makes turnaround times quicker.

This closes build performance with CMake. autotools still lags behind
because it builds shared and static libcurl in two, separate passes.
CMake does it in one. Manpage compilation isn't batched, it is in CMake.
After unity and test bundle support the slowest parts of the build are
the configuration phase (which is effectively a tedious, non-parallel,
compilation and/or linking of 300+ tiny programs. The next bottleneck
is compiling individual examples and finally test servers (only slow
with autotools).

The autotools implementation is slightly less efficient than CMake,
because 3 sources are permanently excluded while in CMake this isn't
necessary and solved more efficiently while building libtests. There is
also no 'unity' support for tests, making them a less efficient also.

Enable it in CI for most `configure` jobs. Except in GHA/dist (though
it works fine there too), to use the default config there. Also skip for
the Linux AWC-LC job where it made builds time a few seconds longer
(reason undiscovered.)

Autotools test suite builds compared between master -> `--enable-unity`:
- GHA/Linux: 32s -> 12s
  https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
  https://github.com/curl/curl/actions/runs/10742978889/job/29796766297
- GHA/macOS: 37s -> 10s
  https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
  https://github.com/curl/curl/actions/runs/10742978699/job/29796768875
- GHA/FreeBSD: 15m25 -> 10m58 (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
  https://github.com/curl/curl/actions/runs/10742978937/job/29796766115
- GHA/Cygwin: 3m32 -> 1m21
  https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
  https://github.com/curl/curl/actions/runs/10742978645/job/29796756933
- GHA/MSYS2: 2m42 -> 50s
  https://github.com/curl/curl/actions/runs/10705668808/job/29681621166
  https://github.com/curl/curl/actions/runs/10742978662/job/29799739289
- GHA/mingw-w64: 5m32 -> 1m23
  https://github.com/curl/curl/actions/runs/10705668808/job/29681628787
  https://github.com/curl/curl/actions/runs/10742978662/job/29799741568

Closes #14815
2024-09-20 23:53:33 +02:00
Daniel Stenberg
e666a678bd
checksrc: check for spaces around '?', '>' and '<'
Closes #14921
2024-09-19 14:59:12 +02:00
Daniel Stenberg
fbf5d507ce
lib/src: white space edits to comply better with code style
... as checksrc now finds and complains about these.

Closes #14921
2024-09-19 14:59:12 +02:00