https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.html
as of 2023-11-29 [1].
Enable new recommended warnings (except `-Wsign-conversion`):
- enable `-Wformat=2` for clang (in both cmake and autotools).
- add `CURL_PRINTF()` internal attribute and mark functions accepting
printf arguments with it. This is a copy of existing
`CURL_TEMP_PRINTF()` but using `__printf__` to make it compatible
with redefinting the `printf` symbol:
https://gcc.gnu.org/onlinedocs/gcc-3.0.4/gcc_5.html#SEC94
- fix `CURL_PRINTF()` and existing `CURL_TEMP_PRINTF()` for
mingw-w64 and enable it on this platform.
- enable `-Wimplicit-fallthrough`.
- enable `-Wtrampolines`.
- add `-Wsign-conversion` commented with a FIXME.
- cmake: enable `-pedantic-errors` the way we do it with autotools.
Follow-up to d5c0351055#2747
- lib/curl_trc.h: use `CURL_FORMAT()`, this also fixes it to enable format
checks. Previously it was always disabled due to the internal `printf`
macro.
Fix them:
- fix bug where an `set_ipv6_v6only()` call was missed in builds with
`--disable-verbose` / `CURL_DISABLE_VERBOSE_STRINGS=ON`.
- add internal `FALLTHROUGH()` macro.
- replace obsolete fall-through comments with `FALLTHROUGH()`.
- fix fallthrough markups: Delete redundant ones (showing up as
warnings in most cases). Add missing ones. Fix indentation.
- silence `-Wformat-nonliteral` warnings with llvm/clang.
- fix one `-Wformat-nonliteral` warning.
- fix new `-Wformat` and `-Wformat-security` warnings.
- fix `CURL_FORMAT_SOCKET_T` value for mingw-w64. Also move its
definition to `lib/curl_setup.h` allowing use in `tests/server`.
- lib: fix two wrongly passed string arguments in log outputs.
Co-authored-by: Jay Satiro
- fix new `-Wformat` warnings on mingw-w64.
[1] 56c0fde389/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C%2B%2B.mdCloses#12489
They are accepted schemes in URLs passed to curl (the tool, not the
library).
Also makes curl-config show the same list.
Co-Authored-by: Jay Satiro
Reported-by: Chara White
Bug: https://curl.se/mail/archive-2023-12/0026.htmlCloses#12508
char variables if unspecified can be either signed or unsigned depending
on the platform according to the C standard; in most platforms, they are
signed.
This meant that the *i<32 waas always true for bytes with the top bit
set. So they were always getting encoded as \uXXXX, and then since they
were also signed negative, they were getting extended with 1s causing
'\xe2' to be expanded to \uffffffe2, for example:
$ curl --variable 'v=“' --expand-write-out '{{v:json}}\n' file:///dev/null
\uffffffe2\uffffff80\uffffff9c
I fixed this bug by making the code use explicitly unsigned char*
variables instead of char* variables.
Test 268 verifies
Reported-by: iconoclasthero
Closes#12434
When the progress bar was made max width (256 columns), the fly()
function attempted to generate its output buffer too long so that the
trailing carriage return would not fit and then the output would show
wrongly. The fly function is called when the expected total transfer is
unknown, which could be one or more progress calls before the actual
progress meter get shown when the expected transfer size is provided.
This new take also replaces the msnprintf() call with a much simpler
memset() for speed.
Reported-by: Tim Hill
Fixes#12407Closes#12415
When the config file parser detects a word that *probably* should be
quoted, mention double-quotes as a possible remedy.
Test 459 verifies.
Proposed-by: Jiehong on github
Fixes#12409Closes#12412
Windows compilers define `_WIN32` automatically. Windows SDK headers
or build env defines `WIN32`, or we have to take care of it. The
agreement seems to be that `_WIN32` is the preferred practice here.
Make the source code rely on that to detect we're building for Windows.
Public `curl.h` was using `WIN32`, `__WIN32__` and `CURL_WIN32` for
Windows detection, next to the official `_WIN32`. After this patch it
only uses `_WIN32` for this. Also, make it stop defining `CURL_WIN32`.
There is a slight chance these break compatibility with Windows
compilers that fail to define `_WIN32`. I'm not aware of any obsolete
or modern compiler affected, but in case there is one, one possible
solution is to define this macro manually.
grepping for `WIN32` remains useful to discover Windows-specific code.
Also:
- extend `checksrc` to ensure we're not using `WIN32` anymore.
- apply minor formatting here and there.
- delete unnecessary checks for `!MSDOS` when `_WIN32` is present.
Co-authored-by: Jay Satiro
Reviewed-by: Daniel Stenberg
Closes#12376
Enable more picky compiler warnings. I've found these options in the
nghttp3 project when implementing the CMake quick picky warning
functionality for it [1].
`-Wunused-macros` was too noisy to keep around, but fixed a few issues
it revealed while testing.
- autotools: reflect the more precisely-versioned clang warnings.
Follow-up to 033f8e2a08#12324
- autotools: sync between clang and gcc the way we set `no-multichar`.
- autotools: avoid setting `-Wstrict-aliasing=3` twice.
- autotools: disable `-Wmissing-noreturn` for MSYS gcc targets [2].
It triggers in libtool-generated stub code.
- lib/timeval: delete a redundant `!MSDOS` guard from a `WIN32` branch.
- lib/curl_setup.h: delete duplicate declaration for `fileno`.
Added in initial commit ae1912cb0d
(1999-12-29). This suggests this may not be needed anymore, but if
it does, we may restore this for those specific (non-Windows) systems.
- lib: delete unused macro `FTP_BUFFER_ALLOCSIZE` since
c1d6fe2aaa.
- lib: delete unused macro `isxdigit_ascii` since
f65f750742.
- lib/mqtt: delete unused macro `MQTT_HEADER_LEN`.
- lib/multi: delete unused macro `SH_READ`/`SH_WRITE`.
- lib/hostip: add `noreturn` function attribute via new `CURL_NORETURN`
macro.
- lib/mprintf: delete duplicate declaration for `Curl_dyn_vprintf`.
- lib/rand: fix `-Wunreachable-code` and related fallouts [3].
- lib/setopt: fix `-Wunreachable-code-break`.
- lib/system_win32 and lib/timeval: fix double declarations for
`Curl_freq` and `Curl_isVistaOrGreater` in CMake UNITY mode [4].
- lib/warnless: fix double declarations in CMake UNITY mode [5].
This was due to force-disabling the header guard of `warnless.h` to
to reapply it to source code coming after `warnless.c` in UNITY
builds. This reapplied declarations too, causing the warnings.
Solved by adding a header guard for the lines that actually need
to be reapplied.
- lib/vauth/digest: fix `-Wunreachable-code-break` [6].
- lib/vssh/libssh2: fix `-Wunreachable-code-break` and delete redundant
block.
- lib/vtls/sectransp: fix `-Wunreachable-code-break` [7].
- lib/vtls/sectransp: suppress `-Wunreachable-code`.
Detected in `else` branches of dynamic feature checks, with results
known at compile-time, e.g.
```c
if(SecCertificateCopySubjectSummary) /* -> true */
```
Likely fixable as a separate micro-project, but given SecureTransport
is deprecated anyway, let's just silence these locally.
- src/tool_help: delete duplicate declaration for `helptext`.
- src/tool_xattr: fix `-Wunreachable-code`.
- tests: delete duplicate declaration for `unitfail` [8].
- tests: delete duplicate declaration for `strncasecompare`.
- tests/libtest: delete duplicate declaration for `gethostname`.
Originally added in 687df5c8c3
(2010-08-02).
Got complicated later: c49e9683b8
If there are still systems around with warnings, we may restore the
prototype, but limited for those systems.
- tests/lib2305: delete duplicate declaration for
`libtest_debug_config`.
- tests/h2-download: fix `-Wunreachable-code-break`.
[1] a70edb08e9/cmake/PickyWarningsC.cmake
[2] https://ci.appveyor.com/project/curlorg/curl/builds/48553586/job/3qkgjauiqla5fj45?fullLog=true#L1675
[3] https://github.com/curl/curl/actions/runs/6880886309/job/18716044703?pr=12331#step:7:72https://github.com/curl/curl/actions/runs/6883016087/job/18722707368?pr=12331#step:7:109
[4] https://ci.appveyor.com/project/curlorg/curl/builds/48555101/job/9g15qkrriklpf1ut#L204
[5] https://ci.appveyor.com/project/curlorg/curl/builds/48555101/job/9g15qkrriklpf1ut#L218
[6] https://github.com/curl/curl/actions/runs/6880886309/job/18716042927?pr=12331#step:7:290
[7] https://github.com/curl/curl/actions/runs/6891484996/job/18746659406?pr=12331#step:9:1193
[8] https://github.com/curl/curl/actions/runs/6882803986/job/18722082562?pr=12331#step:33:1870Closes#12331
Before this patch some source files were overriding gcc warning options,
but without restoring them at the end of the file. In CMake UNITY builds
these options spilled over to the remainder of the source code,
effecitvely disabling them for a larger portion of the codebase than
intended.
`#pragma clang diagnostic` didn't have such issue in the codebase.
Reviewed-by: Marcel Raad
Closes#12352
To allow passing `LDFLAGS` specific to libcurl (`CURL_LDFLAGS_LIB`) and
curl tool (`CURL_LDFLAGS_BIN`).
This makes it possible to build libcurl and curl with a single
invocation with lib- and tool-specific custom linker flags.
Such flag can be enabling `.map` files, a `.def` file for libcurl DLL,
controlling static/shared, incl. requesting a static curl tool (with
`-static-libtool-libs`) while building both shared and static libcurl.
curl-for-win uses the above and some more.
These options are already supported in `Makefile.mk`. CMake has built-in
variables for this.
Closes#12312
- Don't dereference the past-the-end element when parsing the server's
Content-disposition header.
As 'p' is advanced it can point to the past-the-end element and prior
to this change 'p' could be dereferenced in that case.
Technically the past-the-end element is not out of bounds because dynbuf
(which manages the header line) automatically adds a null terminator to
every buffer and that is not included in the buffer length passed to
the header callback.
Closes https://github.com/curl/curl/pull/12320
GCC 14 introduces a new -Walloc-size included in -Wextra which gives:
```
src/tool_operate.c: In function ‘add_per_transfer’:
src/tool_operate.c:213:5: warning: allocation of insufficient size ‘1’ for type ‘struct per_transfer’ with size ‘480’ [-Walloc-size]
213 | p = calloc(sizeof(struct per_transfer), 1);
| ^
src/var.c: In function ‘addvariable’:
src/var.c:361:5: warning: allocation of insufficient size ‘1’ for type ‘struct var’ with size ‘32’ [-Walloc-size]
361 | p = calloc(sizeof(struct var), 1);
| ^
```
The calloc prototype is:
```
void *calloc(size_t nmemb, size_t size);
```
So, just swap the number of members and size arguments to match the
prototype, as we're initialising 1 struct of size `sizeof(struct
...)`. GCC then sees we're not doing anything wrong.
Closes#12292
- convert ensure_trailing into ensure_trailing_slash
- strdup the URL string to own it proper
- use shorter variable names
- combine some expressions
- simplify error handling in ipfs_gateway()
- add MAX_GATEWAY_URL_LEN + proper bailout if maximum is reached
- ipfs-gateway.d polish and simplification
- shorten ipfs error message + make them "synthetic"
Closes#12281
- Pelles C: Unclear status, failed to obtain a fresh copy a few months
ago. Possible website is HTTP-only. ~10 years ago I left this compiler
dealing with crashes and other issues with no response on the forum
for years. It has seen some activity in curl back in 2021.
- LCC: Last stable release in September 2002.
- Salford C: Misses winsock2 support, possibly abandoned? Last mentioned
in 2006.
- Borland C++: We dropped Borland C++ support in 2018.
- MS Visual C++ 6.0: Released in 1998. curl already requires VS 2010
(or possibly 2008) as a minimum.
Closes#12222
Make sure 'inputpath' only points to memory allocated by libcurl so that
curl_free works correctly.
Pointed out by Coverity
Follow-up to 859e88f653Closes#12280
Previously just ipfs://<cid> and ipns://<cid> was supported, which is
too strict for some usecases.
This patch allows paths and query arguments to be used too.
Making this work according to normal http semantics:
ipfs://<cid>/foo/bar?key=val
ipns://<cid>/foo/bar?key=val
The gateway url support is changed.
It now only supports gateways in the form of:
http://<gateway>/foo/bar
http://<gateway>
Query arguments here are explicitly not allowed and trigger an intended
malformed url error.
There also was a crash when IPFS_PATH was set with a non trailing
forward slash. This has been fixed.
Lastly, a load of test cases have been added to verify the above.
Reported-by: Steven Allen
Fixes#12148Closes#12152
- Pass missing parameter for 'lpNumberOfCharsWritten' to WriteConsoleW()
function.
Apparently this parameter was *not* optional on older Windows versions.
Issue observed on Windows XP SP2. Issue not observed on Windows 7 SP1.
So at some point between those two Microsoft changed the behavior.
Prior to this change, on those versions if parameter is NULL then the
function call fails with error ERROR_INVALID_ACCESS.
Regression since af3f4e41.
Ref: https://github.com/MicrosoftDocs/Console-Docs/issues/299
Fixes https://github.com/curl/curl/issues/12131
Closes https://github.com/curl/curl/pull/12130
Currently, curl allows users to specify absurd request rates that might
be higher than the number of milliseconds in the unit (ex: curl --rate
3600050/h http://localhost:8080 does not error out despite there being
only 3600000ms in a hour).
This change adds a conditional check before the millisecond calculation
making sure that the number is not higher than the numerator (the unit)
If the number is higher, curl errors out with PARAM_NUMBER_TOO_LARGE
Closes#12116
Since the tool itself now uses the base64 code using the curlx way, it
needs to build also when the tool needs it. Starting now, the tool build
defines BULDING_CURL to allow lib-side code to use it.
Follow-up to 2e160c9c65Closes#12010
Found the root cause of the startup crash in unity builds with Unicode
and TrackMemory enabled at the same time.
We must make sure that the `memdebug.h` header doesn't apply to
`lib/curl_multibyte.c` (as even noted in a comment there.) In unity
builds all headers apply to all sources, including `curl_multibyte.c`.
This probably resulted in an infinite loop on startup.
Exclude this source from unity compilation with TrackMemory enabled,
in both libcurl and curl tool. Enable unity mode for a debug Unicode
CI job to keep it tested. Also delete the earlier workaround that
fully disabled unity for affected builds.
Follow-up to d82b080f63#12005
Follow-up to 3f8fc25720#11095Closes#11928
Earlier this year we changed our own stderr variable to use the standard
name `stderr` (to avoid bugs where someone is using `stderr` instead of
the curl-tool specific variable). This solution needed to override the
standard `stderr` symbol via the preprocessor. This in turn didn't play
well with unity builds and caused curl tool to crash or stay silent due
to an uninitialized stderr. This was a hard to find issue, fixed by
manually breaking out one file from the unity sources.
To avoid two these two tricks, this patch implements a different
solution: Restore using our own local variable for our stderr output and
leave `stderr` as-is. To avoid using `stderr` by mistake, add a
`checksrc` rule (based on logic we already used in lib for `strerror`)
that detects any `stderr` use in `src` and points to using our own
variable instead: `tool_stderr`.
Follow-up to 06133d3e9b
Follow-up to 2f17a9b654Closes#11958
Switch to plain getenv() from curl_getenv() to avoid the allocation and
having to keep track of which free() or curl_free() that need to be
used.
Coverity found issues and a memory leak.
Follow-up to 65b563a96aCloses#11959
Delete checks and guards for standard C89 headers and assume these are
available: `stdio.h`, `string.h`, `time.h`, `setjmp.h`, `stdlib.h`,
`stddef.h`, `signal.h`.
Some of these we already used unconditionally, some others we only used
for feature checks.
Follow-up to 9c7165e96a#11918 (for `stdio.h` in CMake)
Closes#11940
Before this patch, in certain build configurations the curl tool may
not have displayed anything (debug, macOS), or crashed at startup
(debug, Windows).
Follow-up to 3f8fc25720
Necessary after 2f17a9b654Closes#11929
Came up while testing debug/testing build for Windows. I'm not sure why
it didn't come up in earlier tests with similar config.
`tool_hugehelp.c` might indeed require `zlib.h` and without linking
`CURL_LIBS` to the `curltool` target, CMake doesn't seem to add detected
dependency headers to the compiler command.
```
[ 25%] Building C object src/CMakeFiles/curltool.dir/tool_hugehelp.c.obj
cd .../curl/bld-cmake-llvm-x64/src && /usr/local/opt/llvm/bin/clang
--target=x86_64-w64-mingw32 --sysroot=/usr/local/opt/mingw-w64/toolchain-x86_64
-DCURLDEBUG -DCURL_STATICLIB -DHAVE_CONFIG_H -DUNICODE -DUNITTESTS -D_UNICODE
-I.../curl/include -I.../curl/lib -I.../curl/bld-cmake-llvm-x64/lib
-I.../curl/bld-cmake-llvm-x64/include -I.../curl/src -Wno-unused-command-line-argument
-D_UCRT -DDEBUGBUILD -DHAS_ALPN -DUSE_MANUAL=1 -fuse-ld=lld -Wl,-s -static-libgcc
-lucrt [...] -O3 -DNDEBUG -municode -MD
-MT src/CMakeFiles/curltool.dir/tool_hugehelp.c.obj
-MF CMakeFiles/curltool.dir/tool_hugehelp.c.obj.d
-o CMakeFiles/curltool.dir/tool_hugehelp.c.obj -c .../curl/bld-cmake-llvm-x64/src/tool_hugehelp.c
.../curl/bld-cmake-llvm-x64/src/tool_hugehelp.c:6:10: fatal error: 'zlib.h' file not found
6 | #include <zlib.h>
| ^~~~~~~~
```
Follow-up to 39e7c22bb4Closes#11927
The linker resolves this automatically in non-unity builds. In unity
builds the linker cannot drop a single object with the duplicates,
resulting in these errors. The root issue is that we started including
certain objects both via both libcurlu and libcurltool libs.
Regression from 39e7c22bb4
Windows errors:
```
[ 3%] Linking C executable unit1303.exe
[ 3%] Building C object tests/server/CMakeFiles/rtspd.dir/__/__/lib/curl_multibyte.c.obj
../../lib/libcurlu-d.a(unity_0.c.obj): In function `curlx_convert_UTF8_to_wchar':
C:/projects/curl/lib/curl_multibyte.c:44: multiple definition of `curlx_convert_UTF8_to_wchar'
../../src/libcurltool-d.a(unity_0.c.obj):C:/projects/curl/lib/curl_multibyte.c:44: first defined here
../../lib/libcurlu-d.a(unity_0.c.obj): In function `curlx_convert_wchar_to_UTF8':
C:/projects/curl/lib/curl_multibyte.c:66: multiple definition of `curlx_convert_wchar_to_UTF8'
../../src/libcurltool-d.a(unity_0.c.obj):C:/projects/curl/lib/curl_multibyte.c:66: first defined here
../../lib/libcurlu-d.a(unity_0.c.obj): In function `curlx_win32_open':
C:/projects/curl/lib/curl_multibyte.c:92: multiple definition of `curlx_win32_open'
../../src/libcurltool-d.a(unity_0.c.obj):C:/projects/curl/lib/curl_multibyte.c:92: first defined here
../../lib/libcurlu-d.a(unity_0.c.obj): In function `curlx_win32_fopen':
C:/projects/curl/lib/curl_multibyte.c:120: multiple definition of `curlx_win32_fopen'
../../src/libcurltool-d.a(unity_0.c.obj):C:/projects/curl/lib/curl_multibyte.c:120: first defined here
../../lib/libcurlu-d.a(unity_0.c.obj): In function `curlx_win32_stat':
[...]
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/48110107/job/nvlhpt9aa4ehny5q#L247
macOS errors:
```
[ 56%] Linking C executable unit1302
duplicate symbol '_curlx_sotouz' in:
../../lib/libcurlu.a(unity_0_c.c.o)
../../src/libcurltool.a(unity_0_c.c.o)
duplicate symbol '_curlx_sitouz' in:
../../lib/libcurlu.a(unity_0_c.c.o)
../../src/libcurltool.a(unity_0_c.c.o)
duplicate symbol '_curlx_uztosz' in:
../../lib/libcurlu.a(unity_0_c.c.o)
../../src/libcurltool.a(unity_0_c.c.o)
[...]
```
with config:
```
-DCMAKE_UNITY_BUILD=ON \
-DENABLE_DEBUG=ON -DBUILD_TESTING=ON -DCMAKE_C_FLAGS=-DDEBUGBUILD \
-DBUILD_SHARED_LIBS=ON \
-DBUILD_STATIC_LIBS=OFF
```
Closes#11926
Seen with llvm 17 on Windows x64.
```
.../curl/tests/server/rtspd.c:136:13: warning: no previous extern declaration for non-static variable 'logdir' [-Wmissing-variable-declarations]
136 | const char *logdir = "log";
| ^
.../curl/tests/server/rtspd.c:136:7: note: declare 'static' if the variable is not intended to be used outside of this translation unit
136 | const char *logdir = "log";
| ^
.../curl/tests/server/rtspd.c:137:6: warning: no previous extern declaration for non-static variable 'loglockfile' [-Wmissing-variable-declarations]
137 | char loglockfile[256];
| ^
.../curl/tests/server/rtspd.c:137:1: note: declare 'static' if the variable is not intended to be used outside of this translation unit
137 | char loglockfile[256];
| ^
.../curl/tests/server/fake_ntlm.c:43:13: warning: no previous extern declaration for non-static variable 'logdir' [-Wmissing-variable-declarations]
43 | const char *logdir = "log";
| ^
.../curl/tests/server/fake_ntlm.c:43:7: note: declare 'static' if the variable is not intended to be used outside of this translation unit
43 | const char *logdir = "log";
| ^
.../curl/src/tool_doswin.c:350:8: warning: possible misuse of comma operator here [-Wcomma]
350 | ++d, ++s;
| ^
.../curl/src/tool_doswin.c:350:5: note: cast expression to void to silence warning
350 | ++d, ++s;
| ^~~
| (void)( )
```
```
.../curl/tests/libtest/lib540.c:146:27: warning: result of comparison 'long' > 2147483647 is always false [-Wtautological-type-limit-compare]
146 | int itimeout = (L > (long)INT_MAX) ? INT_MAX : (int)L;
| ~ ^ ~~~~~~~~~~~~~
1 warning generated.
.../curl/tests/libtest/libntlmconnect.c:195:31: warning: result of comparison 'long' > 2147483647 is always false [-Wtautological-type-limit-compare]
195 | int itimeout = (timeout > (long)INT_MAX) ? INT_MAX : (int)timeout;
| ~~~~~~~ ^ ~~~~~~~~~~~~~
1 warning generated.
.../curl/tests/libtest/lib591.c:117:31: warning: result of comparison 'long' > 2147483647 is always false [-Wtautological-type-limit-compare]
117 | int itimeout = (timeout > (long)INT_MAX) ? INT_MAX : (int)timeout;
| ~~~~~~~ ^ ~~~~~~~~~~~~~
1 warning generated.
.../curl/tests/libtest/lib597.c:99:31: warning: result of comparison 'long' > 2147483647 is always false [-Wtautological-type-limit-compare]
99 | int itimeout = (timeout > (long)INT_MAX) ? INT_MAX : (int)timeout;
| ~~~~~~~ ^ ~~~~~~~~~~~~~
1 warning generated.
```
Seen on macOS Intel:
```
.../curl/tests/server/sws.c:440:64: warning: field precision should have type 'int', but argument has type 'size_t' (aka 'unsigned long') [-Wformat]
msnprintf(logbuf, sizeof(logbuf), "Got request: %s %.*s HTTP/%d.%d",
~~^~
1 warning generated.
```
Closes#11925
- ipfs://<cid>
- ipns://<cid>
This allows you tu use ipfs in curl like:
curl ipfs://<cid>
and
curl ipns://<cid>
For more information consult the readme at:
https://curl.se/docs/ipfs.htmlCloses#8805
