To avoid URL tricks, use the URL parser for this.
This update changes curl's behavior slightly in that it will ignore the
possible query part from the URL and only use the file name from the
actual path from the URL. I consider it a bugfix.
"curl -O localhost/name?giveme-giveme" will now save the output in the
local file named 'name'
Updated test 1210 to verify
Assisted-by: Jay Satiro
Closes#9684
"You never needed a pass phrase" reads like it's about to be followed by
something like "until version so-and-so", but that is not what is
intended. Change to "You never need a pass phrase". There are two
instances of this text, so make sure to update both.
When I explicitly declare, that I would like to have curl built with
wolfSSL support using `--with-wolfssl` configure option, then I would
expect, that either I endup with curl having that support, for example
in form of https support or it wouldn't be available at all.
Downstream projects like for example OpenWrt build curl wolfSSL variant
with `--with-wolfssl` already, but in certain corner cases it does fail:
configure:25299: checking for wolfSSL_Init in -lwolfssl
configure:25321: x86_64-openwrt-linux-musl-gcc -o conftest [snip]
In file included from target-x86_64_musl/usr/include/wolfssl/wolfcrypt/dsa.h:33,
from target-x86_64_musl/usr/include/wolfssl/wolfcrypt/asn_public.h:35,
from target-x86_64_musl/usr/include/wolfssl/ssl.h:35,
from conftest.c:47:
target-x86_64_musl/usr/include/wolfssl/wolfcrypt/integer.h:37:14: fatal error: wolfssl/wolfcrypt/sp_int.h: No such file or directory
#include <wolfssl/wolfcrypt/sp_int.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
and in the end thus produces curl without https support:
curl: (1) Protocol "https" not supported or disabled in libcurl
So fix it, by making the working wolfSSL mandatory and error out in
configure step when that's not the case:
checking for wolfSSL_Init in -lwolfssl... no
configure: error: --with-wolfssl but wolfSSL was not found or doesn't work
References: https://github.com/openwrt/packages/issues/19005
References: https://github.com/openwrt/packages/issues/19547
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Closes#9682
- Fixed an issue where is_in_callback was getting cleared when using web
sockets with debug logging enabled
- Ensure the handle is is_in_callback when calling out to fwrite_func
- Change the write vs. send_data decision to whether or not the handle
is in CONNECT_ONLY mode.
- Account for buflen not including the header length in curl_ws_send
Closes#9665
This is a strcmp() alternative function for comparing "secrets",
designed to take the same time no matter the content to not leak
match/non-match info to observers based on how fast it is.
The time this function takes is only a function of the shortest input
string.
Reported-by: Trail of Bits
Closes#9658
If the main_checkfds function cannot create new file descriptors in an
attempt to detect of stdin, stdout or stderr are closed.
Also changed the check to use fcntl() to check if the descriptors are
open, which avoids superfluously calling pipe() if they all already are.
Follow-up to facfa19cdd
Reported-by: Trail of Bits
Closes#9663
curl_ws_recv() now receives data to fill up the provided buffer, but can
return a partial fragment. The function now also get a pointer to a
curl_ws_frame struct with metadata that also mentions the offset and
total size of the fragment (of which you might be receiving a smaller
piece). This way, large incoming fragments will be "streamed" to the
application. When the curl_ws_frame struct field 'bytesleft' is 0, the
final fragment piece has been delivered.
curl_ws_recv() was also adjusted to work with a buffer size smaller than
the fragment size. (Possibly needless to say as the fragment size can
now be 63 bit large).
curl_ws_send() now supports sending a piece of a fragment, in a
streaming manner, in addition to sending the entire fragment in a single
call if it is small enough. To send a huge fragment, curl_ws_send() can
be used to send it in many small calls by first telling libcurl about
the total expected fragment size, and then send the payload in N number
of separate invokes and libcurl will stream those over the wire.
The struct curl_ws_meta() returns is now called 'curl_ws_frame' and it
has been extended with two new fields: *offset* and *bytesleft*. To help
describe the passed on data chunk when a fragment is delivered in many
smaller pieces.
The documentation has been updated accordingly.
Closes#9636
Example programs targeting a deprecated feature/option are commented with
a warning about it.
Other examples are adapted to not use deprecated options.
Closes#9661
Detecting headers and lib separately makes sense when headers come in
variations or with extra ones, but this wasn't the case here. These were
duplicate/parallel macros that we had to keep in sync with each other
for a working build. This patch leaves a single macro for each of these
dependencies:
- Rely on `HAVE_LIBZ`, delete parallel `HAVE_ZLIB_H`.
Also delete CMake logic making sure these two were in sync, along with
a toggle to turn off that logic, called `CURL_SPECIAL_LIBZ`.
Also delete stray `HAVE_ZLIB` defines.
There is also a `USE_ZLIB` variant in `lib/config-dos.h`. This patch
retains it for compatibility and deprecates it.
- Rely on `USE_LIBSSH2`, delete parallel `HAVE_LIBSSH2_H`.
Also delete `LIBSSH2_WIN32`, `LIBSSH2_LIBRARY` from
`winbuild/MakefileBuild.vc`, these have a role when building libssh2
itself. And `CURL_USE_LIBSSH`, which had no use at all.
Also delete stray `HAVE_LIBSSH2` defines.
- Rely on `USE_LIBSSH`, delete parallel `HAVE_LIBSSH_LIBSSH_H`.
Also delete `LIBSSH_WIN32`, `LIBSSH_LIBRARY` and `HAVE_LIBSSH` from
`winbuild/MakefileBuild.vc`, these were the result of copy-pasting the
libssh2 line, and were not having any use.
- Delete unused `HAVE_LIBPSL_H` and `HAVE_LIBPSL`.
Reviewed-by: Daniel Stenberg
Closes#9652
The former way that also suggested using a non-existing file to just
enable the cookie engine could lead to developers maybe a bit carelessly
guessing a file name that will not exist, and then in a future due to
circumstances, such a file could be made to exist and then accidentally
libcurl would read cookies not actually meant to.
Reported-by: Trail of bits
Closes#9654
The ci-test is the normal makefile target invoked in CI jobs. This has
been using the -r option to runtests.pl since a long time, but I find
that it mostly just adds many lines to the test output report without
anyone caring much about those stats.
Remove it.
Closes#9656
This is a bit shorter and a lot safer.
Substrings of unescaped characters are added by a single call to reduce
overhead.
Extend test 1465 to handle more kind of escapes.
Closes#9653
C string hexadecimal-escaped characters may have more than 2 digits.
This results in a wrong C compiler interpretation of a 2-digit escaped
character when followed by an hex digit character.
The solution retained here is to represent such characters as 3-digit
octal escapes.
Adjust and extend test 1465 for this case.
Closes#9643
Exclude linker flags specifying depedency libs and libpaths, when
building against `libcurl.dll`. In such case these options are not
necessary (but may cause errors if not/wrongly configured.)
Also move and reword a comment on `CPPFLAGS` to not apply to
`UNICODE` options. These are necessary for all build targets.
Closes#9651
- Don't show TESTFAIL message (ie tests failed which aren't ignored) if
only ignored tests failed.
Before:
IGNORED: failed tests: 571 612 1056
TESTDONE: 1214 tests out of 1217 reported OK: 99%
Use of uninitialized value $failed in concatenation (.) or string at
./runtests.pl line 6290.
TESTFAIL: These test cases failed:
After:
IGNORED: failed tests: 571 612 1056
TESTDONE: 1214 tests out of 1217 reported OK: 99%
Closes https://github.com/curl/curl/pull/9648
- Correct the use of -all-static for static Windows CI builds.
curl_LDFLAGS was removed from the makefile when metalink support was
removed. LDFLAGS=-all-static is passed to make only, because it is not a
valid option for configure compilation tests.
Closes https://github.com/curl/curl/pull/9633
In a recent commit I mistakenly deleted this logic, after seeing a
reference to a filename ending with `.cvs` and thinking it must have
been long gone. Turns out this is an existing file. Restore the rule
and the necessary `COPY` definitions with it.
The restored logic is required for a successful build on a bare source
tree (as opposed to a source release tarball).
Also shorten an existing condition similar to the one added in this
patch.
Regression since 07a0047882Closes#9645
After this patch, we reduce the three copies of most `Makefile.m32`
logic to one. This now resides in `lib/Makefile.m32`. It makes future
updates easier, the code shorter, with a small amount of added
complexity.
`Makefile.m32` reduction:
| | bytes | LOC total | blank | comment | code |
|-------------------|-------:|----------:|-------:|---------:|------:|
| 7.85.0 | 34772 | 1337 | 79 | 192 | 1066 |
| before this patch | 17601 | 625 | 62 | 106 | 457 |
| after this patch | 11680 | 392 | 52 | 104 | 236 |
Details:
- Change rules to create objects for the `v*` subdirs in the `lib` dir.
This allows to use a shared compile rule and assumes that filenames
are not (and will not be) colliding across these directories.
`Makefile.m32` now also stores a list of these subdirs. They are
changing rarely though.
- Sync as much as possible between the three `Makefile.m32` scripts'
rules and their source/target sections.
- After this patch `CPPFLAGS` are all applied to the `src` sources once
again. This matches the behaviour of cmake/autotools. Only zlib ones
are actually required there.
- Use `.rc` names from `Makefile.inc` instead of keeping a duplicate.
- Change examples to link `libcurl.dll` by default. This makes building
trivial, even as a cross-build:
`CC=x86_64-w64-mingw32-gcc make -f Makefile.m32`
To run them, you need to move/copy or add-to-path `libcurl.dll`.
You can select static mode via `CFG=-static`.
- List more of the `Makefile.m32` config variables.
- Drop `.rc` support from examples. It made it fragile without much
benefit.
- Include a necessary system lib for the `externalsocket.c` example.
- Exclude unnecessary systems libs when building in `-dyn` mode.
Closes#9642
- Use brackets for the IPv6 address shown in verbose message when the
format is address:port so that it is less confusing.
Before: Trying 2606:4700:4700::1111:443...
After: Trying [2606:4700:4700::1111]:443...
Bug: https://curl.se/mail/archive-2022-02/0041.html
Reported-by: David Hu
Closes#9635
This patch overhauls `Makefile.m32` scripts, fixing a list of quirks,
making its behaviour and customization envvars align better with other
build systems, aiming for less code, that is easier to read, use and
maintain.
Details:
- Rename customization envvars:
`CURL_CC` -> `CC`
`CURL_RC` -> `RC`
`CURL_AR` -> `AR`
`CURL_LDFLAG_EXTRAS_DLL` -> `CURL_LDFLAGS_LIB`
`CURL_LDFLAG_EXTRAS_EXE` -> `CURL_LDFLAGS_BIN`
- Drop `CURL_STRIP` and `CURL_RANLIB`. These tools are no longer used.
- Accept `CFLAGS`, `CPPFLAGS`, `RCFLAGS`, `LDFLAGS` and `LIBS` envvars.
- Drop `CURL_CFLAG_EXTRAS`, `CURL_LDFLAG_EXTRAS`, `CURL_RCFLAG_EXTRAS` in
favor of the above.
- Do not automatically enable `zlib` with `libssh2`. `zlib` is optional
with `libssh2`.
- Omit unnecessary `CPPFLAGS` options when building `curl.exe` and
examples.
- Drop support for deprecated `-winssl` `CFG` option. Use `-schannel`
instead.
- Avoid late evaluation where not necessary (`=` -> `:=`).
- Drop support for `CURL_DLL_A_SUFFIX` to override the implib suffix.
Instead, use the standard naming scheme by default: `libcurl.dll.a`.
The toolchain recognizes the name, and selects it automatically when
asking for a `-shared` vs. `-static` build.
- Stop applying `strip` to `libcurl.a`. Follow-up from
16a58e9f93. There was no debug info to
strip since then.
- Stop setting `-O3`, `-W`, `-Wall` options. You can add these to
`CFLAGS` as desired.
- Always enable `-DCURL_DISABLE_OPENSSL_AUTO_LOAD_CONFIG` with OpenSSL,
to avoid that vulnerability on Windows.
- Add `-lbrotlicommon` to `LIBS` when using `brotli`.
- Do not enable `-nghttp3` without `-ngtcp2`.
- `-ssh2` and `-rtmp` options no longer try to auto-select a TLS-backend.
You need to set the backend explicitly. This scales better and avoids
issues with certain combinations (e.g. `libssh2` + `wolfssl` with no
`schannel`).
- Default to OpenSSL TLS-backend with `ngtcp2`. Possible to override via
`NGTCP2_LIBS`.
- Old, alternate method of enabling components (e.g. `SSH2=1`) no longer
supported.
- Delete `SPNEGO` references. They were no-ops.
- Drop support for Win9x environments.
- Allow setting `OPENSSL_LIBS` independently from `OPENSSL_LIBPATH`.
- Support autotools/CMake `libssh2` builds by default.
- Respect `CURL_DLL_SUFFIX` in `-dyn` mode when building `curl.exe` and
examples.
- Assume standard directory layout with `LIBCARES_PATH`. (Instead of the
long gone embedded one.)
- Stop static linking with c-ares by default. Add
`CPPFLAGS=-DCARES_STATICLIB` to enable it.
- Reorganize internal layout to avoid redundancy and emit clean diffs
between src/lib and example make files.
- Delete unused variables.
- Code cleanups/rework.
- Comment and indentation fixes.
Closes#9632
- Drop auto-detection of OpenSSL 1.0.2 and earlier. Now always defaulting
to OpenSSL 1.1.0 and later, LibreSSL and BoringSSL.
- Drop `Invalid path to OpenSSL package` detection. OpenSSL has been
using a standard file layout since 1.1.0, so this seems unnecessary
now.
- Drop special logic to enable Novell LDAP SDK support.
- Drop special logic to enable OpenLDAP LDAP SDK support. This seems
to be distinct from native OpenLDAP, with support implemented inside
`lib/ldap.c` (vs. `lib/openldap.c`) back when the latter did not exist
yet in curl.
- Add `-lwldap32` only if there is no other LDAP library (either native
OpenLDAP, or SDKs above) present.
- Update `doc/INSTALL.md` accordingly.
After this patch, it's necessary to make configration changes when using
OpenSSL 1.0.2 or earlier, or the two LDAP SDKs.
OpenSSL 1.0.2 and earlier:
```
export OPENSSL_INCLUDE = <path-to-openssl>/outinc
export OPENSSL_LIBPATH = <path-to-openssl>/out
export OPENSSL_LIBS = -lssl32 -leay32 -lgdi32
```
Novell LDAP SDK, previously enabled via `USE_LDAP_NOVELL=1`:
```
export CURL_CFLAG_EXTRAS = -I<path-to-sdk>/inc -DCURL_HAS_NOVELL_LDAPSDK
export CURL_LDFLAG_EXTRAS = -L<path-to-sdk>/lib/mscvc -lldapsdk -lldapssl -lldapx
```
OpenLDAP LDAP SDK, previously enabled via `USE_LDAP_OPENLDAP=1`:
```
export CURL_CFLAG_EXTRAS = -I<path-to-sdk>/include -DCURL_HAS_OPENLDAP_LDAPSDK
export CURL_LDFLAG_EXTRAS = -L<path-to-sdk>/lib -lldap -llber
```
I haven't tested these scenarios, and in general we recommend using
a recent OpenSSL release. Also, WinLDAP (the Windows default) and
OpenLDAP (via `-DUSE_OPENLDAP`) are the LDAP options actively worked on
in curl.
Closes#9631
SourceForge projects can now choose between two hostnames, with .io and
.net ending. Both support HTTPS by default now. Opening the other variant
will perm-redirected to the one chosen by the project.
The .io -> .net redirection is done insecurely.
Let's update the URLs to point to the current canonical endpoints to
avoid any redirects.
Closes#9630
- Add `-lcrypt32` once, and add it always for simplicity.
- Delete broken link and reference to the pre-Vista WinIDN add-on.
MS no longer distribute it.
- Delete related `WINIDN_PATH` option. IDN is a system lib since Vista.
- Sync `LIBCARES_PATH` default with the rest of dependencies.
- Delete version numbers from dependency path defaults.
- `libgsasl` package is now called `gsasl`.
- Delete `libexpat` and `libxml2` references. No longer used by curl.
- Delete `Edit the path below...` comments. We recommend to predefine
those envvars instead.
- `libcares.a` is not an internal dependency anymore. Stop using it as
such.
- `windres` `--include-dir` -> `-I`, `-F` -> `--target=` for readability.
- Delete `STRIP`, `CURL_STRIP`, `AR` references from `src/Makefile.m32`.
They were never used.
- Stop to `clean` some objects twice in `src/Makefile.m32`.
- Delete cvs-specific leftovers.
- Finish resource support in examples make file.
- Delete `-I<root>/lib` from examples make file.
- Fix copyright start year in examples make file.
- Delete duplicate `ftpuploadresume` input in examples make file.
- Sync OpenSSL lib order, `SYNC` support, `PROOT` use, dependency path
defaults, variables names and other internal bits between the three
make files.
- `lib/Makefile.m32` accepted custom options via `DLL_LIBS` envvar. This
was lib-specific and possibly accidental. Use `CURL_LDFLAG_EXTRAS_DLL`
envvar for the same effect.
- Fix linking `curl.exe` and examples to wrong static libs with
auto-detected OpenSSL 1.0.2 or earlier.
- Add `-lgdi32` for OpenSSL 1.0.2 and earlier only.
- Add link to Novell LDAP SDK and use a relative default path. Latest
version is from 2016, linked to an outdated OpenSSL 1.0.1.
- Whitespace and comment cleanups.
TODO in a next commit:
Delete built-in detection/logic for OpenSSL 1.0.2 and earlier, the Novell
LDAP SDK and the other LDAP SDK (which is _not_ OpenLDAP). Write up the
necessary custom envvars to configure them.
Closes#9616
