Commit 91927b7c76 (Rewrite iconv option parsing [BZ #19519]) did not
handle cases where the output codeset for translations (via the `gettext'
family of functions) might have a caller specified encoding suffix such as
TRANSLIT or IGNORE. This led to a regression where translations did not
work when the codeset had a suffix.
This commit fixes the above issue by parsing any suffixes passed to
__dcigettext and adds two new test-cases to intl/tst-codeset.c to
verify correct behaviour. The iconv-internal function __gconv_create_spec
and the static iconv-internal function gconv_destroy_spec are now visible
internally within glibc and used in intl/dcigettext.c.
(cherry picked from commit 7d4ec75e11)
This commit replaces string manipulation during `iconv_open' and iconv_prog
option parsing with a structured, flag based conversion specification. In
doing so, it alters the internal `__gconv_open' interface and accordingly
adjusts its uses.
This change fixes several hangs in the iconv program and therefore includes
a new test to exercise iconv_prog options that originally led to these hangs.
It also includes a new regression test for option handling in the iconv
function.
Reviewed-by: Florian Weimer <fweimer@redhat.com>
Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
Reviewed-by: Carlos O'Donell <carlos@redhat.com>
calls with constant strings shows a small (~10%) performance gain, strdup is
typically used in error reporting code, so not performance critical.
Remove the now unused __need_malloc_and_calloc related defines from stdlib.h.
Rename existing uses of str(n)dup to __str(n)dup so it no longer needs to be
redirected to a builtin. Also building GLIBC with -Os now no longer shows
localplt or linkname space failures (partial fix for BZ #15105 and BZ #19463).
[BZ #15105]
[BZ #19463]
* elf/dl-cache.c (_dl_load_cache_lookup): Use __strdup.
* inet/rcmd.c (rcmd_af): Likewise.
* inet/rexec.c (rexec_af): Likewise.
* intl/dcigettext.c (_LIBC): Likewise.
* intl/finddomain.c (_nl_find_domain): Use strdup expansion.
* locale/loadarchive.c (_nl_load_locale_from_archive): Use __strdup.
* locale/setlocale.c (setlocale): Likewise.
* posix/spawn_faction_addopen.c
(posix_spawn_file_actions_addopen): Likewise.
* stdlib/putenv.c (putenv): Use __strndup.
* sunrpc/svc_simple.c (__registerrpc): Use __strdup.
* sysdeps/posix/getaddrinfo.c (gaih_inet): Use __strdup/__strndup.
* include/stdlib.h (__need_malloc_and_calloc): Remove uses.
(__Need_M_And_C) Remove define/undef.
* stdlib/stdlib.h (__need_malloc_and_calloc): Remove uses.
(__malloc_and_calloc_defined): Remove define.
* string/bits/string2.h (__strdup): Remove define.
(strdup): Likewise.
(__strndup): Likewise.
(strndup): Likewise.
This patch merges the latest release of gettext into the intl
subdirectory. The initial motivation was to include the plural.y
changes which enable building with bison 3.0, but the majority
of the other changes are merely cosmetic so it seemed like merging
the whole directory was simpler than trying to take it piecemeal.
The merge was done by copying across the latext gettext code and
adding in a few small glibc changes that have been added over the
years that seemed beneficial, as well as a couple of small build
fixes that should be merged back to gettext. I also reverted the
gettext commit:
commit 279b57fc367251666f00e8e2b599b83703451afb
Author: Bruno Haible <bruno@clisp.org>
Date: Fri Jun 14 12:03:49 2002 +0000
Make absolute pathnames inside $LANGUAGE work.
As it caused localedata/tst-setlocale3 to fail and it wasn't clear
that glibc wanted that behaviour.
The merge has dropped many uses of __glibc_likely/unlikely. This is
intentional given that it eases merging. It seems to me that the cost
of continually rewriting these lines when merging and the risk of adding
bugs when doing so outweighs the benefits of using these macros when
code is shared with another project.
Tested with make check on x86_64.
ChangeLog:
2014-12-11 Will Newton <will.newton@linaro.org>
Merge gettext 0.19.3 into intl/.
This involves a number of cosmetic changes to comments
and ANSI function definitions and prototypes throughout
all the files. The gettext copyright header is used but
with the date ranges taken from the glibc copy.
* NEWS: Add gettext merge to 2.21.
* intl/bindtextdom.c: Switch to gettext copyright.
Use ANSI definitions and prototypes.
Use gl_* locking primitives rather than __libc_* ones.
Use __builtin_expect rather than __glibc_likely/unlikely.
* intl/dcgettext.c: Switch to gettext copyright.
Use ANSI definitions and prototypes.
* intl/dcigettext.c: Switch to gettext copyright.
Use ANSI definitions and prototypes.
(INTDIV0_RAISES_SIGFPE): New define.
Use gl_* locking primitives rather than __libc_* ones.
Include eval-plural.h instead of plural-eval.c.
Use __builtin_expect rather than __glibc_likely/unlikely.
* intl/dcngettext.c: Switch to gettext copyright.
Use ANSI definitions and prototypes.
* intl/dgettext.c: Likewise.
* intl/dngettext.c: Likewise.
* intl/plural-eval.c: Renamed to...
* intl/eval-plural.h: ...this.
* intl/explodename.c: Switch to gettext copyright.
Use ANSI definitions and prototypes.
(_nl_explode_name): Use strchr instead of __rawmemchr.
* intl/finddomain.c: Switch to gettext copyright.
Use ANSI definitions and prototypes.
Use gl_* locking primitives rather than __libc_* ones.
(_nl_find_domain): Use malloc rather than alloca for
allocation of temporary locale name.
* intl/gettext.c: Switch to gettext copyright.
Use ANSI definitions and prototypes.
* intl/gettextP.h: Switch to gettext copyright.
Use ANSI definitions and prototypes.
Use gl_* locking primitives rather than __libc_* ones.
* intl/gmo.h: Switch to gettext copyright.
(struct sysdep_string): Move struct segment_pair outside of
struct definition.
* intl/hash-string.c: Use ANSI definitions and prototypes.
* intl/hash-string.h: Switch to gettext copyright.
Use ANSI definitions and prototypes.
* intl/l10nflist.c: Switch to gettext copyright.
Use ANSI definitions and prototypes.
(_nl_normalize_codeset): Avoid integer overflow.
* intl/loadinfo.h: Switch to gettext copyright.
Use ANSI definitions and prototypes.
(LIBINTL_DLL_EXPORTED): New define.
(PATH_SEPARATOR): New define.
* intl/loadmsgcat.c: Switch to gettext copyright.
* intl/localealias.c: Switch to gettext copyright.
Use ANSI definitions and prototypes.
(_nl_expand_alias): Use PATH_SEPARATOR.
* intl/ngettext.c: Switch to gettext copyright.
Use ANSI definitions and prototypes.
* intl/plural-exp.c: Likewise.
* intl/plural-exp.h: Switch to gettext copyright.
Use ANSI definitions and prototypes.
(struct expression): Move definition of enum operator outside
of struct definition.
* intl/plural.c: Regenerate.
* intl/plural.y: Switch to gettext copyright.
Use ANSI definitions and prototypes.
Port to bison 3.0.
* intl/textdomain.c: Switch to gettext copyright.
Use ANSI definitions and prototypes.
Use gl_* locking primitives rather than __libc_* ones.
This patch fixes two issues, and perhaps should be two distinct commits,
but I present it here as one for the sake of completeness.
Commit 006dd86111 fails to check malloc's
return in intl/dcigettext.c (_nl_find_msg):
~~~
freemem_size = INITIAL_BLOCK_SIZE;
newmem = (transmem_block_t *) malloc (freemem_size);
...
newmem->next = transmem_list;
transmem_list = newmem;
~~~
If malloc fails then newmem is NULL then newmem->next results in a
fault.
The fix is easy enough, check for newmem != NULL, and fall through to
the error condition below which returns (char *) -1 e.g. resource error.
The problem is that returning (char *) -1 will break all sorts of other
code, so while what we did is correct, the real failure case fix is
slightly broader.
There are 4 other places where _nl_find_msg is called, one is OK, the
other three are fixed to handle -1 error return value.
No regressions on x86-64 or x86.
However, no regressions isn't really a useful metric for this code.
The change was tested as documented here:
http://sourceware.org/glibc/wiki/Testing/WhiteBox
using SystemTap for fault injection to simulate malloc failure.
---
2013-05-03 Carlos O'Donell <carlos at redhat.com>
[BZ #15441]
* intl/dcigettext.c (DCIGETTEXT): Skip translating if _nl_find_msg
returns -1.
(_nl_find_msg): Return -1 if recursive call returned -1. If newmem is
null return -1.
* intl/loadmsgcat.c (_nl_load_domain): If _nl_find_msg returns -1 abort
loading the domain.
2008-03-30 Ulrich Drepper <drepper@redhat.com>
[BZ #5443]
* intl/dcigettext.c (__dcigettext): Get reader lock for locale data
before looking for translation.
* locale/duplocale.c: Transform __libc_setlocale_lock into rwlock.
* locale/freelocale.c: Likewise.
* locale/newlocale.c: Likewise.
* locale/setlocale.c: Likewise.
Based partially on a patch by ryo@np.css.fujitsu.com.
acquiring wrlock. Do conv_tab allocation while holding lock.
* intl/Makefile: Add rules to build and run tst-gettext6.
* intl/tst-gettext6.c: New test.
* intl/tst-gettext6.sh: New file.
failed.
* intl/finddomain.c (_nl_find_domain): Free normalized_codeset
on failure.
* elf/dl-load.c (decompose_rpath): Free copy if result couldn't be
allocated.
2007-08-03 Jakub Jelinek <jakub@redhat.com>
* iconv/gconv_cache.c (__gconv_lookup_cache): Return __GCONV_NULCONV
if from and to charsets are the same.
* iconv/gconv_db.c (__gconv_find_transform): Likewise.
* intl/dcigettext.c (_nl_find_msg): Return NULL even if __gconv_open
returns __GCONV_NOCONV, but not for __GCONV_NULCONV.
2007-07-17 Jakub Jelinek <jakub@redhat.com>
* wcsmbs/wchar.h: Only define wint_t if __need_wint_t.
Don't define wint_t when __need_mbstate_t unless it
is necessary.
(__mbstate_t): Use __WINT_TYPE__ rather than wint_t
in the typedef if possible.
* wctype/wctype.h (wint_t): Define by including
wchar.h with __need_wint_t instead of including stddef.h
with __need_wint_t and as fallback definining it ourselves.
* iconv/gconv.h (__need_wint_t): Define before including
wchar.h.
* sysdeps/gnu/_G_config.h: Don't include gconv.h if not _LIBC
or _GLIBCPP_USE_WCHAR_T.
(__need_wchar_t): Don't define
if not _LIBC or _GLIBCPP_USE_WCHAR_T.
(__need_wint_t): Don't define before including stddef.h,
define before including wchar.h only if _LIBC or
_GLIBCPP_USE_WCHAR_T.
(_G_iconv_t): Don't define if not _LIBC or _GLIBCPP_USE_WCHAR_T.
* sysdeps/mach/hurd/_G_config.h: Likewise.
* sysdeps/generic/_G_config.h: Likewise.
* libio/libio.h (__wunderflow, __wuflow, __woverflow): Only
prototype if _LIBC or _GLIBCPP_USE_WCHAR_T.
(_IO_getwc_unlocked, _IO_putwc_unlocked): Only define
if _LIBC or _GLIBCPP_USE_WCHAR_T.
look further, return original strings.
(_nl_find_msg): Do not return found translation if the conversion
failed. Either signal the string is unusable or that something went
wrong and the original should be used.
2006-06-21 Ulrich Drepper <drepper@redhat.com>
* string/_strerror.c (__strerror_r): Add __builtin_expect.
2005-03-27 Bruno Haible <bruno@clisp.org>
Make it possible for multiple threads to use gettext() in different
locales.
* intl/dcigettext.c (HAVE_PER_THREAD_LOCALE): New macro.
(struct known_translation_t): If HAVE_PER_THREAD_LOCALE, add localename
field.
(transcmp): If HAVE_PER_THREAD_LOCALE, compare localename fields.
(DCIGETTEXT): If HAVE_PER_THREAD_LOCALE, fill the localename field in
search and newp.
* intl/tst-gettext4.c: New file.
* intl/tst-gettext4.sh: New file.
* intl/tst-gettext4-de.po: New file.
* intl/tst-gettext4-fr.po: New file.
* intl/tst-gettext5.c: New file.
* intl/tst-gettext5.sh: New file.
* intl/Makefile (distribute): Add tst-gettext4.sh, tst-gettext4-de.po,
tst-gettext4-fr.po, tst-gettext5.sh.
(multithread-test-srcs): New variable.
(test-srcs): Add its contents.
(tests): Depend on tst-gettext4.out, tst-gettext5.out.
(tst-gettext4.out, tst-gettext5.out): New rules.
(CFLAGS-tst-gettext4.c, CFLAGS-tst-gettext5.c): New variables.
Add rule for linking the multithread-test-srcs with the appropriate
thread-library.
2005-04-28 Ulrich Drepper <drepper@redhat.com>
* po/rw.po: New file. From translation team.
* intl/tst-gettext3.sh: New file.
* intl/Makefile (distribute): Add tst-gettext3.sh.
(test-srcs): Add tst-gettext3.
(tests): Depend on tst-gettext3.out.
(tst-gettext3.out): New rule.
(CFLAGS-tst-gettext3.c): New variable.
Fix bug exposed by tst-gettext3.
* intl/gettextP.h (struct converted_domain): New type.
(struct loaded_domain): Remove the conv, conv_tab fields. Add
conversions, nconversions fields.
(_nl_init_domain_conv): Remove declaration.
(_nl_free_domain_conv): Remove declaration.
(_nl_find_msg): Add convert argument.
* intl/dcigettext.c (DCIGETTEXT): Call _nl_find_msg with convert=1.
(_nl_find_msg): Add convert argument. When a conversion to a different
charset is needed, create a new converted_domain element, instead of
throwing away the old converted translations.
(get_output_charset): New function.
* intl/loadmsgcat.c (_nl_init_domain_conv): Remove function.
(_nl_free_domain_conv): Remove function.
(_nl_load_domain): Initialize the conversions array to empty. Use
_nl_find_msg instead of _nl_init_domain_conv to retrieve the header
entry.
(_nl_unload_domain): Free the conversions array and its contents.
* intl/gettextP.h (struct loaded_domain): Remove codeset_cntr field.
(struct binding): Likewise.
* intl/bindtextdom.c (set_binding_values): Drop codeset_cntr
modifications.
2005-03-19 Bruno Haible <bruno@clisp.org>
* intl/dcigettext.c (struct known_translation_t): Change type of
domainname field to 'const char *'.
(DCIGETTEXT): Remove const-cast.
[BZ #661]
* grp/initgroups.c (internal_getgrouplist): Check if we have
enough space before adding the primary group to the list.
* intl/dcigettext.c (_nl_find_msg): Add a cast.
* nis/nis_clone_dir.c (nis_clone_directory): Use char * for ADDR.
* nis/nis_clone_obj.c (nis_clone_object): Likewise.
* nis/nis_clone_res.c (nis_clone_result): Likewise.
* resolv/nss_dns/dns-network.c (getanswer_r): Use const unsigned char *
for END_OF_MESSAGE and CP.
* resolv/res_send.c (send_dg): Add else branch for case impossible
unless `poll' is buggy.
* crypt/crypt_util.c (__setkey_r): Add a cast.
* locale/programs/linereader.c (get_toplvl_escape): Use size_t for
NBYTES, and unsigned char * for BYTES.
* locale/programs/charmap.c (charmap_new_char): Use size_t and
unsighed char * for NBYTES, BYTES parameters.
* sysdeps/generic/dl-hash.h (_dl_elf_hash): Take const char * argument
and cast it.
* sysdeps/i386/i686/dl-hash.h (_dl_elf_hash): Likewise.
* sunrpc/create_xid.c (_create_xid): Don't use unsigned long for RES.
* sunrpc/svcauth_des.c (_svcauth_des): Fix cast type.
* sunrpc/auth_des.c (authdes_create): Don't use u_char for PKEY_DATA.
(authdes_marshal): Don't use unsigned int for LEN.
* sunrpc/xdr.c (xdr_hyper): Don't use unsigned long for T2.
(xdr_u_hyper): Likewise.
(xdr_u_short): Don't use u_long for L.
* sunrpc/xdr_intXX_t.c (xdr_int64_t): Don't use uint32_t for T2.
* inet/rexec.c (rexec_af): Use socklen_t.
* sunrpc/key_call.c (getkeyserv_handle): Likewise.
* sunrpc/rtime.c (rtime): Likewise.
* resolv/res_send.c (send_vc, send_dg): Likewise.
* nis/nis_callback.c (__nis_create_callback): Likewise.
* sysdeps/generic/libc-start.c: Use unsigned int for nthreads ptr.
* sysdeps/posix/getaddrinfo.c (gaih_inet): Fix type of ADDR local.
* libio/libio.h (_IO_BE): Add parenthesis around EXPR.
* intl/dcigettext.c (INTVARDEF, INTUSE): Macros removed.
(_nl_default_dirname): Use libc_hidden_data_def instead of INTVARDEF.
(libc_freeres_fn, DCIGETTEXT): Don't use INTUSE.
* intl/bindtextdom.c (INTUSE): Macro removed.
(_nl_default_dirname): Use libc_hidden_proto.
(set_binding_values): Don't use INTUSE.
* include/libintl.h (_libc_intl_domainname_internal): Decl removed.
(_libc_intl_domainname): Use libc_hidden_proto.
* posix/regex_internal.h (gettext): Remove INTUSE on it.
* locale/SYS_libc.c (_libc_intl_domainname): Use libc_hidden_data_def
rather than INTDEF.
* include/libintl.h (_): Don't use *_internal name.
* ctype/ctype-extn.c (__ctype_tolower, __ctype_toupper): Use int32_t,
not uint32_t.
* locale/lc-ctype.c (_nl_postload_ctype): Likewise for assignments.
* iconv/gconv_open.c (__gconv_open): Remove useless cast.
[BZ #721]
* sysdeps/i386/dl-machine.h (ELF_MACHINE_NO_RELA): Define this outside
of [RESOLVE_MAP].
* sysdeps/sh/dl-machine.h (ELF_MACHINE_NO_REL): Likewise.
* sysdeps/powerpc/powerpc32/dl-machine.h
(elf_machine_rel, elf_machine_rel_relative): Removed.
* sysdeps/powerpc/powerpc64/dl-machine.h
(elf_machine_rel, elf_machine_rel_relative): Removed.
2005-02-03 Alexandre Oliva <aoliva@redhat.com>
[BZ #721]
* elf/dynamic-link.h: Don't declare nested auto functions that are
not going to be defined.
2004-07-23 Jakub Jelinek <jakub@redhat.com>
[BZ #284]
* include/features.h (_POSIX_SOURCE, _POSIX_C_SOURCE): Define
if _XOPEN_SOURCE >= 500 even if __STRICT_ANSI__ is defined.
2005-02-16 Roland McGrath <roland@redhat.com>
Update.
* intl/dcigettext.c (DCIGETTEXT): Protect tfind/tsearch calls.
* intl/dcigettext.c (_nl_find_msg): Call _nl_load_domain also if
decided < 0.
* intl/finddomain.c (_nl_find_domain): Likewise.
* intl/l10nflist.c (_nl_make_l10nflist): Initialize lock.
* intl/loadinfo.h (struct loaded_l10nfile): Add lock element.
* intl/loadmsgcat.c (_nl_load_domain): Set decided to 1 only once we
are done. First set to -1 to signal initialization is ongoing.
Protect against concurrent callers with recursive lock.
duplicate address recognition does not copy junk. [BZ #322]
2002-08-04 Ulrich Drepper <drepper@redhat.com>
* stdio-common/psignal.c: Declare _sys_siglist_internal. Use USEINT
to access _sys_siglist.
* string/strsignal.c: Likewise.
* sysdeps/generic/siglist.c: Add _sys_siglist_internal alias.
* sysdeps/gnu/siglist.c: Likewise.
* sysdeps/unix/siglist.c: Likewise.
* sysdeps/unix/sysv/linux/arm/siglist.c: Likewise.
* libio/fileops.c: Add missing INTUSEs for _IO_file_jumps.
* libio/wfileops.c: Add missing INTUSE for _IO_file_close.
* intl/dcigettext.c: Define _nl_default_dirname_internal as hidden
alias and use it.
* intl/bindtextdom.c: Use _nl_default_dirname_internal.
* include/netinet/in.h: Add declaration of in6addr_loopback_internal.
* inet/in6_addr.c: Add INTVARDEF for in6addr_loopback.
* sysdeps/posix/getaddrinfo.c: Use INTUSE for in6addr_loopback access.
* include/time.h: Add libc_hidden_proto for __gmtime_r.
* time/gmtime.c (__gmtime_r): Add libc_hidden_def.
* iconv/Versions: Replace __gconv_alias_db, __gconv_modules_db,
and __gconv_cache with __gconv_get_alias_db, __gconv_get_modules_db,
and __gconv_get_cache respectively.
* iconv/gconv_cache.c (gconv_cache): Renamed for __gconv_cache and
defined static. Change all users.
(__gconv_get_cache): New function.
* iconv/gconv_db.c (__gconv_get_modules_db): New function.
(__gconv_get_alias_db): New function.
* iconv/gconv_int.h (__gconv_alias_db): Declare as hidden.
(__conv_modules_db): Likewise.
Add prototypes for __gconv_get_cache, __gconv_get_modules_db,
and __gconv_get_alias_db.
* iconv/iconv_prog.c: Use the new functions instead of accessing the
variables.
* include/stdlib.h: Add prototype and libc_hidden_proto for
__default_morecore.
* sysdeps/generic/morecore.c: Include <stdlib.h>.
* malloc/obstack.c: Remove fputs macro.
* malloc/mtrace.c: Remove fopen macro.
* locale/localename.c (_nl_current_names): ... here, new file.
Make it global, with attribute_hidden.
* locale/localeinfo.h: Declare it.
* locale/Makefile (aux): Add localename.
* locale/localename.c (__current_locale_name): New function.
* include/locale.h (__current_locale_name): Declare it.
* intl/dcigettext.c (guess_category_value): Use that instead of
calling setlocale.
* locale/locale.h [__USE_GNU] (locale_t): New type alias of __locale_t.
[__USE_GNU] (newlocale, duplocale, freelocale, uselocale): New decls.
[__USE_GNU] (LC_GLOBAL_LOCALE): New macro.
* locale/newlocale.c: Add alias to __ name.
* locale/duplocale.c: Likewise.
* locale/freelocale.c: Likewise.
* locale/uselocale.c: New file.
* locale/Makefile (routines): Add it.
* locale/Versions (libc: GLIBC_2.3): New set.
Add newlocale, duplocale, freelocale, uselocale.
(libc: GLIBC_PRIVATE): Add __uselocale.
* locale/localeinfo.h [SHARED]
(_NL_CURRENT_LOCALE): New macro, defined to fetch a tsd value.
(_NL_CURRENT_DATA): Define using that.
(_NL_CURRENT, _NL_CURRENT_WSTR, _NL_CURRENT_WORD): Define using that.
(_NL_CURRENT_DEFINE): Define to empty.
[! SHARED]: Decls of _nl_current_CATEGORY and _nl_current
conditionalized on this.
* locale/xlocale.c (NL_C_INTIIALIZER): New macro,
taking initializer of _nl_C_locobj.
[SHARED] (_nl_global_locale): New variable, using that initializer.
(_nl_C_locobj): Use new macro for initialzier.
* locale/setlocale.c [! SHARED] (_nl_current): Conditionalize on this.
[! SHARED] (CATEGORY_USED): New macro.
[SHARED] (CATEGORY_USED, _nl_C): New macros.
(setdata, setlocale): Use that macro instead of examining _nl_current.
(setdata): Set the slot in _nl_global_locale.
Conditionalize setting of _nl_current on [! SHARED].
* locale/findlocale.c [SHARED] (_nl_C): Define as a macro instead of
declaration as an extern.
* locale/newlocale.c (__newlocale): Use _nl_C_locobj instead of _nl_C.
* include/locale.h: Use libc_hidden_proto for setlocale.
* locale/setlocale.c: Add libc_hidden_def.
* locale/setlocale.c (free_mem): Use _NL_CURRENT_DATA.
