glibc/localedata
Jules Bertholet 25c9c3789e localedata: Fix several issues with the set of characters considered 0-width [BZ #31370]
= `Default_Ignorable_Code_Point`s should have width 0 =

Unicode specifies (https://www.unicode.org/faq/unsup_char.html#3) that characters
with the `Default_Ignorable_Code_Point` property

> should be rendered as completely invisible (and non advancing, i.e. “zero width”),
if not explicitly supported in rendering.

Hence, `wcwidth()` should give them all a width of 0, with two exceptions:

- the soft hyphen (U+00AD SOFT HYPHEN) is assigned width 1 by longstanding precedent
- U+115F HANGUL CHOSEONG FILLER needs a carveout
  due to the unique behavior of the conjoining Korean jamo characters.
  One composed Hangul "syllable block" like 퓛
  is made up of two to three individual component characters, or "jamo".
  These are all assigned an `East_Asian_Width` of `Wide`
  by Unicode, which would normally mean they would all be assigned
  width 2 by glibc; a combination of (leading choseong jamo) +
  (medial jungseong jamo) + (trailing jongseong jamo) would then have width 2 + 2 + 2 = 6.
  However, glibc (and other wcwidth implementations) special-cases jungseong and jongseong,
  assigning them all width 0,
  to ensure that the complete block has width 2 + 0 + 0 = 2 as it should.
  U+115F is meant for use in syllable blocks
  that are intentionally missing a leading jamo;
  it must be assigned a width of 2 even though it has no visible display
  to ensure that the complete block has width 2.

However, `wcwidth()` currently (before this patch)
incorrectly assigns non-zero width to
U+3164 HANGUL FILLER and U+FFA0 HALFWIDTH HANGUL FILLER;
this commit fixes that.

Unicode spec references:
- Hangul:  §3.12 https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G24646 and
  §18.6 https://www.unicode.org/versions/Unicode15.0.0/ch18.pdf#G31028
- `Default_Ignorable_Code_Point`: §5.21 https://www.unicode.org/versions/Unicode15.0.0/ch05.pdf#G40095.

= Non-`Default_Ignorable_Code_Point` format controls should be visible =

The Unicode Standard, §5.21 - Characters Ignored for Display
(https://www.unicode.org/versions/Unicode15.0.0/ch05.pdf#G40095)
says the following:

> A small number of format characters (General_Category = Cf )
> are also not given the Default_Ignorable_Code_Point property.
> This may surprise implementers, who often assume
> that all format characters are generally ignored in fallback display.
> The exact list of these exceptional format characters
> can be found in the Unicode Character Database.
> There are, however, three important sets of such format characters to note:
>
> - prepended concatenation marks
> - interlinear annotation characters
> - Egyptian hieroglyph format controls
>
> The prepended concatenation marks always have a visible display.
> See “Prepended Concatenation Marks” in [*Section 23.2, Layout Controls*](https://www.unicode.org/versions/Unicode15.1.0/ch23.pdf#M9.35858.HeadingBreak.132.Layout.Controls)
> for more discussion of the use and display of these signs.
>
> The other two notable sets of format characters that exceptionally are not ignored
> in fallback display consist of the interlinear annotation characters,
> U+FFF9 INTERLINEAR ANNOTATION ANCHOR through
> U+FFFB INTERLINEAR ANNOTATION TERMINATOR,
> and the Egyptian hieroglyph format controls,
> U+13430 EGYPTIAN HIEROGLYPH VERTICAL JOINER through
> U+1343F EGYPTIAN HIEROGLYPH END WALLED ENCLOSURE.
> These characters should have a visible glyph display for fallback rendering,
> because if they are not displayed,
> it is too easy to misread the resulting displayed text.
> See “Annotation Characters” in [*Section 23.8, Specials*](https://www.unicode.org/versions/Unicode15.1.0/ch23.pdf#M9.21335.Heading.133.Specials),
> as well as [*Section 11.4, Egyptian Hieroglyphs*](https://www.unicode.org/versions/Unicode15.1.0/ch11.pdf#M9.73291.Heading.1418.Egyptian.Hieroglyphs)
> for more discussion of the use and display of these characters.

glibc currently correctly assigns non-zero width to the prepended concatenation marks,
but it incorrectly gives zero width to the interlinear annotation characters
(which a generic terminal cannot interpret)
and the Egyptian hieroglyph format controls
(which are not widely supported in rendering implementations at present).
This commit fixes both these issues as well.

= Derive Hangul syllable type from Unicode data =

Previosuly, the jungseong and jongseong jamo ranges
were hard-coded into the script. With this commit, they are instead parsed
from the HangulSyllableType.txt data file published by Unicode.
This does not affect the end result.

Signed-off-by: Jules Bertholet <julesbertholet@quoi.xyz>
2024-05-15 14:31:06 +02:00
..
charmaps localedata: Fix several issues with the set of characters considered 0-width [BZ #31370] 2024-05-15 14:31:06 +02:00
locales localedata: fix weekdays in mdf_RU locale 2024-05-08 14:27:40 +02:00
tests Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tests-mbwc localedate: Fix printf type on tst_mbrtowc 2022-03-31 08:49:55 -03:00
tst-fmon-locales
tst-localedef-hardlinks.root
unicode-gen localedata: Fix several issues with the set of characters considered 0-width [BZ #31370] 2024-05-15 14:31:06 +02:00
am_ET.UTF-8.in
az_AZ.UTF-8.in
be_BY.UTF-8.in
ber_DZ.UTF-8.in
ber_MA.UTF-8.in
bg_BG.UTF-8.in
br_FR.UTF-8.in
bs_BA.UTF-8.in
bug-iconv-trans.c Use '%z' instead of '%Z' on printf functions 2022-09-22 08:48:04 -03:00
bug-setlocale1-static.c
bug-setlocale1.c
bug-usesetlocale.c
C.UTF-8.in Add generic C.UTF-8 locale (Bug 17318) 2021-09-06 11:30:28 -04:00
ckb_IQ.UTF-8.in
cmn_TW.UTF-8.in
collate-test.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
crh_UA.UTF-8.in
cs_CZ.in
cs_CZ.UTF-8.in
csb_PL.UTF-8.in
cv_RU.UTF-8.in
cy_GB.UTF-8.in
da_DK.ISO-8859-1.in
de_DE.ISO-8859-1.in
Depend
dsb_DE.UTF-8.in
dump-ctype.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
dz_BT.UTF-8.in
en_US.ISO-8859-1.in
en_US.UTF-8.in
eo.UTF-8.in
es_ES.UTF-8.in
et_EE.UTF-8.in
fa_IR.UTF-8.in
fi_FI.UTF-8.in
fil_PH.UTF-8.in
fr_CA.UTF-8.in
fr_FR.UTF-8.in
fur_IT.UTF-8.in
gen-locale.sh locale: Handle loading a missing locale twice (Bug 14247) 2024-04-22 16:03:00 -04:00
gez_ER.UTF-8@abegede.in
ha_NG.UTF-8.in
hr_HR.ISO-8859-2.in
hr_HR.UTF-8.in
hsb_DE.UTF-8.in
hu_HU.UTF-8.in
ig_NG.UTF-8.in
ik_CA.UTF-8.in
is_IS.UTF-8.in
kk_KZ.UTF-8.in
ku_TR.UTF-8.in
ky_KG.UTF-8.in
ln_CD.UTF-8.in
lt_LT.UTF-8.in
lv_LV.UTF-8.in Change lv_LV collation to agree with the recent change in CLDR 2024-02-08 08:13:37 +01:00
Makefile localedata: add mdf_RU locale 2024-05-08 14:27:40 +02:00
mdf_RU.UTF-8.in localedata: add mdf_RU locale 2024-05-08 14:27:40 +02:00
mi_NZ.UTF-8.in
ml_IN.UTF-8.in
mn_MN.UTF-8.in
mr_IN.UTF-8.in
mt_MT.UTF-8.in
nan_TW.UTF-8@latin.in
nb_NO.UTF-8.in
om_KE.UTF-8.in
or_IN.UTF-8.in
os_RU.UTF-8.in
pl_PL.UTF-8.in
ps_AF.UTF-8.in
README
rif_MA.UTF-8.in Add rif_MA locale [BZ #27781] 2022-04-07 14:59:41 +02:00
ro_RO.UTF-8.in
ru_RU.UTF-8.in
sah_RU.UTF-8.in
sc_IT.UTF-8.in
se_NO.UTF-8.in
show-ucs-data.c
si_LK.UTF-8.in
sort-test.sh Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
sq_AL.UTF-8.in
sr_RS.UTF-8.in
SUPPORTED localedata: add mdf_RU locale 2024-05-08 14:27:40 +02:00
sv_SE.ISO-8859-1.in
sv_SE.UTF-8.in
syr.UTF-8.in Add locale for syr_SY 2022-04-21 13:05:40 +02:00
szl_PL.UTF-8.in
tg_TJ.UTF-8.in
th_TH.UTF-8.in Adapt collation in th_TH locale to use the iso14651_t1_common file and sync the collation with CLDR 2023-09-21 10:34:35 +02:00
tk_TM.UTF-8.in
tr_TR.UTF-8.in
tst-c-utf8-consistency.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-ctype-de_DE.ISO-8859-1.in
tst-ctype.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-ctype.sh Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-digits.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-fmon.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-fmon.data Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-fmon.sh Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-iconv-emojis-trans.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-iconv-math-trans.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-langinfo-newlocale-static.c
tst-langinfo-newlocale.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-langinfo-setlocale-static.c
tst-langinfo-setlocale.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-langinfo-static.c
tst-langinfo.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-langinfo.sh Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-leaks.c
tst-locale-loadlocale.c locale: Handle loading a missing locale twice (Bug 14247) 2024-04-22 16:03:00 -04:00
tst-locale.sh Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-localedef-hardlinks.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-mbswcs1.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-mbswcs2.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-mbswcs3.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-mbswcs4.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-mbswcs5.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-mbswcs6.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-numeric.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-numeric.data Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-numeric.sh Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-rpmatch.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-rpmatch.sh Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-setlocale2.c
tst-setlocale3.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-setlocale.c
tst-sscanf.c
tst-strfmon1.c
tst-trans.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-trans.sh Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-wctype.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
tst-wctype.input
tst-xlocale1.c
tst-xlocale2.c
tt_RU.UTF-8.in
tt_RU.UTF-8@iqtelif.in
ug_CN.UTF-8.in
uk_UA.UTF-8.in
uz_UZ.UTF-8.in
vi_VN.UTF-8.in
xfrm-test.c Update copyright dates with scripts/update-copyrights 2024-01-01 10:53:40 -08:00
yi_US.UTF-8.in
yo_NG.UTF-8.in
zh_CN.UTF-8.in

		       POSIX locale descriptions
				  and
		    POSIX character set descriptions

Ulrich Drepper			Time-stamp: <2004/11/27 13:06:54 drepper>
drepper@redhat.com


This directory contains the data needed to build the locale data files
to use the internationalization features of the GNU libc.

POSIX.2 describes the `localedef' utility which is part of the GNU libc.
You need this program to "compile" the locale description in a form
suitable for fast access by the GNU libc functions.  Any compilation is
based on a given character set.

Once you run `make install' for the GNU libc the data files are
automatically installed in the right place, ready for use by the
`localedef' program.

To compile the locale data files you simply have to decide which locale
(based on the location and the language) and which character set you
use.  E.g., French speaking Canadians would use the locale `fr_CA' and
the character set `ISO_8859-1,1987'.  Calling `localedef' to get the
desired data should happen like this:

	localedef -i fr_CA -f ISO-8859-1 fr_CA

This will place the 6 output files in the appropriate directory where
the GNU libc functions can find them.  Please note that you need
permission to write to this directory ($(prefix)/share/locale, where
$(prefix) is the value you specified while configuring GNU libc).  If
you do not have the necessary permissions, you can write the files into an
arbitrary directory by giving a path including a '/' character instead
of `fr_CA'.  E.g., to put the new files in a subdirectory of the
current directory simply use

	localedef -i fr_CA -f ISO-8859-1 ./fr_CA

How to use these data files is described in the GNU libc manual,
especially in the section describing the `setlocale' function.

All problems should be reported using

  https://sourceware.org/bugzilla/


One more note: the `POSIX' locale definition is not meant to be used
as an input file for `localedef'.  It is rather there to show the
values with are built in the libc binaries as default values when no
legal locale is found or the "C" or "POSIX" locale is selected.


		       The collation test suite
		       ########################

This package also contains a (beginning of a) test suite for the
collation functions in the GNU libc.  The files are provided sorted.
The test program shuffles the lines and sort them afterwards.

Some of the files are provided in 8bit form, i.e., not only ASCII
characters.  So the tools you use to process the files should be 8bit
clean.

To run the test program the appropriate locale information must be
installed.  Therefore the localedef program is used to generate this
data used the locale and charmap description files contained here.
Since we cannot run the localedef program in case of cross-compilation
no tests at all are performed.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Local Variables:
 mode:text
 eval:(load-library "time-stamp")
 eval:(make-local-variable 'write-file-hooks)
 eval:(add-hook 'write-file-hooks 'time-stamp)
 eval:(setq time-stamp-format '(time-stamp-yyyy/mm/dd time-stamp-hh:mm:ss user-login-name))
End: