linux-kernel - RE: vfat: Broken case-insensitive support for UTF-8

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <b42888a01c8847e48116873ebbbbb261@AcuMS.aculab.com>
Date:   Mon, 20 Jan 2020 16:43:21 +0000
From:   David Laight <David.Laight@...LAB.COM>
To:     'Pali Rohár' <pali.rohar@...il.com>
CC:     OGAWA Hirofumi <hirofumi@...l.parknet.co.jp>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-fsdevel@...r.kernel.org" <linux-fsdevel@...r.kernel.org>,
        "Theodore Y. Ts'o" <tytso@....edu>,
        Namjae Jeon <linkinjeon@...il.com>,
        "Gabriel Krisman Bertazi" <krisman@...labora.com>
Subject: RE: vfat: Broken case-insensitive support for UTF-8

From: Pali Rohár
> Sent: 20 January 2020 16:27
...
> > Unfortunately there is neither a 1:1 mapping of all possible byte sequences
> > to wchar_t (or unicode code points),
> 
> I was talking about valid UTF-8 sequence (invalid, illformed is out of
> game and for sure would always cause problems).

Except that they are always likely to happen.
I've been pissed off by programs crashing because they assume that
a input string (eg an email) is UTF-8 but happens to contain a single
0xa3 byte in the otherwise 7-bit data.

The standard ought to have defined a translation for such sequences
and just a 'warning' from the function(s) that unexpected bytes were
processed.

> > nor a 1:1 mapping of all possible wchar_t values to UTF-8.
> 
> This is not truth. There is exactly only one way how to convert sequence
> of Unicode code points to UTF-8. UTF is Unicode Transformation Format
> and has exact definition how is Unicode Transformed.

But a wchar_t can hold lots of values that aren't Unicode code points.
Prior to the 2003 changes half of the 2^32 values could be converted.
Afterwards only a small fraction.

> If you have valid UTF-8 sequence then it describe one exact sequence of
> Unicode code points. And if you have sequence (ordinals) of Unicode code
> points there is exactly one and only one its representation in UTF-8.
> 
> I would suggest you to read Unicode standard, section 2.5 Encoding Forms.

That all assumes everyone is playing the correct game

> > Really both need to be defined - even for otherwise 'invalid' sequences.
> >
> > Even the 16-bit values above 0xd000 can appear on their own in
> > windows filesystems (according to wikipedia).
> 
> If you are talking about UTF-16 (which is _not_ 16-bit as you wrote),
> look at my previous email:

UFT-16 is a sequence of 16-bit values....
It can contain 0xd000 to 0xffff (usually in pairs) but they aren't UTF-8 codepoints.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)