Message-ID: <20200119221455.bac7dc55g56q2l4r@pali>
Date:   Sun, 19 Jan 2020 23:14:55 +0100
From:   Pali Rohár <pali.rohar@...il.com>
To:     linux-kernel@...r.kernel.org, linux-fsdevel@...r.kernel.org,
        "Theodore Y. Ts'o" <tytso@....edu>,
        OGAWA Hirofumi <hirofumi@...l.parknet.co.jp>,
        Namjae Jeon <linkinjeon@...il.com>,
        Gabriel Krisman Bertazi <krisman@...labora.com>
Subject: vfat: Broken case-insensitive support for UTF-8

Hello!

I have looked more deeply at vfat kernel code how is UTF-8 encoding
handled and I found out that case-insensitivity is broken, or rather not
implemented at all.

In fat_fill_super() function is already FIXME comment about this problem

	/* FIXME: utf8 is using iocharset for upper/lower conversion */
	if (sbi->options.isvfat) {
		sbi->nls_io = load_nls(sbi->options.iocharset);

Basically vfat always loads NLS table which is used for strnicmp and
tolower functions. When no is specified, then default (iso8859-1) is
used. And this applies also when utf8=1 mount option is specified. Also
note that kernel's utf8 NLS table does not implement toupper/tolower
functions (kernel's NLS API does not support tolower/toupper for
non-fixed-8bit encodings, like UTF-8).

So when UTF-8 on VFS for VFAT is enabled, then for VFS <--> VFAT
conversion are used utf16s_to_utf8s() and utf8s_to_utf16s() functions.
But in fat_name_match(), vfat_hashi() and vfat_cmpi() functions is used
NLS table (default iso8859-1) with nls_strnicmp() and nls_tolower().

Which means that fat_name_match(), vfat_hashi() and vfat_cmpi() are
broken for vfat in UTF-8 mode.

I was thinking how to fix it, and the only possible way is to write a
uni_tolower() function which takes one Unicode code point and returns
lowercase of input's Unicode code point. We cannot do any Unicode
normalization as VFAT specification does not say anything about it and
MS reference fastfat.sys implementation does not do it neither.

So, what would be the best option for implementing that function?

  unicode_t uni_tolower(unicode_t u);

Could a new fs/unicode code help with it? Or it is too tied with NFD
normalization and therefore cannot be easily used or extended?

New exfat code which is under review and hopefully would be merged,
contains own unicode upcase table (as defined by exfat specification) so
as exfat is similar to FAT32, maybe reusing it would be a better option?


========================================================================

Proof that vfat in UTF-8 mode is broken and must be fixed:

$ mount | grep /mnt/fat
/tmp/fat2 on /mnt/fat type vfat
(rw,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)
$ ll /mnt/fat/
total 1
drwxr-xr-x 2 pali pali 512 Jan 19 22:50 ./
drwxrwxrwt 4 root root  80 Jan 19 22:45 ../
$ touch /mnt/fat/č
$ ll /mnt/fat/
total 1
drwxr-xr-x 2 pali pali 512 Jan 19 22:50 ./
drwxrwxrwt 4 root root  80 Jan 19 22:45 ../
-rwxr-xr-x 1 pali pali   0 Jan 19 22:50 č*
$ touch /mnt/fat/Č
$ ll /mnt/fat/
total 1
drwxr-xr-x 2 pali pali 512 Jan 19 22:50 ./
drwxrwxrwt 4 root root  80 Jan 19 22:45 ../
-rwxr-xr-x 1 pali pali   0 Jan 19 22:50 Č*
-rwxr-xr-x 1 pali pali   0 Jan 19 22:50 č*
$ touch /mnt/fat/d
$ ll /mnt/fat/
total 1
drwxr-xr-x 2 pali pali 512 Jan 19 22:50 ./
drwxrwxrwt 4 root root  80 Jan 19 22:45 ../
-rwxr-xr-x 1 pali pali   0 Jan 19 22:50 d*
-rwxr-xr-x 1 pali pali   0 Jan 19 22:50 Č*
-rwxr-xr-x 1 pali pali   0 Jan 19 22:50 č*
$ touch /mnt/fat/D
$ ll /mnt/fat/
total 1
drwxr-xr-x 2 pali pali 512 Jan 19 22:50 ./
drwxrwxrwt 4 root root  80 Jan 19 22:45 ../
-rwxr-xr-x 1 pali pali   0 Jan 19 22:51 d*
-rwxr-xr-x 1 pali pali   0 Jan 19 22:50 Č*
-rwxr-xr-x 1 pali pali   0 Jan 19 22:50 č*

As you can see lowercase 'd' and uppercase 'D' are same, but lowercase
'č' and uppercase 'Č' are not same. This is because 'č' is two bytes
0xc4 0x8d sequence and comparing is done by Latin1 table. 0xc4 is in
Latin 'Ä' which is already in uppercase. 0x8d is control char so is not
changed by tolower/toupper function.

Bigger problem can be with U+C9FF code point. In UTF-8 it is encoded as
bytes 0xe3 0xa7 0xbf (in Latin1 㧿). If you convert it by Latin1 upper
case table you get ç¿ (bytes 0xc3 0xa7 0xbf). First two bytes is valid
UTF-8 sequence for character ç = U+00E7.

Therefore U+C9FF and U+00E7 may be treated in some cases as same
character (when comparing just prefixes), difference only in upper case,
which is fully wrong.

-- 
Pali Rohár
pali.rohar@...il.com

Download attachment "signature.asc" of type "application/pgp-signature" (196 bytes)

Powered by blists - more mailing lists