linux-kernel - Re: OT: character encodings

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <45A14A2A.9060306@imap.cc>
Date:	Sun, 07 Jan 2007 20:29:46 +0100
From:	Tilman Schmidt <tilman@...p.cc>
To:	Russell King <rmk+lkml@....linux.org.uk>
CC:	David Woodhouse <dwmw2@...radead.org>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: OT: character encodings

Am 07.01.2007 18:06 schrieb Russell King:
> 
> $ git log | head -n 1000 | tail -n 200 > o
> $ file -i o
> o: text/plain; charset=us-ascii
> $ git log | head -n 1000 | tail -n 300 > o
> $ file -i o
> o: text/plain; charset=us-ascii
> $ git log | head -n 1000 | tail -n 400 > o
> $ file -i o
> o: text/plain; charset=utf-8
> 
> (and you know what charset the file is thought to have with all 1000
> lines in it.)

What the "file" command thinks is hardly relevant here. "file" just
attempts to guess what the contents of a file might be, by applying
a simple set of heuristics. Your results only highlight the actual
problem: "git" is apparently unable to handle character sets properly
and instead produces a mix of encodings as output.

> All on a system with LANG set to en_GB (iow ISO-8859-1).

For software with proper multilingual support, that should have been
enough to make sure that all its output would be in iso-8859-1, too.
Obviously "git" doesn't fall into that category.

>> Yes. When you stored it on disk, the character set information was lost.
> 
> The same thing actually happens when I look at it via:
> 
>   $ git log | head -n 1000 | less

The loss has happened long before you run that command, when the
data was committed into "git".

> So, I think you'll find that the contents of git _is_ an ad-hoc collection
> of character sets which people happen to have in use on their machines.

Exactly.

>> A mixed charset environment was _already_ a pain in the butt, because
>> almost nobody got labelling right. It's wrong to blame that on UTF-8.
> 
> I'm not talking about a mixed charset environment.  I'm talking about
> non-UTF-8 single charset environments being broken by programs which
> universally think the universe is UTF-8 only.

The problem is not programs thinking the universe is UTF-8 only; it's
people mixing different charsets, in conjunction with programs not
caring about charsets at all.

Specifically, your non-UTF-8 single charset environment was not broken
by git thinking everything was UTF-8, but to the contrary by some data
in the git repository actually being UTF-8 and git *not* thinking that.

And that problem is, I repeat, much older than UTF-8.

HTH
Tilman

-- 
Tilman Schmidt                          E-Mail: tilman@...p.cc
Bonn, Germany
Diese Nachricht besteht zu 100% aus wiederverwerteten Bits.
Ungeöffnet mindestens haltbar bis: (siehe Rückseite)

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/