linux-kernel - Re: OT: character encodings (was: Linux 2.6.20-rc4)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20070107204834.GU24090@1wt.eu>
Date:	Sun, 7 Jan 2007 21:48:34 +0100
From:	Willy Tarreau <w@....eu>
To:	Jan Engelhardt <jengelh@...ux01.gwdg.de>
Cc:	Russell King <rmk+lkml@....linux.org.uk>,
	David Woodhouse <dwmw2@...radead.org>,
	Tilman Schmidt <tilman@...p.cc>,
	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: OT: character encodings (was: Linux 2.6.20-rc4)

On Sun, Jan 07, 2007 at 08:11:38PM +0100, Jan Engelhardt wrote:
> 
> On Jan 7 2007 17:06, Russell King wrote:
> >On Mon, Jan 08, 2007 at 12:29:05AM +0800, David Woodhouse wrote:
> >
> >$ git log | head -n 1000 | tail -n 200 > o
> >$ file -i o
> >o: text/plain; charset=us-ascii
> >$ git log | head -n 1000 | tail -n 300 > o
> >$ file -i o
> >o: text/plain; charset=us-ascii
> >$ git log | head -n 1000 | tail -n 400 > o
> >$ file -i o
> >o: text/plain; charset=utf-8
> 
> I am inclined to say that "file" does not count, because it tries to guess an
> ambiguous mapping from bytes to character set. Even more, file should be
> _unable at all_ to distinguish an iso-8859-1 from an iso-8859-2 (or worse: 15)
> file. This program is soo... forget it, it's not an argument. It works well for
> headerful files, but text files don't really contain one. The next best thing
> would be html, with a proper <meta http-equiv=Content> tag.

The stupidity from the start up with those character sets is that they
consider that a whole file is written with a given set. In fact, the
charset should apply to characters themselves. At least, the
quoted-printable, non-human friendly, encoding was the least stupid.

Now that UTF8 comes everywhere, everyone receives tons of mangled mails,
and even mailers which correctly support UTF8 and use it by default manage
to shoot themselves in the foot when they reply to, or forward a mail. The
system is completely broken because limited by design, and we have to learn
to live with this brokenness.

Willy

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/