linux-kernel - Re: [PATCH 1/2] scripts/spdxcheck.py: Always open files in binary mode

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Thu, 13 Dec 2018 10:10:52 -0500
From:   Jeremy Cline <jcline@...hat.com>
To:     Uwe Kleine-König <u.kleine-koenig@...gutronix.de>
Cc:     Thierry Reding <thierry.reding@...il.com>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Jonathan Corbet <corbet@....net>,
        Joe Perches <joe@...ches.com>, linux-kernel@...r.kernel.org
Subject: Re: [PATCH 1/2] scripts/spdxcheck.py: Always open files in binary
 mode

On Thu, Dec 13, 2018 at 08:37:08AM +0100, Uwe Kleine-König wrote:
> On Wed, Dec 12, 2018 at 01:14:10PM -0500, Jeremy Cline wrote:
> > Hi,
> > 
> > On Wed, Dec 12, 2018 at 02:12:09PM +0100, Thierry Reding wrote:
> > > From: Thierry Reding <treding@...dia.com>
> > > 
> > > The spdxcheck script currently falls over when confronted with a binary
> > > file (such as Documentation/logo.gif). To avoid that, always open files
> > > in binary mode and decode line-by-line, ignoring encoding errors.
> 
> I suggest pointing out that the breakage only happens with python3 and
> results in a UnicodeDecodeError.
> 
> > > One tricky case is when piping data into the script and reading it from
> > > standard input. By default, standard input will be opened in text mode,
> > > so we need to reopen it in binary mode.
> > > 
> > > Signed-off-by: Thierry Reding <treding@...dia.com>
> > > ---
> > >  scripts/spdxcheck.py | 6 ++++--
> > >  1 file changed, 4 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/scripts/spdxcheck.py b/scripts/spdxcheck.py
> > > index 5056fb3b897d..e559c6294c39 100755
> > > --- a/scripts/spdxcheck.py
> > > +++ b/scripts/spdxcheck.py
> > > @@ -168,6 +168,7 @@ class id_parser(object):
> > >          self.curline = 0
> > >          try:
> > >              for line in fd:
> > > +                line = line.decode(locale.getpreferredencoding(False), errors='ignore')
> > >                  self.curline += 1
> > >                  if self.curline > maxlines:
> > >                      break
> > > @@ -249,12 +250,13 @@ if __name__ == '__main__':
> > >  
> > >      try:
> > >          if len(args.path) and args.path[0] == '-':
> > > -            parser.parse_lines(sys.stdin, args.maxlines, '-')
> > > +            stdin = os.fdopen(sys.stdin.fileno(), 'rb')
> > > +            parser.parse_lines(stdin, args.maxlines, '-')
> > >          else:
> > >              if args.path:
> > >                  for p in args.path:
> > >                      if os.path.isfile(p):
> > > -                        parser.parse_lines(open(p), args.maxlines, p)
> > > +                        parser.parse_lines(open(p, 'rb'), args.maxlines, p)
> > >                      elif os.path.isdir(p):
> > >                          scan_git_subtree(repo.head.reference.commit.tree, p)
> > >                      else:
> > > -- 
> > > 2.19.1
> > > 
> > 
> > It might be worth noting this fixes commit 6f4d29df66ac
> > ("scripts/spdxcheck.py: make python3 compliant") and also Cc this for
> > stable since 6f4d29df66ac got backported to v4.19. While that commit
> > did indeed make the script work with Python 3 for piping data, it broke
> > Python 2 and made its way to stable.
> 
> It didn't break for me. Can you provide details about how and when it
> broke for you?

I was wrong about it being Python 2 that broke, sorry about that.
6f4d29df66ac broke Python 3 when you run it against a sub-tree because
scan_git_tree() opens the files in binary mode, but then find is run
with a text string:

$ python3 scripts/spdxcheck.py net/
FAIL: argument should be integer or bytes-like object, not 'str'
Traceback (most recent call last):
  File "scripts/spdxcheck.py", line 259, in <module>
    scan_git_subtree(repo.head.reference.commit.tree, p)
  File "scripts/spdxcheck.py", line 211, in scan_git_subtree
    scan_git_tree(tree)
  File "scripts/spdxcheck.py", line 206, in scan_git_tree
    parser.parse_lines(fd, args.maxlines, el.path)
  File "scripts/spdxcheck.py", line 175, in parse_lines
    if line.find("SPDX-License-Identifier:") < 0:
TypeError: argument should be integer or bytes-like object, not 'str'

The reason I opened things in binary mode when I started adding Python 3
support was because not all files were valid UTF-8 (and some were
binary) so I decoded the text line-by-line and ignored any decoding
errors for simplicity's sake.

Regards,
Jeremy