linux-kernel - Re: rfc: treewide scripted patch mechanism? (was: Re: [PATCH] Makefile: Convert -Wimplicit-fallthrough=3 to just -Wimplicit-fallthrough for clang)QUILT

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <14723fccc2c3362cc045df17fc8554f37c8a8529.camel@perches.com>
Date:   Mon, 19 Aug 2019 17:08:00 -0700
From:   Joe Perches <joe@...ches.com>
To:     Stephen Rothwell <sfr@...b.auug.org.au>
Cc:     Linus Torvalds <torvalds@...ux-foundation.org>,
        Julia Lawall <julia.lawall@...6.fr>,
        "Gustavo A. R. Silva" <gustavo@...eddedor.com>,
        LKML <linux-kernel@...r.kernel.org>,
        clang-built-linux@...glegroups.com,
        Linux Next Mailing List <linux-next@...r.kernel.org>
Subject: Re: rfc: treewide scripted patch mechanism? (was: Re: [PATCH]
 Makefile: Convert -Wimplicit-fallthrough=3 to just -Wimplicit-fallthrough
 for clang)QUILT

On Tue, 2019-08-20 at 09:24 +1000, Stephen Rothwell wrote:
> Hi Joe,

Hi Stephen

> Sorry for the slow response.

No worries. thanks for picking up the thread.

> On Fri, 16 Aug 2019 12:58:27 -0700 Joe Perches <joe@...ches.com> wrote:
> > On Sat, 2019-08-10 at 13:33 -0700, Joe Perches wrote:
> > > On Sat, 2019-08-10 at 13:18 -0700, Joe Perches wrote:  
> > []
> > > > There are classes of patches generated by scripts that have
> > > > no real mechanism to be applied today.
> > > > 
> > > > For instance: global coccinelle scripted changes to use stracpy
> > > > https://lore.kernel.org/lkml/alpine.DEB.2.21.1907251747560.2494@hadrien/
> > > > 
> > > > and trivial scripted changes to MAINTAINERS
> > > > https://lore.kernel.org/lkml/6482e6546dc328ec47b07dba9a78a9573ebb3e56.camel@perches.com/
> > > > 
> > > > that are basically impossible to be applied by anyone but you.
> > > > 
> > > > Otherwise there are hundreds of little micro patches most of
> > > > which would not otherwise be applied.
> > > > 
> > > > There should be some process available to get these treewide
> > > > or difficult to keep up-to-date and apply patches handled.
> > > > 
> > > > I believe these sorts of scripted patches should ideally
> > > > be handled immediately before an RC1 so other trees can be 
> > > > synchronized in the simplest way possible.  
> > > 
> > > Hey Stephen
> > > 
> > > Question for you about a possible -next process change.
> > > 
> > > Would it be reasonable to have some mechanism to script
> > > treewide patches to generate and apply after Andrew Morton's
> > > mmotm patches are applied to -next?
> 
> I don't see why not (its all just software, right? :-)).  I would have
> to refresh my understanding of how Andrew constructs his mmot{s,m} quilt
> series, but I should be able to sort that out.  The only other issue is
> the time it takes to apply these changes and test them.  The total time
> it takes to construct linux-next each day increases towards the opening
> of the merge window (we are currently at -rc5 and I am already taking
> about 12 hours each day).
> 
> > > This could allow treewide scripted patches to have
> > > compilation and test coverage before possibly being
> > > applied to Linus' tree.
> 
> Always a good thing :-)
> 
> So, do we have a pending example, or can you give my some idea of what
> they would look like?

A few examples:

1: a patch just to MAINTAINERS done via bash script:

https://lore.kernel.org/lkml/904551f1f198ffac9a0f9c3c99aa966b0a7c76c1.camel@perches.com/

$ git grep -h "^[FX]:" MAINTAINERS | \
  cut -f2- | grep -vP '/$|\*|\?|\[' | \
  while read file ; do \
    if [ -d $file ]; then \
      sed -i -e "s@...ile}\$@...ile}/@" MAINTAINERS ; \
    fi ; \
  done

This one is trivial and takes almost no time.

2: would be Julia Lawall's stracpy change done
with coccinelle: (attached)

This one takes quite a bit longer as it has to do a
cocci --all-includes scan of each source file and each
of its #include files.

The 1st MAINTAINERS change is an annoyance because it
either is individual patches for each of 50 subsystems
or a single patch that changes constantly.  Either
tends to get elements dropped on the floor.

The 2nd is treewide and quite a large patch which
spans nearly every subsystem.  These types of patches
generally are not always acceptable to one party or
another but do allow whatever exceptional uses of
strlcpy or strncpy that remain to be analyzed for
defects.

3: might be the /* fallthrough */ to fallthrough;
script attached to this email:

https://lore.kernel.org/lkml/61ddbb86d5e68a15e24ccb06d9b399bbf5ce2da7.camel@perches.com/

$ git ls-files -- '*.[ch]' | \
  xargs perl cvt_style.pl -o --convert=fallthrough

but this depends on the acceptance of another currently
rfc patch:

https://lore.kernel.org/lkml/1d2830aadbe9d8151728a7df5b88528fc72a0095.1564549413.git.joe@perches.com/

This script takes around 15 minutes on my 3 year old
laptop with an ssd.

cheers, Joe

View attachment "stracpy.cocci" of type "text/x-csrc" (589 bytes)

View attachment "stracpy.out" of type "text/x-patch" (93956 bytes)