linux-kernel - Re: [PATCH] [RFC] dt-bindings: Consider DT_SCHEMA

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAL_Jsq+n6DVpoSkym3E4yG7BcUu6jS8O51tCpFD4-CeCtPesqA@mail.gmail.com>
Date:   Thu, 11 Mar 2021 14:14:02 -0700
From:   Rob Herring <robh+dt@...nel.org>
To:     Geert Uytterhoeven <geert@...ux-m68k.org>
Cc:     Masahiro Yamada <masahiroy@...nel.org>,
        "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS" 
        <devicetree@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH] [RFC] dt-bindings: Consider DT_SCHEMA_FILES when finding
 all json-schema

On Wed, Mar 10, 2021 at 1:43 AM Geert Uytterhoeven <geert@...ux-m68k.org> wrote:
>
> Hi Rob,
>
> On Tue, Mar 9, 2021 at 6:42 PM Rob Herring <robh+dt@...nel.org> wrote:
> > On Tue, Mar 9, 2021 at 4:21 AM Geert Uytterhoeven
> > <geert+renesas@...der.be> wrote:
> > > Setting DT_SCHEMA_FILES allows the user to restrict the
> > > "dt_binding_check" make target to a specified set of DT binding files.
> > > However, yamllint is still run on all available files, which can take
> > > quite some time.
> > >
> > > Fix this by changing "find_cmd" to only return the specified files.
> > > Note that this also affects the "cmd_chk_bindings" and "cmd_mk_schema"
> > > rules.
> > >
> > > This reduces the execution time of
> > >
> > >     make dt_binding_check DT_SCHEMA_FILES=/path/to/json/schema/file
> > >
> > > from ca. 22 to less than 2 seconds on an i7-8700K.
> >
> > We could use xargs sharding like 'chk_bindings' does. That goes from
> > 18s to 5s for me (i7-7700HQ). Good enough? Not sure why I didn't other
> > than thinking 20sec was fast enough.
>
> Sounds better, but yamllint (on all files) would still take 80% of the
> time for a single binding check, assuming a reasonably fast multi-core
> machine.  My main objection is that while working on a new binding, and
> using DT_SCHEMA_FILES, I don't want to waste time on checking other
> bindings, and being bothered with warnings about them.

We could move yamllint to when we extract the example. Then it only
runs on modified schema files. The problem with that is we have to
watch out for slow python start-up times. I get about 25sec for the
whole tree 1 by 1 with a quick test:

$ time find Documentation/devicetree/bindings/ -name '*.yaml' | xargs
-P8 -n1 yamllint -f parsable -c
Documentation/devicetree/bindings/.yamllint

real    0m24.939s
user    2m55.846s
sys     0m12.738s

Of course, that's a parallel build compared to 18sec single threaded.
I'm seeing about 8 min for full dt_binding_check without yamllint.

Here's what the kbuild changes would look like(whitespace damaged):

diff --git a/Documentation/devicetree/bindings/Makefile
b/Documentation/devicetree/bindings/Makefile
index 780e5618ec0a..3a213343d587 100644
--- a/Documentation/devicetree/bindings/Makefile
+++ b/Documentation/devicetree/bindings/Makefile
@@ -16,8 +16,13 @@ check_dtschema_version:
 quiet_cmd_extract_ex = DTEX    $@
       cmd_extract_ex = $(DT_EXTRACT_EX) $< > $@

+define rule_extract_ex
+       $(if $(DT_SCHEMA_LINT),$(call cmd,yamllint),)
+       $(call cmd,extract_ex)
+endef
+
 $(obj)/%.example.dts: $(src)/%.yaml check_dtschema_version FORCE
-       $(call if_changed,extract_ex)
+       $(call if_changed_rule,extract_ex)

 # Use full schemas when checking %.example.dts
 DT_TMP_SCHEMA := $(obj)/processed-schema-examples.json
@@ -26,9 +31,8 @@ find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
                -name 'processed-schema*' ! \
                -name '*.example.dt.yaml' \)

-quiet_cmd_yamllint = LINT    $(src)
-      cmd_yamllint = ($(find_cmd) | \
-                     xargs $(DT_SCHEMA_LINT) -f parsable -c
$(srctree)/$(src)/.yamllint) || true
+quiet_cmd_yamllint = LINT    $(patsubst $(srctree)/%,%,$<)
+      cmd_yamllint = $(DT_SCHEMA_LINT) -f parsable -c
$(srctree)/$(src)/.yamllint $(real-prereqs) || true

 quiet_cmd_chk_bindings = CHKDT   $@
       cmd_chk_bindings = ($(find_cmd) | \
@@ -43,7 +47,6 @@ quiet_cmd_mk_schema = SCHEMA  $@
                      rm -f $$f

 define rule_chkdt
-       $(if $(DT_SCHEMA_LINT),$(call cmd,yamllint),)
        $(call cmd,chk_bindings)
        $(call cmd,mk_schema)
 endef