linux-kernel - [PATCH] scripts: Add utility to convert make's output into JSON

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-Id: <20240319-make-to-json-script-v1-1-621cc4d502cb@collabora.com>
Date: Tue, 19 Mar 2024 17:24:26 -0400
From: Nícolas F. R. A. Prado <nfraprado@...labora.com>
To: Masahiro Yamada <masahiroy@...nel.org>, Rob Herring <robh@...nel.org>
Cc: kernel@...labora.com, linux-kernel@...r.kernel.org, 
 Nícolas F. R. A. Prado <nfraprado@...labora.com>
Subject: [PATCH] scripts: Add utility to convert make's output into JSON

The output from make isn't well suited for machines: reading it requires
hardcoding details of the formatting of commands from make.

Add a script that parses the output from make and outputs it in a well
structured format in JSON. This allows third party programs to parse the
make output without caring about formatting details, which stay in the
kernel tree.

For a given command line from make's output:
  DTC_CHK arch/arm64/boot/dts/actions/s700-cubieboard7.dtb

where DTC_CHK is the command, arch/.../s700-cubieboard7.dtb is the
target and any non-command lines that follow it are the command's
output, the generated JSON output is in the format:
[
    ...,
    {
        "command": "DTC_CHK",
        "target": "arch/arm64/boot/dts/actions/s700-cubieboard7.dtb",
        "output": "<any following non-command lines>"
    },
    ...
]

This assumes the output from a command immediately follows it. If
building with multiple jobs (`make -j`), the `--output-sync` argument
also needs to be supplied to make in order for the output fields to be
correctly set.

For an example of the usage of this script, consider `make dtbs_check`
generates a very long output in the following format:

  DTC_CHK arch/arm64/boot/dts/arm/foundation-v8-psci.dtb
  DTC_CHK arch/arm64/boot/dts/allwinner/sun50i-a64-amarula-relic.dtb
/tmp/kci/linux/arch/arm64/boot/dts/allwinner/sun50i-a64-amarula-relic.dtb: thermal-zones: gpu0-thermal: 'trips' is a required property
	from schema $id: http://devicetree.org/schemas/thermal/thermal-zones.yaml#
/tmp/kci/linux/arch/arm64/boot/dts/allwinner/sun50i-a64-amarula-relic.dtb: thermal-zones: gpu1-thermal: 'trips' is a required property
	from schema $id: http://devicetree.org/schemas/thermal/thermal-zones.yaml#
  DTC_CHK arch/arm64/boot/dts/altera/socfpga_stratix10_socdk_nand.dtb

With this output converted into JSON, it will be much simpler to
identify that only the middle DTB in this example produces warnings. By
extension, CIs will be able to more easily report which DTBs are
currently producing warnings and which are not, and identify
regressions.

Signed-off-by: Nícolas F. R. A. Prado <nfraprado@...labora.com>
---
Hi,
I'm not sure which subsystem this script would be a part of, which is
why I haven't added an entry in MAINTAINERS. Suggestions appreciated.
---
 scripts/make-to-json.py | 51 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 51 insertions(+)

diff --git a/scripts/make-to-json.py b/scripts/make-to-json.py
new file mode 100755
index 000000000000..7570134f6151
--- /dev/null
+++ b/scripts/make-to-json.py
@@ -0,0 +1,51 @@
+#!/bin/env python
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2024 Collabora Ltd
+#
+# Transform the output from make into JSON. This makes it easier for machines to
+# parse.
+#
+# NOTE: This assumes the output from a command immediately follows it. If
+# building with multiple jobs (`make -j`), make sure to also pass
+# `--output-sync`.
+#
+# Command lines from make are in the format:
+#   DTC_CHK arch/arm64/boot/dts/actions/s700-cubieboard7.dtb
+#
+# where DTC_CHK is the command, arch/.../s700-cubieboard7.dtb is the target and
+# any non-command lines that follow it are the command's output.
+#
+# The JSON output is in the format:
+# [
+#     {
+#         "command": "DTC_CHK",
+#         "target": "arch/arm64/boot/dts/actions/s700-cubieboard7.dtb",
+#         "output": ""
+#     },
+#     ...
+# ]
+
+import json
+import re
+import sys
+
+make_cmd_re = re.compile(r"^  (?P<command>[A-Z0-9_.]{1,8}( \[M\])?) +(?P<target>.*)")
+
+make_output = []
+step = {}
+
+for line in sys.stdin:
+    match = make_cmd_re.match(line)
+    if match:
+        if step:
+            make_output.append(step)  # save previous step, if any
+        step = {}
+        step["command"] = match.group("command")
+        step["target"] = match.group("target")
+        step["output"] = ""
+    else:
+        step["output"] += line
+if step:
+    make_output.append(step)  # save last step, if any
+
+json.dump(make_output, sys.stdout)

---
base-commit: dad309222e4c3fc7f88b20ce725ce1e0eea07cc7
change-id: 20240319-make-to-json-script-764103bf74d7

Best regards,
-- 
Nícolas F. R. A. Prado <nfraprado@...labora.com>