| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Fri, 17 Jun 2022 22:02:02 +0800
From: Xing Zhengjun <zhengjun.xing@...ux.intel.com>
To: Ian Rogers <irogers@...gle.com>
Cc: Peter Zijlstra <peterz@...radead.org>,
Ingo Molnar <mingo@...hat.com>,
Arnaldo Carvalho de Melo <acme@...nel.org>,
Mark Rutland <mark.rutland@....com>,
Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
Jiri Olsa <jolsa@...nel.org>,
Namhyung Kim <namhyung@...nel.org>,
John Garry <john.garry@...wei.com>,
Kan Liang <kan.liang@...ux.intel.com>,
Andi Kleen <ak@...ux.intel.com>, Felix Fietkau <nbd@....name>,
Qi Liu <liuqi115@...wei.com>, Like Xu <likexu@...cent.com>,
linux-kernel@...r.kernel.org, linux-perf-users@...r.kernel.org,
Nick Forrington <nick.forrington@....com>,
Kajol Jain <kjain@...ux.ibm.com>,
James Clark <james.clark@....com>,
Andrew Kilroy <andrew.kilroy@....com>,
"Paul A . Clarke" <pc@...ibm.com>, Will Deacon <will@...nel.org>,
Mathieu Poirier <mathieu.poirier@...aro.org>,
ananth.narayan@....com, ravi.bangoria@....com,
santosh.shukla@....com, sandipan.das@....com,
Caleb Biggers <caleb.biggers@...el.com>,
Perry Taylor <perry.taylor@...el.com>,
Kshipra Bopardikar <kshipra.bopardikar@...el.com>,
Stephane Eranian <eranian@...gle.com>
Subject: Re: [PATCH v5 2/4] perf jevents: Add python converter script
Hi Ian,
On 6/17/2022 9:33 PM, Ian Rogers wrote:
> On Thu, Jun 16, 2022 at 11:51 PM Xing Zhengjun
> <zhengjun.xing@...ux.intel.com> wrote:
>>
>>
>>
>> On 6/17/2022 10:40 AM, Ian Rogers wrote:
>>> On Thu, Jun 16, 2022 at 6:52 PM Xing Zhengjun
>>> <zhengjun.xing@...ux.intel.com> wrote:
>>>>
>>>> Hi Ian,
>>>>
>>>> On 6/17/2022 9:26 AM, Ian Rogers wrote:
>>>>> On Thu, Jun 16, 2022 at 5:04 PM Xing Zhengjun
>>>>> <zhengjun.xing@...ux.intel.com> wrote:
>>>>>>
>>>>>> Hi Ian,
>>>>>>
>>>>>> I applied this patch series based on the head of the perf/core
>>>>>> branch, but it failed when do make.
>>>>>> My python version:
>>>>>> Python 2.7.18 (default, Mar 8 2021, 13:02:45)
>>>>>> [GCC 9.3.0] on linux2
>>>>>>
>>>>>>
>>>>>> GEN pmu-events/pmu-events.c
>>>>>> make[3]: Nothing to be done for 'install_headers'.
>>>>>> GEN python/perf.so
>>>>>> Traceback (most recent call last):
>>>>>> File "pmu-events/jevents.py", line 23, in <module>
>>>>>> def file_name_to_table_name(parents: list[str], dirname: str) -> str:
>>>>>> TypeError: 'type' object is not subscriptable
>>>>>> make[3]: *** [pmu-events/Build:20: pmu-events/pmu-events.c] Error 1
>>>>>> make[2]: *** [Makefile.perf:663: pmu-events/pmu-events-in.o] Error 2
>>>>>> make[2]: *** Waiting for unfinished jobs....
>>>>>> make[1]: *** [Makefile.perf:240: sub-make] Error 2
>>>>>> make: *** [Makefile:70: all] Error 2
>>>>>
>>>>> Hi Zhengjun,
>>>>>
>>>>> Thanks for testing! You are running the test script and not the main
>>>>> build. Unlike the main build I didn't add a Python version test for
>>>>> the test script. As mentioned in the cover letter you need Python 3.6
>>>>> as Python 2 was sunset in 2020.
>>>>>
>>>>
>>>> In fact, I run the main build,
>>>> $ cd tools/perf
>>>> $ make
>>>>
>>>> $ lsb_release -a
>>>> No LSB modules are available.
>>>> Distributor ID: Ubuntu
>>>> Description: Ubuntu 20.04.3 LTS
>>>> Release: 20.04
>>>> Codename: focal
>>>>
>>>> The default python in Ubuntu 20.04.3 is Python2
>>>>
>>>> I think a lot of developers also use the Ubuntu 20.04.3 LTS,
>>>> it's better to support python2.
>>>
>>> TL;DR "apt install python-dev-is-python3" :-)
>>>
>>> The problem is what you lose by making the code work with python2:
>>> f-strings, various library functions. You end up coding to a worse
>>> standard of Python and the code is considered non-pythonic. Python 3.6
>>> was released in Dec. 2016 and so is over 5 years old now - ie I'm not
>>> pushing for the bleeding edge.
>>>
>>> A problem we face in the regular perf Python build is that Python 2
>>> doesn't support new APIs and Python3 has deprecated the old APIs [1].
>>> I think the sensible thing is to follow Python 3's deprecation plan
>>> which will mean losing Python 2 support, ie we shouldn't look for
>>> python2-config in the build any more as python2 won't support the APIs
>>> our scripts are now using.
>>>
>>> I think arguing that Python 2 was something not worth sunsetting was
>>> an argument to be made 2 years ago, but now we have to live with the
>>> consequences. I don't think there is any hardship except an install
>>> step. For the build changes in these patches if an old version of
>>> Python is detected then you just don't get jevents and there is a
>>> warning at build time.
>>
>> After "sudo apt install python-dev-is-python3", now the python version :
>> Python 3.8.10 (default, Mar 15 2022, 12:22:08)
>> [GCC 9.4.0] on linux
>>
>> But the issue still happened.
>>
>> $cd tools/perf
>> $make
>> GEN pmu-events/pmu-events.c
>> make[3]: Nothing to be done for 'install_headers'.
>> GEN python/perf.so
>> Traceback (most recent call last):
>> File "pmu-events/jevents.py", line 23, in <module>
>> def file_name_to_table_name(parents: list[str], dirname: str) -> str:
>> TypeError: 'type' object is not subscriptable
>> make[3]: *** [pmu-events/Build:20: pmu-events/pmu-events.c] Error 1
>> make[2]: *** [Makefile.perf:663: pmu-events/pmu-events-in.o] Error 2
>> make[2]: *** Waiting for unfinished jobs....
>> make[1]: *** [Makefile.perf:240: sub-make] Error 2
>> make: *** [Makefile:70: all] Error 2
>>
>>
>> For the python version, I think python3 is ok.
>> But during the build, we need to add a warning and how to fix messages
>> when detecting the system is using pthon2, just like it did before:
>>
>> Auto-detecting system features:
>> ... dwarf: [ on ]
>> ... dwarf_getlocations: [ on ]
>> ... glibc: [ on ]
>> ... libbfd: [ on ]
>> ... libbfd-buildid: [ on ]
>> ... libcap: [ on ]
>> ... libelf: [ on ]
>> ... libnuma: [ on ]
>> ... numa_num_possible_cpus: [ on ]
>> ... libperl: [ on ]
>> ... libpython: [ on ]
>> ... libcrypto: [ on ]
>> ... libunwind: [ on ]
>> ... libdw-dwarf-unwind: [ on ]
>> ... zlib: [ on ]
>> ... lzma: [ on ]
>> ... get_cpuid: [ on ]
>> ... bpf: [ on ]
>> ... libaio: [ on ]
>> ... libzstd: [ on ]
>> ... disassembler-four-args: [ on ]
>> ... python-dev-is-python3: [ off ]
>>
>
> Agreed on cleaning up the python Makefile logic. I'd rather do it out
> of these patches. For the issue you see can you try adding the 1
> liner:
>
> ```
> diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py
> index 658eb2696f9a..9f1a37515c0f 100755
> --- a/tools/perf/pmu-events/jevents.py
> +++ b/tools/perf/pmu-events/jevents.py
> @@ -1,6 +1,7 @@
> #!/usr/bin/env python3
> # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> """Convert directories of JSON events to C code."""
> +from __future__ import annotations
> import argparse
> import csv
> import json
> ```
>
With the patch, the build still fails.
GEN pmu-events/pmu-events.c
make[3]: Nothing to be done for 'install_headers'.
GEN python/perf.so
Traceback (most recent call last):
File "pmu-events/jevents.py", line 395, in <module>
main()
File "pmu-events/jevents.py", line 387, in main
ftw(arch_path, [], process_one_file)
File "pmu-events/jevents.py", line 368, in ftw
ftw(item.path, parents + [item.name], action)
File "pmu-events/jevents.py", line 366, in ftw
action(parents, item)
File "pmu-events/jevents.py", line 298, in process_one_file
print_events_table_entries(item, get_topic(item.name))
File "pmu-events/jevents.py", line 270, in get_topic
return topic.removesuffix('.json').replace('-', ' ')
AttributeError: 'str' object has no attribute 'removesuffix'
make[3]: *** [pmu-events/Build:20: pmu-events/pmu-events.c] Error 1
make[3]: *** Deleting file 'pmu-events/pmu-events.c'
make[2]: *** [Makefile.perf:663: pmu-events/pmu-events-in.o] Error 2
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [Makefile.perf:240: sub-make] Error 2
make: *** [Makefile:70: all] Error 2
> Thanks,
> Ian
>
>>>
>>> Thanks,
>>> Ian
>>>
>>> [1] https://lore.kernel.org/lkml/20220615014206.26651-1-irogers@google.com/
>>>
>>>
>>>>> Thanks,
>>>>> Ian
>>>>>
>>>>>
>>>>>> On 6/16/2022 12:48 PM, Ian Rogers wrote:
>>>>>>> jevents.c is large, has a dependency on an old forked version of jsmn,
>>>>>>> and is challenging to work upon. A lot of jevents.c's complexity comes
>>>>>>> from needing to write json and csv parsing from first principles. In
>>>>>>> contrast python has this functionality in standard libraries and is
>>>>>>> already a build pre-requisite for tools like asciidoc (that builds all
>>>>>>> of the perf man pages).
>>>>>>>
>>>>>>> Introduce jevents.py that produces identical output to jevents.c. Add a
>>>>>>> test that runs both converter tools and validates there are no output
>>>>>>> differences. The test can be invoked with a phony build target like:
>>>>>>>
>>>>>>> make -C tools/perf jevents-py-test
>>>>>>>
>>>>>>> The python code deliberately tries to replicate the behavior of
>>>>>>> jevents.c so that the output matches and transitioning tools shouldn't
>>>>>>> introduce regressions. In some cases the code isn't as elegant as hoped,
>>>>>>> but fixing this can be done as follow up.
>>>>>>>
>>>>>>> Signed-off-by: Ian Rogers <irogers@...gle.com>
>>>>>>> ---
>>>>>>> tools/perf/Makefile.perf | 6 +
>>>>>>> tools/perf/pmu-events/jevents-test.sh | 33 +++
>>>>>>> tools/perf/pmu-events/jevents.py | 394 ++++++++++++++++++++++++++
>>>>>>> 3 files changed, 433 insertions(+)
>>>>>>> create mode 100755 tools/perf/pmu-events/jevents-test.sh
>>>>>>> create mode 100755 tools/perf/pmu-events/jevents.py
>>>>>>>
>>>>>>> diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
>>>>>>> index 8f738e11356d..1e29c8936f71 100644
>>>>>>> --- a/tools/perf/Makefile.perf
>>>>>>> +++ b/tools/perf/Makefile.perf
>>>>>>> @@ -669,6 +669,12 @@ $(JEVENTS_IN): FORCE
>>>>>>> $(JEVENTS): $(JEVENTS_IN)
>>>>>>> $(QUIET_LINK)$(HOSTCC) $(JEVENTS_IN) -o $@
>>>>>>>
>>>>>>> +JEVENTS_PY := pmu-events/jevents.py
>>>>>>> +JEVENTS_PY_TEST := pmu-events/jevents-test.sh
>>>>>>> +.PHONY: jevents-py-test
>>>>>>> +jevents-py-test: $(JEVENTS)
>>>>>>> + $(Q)$(call echo-cmd,gen)$(JEVENTS_PY_TEST) $(JEVENTS) $(JEVENTS_PY) pmu-events/arch
>>>>>>> +
>>>>>>> $(PMU_EVENTS_IN): $(JEVENTS) FORCE
>>>>>>> $(Q)$(MAKE) -f $(srctree)/tools/build/Makefile.build dir=pmu-events obj=pmu-events
>>>>>>>
>>>>>>> diff --git a/tools/perf/pmu-events/jevents-test.sh b/tools/perf/pmu-events/jevents-test.sh
>>>>>>> new file mode 100755
>>>>>>> index 000000000000..9ae852292576
>>>>>>> --- /dev/null
>>>>>>> +++ b/tools/perf/pmu-events/jevents-test.sh
>>>>>>> @@ -0,0 +1,33 @@
>>>>>>> +#!/bin/sh
>>>>>>> +# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
>>>>>>> +# Validate that the legacy jevents and jevents.py produce identical output.
>>>>>>> +set -e
>>>>>>> +
>>>>>>> +JEVENTS="$1"
>>>>>>> +JEVENTS_PY="$2"
>>>>>>> +ARCH_PATH="$3"
>>>>>>> +JEVENTS_C_GENERATED=$(mktemp /tmp/jevents_c.XXXXX.c)
>>>>>>> +JEVENTS_PY_GENERATED=$(mktemp /tmp/jevents_py.XXXXX.c)
>>>>>>> +
>>>>>>> +cleanup() {
>>>>>>> + rm "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED"
>>>>>>> + trap - exit term int
>>>>>>> +}
>>>>>>> +trap cleanup exit term int
>>>>>>> +
>>>>>>> +for path in "$ARCH_PATH"/*
>>>>>>> +do
>>>>>>> + arch=$(basename $path)
>>>>>>> + if [ "$arch" = "test" ]
>>>>>>> + then
>>>>>>> + continue
>>>>>>> + fi
>>>>>>> + echo "Checking architecture: $arch"
>>>>>>> + echo "Generating using jevents.c"
>>>>>>> + "$JEVENTS" "$arch" "$ARCH_PATH" "$JEVENTS_C_GENERATED"
>>>>>>> + echo "Generating using jevents.py"
>>>>>>> + "$JEVENTS_PY" "$arch" "$ARCH_PATH" "$JEVENTS_PY_GENERATED"
>>>>>>> + echo "Diffing"
>>>>>>> + diff -u "$JEVENTS_C_GENERATED" "$JEVENTS_PY_GENERATED"
>>>>>>> +done
>>>>>>> +cleanup
>>>>>>> diff --git a/tools/perf/pmu-events/jevents.py b/tools/perf/pmu-events/jevents.py
>>>>>>> new file mode 100755
>>>>>>> index 000000000000..658eb2696f9a
>>>>>>> --- /dev/null
>>>>>>> +++ b/tools/perf/pmu-events/jevents.py
>>>>>>> @@ -0,0 +1,394 @@
>>>>>>> +#!/usr/bin/env python3
>>>>>>> +# SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
>>>>>>> +"""Convert directories of JSON events to C code."""
>>>>>>> +import argparse
>>>>>>> +import csv
>>>>>>> +import json
>>>>>>> +import os
>>>>>>> +import sys
>>>>>>> +from typing import Callable
>>>>>>> +
>>>>>>> +# Global command line arguments.
>>>>>>> +args = None
>>>>>>> +# List of event tables generated from "/sys" directories.
>>>>>>> +sys_event_tables = []
>>>>>>> +# Map from an event name to an architecture standard
>>>>>>> +# JsonEvent. Architecture standard events are in json files in the top
>>>>>>> +# f'{args.starting_dir}/{args.arch}' directory.
>>>>>>> +arch_std_events = dict()
>>>>>>> +# Track whether an events table is currently being defined and needs closing.
>>>>>>> +close_table = False
>>>>>>> +
>>>>>>> +
>>>>>>> +def file_name_to_table_name(parents: list[str], dirname: str) -> str:
>>>>>>> + """Generate a C table name from directory names."""
>>>>>>> + tblname = 'pme'
>>>>>>> + for p in parents:
>>>>>>> + tblname += '_' + p
>>>>>>> + tblname += '_' + dirname
>>>>>>> + return tblname.replace('-', '_')
>>>>>>> +
>>>>>>> +
>>>>>>> +class JsonEvent:
>>>>>>> + """Representation of an event loaded from a json file dictionary."""
>>>>>>> +
>>>>>>> + def __init__(self, dict):
>>>>>>> + """Constructor passed the dictionary of parsed json values."""
>>>>>>> +
>>>>>>> + def llx(x: int) -> str:
>>>>>>> + """Convert an int to a string similar to a printf modifier of %#llx."""
>>>>>>> + return '0' if x == 0 else hex(x)
>>>>>>> +
>>>>>>> + def fixdesc(s: str) -> str:
>>>>>>> + """Fix formatting issue for the desc string."""
>>>>>>> + if s is None:
>>>>>>> + return None
>>>>>>> + return s.removesuffix('. ').removesuffix('. ').removesuffix('.').replace(
>>>>>>> + '\n', '\\n').replace('\"', '\\"').replace('\r', '\\r')
>>>>>>> +
>>>>>>> + def convert_aggr_mode(aggr_mode: str) -> str:
>>>>>>> + """Returns the aggr_mode_class enum value associated with the JSON string."""
>>>>>>> + if not aggr_mode:
>>>>>>> + return None
>>>>>>> + aggr_mode_to_enum = {
>>>>>>> + 'PerChip': '1',
>>>>>>> + 'PerCore': '2',
>>>>>>> + }
>>>>>>> + return aggr_mode_to_enum[aggr_mode]
>>>>>>> +
>>>>>>> + def lookup_msr(num: str) -> str:
>>>>>>> + """Converts the msr number, or first in a list to the appropriate event field."""
>>>>>>> + if not num:
>>>>>>> + return None
>>>>>>> + msrmap = {
>>>>>>> + 0x3F6: 'ldlat=',
>>>>>>> + 0x1A6: 'offcore_rsp=',
>>>>>>> + 0x1A7: 'offcore_rsp=',
>>>>>>> + 0x3F7: 'frontend=',
>>>>>>> + }
>>>>>>> + return msrmap[int(num.split(',', 1)[0], 0)]
>>>>>>> +
>>>>>>> + def real_event(name: str, event: str) -> str:
>>>>>>> + """Convert well known event names to an event string otherwise use the event argument."""
>>>>>>> + fixed = {
>>>>>>> + 'inst_retired.any': 'event=0xc0,period=2000003',
>>>>>>> + 'inst_retired.any_p': 'event=0xc0,period=2000003',
>>>>>>> + 'cpu_clk_unhalted.ref': 'event=0x0,umask=0x03,period=2000003',
>>>>>>> + 'cpu_clk_unhalted.thread': 'event=0x3c,period=2000003',
>>>>>>> + 'cpu_clk_unhalted.core': 'event=0x3c,period=2000003',
>>>>>>> + 'cpu_clk_unhalted.thread_any': 'event=0x3c,any=1,period=2000003',
>>>>>>> + }
>>>>>>> + if not name:
>>>>>>> + return None
>>>>>>> + if name.lower() in fixed:
>>>>>>> + return fixed[name.lower()]
>>>>>>> + return event
>>>>>>> +
>>>>>>> + def unit_to_pmu(unit: str) -> str:
>>>>>>> + """Convert a JSON Unit to Linux PMU name."""
>>>>>>> + if not unit:
>>>>>>> + return None
>>>>>>> + # Comment brought over from jevents.c:
>>>>>>> + # it's not realistic to keep adding these, we need something more scalable ...
>>>>>>> + table = {
>>>>>>> + 'CBO': 'uncore_cbox',
>>>>>>> + 'QPI LL': 'uncore_qpi',
>>>>>>> + 'SBO': 'uncore_sbox',
>>>>>>> + 'iMPH-U': 'uncore_arb',
>>>>>>> + 'CPU-M-CF': 'cpum_cf',
>>>>>>> + 'CPU-M-SF': 'cpum_sf',
>>>>>>> + 'UPI LL': 'uncore_upi',
>>>>>>> + 'hisi_sicl,cpa': 'hisi_sicl,cpa',
>>>>>>> + 'hisi_sccl,ddrc': 'hisi_sccl,ddrc',
>>>>>>> + 'hisi_sccl,hha': 'hisi_sccl,hha',
>>>>>>> + 'hisi_sccl,l3c': 'hisi_sccl,l3c',
>>>>>>> + 'imx8_ddr': 'imx8_ddr',
>>>>>>> + 'L3PMC': 'amd_l3',
>>>>>>> + 'DFPMC': 'amd_df',
>>>>>>> + 'cpu_core': 'cpu_core',
>>>>>>> + 'cpu_atom': 'cpu_atom',
>>>>>>> + }
>>>>>>> + return table[unit] if unit in table else f'uncore_{unit.lower()}'
>>>>>>> +
>>>>>>> + eventcode = 0
>>>>>>> + if 'EventCode' in dict:
>>>>>>> + eventcode = int(dict['EventCode'].split(',', 1)[0], 0)
>>>>>>> + if 'ExtSel' in dict:
>>>>>>> + eventcode |= int(dict['ExtSel']) << 8
>>>>>>> + configcode = int(dict['ConfigCode'], 0) if 'ConfigCode' in dict else None
>>>>>>> + self.name = dict['EventName'].lower() if 'EventName' in dict else None
>>>>>>> + self.compat = dict.get('Compat')
>>>>>>> + self.desc = fixdesc(dict.get('BriefDescription'))
>>>>>>> + self.long_desc = fixdesc(dict.get('PublicDescription'))
>>>>>>> + precise = dict.get('PEBS')
>>>>>>> + msr = lookup_msr(dict.get('MSRIndex'))
>>>>>>> + msrval = dict.get('MSRValue')
>>>>>>> + extra_desc = ''
>>>>>>> + if 'Data_LA' in dict:
>>>>>>> + extra_desc += ' Supports address when precise'
>>>>>>> + if 'Errata' in dict:
>>>>>>> + extra_desc += '.'
>>>>>>> + if 'Errata' in dict:
>>>>>>> + extra_desc += ' Spec update: ' + dict['Errata']
>>>>>>> + self.pmu = unit_to_pmu(dict.get('Unit'))
>>>>>>> + filter = dict.get('Filter')
>>>>>>> + self.unit = dict.get('ScaleUnit')
>>>>>>> + self.perpkg = dict.get('PerPkg')
>>>>>>> + self.aggr_mode = convert_aggr_mode(dict.get('AggregationMode'))
>>>>>>> + self.deprecated = dict.get('Deprecated')
>>>>>>> + self.metric_name = dict.get('MetricName')
>>>>>>> + self.metric_group = dict.get('MetricGroup')
>>>>>>> + self.metric_constraint = dict.get('MetricConstraint')
>>>>>>> + self.metric_expr = dict.get('MetricExpr')
>>>>>>> + if self.metric_expr:
>>>>>>> + self.metric_expr = self.metric_expr.replace('\\', '\\\\')
>>>>>>> + arch_std = dict.get('ArchStdEvent')
>>>>>>> + if precise and self.desc and not '(Precise Event)' in self.desc:
>>>>>>> + extra_desc += ' (Must be precise)' if precise == '2' else (' (Precise '
>>>>>>> + 'event)')
>>>>>>> + event = f'config={llx(configcode)}' if configcode is not None else f'event={llx(eventcode)}'
>>>>>>> + event_fields = [
>>>>>>> + ('AnyThread', 'any='),
>>>>>>> + ('PortMask', 'ch_mask='),
>>>>>>> + ('CounterMask', 'cmask='),
>>>>>>> + ('EdgeDetect', 'edge='),
>>>>>>> + ('FCMask', 'fc_mask='),
>>>>>>> + ('Invert', 'inv='),
>>>>>>> + ('SampleAfterValue', 'period='),
>>>>>>> + ('UMask', 'umask='),
>>>>>>> + ]
>>>>>>> + for key, value in event_fields:
>>>>>>> + if key in dict and dict[key] != '0':
>>>>>>> + event += ',' + value + dict[key]
>>>>>>> + if filter:
>>>>>>> + event += f',{filter}'
>>>>>>> + if msr:
>>>>>>> + event += f',{msr}{msrval}'
>>>>>>> + if self.desc and extra_desc:
>>>>>>> + self.desc += extra_desc
>>>>>>> + if self.long_desc and extra_desc:
>>>>>>> + self.long_desc += extra_desc
>>>>>>> + if self.pmu:
>>>>>>> + if self.desc and not self.desc.endswith('. '):
>>>>>>> + self.desc += '. '
>>>>>>> + self.desc = (self.desc if self.desc else '') + ('Unit: ' + self.pmu + ' ')
>>>>>>> + if arch_std and arch_std.lower() in arch_std_events:
>>>>>>> + event = arch_std_events[arch_std.lower()].event
>>>>>>> + # Copy from the architecture standard event to self for undefined fields.
>>>>>>> + for attr, value in arch_std_events[arch_std.lower()].__dict__.items():
>>>>>>> + if hasattr(self, attr) and not getattr(self, attr):
>>>>>>> + setattr(self, attr, value)
>>>>>>> +
>>>>>>> + self.event = real_event(self.name, event)
>>>>>>> +
>>>>>>> + def __repr__(self) -> str:
>>>>>>> + """String representation primarily for debugging."""
>>>>>>> + s = '{\n'
>>>>>>> + for attr, value in self.__dict__.items():
>>>>>>> + if value:
>>>>>>> + s += f'\t{attr} = {value},\n'
>>>>>>> + return s + '}'
>>>>>>> +
>>>>>>> + def ToCString(self, topic_local: str) -> str:
>>>>>>> + """Representation of the event as a C struct initializer."""
>>>>>>> +
>>>>>>> + def AttrString(attr: str, value: str) -> str:
>>>>>>> + return '\t.%s = \"%s\",\n' % (attr, value)
>>>>>>> +
>>>>>>> + def StrIfPresent(self, attr: str) -> str:
>>>>>>> + if not getattr(self, attr):
>>>>>>> + return ''
>>>>>>> + return AttrString(attr, getattr(self, attr))
>>>>>>> +
>>>>>>> + s = '{\n'
>>>>>>> + for attr in ['name', 'event']:
>>>>>>> + s += StrIfPresent(self, attr)
>>>>>>> + if self.desc is not None:
>>>>>>> + s += AttrString('desc', self.desc)
>>>>>>> + else:
>>>>>>> + s += AttrString('desc', '(null)')
>>>>>>> + s += StrIfPresent(self, 'compat')
>>>>>>> + s += f'\t.topic = "{topic_local}",\n'
>>>>>>> + for attr in [
>>>>>>> + 'long_desc', 'pmu', 'unit', 'perpkg', 'aggr_mode', 'metric_expr',
>>>>>>> + 'metric_name', 'metric_group', 'deprecated', 'metric_constraint'
>>>>>>> + ]:
>>>>>>> + s += StrIfPresent(self, attr)
>>>>>>> + s += '},\n'
>>>>>>> + return s
>>>>>>> +
>>>>>>> +
>>>>>>> +def read_json_events(path: str) -> list[JsonEvent]:
>>>>>>> + """Read json events from the specified file."""
>>>>>>> + return json.load(open(path), object_hook=lambda d: JsonEvent(d))
>>>>>>> +
>>>>>>> +def preprocess_arch_std_files(archpath: str) -> None:
>>>>>>> + """Read in all architecture standard events."""
>>>>>>> + global arch_std_events
>>>>>>> + for item in os.scandir(archpath):
>>>>>>> + if item.is_file() and item.name.endswith('.json'):
>>>>>>> + for event in read_json_events(item.path):
>>>>>>> + if event.name:
>>>>>>> + arch_std_events[event.name.lower()] = event
>>>>>>> +
>>>>>>> +
>>>>>>> +def print_events_table_prefix(tblname: str) -> None:
>>>>>>> + """Called when a new events table is started."""
>>>>>>> + global close_table
>>>>>>> + if close_table:
>>>>>>> + raise IOError('Printing table prefix but last table has no suffix')
>>>>>>> + args.output_file.write('static const struct pmu_event %s[] = {\n' % tblname)
>>>>>>> + close_table = True
>>>>>>> +
>>>>>>> +
>>>>>>> +def print_events_table_entries(item: os.DirEntry, topic: str) -> None:
>>>>>>> + """Create contents of an events table."""
>>>>>>> + if not close_table:
>>>>>>> + raise IOError('Table entries missing prefix')
>>>>>>> + for event in read_json_events(item.path):
>>>>>>> + args.output_file.write(event.ToCString(topic))
>>>>>>> +
>>>>>>> +def print_events_table_suffix() -> None:
>>>>>>> + """Optionally close events table."""
>>>>>>> + global close_table
>>>>>>> + if close_table:
>>>>>>> + args.output_file.write("""{
>>>>>>> +\t.name = 0,
>>>>>>> +\t.event = 0,
>>>>>>> +\t.desc = 0,
>>>>>>> +},
>>>>>>> +};
>>>>>>> +""")
>>>>>>> + close_table = False
>>>>>>> +
>>>>>>> +def process_one_file(parents: list[str], item: os.DirEntry) -> None:
>>>>>>> + """Process a JSON file during the main walk."""
>>>>>>> + global sys_event_tables
>>>>>>> +
>>>>>>> + def get_topic(topic: str) -> str:
>>>>>>> + return topic.removesuffix('.json').replace('-', ' ')
>>>>>>> +
>>>>>>> + def is_leaf_dir(path: str) -> bool:
>>>>>>> + for item in os.scandir(path):
>>>>>>> + if item.is_dir():
>>>>>>> + return False
>>>>>>> + return True
>>>>>>> +
>>>>>>> + # model directory, reset topic
>>>>>>> + if item.is_dir() and is_leaf_dir(item.path):
>>>>>>> + print_events_table_suffix()
>>>>>>> +
>>>>>>> + tblname = file_name_to_table_name(parents, item.name)
>>>>>>> + if item.name == 'sys':
>>>>>>> + sys_event_tables.append(tblname)
>>>>>>> + print_events_table_prefix(tblname)
>>>>>>> + return
>>>>>>> +
>>>>>>> + # base dir or too deep
>>>>>>> + level = len(parents)
>>>>>>> + if level == 0 or level > 4:
>>>>>>> + return
>>>>>>> +
>>>>>>> + # Ignore other directories. If the file name does not have a .json
>>>>>>> + # extension, ignore it. It could be a readme.txt for instance.
>>>>>>> + if not item.is_file() or not item.name.endswith('.json'):
>>>>>>> + return
>>>>>>> +
>>>>>>> + print_events_table_entries(item, get_topic(item.name))
>>>>>>> +
>>>>>>> +
>>>>>>> +def print_mapping_table() -> None:
>>>>>>> + """Read the mapfile and generate the struct from cpuid string to event table."""
>>>>>>> + table = csv.reader(open(f'{args.starting_dir}/{args.arch}/mapfile.csv'))
>>>>>>> + args.output_file.write('const struct pmu_events_map pmu_events_map[] = {\n')
>>>>>>> + first = True
>>>>>>> + for row in table:
>>>>>>> + # Skip the first row or any row beginning with #.
>>>>>>> + if not first and len(row) > 0 and not row[0].startswith('#'):
>>>>>>> + tblname = file_name_to_table_name([], row[2].replace('/', '_'))
>>>>>>> + args.output_file.write("""{
>>>>>>> +\t.cpuid = \"%s\",
>>>>>>> +\t.version = \"%s\",
>>>>>>> +\t.type = \"%s\",
>>>>>>> +\t.table = %s
>>>>>>> +},
>>>>>>> +""" % (row[0].replace('\\', '\\\\'), row[1], row[3], tblname))
>>>>>>> + first = False
>>>>>>> +
>>>>>>> + args.output_file.write("""{
>>>>>>> +\t.cpuid = "testcpu",
>>>>>>> +\t.version = "v1",
>>>>>>> +\t.type = "core",
>>>>>>> +\t.table = pme_test_soc_cpu,
>>>>>>> +},
>>>>>>> +{
>>>>>>> +\t.cpuid = 0,
>>>>>>> +\t.version = 0,
>>>>>>> +\t.type = 0,
>>>>>>> +\t.table = 0,
>>>>>>> +},
>>>>>>> +};
>>>>>>> +""")
>>>>>>> +
>>>>>>> +
>>>>>>> +def print_system_mapping_table() -> None:
>>>>>>> + """C struct mapping table array for tables from /sys directories."""
>>>>>>> + args.output_file.write(
>>>>>>> + '\nconst struct pmu_sys_events pmu_sys_event_tables[] = {\n')
>>>>>>> + for tblname in sys_event_tables:
>>>>>>> + args.output_file.write("""\t{
>>>>>>> +\t\t.table = %s,
>>>>>>> +\t\t.name = \"%s\",
>>>>>>> +\t},
>>>>>>> +""" % (tblname, tblname))
>>>>>>> + args.output_file.write("""\t{
>>>>>>> +\t\t.table = 0
>>>>>>> +\t},
>>>>>>> +};
>>>>>>> +""")
>>>>>>> +
>>>>>>> +
>>>>>>> +def main() -> None:
>>>>>>> + global args
>>>>>>> +
>>>>>>> + def dir_path(path: str) -> str:
>>>>>>> + """Validate path is a directory for argparse."""
>>>>>>> + if os.path.isdir(path):
>>>>>>> + return path
>>>>>>> + else:
>>>>>>> + raise argparse.ArgumentTypeError(f'\'{path}\' is not a valid directory')
>>>>>>> +
>>>>>>> + def ftw(path: str, parents: list[str],
>>>>>>> + action: Callable[[list[str], os.DirEntry], None]) -> None:
>>>>>>> + """Replicate the directory/file walking behavior of C's file tree walk."""
>>>>>>> + for item in os.scandir(path):
>>>>>>> + action(parents, item)
>>>>>>> + if item.is_dir():
>>>>>>> + ftw(item.path, parents + [item.name], action)
>>>>>>> +
>>>>>>> + ap = argparse.ArgumentParser()
>>>>>>> + ap.add_argument('arch', help='Architecture name like x86')
>>>>>>> + ap.add_argument(
>>>>>>> + 'starting_dir',
>>>>>>> + type=dir_path,
>>>>>>> + help='Root of tree containing architecture directories containing json files'
>>>>>>> + )
>>>>>>> + ap.add_argument(
>>>>>>> + 'output_file', type=argparse.FileType('w'), nargs='?', default=sys.stdout)
>>>>>>> + args = ap.parse_args()
>>>>>>> +
>>>>>>> + args.output_file.write("#include \"pmu-events/pmu-events.h\"\n")
>>>>>>> + for path in [args.arch, 'test']:
>>>>>>> + arch_path = f'{args.starting_dir}/{path}'
>>>>>>> + if not os.path.isdir(arch_path):
>>>>>>> + raise IOError(f'Missing architecture directory in \'{arch_path}\'')
>>>>>>> + preprocess_arch_std_files(arch_path)
>>>>>>> + ftw(arch_path, [], process_one_file)
>>>>>>> + print_events_table_suffix()
>>>>>>> +
>>>>>>> + print_mapping_table()
>>>>>>> + print_system_mapping_table()
>>>>>>> +
>>>>>>> +
>>>>>>> +if __name__ == '__main__':
>>>>>>> + main()
>>>>>>
>>>>>> --
>>>>>> Zhengjun Xing
>>>>
>>>> --
>>>> Zhengjun Xing
>>
>> --
>> Zhengjun Xing
--
Zhengjun Xing
Powered by blists - more mailing lists