linux-kernel - Re: [PATCH v4 4/6] perf scripts python: Add trace end processing and PRODUCT and CATEGORIES information

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <ZLlOMR9UjRFnCfS+@kernel.org>
Date:   Thu, 20 Jul 2023 12:09:37 -0300
From:   Arnaldo Carvalho de Melo <acme@...nel.org>
To:     Anup Sharma <anupnewsmail@...il.com>
Cc:     Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...hat.com>,
        Mark Rutland <mark.rutland@....com>,
        Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
        Jiri Olsa <jolsa@...nel.org>,
        Namhyung Kim <namhyung@...nel.org>,
        Ian Rogers <irogers@...gle.com>,
        Adrian Hunter <adrian.hunter@...el.com>,
        linux-perf-users@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH v4 4/6] perf scripts python: Add trace end processing and
 PRODUCT and CATEGORIES information

Em Wed, Jul 19, 2023 at 04:20:08AM +0530, Anup Sharma escreveu:
> The final output will now be presented in JSON format following the Gecko
> profile structure. Additionally, the inclusion of PRODUCT allows easy retrieval
> of header information for UI.
> 
> Furthermore, CATEGORIES have been introduced to enable customization of
> kernel and user colors using input arguments. To facilitate this functionality,
> an argparse-based parser has been implemented.
> 
> Note that the implementation of threads will be addressed in subsequent commits."

It is failing here:

  [root@...e ~]# perf script firefox-gecko-converter.py
  Traceback (most recent call last):
    File "/var/home/acme/libexec/perf-core/scripts/python/firefox-gecko-converter.py", line 202, in trace_end
      "threads": threads,
  NameError: name 'threads' is not defined. Did you mean: 'Thread'?
  Fatal Python error: handler_call_die: problem in Python trace event handler
  Python runtime state: initialized
  
  Current thread 0x00007fde43f98f40 (most recent call first):
    <no Python frame>
  
  Extension modules: perf_trace_context (total: 1)
  Aborted (core dumped)
  [root@...e ~]#

We need to be able to test it at each changeset, not just at the end of
the patchkit, so please reorganize this so that running:

  perf script firefox-gecko-converter.py

After each patch works.

We need this to bisect problems, etc.

- Arnaldo
 
> Signed-off-by: Anup Sharma <anupnewsmail@...il.com>
> ---
>  .../scripts/python/firefox-gecko-converter.py | 64 ++++++++++++++++++-
>  1 file changed, 63 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/perf/scripts/python/firefox-gecko-converter.py b/tools/perf/scripts/python/firefox-gecko-converter.py
> index d9b1ec18997a..a0218e2245f2 100644
> --- a/tools/perf/scripts/python/firefox-gecko-converter.py
> +++ b/tools/perf/scripts/python/firefox-gecko-converter.py
> @@ -11,6 +11,8 @@
>  
>  import os
>  import sys
> +import json
> +import argparse
>  from dataclasses import dataclass, field
>  from typing import List, Dict, Optional, NamedTuple, Set, Tuple, Any
>  
> @@ -30,6 +32,13 @@ Milliseconds = float
>  # start_time is intialiazed only once for the all event traces.
>  start_time = None
>  
> +# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/profile.js#L425
> +# Follow Brendan Gregg's Flamegraph convention: orange for kernel and yellow for user space by default.
> +CATEGORIES = None
> +
> +# The product name is used by the profiler UI to show the Operating system and Processor.
> +PRODUCT = os.popen('uname -op').read().strip()
> +
>  # https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L156
>  class Frame(NamedTuple):
>  	string_id: StringID
> @@ -172,4 +181,57 @@ def process_event(param_dict: Dict) -> None:
>  # Trace_end runs at the end and will be used to aggregate
>  # the data into the final json object and print it out to stdout.
>  def trace_end() -> None:
> -	pass
> +	# Schema: https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L305
> +	gecko_profile_with_meta = {
> +		"meta": {
> +			"interval": 1,
> +			"processType": 0,
> +			"product": PRODUCT,
> +			"stackwalk": 1,
> +			"debug": 0,
> +			"gcpoison": 0,
> +			"asyncstack": 1,
> +			"startTime": start_time,
> +			"shutdownTime": None,
> +			"version": 24,
> +			"presymbolicated": True,
> +			"categories": CATEGORIES,
> +			"markerSchema": [],
> +			},
> +		"libs": [],
> +		"threads": threads,
> +		"processes": [],
> +		"pausedRanges": [],
> +	}
> +	json.dump(gecko_profile_with_meta, sys.stdout, indent=2)
> +
> +def main() -> None:
> +	global CATEGORIES
> +	parser = argparse.ArgumentParser(description="Convert perf.data to Firefox\'s Gecko Profile format")
> +
> +	# Add the command-line options
> +	# Colors must be defined according to this:
> +	# https://github.com/firefox-devtools/profiler/blob/50124adbfa488adba6e2674a8f2618cf34b59cd2/res/css/categories.css
> +	parser.add_argument('--user-color', default='yellow', help='Color for the User category')
> +	parser.add_argument('--kernel-color', default='orange', help='Color for the Kernel category')
> +	# Parse the command-line arguments
> +	args = parser.parse_args()
> +	# Access the values provided by the user
> +	user_color = args.user_color
> +	kernel_color = args.kernel_color
> +
> +	CATEGORIES = [
> +		{
> +			"name": 'User',
> +			"color": user_color,
> +			"subcategories": ['Other']
> +		},
> +		{
> +			"name": 'Kernel',
> +			"color": kernel_color,
> +			"subcategories": ['Other']
> +		},
> +	]
> +
> +if __name__ == '__main__':
> +    main()
> -- 
> 2.34.1
> 

-- 

- Arnaldo