linux-kernel - Re: [RFC] Adding Support for Firefox's Gecko Profile Format

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Tue, 4 Jul 2023 20:08:11 +0530
From:   Anup Sharma <anupnewsmail@...il.com>
To:     Adrian Hunter <adrian.hunter@...el.com>
Cc:     Namhyung Kim <namhyung@...nel.org>,
        Anup Sharma <anupnewsmail@...il.com>,
        Ian Rogers <irogers@...gle.com>,
        linux-perf-users@...r.kernel.org,
        Peter Zijlstra <peterz@...radead.org>,
        Ingo Molnar <mingo@...hat.com>,
        Arnaldo Carvalho de Melo <acme@...nel.org>,
        Mark Rutland <mark.rutland@....com>,
        Alexander Shishkin <alexander.shishkin@...ux.intel.com>,
        Jiri Olsa <jolsa@...nel.org>, linux-kernel@...r.kernel.org
Subject: Re: [RFC] Adding Support for Firefox's Gecko Profile Format

On Thu, Jun 29, 2023 at 09:35:22AM +0300, Adrian Hunter wrote:
> On 29/06/23 09:26, Namhyung Kim wrote:
> > Hi Anup,
> > 
> > On Wed, Jun 28, 2023 at 1:11 PM Anup Sharma <anupnewsmail@...il.com> wrote:
> >>
> >> On Wed, Jun 14, 2023 at 01:17:52PM -0700, Ian Rogers wrote:
> >>> On Wed, Jun 14, 2023 at 1:14 PM Anup Sharma <anupnewsmail@...il.com> wrote:
> >>>>
> >>>> On Thu, Jun 08, 2023 at 06:23:49PM +0530, Anup Sharma wrote:
> >>>> I wanted to take a moment to provide you with an update on the progress
> >>>> of our Firefox Gecko converter work. While I must emphasize that the code
> >>>> I'm sharing is not the final version, I wanted to share the advancements
> >>>> I have made thus far.
> >>>>
> >>>> This script can generate a JSON format from the output of the "perf script" command.
> >>>> I attempted to load this JSON file into profile.firefox.com, and although it
> >>>> successfully loaded, the call tree are not visible. I'm certain this issue
> >>>> is related to the format of the JSON file or if there is another underlying
> >>>> cause. I will continue investigating to determine the cause of this problem.
> >>>
> >>> Great Anup, progress can be frustrating slow at first but it is a good
> >>> milestone to be generating output and having firefox profiler consume
> >>> it. You can open up the JavaScript console for the firefox profiler
> >>> and it will give some debug output. This should hopefully help you
> >>> make progress.
> >>
> >> This week I tried playing with perf-script-python, the first challenge was
> >> figuring out how to parse the data coming from the process_events.
> >> Understanding the structure and extracting the necessary information
> >> proved to be more complex than anticipated. This required me to spend
> >> a significant amount of time researching and experimenting with different
> >> parsing techniques.
> > 
> > I'm not sure what kind of parsing you meant.  IIRC the sample data is
> > in a dict and all the information should be available there.  Maybe we
> > missed some new sample data types though.
> 
> Most data is there.  There are existing scripts in
> tools/perf/scripts/python/ for reference.
> 
> There is also the dlfilter API:
> 
> https://perf.wiki.kernel.org/index.php/Latest_Manual_Page_of_perf-dlfilter.1
>
Hello everyone,

I hope you're all doing well. I have been working on the using process_events function and have made
progress in developing the converter script. Currently, I am in the testing phase. However, I
have encountered a problem after performing some recent compilations. I am no longer receiving
the complete callchains in the output as I used to.
For Example, when I ran the command perf script -F +pid, the output would include detailed
information like the following example:

perf-exec  132554/132554  171854.674582:        356 cycles:P: 
	ffffffff8fab3fc6 arch_static_branch+0x6 (inlined)
	ffffffff8fab3fc6 static_key_false+0x6 (inlined)
	ffffffff8fab3fc6 native_write_msr+0x6 (/lib/modules/6.4.0-rc1/build/vmlinux)
	ffffffff8fa12ca5 intel_pmu_enable_all+0x15 (/lib/modules/6.4.0-rc1/build/vmlinux)

However, in my current situation, the output is limited to the following:

perf  108107/108107  [000] 67650.031827:          1 cycles:P:  
ffffffff83ab3fc4 native_write_msr+0x4 (/lib/modules/6.4.0-rc1/build/vmlinux)
perf  108107/108107  [000] 67650.031832:          1 cycles:P:
ffffffff83ab3fc4 native_write_msr+0x4 (/lib/modules/6.4.0-rc1/build/vmlinux)

It seems to be issue in perf record. I would appreciate any suggestions or assistance
in resolving this issue. Thank you all for your help.

Regards,
Anup
> > 
> >>
> >> The second challenge revolved around the usage of event hooks provided with the perf script
> >> python event handlers. I found myself deliberating between two approaches. The first
> >> approach involved creating custom functions that would be called using the event
> >> handlers. These functions would then save the data in an organized format within
> >> globally created data structures. The alternative approach was to write the entire
> >> logic inside the event handlers themselves.
> >>
> >> Additionally, I contemplated whether it would be more suitable to handle the creation of
> >> a Gecko format for JSON and the profile format within the same script or to separate
> >> them into different scripts.
> >>
> >> I will discuss this points during tomorrow's office hour.
> >>
> >> However, I have gained a deeper understanding of the problem at hand and will use this
> >> knowledge to make more informed decisions and progress more effectively in the coming weeks.
> > 
> > Sounds like you did a good amount of research on how perf script
> > works.  We can talk about the details in the meeting.
> > 
> > Thanks,
> > Namhyung
>