Message-ID: <20170612212005.1bf8c44d@gandalf.local.home>
Date:   Mon, 12 Jun 2017 21:20:05 -0400
From:   Steven Rostedt <rostedt@...dmis.org>
To:     Will Hawkins <whh8b@...ginia.edu>
Cc:     LKML <linux-kernel@...r.kernel.org>
Subject: Re: Ftrace vs perf user page fault statistics differences

On Mon, 12 Jun 2017 20:20:42 -0400
Will Hawkins <whh8b@...ginia.edu> wrote:

> Dear Mr. Rostedt and Kernel community,
> 
> I hope that this is an appropriate place to ask this question, Please
> forgive me for wasting your time if it is not. I've searched for
> answers to this question on LKML and the "net" before asking. I
> apologize if I already missed the question and answer somewhere else.

No no, this is the correct forum.

> 
> I was doing a gut check of my understanding of how the Kernel pages in
> a binary program for execution. I created a very small program that is
> two pages. The entry point to the program is on page a and the code
> simply jumps to the second page (b) for its termination.
> 
> I compiled the program and dropped the kernel's memory caches [1].
> Then I ran the program under perf:
> 
> perf record --call-graph fp -e page-faults ../one_page_play/page

Can you supply this program. So I can see exactly what it does?

> 
> I looked at the results:
> 
> perf report
> 
> and the results were as I expected. There were two page faults for
> loading the code into memory and a page fault to
> copy_user_enhanced_fast_string invoked by execve's implementation when
> loading the binary.

What does perf script show you?

> 
> I decided to run the application under ftrace just for fun. I wanted
> an excuse to learn more about it and this seemed like the perfect
> chance. I used the incredible trace-cmd suite for the actual
> incantation of ftrace. I won't include the actual incantations here
> because I used many of them while digging around.

what events did you enable?

> 
> The results are both expected and unexpected. I see output like this:
> 
> Event: page_fault_user:0x4000e0
> 
> which indicates that there is a page fault at the program's entry
> point (and matches what I saw with the perf output). I have another
> similar entry that confirms the other expected page fault when loading
> the second page of the test application.
> 
> However, I also see entries like this:
> 
> Event: page_fault_user:0x7f4f590934c4 (1)

This could very well be the dynamic linker.

> 
> The addresses of the faults I see that match that pattern are not
> loaded into the application binary. What I discovered as I
> investigated, is that those page faults seem to occur when the kernel
> is attempting to record the output of stack traces, etc.

Are you sure? What does ldd give you of your program?

> 
> After thinking through this, I came up with the following hypothesis
> which is the crux of this email:
> 
> Ftrace's act of recording the traces that I requested to its ring
> buffer generated page faults of their own. These page faults are
> generated on behalf of the traced program and get reported in the
> results.

There are no page faults that happen by the ftrace ring buffer that
would be associated with the program.

> 
> If that is correct/reasonable, it explains the differences between
> what perf is reporting and what ftrace is reporting and I am happy.
> If, however, that is a bogus conclusion, please help me understand
> what is going on.

I'll need to know more about exactly what you are doing to help out.

> 
> I know that everyone who is on this email is incredibly busy and has
> much to do. I hope that I've included enough information to make it
> possible for you experts to advise, but not included too much to waste
> your time.
> 
> If you have the time or interest in answering, I would love to hear
> your responses. Please CC me directly on all responses.

No worries about Ccing you directly. LKML gets over 600 emails a day.
Nobody reads it all. If someone isn't Cc'd directly, they will most
likely never respond.

> 
> Thanks again for your time!

No problem.

-- Steve

> 
> Will
> 
> [1] I used echo 3 > /proc/sys/vm/drop_caches to accomplish this and
> issued it between every run. It may have been overkill, but I did it
> anyway.

Powered by blists - more mailing lists