Message-ID: <20101029104858.GA1800@linux.vnet.ibm.com>
Date:	Fri, 29 Oct 2010 16:18:58 +0530
From:	Srikar Dronamraju <srikar@...ux.vnet.ibm.com>
To:	Christoph Hellwig <hch@...radead.org>
Cc:	Peter Zijlstra <peterz@...radead.org>, Ingo Molnar <mingo@...e.hu>,
	Steven Rostedt <rostedt@...dmis.org>,
	Randy Dunlap <rdunlap@...otime.net>,
	Arnaldo Carvalho de Melo <acme@...radead.org>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Naren A Devaiah <naren.devaiah@...ibm.com>,
	Ananth N Mavinakayanahalli <ananth@...ibm.com>,
	Masami Hiramatsu <masami.hiramatsu.pt@...achi.com>,
	Oleg Nesterov <oleg@...hat.com>,
	Mark Wielaard <mjw@...hat.com>,
	Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
	LKML <linux-kernel@...r.kernel.org>,
	Jim Keniston <jkenisto@...ux.vnet.ibm.com>,
	Frederic Weisbecker <fweisbec@...il.com>,
	"Frank Ch. Eigler" <fche@...hat.com>,
	Andrew Morton <akpm@...ux-foundation.org>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>
Subject: Re: [PATCHv11 2.6.36-rc2-tip 0/15]  0: Uprobes Patches


> It's been a while since the last posting.  Did you make any progress on
> uprobes, especially allowing to define probes based on a file name?
> 

Thanks for checking, I discussed with Peter offline and ironed out most
of the issues. I am thankful for Peter for all the suggestions

I am still getting the inode based uprobes to shape. 
Here is the brief summary of the discussion.

Significant differences from the previous patchset are:

- All probes would be maintained in a global rbtree sorted by inode and
  offset.
- There can be one or more consumers per probe. With each consumer there
  will be one handler and one (optional) filter.
- Filter restricts the processes/tasks that the handler is active.
- uprobe structure is dynamically created when the first consumer
  registers to the probe. It gets deallocated when all consumers
  unregisters from the probe.
- While registering a probe, we walk thro the list of vmas that are
  mapped to the inode, check if the consumer wants to probe the task
  corresponding to the vma and inserts the breakpoint.
- unregistering a probe also does something similar except for deleting
  the probe.
- There will be a hook in mmap/unmap to install probes as and when the
  vma gets loaded into process address space. This hook would walk thro
  the tree of probes for that inode and for each probe, walk thro the
  list of consumers and insert/delete breakpoints accordingly.
- There will be a hook in fork to install probes in newly created
  processes. This hook would walk thro the tree of probes for that inode
  and for each probe, walk thro the list of consumers and insert/delete
  breakpoints accordingly.
- Slots will still hang-out of mm_struct.
- Instead of the per-probe slot, we would have to use a per-thread slot.
  (This slot is for single stepping out of line). On every probehit, the
  slot has to be refreshed with the correct contents. 
- Since probe information is stored as inode:offset, probe
  identification on a breakpoint hit can only happen in task context.

Current issues: Given a vma; finding all tasks that have this vma
mapped. The current solution seems to walk thro all tasks in the system
and check if each task's mm field. But this could be inefficient
especially if the vma is part of all processes like libc etc.
I dont think we can use mm->owner to walk thro all the tasks that map to
this mm_struct.

Slot allocation delays: Using a per-thread slot instead of a per-probe
slot leads to a 3.5X degradation in performance. If we need to use
per-probe slots, we would have to have maintain metadata, i.e the slot
to virtual address mapping per process. However Peter prefers that we
reduce such metadata.

If a sigtrap occurs not because of a uprobe, we queue the signal only in
the task context. This is because we cannot identify the cause for trap
in interrupt context. If we use per-probe slots, we could avoid this.

Uprobes may not be able to trace a long-time running process if its
mapped a file that got updated after the process started. This assumes
the probes were placed after the process was started.

To cover the race where a breakpoint is hit while the probe is being
deleted, Paul has agreed to enhance srcu to work in interrupt context.

>From a perf/ftrace perspective:

Assuming 3551 as pid 
0x0000000000446420 as the vaddr of the probe for zfree
0x0000000000046420 as the offset of zfree from start of text address.
130904 as the inode number for the /bin/zsh

ftrace interface:
Previously:

# echo 'p 3551:0x0000000000446420 %ip %ax' > uprobe_events
# cat uprobe_events
p:uprobes/p_3551_0x0000000000446420 3551:0x0000000000446420 %ip=%ip %ax=%ax

Now:

# echo 'p /path-to-/file:0x0000000000046420 %ip %ax' > uprobe_events
# cat uprobe_events
p:uprobes/p_file_0x0000000000446420 130904:0x0000000000046420 %ip=%ip %ax=%ax


perf interface:
Previously:

# perf probe -p 3551 zfree@zsh
Added new event:
  probe_3551:zfree                       (on 0x446420)
# perf probe --list
probe_3551:zfree                       (on 3551:0x0000000000446420)

Now: 

# perf probe zfree@zsh
Added new event:
  probe_zsh:zfree                      (on 130904:0x46420)
# perf probe --list
probe_zsh:zfree                       (on 130904:0x0000000000046420)

JFYI: v11 patchset code that I last posted to LKML is available at:
git://git.kernel.org/pub/scm/linux/kernel/git/srikar/linux-uprobes.git
 branch : uprobes_v11_patchset 

Peter knows the details/trade-offs of both the designs.(v11 and the
inode based probes)

-- 
Thanks and Regards
Srikar
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists