linux-kernel - Re: Is it a workqueue related issue in 2.6.37 (Was: Re: [libvirt] blkio cgroup [solved])

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 25 Feb 2011 14:53:08 -0500
From:	Steven Rostedt <rostedt@...dmis.org>
To:	Dominik Klein <dk@...telegence.net>
Cc:	Tejun Heo <tj@...nel.org>, Vivek Goyal <vgoyal@...hat.com>,
	linux kernel mailing list <linux-kernel@...r.kernel.org>,
	libvir-list@...hat.com
Subject: Re: Is it a workqueue related issue in 2.6.37 (Was: Re: [libvirt]
 blkio cgroup [solved])

On Fri, Feb 25, 2011 at 08:24:15AM +0100, Dominik Klein wrote:
> > Maybe watching what the workqueue is doing using the following trace
> > events could be helpful.
> > 
> > # grep workqueue /sys/kernel/debug/tracing/available_events
> > workqueue:workqueue_insertion
> > workqueue:workqueue_execution
> > workqueue:workqueue_creation
> > workqueue:workqueue_destruction
> 
> Since I've never done this before, I will tell you how I captured the
> trace so you know what I did and can see whether I did something wrong
> and can correct me if necessary.
> 
> echo blk > /sys/kernel/debug/tracing/current_tracer
> echo 1 > /sys/block/sdb/trace/enable
> echo workqueue_queue_work >> /sys/kernel/debug/tracing/set_event
> echo workqueue_activate_work >> /sys/kernel/debug/tracing/set_event
> echo workqueue_execute_start >> /sys/kernel/debug/tracing/set_event
> echo workqueue_execute_end >> /sys/kernel/debug/tracing/set_event
> cat /sys/kernel/debug/tracing/trace_pipe

Just an FYI,

If you download trace-cmd from:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/trace-cmd.git

You could do the following:

# trace-cmd record -p blk -e workqueue_queue_work -e workqueue_activate_work -e workqueue_execute_start -e workqueue_execute_end <cmd>

Where <cmd> could be 
sh -c "echo 1 > /sys/block/sdb/trace/enable; sleep 10"

This will create a trace.dat file that you can read with 
trace-cmd report

Another way, if you want to do more than just that echo is to use:

# trace-cmd start -p blk -e workqueue_queue_work -e workqueue_activate_work -e workqueue_execute_start -e workqueue_execute_end
# echo 1 > /sys/block/sdb/trace/enable
# <do anyting you want>
# trace-cmd stop
# trace-cmd extract

The start just enable ftrace (like you did with the echos).
The extract will create the trace.dat file from the ftrace ring buffers.
You could alse just cat trace_pipe, which would do the same, or you
could do the echos, and then the trace-cmd stop and extract to get the
file. It's your choice ;)

You can then gzip the trace.dat file and send it to others that can read
it as well, as all the information needed to read the trace is recorded
in the file.

You could even send the data over the network instead of writing the
trace.dat locally.

On another box:

$ trace-cmd listen -p 12345

Then on the target:

# trace-cmd record -N host:12345 ...

This will send the data to the listening host and the file will be
created on the host side.

One added benefit of having the trace.dat file. kernelshark can read it.

-- Steve



> 
> And that output i gzip'd.
> 
> This was taken with 2.6.37 plus the patch you mentioned on a Dell R815
> with 2 12 core AMD Opteron 6174 CPUs. If you need any more information,
> please let me know.
> 


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/