linux-kernel - Re: Is it a workqueue related issue in 2.6.37 (Was: Re: [libvirt] blkio cgroup [solved])

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4D67BFAB.1090306@in-telegence.net>
Date:	Fri, 25 Feb 2011 15:41:47 +0100
From:	Dominik Klein <dk@...telegence.net>
To:	Tejun Heo <tj@...nel.org>
CC:	Vivek Goyal <vgoyal@...hat.com>,
	linux kernel mailing list <linux-kernel@...r.kernel.org>,
	libvir-list@...hat.com
Subject: Re: Is it a workqueue related issue in 2.6.37 (Was: Re: [libvirt]
 blkio cgroup [solved])

On 02/25/2011 02:18 PM, Tejun Heo wrote:
> Hello,
> 
> On Fri, Feb 25, 2011 at 12:46:16PM +0100, Dominik Klein wrote:
>> With 2.6.37 (also tried .1 and .2) it does not work but end up like I
>> documented. With 2.6.38-rc1, it does work. With deadline scheduler, it
>> also works in 2.6.37.
> 
> Okay, here's the problematic part.
> 
>           <idle>-0     [013]  1640.975562: workqueue_queue_work: work struct=ffff88080f14f270 function=blk_throtl_work workqueue=ffff88102c8fc700 req_cpu=13 cpu=13
>           <idle>-0     [013]  1640.975564: workqueue_activate_work: work struct ffff88080f14f270
>            <...>-477   [013]  1640.975574: workqueue_execute_start: work struct ffff88080f14f270: function blk_throtl_work
> 	  <idle>-0     [013]  1641.087450: workqueue_queue_work: work struct=ffff88080f14f270 function=blk_throtl_work workqueue=ffff88102c8fc700 req_cpu=13 cpu=13
> 		     
> The workqueue is per-cpu, so we only need to follow cpu=13 cases.
> @1640, blk_throtl_work() is queued, activated and starts executing but
> never finishes.  The same work item is never executed more than once
> at the same on the same CPU, so when the next work item is queued, it
> doesn't get activated until the previous execution is complete.
> 
> The next thing to do would be finding out why blk_throtl_work() isn't
> finishing.  sysrq-t or /proc/PID/stack should show us where it's
> stalled.
> 
> Thanks.
> 


See attached logs of another run.

sysctl -w kernel.sysrq=1


echo blk > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/block/sdb/trace/enable
echo workqueue_queue_work >> /sys/kernel/debug/tracing/set_event
echo workqueue_activate_work >> /sys/kernel/debug/tracing/set_event
echo workqueue_execute_start >> /sys/kernel/debug/tracing/set_event
echo workqueue_execute_end >> /sys/kernel/debug/tracing/set_event

That makes attachment trace_pipe5.gz

echo 8 > /proc/sysrq-trigger
echo t > /proc/sysrq-trigger

That makes attachment console.gz

hth
Dominik

Download attachment "console.gz" of type "application/gzip" (45434 bytes)

Download attachment "trace_pipe5.gz" of type "application/gzip" (100324 bytes)