linux-kernel - Re: corruption causing crash in __queue

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <5672D810.2010404@kyup.com>
Date:	Thu, 17 Dec 2015 17:43:12 +0200
From:	Nikolay Borisov <kernel@...p.com>
To:	Tejun Heo <tj@...nel.org>
Cc:	Mike Snitzer <snitzer@...hat.com>,
	"Linux-Kernel@...r. Kernel. Org" <linux-kernel@...r.kernel.org>,
	SiteGround Operations <operations@...eground.com>,
	Alasdair Kergon <agk@...hat.com>,
	device-mapper development <dm-devel@...hat.com>
Subject: Re: corruption causing crash in __queue_work



On 12/17/2015 05:33 PM, Tejun Heo wrote:
> Hello, Nikolay.
> 
> On Thu, Dec 17, 2015 at 12:46:10PM +0200, Nikolay Borisov wrote:
>> diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c
>> index 493c38e08bd2..ccbbf7823cf3 100644
>> --- a/drivers/md/dm-thin.c
>> +++ b/drivers/md/dm-thin.c
>> @@ -3506,8 +3506,8 @@ static void pool_postsuspend(struct dm_target *ti)
>>         struct pool_c *pt = ti->private;
>>         struct pool *pool = pt->pool;
>>
>> -       cancel_delayed_work(&pool->waker);
>> -       cancel_delayed_work(&pool->no_space_timeout);
>> +       cancel_delayed_work_sync(&pool->waker);
>> +       cancel_delayed_work_sync(&pool->no_space_timeout);
>>         flush_workqueue(pool->wq);
>>         (void) commit(pool);
>>  }
>>
>> And this seems to have resolved the crashes. For the past 24 hours I
>> haven't seen a single server crash whereas before at least 3-5 servers
>> would crash.
> 
> So, that's an obvious bug on dm-thin side.


Mike if you are ok with this I will submit a proper patch ?
> 
>> Given that, it seems like a race condition between destroying the
>> workqueue from dm-thin and cancelling all the delayed work.
>>
>> Tejun, I've looked at cancel_delayed_work/cancel_delayed_work_sync and
>> they both call try_to_grab_pending and then their function diverges. Is
>> it possible that there is a latent race condition between canceling the
>> delayed work and the subsequent re-scheduling of the work item?
> 
> It's just the wrong variant being used.  cancel_delayed_work() doesn't
> guarantee that the work item isn't running on return.  If the work
> item was running and the workqueue is destroyed afterwards, it may end
> up trying to requeue itself on a destroyed workqueue.

Right, but my initial understanding was that when canceling the delayed
work and then issuing flush_workqueue would act the same way as if
cancel_delayed_work_sync is called wrt to this particular delayed item, no?



> 
> Thanks.
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/