Message-ID: <CAOMFOmWB4yxj65N2x5+KREmiQBdPoWsTiA7EBW6ONPBOyFo11w@mail.gmail.com>
Date:	Fri, 14 Sep 2012 14:06:10 -0700
From:	Anatol Pomozov <anatol.pomozov@...il.com>
To:	linux-ext4@...r.kernel.org, "Theodore Ts'o" <tytso@....edu>
Subject: ext4_orphan_del() sleeps in non-journal mode

Hi,

I am debugging one issue that happens on our servers. We use ext4 with
non-journaling mode (2.6.34 kernel) and when we try to use
asynchronous IO we see following oops in dmesg:

<3>[ 3983.762966] bad: scheduling from the idle thread!
<4>[ 3983.762968] Pid: 0, comm: swapper
<4>[ 3983.762970] Call Trace:
<4>[ 3983.762972]  <IRQ>  [<ffffffff811d3fde>] dequeue_task_idle+0x24/0x30
<4>[ 3983.762980]  [<ffffffff81002f58>] schedule+0x2a98/0x3310
<4>[ 3983.762985]  [<ffffffff8101a08a>] ? sched_clock_cpu+0x2a/0xe0
<4>[ 3983.762988]  [<ffffffff8102b5d7>] ? mempool_alloc+0xa7/0x1a0
<4>[ 3983.762992]  [<ffffffff8100441b>] __mutex_lock_common.isra.3+0x14b/0x1d0
<4>[ 3983.762996]  [<ffffffff810045c3>] __mutex_lock_slowpath+0x13/0x20
<4>[ 3983.762999]  [<ffffffff81004242>] mutex_lock+0x22/0x40
<4>[ 3983.763004]  [<ffffffff8111918f>] ext4_orphan_del+0x4f/0x2e0
<4>[ 3983.763008]  [<ffffffff810b2e8c>] ? insert_work+0x6c/0xb0
<4>[ 3983.763011]  [<ffffffff81027af8>] ? diskmon_bio_complete+0x798/0xda0
<4>[ 3983.763016]  [<ffffffff812a33e8>] ext4_end_io_dio+0xb7/0x1d7
<4>[ 3983.763021]  [<ffffffff81050f3c>] dio_fast_end_async+0x1bc/0x1d0
<4>[ 3983.763025]  [<ffffffff8112c93a>] ? blk_complete_request+0x1a/0x20
<4>[ 3983.763028]  [<ffffffff81050a2d>] bio_endio+0x6d/0x80
<4>[ 3983.763033]  [<ffffffff81129002>] req_bio_endio+0x62/0xb0
<4>[ 3983.763036]  [<ffffffff81129202>] blk_update_request+0x142/0x3f0
<4>[ 3983.763041]  [<ffffffff8114232e>] ? ata_qc_complete+0xae/0x1f0
<4>[ 3983.763044]  [<ffffffff811299fc>] blk_end_bidi_request+0x2c/0xa0
<4>[ 3983.763047]  [<ffffffff81129a80>] blk_end_request+0x10/0x20
<4>[ 3983.763050]  [<ffffffff8113ffac>] scsi_io_completion+0xac/0x520
<4>[ 3983.763053]  [<ffffffff8113dca7>] scsi_finish_command+0xb7/0x110
<4>[ 3983.763056]  [<ffffffff8113fddf>] scsi_softirq_done+0x6f/0x140
<4>[ 3983.763059]  [<ffffffff8112c7d7>] blk_done_softirq+0x77/0x80
<4>[ 3983.763062]  [<ffffffff810156cf>] __do_softirq+0x37f/0x3e0
<4>[ 3983.763066]  [<ffffffff8109e7bc>] ? ack_apic_level+0x7c/0x1f0
<4>[ 3983.763070]  [<ffffffff810995cc>] call_softirq+0x1c/0x30
<4>[ 3983.763072]  [<ffffffff81005cf1>] do_softirq+0x41/0x80
<4>[ 3983.763074]  [<ffffffff81015879>] irq_exit+0x49/0xa0
<4>[ 3983.763077]  [<ffffffff810055b2>] do_IRQ+0x72/0xe0
<4>[ 3983.763083]  [<ffffffff814a0c13>] ret_from_intr+0x0/0xa
<4>[ 3983.763084]  <EOI>  [<ffffffff81005da0>] ? c1e_idle+0x70/0x170
<4>[ 3983.763089]  [<ffffffff81005860>] cpu_idle+0x90/0x130
<4>[ 3983.763091]  [<ffffffff8117b45a>] rest_init+0x7e/0x80
<4>[ 3983.763094]  [<ffffffff81b45c62>] start_kernel+0x3b7/0x3c3
<4>[ 3983.763097]  [<ffffffff81b45331>] x86_64_start_reservations+0x141/0x145
<4>[ 3983.763101]  [<ffffffff81b4544c>] x86_64_start_kernel+0x117/0x11e



So the problem is that ext4_orphan_del() wants to sleep in softirq
context. I started debugging and here are some questions.

The first question is why ext4_orphan_del() sleeps in no-journal mode
at all. It gets mutex to manipulate with i_orphan list but this list
is used only in journaling mode. In non-journal mode (in my case) both
ext4_orphan_del() and ext4_orphan_add() should be no-op.

ext4_orphan_del() gets mutex in no-journal mode when it is called with
NULL as a first parameter. There are 10 places in fs/ext4 where it
happens:

$ git grep "ext4_orphan_del(NULL"
fs/ext4/indirect.c:845:                         ext4_orphan_del(NULL, inode);
fs/ext4/inode.c:249:            ext4_orphan_del(NULL, inode);
fs/ext4/inode.c:281:                    ext4_orphan_del(NULL, inode);
fs/ext4/inode.c:956:                            ext4_orphan_del(NULL, inode);
fs/ext4/inode.c:1069:                   ext4_orphan_del(NULL, inode);
fs/ext4/inode.c:1111:                   ext4_orphan_del(NULL, inode);
fs/ext4/inode.c:1177:                   ext4_orphan_del(NULL, inode);
fs/ext4/inode.c:4338:
ext4_orphan_del(NULL, inode);
fs/ext4/inode.c:4365:           ext4_orphan_del(NULL, inode);
fs/ext4/migrate.c:516:          ext4_orphan_del(NULL, tmp_inode);


There was a change that fixes ext4_orphan_del(NULL) issue in
ext4_setattr for no-journal mode 3d287de3b828 . And I think we should
fix all other places as well.

There are several possible solutions for this issue:
1) Pass handle received by ext4_journal_current_handle() or similar.
Why do we pass NULL at all when we can use the handle? I see that in
some functions we already have "handle" variable that we can re-use.
2) Follow the way used by Dmitry and call ext4_orphan_del only if
ext4_orphan_add was successful *and* handle is valid. This is not
always possible as not all _del() are paired with _add() in the same
function.
3) Inside ext4_orphan_del() and ext4_orphan_add() check if journal is
enabled. Do nothing if this is no-journal mode. What is the best way
to check no-journal mode? Is it just "if (EXT4_SB(sb)->s_journal) ..."

It seems that #1 is the best way.

PS once this no-journal issue will be clarified I'll take a look at
sleeping issue in journaling mode.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists