| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <da90f64c-260a-4329-87bf-1f9ff20a5951@gmail.com>
Date: Sun, 15 Dec 2024 04:17:39 +0200
From: Sergey Ryazanov <ryazanov.s.a@...il.com>
To: Jinjian Song <jinjian.song@...ocom.com>,
chandrashekar.devegowda@...el.com, chiranjeevi.rapolu@...ux.intel.com,
haijun.liu@...iatek.com, ricardo.martinez@...ux.intel.com,
loic.poulain@...aro.org, johannes@...solutions.net, davem@...emloft.net,
edumazet@...gle.com, kuba@...nel.org, pabeni@...hat.com
Cc: linux-kernel@...r.kernel.org, netdev@...r.kernel.org,
linux-doc@...r.kernel.org, angelogioacchino.delregno@...labora.com,
linux-arm-kernel@...ts.infradead.org, matthias.bgg@...il.com,
corbet@....net, linux-mediatek@...ts.infradead.org, helgaas@...nel.org,
danielwinkler@...gle.com, korneld@...gle.com, andrew+netdev@...n.ch,
horms@...nel.org
Subject: Re: [net v2] net: wwan: t7xx: Fix FSM command timeout issue
Hello Jinjian,
Nice catch! Suddenly, the proposed fix is not complete, we still have
issues with memory use after free. Please find details below.
On 13.12.2024 08:47, Jinjian Song wrote:
> When driver processes the internal state change command, it use an
> asynchronous thread to process the command operation. If the main
> thread detects that the task has timed out, the asynchronous thread
> will panic when executing the completion notification because the
> main thread completion object has been released.
>
> BUG: unable to handle page fault for address: fffffffffffffff8
> PGD 1f283a067 P4D 1f283a067 PUD 1f283c067 PMD 0
> Oops: 0000 [#1] PREEMPT SMP NOPTI
> RIP: 0010:complete_all+0x3e/0xa0
> [...]
> Call Trace:
> <TASK>
> ? __die_body+0x68/0xb0
> ? page_fault_oops+0x379/0x3e0
> ? exc_page_fault+0x69/0xa0
> ? asm_exc_page_fault+0x22/0x30
> ? complete_all+0x3e/0xa0
> fsm_main_thread+0xa3/0x9c0 [mtk_t7xx (HASH:1400 5)]
> ? __pfx_autoremove_wake_function+0x10/0x10
> kthread+0xd8/0x110
> ? __pfx_fsm_main_thread+0x10/0x10 [mtk_t7xx (HASH:1400 5)]
> ? __pfx_kthread+0x10/0x10
> ret_from_fork+0x38/0x50
> ? __pfx_kthread+0x10/0x10
> ret_from_fork_asm+0x1b/0x30
> </TASK>
> [...]
> CR2: fffffffffffffff8
> ---[ end trace 0000000000000000 ]---
>
> After the main thread determines that the task has timed out, mark
> the completion invalid, and add judgment in the asynchronous task.
>
> Fixes: d785ed945de6 ("net: wwan: t7xx: PCIe reset rescan")
The completion waiting was introduced in a different commit. I believe,
the fix tag should be 13e920d93e37 ("net: wwan: t7xx: Add core components")
> Signed-off-by: Jinjian Song <jinjian.song@...ocom.com>
> ---
> drivers/net/wwan/t7xx/t7xx_state_monitor.c | 7 +++++--
> 1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/net/wwan/t7xx/t7xx_state_monitor.c b/drivers/net/wwan/t7xx/t7xx_state_monitor.c
> index 3931c7a13f5a..57f1a7730fff 100644
> --- a/drivers/net/wwan/t7xx/t7xx_state_monitor.c
> +++ b/drivers/net/wwan/t7xx/t7xx_state_monitor.c
> @@ -108,7 +108,8 @@ static void fsm_finish_command(struct t7xx_fsm_ctl *ctl, struct t7xx_fsm_command
> {
> if (cmd->flag & FSM_CMD_FLAG_WAIT_FOR_COMPLETION) {
> *cmd->ret = result;
The memory for the result storage is allocated on the stack as well. And
writing it unconditionally can cause unexpected consequences.
> - complete_all(cmd->done);
> + if (cmd->done)
> + complete_all(cmd->done);
> }
>
> kfree(cmd);
> @@ -503,8 +504,10 @@ int t7xx_fsm_append_cmd(struct t7xx_fsm_ctl *ctl, enum t7xx_fsm_cmd_state cmd_id
>
> wait_ret = wait_for_completion_timeout(&done,
> msecs_to_jiffies(FSM_CMD_TIMEOUT_MS));
> - if (!wait_ret)
> + if (!wait_ret) {
> + cmd->done = NULL;
We cannot access the command memory here, since fsm_finish_command()
could release it already.
> return -ETIMEDOUT;
> + }
>
> return ret;
> }
Here we have an ownership transfer problem and a driver author has tried
to solve it, but as noticed, we are still experiencing issues in case of
timeout.
The command completion routine should not release the command memory
unconditionally. Looks like the references counting approach should help
us here. E.g.
1. grab a reference before we put a command into the queue
1.1. grab an extra reference if we are going to wait the completion
2. release the reference as soon as we are done with the command execution
3. in case of completion waiting release the reference as soon as we are
done with waiting due to completion or timeout
Could you try the following patch? Please note, besides the reference
counter introduction it also moves completion and result storage inside
the command structure as advised by the completion documentation.
--- a/drivers/net/wwan/t7xx/t7xx_state_monitor.h
+++ b/drivers/net/wwan/t7xx/t7xx_state_monitor.h
@@ -109,8 +109,9 @@ struct t7xx_fsm_command {
struct list_head entry;
enum t7xx_fsm_cmd_state cmd_id;
unsigned int flag;
- struct completion *done;
- int *ret;
+ struct completion done;
+ int result;
+ struct struct kref refcnt;
};
struct t7xx_fsm_notifier {
--- a/drivers/net/wwan/t7xx/t7xx_state_monitor.c
+++ b/drivers/net/wwan/t7xx/t7xx_state_monitor.c
@@ -97,14 +97,20 @@ void t7xx_fsm_broadcast_state(struct t7xx_fsm_ctl
*ctl, enum md_state state)
fsm_state_notify(ctl->md, state);
}
+static void fsm_release_command(struct kref *ref)
+{
+ struct t7xx_fsm_command *cmd = container_of(ref, typeof(*cmd), refcnt);
+ kfree(cmd);
+}
+
static void fsm_finish_command(struct t7xx_fsm_ctl *ctl, struct
t7xx_fsm_command *cmd, int result)
{
if (cmd->flag & FSM_CMD_FLAG_WAIT_FOR_COMPLETION) {
- *cmd->ret = result;
+ cmd->result = result;
complete_all(cmd->done);
}
- kfree(cmd);
+ kref_put(&cmd->refcnt, fsm_release_command);
}
static void fsm_del_kf_event(struct t7xx_fsm_event *event)
@@ -396,7 +402,6 @@ static int fsm_main_thread(void *data)
int t7xx_fsm_append_cmd(struct t7xx_fsm_ctl *ctl, enum
t7xx_fsm_cmd_state cmd_id, unsigned int flag)
{
- DECLARE_COMPLETION_ONSTACK(done);
struct t7xx_fsm_command *cmd;
unsigned long flags;
int ret;
@@ -408,11 +413,13 @@ int t7xx_fsm_append_cmd(struct t7xx_fsm_ctl *ctl,
enum t7xx_fsm_cmd_state cmd_id
INIT_LIST_HEAD(&cmd->entry);
cmd->cmd_id = cmd_id;
cmd->flag = flag;
+ kref_init(&cmd->refcnt);
if (flag & FSM_CMD_FLAG_WAIT_FOR_COMPLETION) {
- cmd->done = &done;
- cmd->ret = &ret;
+ init_completion(&cmd->done);
+ kref_get(&cmd->refcnt);
}
+ kref_get(&cmd->refcnt);
spin_lock_irqsave(&ctl->command_lock, flags);
list_add_tail(&cmd->entry, &ctl->command_queue);
spin_unlock_irqrestore(&ctl->command_lock, flags);
@@ -422,10 +429,10 @@ int t7xx_fsm_append_cmd(struct t7xx_fsm_ctl *ctl,
enum t7xx_fsm_cmd_state cmd_id
if (flag & FSM_CMD_FLAG_WAIT_FOR_COMPLETION) {
unsigned long wait_ret;
- wait_ret = wait_for_completion_timeout(&done,
+ wait_ret = wait_for_completion_timeout(&cmd->done,
msecs_to_jiffies(FSM_CMD_TIMEOUT_MS));
- if (!wait_ret)
- return -ETIMEDOUT;
+ ret = wait_ret ? cmd->result : -ETIMEDOUT;
+ kref_put(&cmd->refcnt, fsm_release_command);
return ret;
}
--
Sergey
Powered by blists - more mailing lists