linux-kernel - Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in process

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <alpine.LFD.2.11.1509290957130.2072@ja.home.ssi.bg>
Date:	Tue, 29 Sep 2015 10:40:43 +0300 (EEST)
From:	Julian Anastasov <ja@....bg>
To:	"Andre Tomt (LKML)" <lkml@...t.net>
cc:	Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
	linux-kernel@...r.kernel.org,
	"David S. Miller" <davem@...emloft.net>, stable@...r.kernel.org,
	"Eric W. Biederman" <ebiederm@...ssion.com>,
	Stephen Hemminger <stephen@...workplumber.org>,
	holger.hoffstaette@...glemail.com
Subject: Re: [PATCH 4.1 125/159] net: call rcu_read_lock early in
 process_backlog


	Hello,

On Tue, 29 Sep 2015, Andre Tomt (LKML) wrote:

> On 26. sep. 2015 22:56, Greg Kroah-Hartman wrote:
> > 4.1-stable review patch.  If anyone has any objections, please let me know.
> >
> > ------------------
> >
> > From: Julian Anastasov <ja@....bg>
> >
> > [ Upstream commit 2c17d27c36dcce2b6bf689f41a46b9e909877c21 ]
> >
> > Incoming packet should be either in backlog queue or
> > in RCU read-side section. Otherwise, the final sequence of
> > flush_backlog() and synchronize_net() may miss packets
> > that can run without device reference:
> <snip>
> 
> Several of our 4.1.9-rc1 running systems are experiencing hangs requiring
> hardware/sysrq reset with this patch applied. Reverting it fixes the hangs
> completely.

	Bad to hear that...

> 4.2 includes this patch as well but I have no such problems there. 4.2.2-rc1
> works fine as well.

	No problem with 4.2+? Same setup/config?

> For now I think this patch should be reverted in 4.1.9.
> 
> The hangs have occured so far on Xen PV and KVM x86_64 virtual machines, they
> will hang completely within minutes or hours depending on the type of
> workload. The workloads are all fairly light, one running low traffic
> email/antispam, another running monitoring and metrics of ~5 hosts and one
> running a single terminal IRC client. All but the IRC one will hang within a
> few minutes of booting.
> 
> When they lock up they only respond to sysrq, with ttyS0/hvc0 not echoing
> anything typed in back, and are completely dead on the network. One system
> managed to report rcu stalls but no backtraces (I'll look over the debug
> config, if there is any interest).
> 
> My bare metal desktop has yet to be able to hit it, but it might be entirely
> down to a different type of workload.
> 
> Something missing in 4.1?

	They are 2 related patches, the first one is
[PATCH 4.1 124/159] net: do not process device backlog during unregistration

	But the problematic patch calls rcu_read_lock while
local IRQ is disabled (in process_backlog), this is something
that should be noted for the patch. I'll try to see what Xen does.
It would be useful to see .config and any kind of backtraces/stalls,
it will help also to other developers to catch the problem...

Regards

--
Julian Anastasov <ja@....bg>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/