Message-ID: <1390427824.27806.36.camel@edumazet-glaptop2.roam.corp.google.com>
Date:	Wed, 22 Jan 2014 13:57:04 -0800
From:	Eric Dumazet <eric.dumazet@...il.com>
To:	Arnaud Ebalard <arno@...isbad.org>
Cc:	David Miller <davem@...emloft.net>,
	Eric Dumazet <edumazet@...gle.com>,
	Daniel Borkmann <dborkman@...hat.com>,
	Herbert Xu <herbert@...dor.apana.org.au>,
	Willy Tarreau <w@....eu>, netdev@...r.kernel.org
Subject: Re: [BUG] null pointer dereference in tcp_gso_segment()

On Wed, 2014-01-22 at 22:46 +0100, Arnaud Ebalard wrote:
> Hi,
> 
> The following is a backtrace I got while doing a simple transfer from my
> NAS (ARMv7-based ReadyNAS 102) running a 3.13.0 kernel. Usage was a
> simple file served by nginx. I managed to get a second identical one
> in another session. As this did not looked like some random race
> condition or some missing lock, I decided to take a look. Details are
> given below. FWIW, driver is mvneta (w/ fixes and performance patches
> you accepted for v3.14, David) and ethtool reports gso is enabled and tso
> disabled.
> 
> [  943.860383] Unable to handle kernel NULL pointer dereference at virtual address 00000090
> [  943.868497] pgd = c0004000
> [  943.871217] [00000090] *pgd=00000000
> [  943.874813] Internal error: Oops: 17 [#1] ARM
> [  943.879175] Modules linked in:
> [  943.882248] CPU: 0 PID: 0 Comm: swapper Not tainted 3.13.0.rn102-00594-ga0fa1dd3cdbc-dirty #63
> [  943.890873] task: c0775620 ti: c0768000 task.ti: c0768000
> [  943.896291] PC is at tcp_gso_segment+0x1d8/0x390
> [  943.900925] LR is at skb_segment+0x510/0x788
> [  943.905202] pc : [<c04b2094>]    lr : [<c0448b90>]    psr: 200f0113
> [  943.905202] sp : c0769aa8  ip : 00005b27  fp : 00000001
> [  943.916697] r10: 00005b27  r9 : 00000868  r8 : dfab5cb0
> [  943.921930] r7 : 00000000  r6 : 4ef79279  r5 : 00000020  r4 : deeba830
> [  943.928467] r3 : 000005a8  r2 : df993200  r1 : c04a63f8  r0 : 000005a8
> [  943.935005] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
> [  943.942325] Control: 10c5387d  Table: 1534c019  DAC: 00000015
> [  943.948079] Process swapper (pid: 0, stack limit = 0xc0768238)
> [  943.953921] Stack: (0xc0769aa8 to 0xc076a000)
> [  943.958288] 9aa0:                   c077000c 000005a8 0000e397 d52f3540 c0771230 00001c48
> [  943.966481] 9ac0: 00000006 deeba830 0000c6bc 00000014 0000009c 00000000 00000014 c04beeb4
> [  943.974674] 9ae0: c0fbb8ec dec2ab80 0000008e 00000000 00000001 00004803 00000002 deeba830
> [  943.982867] 9b00: c07739cc deeba830 df27d15c df1fd800 c05a045c c0452dcc 00000004 c07739e0
> [  943.991061] 9b20: 00004803 00004803 00000002 00010000 00000000 deeba830 df27d15c c04530e8
> [  943.999254] 9b40: 00000001 c046fc40 c0492410 df1fd800 00000000 00000000 00000000 def06e00
> [  944.007447] 9b60: df1fd800 deeba830 df1fd800 df27d15c 00000010 c0469af4 def06e00 00000000
> [  944.015640] 9b80: 00000042 00000000 df27d15c df1fd800 deeba830 c04535b8 def06e60 000333c5
> [  944.023834] 9ba0: 00000000 dfaf5180 dfaf51fc 0000000e 00000000 deeba830 df1fd800 def35580
> [  944.032027] 9bc0: 00000010 c0492604 80000000 0b50a8c0 00000000 deeba830 df993200 00000000
> [  944.040221] 9be0: df993404 def35580 00000000 009c0000 00000000 c0492a2c deeba830 c0492b5c
> [  944.048414] 9c00: deeba780 df993200 deeba780 00000020 00000000 0000fb7b 00000020 df993200
> [  944.056608] 9c20: deeba830 c0778368 00000000 def35580 00000000 c04a76e8 c3a0f5c7 134b291d
> [  944.064802] 9c40: 00000001 00000000 00000002 00000000 00000000 0000fb7b 002411ee 00000000
> [  944.072995] 9c60: 79e31870 df993200 deeba780 000005a8 4ef79279 00001c48 00000000 000021f0
> [  944.081188] 9c80: 00000000 c04a7d68 c39c239b 134b291d 00000006 000016a0 000021f0 00000001
> [  944.089381] 9ca0: 00000000 df9932bc 00000002 00000000 00000006 134b291d d52f3540 df993200
> [  944.097575] 9cc0: d567e7e4 d52f3540 00000020 d567e7d0 d52f3540 0000ddc7 c0770618 c04a86d8
> [  944.105768] 9ce0: 00000020 df993200 df993200 c04a4954 00000000 00000020 00000001 00000002
> [  944.113961] 9d00: d52f3540 df993200 def35a00 df993200 d567e7d0 d52f3540 0000ddc7 c04ac61c
> [  944.122154] 9d20: c048da1c c046fc40 c048da1c c046fc40 dec12140 c0770b64 d52f3540 d52f3540
> [  944.130348] 9d40: c07a3dc0 00000000 df993200 c04aede0 df1fd800 c046fcc0 00000000 c0769d74
> [  944.138541] 9d60: c048da1c 80000000 c048d734 c0770b64 0000000e c0770b64 3050a8c0 c05aacb4
> [  944.146734] 9d80: c0773470 00000000 c07a3dc0 d52f3540 d52f3540 00000000 c0770618 c048dab0
> [  944.154927] 9da0: d567e7d0 c07739e8 c0770628 df1fd800 d52f3540 c048d85c df1b4400 c0770614
> [  944.163120] 9dc0: c0770614 c07739e8 c0770628 df1fd800 00000008 c044f018 00000003 00000000
> [  944.171314] 9de0: e1424800 42000000 df1fd800 d52f3540 00000001 c0770628 00000000 d52f3540
> [  944.179507] 9e00: 00000003 df3bd400 e1424800 42000000 df1fd800 d52f3540 00000001 c045153c
> [  944.187700] 9e20: 00000004 c0019430 e1424000 c0382238 00000000 00000000 00000002 dfbc4600
> [  944.195893] 9e40: 00000001 df1fdbe0 00000001 00000042 00000000 00000000 dee34000 00000040
> [  944.204086] 9e60: df1fdbe0 00000100 00000000 00000100 df3bd400 c079b684 00000000 c0382354
> [  944.212279] 9e80: c04ab2fc c04aa344 df993200 df1fd800 00000100 df1fdbfc 00000000 df993200
> [  944.220473] 9ea0: 00000100 c04ab360 c0768000 c0023cf4 c0fba140 c03822b0 df1fdbfc 00000040
> [  944.228666] 9ec0: 0000012c c07abb40 c07abb48 c0778368 c07abb40 c0451288 c0769ed8 0000fb7d
> [  944.236859] 9ee0: c0fba140 00000001 0000000c c07ac850 c07ac840 c0768000 00000003 00000100
> [  944.245052] 9f00: 0000000c c001ebd4 00000000 c004782c c0fba140 0000000a 0000fb7c 00200000
> [  944.253245] 9f20: c0769f78 c078af20 00000018 00000000 c0769f78 c07abc01 00000001 c07abc01
> [  944.261438] 9f40: 00000000 c001ef18 c078af20 c000ea44 c07f6f40 000003ff c07f6f40 c00084d8
> [  944.269632] 9f60: c000ebc0 c0041c54 600f0013 ffffffff c0769fac c00110c0 00000000 00000000
> [  944.277825] 9f80: 00000000 c0779a48 c0768000 c0768000 c0768000 c07700cc c07abc01 00000001
> [  944.286018] 9fa0: c07abc01 00000000 01000000 c0769fc0 c000ebc0 c0041c54 600f0013 ffffffff
> [  944.294211] 9fc0: c0761530 c073ba14 ffffffff ffffffff c073b4f0 00000000 00000000 c0761530
> [  944.302405] 9fe0: 00000000 10c53c7d c0770078 c076152c c07765f8 00008070 00000000 00000000
> [  944.310608] [<c04b2094>] (tcp_gso_segment+0x1d8/0x390) from [<c04beeb4>] (inet_gso_segment+0x118/0x2dc)
> [  944.320029] [<c04beeb4>] (inet_gso_segment+0x118/0x2dc) from [<c0452dcc>] (skb_mac_gso_segment+0xa4/0x178)
> [  944.329704] [<c0452dcc>] (skb_mac_gso_segment+0xa4/0x178) from [<c04530e8>] (dev_hard_start_xmit+0x170/0x484)
> [  944.339643] [<c04530e8>] (dev_hard_start_xmit+0x170/0x484) from [<c0469af4>] (sch_direct_xmit+0xa4/0x19c)
> [  944.349231] [<c0469af4>] (sch_direct_xmit+0xa4/0x19c) from [<c04535b8>] (__dev_queue_xmit+0x1bc/0x3dc)
> [  944.358560] [<c04535b8>] (__dev_queue_xmit+0x1bc/0x3dc) from [<c0492604>] (ip_finish_output+0x1f4/0x440)
> [  944.368060] [<c0492604>] (ip_finish_output+0x1f4/0x440) from [<c0492a2c>] (ip_local_out+0x28/0x2c)
> [  944.377037] [<c0492a2c>] (ip_local_out+0x28/0x2c) from [<c0492b5c>] (ip_queue_xmit+0x12c/0x384)
> [  944.385754] [<c0492b5c>] (ip_queue_xmit+0x12c/0x384) from [<c04a76e8>] (tcp_transmit_skb+0x42c/0x86c)
> [  944.394992] [<c04a76e8>] (tcp_transmit_skb+0x42c/0x86c) from [<c04a7d68>] (tcp_write_xmit+0x174/0xa74)
> [  944.404317] [<c04a7d68>] (tcp_write_xmit+0x174/0xa74) from [<c04a86d8>] (__tcp_push_pending_frames+0x30/0x98)
> [  944.414252] [<c04a86d8>] (__tcp_push_pending_frames+0x30/0x98) from [<c04a4954>] (tcp_rcv_established+0x334/0x5a0)
> [  944.424622] [<c04a4954>] (tcp_rcv_established+0x334/0x5a0) from [<c04ac61c>] (tcp_v4_do_rcv+0x104/0x240)
> [  944.434122] [<c04ac61c>] (tcp_v4_do_rcv+0x104/0x240) from [<c04aede0>] (tcp_v4_rcv+0x6ec/0x728)
> [  944.442838] [<c04aede0>] (tcp_v4_rcv+0x6ec/0x728) from [<c048dab0>] (ip_local_deliver_finish+0x94/0x21c)
> [  944.452337] [<c048dab0>] (ip_local_deliver_finish+0x94/0x21c) from [<c048d85c>] (ip_rcv_finish+0x128/0x2e8)
> [  944.462098] [<c048d85c>] (ip_rcv_finish+0x128/0x2e8) from [<c044f018>] (__netif_receive_skb_core+0x4c4/0x5d0)
> [  944.472032] [<c044f018>] (__netif_receive_skb_core+0x4c4/0x5d0) from [<c045153c>] (napi_gro_receive+0x74/0xa0)
> [  944.482054] [<c045153c>] (napi_gro_receive+0x74/0xa0) from [<c0382238>] (mvneta_rx+0x420/0x498)
> [  944.490771] [<c0382238>] (mvneta_rx+0x420/0x498) from [<c0382354>] (mvneta_poll+0xa4/0x3b8)
> [  944.499139] [<c0382354>] (mvneta_poll+0xa4/0x3b8) from [<c0451288>] (net_rx_action+0x98/0x180)
> [  944.507773] [<c0451288>] (net_rx_action+0x98/0x180) from [<c001ebd4>] (__do_softirq+0xc8/0x1f4)
> [  944.516491] [<c001ebd4>] (__do_softirq+0xc8/0x1f4) from [<c001ef18>] (irq_exit+0x6c/0xa8)
> [  944.524690] [<c001ef18>] (irq_exit+0x6c/0xa8) from [<c000ea44>] (handle_IRQ+0x34/0x84)
> [  944.532624] [<c000ea44>] (handle_IRQ+0x34/0x84) from [<c00084d8>] (armada_370_xp_handle_irq+0x48/0x50)
> [  944.541953] [<c00084d8>] (armada_370_xp_handle_irq+0x48/0x50) from [<c00110c0>] (__irq_svc+0x40/0x50)
> [  944.551186] Exception stack(0xc0769f78 to 0xc0769fc0)
> [  944.556246] 9f60:                                                       00000000 00000000
> [  944.564439] 9f80: 00000000 c0779a48 c0768000 c0768000 c0768000 c07700cc c07abc01 00000001
> [  944.572632] 9fa0: c07abc01 00000000 01000000 c0769fc0 c000ebc0 c0041c54 600f0013 ffffffff
> [  944.580835] [<c00110c0>] (__irq_svc+0x40/0x50) from [<c0041c54>] (cpu_startup_entry+0x44/0xe0)
> [  944.589470] [<c0041c54>] (cpu_startup_entry+0x44/0xe0) from [<c073ba14>] (start_kernel+0x2fc/0x358)
> [  944.598533] Code: e5942010 e5872010 e5977000 e59d3004 (e1d729b0) 
> [  944.604662] ---[ end trace 39b798f37a10efc0 ]---
> [  944.609288] Kernel panic - not syncing: Fatal exception in interrupt
> 
> 
> Here is the beginning of the assembly dump of tcp_gso_segment() provided by
> arm-linux-gnueabi-objdump -S -EL -D -b binary -m arm --start-address=0x4a9ebc
> --stop-address=0x4aa24c Image. If you wonder, LOADADDR is 0x8000 during
> compilation, which explains the address shift.

> The line marked with the HERE! above is where the null pointer derefence
> occurs. The code is in fact the (inlined) code of tcp_hdr(), which is in
> turn the inlined code of tcp_transport_header() on given skb:
> 
> static inline unsigned char *skb_transport_header(const struct sk_buff *skb)
> {
>         return skb->head + skb->transport_header;
> }
> 
> The call to tcp_hdr() is in the following loop in tcp_gso_segment():
> 
>      ...
>   1  do {
>   2          th->fin = th->psh = 0;
>   3          th->check = newcheck;
>   4
>   5          if (skb->ip_summed != CHECKSUM_PARTIAL)
>   6                  th->check =
>   7                       csum_fold(csum_partial(skb_transport_header(skb),
>   8                                              thlen, skb->csum));
>   9
>  10          seq += mss;
>  11          if (copy_destructor) {
>  12                  skb->destructor = gso_skb->destructor;
>  13                  skb->sk = gso_skb->sk;
>  14                  sum_truesize += skb->truesize;
>  15          }
>  16          skb = skb->next;
>  17          th = tcp_hdr(skb);         <-- HERE!
>  18
>  19          th->seq = htonl(seq);
>  20          th->cwr = 0;
>  21  } while (skb->next);
>      ...
> 
> Unless there is an assumption I missed somewhere in the function, the
> problem may occur during the first round of the loop, because (unlike
> the 'while' condition does at line 21) skb->next is not checked against
> null at lines 17 above before it is passed to tcp_hdr() at line 18.
> 
> To be honest, I am asking because I am not familiar w/ the code and it
> is somewhat old so I wonder why noone got hit before. AFAICT,
> f4c50d990dcf ([NET]: Add software TSOv4) added TSOv4 support in 2006 via
> introduction of tcp_tso_segmen() (with the same kind of deref but
> possibly different assumptions) which was more recently modified via
> 28850dc7c7 (net: tcp: move GRO/GSO functions to tcp_offload) to become
> tcp_gso_segment().
> 
> David, can you confirm the analysis and possibly comment on the
> conditions needed for the bug to manifest?

A gso packet contains at least 2 segments.

So the NULL deref should not happen.

Something strange is happening here...



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists