Message-ID: <ae631eefb45947ac84cfe0468d0b7508@AcuMS.aculab.com>
Date:   Fri, 20 May 2022 16:08:10 +0000
From:   David Laight <David.Laight@...LAB.COM>
To:     David Laight <David.Laight@...LAB.COM>,
        'Pavan Chebbi' <pavan.chebbi@...adcom.com>
CC:     Paolo Abeni <pabeni@...hat.com>,
        Michael Chan <michael.chan@...adcom.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "mchan@...adcom.com" <mchan@...adcom.com>,
        David Miller <davem@...emloft.net>
Subject: RE: tg3 dropping packets at high packet rates

I've hijacked 'ethtool -d' to output some instrumentation.

Basically doing:
        tp->rx_avail[(hw_idx - sw_idx) & tp->rx_ret_ring_mask]++;
in both the hardware interrupt and napi callback.
(To different arrays.)

These tests were done after setting:
ethtool -G em2 rx 2047
ethtool -C em2 rx-usecs 1
ethtool -C em2 rx-frames 1
ethtool -C em2 rx-frames-irq 1
while the system was idle.

After running or a few seconds the number of pending packets
in the hardware interrupt isn't too bad:
# ethtool -d em2|grep '^0.[4-7]'|xargs -n8
0x4000 0x000e6004 0x4004 0x009c794f 0x4008 0x0024ccf0 0x400c 0x00037fb1
0x4010 0x00004ddd 0x4014 0x00001a05 0x4018 0x00000d92 0x401c 0x00000603
0x4020 0x000002a4 0x4024 0x000000f0 0x4028 0x00000061 0x402c 0x00000044
0x4030 0x0000001c 0x4034 0x00000014 0x4038 0x0000000c 0x403c 0x0000000b
0x4040 0x00000005 0x4044 0x00000001 0x4048 0x00000001 0x404c 0x00000004
0x4054 0x00000001 0x4090 0x00000001 0x4094 0x00000001 0x40a8 0x00000001
0x40c4 0x00000001 0x4138 0x00000001 0x4148 0x00000001 0x4154 0x00000001
0x4160 0x00000002 0x4178 0x00000001 0x4198 0x00000001 0x41a4 0x00000001
0x41c8 0x00000001 0x41cc 0x00000001 0x41d0 0x00000001 0x41d8 0x00000001
0x41e8 0x00000001 0x41f4 0x00000001 0x4200 0x00000001 0x4204 0x00000001
0x420c 0x00000001 0x4210 0x00000001 0x4218 0x00000001 0x4220 0x00000002
0x4224 0x00000004 0x4228 0x00000003 0x422c 0x00000002 0x4230 0x00000001
0x4234 0x00000001 0x4238 0x00000002 0x423c 0x00000002 0x4240 0x00000003
0x4244 0x00000004 0x4248 0x00000008 0x424c 0x00000008 0x4250 0x00000002
0x4254 0x00000008 0x4258 0x00000003 0x425c 0x00000004 0x4260 0x00000003
0x4264 0x0000000b 0x4268 0x00000006 0x426c 0x00000004 0x4270 0x00000004
0x4274 0x00000004 0x4278 0x00000003 0x427c 0x00000005 0x4280 0x00000003
0x4284 0x00000004 0x4288 0x00000005 0x428c 0x00000004 0x4290 0x00000003
0x4294 0x00000002 0x4298 0x00000003 0x429c 0x00000003 0x42a0 0x00000001
0x42a4 0x00000001 0x42a8 0x00000001 0x42ac 0x00000004 0x42b0 0x00000001
0x42b8 0x00000001 0x42c0 0x00000001 0x42c8 0x00000002 0x42d0 0x00000001
0x42d4 0x00000001 0x42e0 0x00000001 0x4304 0x00000001 0x4314 0x00000001
0x4340 0x00000001 0x439c 0x00000001 0x43b0 0x00000001 0x43dc 0x00000001
0x4410 0x00000001 0x44a0 0x00000001

In the napi callback (tg3_rx()) it is a different story:
# ethtool -d em2|grep '^0.[0-3]'|xargs -n8
0x0000 0x010778da 0x0004 0x011537a8 0x0008 0x004229f2 0x000c 0x000a583d
0x0010 0x0001dd9f 0x0014 0x0000afa6 0x0018 0x0000606d 0x001c 0x00003b80
0x0020 0x0000273f 0x0024 0x00001b72 0x0028 0x000013f9 0x002c 0x00000f15
0x0030 0x00000ce6 0x0034 0x00000a4e 0x0038 0x000008ac 0x003c 0x000007c5
0x0040 0x00000647 0x0044 0x0000059f 0x0048 0x0000053f 0x004c 0x00000486
0x0050 0x0000043a 0x0054 0x000003ad 0x0058 0x00000364 0x005c 0x0000034d
0x0060 0x00000305 0x0064 0x00000301 0x0068 0x000002cf 0x006c 0x00000297
0x0070 0x000002a3 0x0074 0x00000295 0x0078 0x0000027f 0x007c 0x0000028f
0x0080 0x00000278 0x0084 0x00000260 0x0088 0x00000251 0x008c 0x00000253
0x0090 0x0000024d 0x0094 0x0000023a 0x0098 0x00000245 0x009c 0x00000233
0x00a0 0x0000023f 0x00a4 0x00000207 0x00a8 0x0000022e 0x00ac 0x00000203
0x00b0 0x0000022e 0x00b4 0x00000212 0x00b8 0x00000203 0x00bc 0x0000020b
...
lots of numbers that go down slowly
...
0x0d00 0x0000012e 0x0d04 0x00000127 0x0d08 0x00000138 0x0d0c 0x00000125
0x0d10 0x00000116 0x0d14 0x00000117 0x0d18 0x00000120 0x0d1c 0x0000011f
0x0d20 0x00000135 0x0d24 0x00000120 0x0d28 0x00000111 0x0d2c 0x00000131
0x0d30 0x00000131 0x0d34 0x0000010e 0x0d38 0x00000110 0x0d3c 0x00000114
0x0d40 0x00000100 0x0d44 0x0000011b 0x0d48 0x00000113 0x0d4c 0x00000108
0x0d50 0x000000fe 0x0d54 0x00000103 0x0d58 0x000000f2 0x0d5c 0x00000124
0x0d60 0x000000fb 0x0d64 0x000000fe 0x0d68 0x000000f5 0x0d6c 0x00000114
0x0d70 0x000000f5 0x0d74 0x000000da 0x0d78 0x000000d8 0x0d7c 0x000000f8
0x0d80 0x000000db 0x0d84 0x000000e6 0x0d88 0x00000100 0x0d8c 0x000000d7
0x0d90 0x000000e3 0x0d94 0x000000d6 0x0d98 0x000000cf 0x0d9c 0x000000cf
0x0da0 0x000000d0 0x0da4 0x000000df 0x0da8 0x000000cf 0x0dac 0x000000bb
0x0db0 0x000000d1 0x0db4 0x000000ba 0x0db8 0x000000cc 0x0dbc 0x000000bc
0x0dc0 0x000000b5 0x0dc4 0x000000c5 0x0dc8 0x000000c3 0x0dcc 0x000000b7
0x0dd0 0x000000bd 0x0dd4 0x000000a1 0x0dd8 0x0000009c 0x0ddc 0x000000bc
0x0de0 0x000000ba 0x0de4 0x000000c5 0x0de8 0x0000009d 0x0dec 0x000000a0
0x0df0 0x0000009f 0x0df4 0x00000092 0x0df8 0x000000be 0x0dfc 0x00000089
0x0e00 0x0000007c 0x0e04 0x00000089 0x0e08 0x00000090 0x0e0c 0x0000008a
0x0e10 0x00000081 0x0e14 0x0000007d 0x0e18 0x00000085 0x0e1c 0x00000091
0x0e20 0x00000077 0x0e24 0x0000007e 0x0e28 0x00000076 0x0e2c 0x00000071
0x0e30 0x00000067 0x0e34 0x00000063 0x0e38 0x0000005d 0x0e3c 0x00000067
0x0e40 0x00000070 0x0e44 0x00000068 0x0e48 0x00000060 0x0e4c 0x00000071
0x0e50 0x0000005b 0x0e54 0x00000055 0x0e58 0x00000068 0x0e5c 0x0000004b
0x0e60 0x0000004f 0x0e64 0x00000052 0x0e68 0x00000053 0x0e6c 0x00000041
0x0e70 0x00000049 0x0e74 0x00000055 0x0e78 0x0000005a 0x0e7c 0x00000050
0x0e80 0x0000003f 0x0e84 0x0000003c 0x0e88 0x0000004d 0x0e8c 0x0000003d
0x0e90 0x0000003d 0x0e94 0x00000035 0x0e98 0x0000003a 0x0e9c 0x00000034
0x0ea0 0x00000036 0x0ea4 0x0000003f 0x0ea8 0x00000035 0x0eac 0x0000003d
0x0eb0 0x00000033 0x0eb4 0x0000003b 0x0eb8 0x00000032 0x0ebc 0x00000032
0x0ec0 0x00000027 0x0ec4 0x0000002c 0x0ec8 0x00000030 0x0ecc 0x00000038
0x0ed0 0x0000002e 0x0ed4 0x00000027 0x0ed8 0x00000029 0x0edc 0x00000021
0x0ee0 0x00000026 0x0ee4 0x0000002a 0x0ee8 0x0000002b 0x0eec 0x00000020
0x0ef0 0x0000001d 0x0ef4 0x0000001b 0x0ef8 0x00000028 0x0efc 0x00000028
0x0f00 0x0000001d 0x0f04 0x00000018 0x0f08 0x00000025 0x0f0c 0x00000022
0x0f10 0x00000018 0x0f14 0x00000021 0x0f18 0x0000001d 0x0f1c 0x00000017
0x0f20 0x00000019 0x0f24 0x0000001d 0x0f28 0x0000001d 0x0f2c 0x00000021
0x0f30 0x0000001c 0x0f34 0x00000013 0x0f38 0x00000010 0x0f3c 0x00000025
0x0f40 0x00000016 0x0f44 0x0000001c 0x0f48 0x00000015 0x0f4c 0x00000014
0x0f50 0x0000001c 0x0f54 0x00000010 0x0f58 0x0000001a 0x0f5c 0x00000012
0x0f60 0x00000013 0x0f64 0x00000016 0x0f68 0x00000018 0x0f6c 0x0000000e
0x0f70 0x0000000a 0x0f74 0x00000013 0x0f78 0x00000010 0x0f7c 0x00000019
0x0f80 0x00000019 0x0f84 0x00000014 0x0f88 0x00000015 0x0f8c 0x00000012
0x0f90 0x00000017 0x0f94 0x00000012 0x0f98 0x00000011 0x0f9c 0x00000015
0x0fa0 0x00000010 0x0fa4 0x00000019 0x0fa8 0x00000013 0x0fac 0x00000013
0x0fb0 0x00000015 0x0fb4 0x0000000e 0x0fb8 0x0000000d 0x0fbc 0x0000000e
0x0fc0 0x00000012 0x0fc4 0x00000011 0x0fc8 0x00000014 0x0fcc 0x00000011
0x0fd0 0x00000013 0x0fd4 0x0000000f 0x0fd8 0x0000000e 0x0fdc 0x0000000a
0x0fe0 0x0000000c 0x0fe4 0x0000000f 0x0fe8 0x0000000a 0x0fec 0x00000008
0x0ff0 0x0000000b 0x0ff4 0x00000011 0x0ff8 0x00000010 0x0ffc 0x0000000b
0x1000 0x0000000e 0x1004 0x0000000c 0x1008 0x0000000d 0x100c 0x0000000e
0x1010 0x00000007 0x1014 0x0000000a 0x1018 0x00000013 0x101c 0x00000013
0x1020 0x00000005 0x1024 0x00000007 0x1028 0x00000011 0x102c 0x00000017
0x1030 0x00000010 0x1034 0x00000009 0x1038 0x0000000b 0x103c 0x0000000f
0x1040 0x00000009 0x1044 0x0000000f 0x1048 0x0000000a 0x104c 0x00000009
0x1050 0x0000000c 0x1054 0x0000000b 0x1058 0x00000009 0x105c 0x00000007
0x1060 0x0000000b 0x1064 0x0000000b 0x1068 0x00000008 0x106c 0x00000006
0x1070 0x00000006 0x1074 0x00000010 0x1078 0x00000009 0x107c 0x00000006
0x1080 0x00000009 0x1084 0x00000002 0x1088 0x0000000b 0x108c 0x00000006
0x1090 0x0000000b 0x1094 0x00000009 0x1098 0x0000000b 0x109c 0x00000006
0x10a0 0x00000008 0x10a4 0x00000007 0x10a8 0x00000009 0x10ac 0x00000008
0x10b0 0x00000008 0x10b4 0x00000010 0x10b8 0x00000007 0x10bc 0x00000006
0x10c0 0x00000005 0x10c4 0x00000008 0x10c8 0x00000003 0x10cc 0x00000004
0x10d0 0x00000006 0x10d4 0x00000009 0x10d8 0x00000007 0x10dc 0x00000005
0x10e0 0x00000007 0x10e4 0x00000006 0x10e8 0x00000007 0x10ec 0x00000008
0x10f0 0x00000007 0x10f4 0x0000000c 0x10f8 0x00000006 0x10fc 0x00000008
0x1100 0x00000004 0x1104 0x0000000d 0x1108 0x00000008 0x110c 0x00000005
0x1110 0x00000008 0x1114 0x00000002 0x1118 0x00000007 0x111c 0x00000002
0x1120 0x0000000a 0x1124 0x00000007 0x1128 0x00000001 0x112c 0x00000006
0x1130 0x00000003 0x1134 0x00000002 0x1138 0x00000004 0x113c 0x00000005
0x1140 0x00000007 0x1144 0x00000003 0x1148 0x00000003 0x114c 0x00000002
0x1150 0x00000004 0x1154 0x00000006 0x1158 0x00000005 0x115c 0x00000004
0x1160 0x00000006 0x1164 0x00000003 0x1168 0x00000007 0x116c 0x00000005
0x1170 0x00000006 0x1174 0x00000008 0x1178 0x00000004 0x117c 0x00000005
0x1180 0x00000004 0x1184 0x00000006 0x1188 0x00000003 0x118c 0x00000008
0x1190 0x00000005 0x1194 0x00000004 0x1198 0x00000003 0x119c 0x00000004
0x11a0 0x00000002 0x11a4 0x00000003 0x11a8 0x00000003 0x11ac 0x00000005
0x11b0 0x00000006 0x11b4 0x00000007 0x11b8 0x00000002 0x11bc 0x00000003
0x11c0 0x00000005 0x11c4 0x00000003 0x11c8 0x00000004 0x11cc 0x00000001
0x11d0 0x00000002 0x11d4 0x00000002 0x11d8 0x00000004 0x11dc 0x00000001
0x11e0 0x00000003 0x11e4 0x00000001 0x11e8 0x00000008 0x11ec 0x00000004
0x11f0 0x00000002 0x11f4 0x00000002 0x11f8 0x00000003 0x11fc 0x00000003
0x1200 0x00000001 0x1204 0x00000008 0x1208 0x00000005 0x120c 0x00000003
0x1210 0x00000002 0x1214 0x00000002 0x1218 0x00000002 0x121c 0x00000004
0x1220 0x00000002 0x1224 0x00000002 0x1228 0x00000004 0x122c 0x00000003
0x1230 0x00000003 0x1234 0x00000001 0x1238 0x00000005 0x123c 0x00000004
0x1240 0x00000005 0x1244 0x00000002 0x1248 0x00000001 0x124c 0x00000004
0x1250 0x00000003 0x1254 0x00000003 0x1258 0x00000001 0x125c 0x00000004
0x1260 0x00000003 0x1264 0x00000001 0x1268 0x00000004 0x126c 0x00000002
0x1270 0x00000007 0x1274 0x00000003 0x1278 0x00000001 0x127c 0x00000001
0x1280 0x00000002 0x1284 0x00000004 0x1288 0x00000003 0x128c 0x00000002
0x1290 0x00000003 0x1294 0x00000004 0x1298 0x00000004 0x12a0 0x00000002
0x12a4 0x00000002 0x12a8 0x00000002 0x12ac 0x00000005 0x12b4 0x00000002
0x12b8 0x00000004 0x12bc 0x00000001 0x12c0 0x00000003 0x12c4 0x00000001
0x12cc 0x00000001 0x12d0 0x00000001 0x12d8 0x00000001 0x12dc 0x00000004
0x12e0 0x00000005 0x12e4 0x00000001 0x12e8 0x00000003 0x12ec 0x00000001
0x12f0 0x00000001 0x12f4 0x00000006 0x12f8 0x00000002 0x12fc 0x00000002
0x1300 0x00000004 0x1304 0x00000001 0x1308 0x00000001 0x130c 0x00000001
0x1310 0x00000001 0x1314 0x00000004 0x1318 0x00000002 0x1320 0x00000003
0x1324 0x00000002 0x132c 0x00000001 0x1330 0x00000002 0x1334 0x00000003
0x1338 0x00000003 0x1340 0x00000002 0x1344 0x00000002 0x1350 0x00000001
0x1354 0x00000002 0x1358 0x00000002 0x1360 0x00000003 0x1364 0x00000002
0x1368 0x00000002 0x136c 0x00000002 0x1370 0x00000003 0x1374 0x00000001
0x137c 0x00000002 0x1380 0x00000001 0x1384 0x00000002 0x1388 0x00000003
0x138c 0x00000001 0x1390 0x00000001 0x1394 0x00000001 0x13a0 0x00000002
0x13a4 0x00000001 0x13a8 0x00000002 0x13ac 0x00000001 0x13b8 0x00000002
0x13bc 0x00000001 0x13c4 0x00000004 0x13c8 0x00000005 0x13cc 0x00000001
0x13d4 0x00000002 0x13d8 0x00000001 0x13e0 0x00000004 0x13e8 0x00000002
0x13f0 0x00000001 0x13f4 0x00000001 0x13f8 0x00000002 0x13fc 0x00000001
0x1400 0x00000002 0x1408 0x00000001 0x1410 0x00000003 0x1418 0x00000001
0x1420 0x00000001 0x1424 0x00000002 0x1428 0x00000001 0x1434 0x00000001
0x143c 0x00000001 0x1440 0x00000003 0x1444 0x00000001 0x1448 0x00000001
0x1450 0x00000001 0x1458 0x00000001 0x1464 0x00000002 0x1468 0x00000001
0x146c 0x00000004 0x1474 0x00000001 0x1478 0x00000003 0x147c 0x00000002
0x148c 0x00000001 0x1490 0x00000001 0x1494 0x00000001 0x1498 0x00000001
0x14a0 0x00000001 0x14a4 0x00000002 0x14ac 0x00000002 0x14bc 0x00000003
0x14c4 0x00000001 0x14cc 0x00000001 0x14d0 0x00000002 0x14d4 0x00000001
0x14d8 0x00000001 0x14dc 0x00000001 0x14e8 0x00000002 0x14f0 0x00000002
0x1500 0x00000001 0x150c 0x00000001 0x1510 0x00000001 0x1514 0x00000002
0x151c 0x00000001 0x1520 0x00000001 0x1524 0x00000001 0x1528 0x00000001
0x152c 0x00000001 0x1530 0x00000001 0x153c 0x00000001 0x154c 0x00000001
0x155c 0x00000001 0x1560 0x00000003 0x1568 0x00000001 0x156c 0x00000001
0x1578 0x00000001 0x1580 0x00000001 0x1594 0x00000001 0x159c 0x00000001
0x15a0 0x00000001 0x15a4 0x00000001 0x15a8 0x00000001 0x15b4 0x00000002
0x15b8 0x00000001 0x15c0 0x00000002 0x15d0 0x00000001 0x15e0 0x00000002
0x15ec 0x00000001 0x160c 0x00000001 0x1610 0x00000001 0x1614 0x00000001
0x161c 0x00000001 0x1638 0x00000001 0x163c 0x00000001 0x1654 0x00000001
0x1658 0x00000001 0x1668 0x00000001 0x1670 0x00000001 0x1678 0x00000002
0x168c 0x00000002 0x1690 0x00000001 0x16a0 0x00000001 0x16b0 0x00000001
0x16c0 0x00000001 0x16e8 0x00000001 0x16ec 0x00000002 0x16f4 0x00000001
0x1710 0x00000001 0x1718 0x00000001 0x1730 0x00000001 0x1740 0x00000002
0x1748 0x00000001 0x175c 0x00000001 0x1760 0x00000002 0x178c 0x00000001
0x17ac 0x00000001 0x17d0 0x00000001 0x17d8 0x00000001 0x17e0 0x00000001
0x1810 0x00000001 0x1820 0x00000001 0x1824 0x00000001 0x1838 0x00000001
0x1850 0x00000001 0x1854 0x00000001 0x1858 0x00000001 0x1868 0x00000001
0x188c 0x00000001 0x1894 0x00000001 0x18cc 0x00000001 0x18d8 0x00000001
0x18f8 0x00000001 0x18fc 0x00000001 0x1900 0x00000001 0x1954 0x00000003
0x196c 0x00000001 0x19d8 0x00000001 0x19e4 0x00000003 0x1a50 0x00000001
0x1a64 0x00000001 0x1ab0 0x00000001 0x1ac8 0x00000001 0x1ae4 0x00000001
0x1b20 0x00000001 0x1b60 0x00000001 0x1b64 0x00000001 0x1be4 0x00000001
0x1c5c 0x00000001 0x1ce0 0x00000001

The last entry is for 1848 pending packets.

I've done some other measurements:

tg3_rx_prodring_xfer() very rarely copied more than 64 items or
finds a 'busy' slot (ie err set).

Processing 4 rx packets for every napi budget doesn't make any
difference.

It seems like there can be a significant delay between the hardware
interrupt and the napi/softint.

I'm not sure of how to instrument that though.
But it is likely to affect other ethernet drivers.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Powered by blists - more mailing lists