netdev - PROBLEM: IPv4 route exception patch breaks BIRD2

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <8da4a321-a676-0464-f5cf-d467840dbfa7@arctarus.co.uk>
Date:   Thu, 11 Jun 2020 19:10:44 +0100
From:   Joseph Marsden <joseph@...tarus.co.uk>
To:     davem@...emloft.net
Cc:     netdev@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: PROBLEM: IPv4 route exception patch breaks BIRD2

Hi all,

While upgrading my kernel from 5.2 to 5.4.43, I noticed a very sharp
increase in CPU usage - coming from BIRD, the routing daemon on my system.
I ran a Netlink capture and saw that BIRD was attempting to insert
routes twice into the kernel routing table. This was causing routes to
be re-inserted rapidly, and explained the increase of CPU usage.

I'm running IPv4 and IPv6 full tables - about 789,360 v4 routes and
78,185 v6 routes. About 10,000 routes fail to be re-inserted and this
number fluctuated over time as the routing table was updated. The routes
themselves are sourced from BGP peerings with upstreams on the router.

I tried downgrading BIRD itself several times before realising that it
must be a kernel issue, so I performed a bisect, and found this patch to
be the issue:

[PATCH net-next v7 04/11] ipv4: Dump route exceptions if requested
https://lore.kernel.org/netdev/8d3b68cd37fb5fddc470904cdd6793fcf480c6c1.1561131177.git.sbrivio@redhat.com/

When I added some debug logging to BIRD, I could see that before the
patch BIRD was not receiving any FIB exception routes, but after the
patch, it was receiving many.
I suspect this is unrelated to the bigger issue here - even if I patch
BIRD to reject these routes, the problem still occurs and BIRD keeps
trying to re-insert routes into the kernel routing table twice.

- Joseph

Extra information:
Kernel version: Linux version 5.4.43 (nixbld@...alhost) (gcc version
9.2.0 (GCC)) #1-NixOS SMP Wed May 27 15:46:53 UTC 2020
Most recent kernel version which did not have the bug: 5.2.0
BIRD version I tested this on: 2.0.7 (also tested on 2.0.2 and other
older versions)

ver_linux output:
Linux bdr1 5.4.43 #1-NixOS SMP Wed May 27 15:46:53 UTC 2020 x86_64 GNU/Linux

Util-linux          	2.33.2
Mount               	2.33.2
Module-init-tools   	26
E2fsprogs           	1.45.5
Linux C Library     	2.30
Dynamic linker (ldd)	2.30
Procps              	3.3.16
Net-tools           	1.60
Kbd                 	2.0.4
Console-tools       	2.0.4
Sh-utils            	8.31
Udev                	243
Modules Loaded      	agpgart ata_generic ata_piix atkbd autofs4
bochs_drm bridge br_netfilter button cdrom crc16 crc32c_generic
crc_ccitt dm_mod drm drm_kms_helper drm_vram_helper dummy ehci_hcd evdev
ext4 failover fb_sys_fops floppy gre hid hid_generic i2c_core i2c_piix4
i8042 input_leds intel_agp intel_gtt ip6_gre ip6table_filter
ip6table_nat ip6table_raw ip6_tables ip6_tunnel iptable_filter
iptable_nat iptable_raw ip_tables ipv6 jbd2 joydev led_class libata
libcrc32c libps2 llc loop mac_hid macvlan mbcache mousedev net_failover
nf_conntrack nf_conntrack_netlink nf_defrag_ipv4 nf_defrag_ipv6
nf_log_common nf_log_ipv4 nf_log_ipv6 nf_nat nfnetlink overlay pata_acpi
psmouse qemu_fw_cfg rng_core rtc_cmos sch_fq_codel scsi_mod serio
serio_raw sr_mod stp syscopyarea sysfillrect sysimgblt tap ttm tun
tunnel6 uhci_hcd usb_common usbcore usbhid veth virtio virtio_balloon
virtio_blk virtio_console virtio_net virtio_pci virtio_ring virtio_rng
xfrm_algo xfrm_user x_tables xt_addrtype xt_conntrack xt_LOG
xt_MASQUERADE xt_multiport xt_nat xt_pkttype xt_tcpudp