Message-ID: <CACna6rwaa8xkqpt74aDouHjwgqkomYEtpCO3kTX7eVKBPanNzg@mail.gmail.com>
Date:	Sat, 24 Aug 2013 17:19:53 +0200
From:	Rafał Miłecki <zajec5@...il.com>
To:	OpenWrt Development List <openwrt-devel@...ts.openwrt.org>,
	Network Development <netdev@...r.kernel.org>
Subject: Understanding/reimplementing forwarding acceleration used by Broadcom (ctf)

Recently I've finally discovered what makes packets forwarding much
faster when using Broadcom's firmware for their routers. It seems
pretty obvious for some DD-WRT guys, but I guess noone of us knew
about it.

It's the magic ctf.ko module that does the trick. It's a closed source
kernel module, that interferences with kernel using simple symbol
exported in setup.c:
ctf_attach_t ctf_attach_fn = NULL;
EXPORT_SYMBOL(ctf_attach_fn);

CTF probably stands for Cut-Through Forwarding.

Removing this module (rmmod ctf) resulted in performance drop on my
WNDR4500 from 505 Mbits/sec to 379 Mbits/sec for LAN to WAN
iperf-tested traffic. My friend tested this on his BCM4706-based
router and removing ctf.ko module resulted in 850Mb/s to 120Mb/s drop.

When ctf.ko gets loaded, it probably just replaces NULL with it's
struct allowing kernel to use it's mysterious API. If you wonder what
part of Linux kernel was modified to make use of ctf.ko, see attached
diff, that's what Broadcom modified in the net tree.

I don't have so good understanding of the net layer, so your help on
that would be more than appreciated.

AFAIK this magic kernel module allows some packets forwarding to be
done directly, with skipping packets analyze in the OS. It means some
features can't be used with it (like QoS), but it still looks like a
nice solution for basic routing.
It looks for me that ctf uses two connections types:
1) brc which may stand for Bridge Connection
2) ipc which may stand for IP Connection

You can see struct ctf_brc and struct ctf_ipc in the attached
hndctf.h. I didn't focus on ctf_brc yet, but I tried to analyze
ctf_ipc. It contains set of data allowing detection of the specific IP
connection. Pretty obviously, it means storing protocol, source (MAC,
IP, port), destination (MAC, IP, port) and few others (like VLAN id,
NAT rules, target interface) values. I guess that ctf.ko analyzes
every received packet using rules stored in struct ctf_ipc. When it
gets some packet covered in it's struct ctf_ipc database, it doesn't
pass it to the OS, but modifies it itself (if needed) and transmits to
the target.
You can see that nf_nat_packet was modified to call
ip_conntrack_ipct_add which tests if the packets matches it's
requirements. If it does, a new IP forwarding rule is added using
ctf_ipc_add.

I wonder what do you think about this solution. Is this something we
could try to implement ourself? Is it worth it? Is there some existing
project doing similar thing?

While ctf.ko module is closed source, it should be possible to
re-implement it. It probably just keeps a list of rules and does some
simple modifications of the received packets (like VLAN id change,
IP/port hacking for SNAT/DNAT), etc.

I don't have any experience in hacking net layer, so any comments on
that would be great!

-- 
Rafał

View attachment "hndctf.h" of type "text/x-chdr" (10072 bytes)

Download attachment "ctf.diff" of type "application/octet-stream" (35155 bytes)

Powered by blists - more mailing lists