| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <20130423160438.2479.C42C3789@sakura.ad.jp>
Date: Tue, 23 Apr 2013 16:04:39 +0900
From: Naoto MATSUMOTO <n-matsumoto@...ura.ad.jp>
To: Stephen Hemminger <stephen@...workplumber.org>
Cc: David Miller <davem@...emloft.net>, Greg KH <greg@...ah.com>,
netdev@...r.kernel.org
Subject: Re: [RFC 1/2] netmap: infrastructure (in staging)
Hi all
Various10GbE-NIC 64Byte Short Packet Processing Bechmark (RX) with
netmap memo share for you. http://twitpic.com/clb97k
FYI: Another netmap benchmark with DPDK result.
Disruptive IP Networking with Intel DPDK on Linux
http://slidesha.re/SeVFZn
enjoy!
--
Naoto
On Fri, 19 Apr 2013 12:06:51 -0700
Stephen Hemminger <stephen@...workplumber.org> wrote:
> Netmap is a framework for packet generation and capture from user
> space. It allows for efficient packet handling (up to line rate on
> 10Gb) with minimum system load. For more info see:
> http://info.iet.unipi.it/~luigi/netmap/
>
> This version is based on the latest version from Luigi Rizzo. It has
> been modified to live to work with current net-next kernel. It still
> has all the BSD ugliness, and that is why it needs to spend some time
> in the staging penalty box.
>
> It builds and loads, but definitely needs more work.
>
> Signed-off-by: Stephen Hemminger <stephen@...workplumger.org>
>
> ---
> Patch is against net-next
>
> drivers/staging/Kconfig | 2
> drivers/staging/Makefile | 1
> drivers/staging/netmap/Kconfig | 16
> drivers/staging/netmap/Makefile | 2
> drivers/staging/netmap/README | 127 +
> drivers/staging/netmap/TODO | 16
> drivers/staging/netmap/netmap.c | 2514 +++++++++++++++++++++++++++++++++++
> drivers/staging/netmap/netmap_mem2.c | 974 +++++++++++++
> include/netmap/bsd_glue.h | 263 +++
> include/netmap/netmap_kern.h | 474 ++++++
> include/uapi/Kbuild | 1
> include/uapi/linux/if.h | 1
> include/uapi/netmap/Kbuild | 3
> include/uapi/netmap/netmap.h | 289 ++++
> include/uapi/netmap/netmap_user.h | 95 +
> 15 files changed, 4778 insertions(+)
>
> --- a/drivers/staging/Kconfig 2013-02-26 10:19:35.000000000 -0800
> +++ b/drivers/staging/Kconfig 2013-03-10 10:08:20.323671490 -0700
> @@ -140,4 +140,6 @@ source "drivers/staging/zcache/Kconfig"
>
> source "drivers/staging/goldfish/Kconfig"
>
> +source "drivers/staging/netmap/Kconfig"
> +
> endif # STAGING
> --- a/drivers/staging/Makefile 2013-02-26 10:19:35.000000000 -0800
> +++ b/drivers/staging/Makefile 2013-03-10 10:08:48.555305267 -0700
> @@ -62,3 +62,4 @@ obj-$(CONFIG_SB105X) += sb105x/
> obj-$(CONFIG_FIREWIRE_SERIAL) += fwserial/
> obj-$(CONFIG_ZCACHE) += zcache/
> obj-$(CONFIG_GOLDFISH) += goldfish/
> +obj-$(CONFIG_NETMAP) += netmap/
> --- /dev/null 1970-01-01 00:00:00.000000000 +0000
> +++ b/drivers/staging/netmap/Kconfig 2013-03-10 10:08:20.323671490 -0700
> @@ -0,0 +1,16 @@
> +#
> +# Netmap - user mode packet processing framework
> +#
> +
> +config NETMAP
> + tristate "Netmap - user mode networking"
> + depends on NET && !IOMMU_API
> + default n
> + help
> + If you say Y here, you will get experimental support for
> + netmap a framework for fast packet I/O using memory mapped
> + buffers.
> +
> + See <http:/nfo.iet.unipi.it/~luigi/netmap/> for more information.
> +
> + If unsure, say N.
> --- /dev/null 1970-01-01 00:00:00.000000000 +0000
> +++ b/drivers/staging/netmap/Makefile 2013-03-10 10:08:20.323671490 -0700
> @@ -0,0 +1,2 @@
> +EXTRA_CFLAGS := -DDEBUG
> +obj-$(CONFIG_NETMAP) += netmap.o
> --- /dev/null 1970-01-01 00:00:00.000000000 +0000
> +++ b/drivers/staging/netmap/netmap.c 2013-03-10 16:52:34.160592316 -0700
> @@ -0,0 +1,2515 @@
> +/*
> + * Copyright (C) 2011-2012 Matteo Landi, Luigi Rizzo. All rights reserved.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + * notice, this list of conditions and the following disclaimer.
> + * 2. Redistributions in binary form must reproduce the above copyright
> + * notice, this list of conditions and the following disclaimer in the
> + * documentation and/or other materials provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
> + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
> + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
> + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +#define NM_BRIDGE
> +
> +/*
> + * This module supports memory mapped access to network devices,
> + * see netmap(4).
> + *
> + * The module uses a large, memory pool allocated by the kernel
> + * and accessible as mmapped memory by multiple userspace threads/processes.
> + * The memory pool contains packet buffers and "netmap rings",
> + * i.e. user-accessible copies of the interface's queues.
> + *
> + * Access to the network card works like this:
> + * 1. a process/thread issues one or more open() on /dev/netmap, to create
> + * select()able file descriptor on which events are reported.
> + * 2. on each descriptor, the process issues an ioctl() to identify
> + * the interface that should report events to the file descriptor.
> + * 3. on each descriptor, the process issues an mmap() request to
> + * map the shared memory region within the process' address space.
> + * The list of interesting queues is indicated by a location in
> + * the shared memory region.
> + * 4. using the functions in the netmap(4) userspace API, a process
> + * can look up the occupation state of a queue, access memory buffers,
> + * and retrieve received packets or enqueue packets to transmit.
> + * 5. using some ioctl()s the process can synchronize the userspace view
> + * of the queue with the actual status in the kernel. This includes both
> + * receiving the notification of new packets, and transmitting new
> + * packets on the output interface.
> + * 6. select() or poll() can be used to wait for events on individual
> + * transmit or receive queues (or all queues for a given interface).
> + */
> +
> +#ifdef linux
> +#include <netmap/bsd_glue.h>
> +static netdev_tx_t linux_netmap_start(struct sk_buff *skb, struct net_device *dev);
> +#endif /* linux */
> +
> +#ifdef __APPLE__
> +#include "osx_glue.h"
> +#endif /* __APPLE__ */
> +
> +#ifdef __FreeBSD__
> +#include <sys/cdefs.h> /* prerequisite */
> +__FBSDID("$FreeBSD: head/sys/dev/netmap/netmap.c 241723 2012-10-19 09:41:45Z glebius $");
> +
> +#include <sys/types.h>
> +#include <sys/module.h>
> +#include <sys/errno.h>
> +#include <sys/param.h> /* defines used in kernel.h */
> +#include <sys/jail.h>
> +#include <sys/kernel.h> /* types used in module initialization */
> +#include <sys/conf.h> /* cdevsw struct */
> +#include <sys/uio.h> /* uio struct */
> +#include <sys/sockio.h>
> +#include <sys/socketvar.h> /* struct socket */
> +#include <sys/malloc.h>
> +#include <sys/mman.h> /* PROT_EXEC */
> +#include <sys/poll.h>
> +#include <sys/proc.h>
> +#include <vm/vm.h> /* vtophys */
> +#include <vm/pmap.h> /* vtophys */
> +#include <sys/socket.h> /* sockaddrs */
> +#include <machine/bus.h>
> +#include <sys/selinfo.h>
> +#include <sys/sysctl.h>
> +#include <net/if.h>
> +#include <net/bpf.h> /* BIOCIMMEDIATE */
> +#include <net/vnet.h>
> +#include <machine/bus.h> /* bus_dmamap_* */
> +
> +MALLOC_DEFINE(M_NETMAP, "netmap", "Network memory map");
> +#endif /* __FreeBSD__ */
> +
> +#include <netmap/netmap.h>
> +#include <netmap/netmap_kern.h>
> +
> +u_int netmap_total_buffers;
> +u_int netmap_buf_size;
> +char *netmap_buffer_base; /* address of an invalid buffer */
> +
> +/* user-controlled variables */
> +int netmap_verbose = 1;
> +
> +static int netmap_no_timestamp; /* don't timestamp on rxsync */
> +
> +SYSCTL_NODE(_dev, OID_AUTO, netmap, CTLFLAG_RW, 0, "Netmap args");
> +SYSCTL_INT(_dev_netmap, OID_AUTO, verbose,
> + CTLFLAG_RW, &netmap_verbose, 0, "Verbose mode");
> +SYSCTL_INT(_dev_netmap, OID_AUTO, no_timestamp,
> + CTLFLAG_RW, &netmap_no_timestamp, 0, "no_timestamp");
> +int netmap_mitigate = 1;
> +SYSCTL_INT(_dev_netmap, OID_AUTO, mitigate, CTLFLAG_RW, &netmap_mitigate, 0, "");
> +int netmap_no_pendintr = 1;
> +SYSCTL_INT(_dev_netmap, OID_AUTO, no_pendintr,
> + CTLFLAG_RW, &netmap_no_pendintr, 0, "Always look for new received packets.");
> +
> +int netmap_drop = 0; /* debugging */
> +int netmap_flags = 0; /* debug flags */
> +int netmap_fwd = 0; /* force transparent mode */
> +int netmap_copy = 0; /* debugging, copy content */
> +
> +SYSCTL_INT(_dev_netmap, OID_AUTO, drop, CTLFLAG_RW, &netmap_drop, 0 , "");
> +SYSCTL_INT(_dev_netmap, OID_AUTO, flags, CTLFLAG_RW, &netmap_flags, 0 , "");
> +SYSCTL_INT(_dev_netmap, OID_AUTO, fwd, CTLFLAG_RW, &netmap_fwd, 0 , "");
> +SYSCTL_INT(_dev_netmap, OID_AUTO, copy, CTLFLAG_RW, &netmap_copy, 0 , "");
> +
> +#ifdef NM_BRIDGE /* support for netmap bridge */
> +
> +/*
> + * system parameters.
> + *
> + * All switched ports have prefix NM_NAME.
> + * The switch has a max of NM_BDG_MAXPORTS ports (often stored in a bitmap,
> + * so a practical upper bound is 64).
> + * Each tx ring is read-write, whereas rx rings are readonly (XXX not done yet).
> + * The virtual interfaces use per-queue lock instead of core lock.
> + * In the tx loop, we aggregate traffic in batches to make all operations
> + * faster. The batch size is NM_BDG_BATCH
> + */
> +#define NM_NAME "vale" /* prefix for the interface */
> +#define NM_BDG_MAXPORTS 16 /* up to 64 ? */
> +#define NM_BRIDGE_RINGSIZE 1024 /* in the device */
> +#define NM_BDG_HASH 1024 /* forwarding table entries */
> +#define NM_BDG_BATCH 1024 /* entries in the forwarding buffer */
> +#define NM_BRIDGES 4 /* number of bridges */
> +int netmap_bridge = NM_BDG_BATCH; /* bridge batch size */
> +SYSCTL_INT(_dev_netmap, OID_AUTO, bridge, CTLFLAG_RW, &netmap_bridge, 0 , "");
> +
> +#ifdef linux
> +#define ADD_BDG_REF(ifp) (NA(ifp)->if_refcount++)
> +#define DROP_BDG_REF(ifp) (NA(ifp)->if_refcount-- <= 1)
> +#else /* !linux */
> +#define ADD_BDG_REF(ifp) (ifp)->if_refcount++
> +#define DROP_BDG_REF(ifp) refcount_release(&(ifp)->if_refcount)
> +#ifdef __FreeBSD__
> +#include <sys/endian.h>
> +#include <sys/refcount.h>
> +#endif /* __FreeBSD__ */
> +#define prefetch(x) __builtin_prefetch(x)
> +#endif /* !linux */
> +
> +static void bdg_netmap_attach(struct ifnet *ifp);
> +static int bdg_netmap_reg(struct ifnet *ifp, int onoff);
> +/* per-tx-queue entry */
> +struct nm_bdg_fwd { /* forwarding entry for a bridge */
> + void *buf;
> + uint64_t dst; /* dst mask */
> + uint32_t src; /* src index ? */
> + uint16_t len; /* src len */
> +};
> +
> +struct nm_hash_ent {
> + uint64_t mac; /* the top 2 bytes are the epoch */
> + uint64_t ports;
> +};
> +
> +/*
> + * Interfaces for a bridge are all in ports[].
> + * The array has fixed size, an empty entry does not terminate
> + * the search.
> + */
> +struct nm_bridge {
> + struct ifnet *bdg_ports[NM_BDG_MAXPORTS];
> + int n_ports;
> + uint64_t act_ports;
> + int freelist; /* first buffer index */
> + NM_SELINFO_T si; /* poll/select wait queue */
> + NM_LOCK_T bdg_lock; /* protect the selinfo ? */
> +
> + /* the forwarding table, MAC+ports */
> + struct nm_hash_ent ht[NM_BDG_HASH];
> +
> + int namelen; /* 0 means free */
> + char basename[IFNAMSIZ];
> +};
> +
> +struct nm_bridge nm_bridges[NM_BRIDGES];
> +
> +#define BDG_LOCK(b) mtx_lock(&(b)->bdg_lock)
> +#define BDG_UNLOCK(b) mtx_unlock(&(b)->bdg_lock)
> +
> +/*
> + * NA(ifp)->bdg_port port index
> + */
> +
> +// XXX only for multiples of 64 bytes, non overlapped.
> +static inline void
> +pkt_copy(void *_src, void *_dst, int l)
> +{
> + uint64_t *src = _src;
> + uint64_t *dst = _dst;
> + if (unlikely(l >= 1024)) {
> + bcopy(src, dst, l);
> + return;
> + }
> + for (; likely(l > 0); l-=64) {
> + *dst++ = *src++;
> + *dst++ = *src++;
> + *dst++ = *src++;
> + *dst++ = *src++;
> + *dst++ = *src++;
> + *dst++ = *src++;
> + *dst++ = *src++;
> + *dst++ = *src++;
> + }
> +}
> +
> +/*
> + * locate a bridge among the existing ones.
> + * a ':' in the name terminates the bridge name. Otherwise, just NM_NAME.
> + * We assume that this is called with a name of at least NM_NAME chars.
> + */
> +static struct nm_bridge *
> +nm_find_bridge(const char *name)
> +{
> + int i, l, namelen, e;
> + struct nm_bridge *b = NULL;
> +
> + namelen = strlen(NM_NAME); /* base length */
> + l = strlen(name); /* actual length */
> + for (i = namelen + 1; i < l; i++) {
> + if (name[i] == ':') {
> + namelen = i;
> + break;
> + }
> + }
> + if (namelen >= IFNAMSIZ)
> + namelen = IFNAMSIZ;
> + ND("--- prefix is '%.*s' ---", namelen, name);
> +
> + /* use the first entry for locking */
> + BDG_LOCK(nm_bridges); // XXX do better
> + for (e = -1, i = 1; i < NM_BRIDGES; i++) {
> + b = nm_bridges + i;
> + if (b->namelen == 0)
> + e = i; /* record empty slot */
> + else if (strncmp(name, b->basename, namelen) == 0) {
> + ND("found '%.*s' at %d", namelen, name, i);
> + break;
> + }
> + }
> + if (i == NM_BRIDGES) { /* all full */
> + if (e == -1) { /* no empty slot */
> + b = NULL;
> + } else {
> + b = nm_bridges + e;
> + strncpy(b->basename, name, namelen);
> + b->namelen = namelen;
> + }
> + }
> + BDG_UNLOCK(nm_bridges);
> + return b;
> +}
> +#endif /* NM_BRIDGE */
> +
> +
> +/*
> + * Fetch configuration from the device, to cope with dynamic
> + * reconfigurations after loading the module.
> + */
> +static int
> +netmap_update_config(struct netmap_adapter *na)
> +{
> + struct ifnet *ifp = na->ifp;
> + u_int txr, txd, rxr, rxd;
> +
> + txr = txd = rxr = rxd = 0;
> + if (na->nm_config) {
> + na->nm_config(ifp, &txr, &txd, &rxr, &rxd);
> + } else {
> + /* take whatever we had at init time */
> + txr = na->num_tx_rings;
> + txd = na->num_tx_desc;
> + rxr = na->num_rx_rings;
> + rxd = na->num_rx_desc;
> + }
> +
> + if (na->num_tx_rings == txr && na->num_tx_desc == txd &&
> + na->num_rx_rings == rxr && na->num_rx_desc == rxd)
> + return 0; /* nothing changed */
> + if (netmap_verbose || na->refcount > 0) {
> + D("stored config %s: txring %d x %d, rxring %d x %d",
> + ifp->if_xname,
> + na->num_tx_rings, na->num_tx_desc,
> + na->num_rx_rings, na->num_rx_desc);
> + D("new config %s: txring %d x %d, rxring %d x %d",
> + ifp->if_xname, txr, txd, rxr, rxd);
> + }
> + if (na->refcount == 0) {
> + D("configuration changed (but fine)");
> + na->num_tx_rings = txr;
> + na->num_tx_desc = txd;
> + na->num_rx_rings = rxr;
> + na->num_rx_desc = rxd;
> + return 0;
> + }
> + D("configuration changed while active, this is bad...");
> + return 1;
> +}
> +
> +/*------------- memory allocator -----------------*/
> +#ifdef NETMAP_MEM2
> +#include "netmap_mem2.c"
> +#else /* !NETMAP_MEM2 */
> +#include "netmap_mem1.c"
> +#endif /* !NETMAP_MEM2 */
> +/*------------ end of memory allocator ----------*/
> +
> +
> +/* Structure associated to each thread which registered an interface.
> + *
> + * The first 4 fields of this structure are written by NIOCREGIF and
> + * read by poll() and NIOC?XSYNC.
> + * There is low contention among writers (actually, a correct user program
> + * should have no contention among writers) and among writers and readers,
> + * so we use a single global lock to protect the structure initialization.
> + * Since initialization involves the allocation of memory, we reuse the memory
> + * allocator lock.
> + * Read access to the structure is lock free. Readers must check that
> + * np_nifp is not NULL before using the other fields.
> + * If np_nifp is NULL initialization has not been performed, so they should
> + * return an error to userlevel.
> + *
> + * The ref_done field is used to regulate access to the refcount in the
> + * memory allocator. The refcount must be incremented at most once for
> + * each open("/dev/netmap"). The increment is performed by the first
> + * function that calls netmap_get_memory() (currently called by
> + * mmap(), NIOCGINFO and NIOCREGIF).
> + * If the refcount is incremented, it is then decremented when the
> + * private structure is destroyed.
> + */
> +struct netmap_priv_d {
> + struct netmap_if * volatile np_nifp; /* netmap interface descriptor. */
> +
> + struct ifnet *np_ifp; /* device for which we hold a reference */
> + int np_ringid; /* from the ioctl */
> + u_int np_qfirst, np_qlast; /* range of rings to scan */
> + uint16_t np_txpoll;
> +
> + unsigned long ref_done; /* use with NMA_LOCK held */
> +};
> +
> +
> +static int
> +netmap_get_memory(struct netmap_priv_d* p)
> +{
> + int error = 0;
> + NMA_LOCK();
> + if (!p->ref_done) {
> + error = netmap_memory_finalize();
> + if (!error)
> + p->ref_done = 1;
> + }
> + NMA_UNLOCK();
> + return error;
> +}
> +
> +/*
> + * File descriptor's private data destructor.
> + *
> + * Call nm_register(ifp,0) to stop netmap mode on the interface and
> + * revert to normal operation. We expect that np_ifp has not gone.
> + */
> +/* call with NMA_LOCK held */
> +static void
> +netmap_dtor_locked(void *data)
> +{
> + struct netmap_priv_d *priv = data;
> + struct ifnet *ifp = priv->np_ifp;
> + struct netmap_adapter *na = NA(ifp);
> + struct netmap_if *nifp = priv->np_nifp;
> +
> + na->refcount--;
> + if (na->refcount <= 0) { /* last instance */
> + u_int i, j, lim;
> +
> + if (netmap_verbose)
> + D("deleting last instance for %s", ifp->if_xname);
> + /*
> + * there is a race here with *_netmap_task() and
> + * netmap_poll(), which don't run under NETMAP_REG_LOCK.
> + * na->refcount == 0 && na->ifp->if_capenable & IFCAP_NETMAP
> + * (aka NETMAP_DELETING(na)) are a unique marker that the
> + * device is dying.
> + * Before destroying stuff we sleep a bit, and then complete
> + * the job. NIOCREG should realize the condition and
> + * loop until they can continue; the other routines
> + * should check the condition at entry and quit if
> + * they cannot run.
> + */
> + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0);
> + tsleep(na, 0, "NIOCUNREG", 4);
> + na->nm_lock(ifp, NETMAP_REG_LOCK, 0);
> + na->nm_register(ifp, 0); /* off, clear IFCAP_NETMAP */
> + /* Wake up any sleeping threads. netmap_poll will
> + * then return POLLERR
> + */
> + for (i = 0; i < na->num_tx_rings + 1; i++)
> + selwakeuppri(&na->tx_rings[i].si, PI_NET);
> + for (i = 0; i < na->num_rx_rings + 1; i++)
> + selwakeuppri(&na->rx_rings[i].si, PI_NET);
> + selwakeuppri(&na->tx_si, PI_NET);
> + selwakeuppri(&na->rx_si, PI_NET);
> + /* release all buffers */
> + for (i = 0; i < na->num_tx_rings + 1; i++) {
> + struct netmap_ring *ring = na->tx_rings[i].ring;
> + lim = na->tx_rings[i].nkr_num_slots;
> + for (j = 0; j < lim; j++)
> + netmap_free_buf(nifp, ring->slot[j].buf_idx);
> + /* knlist_destroy(&na->tx_rings[i].si.si_note); */
> + mtx_destroy(&na->tx_rings[i].q_lock);
> + }
> + for (i = 0; i < na->num_rx_rings + 1; i++) {
> + struct netmap_ring *ring = na->rx_rings[i].ring;
> + lim = na->rx_rings[i].nkr_num_slots;
> + for (j = 0; j < lim; j++)
> + netmap_free_buf(nifp, ring->slot[j].buf_idx);
> + /* knlist_destroy(&na->rx_rings[i].si.si_note); */
> + mtx_destroy(&na->rx_rings[i].q_lock);
> + }
> + /* XXX kqueue(9) needed; these will mirror knlist_init. */
> + /* knlist_destroy(&na->tx_si.si_note); */
> + /* knlist_destroy(&na->rx_si.si_note); */
> + netmap_free_rings(na);
> + wakeup(na);
> + }
> + netmap_if_free(nifp);
> +}
> +
> +static void
> +nm_if_rele(struct ifnet *ifp)
> +{
> +#ifndef NM_BRIDGE
> + if_rele(ifp);
> +#else /* NM_BRIDGE */
> + int i, full;
> + struct nm_bridge *b;
> +
> + if (strncmp(ifp->if_xname, NM_NAME, sizeof(NM_NAME) - 1)) {
> + if_rele(ifp);
> + return;
> + }
> + if (!DROP_BDG_REF(ifp))
> + return;
> + b = ifp->if_bridge;
> + BDG_LOCK(nm_bridges);
> + BDG_LOCK(b);
> + ND("want to disconnect %s from the bridge", ifp->if_xname);
> + full = 0;
> + for (i = 0; i < NM_BDG_MAXPORTS; i++) {
> + if (b->bdg_ports[i] == ifp) {
> + b->bdg_ports[i] = NULL;
> + bzero(ifp, sizeof(*ifp));
> + free(ifp, M_DEVBUF);
> + break;
> + }
> + else if (b->bdg_ports[i] != NULL)
> + full = 1;
> + }
> + BDG_UNLOCK(b);
> + if (full == 0) {
> + ND("freeing bridge %d", b - nm_bridges);
> + b->namelen = 0;
> + }
> + BDG_UNLOCK(nm_bridges);
> + if (i == NM_BDG_MAXPORTS)
> + D("ouch, cannot find ifp to remove");
> +#endif /* NM_BRIDGE */
> +}
> +
> +static void
> +netmap_dtor(void *data)
> +{
> + struct netmap_priv_d *priv = data;
> + struct ifnet *ifp = priv->np_ifp;
> + struct netmap_adapter *na;
> +
> + NMA_LOCK();
> + if (ifp) {
> + na = NA(ifp);
> + na->nm_lock(ifp, NETMAP_REG_LOCK, 0);
> + netmap_dtor_locked(data);
> + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0);
> +
> + nm_if_rele(ifp);
> + }
> + if (priv->ref_done) {
> + netmap_memory_deref();
> + }
> + NMA_UNLOCK();
> + bzero(priv, sizeof(*priv)); /* XXX for safety */
> + free(priv, M_DEVBUF);
> +}
> +
> +#ifdef __FreeBSD__
> +#include <vm/vm.h>
> +#include <vm/vm_param.h>
> +#include <vm/vm_object.h>
> +#include <vm/vm_page.h>
> +#include <vm/vm_pager.h>
> +#include <vm/uma.h>
> +
> +static struct cdev_pager_ops saved_cdev_pager_ops;
> +
> +static int
> +netmap_dev_pager_ctor(void *handle, vm_ooffset_t size, vm_prot_t prot,
> + vm_ooffset_t foff, struct ucred *cred, u_short *color)
> +{
> + if (netmap_verbose)
> + D("first mmap for %p", handle);
> + return saved_cdev_pager_ops.cdev_pg_ctor(handle,
> + size, prot, foff, cred, color);
> +}
> +
> +static void
> +netmap_dev_pager_dtor(void *handle)
> +{
> + saved_cdev_pager_ops.cdev_pg_dtor(handle);
> + ND("ready to release memory for %p", handle);
> +}
> +
> +
> +static struct cdev_pager_ops netmap_cdev_pager_ops = {
> + .cdev_pg_ctor = netmap_dev_pager_ctor,
> + .cdev_pg_dtor = netmap_dev_pager_dtor,
> + .cdev_pg_fault = NULL,
> +};
> +
> +static int
> +netmap_mmap_single(struct cdev *cdev, vm_ooffset_t *foff,
> + vm_size_t objsize, vm_object_t *objp, int prot)
> +{
> + vm_object_t obj;
> +
> + ND("cdev %p foff %jd size %jd objp %p prot %d", cdev,
> + (intmax_t )*foff, (intmax_t )objsize, objp, prot);
> + obj = vm_pager_allocate(OBJT_DEVICE, cdev, objsize, prot, *foff,
> + curthread->td_ucred);
> + ND("returns obj %p", obj);
> + if (obj == NULL)
> + return EINVAL;
> + if (saved_cdev_pager_ops.cdev_pg_fault == NULL) {
> + ND("initialize cdev_pager_ops");
> + saved_cdev_pager_ops = *(obj->un_pager.devp.ops);
> + netmap_cdev_pager_ops.cdev_pg_fault =
> + saved_cdev_pager_ops.cdev_pg_fault;
> + };
> + obj->un_pager.devp.ops = &netmap_cdev_pager_ops;
> + *objp = obj;
> + return 0;
> +}
> +#endif /* __FreeBSD__ */
> +
> +
> +/*
> + * mmap(2) support for the "netmap" device.
> + *
> + * Expose all the memory previously allocated by our custom memory
> + * allocator: this way the user has only to issue a single mmap(2), and
> + * can work on all the data structures flawlessly.
> + *
> + * Return 0 on success, -1 otherwise.
> + */
> +
> +#ifdef __FreeBSD__
> +static int
> +netmap_mmap(__unused struct cdev *dev,
> +#if __FreeBSD_version < 900000
> + vm_offset_t offset, vm_paddr_t *paddr, int nprot
> +#else
> + vm_ooffset_t offset, vm_paddr_t *paddr, int nprot,
> + __unused vm_memattr_t *memattr
> +#endif
> + )
> +{
> + int error = 0;
> + struct netmap_priv_d *priv;
> +
> + if (nprot & PROT_EXEC)
> + return (-1); // XXX -1 or EINVAL ?
> +
> + error = devfs_get_cdevpriv((void **)&priv);
> + if (error == EBADF) { /* called on fault, memory is initialized */
> + ND(5, "handling fault at ofs 0x%x", offset);
> + error = 0;
> + } else if (error == 0) /* make sure memory is set */
> + error = netmap_get_memory(priv);
> + if (error)
> + return (error);
> +
> + ND("request for offset 0x%x", (uint32_t)offset);
> + *paddr = netmap_ofstophys(offset);
> +
> + return (*paddr ? 0 : ENOMEM);
> +}
> +
> +static int
> +netmap_close(struct cdev *dev, int fflag, int devtype, struct thread *td)
> +{
> + if (netmap_verbose)
> + D("dev %p fflag 0x%x devtype %d td %p",
> + dev, fflag, devtype, td);
> + return 0;
> +}
> +
> +static int
> +netmap_open(struct cdev *dev, int oflags, int devtype, struct thread *td)
> +{
> + struct netmap_priv_d *priv;
> + int error;
> +
> + priv = malloc(sizeof(struct netmap_priv_d), M_DEVBUF,
> + M_NOWAIT | M_ZERO);
> + if (priv == NULL)
> + return ENOMEM;
> +
> + error = devfs_set_cdevpriv(priv, netmap_dtor);
> + if (error)
> + return error;
> +
> + return 0;
> +}
> +#endif /* __FreeBSD__ */
> +
> +
> +/*
> + * Handlers for synchronization of the queues from/to the host.
> + * Netmap has two operating modes:
> + * - in the default mode, the rings connected to the host stack are
> + * just another ring pair managed by userspace;
> + * - in transparent mode (XXX to be defined) incoming packets
> + * (from the host or the NIC) are marked as NS_FORWARD upon
> + * arrival, and the user application has a chance to reset the
> + * flag for packets that should be dropped.
> + * On the RXSYNC or poll(), packets in RX rings between
> + * kring->nr_kcur and ring->cur with NS_FORWARD still set are moved
> + * to the other side.
> + * The transfer NIC --> host is relatively easy, just encapsulate
> + * into mbufs and we are done. The host --> NIC side is slightly
> + * harder because there might not be room in the tx ring so it
> + * might take a while before releasing the buffer.
> + */
> +
> +/*
> + * pass a chain of buffers to the host stack as coming from 'dst'
> + */
> +static void
> +netmap_send_up(struct ifnet *dst, struct mbuf *head)
> +{
> + struct mbuf *m;
> +
> + /* send packets up, outside the lock */
> + while ((m = head) != NULL) {
> + head = head->m_nextpkt;
> + m->m_nextpkt = NULL;
> + if (netmap_verbose & NM_VERB_HOST)
> + D("sending up pkt %p size %d", m, MBUF_LEN(m));
> + NM_SEND_UP(dst, m);
> + }
> +}
> +
> +struct mbq {
> + struct mbuf *head;
> + struct mbuf *tail;
> + int count;
> +};
> +
> +/*
> + * put a copy of the buffers marked NS_FORWARD into an mbuf chain.
> + * Run from hwcur to cur - reserved
> + */
> +static void
> +netmap_grab_packets(struct netmap_kring *kring, struct mbq *q, int force)
> +{
> + /* Take packets from hwcur to cur-reserved and pass them up.
> + * In case of no buffers we give up. At the end of the loop,
> + * the queue is drained in all cases.
> + * XXX handle reserved
> + */
> + int k = kring->ring->cur - kring->ring->reserved;
> + u_int n, lim = kring->nkr_num_slots - 1;
> + struct mbuf *m, *tail = q->tail;
> +
> + if (k < 0)
> + k = k + kring->nkr_num_slots;
> + for (n = kring->nr_hwcur; n != k;) {
> + struct netmap_slot *slot = &kring->ring->slot[n];
> +
> + n = (n == lim) ? 0 : n + 1;
> + if ((slot->flags & NS_FORWARD) == 0 && !force)
> + continue;
> + if (slot->len < 14 || slot->len > NETMAP_BUF_SIZE) {
> + D("bad pkt at %d len %d", n, slot->len);
> + continue;
> + }
> + slot->flags &= ~NS_FORWARD; // XXX needed ?
> + m = m_devget(NMB(slot), slot->len, 0, kring->na->ifp, NULL);
> +
> + if (m == NULL)
> + break;
> + if (tail)
> + tail->m_nextpkt = m;
> + else
> + q->head = m;
> + tail = m;
> + q->count++;
> + m->m_nextpkt = NULL;
> + }
> + q->tail = tail;
> +}
> +
> +/*
> + * called under main lock to send packets from the host to the NIC
> + * The host ring has packets from nr_hwcur to (cur - reserved)
> + * to be sent down. We scan the tx rings, which have just been
> + * flushed so nr_hwcur == cur. Pushing packets down means
> + * increment cur and decrement avail.
> + * XXX to be verified
> + */
> +static void
> +netmap_sw_to_nic(struct netmap_adapter *na)
> +{
> + struct netmap_kring *kring = &na->rx_rings[na->num_rx_rings];
> + struct netmap_kring *k1 = &na->tx_rings[0];
> + int i, howmany, src_lim, dst_lim;
> +
> + howmany = kring->nr_hwavail; /* XXX otherwise cur - reserved - nr_hwcur */
> +
> + src_lim = kring->nkr_num_slots;
> + for (i = 0; howmany > 0 && i < na->num_tx_rings; i++, k1++) {
> + ND("%d packets left to ring %d (space %d)", howmany, i, k1->nr_hwavail);
> + dst_lim = k1->nkr_num_slots;
> + while (howmany > 0 && k1->ring->avail > 0) {
> + struct netmap_slot *src, *dst, tmp;
> + src = &kring->ring->slot[kring->nr_hwcur];
> + dst = &k1->ring->slot[k1->ring->cur];
> + tmp = *src;
> + src->buf_idx = dst->buf_idx;
> + src->flags = NS_BUF_CHANGED;
> +
> + dst->buf_idx = tmp.buf_idx;
> + dst->len = tmp.len;
> + dst->flags = NS_BUF_CHANGED;
> + ND("out len %d buf %d from %d to %d",
> + dst->len, dst->buf_idx,
> + kring->nr_hwcur, k1->ring->cur);
> +
> + if (++kring->nr_hwcur >= src_lim)
> + kring->nr_hwcur = 0;
> + howmany--;
> + kring->nr_hwavail--;
> + if (++k1->ring->cur >= dst_lim)
> + k1->ring->cur = 0;
> + k1->ring->avail--;
> + }
> + kring->ring->cur = kring->nr_hwcur; // XXX
> + k1++;
> + }
> +}
> +
> +/*
> + * netmap_sync_to_host() passes packets up. We are called from a
> + * system call in user process context, and the only contention
> + * can be among multiple user threads erroneously calling
> + * this routine concurrently.
> + */
> +static void
> +netmap_sync_to_host(struct netmap_adapter *na)
> +{
> + struct netmap_kring *kring = &na->tx_rings[na->num_tx_rings];
> + struct netmap_ring *ring = kring->ring;
> + u_int k, lim = kring->nkr_num_slots - 1;
> + struct mbq q = { NULL, NULL };
> +
> + k = ring->cur;
> + if (k > lim) {
> + netmap_ring_reinit(kring);
> + return;
> + }
> + // na->nm_lock(na->ifp, NETMAP_CORE_LOCK, 0);
> +
> + /* Take packets from hwcur to cur and pass them up.
> + * In case of no buffers we give up. At the end of the loop,
> + * the queue is drained in all cases.
> + */
> + netmap_grab_packets(kring, &q, 1);
> + kring->nr_hwcur = k;
> + kring->nr_hwavail = ring->avail = lim;
> + // na->nm_lock(na->ifp, NETMAP_CORE_UNLOCK, 0);
> +
> + netmap_send_up(na->ifp, q.head);
> +}
> +
> +/*
> + * rxsync backend for packets coming from the host stack.
> + * They have been put in the queue by netmap_start() so we
> + * need to protect access to the kring using a lock.
> + *
> + * This routine also does the selrecord if called from the poll handler
> + * (we know because td != NULL).
> + *
> + * NOTE: on linux, selrecord() is defined as a macro and uses pwait
> + * as an additional hidden argument.
> + */
> +static void
> +netmap_sync_from_host(struct netmap_adapter *na, struct thread *td, void *pwait)
> +{
> + struct netmap_kring *kring = &na->rx_rings[na->num_rx_rings];
> + struct netmap_ring *ring = kring->ring;
> + u_int j, n, lim = kring->nkr_num_slots;
> + u_int k = ring->cur, resvd = ring->reserved;
> +
> + (void)pwait; /* disable unused warnings */
> + na->nm_lock(na->ifp, NETMAP_CORE_LOCK, 0);
> + if (k >= lim) {
> + netmap_ring_reinit(kring);
> + return;
> + }
> + /* new packets are already set in nr_hwavail */
> + /* skip past packets that userspace has released */
> + j = kring->nr_hwcur;
> + if (resvd > 0) {
> + if (resvd + ring->avail >= lim + 1) {
> + D("XXX invalid reserve/avail %d %d", resvd, ring->avail);
> + ring->reserved = resvd = 0; // XXX panic...
> + }
> + k = (k >= resvd) ? k - resvd : k + lim - resvd;
> + }
> + if (j != k) {
> + n = k >= j ? k - j : k + lim - j;
> + kring->nr_hwavail -= n;
> + kring->nr_hwcur = k;
> + }
> + k = ring->avail = kring->nr_hwavail - resvd;
> + if (k == 0 && td)
> + selrecord(td, &kring->si);
> + if (k && (netmap_verbose & NM_VERB_HOST))
> + D("%d pkts from stack", k);
> + na->nm_lock(na->ifp, NETMAP_CORE_UNLOCK, 0);
> +}
> +
> +
> +/*
> + * get a refcounted reference to an interface.
> + * Return ENXIO if the interface does not exist, EINVAL if netmap
> + * is not supported by the interface.
> + * If successful, hold a reference.
> + */
> +static int
> +get_ifp(const char *name, struct ifnet **ifp)
> +{
> +#ifdef NM_BRIDGE
> + struct ifnet *iter = NULL;
> +
> + do {
> + struct nm_bridge *b;
> + int i, l, cand = -1;
> +
> + if (strncmp(name, NM_NAME, sizeof(NM_NAME) - 1))
> + break;
> + b = nm_find_bridge(name);
> + if (b == NULL) {
> + D("no bridges available for '%s'", name);
> + return (ENXIO);
> + }
> + /* XXX locking */
> + BDG_LOCK(b);
> + /* lookup in the local list of ports */
> + for (i = 0; i < NM_BDG_MAXPORTS; i++) {
> + iter = b->bdg_ports[i];
> + if (iter == NULL) {
> + if (cand == -1)
> + cand = i; /* potential insert point */
> + continue;
> + }
> + if (!strcmp(iter->if_xname, name)) {
> + ADD_BDG_REF(iter);
> + ND("found existing interface");
> + BDG_UNLOCK(b);
> + break;
> + }
> + }
> + if (i < NM_BDG_MAXPORTS) /* already unlocked */
> + break;
> + if (cand == -1) {
> + D("bridge full, cannot create new port");
> +no_port:
> + BDG_UNLOCK(b);
> + *ifp = NULL;
> + return EINVAL;
> + }
> + ND("create new bridge port %s", name);
> + /* space for forwarding list after the ifnet */
> + l = sizeof(*iter) +
> + sizeof(struct nm_bdg_fwd)*NM_BDG_BATCH ;
> + iter = malloc(l, M_DEVBUF, M_NOWAIT | M_ZERO);
> + if (!iter)
> + goto no_port;
> + strcpy(iter->if_xname, name);
> + bdg_netmap_attach(iter);
> + b->bdg_ports[cand] = iter;
> + iter->if_bridge = b;
> + ADD_BDG_REF(iter);
> + BDG_UNLOCK(b);
> + ND("attaching virtual bridge %p", b);
> + } while (0);
> + *ifp = iter;
> + if (! *ifp)
> +#endif /* NM_BRIDGE */
> + *ifp = ifunit_ref(name);
> + if (*ifp == NULL)
> + return (ENXIO);
> + /* can do this if the capability exists and if_pspare[0]
> + * points to the netmap descriptor.
> + */
> + if (NETMAP_CAPABLE(*ifp))
> + return 0; /* valid pointer, we hold the refcount */
> + nm_if_rele(*ifp);
> + return EINVAL; // not NETMAP capable
> +}
> +
> +
> +/*
> + * Error routine called when txsync/rxsync detects an error.
> + * Can't do much more than resetting cur = hwcur, avail = hwavail.
> + * Return 1 on reinit.
> + *
> + * This routine is only called by the upper half of the kernel.
> + * It only reads hwcur (which is changed only by the upper half, too)
> + * and hwavail (which may be changed by the lower half, but only on
> + * a tx ring and only to increase it, so any error will be recovered
> + * on the next call). For the above, we don't strictly need to call
> + * it under lock.
> + */
> +int
> +netmap_ring_reinit(struct netmap_kring *kring)
> +{
> + struct netmap_ring *ring = kring->ring;
> + u_int i, lim = kring->nkr_num_slots - 1;
> + int errors = 0;
> +
> + RD(10, "called for %s", kring->na->ifp->if_xname);
> + if (ring->cur > lim)
> + errors++;
> + for (i = 0; i <= lim; i++) {
> + u_int idx = ring->slot[i].buf_idx;
> + u_int len = ring->slot[i].len;
> + if (idx < 2 || idx >= netmap_total_buffers) {
> + if (!errors++)
> + D("bad buffer at slot %d idx %d len %d ", i, idx, len);
> + ring->slot[i].buf_idx = 0;
> + ring->slot[i].len = 0;
> + } else if (len > NETMAP_BUF_SIZE) {
> + ring->slot[i].len = 0;
> + if (!errors++)
> + D("bad len %d at slot %d idx %d",
> + len, i, idx);
> + }
> + }
> + if (errors) {
> + int pos = kring - kring->na->tx_rings;
> + int n = kring->na->num_tx_rings + 1;
> +
> + RD(10, "total %d errors", errors);
> + errors++;
> + RD(10, "%s %s[%d] reinit, cur %d -> %d avail %d -> %d",
> + kring->na->ifp->if_xname,
> + pos < n ? "TX" : "RX", pos < n ? pos : pos - n,
> + ring->cur, kring->nr_hwcur,
> + ring->avail, kring->nr_hwavail);
> + ring->cur = kring->nr_hwcur;
> + ring->avail = kring->nr_hwavail;
> + }
> + return (errors ? 1 : 0);
> +}
> +
> +
> +/*
> + * Set the ring ID. For devices with a single queue, a request
> + * for all rings is the same as a single ring.
> + */
> +static int
> +netmap_set_ringid(struct netmap_priv_d *priv, u_int ringid)
> +{
> + struct ifnet *ifp = priv->np_ifp;
> + struct netmap_adapter *na = NA(ifp);
> + u_int i = ringid & NETMAP_RING_MASK;
> + /* initially (np_qfirst == np_qlast) we don't want to lock */
> + int need_lock = (priv->np_qfirst != priv->np_qlast);
> + int lim = na->num_rx_rings;
> +
> + if (na->num_tx_rings > lim)
> + lim = na->num_tx_rings;
> + if ( (ringid & NETMAP_HW_RING) && i >= lim) {
> + D("invalid ring id %d", i);
> + return (EINVAL);
> + }
> + if (need_lock)
> + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0);
> + priv->np_ringid = ringid;
> + if (ringid & NETMAP_SW_RING) {
> + priv->np_qfirst = NETMAP_SW_RING;
> + priv->np_qlast = 0;
> + } else if (ringid & NETMAP_HW_RING) {
> + priv->np_qfirst = i;
> + priv->np_qlast = i + 1;
> + } else {
> + priv->np_qfirst = 0;
> + priv->np_qlast = NETMAP_HW_RING ;
> + }
> + priv->np_txpoll = (ringid & NETMAP_NO_TX_POLL) ? 0 : 1;
> + if (need_lock)
> + na->nm_lock(ifp, NETMAP_CORE_UNLOCK, 0);
> + if (netmap_verbose) {
> + if (ringid & NETMAP_SW_RING)
> + D("ringid %s set to SW RING", ifp->if_xname);
> + else if (ringid & NETMAP_HW_RING)
> + D("ringid %s set to HW RING %d", ifp->if_xname,
> + priv->np_qfirst);
> + else
> + D("ringid %s set to all %d HW RINGS", ifp->if_xname, lim);
> + }
> + return 0;
> +}
> +
> +/*
> + * ioctl(2) support for the "netmap" device.
> + *
> + * Following a list of accepted commands:
> + * - NIOCGINFO
> + * - SIOCGIFADDR just for convenience
> + * - NIOCREGIF
> + * - NIOCUNREGIF
> + * - NIOCTXSYNC
> + * - NIOCRXSYNC
> + *
> + * Return 0 on success, errno otherwise.
> + */
> +static int
> +netmap_ioctl(struct cdev *dev, u_long cmd, caddr_t data,
> + int fflag, struct thread *td)
> +{
> + struct netmap_priv_d *priv = NULL;
> + struct ifnet *ifp;
> + struct nmreq *nmr = (struct nmreq *) data;
> + struct netmap_adapter *na;
> + int error;
> + u_int i, lim;
> + struct netmap_if *nifp;
> +
> + (void)dev; /* UNUSED */
> + (void)fflag; /* UNUSED */
> +#ifdef linux
> +#define devfs_get_cdevpriv(pp) \
> + ({ *(struct netmap_priv_d **)pp = ((struct file *)td)->private_data; \
> + (*pp ? 0 : ENOENT); })
> +
> +/* devfs_set_cdevpriv cannot fail on linux */
> +#define devfs_set_cdevpriv(p, fn) \
> + ({ ((struct file *)td)->private_data = p; (p ? 0 : EINVAL); })
> +
> +
> +#define devfs_clear_cdevpriv() do { \
> + netmap_dtor(priv); ((struct file *)td)->private_data = 0; \
> + } while (0)
> +#endif /* linux */
> +
> + CURVNET_SET(TD_TO_VNET(td));
> +
> + error = devfs_get_cdevpriv((void **)&priv);
> + if (error) {
> + CURVNET_RESTORE();
> + /* XXX ENOENT should be impossible, since the priv
> + * is now created in the open */
> + return (error == ENOENT ? ENXIO : error);
> + }
> +
> + nmr->nr_name[sizeof(nmr->nr_name) - 1] = '\0'; /* truncate name */
> + switch (cmd) {
> + case NIOCGINFO: /* return capabilities etc */
> + if (nmr->nr_version != NETMAP_API) {
> + D("API mismatch got %d have %d",
> + nmr->nr_version, NETMAP_API);
> + nmr->nr_version = NETMAP_API;
> + error = EINVAL;
> + break;
> + }
> + /* update configuration */
> + error = netmap_get_memory(priv);
> + ND("get_memory returned %d", error);
> + if (error)
> + break;
> + /* memsize is always valid */
> + nmr->nr_memsize = nm_mem.nm_totalsize;
> + nmr->nr_offset = 0;
> + nmr->nr_rx_rings = nmr->nr_tx_rings = 0;
> + nmr->nr_rx_slots = nmr->nr_tx_slots = 0;
> + if (nmr->nr_name[0] == '\0') /* just get memory info */
> + break;
> + error = get_ifp(nmr->nr_name, &ifp); /* get a refcount */
> + if (error)
> + break;
> + na = NA(ifp); /* retrieve netmap_adapter */
> + netmap_update_config(na);
> + nmr->nr_rx_rings = na->num_rx_rings;
> + nmr->nr_tx_rings = na->num_tx_rings;
> + nmr->nr_rx_slots = na->num_rx_desc;
> + nmr->nr_tx_slots = na->num_tx_desc;
> + nm_if_rele(ifp); /* return the refcount */
> + break;
> +
> + case NIOCREGIF:
> + if (nmr->nr_version != NETMAP_API) {
> + nmr->nr_version = NETMAP_API;
> + error = EINVAL;
> + break;
> + }
> + /* ensure allocators are ready */
> + error = netmap_get_memory(priv);
> + ND("get_memory returned %d", error);
> + if (error)
> + break;
> +
> + /* protect access to priv from concurrent NIOCREGIF */
> + NMA_LOCK();
> + if (priv->np_ifp != NULL) { /* thread already registered */
> + error = netmap_set_ringid(priv, nmr->nr_ringid);
> + NMA_UNLOCK();
> + break;
> + }
> + /* find the interface and a reference */
> + error = get_ifp(nmr->nr_name, &ifp); /* keep reference */
> + if (error) {
> + NMA_UNLOCK();
> + break;
> + }
> + na = NA(ifp); /* retrieve netmap adapter */
> +
> + for (i = 10; i > 0; i--) {
> + na->nm_lock(ifp, NETMAP_REG_LOCK, 0);
> + if (!NETMAP_DELETING(na))
> + break;
> + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0);
> + tsleep(na, 0, "NIOCREGIF", hz/10);
> + }
> + if (i == 0) {
> + D("too many NIOCREGIF attempts, give up");
> + error = EINVAL;
> + nm_if_rele(ifp); /* return the refcount */
> + NMA_UNLOCK();
> + break;
> + }
> +
> + /* ring configuration may have changed, fetch from the card */
> + netmap_update_config(na);
> + priv->np_ifp = ifp; /* store the reference */
> + error = netmap_set_ringid(priv, nmr->nr_ringid);
> + if (error)
> + goto error;
> + nifp = netmap_if_new(nmr->nr_name, na);
> + if (nifp == NULL) { /* allocation failed */
> + error = ENOMEM;
> + } else if (ifp->if_capenable & IFCAP_NETMAP) {
> + /* was already set */
> + } else {
> + /* Otherwise set the card in netmap mode
> + * and make it use the shared buffers.
> + */
> + for (i = 0 ; i < na->num_tx_rings + 1; i++)
> + mtx_init(&na->tx_rings[i].q_lock, "nm_txq_lock", MTX_NETWORK_LOCK, MTX_DEF);
> + for (i = 0 ; i < na->num_rx_rings + 1; i++) {
> + mtx_init(&na->rx_rings[i].q_lock, "nm_rxq_lock", MTX_NETWORK_LOCK, MTX_DEF);
> + }
> + error = na->nm_register(ifp, 1); /* mode on */
> + if (error) {
> + netmap_dtor_locked(priv);
> + netmap_if_free(nifp);
> + }
> + }
> +
> + if (error) { /* reg. failed, release priv and ref */
> +error:
> + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0);
> + nm_if_rele(ifp); /* return the refcount */
> + priv->np_ifp = NULL;
> + priv->np_nifp = NULL;
> + NMA_UNLOCK();
> + break;
> + }
> +
> + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0);
> +
> + /* the following assignment is a commitment.
> + * Readers (i.e., poll and *SYNC) check for
> + * np_nifp != NULL without locking
> + */
> + wmb(); /* make sure previous writes are visible to all CPUs */
> + priv->np_nifp = nifp;
> + NMA_UNLOCK();
> +
> + /* return the offset of the netmap_if object */
> + nmr->nr_rx_rings = na->num_rx_rings;
> + nmr->nr_tx_rings = na->num_tx_rings;
> + nmr->nr_rx_slots = na->num_rx_desc;
> + nmr->nr_tx_slots = na->num_tx_desc;
> + nmr->nr_memsize = nm_mem.nm_totalsize;
> + nmr->nr_offset = netmap_if_offset(nifp);
> + break;
> +
> + case NIOCUNREGIF:
> + // XXX we have no data here ?
> + D("deprecated, data is %p", nmr);
> + error = EINVAL;
> + break;
> +
> + case NIOCTXSYNC:
> + case NIOCRXSYNC:
> + nifp = priv->np_nifp;
> +
> + if (nifp == NULL) {
> + error = ENXIO;
> + break;
> + }
> + rmb(); /* make sure following reads are not from cache */
> +
> +
> + ifp = priv->np_ifp; /* we have a reference */
> +
> + if (ifp == NULL) {
> + D("Internal error: nifp != NULL && ifp == NULL");
> + error = ENXIO;
> + break;
> + }
> +
> + na = NA(ifp); /* retrieve netmap adapter */
> + if (priv->np_qfirst == NETMAP_SW_RING) { /* host rings */
> + if (cmd == NIOCTXSYNC)
> + netmap_sync_to_host(na);
> + else
> + netmap_sync_from_host(na, NULL, NULL);
> + break;
> + }
> + /* find the last ring to scan */
> + lim = priv->np_qlast;
> + if (lim == NETMAP_HW_RING)
> + lim = (cmd == NIOCTXSYNC) ?
> + na->num_tx_rings : na->num_rx_rings;
> +
> + for (i = priv->np_qfirst; i < lim; i++) {
> + if (cmd == NIOCTXSYNC) {
> + struct netmap_kring *kring = &na->tx_rings[i];
> + if (netmap_verbose & NM_VERB_TXSYNC)
> + D("pre txsync ring %d cur %d hwcur %d",
> + i, kring->ring->cur,
> + kring->nr_hwcur);
> + na->nm_txsync(ifp, i, 1 /* do lock */);
> + if (netmap_verbose & NM_VERB_TXSYNC)
> + D("post txsync ring %d cur %d hwcur %d",
> + i, kring->ring->cur,
> + kring->nr_hwcur);
> + } else {
> + na->nm_rxsync(ifp, i, 1 /* do lock */);
> + microtime(&na->rx_rings[i].ring->ts);
> + }
> + }
> +
> + break;
> +
> +#ifdef __FreeBSD__
> + case BIOCIMMEDIATE:
> + case BIOCGHDRCMPLT:
> + case BIOCSHDRCMPLT:
> + case BIOCSSEESENT:
> + D("ignore BIOCIMMEDIATE/BIOCSHDRCMPLT/BIOCSHDRCMPLT/BIOCSSEESENT");
> + break;
> +
> + default: /* allow device-specific ioctls */
> + {
> + struct socket so;
> + bzero(&so, sizeof(so));
> + error = get_ifp(nmr->nr_name, &ifp); /* keep reference */
> + if (error)
> + break;
> + so.so_vnet = ifp->if_vnet;
> + // so->so_proto not null.
> + error = ifioctl(&so, cmd, data, td);
> + nm_if_rele(ifp);
> + break;
> + }
> +
> +#else /* linux */
> + default:
> + error = EOPNOTSUPP;
> +#endif /* linux */
> + }
> +
> + CURVNET_RESTORE();
> + return (error);
> +}
> +
> +
> +/*
> + * select(2) and poll(2) handlers for the "netmap" device.
> + *
> + * Can be called for one or more queues.
> + * Return true the event mask corresponding to ready events.
> + * If there are no ready events, do a selrecord on either individual
> + * selfd or on the global one.
> + * Device-dependent parts (locking and sync of tx/rx rings)
> + * are done through callbacks.
> + *
> + * On linux, arguments are really pwait, the poll table, and 'td' is struct file *
> + * The first one is remapped to pwait as selrecord() uses the name as an
> + * hidden argument.
> + */
> +static int
> +netmap_poll(struct cdev *dev, int events, struct thread *td)
> +{
> + struct netmap_priv_d *priv = NULL;
> + struct netmap_adapter *na;
> + struct ifnet *ifp;
> + struct netmap_kring *kring;
> + u_int core_lock, i, check_all, want_tx, want_rx, revents = 0;
> + u_int lim_tx, lim_rx, host_forwarded = 0;
> + struct mbq q = { NULL, NULL, 0 };
> + enum {NO_CL, NEED_CL, LOCKED_CL }; /* see below */
> + void *pwait = dev; /* linux compatibility */
> +
> + (void)pwait;
> +
> + if (devfs_get_cdevpriv((void **)&priv) != 0 || priv == NULL)
> + return POLLERR;
> +
> + if (priv->np_nifp == NULL) {
> + D("No if registered");
> + return POLLERR;
> + }
> + rmb(); /* make sure following reads are not from cache */
> +
> + ifp = priv->np_ifp;
> + // XXX check for deleting() ?
> + if ( (ifp->if_capenable & IFCAP_NETMAP) == 0)
> + return POLLERR;
> +
> + if (netmap_verbose & 0x8000)
> + D("device %s events 0x%x", ifp->if_xname, events);
> + want_tx = events & (POLLOUT | POLLWRNORM);
> + want_rx = events & (POLLIN | POLLRDNORM);
> +
> + na = NA(ifp); /* retrieve netmap adapter */
> +
> + lim_tx = na->num_tx_rings;
> + lim_rx = na->num_rx_rings;
> + /* how many queues we are scanning */
> + if (priv->np_qfirst == NETMAP_SW_RING) {
> + if (priv->np_txpoll || want_tx) {
> + /* push any packets up, then we are always ready */
> + kring = &na->tx_rings[lim_tx];
> + netmap_sync_to_host(na);
> + revents |= want_tx;
> + }
> + if (want_rx) {
> + kring = &na->rx_rings[lim_rx];
> + if (kring->ring->avail == 0)
> + netmap_sync_from_host(na, td, dev);
> + if (kring->ring->avail > 0) {
> + revents |= want_rx;
> + }
> + }
> + return (revents);
> + }
> +
> + /* if we are in transparent mode, check also the host rx ring */
> + kring = &na->rx_rings[lim_rx];
> + if ( (priv->np_qlast == NETMAP_HW_RING) // XXX check_all
> + && want_rx
> + && (netmap_fwd || kring->ring->flags & NR_FORWARD) ) {
> + if (kring->ring->avail == 0)
> + netmap_sync_from_host(na, td, dev);
> + if (kring->ring->avail > 0)
> + revents |= want_rx;
> + }
> +
> + /*
> + * check_all is set if the card has more than one queue and
> + * the client is polling all of them. If true, we sleep on
> + * the "global" selfd, otherwise we sleep on individual selfd
> + * (we can only sleep on one of them per direction).
> + * The interrupt routine in the driver should always wake on
> + * the individual selfd, and also on the global one if the card
> + * has more than one ring.
> + *
> + * If the card has only one lock, we just use that.
> + * If the card has separate ring locks, we just use those
> + * unless we are doing check_all, in which case the whole
> + * loop is wrapped by the global lock.
> + * We acquire locks only when necessary: if poll is called
> + * when buffers are available, we can just return without locks.
> + *
> + * rxsync() is only called if we run out of buffers on a POLLIN.
> + * txsync() is called if we run out of buffers on POLLOUT, or
> + * there are pending packets to send. The latter can be disabled
> + * passing NETMAP_NO_TX_POLL in the NIOCREG call.
> + */
> + check_all = (priv->np_qlast == NETMAP_HW_RING) && (lim_tx > 1 || lim_rx > 1);
> +
> + /*
> + * core_lock indicates what to do with the core lock.
> + * The core lock is used when either the card has no individual
> + * locks, or it has individual locks but we are cheking all
> + * rings so we need the core lock to avoid missing wakeup events.
> + *
> + * It has three possible states:
> + * NO_CL we don't need to use the core lock, e.g.
> + * because we are protected by individual locks.
> + * NEED_CL we need the core lock. In this case, when we
> + * call the lock routine, move to LOCKED_CL
> + * to remember to release the lock once done.
> + * LOCKED_CL core lock is set, so we need to release it.
> + */
> + core_lock = (check_all || !na->separate_locks) ? NEED_CL : NO_CL;
> +#ifdef NM_BRIDGE
> + /* the bridge uses separate locks */
> + if (na->nm_register == bdg_netmap_reg) {
> + ND("not using core lock for %s", ifp->if_xname);
> + core_lock = NO_CL;
> + }
> +#endif /* NM_BRIDGE */
> + if (priv->np_qlast != NETMAP_HW_RING) {
> + lim_tx = lim_rx = priv->np_qlast;
> + }
> +
> + /*
> + * We start with a lock free round which is good if we have
> + * data available. If this fails, then lock and call the sync
> + * routines.
> + */
> + for (i = priv->np_qfirst; want_rx && i < lim_rx; i++) {
> + kring = &na->rx_rings[i];
> + if (kring->ring->avail > 0) {
> + revents |= want_rx;
> + want_rx = 0; /* also breaks the loop */
> + }
> + }
> + for (i = priv->np_qfirst; want_tx && i < lim_tx; i++) {
> + kring = &na->tx_rings[i];
> + if (kring->ring->avail > 0) {
> + revents |= want_tx;
> + want_tx = 0; /* also breaks the loop */
> + }
> + }
> +
> + /*
> + * If we to push packets out (priv->np_txpoll) or want_tx is
> + * still set, we do need to run the txsync calls (on all rings,
> + * to avoid that the tx rings stall).
> + */
> + if (priv->np_txpoll || want_tx) {
> +flush_tx:
> + for (i = priv->np_qfirst; i < lim_tx; i++) {
> + kring = &na->tx_rings[i];
> + /*
> + * Skip the current ring if want_tx == 0
> + * (we have already done a successful sync on
> + * a previous ring) AND kring->cur == kring->hwcur
> + * (there are no pending transmissions for this ring).
> + */
> + if (!want_tx && kring->ring->cur == kring->nr_hwcur)
> + continue;
> + if (core_lock == NEED_CL) {
> + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0);
> + core_lock = LOCKED_CL;
> + }
> + if (na->separate_locks)
> + na->nm_lock(ifp, NETMAP_TX_LOCK, i);
> + if (netmap_verbose & NM_VERB_TXSYNC)
> + D("send %d on %s %d",
> + kring->ring->cur,
> + ifp->if_xname, i);
> + if (na->nm_txsync(ifp, i, 0 /* no lock */))
> + revents |= POLLERR;
> +
> + /* Check avail/call selrecord only if called with POLLOUT */
> + if (want_tx) {
> + if (kring->ring->avail > 0) {
> + /* stop at the first ring. We don't risk
> + * starvation.
> + */
> + revents |= want_tx;
> + want_tx = 0;
> + } else if (!check_all)
> + selrecord(td, &kring->si);
> + }
> + if (na->separate_locks)
> + na->nm_lock(ifp, NETMAP_TX_UNLOCK, i);
> + }
> + }
> +
> + /*
> + * now if want_rx is still set we need to lock and rxsync.
> + * Do it on all rings because otherwise we starve.
> + */
> + if (want_rx) {
> + for (i = priv->np_qfirst; i < lim_rx; i++) {
> + kring = &na->rx_rings[i];
> + if (core_lock == NEED_CL) {
> + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0);
> + core_lock = LOCKED_CL;
> + }
> + if (na->separate_locks)
> + na->nm_lock(ifp, NETMAP_RX_LOCK, i);
> + if (netmap_fwd ||kring->ring->flags & NR_FORWARD) {
> + ND(10, "forwarding some buffers up %d to %d",
> + kring->nr_hwcur, kring->ring->cur);
> + netmap_grab_packets(kring, &q, netmap_fwd);
> + }
> +
> + if (na->nm_rxsync(ifp, i, 0 /* no lock */))
> + revents |= POLLERR;
> + if (netmap_no_timestamp == 0 ||
> + kring->ring->flags & NR_TIMESTAMP) {
> + microtime(&kring->ring->ts);
> + }
> +
> + if (kring->ring->avail > 0)
> + revents |= want_rx;
> + else if (!check_all)
> + selrecord(td, &kring->si);
> + if (na->separate_locks)
> + na->nm_lock(ifp, NETMAP_RX_UNLOCK, i);
> + }
> + }
> + if (check_all && revents == 0) { /* signal on the global queue */
> + if (want_tx)
> + selrecord(td, &na->tx_si);
> + if (want_rx)
> + selrecord(td, &na->rx_si);
> + }
> +
> + /* forward host to the netmap ring */
> + kring = &na->rx_rings[lim_rx];
> + if (kring->nr_hwavail > 0)
> + ND("host rx %d has %d packets", lim_rx, kring->nr_hwavail);
> + if ( (priv->np_qlast == NETMAP_HW_RING) // XXX check_all
> + && (netmap_fwd || kring->ring->flags & NR_FORWARD)
> + && kring->nr_hwavail > 0 && !host_forwarded) {
> + if (core_lock == NEED_CL) {
> + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0);
> + core_lock = LOCKED_CL;
> + }
> + netmap_sw_to_nic(na);
> + host_forwarded = 1; /* prevent another pass */
> + want_rx = 0;
> + goto flush_tx;
> + }
> +
> + if (core_lock == LOCKED_CL)
> + na->nm_lock(ifp, NETMAP_CORE_UNLOCK, 0);
> + if (q.head)
> + netmap_send_up(na->ifp, q.head);
> +
> + return (revents);
> +}
> +
> +/*------- driver support routines ------*/
> +
> +/*
> + * default lock wrapper.
> + */
> +static void
> +netmap_lock_wrapper(struct ifnet *dev, int what, u_int queueid)
> +{
> + struct netmap_adapter *na = NA(dev);
> +
> + switch (what) {
> +#ifdef linux /* some system do not need lock on register */
> + case NETMAP_REG_LOCK:
> + case NETMAP_REG_UNLOCK:
> + break;
> +#endif /* linux */
> +
> + case NETMAP_CORE_LOCK:
> + mtx_lock(&na->core_lock);
> + break;
> +
> + case NETMAP_CORE_UNLOCK:
> + mtx_unlock(&na->core_lock);
> + break;
> +
> + case NETMAP_TX_LOCK:
> + mtx_lock(&na->tx_rings[queueid].q_lock);
> + break;
> +
> + case NETMAP_TX_UNLOCK:
> + mtx_unlock(&na->tx_rings[queueid].q_lock);
> + break;
> +
> + case NETMAP_RX_LOCK:
> + mtx_lock(&na->rx_rings[queueid].q_lock);
> + break;
> +
> + case NETMAP_RX_UNLOCK:
> + mtx_unlock(&na->rx_rings[queueid].q_lock);
> + break;
> + }
> +}
> +
> +
> +/*
> + * Initialize a ``netmap_adapter`` object created by driver on attach.
> + * We allocate a block of memory with room for a struct netmap_adapter
> + * plus two sets of N+2 struct netmap_kring (where N is the number
> + * of hardware rings):
> + * krings 0..N-1 are for the hardware queues.
> + * kring N is for the host stack queue
> + * kring N+1 is only used for the selinfo for all queues.
> + * Return 0 on success, ENOMEM otherwise.
> + *
> + * By default the receive and transmit adapter ring counts are both initialized
> + * to num_queues. na->num_tx_rings can be set for cards with different tx/rx
> + * setups.
> + */
> +int
> +netmap_attach(struct netmap_adapter *arg, int num_queues)
> +{
> + struct netmap_adapter *na = NULL;
> + struct ifnet *ifp = arg ? arg->ifp : NULL;
> +
> + if (arg == NULL || ifp == NULL)
> + goto fail;
> + na = malloc(sizeof(*na), M_DEVBUF, M_NOWAIT | M_ZERO);
> + if (na == NULL)
> + goto fail;
> + WNA(ifp) = na;
> + *na = *arg; /* copy everything, trust the driver to not pass junk */
> + NETMAP_SET_CAPABLE(ifp);
> + if (na->num_tx_rings == 0)
> + na->num_tx_rings = num_queues;
> + na->num_rx_rings = num_queues;
> + na->refcount = na->na_single = na->na_multi = 0;
> + /* Core lock initialized here, others after netmap_if_new. */
> + mtx_init(&na->core_lock, "netmap core lock", MTX_NETWORK_LOCK, MTX_DEF);
> + if (na->nm_lock == NULL) {
> + ND("using default locks for %s", ifp->if_xname);
> + na->nm_lock = netmap_lock_wrapper;
> + }
> +#ifdef linux
> + if (ifp->netdev_ops) {
> + ND("netdev_ops %p", ifp->netdev_ops);
> + /* prepare a clone of the netdev ops */
> + na->nm_ndo = *ifp->netdev_ops;
> + }
> + na->nm_ndo.ndo_start_xmit = linux_netmap_start;
> +#endif
> + D("success for %s", ifp->if_xname);
> + return 0;
> +
> +fail:
> + D("fail, arg %p ifp %p na %p", arg, ifp, na);
> + return (na ? EINVAL : ENOMEM);
> +}
> +
> +
> +/*
> + * Free the allocated memory linked to the given ``netmap_adapter``
> + * object.
> + */
> +void
> +netmap_detach(struct ifnet *ifp)
> +{
> + struct netmap_adapter *na = NA(ifp);
> +
> + if (!na)
> + return;
> +
> + mtx_destroy(&na->core_lock);
> +
> + if (na->tx_rings) { /* XXX should not happen */
> + D("freeing leftover tx_rings");
> + free(na->tx_rings, M_DEVBUF);
> + }
> + bzero(na, sizeof(*na));
> + WNA(ifp) = NULL;
> + free(na, M_DEVBUF);
> +}
> +
> +
> +/*
> + * Intercept packets from the network stack and pass them
> + * to netmap as incoming packets on the 'software' ring.
> + * We are not locked when called.
> + */
> +int
> +netmap_start(struct ifnet *ifp, struct mbuf *m)
> +{
> + struct netmap_adapter *na = NA(ifp);
> + struct netmap_kring *kring = &na->rx_rings[na->num_rx_rings];
> + u_int i, len = MBUF_LEN(m);
> + u_int error = EBUSY, lim = kring->nkr_num_slots - 1;
> + struct netmap_slot *slot;
> +
> + if (netmap_verbose & NM_VERB_HOST)
> + D("%s packet %d len %d from the stack", ifp->if_xname,
> + kring->nr_hwcur + kring->nr_hwavail, len);
> + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0);
> + if (kring->nr_hwavail >= lim) {
> + if (netmap_verbose)
> + D("stack ring %s full\n", ifp->if_xname);
> + goto done; /* no space */
> + }
> + if (len > NETMAP_BUF_SIZE) {
> + D("%s from_host, drop packet size %d > %d", ifp->if_xname,
> + len, NETMAP_BUF_SIZE);
> + goto done; /* too long for us */
> + }
> +
> + /* compute the insert position */
> + i = kring->nr_hwcur + kring->nr_hwavail;
> + if (i > lim)
> + i -= lim + 1;
> + slot = &kring->ring->slot[i];
> + m_copydata(m, 0, len, NMB(slot));
> + slot->len = len;
> + slot->flags = kring->nkr_slot_flags;
> + kring->nr_hwavail++;
> + if (netmap_verbose & NM_VERB_HOST)
> + D("wake up host ring %s %d", na->ifp->if_xname, na->num_rx_rings);
> + selwakeuppri(&kring->si, PI_NET);
> + error = 0;
> +done:
> + na->nm_lock(ifp, NETMAP_CORE_UNLOCK, 0);
> +
> + /* release the mbuf in either cases of success or failure. As an
> + * alternative, put the mbuf in a free list and free the list
> + * only when really necessary.
> + */
> + m_freem(m);
> +
> + return (error);
> +}
> +
> +
> +/*
> + * netmap_reset() is called by the driver routines when reinitializing
> + * a ring. The driver is in charge of locking to protect the kring.
> + * If netmap mode is not set just return NULL.
> + */
> +struct netmap_slot *
> +netmap_reset(struct netmap_adapter *na, enum txrx tx, int n,
> + u_int new_cur)
> +{
> + struct netmap_kring *kring;
> + int new_hwofs, lim;
> +
> + if (na == NULL)
> + return NULL; /* no netmap support here */
> + if (!(na->ifp->if_capenable & IFCAP_NETMAP))
> + return NULL; /* nothing to reinitialize */
> +
> + if (tx == NR_TX) {
> + if (n >= na->num_tx_rings)
> + return NULL;
> + kring = na->tx_rings + n;
> + new_hwofs = kring->nr_hwcur - new_cur;
> + } else {
> + if (n >= na->num_rx_rings)
> + return NULL;
> + kring = na->rx_rings + n;
> + new_hwofs = kring->nr_hwcur + kring->nr_hwavail - new_cur;
> + }
> + lim = kring->nkr_num_slots - 1;
> + if (new_hwofs > lim)
> + new_hwofs -= lim + 1;
> +
> + /* Alwayws set the new offset value and realign the ring. */
> + kring->nkr_hwofs = new_hwofs;
> + if (tx == NR_TX)
> + kring->nr_hwavail = kring->nkr_num_slots - 1;
> + ND(10, "new hwofs %d on %s %s[%d]",
> + kring->nkr_hwofs, na->ifp->if_xname,
> + tx == NR_TX ? "TX" : "RX", n);
> +
> +#if 0 // def linux
> + /* XXX check that the mappings are correct */
> + /* need ring_nr, adapter->pdev, direction */
> + buffer_info->dma = dma_map_single(&pdev->dev, addr, adapter->rx_buffer_len, DMA_FROM_DEVICE);
> + if (dma_mapping_error(&adapter->pdev->dev, buffer_info->dma)) {
> + D("error mapping rx netmap buffer %d", i);
> + // XXX fix error handling
> + }
> +
> +#endif /* linux */
> + /*
> + * Wakeup on the individual and global lock
> + * We do the wakeup here, but the ring is not yet reconfigured.
> + * However, we are under lock so there are no races.
> + */
> + selwakeuppri(&kring->si, PI_NET);
> + selwakeuppri(tx == NR_TX ? &na->tx_si : &na->rx_si, PI_NET);
> + return kring->ring->slot;
> +}
> +
> +
> +/*
> + * Default functions to handle rx/tx interrupts
> + * we have 4 cases:
> + * 1 ring, single lock:
> + * lock(core); wake(i=0); unlock(core)
> + * N rings, single lock:
> + * lock(core); wake(i); wake(N+1) unlock(core)
> + * 1 ring, separate locks: (i=0)
> + * lock(i); wake(i); unlock(i)
> + * N rings, separate locks:
> + * lock(i); wake(i); unlock(i); lock(core) wake(N+1) unlock(core)
> + * work_done is non-null on the RX path.
> + */
> +int
> +netmap_rx_irq(struct ifnet *ifp, int q, int *work_done)
> +{
> + struct netmap_adapter *na;
> + struct netmap_kring *r;
> + NM_SELINFO_T *main_wq;
> +
> + if (!(ifp->if_capenable & IFCAP_NETMAP))
> + return 0;
> + ND(5, "received %s queue %d", work_done ? "RX" : "TX" , q);
> + na = NA(ifp);
> + if (na->na_flags & NAF_SKIP_INTR) {
> + ND("use regular interrupt");
> + return 0;
> + }
> +
> + if (work_done) { /* RX path */
> + if (q >= na->num_rx_rings)
> + return 0; // regular queue
> + r = na->rx_rings + q;
> + r->nr_kflags |= NKR_PENDINTR;
> + main_wq = (na->num_rx_rings > 1) ? &na->rx_si : NULL;
> + } else { /* tx path */
> + if (q >= na->num_tx_rings)
> + return 0; // regular queue
> + r = na->tx_rings + q;
> + main_wq = (na->num_tx_rings > 1) ? &na->tx_si : NULL;
> + work_done = &q; /* dummy */
> + }
> + if (na->separate_locks) {
> + mtx_lock(&r->q_lock);
> + selwakeuppri(&r->si, PI_NET);
> + mtx_unlock(&r->q_lock);
> + if (main_wq) {
> + mtx_lock(&na->core_lock);
> + selwakeuppri(main_wq, PI_NET);
> + mtx_unlock(&na->core_lock);
> + }
> + } else {
> + mtx_lock(&na->core_lock);
> + selwakeuppri(&r->si, PI_NET);
> + if (main_wq)
> + selwakeuppri(main_wq, PI_NET);
> + mtx_unlock(&na->core_lock);
> + }
> + *work_done = 1; /* do not fire napi again */
> + return 1;
> +}
> +
> +
> +#ifdef linux /* linux-specific routines */
> +
> +/*
> + * Remap linux arguments into the FreeBSD call.
> + * - pwait is the poll table, passed as 'dev';
> + * If pwait == NULL someone else already woke up before. We can report
> + * events but they are filtered upstream.
> + * If pwait != NULL, then pwait->key contains the list of events.
> + * - events is computed from pwait as above.
> + * - file is passed as 'td';
> + */
> +static u_int
> +linux_netmap_poll(struct file * file, struct poll_table_struct *pwait)
> +{
> +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,4,0)
> + int events = pwait ? pwait->key : POLLIN | POLLOUT;
> +#else /* in 3.4.0 field 'key' was renamed to '_key' */
> + int events = pwait ? pwait->_key : POLLIN | POLLOUT;
> +#endif
> + return netmap_poll((void *)pwait, events, (void *)file);
> +}
> +
> +static int
> +linux_netmap_mmap(struct file *f, struct vm_area_struct *vma)
> +{
> + int lut_skip, i, j;
> + int user_skip = 0;
> + struct lut_entry *l_entry;
> + int error = 0;
> + unsigned long off, tomap;
> + /*
> + * vma->vm_start: start of mapping user address space
> + * vma->vm_end: end of the mapping user address space
> + * vma->vm_pfoff: offset of first page in the device
> + */
> +
> + // XXX security checks
> +
> + error = netmap_get_memory(f->private_data);
> + ND("get_memory returned %d", error);
> + if (error)
> + return -error;
> +
> + off = vma->vm_pgoff << PAGE_SHIFT; /* offset in bytes */
> + tomap = vma->vm_end - vma->vm_start;
> + for (i = 0; i < NETMAP_POOLS_NR; i++) { /* loop through obj_pools */
> + const struct netmap_obj_pool *p = &nm_mem.pools[i];
> + /*
> + * In each pool memory is allocated in clusters
> + * of size _clustsize, each containing clustentries
> + * entries. For each object k we already store the
> + * vtophys mapping in lut[k] so we use that, scanning
> + * the lut[] array in steps of clustentries,
> + * and we map each cluster (not individual pages,
> + * it would be overkill).
> + */
> +
> + /*
> + * We interpret vm_pgoff as an offset into the whole
> + * netmap memory, as if all clusters where contiguous.
> + */
> + for (lut_skip = 0, j = 0; j < p->_numclusters; j++, lut_skip += p->clustentries) {
> + unsigned long paddr, mapsize;
> + if (p->_clustsize <= off) {
> + off -= p->_clustsize;
> + continue;
> + }
> + l_entry = &p->lut[lut_skip]; /* first obj in the cluster */
> + paddr = l_entry->paddr + off;
> + mapsize = p->_clustsize - off;
> + off = 0;
> + if (mapsize > tomap)
> + mapsize = tomap;
> + ND("remap_pfn_range(%lx, %lx, %lx)",
> + vma->vm_start + user_skip,
> + paddr >> PAGE_SHIFT, mapsize);
> + if (remap_pfn_range(vma, vma->vm_start + user_skip,
> + paddr >> PAGE_SHIFT, mapsize,
> + vma->vm_page_prot))
> + return -EAGAIN; // XXX check return value
> + user_skip += mapsize;
> + tomap -= mapsize;
> + if (tomap == 0)
> + goto done;
> + }
> + }
> +done:
> +
> + return 0;
> +}
> +
> +static netdev_tx_t
> +linux_netmap_start(struct sk_buff *skb, struct net_device *dev)
> +{
> + netmap_start(dev, skb);
> + return (NETDEV_TX_OK);
> +}
> +
> +
> +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,37) // XXX was 38
> +#define LIN_IOCTL_NAME .ioctl
> +int
> +linux_netmap_ioctl(struct inode *inode, struct file *file, u_int cmd, u_long data /* arg */)
> +#else
> +#define LIN_IOCTL_NAME .unlocked_ioctl
> +long
> +linux_netmap_ioctl(struct file *file, u_int cmd, u_long data /* arg */)
> +#endif
> +{
> + int ret;
> + struct nmreq nmr;
> + bzero(&nmr, sizeof(nmr));
> +
> + if (data && copy_from_user(&nmr, (void *)data, sizeof(nmr) ) != 0)
> + return -EFAULT;
> + ret = netmap_ioctl(NULL, cmd, (caddr_t)&nmr, 0, (void *)file);
> + pr_info("netmap ioctl %u ret = %d\n", cmd, ret);
> + if (data && copy_to_user((void*)data, &nmr, sizeof(nmr) ) != 0)
> + return -EFAULT;
> + return -ret;
> +}
> +
> +
> +static int
> +netmap_release(struct inode *inode, struct file *file)
> +{
> + (void)inode; /* UNUSED */
> + if (file->private_data)
> + netmap_dtor(file->private_data);
> + return (0);
> +}
> +
> +static int
> +linux_netmap_open(struct inode *inode, struct file *file)
> +{
> + struct netmap_priv_d *priv;
> + (void)inode; /* UNUSED */
> +
> + priv = malloc(sizeof(struct netmap_priv_d), M_DEVBUF,
> + M_NOWAIT | M_ZERO);
> + if (priv == NULL)
> + return -ENOMEM;
> +
> + file->private_data = priv;
> +
> + return (0);
> +}
> +
> +static struct file_operations netmap_fops = {
> + .open = linux_netmap_open,
> + .mmap = linux_netmap_mmap,
> + LIN_IOCTL_NAME = linux_netmap_ioctl,
> + .poll = linux_netmap_poll,
> + .release = netmap_release,
> +};
> +
> +static struct miscdevice netmap_cdevsw = { /* same name as FreeBSD */
> + MISC_DYNAMIC_MINOR,
> + "netmap",
> + &netmap_fops,
> +};
> +
> +static int netmap_init(void);
> +static void netmap_fini(void);
> +
> +/* Errors have negative values on linux */
> +static int linux_netmap_init(void)
> +{
> + return -netmap_init();
> +}
> +
> +module_init(linux_netmap_init);
> +module_exit(netmap_fini);
> +/* export certain symbols to other modules */
> +EXPORT_SYMBOL(netmap_attach); // driver attach routines
> +EXPORT_SYMBOL(netmap_detach); // driver detach routines
> +EXPORT_SYMBOL(netmap_ring_reinit); // ring init on error
> +EXPORT_SYMBOL(netmap_buffer_lut);
> +EXPORT_SYMBOL(netmap_total_buffers); // index check
> +EXPORT_SYMBOL(netmap_buffer_base);
> +EXPORT_SYMBOL(netmap_reset); // ring init routines
> +EXPORT_SYMBOL(netmap_buf_size);
> +EXPORT_SYMBOL(netmap_rx_irq); // default irq handler
> +EXPORT_SYMBOL(netmap_no_pendintr); // XXX mitigation - should go away
> +
> +
> +MODULE_AUTHOR("Matteo Landi, Luigi Rizzo");
> +MODULE_DESCRIPTION("The netmap packet I/O framework");
> +MODULE_LICENSE("Dual BSD/GPL"); /* the code here is all BSD. */
> +
> +#else /* __FreeBSD__ */
> +
> +static struct cdevsw netmap_cdevsw = {
> + .d_version = D_VERSION,
> + .d_name = "netmap",
> + .d_open = netmap_open,
> + .d_mmap = netmap_mmap,
> + .d_mmap_single = netmap_mmap_single,
> + .d_ioctl = netmap_ioctl,
> + .d_poll = netmap_poll,
> + .d_close = netmap_close,
> +};
> +#endif /* __FreeBSD__ */
> +
> +#ifdef NM_BRIDGE
> +/*
> + *---- support for virtual bridge -----
> + */
> +
> +/* ----- FreeBSD if_bridge hash function ------- */
> +
> +/*
> + * The following hash function is adapted from "Hash Functions" by Bob Jenkins
> + * ("Algorithm Alley", Dr. Dobbs Journal, September 1997).
> + *
> + * http://www.burtleburtle.net/bob/hash/spooky.html
> + */
> +#define mix(a, b, c) \
> +do { \
> + a -= b; a -= c; a ^= (c >> 13); \
> + b -= c; b -= a; b ^= (a << 8); \
> + c -= a; c -= b; c ^= (b >> 13); \
> + a -= b; a -= c; a ^= (c >> 12); \
> + b -= c; b -= a; b ^= (a << 16); \
> + c -= a; c -= b; c ^= (b >> 5); \
> + a -= b; a -= c; a ^= (c >> 3); \
> + b -= c; b -= a; b ^= (a << 10); \
> + c -= a; c -= b; c ^= (b >> 15); \
> +} while (/*CONSTCOND*/0)
> +
> +static __inline uint32_t
> +nm_bridge_rthash(const uint8_t *addr)
> +{
> + uint32_t a = 0x9e3779b9, b = 0x9e3779b9, c = 0; // hask key
> +
> + b += addr[5] << 8;
> + b += addr[4];
> + a += addr[3] << 24;
> + a += addr[2] << 16;
> + a += addr[1] << 8;
> + a += addr[0];
> +
> + mix(a, b, c);
> +#define BRIDGE_RTHASH_MASK (NM_BDG_HASH-1)
> + return (c & BRIDGE_RTHASH_MASK);
> +}
> +
> +#undef mix
> +
> +
> +static int
> +bdg_netmap_reg(struct ifnet *ifp, int onoff)
> +{
> + int i, err = 0;
> + struct nm_bridge *b = ifp->if_bridge;
> +
> + BDG_LOCK(b);
> + if (onoff) {
> + /* the interface must be already in the list.
> + * only need to mark the port as active
> + */
> + ND("should attach %s to the bridge", ifp->if_xname);
> + for (i=0; i < NM_BDG_MAXPORTS; i++)
> + if (b->bdg_ports[i] == ifp)
> + break;
> + if (i == NM_BDG_MAXPORTS) {
> + D("no more ports available");
> + err = EINVAL;
> + goto done;
> + }
> + ND("setting %s in netmap mode", ifp->if_xname);
> + ifp->if_capenable |= IFCAP_NETMAP;
> + NA(ifp)->bdg_port = i;
> + b->act_ports |= (1<<i);
> + b->bdg_ports[i] = ifp;
> + } else {
> + /* should be in the list, too -- remove from the mask */
> + ND("removing %s from netmap mode", ifp->if_xname);
> + ifp->if_capenable &= ~IFCAP_NETMAP;
> + i = NA(ifp)->bdg_port;
> + b->act_ports &= ~(1<<i);
> + }
> +done:
> + BDG_UNLOCK(b);
> + return err;
> +}
> +
> +
> +static int
> +nm_bdg_flush(struct nm_bdg_fwd *ft, int n, struct ifnet *ifp)
> +{
> + int i, ifn;
> + uint64_t all_dst, dst;
> + uint32_t sh, dh;
> + uint64_t mysrc = 1 << NA(ifp)->bdg_port;
> + uint64_t smac, dmac;
> + struct netmap_slot *slot;
> + struct nm_bridge *b = ifp->if_bridge;
> +
> + ND("prepare to send %d packets, act_ports 0x%x", n, b->act_ports);
> + /* only consider valid destinations */
> + all_dst = (b->act_ports & ~mysrc);
> + /* first pass: hash and find destinations */
> + for (i = 0; likely(i < n); i++) {
> + uint8_t *buf = ft[i].buf;
> + dmac = le64toh(*(uint64_t *)(buf)) & 0xffffffffffff;
> + smac = le64toh(*(uint64_t *)(buf + 4));
> + smac >>= 16;
> + if (unlikely(netmap_verbose)) {
> + uint8_t *s = buf+6, *d = buf;
> + D("%d len %4d %02x:%02x:%02x:%02x:%02x:%02x -> %02x:%02x:%02x:%02x:%02x:%02x",
> + i,
> + ft[i].len,
> + s[0], s[1], s[2], s[3], s[4], s[5],
> + d[0], d[1], d[2], d[3], d[4], d[5]);
> + }
> + /*
> + * The hash is somewhat expensive, there might be some
> + * worthwhile optimizations here.
> + */
> + if ((buf[6] & 1) == 0) { /* valid src */
> + uint8_t *s = buf+6;
> + sh = nm_bridge_rthash(buf+6); // XXX hash of source
> + /* update source port forwarding entry */
> + b->ht[sh].mac = smac; /* XXX expire ? */
> + b->ht[sh].ports = mysrc;
> + if (netmap_verbose)
> + D("src %02x:%02x:%02x:%02x:%02x:%02x on port %d",
> + s[0], s[1], s[2], s[3], s[4], s[5], NA(ifp)->bdg_port);
> + }
> + dst = 0;
> + if ( (buf[0] & 1) == 0) { /* unicast */
> + uint8_t *d = buf;
> + dh = nm_bridge_rthash(buf); // XXX hash of dst
> + if (b->ht[dh].mac == dmac) { /* found dst */
> + dst = b->ht[dh].ports;
> + if (netmap_verbose)
> + D("dst %02x:%02x:%02x:%02x:%02x:%02x to port %x",
> + d[0], d[1], d[2], d[3], d[4], d[5], (uint32_t)(dst >> 16));
> + }
> + }
> + if (dst == 0)
> + dst = all_dst;
> + dst &= all_dst; /* only consider valid ports */
> + if (unlikely(netmap_verbose))
> + D("pkt goes to ports 0x%x", (uint32_t)dst);
> + ft[i].dst = dst;
> + }
> +
> + /* second pass, scan interfaces and forward */
> + all_dst = (b->act_ports & ~mysrc);
> + for (ifn = 0; all_dst; ifn++) {
> + struct ifnet *dst_ifp = b->bdg_ports[ifn];
> + struct netmap_adapter *na;
> + struct netmap_kring *kring;
> + struct netmap_ring *ring;
> + int j, lim, sent, locked;
> +
> + if (!dst_ifp)
> + continue;
> + ND("scan port %d %s", ifn, dst_ifp->if_xname);
> + dst = 1 << ifn;
> + if ((dst & all_dst) == 0) /* skip if not set */
> + continue;
> + all_dst &= ~dst; /* clear current node */
> + na = NA(dst_ifp);
> +
> + ring = NULL;
> + kring = NULL;
> + lim = sent = locked = 0;
> + /* inside, scan slots */
> + for (i = 0; likely(i < n); i++) {
> + if ((ft[i].dst & dst) == 0)
> + continue; /* not here */
> + if (!locked) {
> + kring = &na->rx_rings[0];
> + ring = kring->ring;
> + lim = kring->nkr_num_slots - 1;
> + na->nm_lock(dst_ifp, NETMAP_RX_LOCK, 0);
> + locked = 1;
> + }
> + if (unlikely(kring->nr_hwavail >= lim)) {
> + if (netmap_verbose)
> + D("rx ring full on %s", ifp->if_xname);
> + break;
> + }
> + j = kring->nr_hwcur + kring->nr_hwavail;
> + if (j > lim)
> + j -= kring->nkr_num_slots;
> + slot = &ring->slot[j];
> + ND("send %d %d bytes at %s:%d", i, ft[i].len, dst_ifp->if_xname, j);
> + pkt_copy(ft[i].buf, NMB(slot), ft[i].len);
> + slot->len = ft[i].len;
> + kring->nr_hwavail++;
> + sent++;
> + }
> + if (locked) {
> + ND("sent %d on %s", sent, dst_ifp->if_xname);
> + if (sent)
> + selwakeuppri(&kring->si, PI_NET);
> + na->nm_lock(dst_ifp, NETMAP_RX_UNLOCK, 0);
> + }
> + }
> + return 0;
> +}
> +
> +/*
> + * main dispatch routine
> + */
> +static int
> +bdg_netmap_txsync(struct ifnet *ifp, u_int ring_nr, int do_lock)
> +{
> + struct netmap_adapter *na = NA(ifp);
> + struct netmap_kring *kring = &na->tx_rings[ring_nr];
> + struct netmap_ring *ring = kring->ring;
> + int i, j, k, lim = kring->nkr_num_slots - 1;
> + struct nm_bdg_fwd *ft = (struct nm_bdg_fwd *)(ifp + 1);
> + int ft_i; /* position in the forwarding table */
> +
> + k = ring->cur;
> + if (k > lim)
> + return netmap_ring_reinit(kring);
> + if (do_lock)
> + na->nm_lock(ifp, NETMAP_TX_LOCK, ring_nr);
> +
> + if (netmap_bridge <= 0) { /* testing only */
> + j = k; // used all
> + goto done;
> + }
> + if (netmap_bridge > NM_BDG_BATCH)
> + netmap_bridge = NM_BDG_BATCH;
> +
> + ft_i = 0; /* start from 0 */
> + for (j = kring->nr_hwcur; likely(j != k); j = unlikely(j == lim) ? 0 : j+1) {
> + struct netmap_slot *slot = &ring->slot[j];
> + int len = ft[ft_i].len = slot->len;
> + char *buf = ft[ft_i].buf = NMB(slot);
> +
> + prefetch(buf);
> + if (unlikely(len < 14))
> + continue;
> + if (unlikely(++ft_i == netmap_bridge))
> + ft_i = nm_bdg_flush(ft, ft_i, ifp);
> + }
> + if (ft_i)
> + ft_i = nm_bdg_flush(ft, ft_i, ifp);
> + /* count how many packets we sent */
> + i = k - j;
> + if (i < 0)
> + i += kring->nkr_num_slots;
> + kring->nr_hwavail = kring->nkr_num_slots - 1 - i;
> + if (j != k)
> + D("early break at %d/ %d, avail %d", j, k, kring->nr_hwavail);
> +
> +done:
> + kring->nr_hwcur = j;
> + ring->avail = kring->nr_hwavail;
> + if (do_lock)
> + na->nm_lock(ifp, NETMAP_TX_UNLOCK, ring_nr);
> +
> + if (netmap_verbose)
> + D("%s ring %d lock %d", ifp->if_xname, ring_nr, do_lock);
> + return 0;
> +}
> +
> +static int
> +bdg_netmap_rxsync(struct ifnet *ifp, u_int ring_nr, int do_lock)
> +{
> + struct netmap_adapter *na = NA(ifp);
> + struct netmap_kring *kring = &na->rx_rings[ring_nr];
> + struct netmap_ring *ring = kring->ring;
> + u_int j, n, lim = kring->nkr_num_slots - 1;
> + u_int k = ring->cur, resvd = ring->reserved;
> +
> + ND("%s ring %d lock %d avail %d",
> + ifp->if_xname, ring_nr, do_lock, kring->nr_hwavail);
> +
> + if (k > lim)
> + return netmap_ring_reinit(kring);
> + if (do_lock)
> + na->nm_lock(ifp, NETMAP_RX_LOCK, ring_nr);
> +
> + /* skip past packets that userspace has released */
> + j = kring->nr_hwcur; /* netmap ring index */
> + if (resvd > 0) {
> + if (resvd + ring->avail >= lim + 1) {
> + D("XXX invalid reserve/avail %d %d", resvd, ring->avail);
> + ring->reserved = resvd = 0; // XXX panic...
> + }
> + k = (k >= resvd) ? k - resvd : k + lim + 1 - resvd;
> + }
> +
> + if (j != k) { /* userspace has released some packets. */
> + n = k - j;
> + if (n < 0)
> + n += kring->nkr_num_slots;
> + ND("userspace releases %d packets", n);
> + for (n = 0; likely(j != k); n++) {
> + struct netmap_slot *slot = &ring->slot[j];
> + void *addr = NMB(slot);
> +
> + if (addr == netmap_buffer_base) { /* bad buf */
> + if (do_lock)
> + na->nm_lock(ifp, NETMAP_RX_UNLOCK, ring_nr);
> + return netmap_ring_reinit(kring);
> + }
> + /* decrease refcount for buffer */
> +
> + slot->flags &= ~NS_BUF_CHANGED;
> + j = unlikely(j == lim) ? 0 : j + 1;
> + }
> + kring->nr_hwavail -= n;
> + kring->nr_hwcur = k;
> + }
> + /* tell userspace that there are new packets */
> + ring->avail = kring->nr_hwavail - resvd;
> +
> + if (do_lock)
> + na->nm_lock(ifp, NETMAP_RX_UNLOCK, ring_nr);
> + return 0;
> +}
> +
> +static void
> +bdg_netmap_attach(struct ifnet *ifp)
> +{
> + struct netmap_adapter na;
> +
> + ND("attaching virtual bridge");
> + bzero(&na, sizeof(na));
> +
> + na.ifp = ifp;
> + na.separate_locks = 1;
> + na.num_tx_desc = NM_BRIDGE_RINGSIZE;
> + na.num_rx_desc = NM_BRIDGE_RINGSIZE;
> + na.nm_txsync = bdg_netmap_txsync;
> + na.nm_rxsync = bdg_netmap_rxsync;
> + na.nm_register = bdg_netmap_reg;
> + netmap_attach(&na, 1);
> +}
> +
> +#endif /* NM_BRIDGE */
> +
> +static struct cdev *netmap_dev; /* /dev/netmap character device. */
> +
> +
> +/*
> + * Module loader.
> + *
> + * Create the /dev/netmap device and initialize all global
> + * variables.
> + *
> + * Return 0 on success, errno on failure.
> + */
> +static int
> +netmap_init(void)
> +{
> + int error;
> +
> + error = netmap_memory_init();
> + if (error != 0) {
> + printf("netmap: unable to initialize the memory allocator.\n");
> + return (error);
> + }
> + printf("netmap: loaded module\n");
> + netmap_dev = make_dev(&netmap_cdevsw, 0, UID_ROOT, GID_WHEEL, 0660,
> + "netmap");
> +
> +#ifdef NM_BRIDGE
> + {
> + int i;
> + for (i = 0; i < NM_BRIDGES; i++)
> + mtx_init(&nm_bridges[i].bdg_lock, "bdg lock", "bdg_lock", MTX_DEF);
> + }
> +#endif
> + return (error);
> +}
> +
> +
> +/*
> + * Module unloader.
> + *
> + * Free all the memory, and destroy the ``/dev/netmap`` device.
> + */
> +static void
> +netmap_fini(void)
> +{
> + destroy_dev(netmap_dev);
> + netmap_memory_fini();
> + printf("netmap: unloaded module.\n");
> +}
> +
> +
> +#ifdef __FreeBSD__
> +/*
> + * Kernel entry point.
> + *
> + * Initialize/finalize the module and return.
> + *
> + * Return 0 on success, errno on failure.
> + */
> +static int
> +netmap_loader(__unused struct module *module, int event, __unused void *arg)
> +{
> + int error = 0;
> +
> + switch (event) {
> + case MOD_LOAD:
> + error = netmap_init();
> + break;
> +
> + case MOD_UNLOAD:
> + netmap_fini();
> + break;
> +
> + default:
> + error = EOPNOTSUPP;
> + break;
> + }
> +
> + return (error);
> +}
> +
> +
> +DEV_MODULE(netmap, netmap_loader, NULL);
> +#endif /* __FreeBSD__ */
> --- /dev/null 1970-01-01 00:00:00.000000000 +0000
> +++ b/drivers/staging/netmap/README 2013-03-10 10:08:20.327671428 -0700
> @@ -0,0 +1,127 @@
> +# $Id: README 10832 2012-03-22 18:22:42Z luigi $
> +
> +NETMAP FOR LINUX
> +----------------
> +
> +This directory contains a version of the "netmap" code for Linux.
> +
> +Netmap is a BSD-licensed framework that supports line-rate direct packet
> +I/O even on 10GBit/s interfaces (14.88Mpps) with limited system load,
> +and includes a libpcap emulation library to port applications. See
> +
> + http://info.iet.unipi.it/~luigi/netmap/
> +
> +for more details. There you can also find the latest versions
> +of the code and documentation as well as pre-built TinyCore
> +images based on linux 3.0.3 and containing the netmap modules
> +and some test applications.
> +
> +This is a preliminary version supporting the ixgbe and e1000/e1000e
> +driver. Patches for other devices (igb, r8169, forcedeth) are
> +untested and probably not working yet.
> +
> +Netmap relies on a kernel module (netmap_lin.ko) and slightly modified
> +device drivers. Userspace programs can use the native API (documented
> +in netmap.4) or a libpcap emulation library.
> +
> + Directory structure for this archive
> +
> + . documentation, patches etc.
> + include/net header files for user programs
> + net/netmap kernel core files,
> + sample applications, manpage
> + net/* patched device drivers for a 3.0.x linux version.
> +
> +HOW TO BUILD THE CODE
> +---------------------
> +
> +1. make sure you have kernel sources/headers matching your installed system
> +
> +2. do the following
> + (cd net; make KSRC=/usr/src/linux-kernel-source-or-headers )
> + this produces net/netmap/netmap_lin.ko and other kernel modules.
> +
> +3. to build sample applications, run
> + (cd net/netmap; make apps )
> + (you will need the pthreads and libpcap-dev packages to build them)
> +
> +HOW TO USE THE CODE
> +-------------------
> +
> + REMEMBER
> + THIS IS EXPERIMENTAL CODE WHICH MAY CRASH YOUR SYSTEM.
> + USE IT AT YOUR OWN RISk.
> +
> +Whether you built your own modules, or are using the prebuilt
> +TinyCore image, the following steps can be used for initial testing:
> +
> +1. unload any modules for the network cards you want to use, e.g.
> + sudo rmmod ixgbe
> + sudo rmmod e1000
> + sudo rmmod e1000e
> +
> +2. load netmap and device driver module
> + sudo insmod net/netmap/netmap_lin.ko
> + sudo insmod net/ixgbe/ixgbe.ko
> + sudo insmod net/ixgbe/e1000.ko
> + sudo insmod net/ixgbe/e1000e.ko
> +
> +3. turn the interface(s) up
> +
> + sudo ifconfig eth0 up # and same for others
> +
> +4. Run test applications -- as an example, pkt-gen is a raw packet
> + sender/receiver which can do line rate on a 10G interface
> +
> + # send about 500 million packets of 64 bytes each.
> + # wait 5s before starting, so the link can go up
> + sudo pkt-gen -i eth0 -t 500111222 -l 64 -w 5
> + # you should see about 14.2 Mpps
> +
> + sudo pkt-gen -i eth0 # act as a receiver
> +
> +
> +COMMON PROBLEMS
> +----------------
> +
> +* switching in/out of netmap mode causes the link to go down and up.
> + If your card is connected to a switch with spanning tree enabled,
> + the switch will likely MUTE THE LINK FOR 10 SECONDS while it is
> + detecting the new topology. Either disable the spanning tree on
> + the switch or use long pauses before sending data;
> +
> +* Not all cards can do line rate no matter how fast is your software or
> + CPU. Several have hardware limitations that prevent reaching the peak
> + speed, especially for small packet sizes. Examples:
> +
> + - ixgbe cannot receive at line rate with packet sizes that are
> + not multiple of 64 (after CRC stripping).
> + This is especially evident with minimum-sized frames (-l 60 )
> +
> + - some of the low-end 'e1000' cards can send 1.2 - 1.3Mpps instead
> + of the theoretical maximum (1.488Mpps)
> +
> + - the 'realtek' cards seem unable to send more than 450-500Kpps
> + even though they can receive at least 1.1Mpps
> +
> +* if the link is not up when the packet generator starts, you will
> + see frequent messages about a link reset. While we work on a fix,
> + use the '-w' argument on the generator to specify a longer timeout
> +
> +* the ixgbe driver (and perhaps others) is severely slowed down if the
> + remote party is senting flow control frames to slow down traffic.
> + If that happens try to use the ethtool command to disable flow control.
> +
> +
> +REVISION HISTORY
> +-----------------
> +
> +20120322 - fixed the 'igb' driver, now it can send and receive correctly
> + (the problem was in netmap_rx_irq() so it might have affected
> + other multiqueue cards).
> + Also tested the 'r8169' in transmit mode.
> + Added comments on switches and spanning tree.
> +
> +20120217 - initial version. Only ixgbe, e1000 and e1000e are working.
> + Other drivers (igb, r8169, forcedeth) are supplied only as a
> + proof of concept.
> --- /dev/null 1970-01-01 00:00:00.000000000 +0000
> +++ b/drivers/staging/netmap/TODO 2013-03-10 16:45:40.377960319 -0700
> @@ -0,0 +1,21 @@
> +
> +This driver is functional but needs some cleanup before it is ready
> +for included in networking subsystem.
> +
> + - Use unifdef to elimiate non Linux code
> + - Get rid of wrapper code including bsd_glue.h
> + - Fix coding style to be Linux rather than BSD
> + - Fix whitespace warnings
> + - Add stubs so that ethernet drivers can use without #ifdef's
> + - Autoload the module, assign a real minor number and have aliases
> + - Rework documentation and put in Documentation/networking/
> + - Remove memory allocator kludge wrapper
> + - Remove devfs kludge wrappers
> + - Review configuration API's and management (counters)
> + - Locking should be consistent (ie get rid of separate_locks option)?
> + - Doesn't work with DMA remapping
> + - Use __u32 rather than uint32_t in headers
> + - netmap_user.h headers should use inline instead of macros
> +
> +Fundamentally this will break source compatiablity with FreeBSD,
> +but this is not our problem.
> --- /dev/null 1970-01-01 00:00:00.000000000 +0000
> +++ b/drivers/staging/netmap/netmap_mem2.c 2013-03-10 10:08:20.327671428 -0700
> @@ -0,0 +1,974 @@
> +/*
> + * Copyright (C) 2012 Matteo Landi, Luigi Rizzo, Giuseppe Lettieri. All rights reserved.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + * notice, this list of conditions and the following disclaimer.
> + * 2. Redistributions in binary form must reproduce the above copyright
> + * notice, this list of conditions and the following disclaimer in the
> + * documentation and/or other materials provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
> + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
> + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
> + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +/*
> + * $FreeBSD: head/sys/dev/netmap/netmap_mem2.c 234290 2012-04-14 16:44:18Z luigi $
> + * $Id: netmap_mem2.c 12010 2013-01-23 03:57:30Z luigi $
> + *
> + * (New) memory allocator for netmap
> + */
> +
> +/*
> + * This allocator creates three memory regions:
> + * nm_if_pool for the struct netmap_if
> + * nm_ring_pool for the struct netmap_ring
> + * nm_buf_pool for the packet buffers.
> + *
> + * All regions need to be multiple of a page size as we export them to
> + * userspace through mmap. Only the latter needs to be dma-able,
> + * but for convenience use the same type of allocator for all.
> + *
> + * Once mapped, the three regions are exported to userspace
> + * as a contiguous block, starting from nm_if_pool. Each
> + * cluster (and pool) is an integral number of pages.
> + * [ . . . ][ . . . . . .][ . . . . . . . . . .]
> + * nm_if nm_ring nm_buf
> + *
> + * The userspace areas contain offsets of the objects in userspace.
> + * When (at init time) we write these offsets, we find out the index
> + * of the object, and from there locate the offset from the beginning
> + * of the region.
> + *
> + * The invididual allocators manage a pool of memory for objects of
> + * the same size.
> + * The pool is split into smaller clusters, whose size is a
> + * multiple of the page size. The cluster size is chosen
> + * to minimize the waste for a given max cluster size
> + * (we do it by brute force, as we have relatively few object
> + * per cluster).
> + *
> + * Objects are aligned to the cache line (64 bytes) rounding up object
> + * sizes when needed. A bitmap contains the state of each object.
> + * Allocation scans the bitmap; this is done only on attach, so we are not
> + * too worried about performance
> + *
> + * For each allocator we can define (thorugh sysctl) the size and
> + * number of each object. Memory is allocated at the first use of a
> + * netmap file descriptor, and can be freed when all such descriptors
> + * have been released (including unmapping the memory).
> + * If memory is scarce, the system tries to get as much as possible
> + * and the sysctl values reflect the actual allocation.
> + * Together with desired values, the sysctl export also absolute
> + * min and maximum values that cannot be overridden.
> + *
> + * struct netmap_if:
> + * variable size, max 16 bytes per ring pair plus some fixed amount.
> + * 1024 bytes should be large enough in practice.
> + *
> + * In the worst case we have one netmap_if per ring in the system.
> + *
> + * struct netmap_ring
> + * variable too, 8 byte per slot plus some fixed amount.
> + * Rings can be large (e.g. 4k slots, or >32Kbytes).
> + * We default to 36 KB (9 pages), and a few hundred rings.
> + *
> + * struct netmap_buffer
> + * The more the better, both because fast interfaces tend to have
> + * many slots, and because we may want to use buffers to store
> + * packets in userspace avoiding copies.
> + * Must contain a full frame (eg 1518, or more for vlans, jumbo
> + * frames etc.) plus be nicely aligned, plus some NICs restrict
> + * the size to multiple of 1K or so. Default to 2K
> + */
> +
> +#ifndef CONSERVATIVE
> +#define NETMAP_BUF_MAX_NUM 20*4096*2 /* large machine */
> +#else /* CONSERVATIVE */
> +#define NETMAP_BUF_MAX_NUM 20000 /* 40MB */
> +#endif
> +
> +#ifdef linux
> +#define NMA_LOCK_T struct semaphore
> +#define NMA_LOCK_INIT() sema_init(&nm_mem.nm_mtx, 1)
> +#define NMA_LOCK_DESTROY()
> +#define NMA_LOCK() down(&nm_mem.nm_mtx)
> +#define NMA_UNLOCK() up(&nm_mem.nm_mtx)
> +#else /* !linux */
> +#define NMA_LOCK_T struct mtx
> +#define NMA_LOCK_INIT() mtx_init(&nm_mem.nm_mtx, "netmap memory allocator lock", NULL, MTX_DEF)
> +#define NMA_LOCK_DESTROY() mtx_destroy(&nm_mem.nm_mtx)
> +#define NMA_LOCK() mtx_lock(&nm_mem.nm_mtx)
> +#define NMA_UNLOCK() mtx_unlock(&nm_mem.nm_mtx)
> +#endif /* linux */
> +
> +enum {
> + NETMAP_IF_POOL = 0,
> + NETMAP_RING_POOL,
> + NETMAP_BUF_POOL,
> + NETMAP_POOLS_NR
> +};
> +
> +
> +struct netmap_obj_params {
> + u_int size;
> + u_int num;
> +};
> +
> +
> +struct netmap_obj_params netmap_params[NETMAP_POOLS_NR] = {
> + [NETMAP_IF_POOL] = {
> + .size = 1024,
> + .num = 100,
> + },
> + [NETMAP_RING_POOL] = {
> + .size = 9*PAGE_SIZE,
> + .num = 200,
> + },
> + [NETMAP_BUF_POOL] = {
> + .size = 2048,
> + .num = NETMAP_BUF_MAX_NUM,
> + },
> +};
> +
> +
> +struct netmap_obj_pool {
> + char name[16]; /* name of the allocator */
> + u_int objtotal; /* actual total number of objects. */
> + u_int objfree; /* number of free objects. */
> + u_int clustentries; /* actual objects per cluster */
> +
> + /* limits */
> + u_int objminsize; /* minimum object size */
> + u_int objmaxsize; /* maximum object size */
> + u_int nummin; /* minimum number of objects */
> + u_int nummax; /* maximum number of objects */
> +
> + /* the total memory space is _numclusters*_clustsize */
> + u_int _numclusters; /* how many clusters */
> + u_int _clustsize; /* cluster size */
> + u_int _objsize; /* actual object size */
> +
> + u_int _memtotal; /* _numclusters*_clustsize */
> + struct lut_entry *lut; /* virt,phys addresses, objtotal entries */
> + uint32_t *bitmap; /* one bit per buffer, 1 means free */
> + uint32_t bitmap_slots; /* number of uint32 entries in bitmap */
> +};
> +
> +
> +struct netmap_mem_d {
> + NMA_LOCK_T nm_mtx; /* protect the allocator */
> + u_int nm_totalsize; /* shorthand */
> +
> + int finalized; /* !=0 iff preallocation done */
> + int lasterr; /* last error for curr config */
> + int refcount; /* existing priv structures */
> + /* the three allocators */
> + struct netmap_obj_pool pools[NETMAP_POOLS_NR];
> +};
> +
> +
> +static struct netmap_mem_d nm_mem = { /* Our memory allocator. */
> + .pools = {
> + [NETMAP_IF_POOL] = {
> + .name = "netmap_if",
> + .objminsize = sizeof(struct netmap_if),
> + .objmaxsize = 4096,
> + .nummin = 10, /* don't be stingy */
> + .nummax = 10000, /* XXX very large */
> + },
> + [NETMAP_RING_POOL] = {
> + .name = "netmap_ring",
> + .objminsize = sizeof(struct netmap_ring),
> + .objmaxsize = 32*PAGE_SIZE,
> + .nummin = 2,
> + .nummax = 1024,
> + },
> + [NETMAP_BUF_POOL] = {
> + .name = "netmap_buf",
> + .objminsize = 64,
> + .objmaxsize = 65536,
> + .nummin = 4,
> + .nummax = 1000000, /* one million! */
> + },
> + },
> +};
> +
> +struct lut_entry *netmap_buffer_lut; /* exported */
> +
> +/* memory allocator related sysctls */
> +
> +#define STRINGIFY(x) #x
> +
> +#define DECLARE_SYSCTLS(id, name) \
> + /* TUNABLE_INT("hw.netmap." STRINGIFY(name) "_size", &netmap_params[id].size); */ \
> + SYSCTL_INT(_dev_netmap, OID_AUTO, name##_size, \
> + CTLFLAG_RW, &netmap_params[id].size, 0, "Requested size of netmap " STRINGIFY(name) "s"); \
> + SYSCTL_INT(_dev_netmap, OID_AUTO, name##_curr_size, \
> + CTLFLAG_RD, &nm_mem.pools[id]._objsize, 0, "Current size of netmap " STRINGIFY(name) "s"); \
> + /* TUNABLE_INT("hw.netmap." STRINGIFY(name) "_num", &netmap_params[id].num); */ \
> + SYSCTL_INT(_dev_netmap, OID_AUTO, name##_num, \
> + CTLFLAG_RW, &netmap_params[id].num, 0, "Requested number of netmap " STRINGIFY(name) "s"); \
> + SYSCTL_INT(_dev_netmap, OID_AUTO, name##_curr_num, \
> + CTLFLAG_RD, &nm_mem.pools[id].objtotal, 0, "Current number of netmap " STRINGIFY(name) "s")
> +
> +DECLARE_SYSCTLS(NETMAP_IF_POOL, if);
> +DECLARE_SYSCTLS(NETMAP_RING_POOL, ring);
> +DECLARE_SYSCTLS(NETMAP_BUF_POOL, buf);
> +
> +/*
> + * Convert a userspace offset to a phisical address.
> + * XXX re-do in a simpler way.
> + *
> + * The idea here is to hide userspace applications the fact that pre-allocated
> + * memory is not contiguous, but fragmented across different clusters and
> + * smaller memory allocators. Consequently, first of all we need to find which
> + * allocator is owning provided offset, then we need to find out the physical
> + * address associated to target page (this is done using the look-up table.
> + */
> +static inline vm_paddr_t
> +netmap_ofstophys(vm_offset_t offset)
> +{
> + int i;
> + vm_offset_t o = offset;
> + struct netmap_obj_pool *p = nm_mem.pools;
> +
> + for (i = 0; i < NETMAP_POOLS_NR; offset -= p[i]._memtotal, i++) {
> + if (offset >= p[i]._memtotal)
> + continue;
> + // XXX now scan the clusters
> + return p[i].lut[offset / p[i]._objsize].paddr +
> + offset % p[i]._objsize;
> + }
> + /* this is only in case of errors */
> + D("invalid ofs 0x%x out of 0x%x 0x%x 0x%x", (u_int)o,
> + p[NETMAP_IF_POOL]._memtotal,
> + p[NETMAP_IF_POOL]._memtotal
> + + p[NETMAP_RING_POOL]._memtotal,
> + p[NETMAP_IF_POOL]._memtotal
> + + p[NETMAP_RING_POOL]._memtotal
> + + p[NETMAP_BUF_POOL]._memtotal);
> + return 0; // XXX bad address
> +}
> +
> +/*
> + * we store objects by kernel address, need to find the offset
> + * within the pool to export the value to userspace.
> + * Algorithm: scan until we find the cluster, then add the
> + * actual offset in the cluster
> + */
> +static ssize_t
> +netmap_obj_offset(struct netmap_obj_pool *p, const void *vaddr)
> +{
> + int i, k = p->clustentries, n = p->objtotal;
> + ssize_t ofs = 0;
> +
> + for (i = 0; i < n; i += k, ofs += p->_clustsize) {
> + const char *base = p->lut[i].vaddr;
> + ssize_t relofs = (const char *) vaddr - base;
> +
> + if (relofs < 0 || relofs > p->_clustsize)
> + continue;
> +
> + ofs = ofs + relofs;
> + ND("%s: return offset %d (cluster %d) for pointer %p",
> + p->name, ofs, i, vaddr);
> + return ofs;
> + }
> + D("address %p is not contained inside any cluster (%s)",
> + vaddr, p->name);
> + return 0; /* An error occurred */
> +}
> +
> +/* Helper functions which convert virtual addresses to offsets */
> +#define netmap_if_offset(v) \
> + netmap_obj_offset(&nm_mem.pools[NETMAP_IF_POOL], (v))
> +
> +#define netmap_ring_offset(v) \
> + (nm_mem.pools[NETMAP_IF_POOL]._memtotal + \
> + netmap_obj_offset(&nm_mem.pools[NETMAP_RING_POOL], (v)))
> +
> +#define netmap_buf_offset(v) \
> + (nm_mem.pools[NETMAP_IF_POOL]._memtotal + \
> + nm_mem.pools[NETMAP_RING_POOL]._memtotal + \
> + netmap_obj_offset(&nm_mem.pools[NETMAP_BUF_POOL], (v)))
> +
> +
> +/*
> + * report the index, and use start position as a hint,
> + * otherwise buffer allocation becomes terribly expensive.
> + */
> +static void *
> +netmap_obj_malloc(struct netmap_obj_pool *p, int len, uint32_t *start, uint32_t *index)
> +{
> + uint32_t i = 0; /* index in the bitmap */
> + uint32_t mask, j; /* slot counter */
> + void *vaddr = NULL;
> +
> + if (len > p->_objsize) {
> + D("%s request size %d too large", p->name, len);
> + // XXX cannot reduce the size
> + return NULL;
> + }
> +
> + if (p->objfree == 0) {
> + D("%s allocator: run out of memory", p->name);
> + return NULL;
> + }
> + if (start)
> + i = *start;
> +
> + /* termination is guaranteed by p->free, but better check bounds on i */
> + while (vaddr == NULL && i < p->bitmap_slots) {
> + uint32_t cur = p->bitmap[i];
> + if (cur == 0) { /* bitmask is fully used */
> + i++;
> + continue;
> + }
> + /* locate a slot */
> + for (j = 0, mask = 1; (cur & mask) == 0; j++, mask <<= 1)
> + ;
> +
> + p->bitmap[i] &= ~mask; /* mark object as in use */
> + p->objfree--;
> +
> + vaddr = p->lut[i * 32 + j].vaddr;
> + if (index)
> + *index = i * 32 + j;
> + }
> + ND("%s allocator: allocated object @ [%d][%d]: vaddr %p", i, j, vaddr);
> +
> + if (start)
> + *start = i;
> + return vaddr;
> +}
> +
> +
> +/*
> + * free by index, not by address
> + */
> +static void
> +netmap_obj_free(struct netmap_obj_pool *p, uint32_t j)
> +{
> + if (j >= p->objtotal) {
> + D("invalid index %u, max %u", j, p->objtotal);
> + return;
> + }
> + p->bitmap[j / 32] |= (1 << (j % 32));
> + p->objfree++;
> + return;
> +}
> +
> +static void
> +netmap_obj_free_va(struct netmap_obj_pool *p, void *vaddr)
> +{
> + int i, j, n = p->_memtotal / p->_clustsize;
> +
> + for (i = 0, j = 0; i < n; i++, j += p->clustentries) {
> + void *base = p->lut[i * p->clustentries].vaddr;
> + ssize_t relofs = (ssize_t) vaddr - (ssize_t) base;
> +
> + /* Given address, is out of the scope of the current cluster.*/
> + if (vaddr < base || relofs > p->_clustsize)
> + continue;
> +
> + j = j + relofs / p->_objsize;
> + KASSERT(j != 0, ("Cannot free object 0"));
> + netmap_obj_free(p, j);
> + return;
> + }
> + D("address %p is not contained inside any cluster (%s)",
> + vaddr, p->name);
> +}
> +
> +#define netmap_if_malloc(len) netmap_obj_malloc(&nm_mem.pools[NETMAP_IF_POOL], len, NULL, NULL)
> +#define netmap_if_free(v) netmap_obj_free_va(&nm_mem.pools[NETMAP_IF_POOL], (v))
> +#define netmap_ring_malloc(len) netmap_obj_malloc(&nm_mem.pools[NETMAP_RING_POOL], len, NULL, NULL)
> +#define netmap_ring_free(v) netmap_obj_free_va(&nm_mem.pools[NETMAP_RING_POOL], (v))
> +#define netmap_buf_malloc(_pos, _index) \
> + netmap_obj_malloc(&nm_mem.pools[NETMAP_BUF_POOL], NETMAP_BUF_SIZE, _pos, _index)
> +
> +
> +/* Return the index associated to the given packet buffer */
> +#define netmap_buf_index(v) \
> + (netmap_obj_offset(&nm_mem.pools[NETMAP_BUF_POOL], (v)) / nm_mem.pools[NETMAP_BUF_POOL]._objsize)
> +
> +
> +/* Return nonzero on error */
> +static int
> +netmap_new_bufs(struct netmap_if *nifp,
> + struct netmap_slot *slot, u_int n)
> +{
> + struct netmap_obj_pool *p = &nm_mem.pools[NETMAP_BUF_POOL];
> + int i = 0; /* slot counter */
> + uint32_t pos = 0; /* slot in p->bitmap */
> + uint32_t index = 0; /* buffer index */
> +
> + (void)nifp; /* UNUSED */
> + for (i = 0; i < n; i++) {
> + void *vaddr = netmap_buf_malloc(&pos, &index);
> + if (vaddr == NULL) {
> + D("unable to locate empty packet buffer");
> + goto cleanup;
> + }
> + slot[i].buf_idx = index;
> + slot[i].len = p->_objsize;
> + /* XXX setting flags=NS_BUF_CHANGED forces a pointer reload
> + * in the NIC ring. This is a hack that hides missing
> + * initializations in the drivers, and should go away.
> + */
> + slot[i].flags = NS_BUF_CHANGED;
> + }
> +
> + ND("allocated %d buffers, %d available, first at %d", n, p->objfree, pos);
> + return (0);
> +
> +cleanup:
> + while (i > 0) {
> + i--;
> + netmap_obj_free(p, slot[i].buf_idx);
> + }
> + bzero(slot, n * sizeof(slot[0]));
> + return (ENOMEM);
> +}
> +
> +
> +static void
> +netmap_free_buf(struct netmap_if *nifp, uint32_t i)
> +{
> + struct netmap_obj_pool *p = &nm_mem.pools[NETMAP_BUF_POOL];
> +
> + if (i < 2 || i >= p->objtotal) {
> + D("Cannot free buf#%d: should be in [2, %d[", i, p->objtotal);
> + return;
> + }
> + netmap_obj_free(p, i);
> +}
> +
> +static void
> +netmap_reset_obj_allocator(struct netmap_obj_pool *p)
> +{
> + if (p == NULL)
> + return;
> + if (p->bitmap)
> + free(p->bitmap, M_NETMAP);
> + p->bitmap = NULL;
> + if (p->lut) {
> + int i;
> + for (i = 0; i < p->objtotal; i += p->clustentries) {
> + if (p->lut[i].vaddr)
> + contigfree(p->lut[i].vaddr, p->_clustsize, M_NETMAP);
> + }
> + bzero(p->lut, sizeof(struct lut_entry) * p->objtotal);
> +#ifdef linux
> + vfree(p->lut);
> +#else
> + free(p->lut, M_NETMAP);
> +#endif
> + }
> + p->lut = NULL;
> +}
> +
> +/*
> + * Free all resources related to an allocator.
> + */
> +static void
> +netmap_destroy_obj_allocator(struct netmap_obj_pool *p)
> +{
> + if (p == NULL)
> + return;
> + netmap_reset_obj_allocator(p);
> +}
> +
> +/*
> + * We receive a request for objtotal objects, of size objsize each.
> + * Internally we may round up both numbers, as we allocate objects
> + * in small clusters multiple of the page size.
> + * In the allocator we don't need to store the objsize,
> + * but we do need to keep track of objtotal' and clustentries,
> + * as they are needed when freeing memory.
> + *
> + * XXX note -- userspace needs the buffers to be contiguous,
> + * so we cannot afford gaps at the end of a cluster.
> + */
> +
> +
> +/* call with NMA_LOCK held */
> +static int
> +netmap_config_obj_allocator(struct netmap_obj_pool *p, u_int objtotal, u_int objsize)
> +{
> + int i, n;
> + u_int clustsize; /* the cluster size, multiple of page size */
> + u_int clustentries; /* how many objects per entry */
> +
> +#define MAX_CLUSTSIZE (1<<17)
> +#define LINE_ROUND 64
> + if (objsize >= MAX_CLUSTSIZE) {
> + /* we could do it but there is no point */
> + D("unsupported allocation for %d bytes", objsize);
> + goto error;
> + }
> + /* make sure objsize is a multiple of LINE_ROUND */
> + i = (objsize & (LINE_ROUND - 1));
> + if (i) {
> + D("XXX aligning object by %d bytes", LINE_ROUND - i);
> + objsize += LINE_ROUND - i;
> + }
> + if (objsize < p->objminsize || objsize > p->objmaxsize) {
> + D("requested objsize %d out of range [%d, %d]",
> + objsize, p->objminsize, p->objmaxsize);
> + goto error;
> + }
> + if (objtotal < p->nummin || objtotal > p->nummax) {
> + D("requested objtotal %d out of range [%d, %d]",
> + objtotal, p->nummin, p->nummax);
> + goto error;
> + }
> + /*
> + * Compute number of objects using a brute-force approach:
> + * given a max cluster size,
> + * we try to fill it with objects keeping track of the
> + * wasted space to the next page boundary.
> + */
> + for (clustentries = 0, i = 1;; i++) {
> + u_int delta, used = i * objsize;
> + if (used > MAX_CLUSTSIZE)
> + break;
> + delta = used % PAGE_SIZE;
> + if (delta == 0) { // exact solution
> + clustentries = i;
> + break;
> + }
> + if (delta > ( (clustentries*objsize) % PAGE_SIZE) )
> + clustentries = i;
> + }
> + // D("XXX --- ouch, delta %d (bad for buffers)", delta);
> + /* compute clustsize and round to the next page */
> + clustsize = clustentries * objsize;
> + i = (clustsize & (PAGE_SIZE - 1));
> + if (i)
> + clustsize += PAGE_SIZE - i;
> + if (netmap_verbose)
> + D("objsize %d clustsize %d objects %d",
> + objsize, clustsize, clustentries);
> +
> + /*
> + * The number of clusters is n = ceil(objtotal/clustentries)
> + * objtotal' = n * clustentries
> + */
> + p->clustentries = clustentries;
> + p->_clustsize = clustsize;
> + n = (objtotal + clustentries - 1) / clustentries;
> + p->_numclusters = n;
> + p->objtotal = n * clustentries;
> + p->objfree = p->objtotal - 2; /* obj 0 and 1 are reserved */
> + p->_memtotal = p->_numclusters * p->_clustsize;
> + p->_objsize = objsize;
> +
> + return 0;
> +
> +error:
> + p->_objsize = objsize;
> + p->objtotal = objtotal;
> +
> + return EINVAL;
> +}
> +
> +
> +/* call with NMA_LOCK held */
> +static int
> +netmap_finalize_obj_allocator(struct netmap_obj_pool *p)
> +{
> + int i, n;
> +
> + n = sizeof(struct lut_entry) * p->objtotal;
> +#ifdef linux
> + p->lut = vmalloc(n);
> +#else
> + p->lut = malloc(n, M_NETMAP, M_NOWAIT | M_ZERO);
> +#endif
> + if (p->lut == NULL) {
> + D("Unable to create lookup table (%d bytes) for '%s'", n, p->name);
> + goto clean;
> + }
> +
> + /* Allocate the bitmap */
> + n = (p->objtotal + 31) / 32;
> + p->bitmap = malloc(sizeof(uint32_t) * n, M_NETMAP, M_NOWAIT | M_ZERO);
> + if (p->bitmap == NULL) {
> + D("Unable to create bitmap (%d entries) for allocator '%s'", n,
> + p->name);
> + goto clean;
> + }
> + p->bitmap_slots = n;
> +
> + /*
> + * Allocate clusters, init pointers and bitmap
> + */
> + for (i = 0; i < p->objtotal;) {
> + int lim = i + p->clustentries;
> + char *clust;
> +
> + clust = contigmalloc(p->_clustsize, M_NETMAP, M_NOWAIT | M_ZERO,
> + 0, -1UL, PAGE_SIZE, 0);
> + if (clust == NULL) {
> + /*
> + * If we get here, there is a severe memory shortage,
> + * so halve the allocated memory to reclaim some.
> + * XXX check boundaries
> + */
> + D("Unable to create cluster at %d for '%s' allocator",
> + i, p->name);
> + lim = i / 2;
> + for (i--; i >= lim; i--) {
> + p->bitmap[ (i>>5) ] &= ~( 1 << (i & 31) );
> + if (i % p->clustentries == 0 && p->lut[i].vaddr)
> + contigfree(p->lut[i].vaddr,
> + p->_clustsize, M_NETMAP);
> + }
> + p->objtotal = i;
> + p->objfree = p->objtotal - 2;
> + p->_numclusters = i / p->clustentries;
> + p->_memtotal = p->_numclusters * p->_clustsize;
> + break;
> + }
> + for (; i < lim; i++, clust += p->_objsize) {
> + p->bitmap[ (i>>5) ] |= ( 1 << (i & 31) );
> + p->lut[i].vaddr = clust;
> + p->lut[i].paddr = vtophys(clust);
> + }
> + }
> + p->bitmap[0] = ~3; /* objs 0 and 1 is always busy */
> + if (netmap_verbose)
> + D("Pre-allocated %d clusters (%d/%dKB) for '%s'",
> + p->_numclusters, p->_clustsize >> 10,
> + p->_memtotal >> 10, p->name);
> +
> + return 0;
> +
> +clean:
> + netmap_reset_obj_allocator(p);
> + return ENOMEM;
> +}
> +
> +/* call with lock held */
> +static int
> +netmap_memory_config_changed(void)
> +{
> + int i;
> +
> + for (i = 0; i < NETMAP_POOLS_NR; i++) {
> + if (nm_mem.pools[i]._objsize != netmap_params[i].size ||
> + nm_mem.pools[i].objtotal != netmap_params[i].num)
> + return 1;
> + }
> + return 0;
> +}
> +
> +
> +/* call with lock held */
> +static int
> +netmap_memory_config(void)
> +{
> + int i;
> +
> +
> + if (!netmap_memory_config_changed())
> + goto out;
> +
> + D("reconfiguring");
> +
> + if (nm_mem.finalized) {
> + /* reset previous allocation */
> + for (i = 0; i < NETMAP_POOLS_NR; i++) {
> + netmap_reset_obj_allocator(&nm_mem.pools[i]);
> + }
> + nm_mem.finalized = 0;
> + }
> +
> + for (i = 0; i < NETMAP_POOLS_NR; i++) {
> + nm_mem.lasterr = netmap_config_obj_allocator(&nm_mem.pools[i],
> + netmap_params[i].num, netmap_params[i].size);
> + if (nm_mem.lasterr)
> + goto out;
> + }
> +
> + D("Have %d KB for interfaces, %d KB for rings and %d MB for buffers",
> + nm_mem.pools[NETMAP_IF_POOL]._memtotal >> 10,
> + nm_mem.pools[NETMAP_RING_POOL]._memtotal >> 10,
> + nm_mem.pools[NETMAP_BUF_POOL]._memtotal >> 20);
> +
> +out:
> +
> + return nm_mem.lasterr;
> +}
> +
> +/* call with lock held */
> +static int
> +netmap_memory_finalize(void)
> +{
> + int i;
> + u_int totalsize = 0;
> +
> + nm_mem.refcount++;
> + if (nm_mem.refcount > 1) {
> + ND("busy (refcount %d)", nm_mem.refcount);
> + goto out;
> + }
> +
> + /* update configuration if changed */
> + if (netmap_memory_config())
> + goto out;
> +
> + if (nm_mem.finalized) {
> + /* may happen if config is not changed */
> + ND("nothing to do");
> + goto out;
> + }
> +
> + for (i = 0; i < NETMAP_POOLS_NR; i++) {
> + nm_mem.lasterr = netmap_finalize_obj_allocator(&nm_mem.pools[i]);
> + if (nm_mem.lasterr)
> + goto cleanup;
> + totalsize += nm_mem.pools[i]._memtotal;
> + }
> + nm_mem.nm_totalsize = totalsize;
> +
> + /* backward compatibility */
> + netmap_buf_size = nm_mem.pools[NETMAP_BUF_POOL]._objsize;
> + netmap_total_buffers = nm_mem.pools[NETMAP_BUF_POOL].objtotal;
> +
> + netmap_buffer_lut = nm_mem.pools[NETMAP_BUF_POOL].lut;
> + netmap_buffer_base = nm_mem.pools[NETMAP_BUF_POOL].lut[0].vaddr;
> +
> + nm_mem.finalized = 1;
> + nm_mem.lasterr = 0;
> +
> + /* make sysctl values match actual values in the pools */
> + for (i = 0; i < NETMAP_POOLS_NR; i++) {
> + netmap_params[i].size = nm_mem.pools[i]._objsize;
> + netmap_params[i].num = nm_mem.pools[i].objtotal;
> + }
> +
> +out:
> + if (nm_mem.lasterr)
> + nm_mem.refcount--;
> +
> + return nm_mem.lasterr;
> +
> +cleanup:
> + for (i = 0; i < NETMAP_POOLS_NR; i++) {
> + netmap_reset_obj_allocator(&nm_mem.pools[i]);
> + }
> + nm_mem.refcount--;
> +
> + return nm_mem.lasterr;
> +}
> +
> +static int
> +netmap_memory_init(void)
> +{
> + NMA_LOCK_INIT();
> + return (0);
> +}
> +
> +static void
> +netmap_memory_fini(void)
> +{
> + int i;
> +
> + for (i = 0; i < NETMAP_POOLS_NR; i++) {
> + netmap_destroy_obj_allocator(&nm_mem.pools[i]);
> + }
> + NMA_LOCK_DESTROY();
> +}
> +
> +static void
> +netmap_free_rings(struct netmap_adapter *na)
> +{
> + int i;
> + if (!na->tx_rings)
> + return;
> + for (i = 0; i < na->num_tx_rings + 1; i++) {
> + netmap_ring_free(na->tx_rings[i].ring);
> + na->tx_rings[i].ring = NULL;
> + }
> + for (i = 0; i < na->num_rx_rings + 1; i++) {
> + netmap_ring_free(na->rx_rings[i].ring);
> + na->rx_rings[i].ring = NULL;
> + }
> + free(na->tx_rings, M_DEVBUF);
> + na->tx_rings = na->rx_rings = NULL;
> +}
> +
> +
> +
> +/* call with NMA_LOCK held */
> +/*
> + * Allocate the per-fd structure netmap_if.
> + * If this is the first instance, also allocate the krings, rings etc.
> + */
> +static void *
> +netmap_if_new(const char *ifname, struct netmap_adapter *na)
> +{
> + struct netmap_if *nifp;
> + struct netmap_ring *ring;
> + ssize_t base; /* handy for relative offsets between rings and nifp */
> + u_int i, len, ndesc, ntx, nrx;
> + struct netmap_kring *kring;
> +
> + if (netmap_update_config(na)) {
> + /* configuration mismatch, report and fail */
> + return NULL;
> + }
> + ntx = na->num_tx_rings + 1; /* shorthand, include stack ring */
> + nrx = na->num_rx_rings + 1; /* shorthand, include stack ring */
> + /*
> + * the descriptor is followed inline by an array of offsets
> + * to the tx and rx rings in the shared memory region.
> + */
> + len = sizeof(struct netmap_if) + (nrx + ntx) * sizeof(ssize_t);
> + nifp = netmap_if_malloc(len);
> + if (nifp == NULL) {
> + return NULL;
> + }
> +
> + /* initialize base fields -- override const */
> + *(int *)(uintptr_t)&nifp->ni_tx_rings = na->num_tx_rings;
> + *(int *)(uintptr_t)&nifp->ni_rx_rings = na->num_rx_rings;
> + strncpy(nifp->ni_name, ifname, IFNAMSIZ);
> +
> + (na->refcount)++; /* XXX atomic ? we are under lock */
> + if (na->refcount > 1) { /* already setup, we are done */
> + goto final;
> + }
> +
> + len = (ntx + nrx) * sizeof(struct netmap_kring);
> + na->tx_rings = malloc(len, M_DEVBUF, M_NOWAIT | M_ZERO);
> + if (na->tx_rings == NULL) {
> + D("Cannot allocate krings for %s", ifname);
> + goto cleanup;
> + }
> + na->rx_rings = na->tx_rings + ntx;
> +
> + /*
> + * First instance, allocate netmap rings and buffers for this card
> + * The rings are contiguous, but have variable size.
> + */
> + for (i = 0; i < ntx; i++) { /* Transmit rings */
> + kring = &na->tx_rings[i];
> + ndesc = na->num_tx_desc;
> + bzero(kring, sizeof(*kring));
> + len = sizeof(struct netmap_ring) +
> + ndesc * sizeof(struct netmap_slot);
> + ring = netmap_ring_malloc(len);
> + if (ring == NULL) {
> + D("Cannot allocate tx_ring[%d] for %s", i, ifname);
> + goto cleanup;
> + }
> + ND("txring[%d] at %p ofs %d", i, ring);
> + kring->na = na;
> + kring->ring = ring;
> + *(int *)(uintptr_t)&ring->num_slots = kring->nkr_num_slots = ndesc;
> + *(ssize_t *)(uintptr_t)&ring->buf_ofs =
> + (nm_mem.pools[NETMAP_IF_POOL]._memtotal +
> + nm_mem.pools[NETMAP_RING_POOL]._memtotal) -
> + netmap_ring_offset(ring);
> +
> + /*
> + * IMPORTANT:
> + * Always keep one slot empty, so we can detect new
> + * transmissions comparing cur and nr_hwcur (they are
> + * the same only if there are no new transmissions).
> + */
> + ring->avail = kring->nr_hwavail = ndesc - 1;
> + ring->cur = kring->nr_hwcur = 0;
> + *(int *)(uintptr_t)&ring->nr_buf_size = NETMAP_BUF_SIZE;
> + ND("initializing slots for txring[%d]", i);
> + if (netmap_new_bufs(nifp, ring->slot, ndesc)) {
> + D("Cannot allocate buffers for tx_ring[%d] for %s", i, ifname);
> + goto cleanup;
> + }
> + }
> +
> + for (i = 0; i < nrx; i++) { /* Receive rings */
> + kring = &na->rx_rings[i];
> + ndesc = na->num_rx_desc;
> + bzero(kring, sizeof(*kring));
> + len = sizeof(struct netmap_ring) +
> + ndesc * sizeof(struct netmap_slot);
> + ring = netmap_ring_malloc(len);
> + if (ring == NULL) {
> + D("Cannot allocate rx_ring[%d] for %s", i, ifname);
> + goto cleanup;
> + }
> + ND("rxring[%d] at %p ofs %d", i, ring);
> +
> + kring->na = na;
> + kring->ring = ring;
> + *(int *)(uintptr_t)&ring->num_slots = kring->nkr_num_slots = ndesc;
> + *(ssize_t *)(uintptr_t)&ring->buf_ofs =
> + (nm_mem.pools[NETMAP_IF_POOL]._memtotal +
> + nm_mem.pools[NETMAP_RING_POOL]._memtotal) -
> + netmap_ring_offset(ring);
> +
> + ring->cur = kring->nr_hwcur = 0;
> + ring->avail = kring->nr_hwavail = 0; /* empty */
> + *(int *)(uintptr_t)&ring->nr_buf_size = NETMAP_BUF_SIZE;
> + ND("initializing slots for rxring[%d]", i);
> + if (netmap_new_bufs(nifp, ring->slot, ndesc)) {
> + D("Cannot allocate buffers for rx_ring[%d] for %s", i, ifname);
> + goto cleanup;
> + }
> + }
> +#ifdef linux
> + // XXX initialize the selrecord structs.
> + for (i = 0; i < ntx; i++)
> + init_waitqueue_head(&na->tx_rings[i].si);
> + for (i = 0; i < nrx; i++)
> + init_waitqueue_head(&na->rx_rings[i].si);
> + init_waitqueue_head(&na->tx_si);
> + init_waitqueue_head(&na->rx_si);
> +#endif
> +final:
> + /*
> + * fill the slots for the rx and tx rings. They contain the offset
> + * between the ring and nifp, so the information is usable in
> + * userspace to reach the ring from the nifp.
> + */
> + base = netmap_if_offset(nifp);
> + for (i = 0; i < ntx; i++) {
> + *(ssize_t *)(uintptr_t)&nifp->ring_ofs[i] =
> + netmap_ring_offset(na->tx_rings[i].ring) - base;
> + }
> + for (i = 0; i < nrx; i++) {
> + *(ssize_t *)(uintptr_t)&nifp->ring_ofs[i+ntx] =
> + netmap_ring_offset(na->rx_rings[i].ring) - base;
> + }
> + return (nifp);
> +cleanup:
> + netmap_free_rings(na);
> + netmap_if_free(nifp);
> + (na->refcount)--;
> + return NULL;
> +}
> +
> +/* call with NMA_LOCK held */
> +static void
> +netmap_memory_deref(void)
> +{
> + nm_mem.refcount--;
> + if (netmap_verbose)
> + D("refcount = %d", nm_mem.refcount);
> +}
> --- a/include/uapi/Kbuild 2013-02-26 10:19:28.000000000 -0800
> +++ b/include/uapi/Kbuild 2013-03-10 10:08:20.327671428 -0700
> @@ -12,3 +12,4 @@ header-y += video/
> header-y += drm/
> header-y += xen/
> header-y += scsi/
> +header-y += netmap/
> --- /dev/null 1970-01-01 00:00:00.000000000 +0000
> +++ b/include/uapi/netmap/Kbuild 2013-03-10 10:08:20.327671428 -0700
> @@ -0,0 +1,3 @@
> +# UAPI Header export list
> +header-y += netmap.h
> +header-y += netmap_user.h
> --- /dev/null 1970-01-01 00:00:00.000000000 +0000
> +++ b/include/uapi/netmap/netmap.h 2013-03-10 10:08:20.327671428 -0700
> @@ -0,0 +1,289 @@
> +/*
> + * Copyright (C) 2011 Matteo Landi, Luigi Rizzo. All rights reserved.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions are
> + * met:
> + *
> + * 1. Redistributions of source code must retain the above copyright
> + * notice, this list of conditions and the following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above copyright
> + * notice, this list of conditions and the following disclaimer in the
> + * documentation and/or other materials provided with the
> + * distribution.
> + *
> + * 3. Neither the name of the authors nor the names of their contributors
> + * may be used to endorse or promote products derived from this
> + * software without specific prior written permission.
> + *
> + * THIS SOFTWARE IS PROVIDED BY MATTEO LANDI AND CONTRIBUTORS "AS IS" AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
> + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL MATTEO LANDI OR CONTRIBUTORS
> + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
> + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
> + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
> + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
> + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
> + * THE POSSIBILITY OF SUCH DAMAGE.
> + */
> +
> +/*
> + * $FreeBSD: head/sys/net/netmap.h 231198 2012-02-08 11:43:29Z luigi $
> + * $Id: netmap.h 10879 2012-04-12 22:48:59Z luigi $
> + *
> + * Definitions of constants and the structures used by the netmap
> + * framework, for the part visible to both kernel and userspace.
> + * Detailed info on netmap is available with "man netmap" or at
> + *
> + * http://info.iet.unipi.it/~luigi/netmap/
> + */
> +
> +#ifndef _NET_NETMAP_H_
> +#define _NET_NETMAP_H_
> +
> +/*
> + * --- Netmap data structures ---
> + *
> + * The data structures used by netmap are shown below. Those in
> + * capital letters are in an mmapp()ed area shared with userspace,
> + * while others are private to the kernel.
> + * Shared structures do not contain pointers but only memory
> + * offsets, so that addressing is portable between kernel and userspace.
> +
> +
> + softc
> ++----------------+
> +| standard fields|
> +| if_pspare[0] ----------+
> ++----------------+ |
> + |
> ++----------------+<------+
> +|(netmap_adapter)|
> +| | netmap_kring
> +| tx_rings *--------------------------------->+---------------+
> +| | netmap_kring | ring *---------.
> +| rx_rings *--------->+---------------+ | nr_hwcur | |
> ++----------------+ | ring *--------. | nr_hwavail | V
> + | nr_hwcur | | | selinfo | |
> + | nr_hwavail | | +---------------+ .
> + | selinfo | | | ... | .
> + +---------------+ | |(ntx+1 entries)|
> + | .... | | | |
> + |(nrx+1 entries)| | +---------------+
> + | | |
> + KERNEL +---------------+ |
> + |
> + ====================================================================
> + |
> + USERSPACE | NETMAP_RING
> + +---->+-------------+
> + / | cur |
> + NETMAP_IF (nifp, one per file desc.) / | avail |
> + +---------------+ / | buf_ofs |
> + | ni_tx_rings | / +=============+
> + | ni_rx_rings | / | buf_idx | slot[0]
> + | | / | len, flags |
> + | | / +-------------+
> + +===============+ / | buf_idx | slot[1]
> + | txring_ofs[0] | (rel.to nifp)--' | len, flags |
> + | txring_ofs[1] | +-------------+
> + (num_rings+1 entries) (nr_num_slots entries)
> + | txring_ofs[n] | | buf_idx | slot[n-1]
> + +---------------+ | len, flags |
> + | rxring_ofs[0] | +-------------+
> + | rxring_ofs[1] |
> + (num_rings+1 entries)
> + | txring_ofs[n] |
> + +---------------+
> +
> + * The private descriptor ('softc' or 'adapter') of each interface
> + * is extended with a "struct netmap_adapter" containing netmap-related
> + * info (see description in dev/netmap/netmap_kernel.h.
> + * Among other things, tx_rings and rx_rings point to the arrays of
> + * "struct netmap_kring" which in turn reache the various
> + * "struct netmap_ring", shared with userspace.
> +
> + * The NETMAP_RING is the userspace-visible replica of the NIC ring.
> + * Each slot has the index of a buffer, its length and some flags.
> + * In user space, the buffer address is computed as
> + * (char *)ring + buf_ofs + index*NETMAP_BUF_SIZE
> + * In the kernel, buffers do not necessarily need to be contiguous,
> + * and the virtual and physical addresses are derived through
> + * a lookup table.
> + * To associate a different buffer to a slot, applications must
> + * write the new index in buf_idx, and set NS_BUF_CHANGED flag to
> + * make sure that the kernel updates the hardware ring as needed.
> + *
> + * Normally the driver is not requested to report the result of
> + * transmissions (this can dramatically speed up operation).
> + * However the user may request to report completion by setting
> + * NS_REPORT.
> + */
> +struct netmap_slot {
> + uint32_t buf_idx; /* buffer index */
> + uint16_t len; /* packet length, to be copied to/from the hw ring */
> + uint16_t flags; /* buf changed, etc. */
> +#define NS_BUF_CHANGED 0x0001 /* must resync the map, buffer changed */
> +#define NS_REPORT 0x0002 /* ask the hardware to report results
> + * e.g. by generating an interrupt
> + */
> +};
> +
> +/*
> + * Netmap representation of a TX or RX ring (also known as "queue").
> + * This is a queue implemented as a fixed-size circular array.
> + * At the software level, two fields are important: avail and cur.
> + *
> + * In TX rings:
> + * avail indicates the number of slots available for transmission.
> + * It is updated by the kernel after every netmap system call.
> + * It MUST BE decremented by the application when it appends a
> + * packet.
> + * cur indicates the slot to use for the next packet
> + * to send (i.e. the "tail" of the queue).
> + * It MUST BE incremented by the application before
> + * netmap system calls to reflect the number of newly
> + * sent packets.
> + * It is checked by the kernel on netmap system calls
> + * (normally unmodified by the kernel unless invalid).
> + *
> + * The kernel side of netmap uses two additional fields in its own
> + * private ring structure, netmap_kring:
> + * nr_hwcur is a copy of nr_cur on an NIOCTXSYNC.
> + * nr_hwavail is the number of slots known as available by the
> + * hardware. It is updated on an INTR (inc by the
> + * number of packets sent) and on a NIOCTXSYNC
> + * (decrease by nr_cur - nr_hwcur)
> + * A special case, nr_hwavail is -1 if the transmit
> + * side is idle (no pending transmits).
> + *
> + * In RX rings:
> + * avail is the number of packets available (possibly 0).
> + * It MUST BE decremented by the application when it consumes
> + * a packet, and it is updated to nr_hwavail on a NIOCRXSYNC
> + * cur indicates the first slot that contains a packet not
> + * processed yet (the "head" of the queue).
> + * It MUST BE incremented by the software when it consumes
> + * a packet.
> + * reserved indicates the number of buffers before 'cur'
> + * that the application has still in use. Normally 0,
> + * it MUST BE incremented by the application when it
> + * does not return the buffer immediately, and decremented
> + * when the buffer is finally freed.
> + *
> + * The kernel side of netmap uses two additional fields in the kring:
> + * nr_hwcur is a copy of nr_cur on an NIOCRXSYNC
> + * nr_hwavail is the number of packets available. It is updated
> + * on INTR (inc by the number of new packets arrived)
> + * and on NIOCRXSYNC (decreased by nr_cur - nr_hwcur).
> + *
> + * DATA OWNERSHIP/LOCKING:
> + * The netmap_ring is owned by the user program and it is only
> + * accessed or modified in the upper half of the kernel during
> + * a system call.
> + *
> + * The netmap_kring is only modified by the upper half of the kernel.
> + */
> +struct netmap_ring {
> + /*
> + * nr_buf_base_ofs is meant to be used through macros.
> + * It contains the offset of the buffer region from this
> + * descriptor.
> + */
> + const ssize_t buf_ofs;
> + const uint32_t num_slots; /* number of slots in the ring. */
> + uint32_t avail; /* number of usable slots */
> + uint32_t cur; /* 'current' r/w position */
> + uint32_t reserved; /* not refilled before current */
> +
> + const uint16_t nr_buf_size;
> + uint16_t flags;
> +#define NR_TIMESTAMP 0x0002 /* set timestamp on *sync() */
> +
> + struct timeval ts; /* time of last *sync() */
> +
> + /* the slots follow. This struct has variable size */
> + struct netmap_slot slot[0]; /* array of slots. */
> +};
> +
> +
> +/*
> + * Netmap representation of an interface and its queue(s).
> + * There is one netmap_if for each file descriptor on which we want
> + * to select/poll. We assume that on each interface has the same number
> + * of receive and transmit queues.
> + * select/poll operates on one or all pairs depending on the value of
> + * nmr_queueid passed on the ioctl.
> + */
> +struct netmap_if {
> + char ni_name[IFNAMSIZ]; /* name of the interface. */
> + const u_int ni_version; /* API version, currently unused */
> + const u_int ni_rx_rings; /* number of rx rings */
> + const u_int ni_tx_rings; /* if zero, same as ni_rx_rings */
> + /*
> + * The following array contains the offset of each netmap ring
> + * from this structure. The first ni_tx_queues+1 entries refer
> + * to the tx rings, the next ni_rx_queues+1 refer to the rx rings
> + * (the last entry in each block refers to the host stack rings).
> + * The area is filled up by the kernel on NIOCREG,
> + * and then only read by userspace code.
> + */
> + const ssize_t ring_ofs[0];
> +};
> +
> +#ifndef NIOCREGIF
> +/*
> + * ioctl names and related fields
> + *
> + * NIOCGINFO takes a struct ifreq, the interface name is the input,
> + * the outputs are number of queues and number of descriptor
> + * for each queue (useful to set number of threads etc.).
> + *
> + * NIOCREGIF takes an interface name within a struct ifreq,
> + * and activates netmap mode on the interface (if possible).
> + *
> + * NIOCUNREGIF unregisters the interface associated to the fd.
> + *
> + * NIOCTXSYNC, NIOCRXSYNC synchronize tx or rx queues,
> + * whose identity is set in NIOCREGIF through nr_ringid
> + */
> +
> +/*
> + * struct nmreq overlays a struct ifreq
> + */
> +struct nmreq {
> + char nr_name[IFNAMSIZ];
> + uint32_t nr_version; /* API version */
> +#define NETMAP_API 3 /* current version */
> + uint32_t nr_offset; /* nifp offset in the shared region */
> + uint32_t nr_memsize; /* size of the shared region */
> + uint32_t nr_tx_slots; /* slots in tx rings */
> + uint32_t nr_rx_slots; /* slots in rx rings */
> + uint16_t nr_tx_rings; /* number of tx rings */
> + uint16_t nr_rx_rings; /* number of rx rings */
> + uint16_t nr_ringid; /* ring(s) we care about */
> +#define NETMAP_HW_RING 0x4000 /* low bits indicate one hw ring */
> +#define NETMAP_SW_RING 0x2000 /* process the sw ring */
> +#define NETMAP_NO_TX_POLL 0x1000 /* no automatic txsync on poll */
> +#define NETMAP_RING_MASK 0xfff /* the ring number */
> + uint16_t spare1;
> + uint32_t spare2[4];
> +};
> +
> +/*
> + * FreeBSD uses the size value embedded in the _IOWR to determine
> + * how much to copy in/out. So we need it to match the actual
> + * data structure we pass. We put some spares in the structure
> + * to ease compatibility with other versions
> + */
> +#define NIOCGINFO _IOWR('i', 145, struct nmreq) /* return IF info */
> +#define NIOCREGIF _IOWR('i', 146, struct nmreq) /* interface register */
> +#define NIOCUNREGIF _IO('i', 147) /* interface unregister */
> +#define NIOCTXSYNC _IO('i', 148) /* sync tx queues */
> +#define NIOCRXSYNC _IO('i', 149) /* sync rx queues */
> +#endif /* !NIOCREGIF */
> +
> +#endif /* _NET_NETMAP_H_ */
> --- /dev/null 1970-01-01 00:00:00.000000000 +0000
> +++ b/include/uapi/netmap/netmap_user.h 2013-03-10 10:08:20.327671428 -0700
> @@ -0,0 +1,95 @@
> +/*
> + * Copyright (C) 2011 Matteo Landi, Luigi Rizzo. All rights reserved.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions are
> + * met:
> + *
> + * 1. Redistributions of source code must retain the above copyright
> + * notice, this list of conditions and the following disclaimer.
> + *
> + * 2. Redistributions in binary form must reproduce the above copyright
> + * notice, this list of conditions and the following disclaimer in the
> + * documentation and/or other materials provided with the
> + * distribution.
> + *
> + * 3. Neither the name of the authors nor the names of their contributors
> + * may be used to endorse or promote products derived from this
> + * software without specific prior written permission.
> + *
> + * THIS SOFTWARE IS PROVIDED BY MATTEO LANDI AND CONTRIBUTORS "AS IS" AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
> + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL MATTEO LANDI OR CONTRIBUTORS
> + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
> + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
> + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
> + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
> + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
> + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
> + * THE POSSIBILITY OF SUCH DAMAGE.
> + */
> +
> +/*
> + * $FreeBSD: head/sys/net/netmap_user.h 231198 2012-02-08 11:43:29Z luigi $
> + * $Id: netmap_user.h 10879 2012-04-12 22:48:59Z luigi $
> + *
> + * This header contains the macros used to manipulate netmap structures
> + * and packets in userspace. See netmap(4) for more information.
> + *
> + * The address of the struct netmap_if, say nifp, is computed from the
> + * value returned from ioctl(.., NIOCREG, ...) and the mmap region:
> + * ioctl(fd, NIOCREG, &req);
> + * mem = mmap(0, ... );
> + * nifp = NETMAP_IF(mem, req.nr_nifp);
> + * (so simple, we could just do it manually)
> + *
> + * From there:
> + * struct netmap_ring *NETMAP_TXRING(nifp, index)
> + * struct netmap_ring *NETMAP_RXRING(nifp, index)
> + * we can access ring->nr_cur, ring->nr_avail, ring->nr_flags
> + *
> + * ring->slot[i] gives us the i-th slot (we can access
> + * directly plen, flags, bufindex)
> + *
> + * char *buf = NETMAP_BUF(ring, index) returns a pointer to
> + * the i-th buffer
> + *
> + * Since rings are circular, we have macros to compute the next index
> + * i = NETMAP_RING_NEXT(ring, i);
> + */
> +
> +#ifndef _NET_NETMAP_USER_H_
> +#define _NET_NETMAP_USER_H_
> +
> +#define NETMAP_IF(b, o) (struct netmap_if *)((char *)(b) + (o))
> +
> +#define NETMAP_TXRING(nifp, index) \
> + ((struct netmap_ring *)((char *)(nifp) + \
> + (nifp)->ring_ofs[index] ) )
> +
> +#define NETMAP_RXRING(nifp, index) \
> + ((struct netmap_ring *)((char *)(nifp) + \
> + (nifp)->ring_ofs[index + (nifp)->ni_tx_rings + 1] ) )
> +
> +#define NETMAP_BUF(ring, index) \
> + ((char *)(ring) + (ring)->buf_ofs + ((index)*(ring)->nr_buf_size))
> +
> +#define NETMAP_BUF_IDX(ring, buf) \
> + ( ((char *)(buf) - ((char *)(ring) + (ring)->buf_ofs) ) / \
> + (ring)->nr_buf_size )
> +
> +#define NETMAP_RING_NEXT(r, i) \
> + ((i)+1 == (r)->num_slots ? 0 : (i) + 1 )
> +
> +#define NETMAP_RING_FIRST_RESERVED(r) \
> + ( (r)->cur < (r)->reserved ? \
> + (r)->cur + (r)->num_slots - (r)->reserved : \
> + (r)->cur - (r)->reserved )
> +
> +/*
> + * Return 1 if the given tx ring is empty.
> + */
> +#define NETMAP_TX_RING_EMPTY(r) ((r)->avail >= (r)->num_slots - 1)
> +
> +#endif /* _NET_NETMAP_USER_H_ */
> --- /dev/null 1970-01-01 00:00:00.000000000 +0000
> +++ b/include/netmap/bsd_glue.h 2013-03-10 10:08:20.327671428 -0700
> @@ -0,0 +1,263 @@
> +/*
> + * (C) 2012 Luigi Rizzo - Universita` di Pisa
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + * notice, this list of conditions and the following disclaimer.
> + * 2. Redistributions in binary form must reproduce the above copyright
> + * notice, this list of conditions and the following disclaimer in the
> + * documentation and/or other materials provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
> + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
> + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
> + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +/*
> + * glue code to build the netmap bsd code under linux.
> + * Some of these tweaks are generic, some are specific for
> + * character device drivers and network code/device drivers.
> + */
> +
> +#ifndef _BSD_GLUE_H
> +#define _BSD_GLUE_H
> +
> +/* a set of headers used in netmap */
> +#include <linux/version.h>
> +#include <linux/if.h>
> +#include <linux/list.h>
> +#include <linux/mutex.h>
> +#include <linux/types.h>
> +#include <linux/time.h>
> +#include <linux/mm.h>
> +#include <linux/poll.h>
> +#include <linux/netdevice.h>
> +#include <linux/sched.h>
> +#include <linux/wait.h>
> +#include <linux/miscdevice.h>
> +//#include <linux/log2.h> // ilog2
> +#include <linux/etherdevice.h> // eth_type_trans
> +#include <linux/module.h>
> +#include <linux/moduleparam.h>
> +#include <linux/virtio.h> // virt_to_phys
> +
> +#define printf(fmt, arg...) printk(KERN_ERR fmt, ##arg)
> +#define KASSERT(a, b) BUG_ON(!(a))
> +
> +/* Type redefinitions. XXX check them */
> +typedef void * bus_dma_tag_t;
> +typedef void * bus_dmamap_t;
> +typedef int bus_size_t;
> +typedef int bus_dma_segment_t;
> +typedef void * bus_addr_t;
> +#define vm_paddr_t phys_addr_t
> +/* XXX the 'off_t' on Linux corresponds to a 'long' */
> +#define vm_offset_t uint32_t
> +struct thread;
> +
> +/* endianness macros/functions */
> +#define le16toh le16_to_cpu
> +#define le32toh le32_to_cpu
> +#define le64toh le64_to_cpu
> +#define be64toh be64_to_cpu
> +#define htole32 cpu_to_le32
> +#define htole64 cpu_to_le64
> +
> +#include <linux/jiffies.h>
> +#define time_second (jiffies_to_msecs(jiffies) / 1000U )
> +
> +#define bzero(a, len) memset(a, 0, len)
> +#define bcopy(_s, _d, len) memcpy(_d, _s, len)
> +
> +
> +// XXX maybe implement it as a proper function somewhere
> +// it is important to set s->len before the copy.
> +#define m_devget(_buf, _len, _ofs, _dev, _fn) ( { \
> + struct sk_buff *s = netdev_alloc_skb(_dev, _len); \
> + if (s) { \
> + s->len += _len; \
> + skb_copy_to_linear_data_offset(s, _ofs, _buf, _len); \
> + s->protocol = eth_type_trans(s, _dev); \
> + } \
> + s; } )
> +
> +#define mbuf sk_buff
> +#define m_nextpkt next // chain of mbufs
> +#define m_freem(m) dev_kfree_skb_any(m) // free a sk_buff
> +
> +/*
> + * m_copydata() copies from mbuf to buffer following the mbuf chain.
> + * XXX check which linux equivalent we should use to follow fragmented
> + * skbufs.
> + */
> +
> +//#define m_copydata(m, o, l, b) skb_copy_bits(m, o, b, l)
> +#define m_copydata(m, o, l, b) skb_copy_from_linear_data_offset(m, o, b, l)
> +
> +/*
> + * struct ifnet is remapped into struct net_device on linux.
> + * ifnet has an if_softc field pointing to the device-specific struct
> + * (adapter).
> + * On linux the ifnet/net_device is at the beginning of the device-specific
> + * structure, so a pointer to the first field of the ifnet works.
> + * We don't use this in netmap, though.
> + *
> + * if_xname name device name
> + * if_capabilities flags // XXX not used
> + * if_capenable priv_flags
> + * we would use "features" but it is all taken.
> + * XXX check for conflict in flags use.
> + *
> + * if_bridge atalk_ptr struct nm_bridge (only for VALE ports)
> + *
> + * In netmap we use if_pspare[0] to point to the netmap_adapter,
> + * in linux we have no spares so we overload ax25_ptr, and the detection
> + * for netmap-capable is some magic in the area pointed by that.
> + */
> +#define WNA(_ifp) (_ifp)->ax25_ptr
> +
> +#define ifnet net_device /* remap */
> +#define if_xname name /* field ifnet-> net_device */
> +//#define if_capabilities flags /* IFCAP_NETMAP */
> +#define if_capenable priv_flags /* IFCAP_NETMAP */
> +#define if_bridge atalk_ptr /* remap, only for VALE ports */
> +#define ifunit_ref(_x) dev_get_by_name(&init_net, _x);
> +#define if_rele(ifp) dev_put(ifp)
> +#define CURVNET_SET(x)
> +#define CURVNET_RESTORE(x)
> +
> +
> +/*
> + * We use spin_lock_irqsave() because we use the lock in the
> + * (hard) interrupt context.
> + */
> +typedef struct {
> + spinlock_t sl;
> + ulong flags;
> +} safe_spinlock_t;
> +
> +static inline void mtx_lock(safe_spinlock_t *m)
> +{
> + spin_lock_irqsave(&(m->sl), m->flags);
> +}
> +
> +static inline void mtx_unlock(safe_spinlock_t *m)
> +{
> + ulong flags = ACCESS_ONCE(m->flags);
> + spin_unlock_irqrestore(&(m->sl), flags);
> +}
> +
> +#define mtx_init(a, b, c, d) spin_lock_init(&((a)->sl))
> +#define mtx_destroy(a) // XXX spin_lock_destroy(a)
> +
> +/* use volatile to fix a probable compiler error on 2.6.25 */
> +#define malloc(_size, type, flags) \
> + ({ volatile int _v = _size; kmalloc(_v, GFP_ATOMIC | __GFP_ZERO); })
> +
> +#define free(a, t) kfree(a)
> +
> +// XXX do we need GPF_ZERO ?
> +// XXX do we need GFP_DMA for slots ?
> +// http://www.mjmwired.net/kernel/Documentation/DMA-API.txt
> +
> +#define contigmalloc(sz, ty, flags, a, b, pgsz, c) \
> + (char *) __get_free_pages(GFP_ATOMIC | __GFP_ZERO, \
> + ilog2(roundup_pow_of_two((sz)/PAGE_SIZE)))
> +#define contigfree(va, sz, ty) free_pages((unsigned long)va, \
> + ilog2(roundup_pow_of_two(sz)/PAGE_SIZE))
> +
> +#define vtophys virt_to_phys
> +
> +/*--- selrecord and friends ---*/
> +/* wake_up() or wake_up_interruptible() ? */
> +#define selwakeuppri(sw, pri) wake_up(sw)
> +#define selrecord(x, y) poll_wait((struct file *)x, y, pwait)
> +#define knlist_destroy(x) // XXX todo
> +
> +/* we use tsleep/wakeup to sleep a bit. */
> +#define tsleep(a, b, c, t) msleep(10) // XXX
> +#define wakeup(sw) // XXX double check
> +#define microtime do_gettimeofday
> +
> +
> +/*
> + * The following trick is to map a struct cdev into a struct miscdevice
> + */
> +#define cdev miscdevice
> +
> +
> +/*
> + * XXX to complete - the dmamap interface
> + */
> +#define BUS_DMA_NOWAIT 0
> +#define bus_dmamap_load(_1, _2, _3, _4, _5, _6, _7)
> +#define bus_dmamap_unload(_1, _2)
> +
> +typedef int (d_mmap_t)(struct file *f, struct vm_area_struct *vma);
> +typedef unsigned int (d_poll_t)(struct file * file, struct poll_table_struct *pwait);
> +
> +/*
> + * make_dev will set an error and return the first argument.
> + * This relies on the availability of the 'error' local variable.
> + */
> +#define make_dev(_cdev, _zero, _uid, _gid, _perm, _name) \
> + ({error = misc_register(_cdev); \
> + D("run mknod /dev/%s c %d %d # error %d", \
> + (_cdev)->name, MISC_MAJOR, (_cdev)->minor, error); \
> + _cdev; } )
> +#define destroy_dev(_cdev) misc_deregister(_cdev)
> +
> +/*--- sysctl API ----*/
> +/*
> + * linux: sysctl are mapped into /sys/module/ipfw_mod parameters
> + * windows: they are emulated via get/setsockopt
> + */
> +#define CTLFLAG_RD 1
> +#define CTLFLAG_RW 2
> +
> +struct sysctl_oid;
> +struct sysctl_req;
> +
> +
> +#define SYSCTL_DECL(_1)
> +#define SYSCTL_OID(_1, _2, _3, _4, _5, _6, _7, _8)
> +#define SYSCTL_NODE(_1, _2, _3, _4, _5, _6)
> +#define _SYSCTL_BASE(_name, _var, _ty, _perm) \
> + module_param_named(_name, *(_var), _ty, \
> + ( (_perm) == CTLFLAG_RD) ? 0444: 0644 )
> +
> +#define SYSCTL_PROC(_base, _oid, _name, _mode, _var, _val, _desc, _a, _b)
> +
> +#define SYSCTL_INT(_base, _oid, _name, _mode, _var, _val, _desc) \
> + _SYSCTL_BASE(_name, _var, int, _mode)
> +
> +#define SYSCTL_LONG(_base, _oid, _name, _mode, _var, _val, _desc) \
> + _SYSCTL_BASE(_name, _var, long, _mode)
> +
> +#define SYSCTL_ULONG(_base, _oid, _name, _mode, _var, _val, _desc) \
> + _SYSCTL_BASE(_name, _var, ulong, _mode)
> +
> +#define SYSCTL_UINT(_base, _oid, _name, _mode, _var, _val, _desc) \
> + _SYSCTL_BASE(_name, _var, uint, _mode)
> +
> +#define TUNABLE_INT(_name, _ptr)
> +
> +#define SYSCTL_VNET_PROC SYSCTL_PROC
> +#define SYSCTL_VNET_INT SYSCTL_INT
> +
> +#define SYSCTL_HANDLER_ARGS \
> + struct sysctl_oid *oidp, void *arg1, int arg2, struct sysctl_req *req
> +int sysctl_handle_int(SYSCTL_HANDLER_ARGS);
> +int sysctl_handle_long(SYSCTL_HANDLER_ARGS);
> +
> +#endif /* _BSD_GLUE_H */
> --- /dev/null 1970-01-01 00:00:00.000000000 +0000
> +++ b/include/netmap/netmap_kern.h 2013-03-10 11:30:37.253528570 -0700
> @@ -0,0 +1,474 @@
> +/*
> + * Copyright (C) 2011-2012 Matteo Landi, Luigi Rizzo. All rights reserved.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + * notice, this list of conditions and the following disclaimer.
> + * 2. Redistributions in binary form must reproduce the above copyright
> + * notice, this list of conditions and the following disclaimer in the
> + * documentation and/or other materials provided with the distribution.
> + *
> + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
> + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
> + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
> + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
> + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
> + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
> + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
> + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
> + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
> + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + */
> +
> +/*
> + * $FreeBSD: head/sys/dev/netmap/netmap_kern.h 238985 2012-08-02 11:59:43Z luigi $
> + * $Id: netmap_kern.h 12089 2013-02-15 01:20:25Z luigi $
> + *
> + * The header contains the definitions of constants and function
> + * prototypes used only in kernelspace.
> + */
> +
> +#ifndef _NET_NETMAP_KERN_H_
> +#define _NET_NETMAP_KERN_H_
> +
> +#define NETMAP_MEM2 // use the new memory allocator
> +
> +#if defined(__FreeBSD__)
> +#define likely(x) __builtin_expect(!!(x), 1)
> +#define unlikely(x) __builtin_expect(!!(x), 0)
> +
> +#define NM_LOCK_T struct mtx
> +#define NM_SELINFO_T struct selinfo
> +#define MBUF_LEN(m) ((m)->m_pkthdr.len)
> +#define NM_SEND_UP(ifp, m) ((ifp)->if_input)(ifp, m)
> +#elif defined (linux)
> +
> +#define NM_LOCK_T safe_spinlock_t // see bsd_glue.h
> +#define NM_SELINFO_T wait_queue_head_t
> +#define MBUF_LEN(m) ((m)->len)
> +#define NM_SEND_UP(ifp, m) netif_rx(m)
> +
> +#ifndef DEV_NETMAP
> +#define DEV_NETMAP
> +#endif
> +
> +/*
> + * IFF_NETMAP goes into net_device's priv_flags (if_capenable).
> + * Map it to BSD style cap until this driver is cleaned up.
> + */
> +#define IFCAP_NETMAP IFF_NETMAP
> +
> +
> +#elif defined (__APPLE__)
> +#warning apple support is incomplete.
> +#define likely(x) __builtin_expect(!!(x), 1)
> +#define unlikely(x) __builtin_expect(!!(x), 0)
> +#define NM_LOCK_T IOLock *
> +#define NM_SELINFO_T struct selinfo
> +#define MBUF_LEN(m) ((m)->m_pkthdr.len)
> +#define NM_SEND_UP(ifp, m) ((ifp)->if_input)(ifp, m)
> +
> +#else
> +#error unsupported platform
> +#endif
> +
> +#define ND(format, ...)
> +#define D(format, ...) \
> + do { \
> + struct timeval __xxts; \
> + microtime(&__xxts); \
> + printf("%03d.%06d %s [%d] " format "\n", \
> + (int)__xxts.tv_sec % 1000, (int)__xxts.tv_usec, \
> + __FUNCTION__, __LINE__, ##__VA_ARGS__); \
> + } while (0)
> +
> +/* rate limited, lps indicates how many per second */
> +#define RD(lps, format, ...) \
> + do { \
> + static int t0, __cnt; \
> + if (t0 != time_second) { \
> + t0 = time_second; \
> + __cnt = 0; \
> + } \
> + if (__cnt++ < lps) \
> + D(format, ##__VA_ARGS__); \
> + } while (0)
> +
> +struct netmap_adapter;
> +
> +/*
> + * private, kernel view of a ring. Keeps track of the status of
> + * a ring across system calls.
> + *
> + * nr_hwcur index of the next buffer to refill.
> + * It corresponds to ring->cur - ring->reserved
> + *
> + * nr_hwavail the number of slots "owned" by userspace.
> + * nr_hwavail =:= ring->avail + ring->reserved
> + *
> + * The indexes in the NIC and netmap rings are offset by nkr_hwofs slots.
> + * This is so that, on a reset, buffers owned by userspace are not
> + * modified by the kernel. In particular:
> + * RX rings: the next empty buffer (hwcur + hwavail + hwofs) coincides with
> + * the next empty buffer as known by the hardware (next_to_check or so).
> + * TX rings: hwcur + hwofs coincides with next_to_send
> + *
> + * For received packets, slot->flags is set to nkr_slot_flags
> + * so we can provide a proper initial value (e.g. set NS_FORWARD
> + * when operating in 'transparent' mode).
> + */
> +struct netmap_kring {
> + struct netmap_ring *ring;
> + u_int nr_hwcur;
> + int nr_hwavail;
> + u_int nr_kflags; /* private driver flags */
> +#define NKR_PENDINTR 0x1 // Pending interrupt.
> + u_int nkr_num_slots;
> +
> + uint16_t nkr_slot_flags; /* initial value for flags */
> + int nkr_hwofs; /* offset between NIC and netmap ring */
> + struct netmap_adapter *na;
> + NM_SELINFO_T si; /* poll/select wait queue */
> + NM_LOCK_T q_lock; /* used if no device lock available */
> +} __attribute__((__aligned__(64)));
> +
> +/*
> + * This struct extends the 'struct adapter' (or
> + * equivalent) device descriptor. It contains all fields needed to
> + * support netmap operation.
> + */
> +struct netmap_adapter {
> + /*
> + * On linux we do not have a good way to tell if an interface
> + * is netmap-capable. So we use the following trick:
> + * NA(ifp) points here, and the first entry (which hopefully
> + * always exists and is at least 32 bits) contains a magic
> + * value which we can use to detect that the interface is good.
> + */
> + uint32_t magic;
> + uint32_t na_flags; /* future place for IFCAP_NETMAP */
> +#define NAF_SKIP_INTR 1 /* use the regular interrupt handler.
> + * useful during initialization
> + */
> + int refcount; /* number of user-space descriptors using this
> + interface, which is equal to the number of
> + struct netmap_if objs in the mapped region. */
> + /*
> + * The selwakeup in the interrupt thread can use per-ring
> + * and/or global wait queues. We track how many clients
> + * of each type we have so we can optimize the drivers,
> + * and especially avoid huge contention on the locks.
> + */
> + int na_single; /* threads attached to a single hw queue */
> + int na_multi; /* threads attached to multiple hw queues */
> +
> + int separate_locks; /* set if the interface suports different
> + locks for rx, tx and core. */
> +
> + u_int num_rx_rings; /* number of adapter receive rings */
> + u_int num_tx_rings; /* number of adapter transmit rings */
> +
> + u_int num_tx_desc; /* number of descriptor in each queue */
> + u_int num_rx_desc;
> +
> + /* tx_rings and rx_rings are private but allocated
> + * as a contiguous chunk of memory. Each array has
> + * N+1 entries, for the adapter queues and for the host queue.
> + */
> + struct netmap_kring *tx_rings; /* array of TX rings. */
> + struct netmap_kring *rx_rings; /* array of RX rings. */
> +
> + NM_SELINFO_T tx_si, rx_si; /* global wait queues */
> +
> + /* copy of if_qflush and if_transmit pointers, to intercept
> + * packets from the network stack when netmap is active.
> + */
> + int (*if_transmit)(struct ifnet *, struct mbuf *);
> +
> + /* references to the ifnet and device routines, used by
> + * the generic netmap functions.
> + */
> + struct ifnet *ifp; /* adapter is ifp->if_softc */
> +
> + NM_LOCK_T core_lock; /* used if no device lock available */
> +
> + int (*nm_register)(struct ifnet *, int onoff);
> + void (*nm_lock)(struct ifnet *, int what, u_int ringid);
> + int (*nm_txsync)(struct ifnet *, u_int ring, int lock);
> + int (*nm_rxsync)(struct ifnet *, u_int ring, int lock);
> + /* return configuration information */
> + int (*nm_config)(struct ifnet *, u_int *txr, u_int *txd,
> + u_int *rxr, u_int *rxd);
> +
> + int bdg_port;
> +#ifdef linux
> + struct net_device_ops nm_ndo;
> + int if_refcount; // XXX additions for bridge
> +#endif /* linux */
> +};
> +
> +/*
> + * The combination of "enable" (ifp->if_capenable & IFCAP_NETMAP)
> + * and refcount gives the status of the interface, namely:
> + *
> + * enable refcount Status
> + *
> + * FALSE 0 normal operation
> + * FALSE != 0 -- (impossible)
> + * TRUE 1 netmap mode
> + * TRUE 0 being deleted.
> + */
> +
> +#define NETMAP_DELETING(_na) ( ((_na)->refcount == 0) && \
> + ( (_na)->ifp->if_capenable & IFCAP_NETMAP) )
> +
> +/*
> + * parameters for (*nm_lock)(adapter, what, index)
> + */
> +enum {
> + NETMAP_NO_LOCK = 0,
> + NETMAP_CORE_LOCK, NETMAP_CORE_UNLOCK,
> + NETMAP_TX_LOCK, NETMAP_TX_UNLOCK,
> + NETMAP_RX_LOCK, NETMAP_RX_UNLOCK,
> +#ifdef __FreeBSD__
> +#define NETMAP_REG_LOCK NETMAP_CORE_LOCK
> +#define NETMAP_REG_UNLOCK NETMAP_CORE_UNLOCK
> +#else
> + NETMAP_REG_LOCK, NETMAP_REG_UNLOCK
> +#endif
> +};
> +
> +/*
> + * The following are support routines used by individual drivers to
> + * support netmap operation.
> + *
> + * netmap_attach() initializes a struct netmap_adapter, allocating the
> + * struct netmap_ring's and the struct selinfo.
> + *
> + * netmap_detach() frees the memory allocated by netmap_attach().
> + *
> + * netmap_start() replaces the if_transmit routine of the interface,
> + * and is used to intercept packets coming from the stack.
> + *
> + * netmap_load_map/netmap_reload_map are helper routines to set/reset
> + * the dmamap for a packet buffer
> + *
> + * netmap_reset() is a helper routine to be called in the driver
> + * when reinitializing a ring.
> + */
> +int netmap_attach(struct netmap_adapter *, int);
> +void netmap_detach(struct ifnet *);
> +int netmap_start(struct ifnet *, struct mbuf *);
> +enum txrx { NR_RX = 0, NR_TX = 1 };
> +struct netmap_slot *netmap_reset(struct netmap_adapter *na,
> + enum txrx tx, int n, u_int new_cur);
> +int netmap_ring_reinit(struct netmap_kring *);
> +
> +extern u_int netmap_buf_size;
> +#define NETMAP_BUF_SIZE netmap_buf_size
> +extern int netmap_mitigate;
> +extern int netmap_no_pendintr;
> +extern u_int netmap_total_buffers;
> +extern char *netmap_buffer_base;
> +extern int netmap_verbose; // XXX debugging
> +enum { /* verbose flags */
> + NM_VERB_ON = 1, /* generic verbose */
> + NM_VERB_HOST = 0x2, /* verbose host stack */
> + NM_VERB_RXSYNC = 0x10, /* verbose on rxsync/txsync */
> + NM_VERB_TXSYNC = 0x20,
> + NM_VERB_RXINTR = 0x100, /* verbose on rx/tx intr (driver) */
> + NM_VERB_TXINTR = 0x200,
> + NM_VERB_NIC_RXSYNC = 0x1000, /* verbose on rx/tx intr (driver) */
> + NM_VERB_NIC_TXSYNC = 0x2000,
> +};
> +
> +/*
> + * NA returns a pointer to the struct netmap adapter from the ifp,
> + * WNA is used to write it.
> + */
> +#ifndef WNA
> +#define WNA(_ifp) (_ifp)->if_pspare[0]
> +#endif
> +#define NA(_ifp) ((struct netmap_adapter *)WNA(_ifp))
> +
> +/*
> + * Macros to determine if an interface is netmap capable or netmap enabled.
> + * See the magic field in struct netmap_adapter.
> + */
> +#ifdef __FreeBSD__
> +/*
> + * on FreeBSD just use if_capabilities and if_capenable.
> + */
> +#define NETMAP_CAPABLE(ifp) (NA(ifp) && \
> + (ifp)->if_capabilities & IFCAP_NETMAP )
> +
> +#define NETMAP_SET_CAPABLE(ifp) \
> + (ifp)->if_capabilities |= IFCAP_NETMAP
> +
> +#else /* linux */
> +
> +/*
> + * on linux:
> + * we check if NA(ifp) is set and its first element has a related
> + * magic value. The capenable is within the struct netmap_adapter.
> + */
> +#define NETMAP_MAGIC 0x52697a7a
> +
> +#define NETMAP_CAPABLE(ifp) (NA(ifp) && \
> + ((uint32_t)(uintptr_t)NA(ifp) ^ NA(ifp)->magic) == NETMAP_MAGIC )
> +
> +#define NETMAP_SET_CAPABLE(ifp) \
> + NA(ifp)->magic = ((uint32_t)(uintptr_t)NA(ifp)) ^ NETMAP_MAGIC
> +
> +#endif /* linux */
> +
> +#ifdef __FreeBSD__
> +/* Callback invoked by the dma machinery after a successfull dmamap_load */
> +static void netmap_dmamap_cb(__unused void *arg,
> + __unused bus_dma_segment_t * segs, __unused int nseg, __unused int error)
> +{
> +}
> +
> +/* bus_dmamap_load wrapper: call aforementioned function if map != NULL.
> + * XXX can we do it without a callback ?
> + */
> +static inline void
> +netmap_load_map(bus_dma_tag_t tag, bus_dmamap_t map, void *buf)
> +{
> + if (map)
> + bus_dmamap_load(tag, map, buf, NETMAP_BUF_SIZE,
> + netmap_dmamap_cb, NULL, BUS_DMA_NOWAIT);
> +}
> +
> +/* update the map when a buffer changes. */
> +static inline void
> +netmap_reload_map(bus_dma_tag_t tag, bus_dmamap_t map, void *buf)
> +{
> + if (map) {
> + bus_dmamap_unload(tag, map);
> + bus_dmamap_load(tag, map, buf, NETMAP_BUF_SIZE,
> + netmap_dmamap_cb, NULL, BUS_DMA_NOWAIT);
> + }
> +}
> +#else /* linux */
> +
> +/*
> + * XXX How do we redefine these functions:
> + *
> + * on linux we need
> + * dma_map_single(&pdev->dev, virt_addr, len, direction)
> + * dma_unmap_single(&adapter->pdev->dev, phys_addr, len, direction
> + * The len can be implicit (on netmap it is NETMAP_BUF_SIZE)
> + * unfortunately the direction is not, so we need to change
> + * something to have a cross API
> + */
> +#define netmap_load_map(_t, _m, _b)
> +#define netmap_reload_map(_t, _m, _b)
> +#if 0
> + struct e1000_buffer *buffer_info = &tx_ring->buffer_info[l];
> + /* set time_stamp *before* dma to help avoid a possible race */
> + buffer_info->time_stamp = jiffies;
> + buffer_info->mapped_as_page = false;
> + buffer_info->length = len;
> + //buffer_info->next_to_watch = l;
> + /* reload dma map */
> + dma_unmap_single(&adapter->pdev->dev, buffer_info->dma,
> + NETMAP_BUF_SIZE, DMA_TO_DEVICE);
> + buffer_info->dma = dma_map_single(&adapter->pdev->dev,
> + addr, NETMAP_BUF_SIZE, DMA_TO_DEVICE);
> +
> + if (dma_mapping_error(&adapter->pdev->dev, buffer_info->dma)) {
> + D("dma mapping error");
> + /* goto dma_error; See e1000_put_txbuf() */
> + /* XXX reset */
> + }
> + tx_desc->buffer_addr = htole64(buffer_info->dma); //XXX
> +
> +#endif
> +
> +/*
> + * The bus_dmamap_sync() can be one of wmb() or rmb() depending on direction.
> + */
> +#define bus_dmamap_sync(_a, _b, _c)
> +
> +#endif /* linux */
> +
> +/*
> + * functions to map NIC to KRING indexes (n2k) and vice versa (k2n)
> + */
> +static inline int
> +netmap_idx_n2k(struct netmap_kring *kr, int idx)
> +{
> + int n = kr->nkr_num_slots;
> + idx += kr->nkr_hwofs;
> + if (idx < 0)
> + return idx + n;
> + else if (idx < n)
> + return idx;
> + else
> + return idx - n;
> +}
> +
> +
> +static inline int
> +netmap_idx_k2n(struct netmap_kring *kr, int idx)
> +{
> + int n = kr->nkr_num_slots;
> + idx -= kr->nkr_hwofs;
> + if (idx < 0)
> + return idx + n;
> + else if (idx < n)
> + return idx;
> + else
> + return idx - n;
> +}
> +
> +
> +#ifdef NETMAP_MEM2
> +/* Entries of the look-up table. */
> +struct lut_entry {
> + void *vaddr; /* virtual address. */
> + vm_paddr_t paddr; /* phisical address. */
> +};
> +
> +struct netmap_obj_pool;
> +extern struct lut_entry *netmap_buffer_lut;
> +#define NMB_VA(i) (netmap_buffer_lut[i].vaddr)
> +#define NMB_PA(i) (netmap_buffer_lut[i].paddr)
> +#else /* NETMAP_MEM1 */
> +#define NMB_VA(i) (netmap_buffer_base + (i * NETMAP_BUF_SIZE) )
> +#endif /* NETMAP_MEM2 */
> +
> +/*
> + * NMB return the virtual address of a buffer (buffer 0 on bad index)
> + * PNMB also fills the physical address
> + */
> +static inline void *
> +NMB(struct netmap_slot *slot)
> +{
> + uint32_t i = slot->buf_idx;
> + return (unlikely(i >= netmap_total_buffers)) ? NMB_VA(0) : NMB_VA(i);
> +}
> +
> +static inline void *
> +PNMB(struct netmap_slot *slot, uint64_t *pp)
> +{
> + uint32_t i = slot->buf_idx;
> + void *ret = (i >= netmap_total_buffers) ? NMB_VA(0) : NMB_VA(i);
> +#ifdef NETMAP_MEM2
> + *pp = (i >= netmap_total_buffers) ? NMB_PA(0) : NMB_PA(i);
> +#else
> + *pp = vtophys(ret);
> +#endif
> + return ret;
> +}
> +
> +/* default functions to handle rx/tx interrupts */
> +int netmap_rx_irq(struct ifnet *, int, int *);
> +#define netmap_tx_irq(_n, _q) netmap_rx_irq(_n, _q, NULL)
> +
> +extern int netmap_copy;
> +#endif /* _NET_NETMAP_KERN_H_ */
> --- a/include/uapi/linux/if.h 2013-02-16 09:10:41.000000000 -0800
> +++ b/include/uapi/linux/if.h 2013-03-10 11:26:44.500548075 -0700
> @@ -83,6 +83,7 @@
> #define IFF_SUPP_NOFCS 0x80000 /* device supports sending custom FCS */
> #define IFF_LIVE_ADDR_CHANGE 0x100000 /* device supports hardware address
> * change when it's running */
> +#define IFF_NETMAP 0x200000 /* device used with netmap */
>
>
> #define IF_GET_IFACE 0x0001 /* for querying only */
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
SAKURA Internet Inc. / Senior Researcher
Naoto MATSUMOTO <n-matsumoto@...ura.ad.jp>
SAKURA Research Center <http://research.sakura.ad.jp/>
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists