132c0f0beSMauro Carvalho Chehab.. SPDX-License-Identifier: GPL-2.0 232c0f0beSMauro Carvalho Chehab.. include:: <isonum.txt> 3*6fb4825eSEdward Cree.. _switchdev: 432c0f0beSMauro Carvalho Chehab 532c0f0beSMauro Carvalho Chehab=============================================== 632c0f0beSMauro Carvalho ChehabEthernet switch device driver model (switchdev) 732c0f0beSMauro Carvalho Chehab=============================================== 832c0f0beSMauro Carvalho Chehab 932c0f0beSMauro Carvalho ChehabCopyright |copy| 2014 Jiri Pirko <jiri@resnulli.us> 1032c0f0beSMauro Carvalho Chehab 1132c0f0beSMauro Carvalho ChehabCopyright |copy| 2014-2015 Scott Feldman <sfeldma@gmail.com> 1232c0f0beSMauro Carvalho Chehab 1332c0f0beSMauro Carvalho Chehab 1432c0f0beSMauro Carvalho ChehabThe Ethernet switch device driver model (switchdev) is an in-kernel driver 1532c0f0beSMauro Carvalho Chehabmodel for switch devices which offload the forwarding (data) plane from the 1632c0f0beSMauro Carvalho Chehabkernel. 1732c0f0beSMauro Carvalho Chehab 1832c0f0beSMauro Carvalho ChehabFigure 1 is a block diagram showing the components of the switchdev model for 1932c0f0beSMauro Carvalho Chehaban example setup using a data-center-class switch ASIC chip. Other setups 2032c0f0beSMauro Carvalho Chehabwith SR-IOV or soft switches, such as OVS, are possible. 2132c0f0beSMauro Carvalho Chehab 2232c0f0beSMauro Carvalho Chehab:: 2332c0f0beSMauro Carvalho Chehab 2432c0f0beSMauro Carvalho Chehab 2532c0f0beSMauro Carvalho Chehab User-space tools 2632c0f0beSMauro Carvalho Chehab 2732c0f0beSMauro Carvalho Chehab user space | 2832c0f0beSMauro Carvalho Chehab +-------------------------------------------------------------------+ 2932c0f0beSMauro Carvalho Chehab kernel | Netlink 3032c0f0beSMauro Carvalho Chehab | 3132c0f0beSMauro Carvalho Chehab +--------------+-------------------------------+ 3232c0f0beSMauro Carvalho Chehab | Network stack | 3332c0f0beSMauro Carvalho Chehab | (Linux) | 3432c0f0beSMauro Carvalho Chehab | | 3532c0f0beSMauro Carvalho Chehab +----------------------------------------------+ 3632c0f0beSMauro Carvalho Chehab 3732c0f0beSMauro Carvalho Chehab sw1p2 sw1p4 sw1p6 3832c0f0beSMauro Carvalho Chehab sw1p1 + sw1p3 + sw1p5 + eth1 3932c0f0beSMauro Carvalho Chehab + | + | + | + 4032c0f0beSMauro Carvalho Chehab | | | | | | | 4132c0f0beSMauro Carvalho Chehab +--+----+----+----+----+----+---+ +-----+-----+ 4232c0f0beSMauro Carvalho Chehab | Switch driver | | mgmt | 4332c0f0beSMauro Carvalho Chehab | (this document) | | driver | 4432c0f0beSMauro Carvalho Chehab | | | | 4532c0f0beSMauro Carvalho Chehab +--------------+----------------+ +-----------+ 4632c0f0beSMauro Carvalho Chehab | 4732c0f0beSMauro Carvalho Chehab kernel | HW bus (eg PCI) 4832c0f0beSMauro Carvalho Chehab +-------------------------------------------------------------------+ 4932c0f0beSMauro Carvalho Chehab hardware | 5032c0f0beSMauro Carvalho Chehab +--------------+----------------+ 5132c0f0beSMauro Carvalho Chehab | Switch device (sw1) | 5232c0f0beSMauro Carvalho Chehab | +----+ +--------+ 5332c0f0beSMauro Carvalho Chehab | | v offloaded data path | mgmt port 5432c0f0beSMauro Carvalho Chehab | | | | 5532c0f0beSMauro Carvalho Chehab +--|----|----+----+----+----+---+ 5632c0f0beSMauro Carvalho Chehab | | | | | | 5732c0f0beSMauro Carvalho Chehab + + + + + + 5832c0f0beSMauro Carvalho Chehab p1 p2 p3 p4 p5 p6 5932c0f0beSMauro Carvalho Chehab 6032c0f0beSMauro Carvalho Chehab front-panel ports 6132c0f0beSMauro Carvalho Chehab 6232c0f0beSMauro Carvalho Chehab 6332c0f0beSMauro Carvalho Chehab Fig 1. 6432c0f0beSMauro Carvalho Chehab 6532c0f0beSMauro Carvalho Chehab 6632c0f0beSMauro Carvalho ChehabInclude Files 6732c0f0beSMauro Carvalho Chehab------------- 6832c0f0beSMauro Carvalho Chehab 6932c0f0beSMauro Carvalho Chehab:: 7032c0f0beSMauro Carvalho Chehab 7132c0f0beSMauro Carvalho Chehab #include <linux/netdevice.h> 7232c0f0beSMauro Carvalho Chehab #include <net/switchdev.h> 7332c0f0beSMauro Carvalho Chehab 7432c0f0beSMauro Carvalho Chehab 7532c0f0beSMauro Carvalho ChehabConfiguration 7632c0f0beSMauro Carvalho Chehab------------- 7732c0f0beSMauro Carvalho Chehab 7832c0f0beSMauro Carvalho ChehabUse "depends NET_SWITCHDEV" in driver's Kconfig to ensure switchdev model 7932c0f0beSMauro Carvalho Chehabsupport is built for driver. 8032c0f0beSMauro Carvalho Chehab 8132c0f0beSMauro Carvalho Chehab 8232c0f0beSMauro Carvalho ChehabSwitch Ports 8332c0f0beSMauro Carvalho Chehab------------ 8432c0f0beSMauro Carvalho Chehab 8532c0f0beSMauro Carvalho ChehabOn switchdev driver initialization, the driver will allocate and register a 8632c0f0beSMauro Carvalho Chehabstruct net_device (using register_netdev()) for each enumerated physical switch 8732c0f0beSMauro Carvalho Chehabport, called the port netdev. A port netdev is the software representation of 8832c0f0beSMauro Carvalho Chehabthe physical port and provides a conduit for control traffic to/from the 8932c0f0beSMauro Carvalho Chehabcontroller (the kernel) and the network, as well as an anchor point for higher 9032c0f0beSMauro Carvalho Chehablevel constructs such as bridges, bonds, VLANs, tunnels, and L3 routers. Using 9132c0f0beSMauro Carvalho Chehabstandard netdev tools (iproute2, ethtool, etc), the port netdev can also 9232c0f0beSMauro Carvalho Chehabprovide to the user access to the physical properties of the switch port such 9332c0f0beSMauro Carvalho Chehabas PHY link state and I/O statistics. 9432c0f0beSMauro Carvalho Chehab 9532c0f0beSMauro Carvalho ChehabThere is (currently) no higher-level kernel object for the switch beyond the 9632c0f0beSMauro Carvalho Chehabport netdevs. All of the switchdev driver ops are netdev ops or switchdev ops. 9732c0f0beSMauro Carvalho Chehab 9832c0f0beSMauro Carvalho ChehabA switch management port is outside the scope of the switchdev driver model. 9932c0f0beSMauro Carvalho ChehabTypically, the management port is not participating in offloaded data plane and 10032c0f0beSMauro Carvalho Chehabis loaded with a different driver, such as a NIC driver, on the management port 10132c0f0beSMauro Carvalho Chehabdevice. 10232c0f0beSMauro Carvalho Chehab 10332c0f0beSMauro Carvalho ChehabSwitch ID 10432c0f0beSMauro Carvalho Chehab^^^^^^^^^ 10532c0f0beSMauro Carvalho Chehab 10632c0f0beSMauro Carvalho ChehabThe switchdev driver must implement the net_device operation 10732c0f0beSMauro Carvalho Chehabndo_get_port_parent_id for each port netdev, returning the same physical ID for 10832c0f0beSMauro Carvalho Chehabeach port of a switch. The ID must be unique between switches on the same 10932c0f0beSMauro Carvalho Chehabsystem. The ID does not need to be unique between switches on different 11032c0f0beSMauro Carvalho Chehabsystems. 11132c0f0beSMauro Carvalho Chehab 11232c0f0beSMauro Carvalho ChehabThe switch ID is used to locate ports on a switch and to know if aggregated 11332c0f0beSMauro Carvalho Chehabports belong to the same switch. 11432c0f0beSMauro Carvalho Chehab 11532c0f0beSMauro Carvalho ChehabPort Netdev Naming 11632c0f0beSMauro Carvalho Chehab^^^^^^^^^^^^^^^^^^ 11732c0f0beSMauro Carvalho Chehab 11832c0f0beSMauro Carvalho ChehabUdev rules should be used for port netdev naming, using some unique attribute 11932c0f0beSMauro Carvalho Chehabof the port as a key, for example the port MAC address or the port PHYS name. 12032c0f0beSMauro Carvalho ChehabHard-coding of kernel netdev names within the driver is discouraged; let the 12132c0f0beSMauro Carvalho Chehabkernel pick the default netdev name, and let udev set the final name based on a 12232c0f0beSMauro Carvalho Chehabport attribute. 12332c0f0beSMauro Carvalho Chehab 12432c0f0beSMauro Carvalho ChehabUsing port PHYS name (ndo_get_phys_port_name) for the key is particularly 12532c0f0beSMauro Carvalho Chehabuseful for dynamically-named ports where the device names its ports based on 12632c0f0beSMauro Carvalho Chehabexternal configuration. For example, if a physical 40G port is split logically 12732c0f0beSMauro Carvalho Chehabinto 4 10G ports, resulting in 4 port netdevs, the device can give a unique 12832c0f0beSMauro Carvalho Chehabname for each port using port PHYS name. The udev rule would be:: 12932c0f0beSMauro Carvalho Chehab 13032c0f0beSMauro Carvalho Chehab SUBSYSTEM=="net", ACTION=="add", ATTR{phys_switch_id}=="<phys_switch_id>", \ 13132c0f0beSMauro Carvalho Chehab ATTR{phys_port_name}!="", NAME="swX$attr{phys_port_name}" 13232c0f0beSMauro Carvalho Chehab 13332c0f0beSMauro Carvalho ChehabSuggested naming convention is "swXpYsZ", where X is the switch name or ID, Y 13432c0f0beSMauro Carvalho Chehabis the port name or ID, and Z is the sub-port name or ID. For example, sw1p1s0 13532c0f0beSMauro Carvalho Chehabwould be sub-port 0 on port 1 on switch 1. 13632c0f0beSMauro Carvalho Chehab 13732c0f0beSMauro Carvalho ChehabPort Features 13832c0f0beSMauro Carvalho Chehab^^^^^^^^^^^^^ 13932c0f0beSMauro Carvalho Chehab 14032c0f0beSMauro Carvalho ChehabNETIF_F_NETNS_LOCAL 14132c0f0beSMauro Carvalho Chehab 14232c0f0beSMauro Carvalho ChehabIf the switchdev driver (and device) only supports offloading of the default 14332c0f0beSMauro Carvalho Chehabnetwork namespace (netns), the driver should set this feature flag to prevent 14432c0f0beSMauro Carvalho Chehabthe port netdev from being moved out of the default netns. A netns-aware 14532c0f0beSMauro Carvalho Chehabdriver/device would not set this flag and be responsible for partitioning 14632c0f0beSMauro Carvalho Chehabhardware to preserve netns containment. This means hardware cannot forward 14732c0f0beSMauro Carvalho Chehabtraffic from a port in one namespace to another port in another namespace. 14832c0f0beSMauro Carvalho Chehab 14932c0f0beSMauro Carvalho ChehabPort Topology 15032c0f0beSMauro Carvalho Chehab^^^^^^^^^^^^^ 15132c0f0beSMauro Carvalho Chehab 15232c0f0beSMauro Carvalho ChehabThe port netdevs representing the physical switch ports can be organized into 15332c0f0beSMauro Carvalho Chehabhigher-level switching constructs. The default construct is a standalone 15432c0f0beSMauro Carvalho Chehabrouter port, used to offload L3 forwarding. Two or more ports can be bonded 15532c0f0beSMauro Carvalho Chehabtogether to form a LAG. Two or more ports (or LAGs) can be bridged to bridge 15632c0f0beSMauro Carvalho ChehabL2 networks. VLANs can be applied to sub-divide L2 networks. L2-over-L3 15732c0f0beSMauro Carvalho Chehabtunnels can be built on ports. These constructs are built using standard Linux 15832c0f0beSMauro Carvalho Chehabtools such as the bridge driver, the bonding/team drivers, and netlink-based 15932c0f0beSMauro Carvalho Chehabtools such as iproute2. 16032c0f0beSMauro Carvalho Chehab 16132c0f0beSMauro Carvalho ChehabThe switchdev driver can know a particular port's position in the topology by 16232c0f0beSMauro Carvalho Chehabmonitoring NETDEV_CHANGEUPPER notifications. For example, a port moved into a 163404a5ad7SRandy Dunlapbond will see its upper master change. If that bond is moved into a bridge, 16432c0f0beSMauro Carvalho Chehabthe bond's upper master will change. And so on. The driver will track such 16532c0f0beSMauro Carvalho Chehabmovements to know what position a port is in in the overall topology by 16632c0f0beSMauro Carvalho Chehabregistering for netdevice events and acting on NETDEV_CHANGEUPPER. 16732c0f0beSMauro Carvalho Chehab 16832c0f0beSMauro Carvalho ChehabL2 Forwarding Offload 16932c0f0beSMauro Carvalho Chehab--------------------- 17032c0f0beSMauro Carvalho Chehab 17132c0f0beSMauro Carvalho ChehabThe idea is to offload the L2 data forwarding (switching) path from the kernel 17232c0f0beSMauro Carvalho Chehabto the switchdev device by mirroring bridge FDB entries down to the device. An 17332c0f0beSMauro Carvalho ChehabFDB entry is the {port, MAC, VLAN} tuple forwarding destination. 17432c0f0beSMauro Carvalho Chehab 17532c0f0beSMauro Carvalho ChehabTo offloading L2 bridging, the switchdev driver/device should support: 17632c0f0beSMauro Carvalho Chehab 17732c0f0beSMauro Carvalho Chehab - Static FDB entries installed on a bridge port 17832c0f0beSMauro Carvalho Chehab - Notification of learned/forgotten src mac/vlans from device 17932c0f0beSMauro Carvalho Chehab - STP state changes on the port 18032c0f0beSMauro Carvalho Chehab - VLAN flooding of multicast/broadcast and unknown unicast packets 18132c0f0beSMauro Carvalho Chehab 18232c0f0beSMauro Carvalho ChehabStatic FDB Entries 18332c0f0beSMauro Carvalho Chehab^^^^^^^^^^^^^^^^^^ 18432c0f0beSMauro Carvalho Chehab 185787a4109SVladimir OlteanA driver which implements the ``ndo_fdb_add``, ``ndo_fdb_del`` and 186787a4109SVladimir Oltean``ndo_fdb_dump`` operations is able to support the command below, which adds a 187787a4109SVladimir Olteanstatic bridge FDB entry:: 18832c0f0beSMauro Carvalho Chehab 189787a4109SVladimir Oltean bridge fdb add dev DEV ADDRESS [vlan VID] [self] static 19032c0f0beSMauro Carvalho Chehab 191787a4109SVladimir Oltean(the "static" keyword is non-optional: if not specified, the entry defaults to 192787a4109SVladimir Olteanbeing "local", which means that it should not be forwarded) 19332c0f0beSMauro Carvalho Chehab 194787a4109SVladimir OlteanThe "self" keyword (optional because it is implicit) has the role of 195787a4109SVladimir Olteaninstructing the kernel to fulfill the operation through the ``ndo_fdb_add`` 196787a4109SVladimir Olteanimplementation of the ``DEV`` device itself. If ``DEV`` is a bridge port, this 197787a4109SVladimir Olteanwill bypass the bridge and therefore leave the software database out of sync 198787a4109SVladimir Olteanwith the hardware one. 199787a4109SVladimir Oltean 200787a4109SVladimir OlteanTo avoid this, the "master" keyword can be used:: 201787a4109SVladimir Oltean 202787a4109SVladimir Oltean bridge fdb add dev DEV ADDRESS [vlan VID] master static 203787a4109SVladimir Oltean 204787a4109SVladimir OlteanThe above command instructs the kernel to search for a master interface of 205787a4109SVladimir Oltean``DEV`` and fulfill the operation through the ``ndo_fdb_add`` method of that. 206787a4109SVladimir OlteanThis time, the bridge generates a ``SWITCHDEV_FDB_ADD_TO_DEVICE`` notification 207787a4109SVladimir Olteanwhich the port driver can handle and use it to program its hardware table. This 208787a4109SVladimir Olteanway, the software and the hardware database will both contain this static FDB 209787a4109SVladimir Olteanentry. 210787a4109SVladimir Oltean 211787a4109SVladimir OlteanNote: for new switchdev drivers that offload the Linux bridge, implementing the 212787a4109SVladimir Oltean``ndo_fdb_add`` and ``ndo_fdb_del`` bridge bypass methods is strongly 213787a4109SVladimir Olteandiscouraged: all static FDB entries should be added on a bridge port using the 214787a4109SVladimir Oltean"master" flag. The ``ndo_fdb_dump`` is an exception and can be implemented to 215787a4109SVladimir Olteanvisualize the hardware tables, if the device does not have an interrupt for 216787a4109SVladimir Olteannotifying the operating system of newly learned/forgotten dynamic FDB 217787a4109SVladimir Olteanaddresses. In that case, the hardware FDB might end up having entries that the 218787a4109SVladimir Olteansoftware FDB does not, and implementing ``ndo_fdb_dump`` is the only way to see 219787a4109SVladimir Olteanthem. 22032c0f0beSMauro Carvalho Chehab 22132c0f0beSMauro Carvalho ChehabNote: by default, the bridge does not filter on VLAN and only bridges untagged 22232c0f0beSMauro Carvalho Chehabtraffic. To enable VLAN support, turn on VLAN filtering:: 22332c0f0beSMauro Carvalho Chehab 22432c0f0beSMauro Carvalho Chehab echo 1 >/sys/class/net/<bridge>/bridge/vlan_filtering 22532c0f0beSMauro Carvalho Chehab 22632c0f0beSMauro Carvalho ChehabNotification of Learned/Forgotten Source MAC/VLANs 22732c0f0beSMauro Carvalho Chehab^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 22832c0f0beSMauro Carvalho Chehab 22932c0f0beSMauro Carvalho ChehabThe switch device will learn/forget source MAC address/VLAN on ingress packets 23032c0f0beSMauro Carvalho Chehaband notify the switch driver of the mac/vlan/port tuples. The switch driver, 23132c0f0beSMauro Carvalho Chehabin turn, will notify the bridge driver using the switchdev notifier call:: 23232c0f0beSMauro Carvalho Chehab 23332c0f0beSMauro Carvalho Chehab err = call_switchdev_notifiers(val, dev, info, extack); 23432c0f0beSMauro Carvalho Chehab 23532c0f0beSMauro Carvalho ChehabWhere val is SWITCHDEV_FDB_ADD when learning and SWITCHDEV_FDB_DEL when 23632c0f0beSMauro Carvalho Chehabforgetting, and info points to a struct switchdev_notifier_fdb_info. On 23732c0f0beSMauro Carvalho ChehabSWITCHDEV_FDB_ADD, the bridge driver will install the FDB entry into the 23832c0f0beSMauro Carvalho Chehabbridge's FDB and mark the entry as NTF_EXT_LEARNED. The iproute2 bridge 23932c0f0beSMauro Carvalho Chehabcommand will label these entries "offload":: 24032c0f0beSMauro Carvalho Chehab 24132c0f0beSMauro Carvalho Chehab $ bridge fdb 24232c0f0beSMauro Carvalho Chehab 52:54:00:12:35:01 dev sw1p1 master br0 permanent 24332c0f0beSMauro Carvalho Chehab 00:02:00:00:02:00 dev sw1p1 master br0 offload 24432c0f0beSMauro Carvalho Chehab 00:02:00:00:02:00 dev sw1p1 self 24532c0f0beSMauro Carvalho Chehab 52:54:00:12:35:02 dev sw1p2 master br0 permanent 24632c0f0beSMauro Carvalho Chehab 00:02:00:00:03:00 dev sw1p2 master br0 offload 24732c0f0beSMauro Carvalho Chehab 00:02:00:00:03:00 dev sw1p2 self 24832c0f0beSMauro Carvalho Chehab 33:33:00:00:00:01 dev eth0 self permanent 24932c0f0beSMauro Carvalho Chehab 01:00:5e:00:00:01 dev eth0 self permanent 25032c0f0beSMauro Carvalho Chehab 33:33:ff:00:00:00 dev eth0 self permanent 25132c0f0beSMauro Carvalho Chehab 01:80:c2:00:00:0e dev eth0 self permanent 25232c0f0beSMauro Carvalho Chehab 33:33:00:00:00:01 dev br0 self permanent 25332c0f0beSMauro Carvalho Chehab 01:00:5e:00:00:01 dev br0 self permanent 25432c0f0beSMauro Carvalho Chehab 33:33:ff:12:35:01 dev br0 self permanent 25532c0f0beSMauro Carvalho Chehab 25632c0f0beSMauro Carvalho ChehabLearning on the port should be disabled on the bridge using the bridge command:: 25732c0f0beSMauro Carvalho Chehab 25832c0f0beSMauro Carvalho Chehab bridge link set dev DEV learning off 25932c0f0beSMauro Carvalho Chehab 26032c0f0beSMauro Carvalho ChehabLearning on the device port should be enabled, as well as learning_sync:: 26132c0f0beSMauro Carvalho Chehab 26232c0f0beSMauro Carvalho Chehab bridge link set dev DEV learning on self 26332c0f0beSMauro Carvalho Chehab bridge link set dev DEV learning_sync on self 26432c0f0beSMauro Carvalho Chehab 26532c0f0beSMauro Carvalho ChehabLearning_sync attribute enables syncing of the learned/forgotten FDB entry to 26632c0f0beSMauro Carvalho Chehabthe bridge's FDB. It's possible, but not optimal, to enable learning on the 26732c0f0beSMauro Carvalho Chehabdevice port and on the bridge port, and disable learning_sync. 26832c0f0beSMauro Carvalho Chehab 26932c0f0beSMauro Carvalho ChehabTo support learning, the driver implements switchdev op 27032c0f0beSMauro Carvalho Chehabswitchdev_port_attr_set for SWITCHDEV_ATTR_PORT_ID_{PRE}_BRIDGE_FLAGS. 27132c0f0beSMauro Carvalho Chehab 27232c0f0beSMauro Carvalho ChehabFDB Ageing 27332c0f0beSMauro Carvalho Chehab^^^^^^^^^^ 27432c0f0beSMauro Carvalho Chehab 27532c0f0beSMauro Carvalho ChehabThe bridge will skip ageing FDB entries marked with NTF_EXT_LEARNED and it is 27632c0f0beSMauro Carvalho Chehabthe responsibility of the port driver/device to age out these entries. If the 27732c0f0beSMauro Carvalho Chehabport device supports ageing, when the FDB entry expires, it will notify the 27832c0f0beSMauro Carvalho Chehabdriver which in turn will notify the bridge with SWITCHDEV_FDB_DEL. If the 27932c0f0beSMauro Carvalho Chehabdevice does not support ageing, the driver can simulate ageing using a 28032c0f0beSMauro Carvalho Chehabgarbage collection timer to monitor FDB entries. Expired entries will be 28132c0f0beSMauro Carvalho Chehabnotified to the bridge using SWITCHDEV_FDB_DEL. See rocker driver for 28232c0f0beSMauro Carvalho Chehabexample of driver running ageing timer. 28332c0f0beSMauro Carvalho Chehab 28432c0f0beSMauro Carvalho ChehabTo keep an NTF_EXT_LEARNED entry "alive", the driver should refresh the FDB 28532c0f0beSMauro Carvalho Chehabentry by calling call_switchdev_notifiers(SWITCHDEV_FDB_ADD, ...). The 28632c0f0beSMauro Carvalho Chehabnotification will reset the FDB entry's last-used time to now. The driver 28732c0f0beSMauro Carvalho Chehabshould rate limit refresh notifications, for example, no more than once a 28832c0f0beSMauro Carvalho Chehabsecond. (The last-used time is visible using the bridge -s fdb option). 28932c0f0beSMauro Carvalho Chehab 29032c0f0beSMauro Carvalho ChehabSTP State Change on Port 29132c0f0beSMauro Carvalho Chehab^^^^^^^^^^^^^^^^^^^^^^^^ 29232c0f0beSMauro Carvalho Chehab 29332c0f0beSMauro Carvalho ChehabInternally or with a third-party STP protocol implementation (e.g. mstpd), the 29432c0f0beSMauro Carvalho Chehabbridge driver maintains the STP state for ports, and will notify the switch 29532c0f0beSMauro Carvalho Chehabdriver of STP state change on a port using the switchdev op 29632c0f0beSMauro Carvalho Chehabswitchdev_attr_port_set for SWITCHDEV_ATTR_PORT_ID_STP_UPDATE. 29732c0f0beSMauro Carvalho Chehab 29832c0f0beSMauro Carvalho ChehabState is one of BR_STATE_*. The switch driver can use STP state updates to 29932c0f0beSMauro Carvalho Chehabupdate ingress packet filter list for the port. For example, if port is 30032c0f0beSMauro Carvalho ChehabDISABLED, no packets should pass, but if port moves to BLOCKED, then STP BPDUs 30132c0f0beSMauro Carvalho Chehaband other IEEE 01:80:c2:xx:xx:xx link-local multicast packets can pass. 30232c0f0beSMauro Carvalho Chehab 30332c0f0beSMauro Carvalho ChehabNote that STP BDPUs are untagged and STP state applies to all VLANs on the port 30432c0f0beSMauro Carvalho Chehabso packet filters should be applied consistently across untagged and tagged 30532c0f0beSMauro Carvalho ChehabVLANs on the port. 30632c0f0beSMauro Carvalho Chehab 30732c0f0beSMauro Carvalho ChehabFlooding L2 domain 30832c0f0beSMauro Carvalho Chehab^^^^^^^^^^^^^^^^^^ 30932c0f0beSMauro Carvalho Chehab 31032c0f0beSMauro Carvalho ChehabFor a given L2 VLAN domain, the switch device should flood multicast/broadcast 31132c0f0beSMauro Carvalho Chehaband unknown unicast packets to all ports in domain, if allowed by port's 31232c0f0beSMauro Carvalho Chehabcurrent STP state. The switch driver, knowing which ports are within which 31332c0f0beSMauro Carvalho Chehabvlan L2 domain, can program the switch device for flooding. The packet may 31432c0f0beSMauro Carvalho Chehabbe sent to the port netdev for processing by the bridge driver. The 31532c0f0beSMauro Carvalho Chehabbridge should not reflood the packet to the same ports the device flooded, 31632c0f0beSMauro Carvalho Chehabotherwise there will be duplicate packets on the wire. 31732c0f0beSMauro Carvalho Chehab 31832c0f0beSMauro Carvalho ChehabTo avoid duplicate packets, the switch driver should mark a packet as already 31932c0f0beSMauro Carvalho Chehabforwarded by setting the skb->offload_fwd_mark bit. The bridge driver will mark 32032c0f0beSMauro Carvalho Chehabthe skb using the ingress bridge port's mark and prevent it from being forwarded 32132c0f0beSMauro Carvalho Chehabthrough any bridge port with the same mark. 32232c0f0beSMauro Carvalho Chehab 32332c0f0beSMauro Carvalho ChehabIt is possible for the switch device to not handle flooding and push the 32432c0f0beSMauro Carvalho Chehabpackets up to the bridge driver for flooding. This is not ideal as the number 32532c0f0beSMauro Carvalho Chehabof ports scale in the L2 domain as the device is much more efficient at 32632c0f0beSMauro Carvalho Chehabflooding packets that software. 32732c0f0beSMauro Carvalho Chehab 32832c0f0beSMauro Carvalho ChehabIf supported by the device, flood control can be offloaded to it, preventing 32932c0f0beSMauro Carvalho Chehabcertain netdevs from flooding unicast traffic for which there is no FDB entry. 33032c0f0beSMauro Carvalho Chehab 33132c0f0beSMauro Carvalho ChehabIGMP Snooping 33232c0f0beSMauro Carvalho Chehab^^^^^^^^^^^^^ 33332c0f0beSMauro Carvalho Chehab 33432c0f0beSMauro Carvalho ChehabIn order to support IGMP snooping, the port netdevs should trap to the bridge 33532c0f0beSMauro Carvalho Chehabdriver all IGMP join and leave messages. 33632c0f0beSMauro Carvalho ChehabThe bridge multicast module will notify port netdevs on every multicast group 33732c0f0beSMauro Carvalho Chehabchanged whether it is static configured or dynamically joined/leave. 33832c0f0beSMauro Carvalho ChehabThe hardware implementation should be forwarding all registered multicast 33932c0f0beSMauro Carvalho Chehabtraffic groups only to the configured ports. 34032c0f0beSMauro Carvalho Chehab 34132c0f0beSMauro Carvalho ChehabL3 Routing Offload 34232c0f0beSMauro Carvalho Chehab------------------ 34332c0f0beSMauro Carvalho Chehab 34432c0f0beSMauro Carvalho ChehabOffloading L3 routing requires that device be programmed with FIB entries from 34532c0f0beSMauro Carvalho Chehabthe kernel, with the device doing the FIB lookup and forwarding. The device 34632c0f0beSMauro Carvalho Chehabdoes a longest prefix match (LPM) on FIB entries matching route prefix and 34732c0f0beSMauro Carvalho Chehabforwards the packet to the matching FIB entry's nexthop(s) egress ports. 34832c0f0beSMauro Carvalho Chehab 34932c0f0beSMauro Carvalho ChehabTo program the device, the driver has to register a FIB notifier handler 35032c0f0beSMauro Carvalho Chehabusing register_fib_notifier. The following events are available: 35132c0f0beSMauro Carvalho Chehab 35232c0f0beSMauro Carvalho Chehab=================== =================================================== 35332c0f0beSMauro Carvalho ChehabFIB_EVENT_ENTRY_ADD used for both adding a new FIB entry to the device, 35432c0f0beSMauro Carvalho Chehab or modifying an existing entry on the device. 35532c0f0beSMauro Carvalho ChehabFIB_EVENT_ENTRY_DEL used for removing a FIB entry 35632c0f0beSMauro Carvalho ChehabFIB_EVENT_RULE_ADD, 35732c0f0beSMauro Carvalho ChehabFIB_EVENT_RULE_DEL used to propagate FIB rule changes 35832c0f0beSMauro Carvalho Chehab=================== =================================================== 35932c0f0beSMauro Carvalho Chehab 36032c0f0beSMauro Carvalho ChehabFIB_EVENT_ENTRY_ADD and FIB_EVENT_ENTRY_DEL events pass:: 36132c0f0beSMauro Carvalho Chehab 36232c0f0beSMauro Carvalho Chehab struct fib_entry_notifier_info { 36332c0f0beSMauro Carvalho Chehab struct fib_notifier_info info; /* must be first */ 36432c0f0beSMauro Carvalho Chehab u32 dst; 36532c0f0beSMauro Carvalho Chehab int dst_len; 36632c0f0beSMauro Carvalho Chehab struct fib_info *fi; 36732c0f0beSMauro Carvalho Chehab u8 tos; 36832c0f0beSMauro Carvalho Chehab u8 type; 36932c0f0beSMauro Carvalho Chehab u32 tb_id; 37032c0f0beSMauro Carvalho Chehab u32 nlflags; 37132c0f0beSMauro Carvalho Chehab }; 37232c0f0beSMauro Carvalho Chehab 37332c0f0beSMauro Carvalho Chehabto add/modify/delete IPv4 dst/dest_len prefix on table tb_id. The ``*fi`` 37432c0f0beSMauro Carvalho Chehabstructure holds details on the route and route's nexthops. ``*dev`` is one 37532c0f0beSMauro Carvalho Chehabof the port netdevs mentioned in the route's next hop list. 37632c0f0beSMauro Carvalho Chehab 37732c0f0beSMauro Carvalho ChehabRoutes offloaded to the device are labeled with "offload" in the ip route 37832c0f0beSMauro Carvalho Chehablisting:: 37932c0f0beSMauro Carvalho Chehab 38032c0f0beSMauro Carvalho Chehab $ ip route show 38132c0f0beSMauro Carvalho Chehab default via 192.168.0.2 dev eth0 38232c0f0beSMauro Carvalho Chehab 11.0.0.0/30 dev sw1p1 proto kernel scope link src 11.0.0.2 offload 38332c0f0beSMauro Carvalho Chehab 11.0.0.4/30 via 11.0.0.1 dev sw1p1 proto zebra metric 20 offload 38432c0f0beSMauro Carvalho Chehab 11.0.0.8/30 dev sw1p2 proto kernel scope link src 11.0.0.10 offload 38532c0f0beSMauro Carvalho Chehab 11.0.0.12/30 via 11.0.0.9 dev sw1p2 proto zebra metric 20 offload 38632c0f0beSMauro Carvalho Chehab 12.0.0.2 proto zebra metric 30 offload 38732c0f0beSMauro Carvalho Chehab nexthop via 11.0.0.1 dev sw1p1 weight 1 38832c0f0beSMauro Carvalho Chehab nexthop via 11.0.0.9 dev sw1p2 weight 1 38932c0f0beSMauro Carvalho Chehab 12.0.0.3 via 11.0.0.1 dev sw1p1 proto zebra metric 20 offload 39032c0f0beSMauro Carvalho Chehab 12.0.0.4 via 11.0.0.9 dev sw1p2 proto zebra metric 20 offload 39132c0f0beSMauro Carvalho Chehab 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.15 39232c0f0beSMauro Carvalho Chehab 39332c0f0beSMauro Carvalho ChehabThe "offload" flag is set in case at least one device offloads the FIB entry. 39432c0f0beSMauro Carvalho Chehab 39532c0f0beSMauro Carvalho ChehabXXX: add/mod/del IPv6 FIB API 39632c0f0beSMauro Carvalho Chehab 39732c0f0beSMauro Carvalho ChehabNexthop Resolution 39832c0f0beSMauro Carvalho Chehab^^^^^^^^^^^^^^^^^^ 39932c0f0beSMauro Carvalho Chehab 40032c0f0beSMauro Carvalho ChehabThe FIB entry's nexthop list contains the nexthop tuple (gateway, dev), but for 40132c0f0beSMauro Carvalho Chehabthe switch device to forward the packet with the correct dst mac address, the 40232c0f0beSMauro Carvalho Chehabnexthop gateways must be resolved to the neighbor's mac address. Neighbor mac 40332c0f0beSMauro Carvalho Chehabaddress discovery comes via the ARP (or ND) process and is available via the 40432c0f0beSMauro Carvalho Chehabarp_tbl neighbor table. To resolve the routes nexthop gateways, the driver 40532c0f0beSMauro Carvalho Chehabshould trigger the kernel's neighbor resolution process. See the rocker 40632c0f0beSMauro Carvalho Chehabdriver's rocker_port_ipv4_resolve() for an example. 40732c0f0beSMauro Carvalho Chehab 40832c0f0beSMauro Carvalho ChehabThe driver can monitor for updates to arp_tbl using the netevent notifier 40932c0f0beSMauro Carvalho ChehabNETEVENT_NEIGH_UPDATE. The device can be programmed with resolved nexthops 41032c0f0beSMauro Carvalho Chehabfor the routes as arp_tbl updates. The driver implements ndo_neigh_destroy 41132c0f0beSMauro Carvalho Chehabto know when arp_tbl neighbor entries are purged from the port. 4120f22ad45SFlorian Fainelli 4130f22ad45SFlorian FainelliDevice driver expected behavior 4140f22ad45SFlorian Fainelli------------------------------- 4150f22ad45SFlorian Fainelli 4160f22ad45SFlorian FainelliBelow is a set of defined behavior that switchdev enabled network devices must 4170f22ad45SFlorian Fainelliadhere to. 4180f22ad45SFlorian Fainelli 4190f22ad45SFlorian FainelliConfiguration-less state 4200f22ad45SFlorian Fainelli^^^^^^^^^^^^^^^^^^^^^^^^ 4210f22ad45SFlorian Fainelli 4220f22ad45SFlorian FainelliUpon driver bring up, the network devices must be fully operational, and the 4230f22ad45SFlorian Fainellibacking driver must configure the network device such that it is possible to 4240f22ad45SFlorian Fainellisend and receive traffic to this network device and it is properly separated 4250f22ad45SFlorian Fainellifrom other network devices/ports (e.g.: as is frequent with a switch ASIC). How 4260f22ad45SFlorian Fainellithis is achieved is heavily hardware dependent, but a simple solution can be to 4270f22ad45SFlorian Fainelliuse per-port VLAN identifiers unless a better mechanism is available 4280f22ad45SFlorian Fainelli(proprietary metadata for each network port for instance). 4290f22ad45SFlorian Fainelli 4300f22ad45SFlorian FainelliThe network device must be capable of running a full IP protocol stack 4310f22ad45SFlorian Fainelliincluding multicast, DHCP, IPv4/6, etc. If necessary, it should program the 4320f22ad45SFlorian Fainelliappropriate filters for VLAN, multicast, unicast etc. The underlying device 4330f22ad45SFlorian Fainellidriver must effectively be configured in a similar fashion to what it would do 4340f22ad45SFlorian Fainelliwhen IGMP snooping is enabled for IP multicast over these switchdev network 4350f22ad45SFlorian Fainellidevices and unsolicited multicast must be filtered as early as possible in 4360f22ad45SFlorian Fainellithe hardware. 4370f22ad45SFlorian Fainelli 4380f22ad45SFlorian FainelliWhen configuring VLANs on top of the network device, all VLANs must be working, 4390f22ad45SFlorian Fainelliirrespective of the state of other network devices (e.g.: other ports being part 4400f22ad45SFlorian Fainelliof a VLAN-aware bridge doing ingress VID checking). See below for details. 4410f22ad45SFlorian Fainelli 4420f22ad45SFlorian FainelliIf the device implements e.g.: VLAN filtering, putting the interface in 4430f22ad45SFlorian Fainellipromiscuous mode should allow the reception of all VLAN tags (including those 4440f22ad45SFlorian Fainellinot present in the filter(s)). 4450f22ad45SFlorian Fainelli 4460f22ad45SFlorian FainelliBridged switch ports 4470f22ad45SFlorian Fainelli^^^^^^^^^^^^^^^^^^^^ 4480f22ad45SFlorian Fainelli 4490f22ad45SFlorian FainelliWhen a switchdev enabled network device is added as a bridge member, it should 4500f22ad45SFlorian Fainellinot disrupt any functionality of non-bridged network devices and they 4510f22ad45SFlorian Fainellishould continue to behave as normal network devices. Depending on the bridge 4520f22ad45SFlorian Fainelliconfiguration knobs below, the expected behavior is documented. 4530f22ad45SFlorian Fainelli 4540f22ad45SFlorian FainelliBridge VLAN filtering 4550f22ad45SFlorian Fainelli^^^^^^^^^^^^^^^^^^^^^ 4560f22ad45SFlorian Fainelli 4570f22ad45SFlorian FainelliThe Linux bridge allows the configuration of a VLAN filtering mode (statically, 4580f22ad45SFlorian Fainelliat device creation time, and dynamically, during run time) which must be 4590f22ad45SFlorian Fainelliobserved by the underlying switchdev network device/hardware: 4600f22ad45SFlorian Fainelli 4610f22ad45SFlorian Fainelli- with VLAN filtering turned off: the bridge is strictly VLAN unaware and its 4620f22ad45SFlorian Fainelli data path will process all Ethernet frames as if they are VLAN-untagged. 4630f22ad45SFlorian Fainelli The bridge VLAN database can still be modified, but the modifications should 4640f22ad45SFlorian Fainelli have no effect while VLAN filtering is turned off. Frames ingressing the 4650f22ad45SFlorian Fainelli device with a VID that is not programmed into the bridge/switch's VLAN table 4660f22ad45SFlorian Fainelli must be forwarded and may be processed using a VLAN device (see below). 4670f22ad45SFlorian Fainelli 4680f22ad45SFlorian Fainelli- with VLAN filtering turned on: the bridge is VLAN-aware and frames ingressing 4690f22ad45SFlorian Fainelli the device with a VID that is not programmed into the bridges/switch's VLAN 4700f22ad45SFlorian Fainelli table must be dropped (strict VID checking). 4710f22ad45SFlorian Fainelli 4720f22ad45SFlorian FainelliWhen there is a VLAN device (e.g: sw0p1.100) configured on top of a switchdev 4730f22ad45SFlorian Fainellinetwork device which is a bridge port member, the behavior of the software 4740f22ad45SFlorian Fainellinetwork stack must be preserved, or the configuration must be refused if that 4750f22ad45SFlorian Fainelliis not possible. 4760f22ad45SFlorian Fainelli 4770f22ad45SFlorian Fainelli- with VLAN filtering turned off, the bridge will process all ingress traffic 4780f22ad45SFlorian Fainelli for the port, except for the traffic tagged with a VLAN ID destined for a 4790f22ad45SFlorian Fainelli VLAN upper. The VLAN upper interface (which consumes the VLAN tag) can even 4800f22ad45SFlorian Fainelli be added to a second bridge, which includes other switch ports or software 4810f22ad45SFlorian Fainelli interfaces. Some approaches to ensure that the forwarding domain for traffic 4820f22ad45SFlorian Fainelli belonging to the VLAN upper interfaces are managed properly: 483cfeb961aSVladimir Oltean 4840f22ad45SFlorian Fainelli * If forwarding destinations can be managed per VLAN, the hardware could be 4850f22ad45SFlorian Fainelli configured to map all traffic, except the packets tagged with a VID 4860f22ad45SFlorian Fainelli belonging to a VLAN upper interface, to an internal VID corresponding to 4870f22ad45SFlorian Fainelli untagged packets. This internal VID spans all ports of the VLAN-unaware 4880f22ad45SFlorian Fainelli bridge. The VID corresponding to the VLAN upper interface spans the 4890f22ad45SFlorian Fainelli physical port of that VLAN interface, as well as the other ports that 4900f22ad45SFlorian Fainelli might be bridged with it. 4910f22ad45SFlorian Fainelli * Treat bridge ports with VLAN upper interfaces as standalone, and let 4920f22ad45SFlorian Fainelli forwarding be handled in the software data path. 4930f22ad45SFlorian Fainelli 4940f22ad45SFlorian Fainelli- with VLAN filtering turned on, these VLAN devices can be created as long as 4950f22ad45SFlorian Fainelli the bridge does not have an existing VLAN entry with the same VID on any 4960f22ad45SFlorian Fainelli bridge port. These VLAN devices cannot be enslaved into the bridge since they 4970f22ad45SFlorian Fainelli duplicate functionality/use case with the bridge's VLAN data path processing. 4980f22ad45SFlorian Fainelli 4990f22ad45SFlorian FainelliNon-bridged network ports of the same switch fabric must not be disturbed in any 5000f22ad45SFlorian Fainelliway by the enabling of VLAN filtering on the bridge device(s). If the VLAN 5010f22ad45SFlorian Fainellifiltering setting is global to the entire chip, then the standalone ports 5020f22ad45SFlorian Fainellishould indicate to the network stack that VLAN filtering is required by setting 5030f22ad45SFlorian Fainelli'rx-vlan-filter: on [fixed]' in the ethtool features. 5040f22ad45SFlorian Fainelli 5050f22ad45SFlorian FainelliBecause VLAN filtering can be turned on/off at runtime, the switchdev driver 5060f22ad45SFlorian Fainellimust be able to reconfigure the underlying hardware on the fly to honor the 5070f22ad45SFlorian Fainellitoggling of that option and behave appropriately. If that is not possible, the 5080f22ad45SFlorian Fainelliswitchdev driver can also refuse to support dynamic toggling of the VLAN 5090f22ad45SFlorian Fainellifiltering knob at runtime and require a destruction of the bridge device(s) and 5100f22ad45SFlorian Fainellicreation of new bridge device(s) with a different VLAN filtering value to 5110f22ad45SFlorian Fainelliensure VLAN awareness is pushed down to the hardware. 5120f22ad45SFlorian Fainelli 5130f22ad45SFlorian FainelliEven when VLAN filtering in the bridge is turned off, the underlying switch 5140f22ad45SFlorian Fainellihardware and driver may still configure itself in a VLAN-aware mode provided 5150f22ad45SFlorian Fainellithat the behavior described above is observed. 5160f22ad45SFlorian Fainelli 5170f22ad45SFlorian FainelliThe VLAN protocol of the bridge plays a role in deciding whether a packet is 5180f22ad45SFlorian Fainellitreated as tagged or not: a bridge using the 802.1ad protocol must treat both 5190f22ad45SFlorian FainelliVLAN-untagged packets, as well as packets tagged with 802.1Q headers, as 5200f22ad45SFlorian Fainelliuntagged. 5210f22ad45SFlorian Fainelli 5220f22ad45SFlorian FainelliThe 802.1p (VID 0) tagged packets must be treated in the same way by the device 5230f22ad45SFlorian Fainellias untagged packets, since the bridge device does not allow the manipulation of 5240f22ad45SFlorian FainelliVID 0 in its database. 5250f22ad45SFlorian Fainelli 5260f22ad45SFlorian FainelliWhen the bridge has VLAN filtering enabled and a PVID is not configured on the 5276b38c571SVladimir Olteaningress port, untagged and 802.1p tagged packets must be dropped. When the bridge 5280f22ad45SFlorian Fainellihas VLAN filtering enabled and a PVID exists on the ingress port, untagged and 5290f22ad45SFlorian Fainellipriority-tagged packets must be accepted and forwarded according to the 5300f22ad45SFlorian Fainellibridge's port membership of the PVID VLAN. When the bridge has VLAN filtering 5310f22ad45SFlorian Fainellidisabled, the presence/lack of a PVID should not influence the packet 5320f22ad45SFlorian Fainelliforwarding decision. 5330f22ad45SFlorian Fainelli 5340f22ad45SFlorian FainelliBridge IGMP snooping 5350f22ad45SFlorian Fainelli^^^^^^^^^^^^^^^^^^^^ 5360f22ad45SFlorian Fainelli 5370f22ad45SFlorian FainelliThe Linux bridge allows the configuration of IGMP snooping (statically, at 5380f22ad45SFlorian Fainelliinterface creation time, or dynamically, during runtime) which must be observed 5390f22ad45SFlorian Fainelliby the underlying switchdev network device/hardware in the following way: 5400f22ad45SFlorian Fainelli 5410f22ad45SFlorian Fainelli- when IGMP snooping is turned off, multicast traffic must be flooded to all 5420f22ad45SFlorian Fainelli ports within the same bridge that have mcast_flood=true. The CPU/management 5430f22ad45SFlorian Fainelli port should ideally not be flooded (unless the ingress interface has 5440f22ad45SFlorian Fainelli IFF_ALLMULTI or IFF_PROMISC) and continue to learn multicast traffic through 5450f22ad45SFlorian Fainelli the network stack notifications. If the hardware is not capable of doing that 5460f22ad45SFlorian Fainelli then the CPU/management port must also be flooded and multicast filtering 5470f22ad45SFlorian Fainelli happens in software. 5480f22ad45SFlorian Fainelli 5490f22ad45SFlorian Fainelli- when IGMP snooping is turned on, multicast traffic must selectively flow 5500f22ad45SFlorian Fainelli to the appropriate network ports (including CPU/management port). Flooding of 5510f22ad45SFlorian Fainelli unknown multicast should be only towards the ports connected to a multicast 5520f22ad45SFlorian Fainelli router (the local device may also act as a multicast router). 5530f22ad45SFlorian Fainelli 5540f22ad45SFlorian FainelliThe switch must adhere to RFC 4541 and flood multicast traffic accordingly 5550f22ad45SFlorian Fainellisince that is what the Linux bridge implementation does. 5560f22ad45SFlorian Fainelli 5570f22ad45SFlorian FainelliBecause IGMP snooping can be turned on/off at runtime, the switchdev driver 5580f22ad45SFlorian Fainellimust be able to reconfigure the underlying hardware on the fly to honor the 5590f22ad45SFlorian Fainellitoggling of that option and behave appropriately. 5600f22ad45SFlorian Fainelli 5610f22ad45SFlorian FainelliA switchdev driver can also refuse to support dynamic toggling of the multicast 5620f22ad45SFlorian Fainellisnooping knob at runtime and require the destruction of the bridge device(s) 5630f22ad45SFlorian Fainelliand creation of a new bridge device(s) with a different multicast snooping 5640f22ad45SFlorian Fainellivalue. 565