1.. SPDX-License-Identifier: GPL-2.0
2
3=====================================
4Network Devices, the Kernel, and You!
5=====================================
6
7
8Introduction
9============
10The following is a random collection of documentation regarding
11network devices.
12
13struct net_device lifetime rules
14================================
15Network device structures need to persist even after module is unloaded and
16must be allocated with alloc_netdev_mqs() and friends.
17If device has registered successfully, it will be freed on last use
18by free_netdev(). This is required to handle the pathological case cleanly
19(example: ``rmmod mydriver </sys/class/net/myeth/mtu``)
20
21alloc_netdev_mqs() / alloc_netdev() reserve extra space for driver
22private data which gets freed when the network device is freed. If
23separately allocated data is attached to the network device
24(netdev_priv()) then it is up to the module exit handler to free that.
25
26There are two groups of APIs for registering struct net_device.
27First group can be used in normal contexts where ``rtnl_lock`` is not already
28held: register_netdev(), unregister_netdev().
29Second group can be used when ``rtnl_lock`` is already held:
30register_netdevice(), unregister_netdevice(), free_netdevice().
31
32Simple drivers
33--------------
34
35Most drivers (especially device drivers) handle lifetime of struct net_device
36in context where ``rtnl_lock`` is not held (e.g. driver probe and remove paths).
37
38In that case the struct net_device registration is done using
39the register_netdev(), and unregister_netdev() functions:
40
41.. code-block:: c
42
43  int probe()
44  {
45    struct my_device_priv *priv;
46    int err;
47
48    dev = alloc_netdev_mqs(...);
49    if (!dev)
50      return -ENOMEM;
51    priv = netdev_priv(dev);
52
53    /* ... do all device setup before calling register_netdev() ...
54     */
55
56    err = register_netdev(dev);
57    if (err)
58      goto err_undo;
59
60    /* net_device is visible to the user! */
61
62  err_undo:
63    /* ... undo the device setup ... */
64    free_netdev(dev);
65    return err;
66  }
67
68  void remove()
69  {
70    unregister_netdev(dev);
71    free_netdev(dev);
72  }
73
74Note that after calling register_netdev() the device is visible in the system.
75Users can open it and start sending / receiving traffic immediately,
76or run any other callback, so all initialization must be done prior to
77registration.
78
79unregister_netdev() closes the device and waits for all users to be done
80with it. The memory of struct net_device itself may still be referenced
81by sysfs but all operations on that device will fail.
82
83free_netdev() can be called after unregister_netdev() returns on when
84register_netdev() failed.
85
86Device management under RTNL
87----------------------------
88
89Registering struct net_device while in context which already holds
90the ``rtnl_lock`` requires extra care. In those scenarios most drivers
91will want to make use of struct net_device's ``needs_free_netdev``
92and ``priv_destructor`` members for freeing of state.
93
94Example flow of netdev handling under ``rtnl_lock``:
95
96.. code-block:: c
97
98  static void my_setup(struct net_device *dev)
99  {
100    dev->needs_free_netdev = true;
101  }
102
103  static void my_destructor(struct net_device *dev)
104  {
105    some_obj_destroy(priv->obj);
106    some_uninit(priv);
107  }
108
109  int create_link()
110  {
111    struct my_device_priv *priv;
112    int err;
113
114    ASSERT_RTNL();
115
116    dev = alloc_netdev(sizeof(*priv), "net%d", NET_NAME_UNKNOWN, my_setup);
117    if (!dev)
118      return -ENOMEM;
119    priv = netdev_priv(dev);
120
121    /* Implicit constructor */
122    err = some_init(priv);
123    if (err)
124      goto err_free_dev;
125
126    priv->obj = some_obj_create();
127    if (!priv->obj) {
128      err = -ENOMEM;
129      goto err_some_uninit;
130    }
131    /* End of constructor, set the destructor: */
132    dev->priv_destructor = my_destructor;
133
134    err = register_netdevice(dev);
135    if (err)
136      /* register_netdevice() calls destructor on failure */
137      goto err_free_dev;
138
139    /* If anything fails now unregister_netdevice() (or unregister_netdev())
140     * will take care of calling my_destructor and free_netdev().
141     */
142
143    return 0;
144
145  err_some_uninit:
146    some_uninit(priv);
147  err_free_dev:
148    free_netdev(dev);
149    return err;
150  }
151
152If struct net_device.priv_destructor is set it will be called by the core
153some time after unregister_netdevice(), it will also be called if
154register_netdevice() fails. The callback may be invoked with or without
155``rtnl_lock`` held.
156
157There is no explicit constructor callback, driver "constructs" the private
158netdev state after allocating it and before registration.
159
160Setting struct net_device.needs_free_netdev makes core call free_netdevice()
161automatically after unregister_netdevice() when all references to the device
162are gone. It only takes effect after a successful call to register_netdevice()
163so if register_netdevice() fails driver is responsible for calling
164free_netdev().
165
166free_netdev() is safe to call on error paths right after unregister_netdevice()
167or when register_netdevice() fails. Parts of netdev (de)registration process
168happen after ``rtnl_lock`` is released, therefore in those cases free_netdev()
169will defer some of the processing until ``rtnl_lock`` is released.
170
171Devices spawned from struct rtnl_link_ops should never free the
172struct net_device directly.
173
174.ndo_init and .ndo_uninit
175~~~~~~~~~~~~~~~~~~~~~~~~~
176
177``.ndo_init`` and ``.ndo_uninit`` callbacks are called during net_device
178registration and de-registration, under ``rtnl_lock``. Drivers can use
179those e.g. when parts of their init process need to run under ``rtnl_lock``.
180
181``.ndo_init`` runs before device is visible in the system, ``.ndo_uninit``
182runs during de-registering after device is closed but other subsystems
183may still have outstanding references to the netdevice.
184
185MTU
186===
187Each network device has a Maximum Transfer Unit. The MTU does not
188include any link layer protocol overhead. Upper layer protocols must
189not pass a socket buffer (skb) to a device to transmit with more data
190than the mtu. The MTU does not include link layer header overhead, so
191for example on Ethernet if the standard MTU is 1500 bytes used, the
192actual skb will contain up to 1514 bytes because of the Ethernet
193header. Devices should allow for the 4 byte VLAN header as well.
194
195Segmentation Offload (GSO, TSO) is an exception to this rule.  The
196upper layer protocol may pass a large socket buffer to the device
197transmit routine, and the device will break that up into separate
198packets based on the current MTU.
199
200MTU is symmetrical and applies both to receive and transmit. A device
201must be able to receive at least the maximum size packet allowed by
202the MTU. A network device may use the MTU as mechanism to size receive
203buffers, but the device should allow packets with VLAN header. With
204standard Ethernet mtu of 1500 bytes, the device should allow up to
2051518 byte packets (1500 + 14 header + 4 tag).  The device may either:
206drop, truncate, or pass up oversize packets, but dropping oversize
207packets is preferred.
208
209
210struct net_device synchronization rules
211=======================================
212ndo_open:
213	Synchronization: rtnl_lock() semaphore.
214	Context: process
215
216ndo_stop:
217	Synchronization: rtnl_lock() semaphore.
218	Context: process
219	Note: netif_running() is guaranteed false
220
221ndo_do_ioctl:
222	Synchronization: rtnl_lock() semaphore.
223	Context: process
224
225        This is only called by network subsystems internally,
226        not by user space calling ioctl as it was in before
227        linux-5.14.
228
229ndo_siocbond:
230        Synchronization: rtnl_lock() semaphore.
231        Context: process
232
233        Used by the bonding driver for the SIOCBOND family of
234        ioctl commands.
235
236ndo_siocwandev:
237	Synchronization: rtnl_lock() semaphore.
238	Context: process
239
240	Used by the drivers/net/wan framework to handle
241	the SIOCWANDEV ioctl with the if_settings structure.
242
243ndo_siocdevprivate:
244	Synchronization: rtnl_lock() semaphore.
245	Context: process
246
247	This is used to implement SIOCDEVPRIVATE ioctl helpers.
248	These should not be added to new drivers, so don't use.
249
250ndo_eth_ioctl:
251	Synchronization: rtnl_lock() semaphore.
252	Context: process
253
254ndo_get_stats:
255	Synchronization: rtnl_lock() semaphore, dev_base_lock rwlock, or RCU.
256	Context: atomic (can't sleep under rwlock or RCU)
257
258ndo_start_xmit:
259	Synchronization: __netif_tx_lock spinlock.
260
261	When the driver sets NETIF_F_LLTX in dev->features this will be
262	called without holding netif_tx_lock. In this case the driver
263	has to lock by itself when needed.
264	The locking there should also properly protect against
265	set_rx_mode. WARNING: use of NETIF_F_LLTX is deprecated.
266	Don't use it for new drivers.
267
268	Context: Process with BHs disabled or BH (timer),
269		 will be called with interrupts disabled by netconsole.
270
271	Return codes:
272
273	* NETDEV_TX_OK everything ok.
274	* NETDEV_TX_BUSY Cannot transmit packet, try later
275	  Usually a bug, means queue start/stop flow control is broken in
276	  the driver. Note: the driver must NOT put the skb in its DMA ring.
277
278ndo_tx_timeout:
279	Synchronization: netif_tx_lock spinlock; all TX queues frozen.
280	Context: BHs disabled
281	Notes: netif_queue_stopped() is guaranteed true
282
283ndo_set_rx_mode:
284	Synchronization: netif_addr_lock spinlock.
285	Context: BHs disabled
286
287struct napi_struct synchronization rules
288========================================
289napi->poll:
290	Synchronization:
291		NAPI_STATE_SCHED bit in napi->state.  Device
292		driver's ndo_stop method will invoke napi_disable() on
293		all NAPI instances which will do a sleeping poll on the
294		NAPI_STATE_SCHED napi->state bit, waiting for all pending
295		NAPI activity to cease.
296
297	Context:
298		 softirq
299		 will be called with interrupts disabled by netconsole.
300