1.. SPDX-License-Identifier: GPL-2.0
2
3===================================
4Linux Ethernet Bonding Driver HOWTO
5===================================
6
7Latest update: 27 April 2011
8
9Initial release: Thomas Davis <tadavis at lbl.gov>
10
11Corrections, HA extensions: 2000/10/03-15:
12
13  - Willy Tarreau <willy at meta-x.org>
14  - Constantine Gavrilov <const-g at xpert.com>
15  - Chad N. Tindel <ctindel at ieee dot org>
16  - Janice Girouard <girouard at us dot ibm dot com>
17  - Jay Vosburgh <fubar at us dot ibm dot com>
18
19Reorganized and updated Feb 2005 by Jay Vosburgh
20Added Sysfs information: 2006/04/24
21
22  - Mitch Williams <mitch.a.williams at intel.com>
23
24Introduction
25============
26
27The Linux bonding driver provides a method for aggregating
28multiple network interfaces into a single logical "bonded" interface.
29The behavior of the bonded interfaces depends upon the mode; generally
30speaking, modes provide either hot standby or load balancing services.
31Additionally, link integrity monitoring may be performed.
32
33The bonding driver originally came from Donald Becker's
34beowulf patches for kernel 2.0. It has changed quite a bit since, and
35the original tools from extreme-linux and beowulf sites will not work
36with this version of the driver.
37
38For new versions of the driver, updated userspace tools, and
39who to ask for help, please follow the links at the end of this file.
40
41.. Table of Contents
42
43   1. Bonding Driver Installation
44
45   2. Bonding Driver Options
46
47   3. Configuring Bonding Devices
48   3.1	Configuration with Sysconfig Support
49   3.1.1		Using DHCP with Sysconfig
50   3.1.2		Configuring Multiple Bonds with Sysconfig
51   3.2	Configuration with Initscripts Support
52   3.2.1		Using DHCP with Initscripts
53   3.2.2		Configuring Multiple Bonds with Initscripts
54   3.3	Configuring Bonding Manually with Ifenslave
55   3.3.1		Configuring Multiple Bonds Manually
56   3.4	Configuring Bonding Manually via Sysfs
57   3.5	Configuration with Interfaces Support
58   3.6	Overriding Configuration for Special Cases
59   3.7 Configuring LACP for 802.3ad mode in a more secure way
60
61   4. Querying Bonding Configuration
62   4.1	Bonding Configuration
63   4.2	Network Configuration
64
65   5. Switch Configuration
66
67   6. 802.1q VLAN Support
68
69   7. Link Monitoring
70   7.1	ARP Monitor Operation
71   7.2	Configuring Multiple ARP Targets
72   7.3	MII Monitor Operation
73
74   8. Potential Trouble Sources
75   8.1	Adventures in Routing
76   8.2	Ethernet Device Renaming
77   8.3	Painfully Slow Or No Failed Link Detection By Miimon
78
79   9. SNMP agents
80
81   10. Promiscuous mode
82
83   11. Configuring Bonding for High Availability
84   11.1	High Availability in a Single Switch Topology
85   11.2	High Availability in a Multiple Switch Topology
86   11.2.1		HA Bonding Mode Selection for Multiple Switch Topology
87   11.2.2		HA Link Monitoring for Multiple Switch Topology
88
89   12. Configuring Bonding for Maximum Throughput
90   12.1	Maximum Throughput in a Single Switch Topology
91   12.1.1		MT Bonding Mode Selection for Single Switch Topology
92   12.1.2		MT Link Monitoring for Single Switch Topology
93   12.2	Maximum Throughput in a Multiple Switch Topology
94   12.2.1		MT Bonding Mode Selection for Multiple Switch Topology
95   12.2.2		MT Link Monitoring for Multiple Switch Topology
96
97   13. Switch Behavior Issues
98   13.1	Link Establishment and Failover Delays
99   13.2	Duplicated Incoming Packets
100
101   14. Hardware Specific Considerations
102   14.1	IBM BladeCenter
103
104   15. Frequently Asked Questions
105
106   16. Resources and Links
107
108
1091. Bonding Driver Installation
110==============================
111
112Most popular distro kernels ship with the bonding driver
113already available as a module. If your distro does not, or you
114have need to compile bonding from source (e.g., configuring and
115installing a mainline kernel from kernel.org), you'll need to perform
116the following steps:
117
1181.1 Configure and build the kernel with bonding
119-----------------------------------------------
120
121The current version of the bonding driver is available in the
122drivers/net/bonding subdirectory of the most recent kernel source
123(which is available on http://kernel.org).  Most users "rolling their
124own" will want to use the most recent kernel from kernel.org.
125
126Configure kernel with "make menuconfig" (or "make xconfig" or
127"make config"), then select "Bonding driver support" in the "Network
128device support" section.  It is recommended that you configure the
129driver as module since it is currently the only way to pass parameters
130to the driver or configure more than one bonding device.
131
132Build and install the new kernel and modules.
133
1341.2 Bonding Control Utility
135---------------------------
136
137It is recommended to configure bonding via iproute2 (netlink)
138or sysfs, the old ifenslave control utility is obsolete.
139
1402. Bonding Driver Options
141=========================
142
143Options for the bonding driver are supplied as parameters to the
144bonding module at load time, or are specified via sysfs.
145
146Module options may be given as command line arguments to the
147insmod or modprobe command, but are usually specified in either the
148``/etc/modprobe.d/*.conf`` configuration files, or in a distro-specific
149configuration file (some of which are detailed in the next section).
150
151Details on bonding support for sysfs is provided in the
152"Configuring Bonding Manually via Sysfs" section, below.
153
154The available bonding driver parameters are listed below. If a
155parameter is not specified the default value is used.  When initially
156configuring a bond, it is recommended "tail -f /var/log/messages" be
157run in a separate window to watch for bonding driver error messages.
158
159It is critical that either the miimon or arp_interval and
160arp_ip_target parameters be specified, otherwise serious network
161degradation will occur during link failures.  Very few devices do not
162support at least miimon, so there is really no reason not to use it.
163
164Options with textual values will accept either the text name
165or, for backwards compatibility, the option value.  E.g.,
166"mode=802.3ad" and "mode=4" set the same mode.
167
168The parameters are as follows:
169
170active_slave
171
172	Specifies the new active slave for modes that support it
173	(active-backup, balance-alb and balance-tlb).  Possible values
174	are the name of any currently enslaved interface, or an empty
175	string.  If a name is given, the slave and its link must be up in order
176	to be selected as the new active slave.  If an empty string is
177	specified, the current active slave is cleared, and a new active
178	slave is selected automatically.
179
180	Note that this is only available through the sysfs interface. No module
181	parameter by this name exists.
182
183	The normal value of this option is the name of the currently
184	active slave, or the empty string if there is no active slave or
185	the current mode does not use an active slave.
186
187ad_actor_sys_prio
188
189	In an AD system, this specifies the system priority. The allowed range
190	is 1 - 65535. If the value is not specified, it takes 65535 as the
191	default value.
192
193	This parameter has effect only in 802.3ad mode and is available through
194	SysFs interface.
195
196ad_actor_system
197
198	In an AD system, this specifies the mac-address for the actor in
199	protocol packet exchanges (LACPDUs). The value cannot be a multicast
200	address. If the all-zeroes MAC is specified, bonding will internally
201	use the MAC of the bond itself. It is preferred to have the
202	local-admin bit set for this mac but driver does not enforce it. If
203	the value is not given then system defaults to using the masters'
204	mac address as actors' system address.
205
206	This parameter has effect only in 802.3ad mode and is available through
207	SysFs interface.
208
209ad_select
210
211	Specifies the 802.3ad aggregation selection logic to use.  The
212	possible values and their effects are:
213
214	stable or 0
215
216		The active aggregator is chosen by largest aggregate
217		bandwidth.
218
219		Reselection of the active aggregator occurs only when all
220		slaves of the active aggregator are down or the active
221		aggregator has no slaves.
222
223		This is the default value.
224
225	bandwidth or 1
226
227		The active aggregator is chosen by largest aggregate
228		bandwidth.  Reselection occurs if:
229
230		- A slave is added to or removed from the bond
231
232		- Any slave's link state changes
233
234		- Any slave's 802.3ad association state changes
235
236		- The bond's administrative state changes to up
237
238	count or 2
239
240		The active aggregator is chosen by the largest number of
241		ports (slaves).  Reselection occurs as described under the
242		"bandwidth" setting, above.
243
244	The bandwidth and count selection policies permit failover of
245	802.3ad aggregations when partial failure of the active aggregator
246	occurs.  This keeps the aggregator with the highest availability
247	(either in bandwidth or in number of ports) active at all times.
248
249	This option was added in bonding version 3.4.0.
250
251ad_user_port_key
252
253	In an AD system, the port-key has three parts as shown below -
254
255	   =====  ============
256	   Bits   Use
257	   =====  ============
258	   00     Duplex
259	   01-05  Speed
260	   06-15  User-defined
261	   =====  ============
262
263	This defines the upper 10 bits of the port key. The values can be
264	from 0 - 1023. If not given, the system defaults to 0.
265
266	This parameter has effect only in 802.3ad mode and is available through
267	SysFs interface.
268
269all_slaves_active
270
271	Specifies that duplicate frames (received on inactive ports) should be
272	dropped (0) or delivered (1).
273
274	Normally, bonding will drop duplicate frames (received on inactive
275	ports), which is desirable for most users. But there are some times
276	it is nice to allow duplicate frames to be delivered.
277
278	The default value is 0 (drop duplicate frames received on inactive
279	ports).
280
281arp_interval
282
283	Specifies the ARP link monitoring frequency in milliseconds.
284
285	The ARP monitor works by periodically checking the slave
286	devices to determine whether they have sent or received
287	traffic recently (the precise criteria depends upon the
288	bonding mode, and the state of the slave).  Regular traffic is
289	generated via ARP probes issued for the addresses specified by
290	the arp_ip_target option.
291
292	This behavior can be modified by the arp_validate option,
293	below.
294
295	If ARP monitoring is used in an etherchannel compatible mode
296	(modes 0 and 2), the switch should be configured in a mode
297	that evenly distributes packets across all links. If the
298	switch is configured to distribute the packets in an XOR
299	fashion, all replies from the ARP targets will be received on
300	the same link which could cause the other team members to
301	fail.  ARP monitoring should not be used in conjunction with
302	miimon.  A value of 0 disables ARP monitoring.  The default
303	value is 0.
304
305arp_ip_target
306
307	Specifies the IP addresses to use as ARP monitoring peers when
308	arp_interval is > 0.  These are the targets of the ARP request
309	sent to determine the health of the link to the targets.
310	Specify these values in ddd.ddd.ddd.ddd format.  Multiple IP
311	addresses must be separated by a comma.  At least one IP
312	address must be given for ARP monitoring to function.  The
313	maximum number of targets that can be specified is 16.  The
314	default value is no IP addresses.
315
316ns_ip6_target
317
318	Specifies the IPv6 addresses to use as IPv6 monitoring peers when
319	arp_interval is > 0.  These are the targets of the NS request
320	sent to determine the health of the link to the targets.
321	Specify these values in ffff:ffff::ffff:ffff format.  Multiple IPv6
322	addresses must be separated by a comma.  At least one IPv6
323	address must be given for NS/NA monitoring to function.  The
324	maximum number of targets that can be specified is 16.  The
325	default value is no IPv6 addresses.
326
327arp_validate
328
329	Specifies whether or not ARP probes and replies should be
330	validated in any mode that supports arp monitoring, or whether
331	non-ARP traffic should be filtered (disregarded) for link
332	monitoring purposes.
333
334	Possible values are:
335
336	none or 0
337
338		No validation or filtering is performed.
339
340	active or 1
341
342		Validation is performed only for the active slave.
343
344	backup or 2
345
346		Validation is performed only for backup slaves.
347
348	all or 3
349
350		Validation is performed for all slaves.
351
352	filter or 4
353
354		Filtering is applied to all slaves. No validation is
355		performed.
356
357	filter_active or 5
358
359		Filtering is applied to all slaves, validation is performed
360		only for the active slave.
361
362	filter_backup or 6
363
364		Filtering is applied to all slaves, validation is performed
365		only for backup slaves.
366
367	Validation:
368
369	Enabling validation causes the ARP monitor to examine the incoming
370	ARP requests and replies, and only consider a slave to be up if it
371	is receiving the appropriate ARP traffic.
372
373	For an active slave, the validation checks ARP replies to confirm
374	that they were generated by an arp_ip_target.  Since backup slaves
375	do not typically receive these replies, the validation performed
376	for backup slaves is on the broadcast ARP request sent out via the
377	active slave.  It is possible that some switch or network
378	configurations may result in situations wherein the backup slaves
379	do not receive the ARP requests; in such a situation, validation
380	of backup slaves must be disabled.
381
382	The validation of ARP requests on backup slaves is mainly helping
383	bonding to decide which slaves are more likely to work in case of
384	the active slave failure, it doesn't really guarantee that the
385	backup slave will work if it's selected as the next active slave.
386
387	Validation is useful in network configurations in which multiple
388	bonding hosts are concurrently issuing ARPs to one or more targets
389	beyond a common switch.  Should the link between the switch and
390	target fail (but not the switch itself), the probe traffic
391	generated by the multiple bonding instances will fool the standard
392	ARP monitor into considering the links as still up.  Use of
393	validation can resolve this, as the ARP monitor will only consider
394	ARP requests and replies associated with its own instance of
395	bonding.
396
397	Filtering:
398
399	Enabling filtering causes the ARP monitor to only use incoming ARP
400	packets for link availability purposes.  Arriving packets that are
401	not ARPs are delivered normally, but do not count when determining
402	if a slave is available.
403
404	Filtering operates by only considering the reception of ARP
405	packets (any ARP packet, regardless of source or destination) when
406	determining if a slave has received traffic for link availability
407	purposes.
408
409	Filtering is useful in network configurations in which significant
410	levels of third party broadcast traffic would fool the standard
411	ARP monitor into considering the links as still up.  Use of
412	filtering can resolve this, as only ARP traffic is considered for
413	link availability purposes.
414
415	This option was added in bonding version 3.1.0.
416
417arp_all_targets
418
419	Specifies the quantity of arp_ip_targets that must be reachable
420	in order for the ARP monitor to consider a slave as being up.
421	This option affects only active-backup mode for slaves with
422	arp_validation enabled.
423
424	Possible values are:
425
426	any or 0
427
428		consider the slave up only when any of the arp_ip_targets
429		is reachable
430
431	all or 1
432
433		consider the slave up only when all of the arp_ip_targets
434		are reachable
435
436arp_missed_max
437
438	Specifies the number of arp_interval monitor checks that must
439	fail in order for an interface to be marked down by the ARP monitor.
440
441	In order to provide orderly failover semantics, backup interfaces
442	are permitted an extra monitor check (i.e., they must fail
443	arp_missed_max + 1 times before being marked down).
444
445	The default value is 2, and the allowable range is 1 - 255.
446
447downdelay
448
449	Specifies the time, in milliseconds, to wait before disabling
450	a slave after a link failure has been detected.  This option
451	is only valid for the miimon link monitor.  The downdelay
452	value should be a multiple of the miimon value; if not, it
453	will be rounded down to the nearest multiple.  The default
454	value is 0.
455
456fail_over_mac
457
458	Specifies whether active-backup mode should set all slaves to
459	the same MAC address at enslavement (the traditional
460	behavior), or, when enabled, perform special handling of the
461	bond's MAC address in accordance with the selected policy.
462
463	Possible values are:
464
465	none or 0
466
467		This setting disables fail_over_mac, and causes
468		bonding to set all slaves of an active-backup bond to
469		the same MAC address at enslavement time.  This is the
470		default.
471
472	active or 1
473
474		The "active" fail_over_mac policy indicates that the
475		MAC address of the bond should always be the MAC
476		address of the currently active slave.  The MAC
477		address of the slaves is not changed; instead, the MAC
478		address of the bond changes during a failover.
479
480		This policy is useful for devices that cannot ever
481		alter their MAC address, or for devices that refuse
482		incoming broadcasts with their own source MAC (which
483		interferes with the ARP monitor).
484
485		The down side of this policy is that every device on
486		the network must be updated via gratuitous ARP,
487		vs. just updating a switch or set of switches (which
488		often takes place for any traffic, not just ARP
489		traffic, if the switch snoops incoming traffic to
490		update its tables) for the traditional method.  If the
491		gratuitous ARP is lost, communication may be
492		disrupted.
493
494		When this policy is used in conjunction with the mii
495		monitor, devices which assert link up prior to being
496		able to actually transmit and receive are particularly
497		susceptible to loss of the gratuitous ARP, and an
498		appropriate updelay setting may be required.
499
500	follow or 2
501
502		The "follow" fail_over_mac policy causes the MAC
503		address of the bond to be selected normally (normally
504		the MAC address of the first slave added to the bond).
505		However, the second and subsequent slaves are not set
506		to this MAC address while they are in a backup role; a
507		slave is programmed with the bond's MAC address at
508		failover time (and the formerly active slave receives
509		the newly active slave's MAC address).
510
511		This policy is useful for multiport devices that
512		either become confused or incur a performance penalty
513		when multiple ports are programmed with the same MAC
514		address.
515
516
517	The default policy is none, unless the first slave cannot
518	change its MAC address, in which case the active policy is
519	selected by default.
520
521	This option may be modified via sysfs only when no slaves are
522	present in the bond.
523
524	This option was added in bonding version 3.2.0.  The "follow"
525	policy was added in bonding version 3.3.0.
526
527lacp_active
528	Option specifying whether to send LACPDU frames periodically.
529
530	off or 0
531		LACPDU frames acts as "speak when spoken to".
532
533	on or 1
534		LACPDU frames are sent along the configured links
535		periodically. See lacp_rate for more details.
536
537	The default is on.
538
539lacp_rate
540
541	Option specifying the rate in which we'll ask our link partner
542	to transmit LACPDU packets in 802.3ad mode.  Possible values
543	are:
544
545	slow or 0
546		Request partner to transmit LACPDUs every 30 seconds
547
548	fast or 1
549		Request partner to transmit LACPDUs every 1 second
550
551	The default is slow.
552
553max_bonds
554
555	Specifies the number of bonding devices to create for this
556	instance of the bonding driver.  E.g., if max_bonds is 3, and
557	the bonding driver is not already loaded, then bond0, bond1
558	and bond2 will be created.  The default value is 1.  Specifying
559	a value of 0 will load bonding, but will not create any devices.
560
561miimon
562
563	Specifies the MII link monitoring frequency in milliseconds.
564	This determines how often the link state of each slave is
565	inspected for link failures.  A value of zero disables MII
566	link monitoring.  A value of 100 is a good starting point.
567	The use_carrier option, below, affects how the link state is
568	determined.  See the High Availability section for additional
569	information.  The default value is 100 if arp_interval is not
570	set.
571
572min_links
573
574	Specifies the minimum number of links that must be active before
575	asserting carrier. It is similar to the Cisco EtherChannel min-links
576	feature. This allows setting the minimum number of member ports that
577	must be up (link-up state) before marking the bond device as up
578	(carrier on). This is useful for situations where higher level services
579	such as clustering want to ensure a minimum number of low bandwidth
580	links are active before switchover. This option only affect 802.3ad
581	mode.
582
583	The default value is 0. This will cause carrier to be asserted (for
584	802.3ad mode) whenever there is an active aggregator, regardless of the
585	number of available links in that aggregator. Note that, because an
586	aggregator cannot be active without at least one available link,
587	setting this option to 0 or to 1 has the exact same effect.
588
589mode
590
591	Specifies one of the bonding policies. The default is
592	balance-rr (round robin).  Possible values are:
593
594	balance-rr or 0
595
596		Round-robin policy: Transmit packets in sequential
597		order from the first available slave through the
598		last.  This mode provides load balancing and fault
599		tolerance.
600
601	active-backup or 1
602
603		Active-backup policy: Only one slave in the bond is
604		active.  A different slave becomes active if, and only
605		if, the active slave fails.  The bond's MAC address is
606		externally visible on only one port (network adapter)
607		to avoid confusing the switch.
608
609		In bonding version 2.6.2 or later, when a failover
610		occurs in active-backup mode, bonding will issue one
611		or more gratuitous ARPs on the newly active slave.
612		One gratuitous ARP is issued for the bonding master
613		interface and each VLAN interfaces configured above
614		it, provided that the interface has at least one IP
615		address configured.  Gratuitous ARPs issued for VLAN
616		interfaces are tagged with the appropriate VLAN id.
617
618		This mode provides fault tolerance.  The primary
619		option, documented below, affects the behavior of this
620		mode.
621
622	balance-xor or 2
623
624		XOR policy: Transmit based on the selected transmit
625		hash policy.  The default policy is a simple [(source
626		MAC address XOR'd with destination MAC address XOR
627		packet type ID) modulo slave count].  Alternate transmit
628		policies may be	selected via the xmit_hash_policy option,
629		described below.
630
631		This mode provides load balancing and fault tolerance.
632
633	broadcast or 3
634
635		Broadcast policy: transmits everything on all slave
636		interfaces.  This mode provides fault tolerance.
637
638	802.3ad or 4
639
640		IEEE 802.3ad Dynamic link aggregation.  Creates
641		aggregation groups that share the same speed and
642		duplex settings.  Utilizes all slaves in the active
643		aggregator according to the 802.3ad specification.
644
645		Slave selection for outgoing traffic is done according
646		to the transmit hash policy, which may be changed from
647		the default simple XOR policy via the xmit_hash_policy
648		option, documented below.  Note that not all transmit
649		policies may be 802.3ad compliant, particularly in
650		regards to the packet mis-ordering requirements of
651		section 43.2.4 of the 802.3ad standard.  Differing
652		peer implementations will have varying tolerances for
653		noncompliance.
654
655		Prerequisites:
656
657		1. Ethtool support in the base drivers for retrieving
658		the speed and duplex of each slave.
659
660		2. A switch that supports IEEE 802.3ad Dynamic link
661		aggregation.
662
663		Most switches will require some type of configuration
664		to enable 802.3ad mode.
665
666	balance-tlb or 5
667
668		Adaptive transmit load balancing: channel bonding that
669		does not require any special switch support.
670
671		In tlb_dynamic_lb=1 mode; the outgoing traffic is
672		distributed according to the current load (computed
673		relative to the speed) on each slave.
674
675		In tlb_dynamic_lb=0 mode; the load balancing based on
676		current load is disabled and the load is distributed
677		only using the hash distribution.
678
679		Incoming traffic is received by the current slave.
680		If the receiving slave fails, another slave takes over
681		the MAC address of the failed receiving slave.
682
683		Prerequisite:
684
685		Ethtool support in the base drivers for retrieving the
686		speed of each slave.
687
688	balance-alb or 6
689
690		Adaptive load balancing: includes balance-tlb plus
691		receive load balancing (rlb) for IPV4 traffic, and
692		does not require any special switch support.  The
693		receive load balancing is achieved by ARP negotiation.
694		The bonding driver intercepts the ARP Replies sent by
695		the local system on their way out and overwrites the
696		source hardware address with the unique hardware
697		address of one of the slaves in the bond such that
698		different peers use different hardware addresses for
699		the server.
700
701		Receive traffic from connections created by the server
702		is also balanced.  When the local system sends an ARP
703		Request the bonding driver copies and saves the peer's
704		IP information from the ARP packet.  When the ARP
705		Reply arrives from the peer, its hardware address is
706		retrieved and the bonding driver initiates an ARP
707		reply to this peer assigning it to one of the slaves
708		in the bond.  A problematic outcome of using ARP
709		negotiation for balancing is that each time that an
710		ARP request is broadcast it uses the hardware address
711		of the bond.  Hence, peers learn the hardware address
712		of the bond and the balancing of receive traffic
713		collapses to the current slave.  This is handled by
714		sending updates (ARP Replies) to all the peers with
715		their individually assigned hardware address such that
716		the traffic is redistributed.  Receive traffic is also
717		redistributed when a new slave is added to the bond
718		and when an inactive slave is re-activated.  The
719		receive load is distributed sequentially (round robin)
720		among the group of highest speed slaves in the bond.
721
722		When a link is reconnected or a new slave joins the
723		bond the receive traffic is redistributed among all
724		active slaves in the bond by initiating ARP Replies
725		with the selected MAC address to each of the
726		clients. The updelay parameter (detailed below) must
727		be set to a value equal or greater than the switch's
728		forwarding delay so that the ARP Replies sent to the
729		peers will not be blocked by the switch.
730
731		Prerequisites:
732
733		1. Ethtool support in the base drivers for retrieving
734		the speed of each slave.
735
736		2. Base driver support for setting the hardware
737		address of a device while it is open.  This is
738		required so that there will always be one slave in the
739		team using the bond hardware address (the
740		curr_active_slave) while having a unique hardware
741		address for each slave in the bond.  If the
742		curr_active_slave fails its hardware address is
743		swapped with the new curr_active_slave that was
744		chosen.
745
746num_grat_arp,
747num_unsol_na
748
749	Specify the number of peer notifications (gratuitous ARPs and
750	unsolicited IPv6 Neighbor Advertisements) to be issued after a
751	failover event.  As soon as the link is up on the new slave
752	(possibly immediately) a peer notification is sent on the
753	bonding device and each VLAN sub-device. This is repeated at
754	the rate specified by peer_notif_delay if the number is
755	greater than 1.
756
757	The valid range is 0 - 255; the default value is 1.  These options
758	affect only the active-backup mode.  These options were added for
759	bonding versions 3.3.0 and 3.4.0 respectively.
760
761	From Linux 3.0 and bonding version 3.7.1, these notifications
762	are generated by the ipv4 and ipv6 code and the numbers of
763	repetitions cannot be set independently.
764
765packets_per_slave
766
767	Specify the number of packets to transmit through a slave before
768	moving to the next one. When set to 0 then a slave is chosen at
769	random.
770
771	The valid range is 0 - 65535; the default value is 1. This option
772	has effect only in balance-rr mode.
773
774peer_notif_delay
775
776	Specify the delay, in milliseconds, between each peer
777	notification (gratuitous ARP and unsolicited IPv6 Neighbor
778	Advertisement) when they are issued after a failover event.
779	This delay should be a multiple of the MII link monitor interval
780	(miimon).
781
782	The valid range is 0 - 300000. The default value is 0, which means
783	to match the value of the MII link monitor interval.
784
785prio
786	Slave priority. A higher number means higher priority.
787	The primary slave has the highest priority. This option also
788	follows the primary_reselect rules.
789
790	This option could only be configured via netlink, and is only valid
791	for active-backup(1), balance-tlb (5) and balance-alb (6) mode.
792	The valid value range is a signed 32 bit integer.
793
794	The default value is 0.
795
796primary
797
798	A string (eth0, eth2, etc) specifying which slave is the
799	primary device.  The specified device will always be the
800	active slave while it is available.  Only when the primary is
801	off-line will alternate devices be used.  This is useful when
802	one slave is preferred over another, e.g., when one slave has
803	higher throughput than another.
804
805	The primary option is only valid for active-backup(1),
806	balance-tlb (5) and balance-alb (6) mode.
807
808primary_reselect
809
810	Specifies the reselection policy for the primary slave.  This
811	affects how the primary slave is chosen to become the active slave
812	when failure of the active slave or recovery of the primary slave
813	occurs.  This option is designed to prevent flip-flopping between
814	the primary slave and other slaves.  Possible values are:
815
816	always or 0 (default)
817
818		The primary slave becomes the active slave whenever it
819		comes back up.
820
821	better or 1
822
823		The primary slave becomes the active slave when it comes
824		back up, if the speed and duplex of the primary slave is
825		better than the speed and duplex of the current active
826		slave.
827
828	failure or 2
829
830		The primary slave becomes the active slave only if the
831		current active slave fails and the primary slave is up.
832
833	The primary_reselect setting is ignored in two cases:
834
835		If no slaves are active, the first slave to recover is
836		made the active slave.
837
838		When initially enslaved, the primary slave is always made
839		the active slave.
840
841	Changing the primary_reselect policy via sysfs will cause an
842	immediate selection of the best active slave according to the new
843	policy.  This may or may not result in a change of the active
844	slave, depending upon the circumstances.
845
846	This option was added for bonding version 3.6.0.
847
848tlb_dynamic_lb
849
850	Specifies if dynamic shuffling of flows is enabled in tlb
851	or alb mode. The value has no effect on any other modes.
852
853	The default behavior of tlb mode is to shuffle active flows across
854	slaves based on the load in that interval. This gives nice lb
855	characteristics but can cause packet reordering. If re-ordering is
856	a concern use this variable to disable flow shuffling and rely on
857	load balancing provided solely by the hash distribution.
858	xmit-hash-policy can be used to select the appropriate hashing for
859	the setup.
860
861	The sysfs entry can be used to change the setting per bond device
862	and the initial value is derived from the module parameter. The
863	sysfs entry is allowed to be changed only if the bond device is
864	down.
865
866	The default value is "1" that enables flow shuffling while value "0"
867	disables it. This option was added in bonding driver 3.7.1
868
869
870updelay
871
872	Specifies the time, in milliseconds, to wait before enabling a
873	slave after a link recovery has been detected.  This option is
874	only valid for the miimon link monitor.  The updelay value
875	should be a multiple of the miimon value; if not, it will be
876	rounded down to the nearest multiple.  The default value is 0.
877
878use_carrier
879
880	Specifies whether or not miimon should use MII or ETHTOOL
881	ioctls vs. netif_carrier_ok() to determine the link
882	status. The MII or ETHTOOL ioctls are less efficient and
883	utilize a deprecated calling sequence within the kernel.  The
884	netif_carrier_ok() relies on the device driver to maintain its
885	state with netif_carrier_on/off; at this writing, most, but
886	not all, device drivers support this facility.
887
888	If bonding insists that the link is up when it should not be,
889	it may be that your network device driver does not support
890	netif_carrier_on/off.  The default state for netif_carrier is
891	"carrier on," so if a driver does not support netif_carrier,
892	it will appear as if the link is always up.  In this case,
893	setting use_carrier to 0 will cause bonding to revert to the
894	MII / ETHTOOL ioctl method to determine the link state.
895
896	A value of 1 enables the use of netif_carrier_ok(), a value of
897	0 will use the deprecated MII / ETHTOOL ioctls.  The default
898	value is 1.
899
900xmit_hash_policy
901
902	Selects the transmit hash policy to use for slave selection in
903	balance-xor, 802.3ad, and tlb modes.  Possible values are:
904
905	layer2
906
907		Uses XOR of hardware MAC addresses and packet type ID
908		field to generate the hash. The formula is
909
910		hash = source MAC[5] XOR destination MAC[5] XOR packet type ID
911		slave number = hash modulo slave count
912
913		This algorithm will place all traffic to a particular
914		network peer on the same slave.
915
916		This algorithm is 802.3ad compliant.
917
918	layer2+3
919
920		This policy uses a combination of layer2 and layer3
921		protocol information to generate the hash.
922
923		Uses XOR of hardware MAC addresses and IP addresses to
924		generate the hash.  The formula is
925
926		hash = source MAC[5] XOR destination MAC[5] XOR packet type ID
927		hash = hash XOR source IP XOR destination IP
928		hash = hash XOR (hash RSHIFT 16)
929		hash = hash XOR (hash RSHIFT 8)
930		And then hash is reduced modulo slave count.
931
932		If the protocol is IPv6 then the source and destination
933		addresses are first hashed using ipv6_addr_hash.
934
935		This algorithm will place all traffic to a particular
936		network peer on the same slave.  For non-IP traffic,
937		the formula is the same as for the layer2 transmit
938		hash policy.
939
940		This policy is intended to provide a more balanced
941		distribution of traffic than layer2 alone, especially
942		in environments where a layer3 gateway device is
943		required to reach most destinations.
944
945		This algorithm is 802.3ad compliant.
946
947	layer3+4
948
949		This policy uses upper layer protocol information,
950		when available, to generate the hash.  This allows for
951		traffic to a particular network peer to span multiple
952		slaves, although a single connection will not span
953		multiple slaves.
954
955		The formula for unfragmented TCP and UDP packets is
956
957		hash = source port, destination port (as in the header)
958		hash = hash XOR source IP XOR destination IP
959		hash = hash XOR (hash RSHIFT 16)
960		hash = hash XOR (hash RSHIFT 8)
961		hash = hash RSHIFT 1
962		And then hash is reduced modulo slave count.
963
964		If the protocol is IPv6 then the source and destination
965		addresses are first hashed using ipv6_addr_hash.
966
967		For fragmented TCP or UDP packets and all other IPv4 and
968		IPv6 protocol traffic, the source and destination port
969		information is omitted.  For non-IP traffic, the
970		formula is the same as for the layer2 transmit hash
971		policy.
972
973		This algorithm is not fully 802.3ad compliant.  A
974		single TCP or UDP conversation containing both
975		fragmented and unfragmented packets will see packets
976		striped across two interfaces.  This may result in out
977		of order delivery.  Most traffic types will not meet
978		this criteria, as TCP rarely fragments traffic, and
979		most UDP traffic is not involved in extended
980		conversations.  Other implementations of 802.3ad may
981		or may not tolerate this noncompliance.
982
983	encap2+3
984
985		This policy uses the same formula as layer2+3 but it
986		relies on skb_flow_dissect to obtain the header fields
987		which might result in the use of inner headers if an
988		encapsulation protocol is used. For example this will
989		improve the performance for tunnel users because the
990		packets will be distributed according to the encapsulated
991		flows.
992
993	encap3+4
994
995		This policy uses the same formula as layer3+4 but it
996		relies on skb_flow_dissect to obtain the header fields
997		which might result in the use of inner headers if an
998		encapsulation protocol is used. For example this will
999		improve the performance for tunnel users because the
1000		packets will be distributed according to the encapsulated
1001		flows.
1002
1003	vlan+srcmac
1004
1005		This policy uses a very rudimentary vlan ID and source mac
1006		hash to load-balance traffic per-vlan, with failover
1007		should one leg fail. The intended use case is for a bond
1008		shared by multiple virtual machines, all configured to
1009		use their own vlan, to give lacp-like functionality
1010		without requiring lacp-capable switching hardware.
1011
1012		The formula for the hash is simply
1013
1014		hash = (vlan ID) XOR (source MAC vendor) XOR (source MAC dev)
1015
1016	The default value is layer2.  This option was added in bonding
1017	version 2.6.3.  In earlier versions of bonding, this parameter
1018	does not exist, and the layer2 policy is the only policy.  The
1019	layer2+3 value was added for bonding version 3.2.2.
1020
1021resend_igmp
1022
1023	Specifies the number of IGMP membership reports to be issued after
1024	a failover event. One membership report is issued immediately after
1025	the failover, subsequent packets are sent in each 200ms interval.
1026
1027	The valid range is 0 - 255; the default value is 1. A value of 0
1028	prevents the IGMP membership report from being issued in response
1029	to the failover event.
1030
1031	This option is useful for bonding modes balance-rr (0), active-backup
1032	(1), balance-tlb (5) and balance-alb (6), in which a failover can
1033	switch the IGMP traffic from one slave to another.  Therefore a fresh
1034	IGMP report must be issued to cause the switch to forward the incoming
1035	IGMP traffic over the newly selected slave.
1036
1037	This option was added for bonding version 3.7.0.
1038
1039lp_interval
1040
1041	Specifies the number of seconds between instances where the bonding
1042	driver sends learning packets to each slaves peer switch.
1043
1044	The valid range is 1 - 0x7fffffff; the default value is 1. This Option
1045	has effect only in balance-tlb and balance-alb modes.
1046
10473. Configuring Bonding Devices
1048==============================
1049
1050You can configure bonding using either your distro's network
1051initialization scripts, or manually using either iproute2 or the
1052sysfs interface.  Distros generally use one of three packages for the
1053network initialization scripts: initscripts, sysconfig or interfaces.
1054Recent versions of these packages have support for bonding, while older
1055versions do not.
1056
1057We will first describe the options for configuring bonding for
1058distros using versions of initscripts, sysconfig and interfaces with full
1059or partial support for bonding, then provide information on enabling
1060bonding without support from the network initialization scripts (i.e.,
1061older versions of initscripts or sysconfig).
1062
1063If you're unsure whether your distro uses sysconfig,
1064initscripts or interfaces, or don't know if it's new enough, have no fear.
1065Determining this is fairly straightforward.
1066
1067First, look for a file called interfaces in /etc/network directory.
1068If this file is present in your system, then your system use interfaces. See
1069Configuration with Interfaces Support.
1070
1071Else, issue the command::
1072
1073	$ rpm -qf /sbin/ifup
1074
1075It will respond with a line of text starting with either
1076"initscripts" or "sysconfig," followed by some numbers.  This is the
1077package that provides your network initialization scripts.
1078
1079Next, to determine if your installation supports bonding,
1080issue the command::
1081
1082    $ grep ifenslave /sbin/ifup
1083
1084If this returns any matches, then your initscripts or
1085sysconfig has support for bonding.
1086
10873.1 Configuration with Sysconfig Support
1088----------------------------------------
1089
1090This section applies to distros using a version of sysconfig
1091with bonding support, for example, SuSE Linux Enterprise Server 9.
1092
1093SuSE SLES 9's networking configuration system does support
1094bonding, however, at this writing, the YaST system configuration
1095front end does not provide any means to work with bonding devices.
1096Bonding devices can be managed by hand, however, as follows.
1097
1098First, if they have not already been configured, configure the
1099slave devices.  On SLES 9, this is most easily done by running the
1100yast2 sysconfig configuration utility.  The goal is for to create an
1101ifcfg-id file for each slave device.  The simplest way to accomplish
1102this is to configure the devices for DHCP (this is only to get the
1103file ifcfg-id file created; see below for some issues with DHCP).  The
1104name of the configuration file for each device will be of the form::
1105
1106    ifcfg-id-xx:xx:xx:xx:xx:xx
1107
1108Where the "xx" portion will be replaced with the digits from
1109the device's permanent MAC address.
1110
1111Once the set of ifcfg-id-xx:xx:xx:xx:xx:xx files has been
1112created, it is necessary to edit the configuration files for the slave
1113devices (the MAC addresses correspond to those of the slave devices).
1114Before editing, the file will contain multiple lines, and will look
1115something like this::
1116
1117	BOOTPROTO='dhcp'
1118	STARTMODE='on'
1119	USERCTL='no'
1120	UNIQUE='XNzu.WeZGOGF+4wE'
1121	_nm_name='bus-pci-0001:61:01.0'
1122
1123Change the BOOTPROTO and STARTMODE lines to the following::
1124
1125	BOOTPROTO='none'
1126	STARTMODE='off'
1127
1128Do not alter the UNIQUE or _nm_name lines.  Remove any other
1129lines (USERCTL, etc).
1130
1131Once the ifcfg-id-xx:xx:xx:xx:xx:xx files have been modified,
1132it's time to create the configuration file for the bonding device
1133itself.  This file is named ifcfg-bondX, where X is the number of the
1134bonding device to create, starting at 0.  The first such file is
1135ifcfg-bond0, the second is ifcfg-bond1, and so on.  The sysconfig
1136network configuration system will correctly start multiple instances
1137of bonding.
1138
1139The contents of the ifcfg-bondX file is as follows::
1140
1141	BOOTPROTO="static"
1142	BROADCAST="10.0.2.255"
1143	IPADDR="10.0.2.10"
1144	NETMASK="255.255.0.0"
1145	NETWORK="10.0.2.0"
1146	REMOTE_IPADDR=""
1147	STARTMODE="onboot"
1148	BONDING_MASTER="yes"
1149	BONDING_MODULE_OPTS="mode=active-backup miimon=100"
1150	BONDING_SLAVE0="eth0"
1151	BONDING_SLAVE1="bus-pci-0000:06:08.1"
1152
1153Replace the sample BROADCAST, IPADDR, NETMASK and NETWORK
1154values with the appropriate values for your network.
1155
1156The STARTMODE specifies when the device is brought online.
1157The possible values are:
1158
1159	======== ======================================================
1160	onboot	 The device is started at boot time.  If you're not
1161		 sure, this is probably what you want.
1162
1163	manual	 The device is started only when ifup is called
1164		 manually.  Bonding devices may be configured this
1165		 way if you do not wish them to start automatically
1166		 at boot for some reason.
1167
1168	hotplug  The device is started by a hotplug event.  This is not
1169		 a valid choice for a bonding device.
1170
1171	off or   The device configuration is ignored.
1172	ignore
1173	======== ======================================================
1174
1175The line BONDING_MASTER='yes' indicates that the device is a
1176bonding master device.  The only useful value is "yes."
1177
1178The contents of BONDING_MODULE_OPTS are supplied to the
1179instance of the bonding module for this device.  Specify the options
1180for the bonding mode, link monitoring, and so on here.  Do not include
1181the max_bonds bonding parameter; this will confuse the configuration
1182system if you have multiple bonding devices.
1183
1184Finally, supply one BONDING_SLAVEn="slave device" for each
1185slave.  where "n" is an increasing value, one for each slave.  The
1186"slave device" is either an interface name, e.g., "eth0", or a device
1187specifier for the network device.  The interface name is easier to
1188find, but the ethN names are subject to change at boot time if, e.g.,
1189a device early in the sequence has failed.  The device specifiers
1190(bus-pci-0000:06:08.1 in the example above) specify the physical
1191network device, and will not change unless the device's bus location
1192changes (for example, it is moved from one PCI slot to another).  The
1193example above uses one of each type for demonstration purposes; most
1194configurations will choose one or the other for all slave devices.
1195
1196When all configuration files have been modified or created,
1197networking must be restarted for the configuration changes to take
1198effect.  This can be accomplished via the following::
1199
1200	# /etc/init.d/network restart
1201
1202Note that the network control script (/sbin/ifdown) will
1203remove the bonding module as part of the network shutdown processing,
1204so it is not necessary to remove the module by hand if, e.g., the
1205module parameters have changed.
1206
1207Also, at this writing, YaST/YaST2 will not manage bonding
1208devices (they do not show bonding interfaces on its list of network
1209devices).  It is necessary to edit the configuration file by hand to
1210change the bonding configuration.
1211
1212Additional general options and details of the ifcfg file
1213format can be found in an example ifcfg template file::
1214
1215	/etc/sysconfig/network/ifcfg.template
1216
1217Note that the template does not document the various ``BONDING_*``
1218settings described above, but does describe many of the other options.
1219
12203.1.1 Using DHCP with Sysconfig
1221-------------------------------
1222
1223Under sysconfig, configuring a device with BOOTPROTO='dhcp'
1224will cause it to query DHCP for its IP address information.  At this
1225writing, this does not function for bonding devices; the scripts
1226attempt to obtain the device address from DHCP prior to adding any of
1227the slave devices.  Without active slaves, the DHCP requests are not
1228sent to the network.
1229
12303.1.2 Configuring Multiple Bonds with Sysconfig
1231-----------------------------------------------
1232
1233The sysconfig network initialization system is capable of
1234handling multiple bonding devices.  All that is necessary is for each
1235bonding instance to have an appropriately configured ifcfg-bondX file
1236(as described above).  Do not specify the "max_bonds" parameter to any
1237instance of bonding, as this will confuse sysconfig.  If you require
1238multiple bonding devices with identical parameters, create multiple
1239ifcfg-bondX files.
1240
1241Because the sysconfig scripts supply the bonding module
1242options in the ifcfg-bondX file, it is not necessary to add them to
1243the system ``/etc/modules.d/*.conf`` configuration files.
1244
12453.2 Configuration with Initscripts Support
1246------------------------------------------
1247
1248This section applies to distros using a recent version of
1249initscripts with bonding support, for example, Red Hat Enterprise Linux
1250version 3 or later, Fedora, etc.  On these systems, the network
1251initialization scripts have knowledge of bonding, and can be configured to
1252control bonding devices.  Note that older versions of the initscripts
1253package have lower levels of support for bonding; this will be noted where
1254applicable.
1255
1256These distros will not automatically load the network adapter
1257driver unless the ethX device is configured with an IP address.
1258Because of this constraint, users must manually configure a
1259network-script file for all physical adapters that will be members of
1260a bondX link.  Network script files are located in the directory:
1261
1262/etc/sysconfig/network-scripts
1263
1264The file name must be prefixed with "ifcfg-eth" and suffixed
1265with the adapter's physical adapter number.  For example, the script
1266for eth0 would be named /etc/sysconfig/network-scripts/ifcfg-eth0.
1267Place the following text in the file::
1268
1269	DEVICE=eth0
1270	USERCTL=no
1271	ONBOOT=yes
1272	MASTER=bond0
1273	SLAVE=yes
1274	BOOTPROTO=none
1275
1276The DEVICE= line will be different for every ethX device and
1277must correspond with the name of the file, i.e., ifcfg-eth1 must have
1278a device line of DEVICE=eth1.  The setting of the MASTER= line will
1279also depend on the final bonding interface name chosen for your bond.
1280As with other network devices, these typically start at 0, and go up
1281one for each device, i.e., the first bonding instance is bond0, the
1282second is bond1, and so on.
1283
1284Next, create a bond network script.  The file name for this
1285script will be /etc/sysconfig/network-scripts/ifcfg-bondX where X is
1286the number of the bond.  For bond0 the file is named "ifcfg-bond0",
1287for bond1 it is named "ifcfg-bond1", and so on.  Within that file,
1288place the following text::
1289
1290	DEVICE=bond0
1291	IPADDR=192.168.1.1
1292	NETMASK=255.255.255.0
1293	NETWORK=192.168.1.0
1294	BROADCAST=192.168.1.255
1295	ONBOOT=yes
1296	BOOTPROTO=none
1297	USERCTL=no
1298
1299Be sure to change the networking specific lines (IPADDR,
1300NETMASK, NETWORK and BROADCAST) to match your network configuration.
1301
1302For later versions of initscripts, such as that found with Fedora
13037 (or later) and Red Hat Enterprise Linux version 5 (or later), it is possible,
1304and, indeed, preferable, to specify the bonding options in the ifcfg-bond0
1305file, e.g. a line of the format::
1306
1307  BONDING_OPTS="mode=active-backup arp_interval=60 arp_ip_target=192.168.1.254"
1308
1309will configure the bond with the specified options.  The options
1310specified in BONDING_OPTS are identical to the bonding module parameters
1311except for the arp_ip_target field when using versions of initscripts older
1312than and 8.57 (Fedora 8) and 8.45.19 (Red Hat Enterprise Linux 5.2).  When
1313using older versions each target should be included as a separate option and
1314should be preceded by a '+' to indicate it should be added to the list of
1315queried targets, e.g.,::
1316
1317    arp_ip_target=+192.168.1.1 arp_ip_target=+192.168.1.2
1318
1319is the proper syntax to specify multiple targets.  When specifying
1320options via BONDING_OPTS, it is not necessary to edit
1321``/etc/modprobe.d/*.conf``.
1322
1323For even older versions of initscripts that do not support
1324BONDING_OPTS, it is necessary to edit /etc/modprobe.d/*.conf, depending upon
1325your distro) to load the bonding module with your desired options when the
1326bond0 interface is brought up.  The following lines in /etc/modprobe.d/*.conf
1327will load the bonding module, and select its options:
1328
1329	alias bond0 bonding
1330	options bond0 mode=balance-alb miimon=100
1331
1332Replace the sample parameters with the appropriate set of
1333options for your configuration.
1334
1335Finally run "/etc/rc.d/init.d/network restart" as root.  This
1336will restart the networking subsystem and your bond link should be now
1337up and running.
1338
13393.2.1 Using DHCP with Initscripts
1340---------------------------------
1341
1342Recent versions of initscripts (the versions supplied with Fedora
1343Core 3 and Red Hat Enterprise Linux 4, or later versions, are reported to
1344work) have support for assigning IP information to bonding devices via
1345DHCP.
1346
1347To configure bonding for DHCP, configure it as described
1348above, except replace the line "BOOTPROTO=none" with "BOOTPROTO=dhcp"
1349and add a line consisting of "TYPE=Bonding".  Note that the TYPE value
1350is case sensitive.
1351
13523.2.2 Configuring Multiple Bonds with Initscripts
1353-------------------------------------------------
1354
1355Initscripts packages that are included with Fedora 7 and Red Hat
1356Enterprise Linux 5 support multiple bonding interfaces by simply
1357specifying the appropriate BONDING_OPTS= in ifcfg-bondX where X is the
1358number of the bond.  This support requires sysfs support in the kernel,
1359and a bonding driver of version 3.0.0 or later.  Other configurations may
1360not support this method for specifying multiple bonding interfaces; for
1361those instances, see the "Configuring Multiple Bonds Manually" section,
1362below.
1363
13643.3 Configuring Bonding Manually with iproute2
1365-----------------------------------------------
1366
1367This section applies to distros whose network initialization
1368scripts (the sysconfig or initscripts package) do not have specific
1369knowledge of bonding.  One such distro is SuSE Linux Enterprise Server
1370version 8.
1371
1372The general method for these systems is to place the bonding
1373module parameters into a config file in /etc/modprobe.d/ (as
1374appropriate for the installed distro), then add modprobe and/or
1375`ip link` commands to the system's global init script.  The name of
1376the global init script differs; for sysconfig, it is
1377/etc/init.d/boot.local and for initscripts it is /etc/rc.d/rc.local.
1378
1379For example, if you wanted to make a simple bond of two e100
1380devices (presumed to be eth0 and eth1), and have it persist across
1381reboots, edit the appropriate file (/etc/init.d/boot.local or
1382/etc/rc.d/rc.local), and add the following::
1383
1384	modprobe bonding mode=balance-alb miimon=100
1385	modprobe e100
1386	ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
1387	ip link set eth0 master bond0
1388	ip link set eth1 master bond0
1389
1390Replace the example bonding module parameters and bond0
1391network configuration (IP address, netmask, etc) with the appropriate
1392values for your configuration.
1393
1394Unfortunately, this method will not provide support for the
1395ifup and ifdown scripts on the bond devices.  To reload the bonding
1396configuration, it is necessary to run the initialization script, e.g.,::
1397
1398	# /etc/init.d/boot.local
1399
1400or::
1401
1402	# /etc/rc.d/rc.local
1403
1404It may be desirable in such a case to create a separate script
1405which only initializes the bonding configuration, then call that
1406separate script from within boot.local.  This allows for bonding to be
1407enabled without re-running the entire global init script.
1408
1409To shut down the bonding devices, it is necessary to first
1410mark the bonding device itself as being down, then remove the
1411appropriate device driver modules.  For our example above, you can do
1412the following::
1413
1414	# ifconfig bond0 down
1415	# rmmod bonding
1416	# rmmod e100
1417
1418Again, for convenience, it may be desirable to create a script
1419with these commands.
1420
1421
14223.3.1 Configuring Multiple Bonds Manually
1423-----------------------------------------
1424
1425This section contains information on configuring multiple
1426bonding devices with differing options for those systems whose network
1427initialization scripts lack support for configuring multiple bonds.
1428
1429If you require multiple bonding devices, but all with the same
1430options, you may wish to use the "max_bonds" module parameter,
1431documented above.
1432
1433To create multiple bonding devices with differing options, it is
1434preferable to use bonding parameters exported by sysfs, documented in the
1435section below.
1436
1437For versions of bonding without sysfs support, the only means to
1438provide multiple instances of bonding with differing options is to load
1439the bonding driver multiple times.  Note that current versions of the
1440sysconfig network initialization scripts handle this automatically; if
1441your distro uses these scripts, no special action is needed.  See the
1442section Configuring Bonding Devices, above, if you're not sure about your
1443network initialization scripts.
1444
1445To load multiple instances of the module, it is necessary to
1446specify a different name for each instance (the module loading system
1447requires that every loaded module, even multiple instances of the same
1448module, have a unique name).  This is accomplished by supplying multiple
1449sets of bonding options in ``/etc/modprobe.d/*.conf``, for example::
1450
1451	alias bond0 bonding
1452	options bond0 -o bond0 mode=balance-rr miimon=100
1453
1454	alias bond1 bonding
1455	options bond1 -o bond1 mode=balance-alb miimon=50
1456
1457will load the bonding module two times.  The first instance is
1458named "bond0" and creates the bond0 device in balance-rr mode with an
1459miimon of 100.  The second instance is named "bond1" and creates the
1460bond1 device in balance-alb mode with an miimon of 50.
1461
1462In some circumstances (typically with older distributions),
1463the above does not work, and the second bonding instance never sees
1464its options.  In that case, the second options line can be substituted
1465as follows::
1466
1467	install bond1 /sbin/modprobe --ignore-install bonding -o bond1 \
1468				     mode=balance-alb miimon=50
1469
1470This may be repeated any number of times, specifying a new and
1471unique name in place of bond1 for each subsequent instance.
1472
1473It has been observed that some Red Hat supplied kernels are unable
1474to rename modules at load time (the "-o bond1" part).  Attempts to pass
1475that option to modprobe will produce an "Operation not permitted" error.
1476This has been reported on some Fedora Core kernels, and has been seen on
1477RHEL 4 as well.  On kernels exhibiting this problem, it will be impossible
1478to configure multiple bonds with differing parameters (as they are older
1479kernels, and also lack sysfs support).
1480
14813.4 Configuring Bonding Manually via Sysfs
1482------------------------------------------
1483
1484Starting with version 3.0.0, Channel Bonding may be configured
1485via the sysfs interface.  This interface allows dynamic configuration
1486of all bonds in the system without unloading the module.  It also
1487allows for adding and removing bonds at runtime.  Ifenslave is no
1488longer required, though it is still supported.
1489
1490Use of the sysfs interface allows you to use multiple bonds
1491with different configurations without having to reload the module.
1492It also allows you to use multiple, differently configured bonds when
1493bonding is compiled into the kernel.
1494
1495You must have the sysfs filesystem mounted to configure
1496bonding this way.  The examples in this document assume that you
1497are using the standard mount point for sysfs, e.g. /sys.  If your
1498sysfs filesystem is mounted elsewhere, you will need to adjust the
1499example paths accordingly.
1500
1501Creating and Destroying Bonds
1502-----------------------------
1503To add a new bond foo::
1504
1505	# echo +foo > /sys/class/net/bonding_masters
1506
1507To remove an existing bond bar::
1508
1509	# echo -bar > /sys/class/net/bonding_masters
1510
1511To show all existing bonds::
1512
1513	# cat /sys/class/net/bonding_masters
1514
1515.. note::
1516
1517   due to 4K size limitation of sysfs files, this list may be
1518   truncated if you have more than a few hundred bonds.  This is unlikely
1519   to occur under normal operating conditions.
1520
1521Adding and Removing Slaves
1522--------------------------
1523Interfaces may be enslaved to a bond using the file
1524/sys/class/net/<bond>/bonding/slaves.  The semantics for this file
1525are the same as for the bonding_masters file.
1526
1527To enslave interface eth0 to bond bond0::
1528
1529	# ifconfig bond0 up
1530	# echo +eth0 > /sys/class/net/bond0/bonding/slaves
1531
1532To free slave eth0 from bond bond0::
1533
1534	# echo -eth0 > /sys/class/net/bond0/bonding/slaves
1535
1536When an interface is enslaved to a bond, symlinks between the
1537two are created in the sysfs filesystem.  In this case, you would get
1538/sys/class/net/bond0/slave_eth0 pointing to /sys/class/net/eth0, and
1539/sys/class/net/eth0/master pointing to /sys/class/net/bond0.
1540
1541This means that you can tell quickly whether or not an
1542interface is enslaved by looking for the master symlink.  Thus:
1543# echo -eth0 > /sys/class/net/eth0/master/bonding/slaves
1544will free eth0 from whatever bond it is enslaved to, regardless of
1545the name of the bond interface.
1546
1547Changing a Bond's Configuration
1548-------------------------------
1549Each bond may be configured individually by manipulating the
1550files located in /sys/class/net/<bond name>/bonding
1551
1552The names of these files correspond directly with the command-
1553line parameters described elsewhere in this file, and, with the
1554exception of arp_ip_target, they accept the same values.  To see the
1555current setting, simply cat the appropriate file.
1556
1557A few examples will be given here; for specific usage
1558guidelines for each parameter, see the appropriate section in this
1559document.
1560
1561To configure bond0 for balance-alb mode::
1562
1563	# ifconfig bond0 down
1564	# echo 6 > /sys/class/net/bond0/bonding/mode
1565	- or -
1566	# echo balance-alb > /sys/class/net/bond0/bonding/mode
1567
1568.. note::
1569
1570   The bond interface must be down before the mode can be changed.
1571
1572To enable MII monitoring on bond0 with a 1 second interval::
1573
1574	# echo 1000 > /sys/class/net/bond0/bonding/miimon
1575
1576.. note::
1577
1578   If ARP monitoring is enabled, it will disabled when MII
1579   monitoring is enabled, and vice-versa.
1580
1581To add ARP targets::
1582
1583	# echo +192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target
1584	# echo +192.168.0.101 > /sys/class/net/bond0/bonding/arp_ip_target
1585
1586.. note::
1587
1588   up to 16 target addresses may be specified.
1589
1590To remove an ARP target::
1591
1592	# echo -192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target
1593
1594To configure the interval between learning packet transmits::
1595
1596	# echo 12 > /sys/class/net/bond0/bonding/lp_interval
1597
1598.. note::
1599
1600   the lp_interval is the number of seconds between instances where
1601   the bonding driver sends learning packets to each slaves peer switch.  The
1602   default interval is 1 second.
1603
1604Example Configuration
1605---------------------
1606We begin with the same example that is shown in section 3.3,
1607executed with sysfs, and without using ifenslave.
1608
1609To make a simple bond of two e100 devices (presumed to be eth0
1610and eth1), and have it persist across reboots, edit the appropriate
1611file (/etc/init.d/boot.local or /etc/rc.d/rc.local), and add the
1612following::
1613
1614	modprobe bonding
1615	modprobe e100
1616	echo balance-alb > /sys/class/net/bond0/bonding/mode
1617	ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
1618	echo 100 > /sys/class/net/bond0/bonding/miimon
1619	echo +eth0 > /sys/class/net/bond0/bonding/slaves
1620	echo +eth1 > /sys/class/net/bond0/bonding/slaves
1621
1622To add a second bond, with two e1000 interfaces in
1623active-backup mode, using ARP monitoring, add the following lines to
1624your init script::
1625
1626	modprobe e1000
1627	echo +bond1 > /sys/class/net/bonding_masters
1628	echo active-backup > /sys/class/net/bond1/bonding/mode
1629	ifconfig bond1 192.168.2.1 netmask 255.255.255.0 up
1630	echo +192.168.2.100 /sys/class/net/bond1/bonding/arp_ip_target
1631	echo 2000 > /sys/class/net/bond1/bonding/arp_interval
1632	echo +eth2 > /sys/class/net/bond1/bonding/slaves
1633	echo +eth3 > /sys/class/net/bond1/bonding/slaves
1634
16353.5 Configuration with Interfaces Support
1636-----------------------------------------
1637
1638This section applies to distros which use /etc/network/interfaces file
1639to describe network interface configuration, most notably Debian and its
1640derivatives.
1641
1642The ifup and ifdown commands on Debian don't support bonding out of
1643the box. The ifenslave-2.6 package should be installed to provide bonding
1644support.  Once installed, this package will provide ``bond-*`` options
1645to be used into /etc/network/interfaces.
1646
1647Note that ifenslave-2.6 package will load the bonding module and use
1648the ifenslave command when appropriate.
1649
1650Example Configurations
1651----------------------
1652
1653In /etc/network/interfaces, the following stanza will configure bond0, in
1654active-backup mode, with eth0 and eth1 as slaves::
1655
1656	auto bond0
1657	iface bond0 inet dhcp
1658		bond-slaves eth0 eth1
1659		bond-mode active-backup
1660		bond-miimon 100
1661		bond-primary eth0 eth1
1662
1663If the above configuration doesn't work, you might have a system using
1664upstart for system startup. This is most notably true for recent
1665Ubuntu versions. The following stanza in /etc/network/interfaces will
1666produce the same result on those systems::
1667
1668	auto bond0
1669	iface bond0 inet dhcp
1670		bond-slaves none
1671		bond-mode active-backup
1672		bond-miimon 100
1673
1674	auto eth0
1675	iface eth0 inet manual
1676		bond-master bond0
1677		bond-primary eth0 eth1
1678
1679	auto eth1
1680	iface eth1 inet manual
1681		bond-master bond0
1682		bond-primary eth0 eth1
1683
1684For a full list of ``bond-*`` supported options in /etc/network/interfaces and
1685some more advanced examples tailored to you particular distros, see the files in
1686/usr/share/doc/ifenslave-2.6.
1687
16883.6 Overriding Configuration for Special Cases
1689----------------------------------------------
1690
1691When using the bonding driver, the physical port which transmits a frame is
1692typically selected by the bonding driver, and is not relevant to the user or
1693system administrator.  The output port is simply selected using the policies of
1694the selected bonding mode.  On occasion however, it is helpful to direct certain
1695classes of traffic to certain physical interfaces on output to implement
1696slightly more complex policies.  For example, to reach a web server over a
1697bonded interface in which eth0 connects to a private network, while eth1
1698connects via a public network, it may be desirous to bias the bond to send said
1699traffic over eth0 first, using eth1 only as a fall back, while all other traffic
1700can safely be sent over either interface.  Such configurations may be achieved
1701using the traffic control utilities inherent in linux.
1702
1703By default the bonding driver is multiqueue aware and 16 queues are created
1704when the driver initializes (see Documentation/networking/multiqueue.rst
1705for details).  If more or less queues are desired the module parameter
1706tx_queues can be used to change this value.  There is no sysfs parameter
1707available as the allocation is done at module init time.
1708
1709The output of the file /proc/net/bonding/bondX has changed so the output Queue
1710ID is now printed for each slave::
1711
1712	Bonding Mode: fault-tolerance (active-backup)
1713	Primary Slave: None
1714	Currently Active Slave: eth0
1715	MII Status: up
1716	MII Polling Interval (ms): 0
1717	Up Delay (ms): 0
1718	Down Delay (ms): 0
1719
1720	Slave Interface: eth0
1721	MII Status: up
1722	Link Failure Count: 0
1723	Permanent HW addr: 00:1a:a0:12:8f:cb
1724	Slave queue ID: 0
1725
1726	Slave Interface: eth1
1727	MII Status: up
1728	Link Failure Count: 0
1729	Permanent HW addr: 00:1a:a0:12:8f:cc
1730	Slave queue ID: 2
1731
1732The queue_id for a slave can be set using the command::
1733
1734	# echo "eth1:2" > /sys/class/net/bond0/bonding/queue_id
1735
1736Any interface that needs a queue_id set should set it with multiple calls
1737like the one above until proper priorities are set for all interfaces.  On
1738distributions that allow configuration via initscripts, multiple 'queue_id'
1739arguments can be added to BONDING_OPTS to set all needed slave queues.
1740
1741These queue id's can be used in conjunction with the tc utility to configure
1742a multiqueue qdisc and filters to bias certain traffic to transmit on certain
1743slave devices.  For instance, say we wanted, in the above configuration to
1744force all traffic bound to 192.168.1.100 to use eth1 in the bond as its output
1745device. The following commands would accomplish this::
1746
1747	# tc qdisc add dev bond0 handle 1 root multiq
1748
1749	# tc filter add dev bond0 protocol ip parent 1: prio 1 u32 match ip \
1750		dst 192.168.1.100 action skbedit queue_mapping 2
1751
1752These commands tell the kernel to attach a multiqueue queue discipline to the
1753bond0 interface and filter traffic enqueued to it, such that packets with a dst
1754ip of 192.168.1.100 have their output queue mapping value overwritten to 2.
1755This value is then passed into the driver, causing the normal output path
1756selection policy to be overridden, selecting instead qid 2, which maps to eth1.
1757
1758Note that qid values begin at 1.  Qid 0 is reserved to initiate to the driver
1759that normal output policy selection should take place.  One benefit to simply
1760leaving the qid for a slave to 0 is the multiqueue awareness in the bonding
1761driver that is now present.  This awareness allows tc filters to be placed on
1762slave devices as well as bond devices and the bonding driver will simply act as
1763a pass-through for selecting output queues on the slave device rather than
1764output port selection.
1765
1766This feature first appeared in bonding driver version 3.7.0 and support for
1767output slave selection was limited to round-robin and active-backup modes.
1768
17693.7 Configuring LACP for 802.3ad mode in a more secure way
1770----------------------------------------------------------
1771
1772When using 802.3ad bonding mode, the Actor (host) and Partner (switch)
1773exchange LACPDUs.  These LACPDUs cannot be sniffed, because they are
1774destined to link local mac addresses (which switches/bridges are not
1775supposed to forward).  However, most of the values are easily predictable
1776or are simply the machine's MAC address (which is trivially known to all
1777other hosts in the same L2).  This implies that other machines in the L2
1778domain can spoof LACPDU packets from other hosts to the switch and potentially
1779cause mayhem by joining (from the point of view of the switch) another
1780machine's aggregate, thus receiving a portion of that hosts incoming
1781traffic and / or spoofing traffic from that machine themselves (potentially
1782even successfully terminating some portion of flows). Though this is not
1783a likely scenario, one could avoid this possibility by simply configuring
1784few bonding parameters:
1785
1786   (a) ad_actor_system : You can set a random mac-address that can be used for
1787       these LACPDU exchanges. The value can not be either NULL or Multicast.
1788       Also it's preferable to set the local-admin bit. Following shell code
1789       generates a random mac-address as described above::
1790
1791	      # sys_mac_addr=$(printf '%02x:%02x:%02x:%02x:%02x:%02x' \
1792				       $(( (RANDOM & 0xFE) | 0x02 )) \
1793				       $(( RANDOM & 0xFF )) \
1794				       $(( RANDOM & 0xFF )) \
1795				       $(( RANDOM & 0xFF )) \
1796				       $(( RANDOM & 0xFF )) \
1797				       $(( RANDOM & 0xFF )))
1798	      # echo $sys_mac_addr > /sys/class/net/bond0/bonding/ad_actor_system
1799
1800   (b) ad_actor_sys_prio : Randomize the system priority. The default value
1801       is 65535, but system can take the value from 1 - 65535. Following shell
1802       code generates random priority and sets it::
1803
1804	    # sys_prio=$(( 1 + RANDOM + RANDOM ))
1805	    # echo $sys_prio > /sys/class/net/bond0/bonding/ad_actor_sys_prio
1806
1807   (c) ad_user_port_key : Use the user portion of the port-key. The default
1808       keeps this empty. These are the upper 10 bits of the port-key and value
1809       ranges from 0 - 1023. Following shell code generates these 10 bits and
1810       sets it::
1811
1812	    # usr_port_key=$(( RANDOM & 0x3FF ))
1813	    # echo $usr_port_key > /sys/class/net/bond0/bonding/ad_user_port_key
1814
1815
18164 Querying Bonding Configuration
1817=================================
1818
18194.1 Bonding Configuration
1820-------------------------
1821
1822Each bonding device has a read-only file residing in the
1823/proc/net/bonding directory.  The file contents include information
1824about the bonding configuration, options and state of each slave.
1825
1826For example, the contents of /proc/net/bonding/bond0 after the
1827driver is loaded with parameters of mode=0 and miimon=1000 is
1828generally as follows::
1829
1830	Ethernet Channel Bonding Driver: 2.6.1 (October 29, 2004)
1831	Bonding Mode: load balancing (round-robin)
1832	Currently Active Slave: eth0
1833	MII Status: up
1834	MII Polling Interval (ms): 1000
1835	Up Delay (ms): 0
1836	Down Delay (ms): 0
1837
1838	Slave Interface: eth1
1839	MII Status: up
1840	Link Failure Count: 1
1841
1842	Slave Interface: eth0
1843	MII Status: up
1844	Link Failure Count: 1
1845
1846The precise format and contents will change depending upon the
1847bonding configuration, state, and version of the bonding driver.
1848
18494.2 Network configuration
1850-------------------------
1851
1852The network configuration can be inspected using the ifconfig
1853command.  Bonding devices will have the MASTER flag set; Bonding slave
1854devices will have the SLAVE flag set.  The ifconfig output does not
1855contain information on which slaves are associated with which masters.
1856
1857In the example below, the bond0 interface is the master
1858(MASTER) while eth0 and eth1 are slaves (SLAVE). Notice all slaves of
1859bond0 have the same MAC address (HWaddr) as bond0 for all modes except
1860TLB and ALB that require a unique MAC address for each slave::
1861
1862  # /sbin/ifconfig
1863  bond0     Link encap:Ethernet  HWaddr 00:C0:F0:1F:37:B4
1864	    inet addr:XXX.XXX.XXX.YYY  Bcast:XXX.XXX.XXX.255  Mask:255.255.252.0
1865	    UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
1866	    RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0
1867	    TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0
1868	    collisions:0 txqueuelen:0
1869
1870  eth0      Link encap:Ethernet  HWaddr 00:C0:F0:1F:37:B4
1871	    UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
1872	    RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0
1873	    TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0
1874	    collisions:0 txqueuelen:100
1875	    Interrupt:10 Base address:0x1080
1876
1877  eth1      Link encap:Ethernet  HWaddr 00:C0:F0:1F:37:B4
1878	    UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
1879	    RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
1880	    TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
1881	    collisions:0 txqueuelen:100
1882	    Interrupt:9 Base address:0x1400
1883
18845. Switch Configuration
1885=======================
1886
1887For this section, "switch" refers to whatever system the
1888bonded devices are directly connected to (i.e., where the other end of
1889the cable plugs into).  This may be an actual dedicated switch device,
1890or it may be another regular system (e.g., another computer running
1891Linux),
1892
1893The active-backup, balance-tlb and balance-alb modes do not
1894require any specific configuration of the switch.
1895
1896The 802.3ad mode requires that the switch have the appropriate
1897ports configured as an 802.3ad aggregation.  The precise method used
1898to configure this varies from switch to switch, but, for example, a
1899Cisco 3550 series switch requires that the appropriate ports first be
1900grouped together in a single etherchannel instance, then that
1901etherchannel is set to mode "lacp" to enable 802.3ad (instead of
1902standard EtherChannel).
1903
1904The balance-rr, balance-xor and broadcast modes generally
1905require that the switch have the appropriate ports grouped together.
1906The nomenclature for such a group differs between switches, it may be
1907called an "etherchannel" (as in the Cisco example, above), a "trunk
1908group" or some other similar variation.  For these modes, each switch
1909will also have its own configuration options for the switch's transmit
1910policy to the bond.  Typical choices include XOR of either the MAC or
1911IP addresses.  The transmit policy of the two peers does not need to
1912match.  For these three modes, the bonding mode really selects a
1913transmit policy for an EtherChannel group; all three will interoperate
1914with another EtherChannel group.
1915
1916
19176. 802.1q VLAN Support
1918======================
1919
1920It is possible to configure VLAN devices over a bond interface
1921using the 8021q driver.  However, only packets coming from the 8021q
1922driver and passing through bonding will be tagged by default.  Self
1923generated packets, for example, bonding's learning packets or ARP
1924packets generated by either ALB mode or the ARP monitor mechanism, are
1925tagged internally by bonding itself.  As a result, bonding must
1926"learn" the VLAN IDs configured above it, and use those IDs to tag
1927self generated packets.
1928
1929For reasons of simplicity, and to support the use of adapters
1930that can do VLAN hardware acceleration offloading, the bonding
1931interface declares itself as fully hardware offloading capable, it gets
1932the add_vid/kill_vid notifications to gather the necessary
1933information, and it propagates those actions to the slaves.  In case
1934of mixed adapter types, hardware accelerated tagged packets that
1935should go through an adapter that is not offloading capable are
1936"un-accelerated" by the bonding driver so the VLAN tag sits in the
1937regular location.
1938
1939VLAN interfaces *must* be added on top of a bonding interface
1940only after enslaving at least one slave.  The bonding interface has a
1941hardware address of 00:00:00:00:00:00 until the first slave is added.
1942If the VLAN interface is created prior to the first enslavement, it
1943would pick up the all-zeroes hardware address.  Once the first slave
1944is attached to the bond, the bond device itself will pick up the
1945slave's hardware address, which is then available for the VLAN device.
1946
1947Also, be aware that a similar problem can occur if all slaves
1948are released from a bond that still has one or more VLAN interfaces on
1949top of it.  When a new slave is added, the bonding interface will
1950obtain its hardware address from the first slave, which might not
1951match the hardware address of the VLAN interfaces (which was
1952ultimately copied from an earlier slave).
1953
1954There are two methods to insure that the VLAN device operates
1955with the correct hardware address if all slaves are removed from a
1956bond interface:
1957
19581. Remove all VLAN interfaces then recreate them
1959
19602. Set the bonding interface's hardware address so that it
1961matches the hardware address of the VLAN interfaces.
1962
1963Note that changing a VLAN interface's HW address would set the
1964underlying device -- i.e. the bonding interface -- to promiscuous
1965mode, which might not be what you want.
1966
1967
19687. Link Monitoring
1969==================
1970
1971The bonding driver at present supports two schemes for
1972monitoring a slave device's link state: the ARP monitor and the MII
1973monitor.
1974
1975At the present time, due to implementation restrictions in the
1976bonding driver itself, it is not possible to enable both ARP and MII
1977monitoring simultaneously.
1978
19797.1 ARP Monitor Operation
1980-------------------------
1981
1982The ARP monitor operates as its name suggests: it sends ARP
1983queries to one or more designated peer systems on the network, and
1984uses the response as an indication that the link is operating.  This
1985gives some assurance that traffic is actually flowing to and from one
1986or more peers on the local network.
1987
19887.2 Configuring Multiple ARP Targets
1989------------------------------------
1990
1991While ARP monitoring can be done with just one target, it can
1992be useful in a High Availability setup to have several targets to
1993monitor.  In the case of just one target, the target itself may go
1994down or have a problem making it unresponsive to ARP requests.  Having
1995an additional target (or several) increases the reliability of the ARP
1996monitoring.
1997
1998Multiple ARP targets must be separated by commas as follows::
1999
2000 # example options for ARP monitoring with three targets
2001 alias bond0 bonding
2002 options bond0 arp_interval=60 arp_ip_target=192.168.0.1,192.168.0.3,192.168.0.9
2003
2004For just a single target the options would resemble::
2005
2006    # example options for ARP monitoring with one target
2007    alias bond0 bonding
2008    options bond0 arp_interval=60 arp_ip_target=192.168.0.100
2009
2010
20117.3 MII Monitor Operation
2012-------------------------
2013
2014The MII monitor monitors only the carrier state of the local
2015network interface.  It accomplishes this in one of three ways: by
2016depending upon the device driver to maintain its carrier state, by
2017querying the device's MII registers, or by making an ethtool query to
2018the device.
2019
2020If the use_carrier module parameter is 1 (the default value),
2021then the MII monitor will rely on the driver for carrier state
2022information (via the netif_carrier subsystem).  As explained in the
2023use_carrier parameter information, above, if the MII monitor fails to
2024detect carrier loss on the device (e.g., when the cable is physically
2025disconnected), it may be that the driver does not support
2026netif_carrier.
2027
2028If use_carrier is 0, then the MII monitor will first query the
2029device's (via ioctl) MII registers and check the link state.  If that
2030request fails (not just that it returns carrier down), then the MII
2031monitor will make an ethtool ETHTOOL_GLINK request to attempt to obtain
2032the same information.  If both methods fail (i.e., the driver either
2033does not support or had some error in processing both the MII register
2034and ethtool requests), then the MII monitor will assume the link is
2035up.
2036
20378. Potential Sources of Trouble
2038===============================
2039
20408.1 Adventures in Routing
2041-------------------------
2042
2043When bonding is configured, it is important that the slave
2044devices not have routes that supersede routes of the master (or,
2045generally, not have routes at all).  For example, suppose the bonding
2046device bond0 has two slaves, eth0 and eth1, and the routing table is
2047as follows::
2048
2049  Kernel IP routing table
2050  Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
2051  10.0.0.0        0.0.0.0         255.255.0.0     U        40 0          0 eth0
2052  10.0.0.0        0.0.0.0         255.255.0.0     U        40 0          0 eth1
2053  10.0.0.0        0.0.0.0         255.255.0.0     U        40 0          0 bond0
2054  127.0.0.0       0.0.0.0         255.0.0.0       U        40 0          0 lo
2055
2056This routing configuration will likely still update the
2057receive/transmit times in the driver (needed by the ARP monitor), but
2058may bypass the bonding driver (because outgoing traffic to, in this
2059case, another host on network 10 would use eth0 or eth1 before bond0).
2060
2061The ARP monitor (and ARP itself) may become confused by this
2062configuration, because ARP requests (generated by the ARP monitor)
2063will be sent on one interface (bond0), but the corresponding reply
2064will arrive on a different interface (eth0).  This reply looks to ARP
2065as an unsolicited ARP reply (because ARP matches replies on an
2066interface basis), and is discarded.  The MII monitor is not affected
2067by the state of the routing table.
2068
2069The solution here is simply to insure that slaves do not have
2070routes of their own, and if for some reason they must, those routes do
2071not supersede routes of their master.  This should generally be the
2072case, but unusual configurations or errant manual or automatic static
2073route additions may cause trouble.
2074
20758.2 Ethernet Device Renaming
2076----------------------------
2077
2078On systems with network configuration scripts that do not
2079associate physical devices directly with network interface names (so
2080that the same physical device always has the same "ethX" name), it may
2081be necessary to add some special logic to config files in
2082/etc/modprobe.d/.
2083
2084For example, given a modules.conf containing the following::
2085
2086	alias bond0 bonding
2087	options bond0 mode=some-mode miimon=50
2088	alias eth0 tg3
2089	alias eth1 tg3
2090	alias eth2 e1000
2091	alias eth3 e1000
2092
2093If neither eth0 and eth1 are slaves to bond0, then when the
2094bond0 interface comes up, the devices may end up reordered.  This
2095happens because bonding is loaded first, then its slave device's
2096drivers are loaded next.  Since no other drivers have been loaded,
2097when the e1000 driver loads, it will receive eth0 and eth1 for its
2098devices, but the bonding configuration tries to enslave eth2 and eth3
2099(which may later be assigned to the tg3 devices).
2100
2101Adding the following::
2102
2103	add above bonding e1000 tg3
2104
2105causes modprobe to load e1000 then tg3, in that order, when
2106bonding is loaded.  This command is fully documented in the
2107modules.conf manual page.
2108
2109On systems utilizing modprobe an equivalent problem can occur.
2110In this case, the following can be added to config files in
2111/etc/modprobe.d/ as::
2112
2113	softdep bonding pre: tg3 e1000
2114
2115This will load tg3 and e1000 modules before loading the bonding one.
2116Full documentation on this can be found in the modprobe.d and modprobe
2117manual pages.
2118
21198.3. Painfully Slow Or No Failed Link Detection By Miimon
2120---------------------------------------------------------
2121
2122By default, bonding enables the use_carrier option, which
2123instructs bonding to trust the driver to maintain carrier state.
2124
2125As discussed in the options section, above, some drivers do
2126not support the netif_carrier_on/_off link state tracking system.
2127With use_carrier enabled, bonding will always see these links as up,
2128regardless of their actual state.
2129
2130Additionally, other drivers do support netif_carrier, but do
2131not maintain it in real time, e.g., only polling the link state at
2132some fixed interval.  In this case, miimon will detect failures, but
2133only after some long period of time has expired.  If it appears that
2134miimon is very slow in detecting link failures, try specifying
2135use_carrier=0 to see if that improves the failure detection time.  If
2136it does, then it may be that the driver checks the carrier state at a
2137fixed interval, but does not cache the MII register values (so the
2138use_carrier=0 method of querying the registers directly works).  If
2139use_carrier=0 does not improve the failover, then the driver may cache
2140the registers, or the problem may be elsewhere.
2141
2142Also, remember that miimon only checks for the device's
2143carrier state.  It has no way to determine the state of devices on or
2144beyond other ports of a switch, or if a switch is refusing to pass
2145traffic while still maintaining carrier on.
2146
21479. SNMP agents
2148===============
2149
2150If running SNMP agents, the bonding driver should be loaded
2151before any network drivers participating in a bond.  This requirement
2152is due to the interface index (ipAdEntIfIndex) being associated to
2153the first interface found with a given IP address.  That is, there is
2154only one ipAdEntIfIndex for each IP address.  For example, if eth0 and
2155eth1 are slaves of bond0 and the driver for eth0 is loaded before the
2156bonding driver, the interface for the IP address will be associated
2157with the eth0 interface.  This configuration is shown below, the IP
2158address 192.168.1.1 has an interface index of 2 which indexes to eth0
2159in the ifDescr table (ifDescr.2).
2160
2161::
2162
2163     interfaces.ifTable.ifEntry.ifDescr.1 = lo
2164     interfaces.ifTable.ifEntry.ifDescr.2 = eth0
2165     interfaces.ifTable.ifEntry.ifDescr.3 = eth1
2166     interfaces.ifTable.ifEntry.ifDescr.4 = eth2
2167     interfaces.ifTable.ifEntry.ifDescr.5 = eth3
2168     interfaces.ifTable.ifEntry.ifDescr.6 = bond0
2169     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 5
2170     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
2171     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 4
2172     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
2173
2174This problem is avoided by loading the bonding driver before
2175any network drivers participating in a bond.  Below is an example of
2176loading the bonding driver first, the IP address 192.168.1.1 is
2177correctly associated with ifDescr.2.
2178
2179     interfaces.ifTable.ifEntry.ifDescr.1 = lo
2180     interfaces.ifTable.ifEntry.ifDescr.2 = bond0
2181     interfaces.ifTable.ifEntry.ifDescr.3 = eth0
2182     interfaces.ifTable.ifEntry.ifDescr.4 = eth1
2183     interfaces.ifTable.ifEntry.ifDescr.5 = eth2
2184     interfaces.ifTable.ifEntry.ifDescr.6 = eth3
2185     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 6
2186     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
2187     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 5
2188     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
2189
2190While some distributions may not report the interface name in
2191ifDescr, the association between the IP address and IfIndex remains
2192and SNMP functions such as Interface_Scan_Next will report that
2193association.
2194
219510. Promiscuous mode
2196====================
2197
2198When running network monitoring tools, e.g., tcpdump, it is
2199common to enable promiscuous mode on the device, so that all traffic
2200is seen (instead of seeing only traffic destined for the local host).
2201The bonding driver handles promiscuous mode changes to the bonding
2202master device (e.g., bond0), and propagates the setting to the slave
2203devices.
2204
2205For the balance-rr, balance-xor, broadcast, and 802.3ad modes,
2206the promiscuous mode setting is propagated to all slaves.
2207
2208For the active-backup, balance-tlb and balance-alb modes, the
2209promiscuous mode setting is propagated only to the active slave.
2210
2211For balance-tlb mode, the active slave is the slave currently
2212receiving inbound traffic.
2213
2214For balance-alb mode, the active slave is the slave used as a
2215"primary."  This slave is used for mode-specific control traffic, for
2216sending to peers that are unassigned or if the load is unbalanced.
2217
2218For the active-backup, balance-tlb and balance-alb modes, when
2219the active slave changes (e.g., due to a link failure), the
2220promiscuous setting will be propagated to the new active slave.
2221
222211. Configuring Bonding for High Availability
2223=============================================
2224
2225High Availability refers to configurations that provide
2226maximum network availability by having redundant or backup devices,
2227links or switches between the host and the rest of the world.  The
2228goal is to provide the maximum availability of network connectivity
2229(i.e., the network always works), even though other configurations
2230could provide higher throughput.
2231
223211.1 High Availability in a Single Switch Topology
2233--------------------------------------------------
2234
2235If two hosts (or a host and a single switch) are directly
2236connected via multiple physical links, then there is no availability
2237penalty to optimizing for maximum bandwidth.  In this case, there is
2238only one switch (or peer), so if it fails, there is no alternative
2239access to fail over to.  Additionally, the bonding load balance modes
2240support link monitoring of their members, so if individual links fail,
2241the load will be rebalanced across the remaining devices.
2242
2243See Section 12, "Configuring Bonding for Maximum Throughput"
2244for information on configuring bonding with one peer device.
2245
224611.2 High Availability in a Multiple Switch Topology
2247----------------------------------------------------
2248
2249With multiple switches, the configuration of bonding and the
2250network changes dramatically.  In multiple switch topologies, there is
2251a trade off between network availability and usable bandwidth.
2252
2253Below is a sample network, configured to maximize the
2254availability of the network::
2255
2256		|                                     |
2257		|port3                           port3|
2258	  +-----+----+                          +-----+----+
2259	  |          |port2       ISL      port2|          |
2260	  | switch A +--------------------------+ switch B |
2261	  |          |                          |          |
2262	  +-----+----+                          +-----++---+
2263		|port1                           port1|
2264		|             +-------+               |
2265		+-------------+ host1 +---------------+
2266			 eth0 +-------+ eth1
2267
2268In this configuration, there is a link between the two
2269switches (ISL, or inter switch link), and multiple ports connecting to
2270the outside world ("port3" on each switch).  There is no technical
2271reason that this could not be extended to a third switch.
2272
227311.2.1 HA Bonding Mode Selection for Multiple Switch Topology
2274-------------------------------------------------------------
2275
2276In a topology such as the example above, the active-backup and
2277broadcast modes are the only useful bonding modes when optimizing for
2278availability; the other modes require all links to terminate on the
2279same peer for them to behave rationally.
2280
2281active-backup:
2282	This is generally the preferred mode, particularly if
2283	the switches have an ISL and play together well.  If the
2284	network configuration is such that one switch is specifically
2285	a backup switch (e.g., has lower capacity, higher cost, etc),
2286	then the primary option can be used to insure that the
2287	preferred link is always used when it is available.
2288
2289broadcast:
2290	This mode is really a special purpose mode, and is suitable
2291	only for very specific needs.  For example, if the two
2292	switches are not connected (no ISL), and the networks beyond
2293	them are totally independent.  In this case, if it is
2294	necessary for some specific one-way traffic to reach both
2295	independent networks, then the broadcast mode may be suitable.
2296
229711.2.2 HA Link Monitoring Selection for Multiple Switch Topology
2298----------------------------------------------------------------
2299
2300The choice of link monitoring ultimately depends upon your
2301switch.  If the switch can reliably fail ports in response to other
2302failures, then either the MII or ARP monitors should work.  For
2303example, in the above example, if the "port3" link fails at the remote
2304end, the MII monitor has no direct means to detect this.  The ARP
2305monitor could be configured with a target at the remote end of port3,
2306thus detecting that failure without switch support.
2307
2308In general, however, in a multiple switch topology, the ARP
2309monitor can provide a higher level of reliability in detecting end to
2310end connectivity failures (which may be caused by the failure of any
2311individual component to pass traffic for any reason).  Additionally,
2312the ARP monitor should be configured with multiple targets (at least
2313one for each switch in the network).  This will insure that,
2314regardless of which switch is active, the ARP monitor has a suitable
2315target to query.
2316
2317Note, also, that of late many switches now support a functionality
2318generally referred to as "trunk failover."  This is a feature of the
2319switch that causes the link state of a particular switch port to be set
2320down (or up) when the state of another switch port goes down (or up).
2321Its purpose is to propagate link failures from logically "exterior" ports
2322to the logically "interior" ports that bonding is able to monitor via
2323miimon.  Availability and configuration for trunk failover varies by
2324switch, but this can be a viable alternative to the ARP monitor when using
2325suitable switches.
2326
232712. Configuring Bonding for Maximum Throughput
2328==============================================
2329
233012.1 Maximizing Throughput in a Single Switch Topology
2331------------------------------------------------------
2332
2333In a single switch configuration, the best method to maximize
2334throughput depends upon the application and network environment.  The
2335various load balancing modes each have strengths and weaknesses in
2336different environments, as detailed below.
2337
2338For this discussion, we will break down the topologies into
2339two categories.  Depending upon the destination of most traffic, we
2340categorize them into either "gatewayed" or "local" configurations.
2341
2342In a gatewayed configuration, the "switch" is acting primarily
2343as a router, and the majority of traffic passes through this router to
2344other networks.  An example would be the following::
2345
2346
2347     +----------+                     +----------+
2348     |          |eth0            port1|          | to other networks
2349     | Host A   +---------------------+ router   +------------------->
2350     |          +---------------------+          | Hosts B and C are out
2351     |          |eth1            port2|          | here somewhere
2352     +----------+                     +----------+
2353
2354The router may be a dedicated router device, or another host
2355acting as a gateway.  For our discussion, the important point is that
2356the majority of traffic from Host A will pass through the router to
2357some other network before reaching its final destination.
2358
2359In a gatewayed network configuration, although Host A may
2360communicate with many other systems, all of its traffic will be sent
2361and received via one other peer on the local network, the router.
2362
2363Note that the case of two systems connected directly via
2364multiple physical links is, for purposes of configuring bonding, the
2365same as a gatewayed configuration.  In that case, it happens that all
2366traffic is destined for the "gateway" itself, not some other network
2367beyond the gateway.
2368
2369In a local configuration, the "switch" is acting primarily as
2370a switch, and the majority of traffic passes through this switch to
2371reach other stations on the same network.  An example would be the
2372following::
2373
2374    +----------+            +----------+       +--------+
2375    |          |eth0   port1|          +-------+ Host B |
2376    |  Host A  +------------+  switch  |port3  +--------+
2377    |          +------------+          |                  +--------+
2378    |          |eth1   port2|          +------------------+ Host C |
2379    +----------+            +----------+port4             +--------+
2380
2381
2382Again, the switch may be a dedicated switch device, or another
2383host acting as a gateway.  For our discussion, the important point is
2384that the majority of traffic from Host A is destined for other hosts
2385on the same local network (Hosts B and C in the above example).
2386
2387In summary, in a gatewayed configuration, traffic to and from
2388the bonded device will be to the same MAC level peer on the network
2389(the gateway itself, i.e., the router), regardless of its final
2390destination.  In a local configuration, traffic flows directly to and
2391from the final destinations, thus, each destination (Host B, Host C)
2392will be addressed directly by their individual MAC addresses.
2393
2394This distinction between a gatewayed and a local network
2395configuration is important because many of the load balancing modes
2396available use the MAC addresses of the local network source and
2397destination to make load balancing decisions.  The behavior of each
2398mode is described below.
2399
2400
240112.1.1 MT Bonding Mode Selection for Single Switch Topology
2402-----------------------------------------------------------
2403
2404This configuration is the easiest to set up and to understand,
2405although you will have to decide which bonding mode best suits your
2406needs.  The trade offs for each mode are detailed below:
2407
2408balance-rr:
2409	This mode is the only mode that will permit a single
2410	TCP/IP connection to stripe traffic across multiple
2411	interfaces. It is therefore the only mode that will allow a
2412	single TCP/IP stream to utilize more than one interface's
2413	worth of throughput.  This comes at a cost, however: the
2414	striping generally results in peer systems receiving packets out
2415	of order, causing TCP/IP's congestion control system to kick
2416	in, often by retransmitting segments.
2417
2418	It is possible to adjust TCP/IP's congestion limits by
2419	altering the net.ipv4.tcp_reordering sysctl parameter.  The
2420	usual default value is 3. But keep in mind TCP stack is able
2421	to automatically increase this when it detects reorders.
2422
2423	Note that the fraction of packets that will be delivered out of
2424	order is highly variable, and is unlikely to be zero.  The level
2425	of reordering depends upon a variety of factors, including the
2426	networking interfaces, the switch, and the topology of the
2427	configuration.  Speaking in general terms, higher speed network
2428	cards produce more reordering (due to factors such as packet
2429	coalescing), and a "many to many" topology will reorder at a
2430	higher rate than a "many slow to one fast" configuration.
2431
2432	Many switches do not support any modes that stripe traffic
2433	(instead choosing a port based upon IP or MAC level addresses);
2434	for those devices, traffic for a particular connection flowing
2435	through the switch to a balance-rr bond will not utilize greater
2436	than one interface's worth of bandwidth.
2437
2438	If you are utilizing protocols other than TCP/IP, UDP for
2439	example, and your application can tolerate out of order
2440	delivery, then this mode can allow for single stream datagram
2441	performance that scales near linearly as interfaces are added
2442	to the bond.
2443
2444	This mode requires the switch to have the appropriate ports
2445	configured for "etherchannel" or "trunking."
2446
2447active-backup:
2448	There is not much advantage in this network topology to
2449	the active-backup mode, as the inactive backup devices are all
2450	connected to the same peer as the primary.  In this case, a
2451	load balancing mode (with link monitoring) will provide the
2452	same level of network availability, but with increased
2453	available bandwidth.  On the plus side, active-backup mode
2454	does not require any configuration of the switch, so it may
2455	have value if the hardware available does not support any of
2456	the load balance modes.
2457
2458balance-xor:
2459	This mode will limit traffic such that packets destined
2460	for specific peers will always be sent over the same
2461	interface.  Since the destination is determined by the MAC
2462	addresses involved, this mode works best in a "local" network
2463	configuration (as described above), with destinations all on
2464	the same local network.  This mode is likely to be suboptimal
2465	if all your traffic is passed through a single router (i.e., a
2466	"gatewayed" network configuration, as described above).
2467
2468	As with balance-rr, the switch ports need to be configured for
2469	"etherchannel" or "trunking."
2470
2471broadcast:
2472	Like active-backup, there is not much advantage to this
2473	mode in this type of network topology.
2474
2475802.3ad:
2476	This mode can be a good choice for this type of network
2477	topology.  The 802.3ad mode is an IEEE standard, so all peers
2478	that implement 802.3ad should interoperate well.  The 802.3ad
2479	protocol includes automatic configuration of the aggregates,
2480	so minimal manual configuration of the switch is needed
2481	(typically only to designate that some set of devices is
2482	available for 802.3ad).  The 802.3ad standard also mandates
2483	that frames be delivered in order (within certain limits), so
2484	in general single connections will not see misordering of
2485	packets.  The 802.3ad mode does have some drawbacks: the
2486	standard mandates that all devices in the aggregate operate at
2487	the same speed and duplex.  Also, as with all bonding load
2488	balance modes other than balance-rr, no single connection will
2489	be able to utilize more than a single interface's worth of
2490	bandwidth.
2491
2492	Additionally, the linux bonding 802.3ad implementation
2493	distributes traffic by peer (using an XOR of MAC addresses
2494	and packet type ID), so in a "gatewayed" configuration, all
2495	outgoing traffic will generally use the same device.  Incoming
2496	traffic may also end up on a single device, but that is
2497	dependent upon the balancing policy of the peer's 802.3ad
2498	implementation.  In a "local" configuration, traffic will be
2499	distributed across the devices in the bond.
2500
2501	Finally, the 802.3ad mode mandates the use of the MII monitor,
2502	therefore, the ARP monitor is not available in this mode.
2503
2504balance-tlb:
2505	The balance-tlb mode balances outgoing traffic by peer.
2506	Since the balancing is done according to MAC address, in a
2507	"gatewayed" configuration (as described above), this mode will
2508	send all traffic across a single device.  However, in a
2509	"local" network configuration, this mode balances multiple
2510	local network peers across devices in a vaguely intelligent
2511	manner (not a simple XOR as in balance-xor or 802.3ad mode),
2512	so that mathematically unlucky MAC addresses (i.e., ones that
2513	XOR to the same value) will not all "bunch up" on a single
2514	interface.
2515
2516	Unlike 802.3ad, interfaces may be of differing speeds, and no
2517	special switch configuration is required.  On the down side,
2518	in this mode all incoming traffic arrives over a single
2519	interface, this mode requires certain ethtool support in the
2520	network device driver of the slave interfaces, and the ARP
2521	monitor is not available.
2522
2523balance-alb:
2524	This mode is everything that balance-tlb is, and more.
2525	It has all of the features (and restrictions) of balance-tlb,
2526	and will also balance incoming traffic from local network
2527	peers (as described in the Bonding Module Options section,
2528	above).
2529
2530	The only additional down side to this mode is that the network
2531	device driver must support changing the hardware address while
2532	the device is open.
2533
253412.1.2 MT Link Monitoring for Single Switch Topology
2535----------------------------------------------------
2536
2537The choice of link monitoring may largely depend upon which
2538mode you choose to use.  The more advanced load balancing modes do not
2539support the use of the ARP monitor, and are thus restricted to using
2540the MII monitor (which does not provide as high a level of end to end
2541assurance as the ARP monitor).
2542
254312.2 Maximum Throughput in a Multiple Switch Topology
2544-----------------------------------------------------
2545
2546Multiple switches may be utilized to optimize for throughput
2547when they are configured in parallel as part of an isolated network
2548between two or more systems, for example::
2549
2550		       +-----------+
2551		       |  Host A   |
2552		       +-+---+---+-+
2553			 |   |   |
2554		+--------+   |   +---------+
2555		|            |             |
2556	 +------+---+  +-----+----+  +-----+----+
2557	 | Switch A |  | Switch B |  | Switch C |
2558	 +------+---+  +-----+----+  +-----+----+
2559		|            |             |
2560		+--------+   |   +---------+
2561			 |   |   |
2562		       +-+---+---+-+
2563		       |  Host B   |
2564		       +-----------+
2565
2566In this configuration, the switches are isolated from one
2567another.  One reason to employ a topology such as this is for an
2568isolated network with many hosts (a cluster configured for high
2569performance, for example), using multiple smaller switches can be more
2570cost effective than a single larger switch, e.g., on a network with 24
2571hosts, three 24 port switches can be significantly less expensive than
2572a single 72 port switch.
2573
2574If access beyond the network is required, an individual host
2575can be equipped with an additional network device connected to an
2576external network; this host then additionally acts as a gateway.
2577
257812.2.1 MT Bonding Mode Selection for Multiple Switch Topology
2579-------------------------------------------------------------
2580
2581In actual practice, the bonding mode typically employed in
2582configurations of this type is balance-rr.  Historically, in this
2583network configuration, the usual caveats about out of order packet
2584delivery are mitigated by the use of network adapters that do not do
2585any kind of packet coalescing (via the use of NAPI, or because the
2586device itself does not generate interrupts until some number of
2587packets has arrived).  When employed in this fashion, the balance-rr
2588mode allows individual connections between two hosts to effectively
2589utilize greater than one interface's bandwidth.
2590
259112.2.2 MT Link Monitoring for Multiple Switch Topology
2592------------------------------------------------------
2593
2594Again, in actual practice, the MII monitor is most often used
2595in this configuration, as performance is given preference over
2596availability.  The ARP monitor will function in this topology, but its
2597advantages over the MII monitor are mitigated by the volume of probes
2598needed as the number of systems involved grows (remember that each
2599host in the network is configured with bonding).
2600
260113. Switch Behavior Issues
2602==========================
2603
260413.1 Link Establishment and Failover Delays
2605-------------------------------------------
2606
2607Some switches exhibit undesirable behavior with regard to the
2608timing of link up and down reporting by the switch.
2609
2610First, when a link comes up, some switches may indicate that
2611the link is up (carrier available), but not pass traffic over the
2612interface for some period of time.  This delay is typically due to
2613some type of autonegotiation or routing protocol, but may also occur
2614during switch initialization (e.g., during recovery after a switch
2615failure).  If you find this to be a problem, specify an appropriate
2616value to the updelay bonding module option to delay the use of the
2617relevant interface(s).
2618
2619Second, some switches may "bounce" the link state one or more
2620times while a link is changing state.  This occurs most commonly while
2621the switch is initializing.  Again, an appropriate updelay value may
2622help.
2623
2624Note that when a bonding interface has no active links, the
2625driver will immediately reuse the first link that goes up, even if the
2626updelay parameter has been specified (the updelay is ignored in this
2627case).  If there are slave interfaces waiting for the updelay timeout
2628to expire, the interface that first went into that state will be
2629immediately reused.  This reduces down time of the network if the
2630value of updelay has been overestimated, and since this occurs only in
2631cases with no connectivity, there is no additional penalty for
2632ignoring the updelay.
2633
2634In addition to the concerns about switch timings, if your
2635switches take a long time to go into backup mode, it may be desirable
2636to not activate a backup interface immediately after a link goes down.
2637Failover may be delayed via the downdelay bonding module option.
2638
263913.2 Duplicated Incoming Packets
2640--------------------------------
2641
2642NOTE: Starting with version 3.0.2, the bonding driver has logic to
2643suppress duplicate packets, which should largely eliminate this problem.
2644The following description is kept for reference.
2645
2646It is not uncommon to observe a short burst of duplicated
2647traffic when the bonding device is first used, or after it has been
2648idle for some period of time.  This is most easily observed by issuing
2649a "ping" to some other host on the network, and noticing that the
2650output from ping flags duplicates (typically one per slave).
2651
2652For example, on a bond in active-backup mode with five slaves
2653all connected to one switch, the output may appear as follows::
2654
2655	# ping -n 10.0.4.2
2656	PING 10.0.4.2 (10.0.4.2) from 10.0.3.10 : 56(84) bytes of data.
2657	64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.7 ms
2658	64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
2659	64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
2660	64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
2661	64 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!)
2662	64 bytes from 10.0.4.2: icmp_seq=2 ttl=64 time=0.216 ms
2663	64 bytes from 10.0.4.2: icmp_seq=3 ttl=64 time=0.267 ms
2664	64 bytes from 10.0.4.2: icmp_seq=4 ttl=64 time=0.222 ms
2665
2666This is not due to an error in the bonding driver, rather, it
2667is a side effect of how many switches update their MAC forwarding
2668tables.  Initially, the switch does not associate the MAC address in
2669the packet with a particular switch port, and so it may send the
2670traffic to all ports until its MAC forwarding table is updated.  Since
2671the interfaces attached to the bond may occupy multiple ports on a
2672single switch, when the switch (temporarily) floods the traffic to all
2673ports, the bond device receives multiple copies of the same packet
2674(one per slave device).
2675
2676The duplicated packet behavior is switch dependent, some
2677switches exhibit this, and some do not.  On switches that display this
2678behavior, it can be induced by clearing the MAC forwarding table (on
2679most Cisco switches, the privileged command "clear mac address-table
2680dynamic" will accomplish this).
2681
268214. Hardware Specific Considerations
2683====================================
2684
2685This section contains additional information for configuring
2686bonding on specific hardware platforms, or for interfacing bonding
2687with particular switches or other devices.
2688
268914.1 IBM BladeCenter
2690--------------------
2691
2692This applies to the JS20 and similar systems.
2693
2694On the JS20 blades, the bonding driver supports only
2695balance-rr, active-backup, balance-tlb and balance-alb modes.  This is
2696largely due to the network topology inside the BladeCenter, detailed
2697below.
2698
2699JS20 network adapter information
2700--------------------------------
2701
2702All JS20s come with two Broadcom Gigabit Ethernet ports
2703integrated on the planar (that's "motherboard" in IBM-speak).  In the
2704BladeCenter chassis, the eth0 port of all JS20 blades is hard wired to
2705I/O Module #1; similarly, all eth1 ports are wired to I/O Module #2.
2706An add-on Broadcom daughter card can be installed on a JS20 to provide
2707two more Gigabit Ethernet ports.  These ports, eth2 and eth3, are
2708wired to I/O Modules 3 and 4, respectively.
2709
2710Each I/O Module may contain either a switch or a passthrough
2711module (which allows ports to be directly connected to an external
2712switch).  Some bonding modes require a specific BladeCenter internal
2713network topology in order to function; these are detailed below.
2714
2715Additional BladeCenter-specific networking information can be
2716found in two IBM Redbooks (www.ibm.com/redbooks):
2717
2718- "IBM eServer BladeCenter Networking Options"
2719- "IBM eServer BladeCenter Layer 2-7 Network Switching"
2720
2721BladeCenter networking configuration
2722------------------------------------
2723
2724Because a BladeCenter can be configured in a very large number
2725of ways, this discussion will be confined to describing basic
2726configurations.
2727
2728Normally, Ethernet Switch Modules (ESMs) are used in I/O
2729modules 1 and 2.  In this configuration, the eth0 and eth1 ports of a
2730JS20 will be connected to different internal switches (in the
2731respective I/O modules).
2732
2733A passthrough module (OPM or CPM, optical or copper,
2734passthrough module) connects the I/O module directly to an external
2735switch.  By using PMs in I/O module #1 and #2, the eth0 and eth1
2736interfaces of a JS20 can be redirected to the outside world and
2737connected to a common external switch.
2738
2739Depending upon the mix of ESMs and PMs, the network will
2740appear to bonding as either a single switch topology (all PMs) or as a
2741multiple switch topology (one or more ESMs, zero or more PMs).  It is
2742also possible to connect ESMs together, resulting in a configuration
2743much like the example in "High Availability in a Multiple Switch
2744Topology," above.
2745
2746Requirements for specific modes
2747-------------------------------
2748
2749The balance-rr mode requires the use of passthrough modules
2750for devices in the bond, all connected to an common external switch.
2751That switch must be configured for "etherchannel" or "trunking" on the
2752appropriate ports, as is usual for balance-rr.
2753
2754The balance-alb and balance-tlb modes will function with
2755either switch modules or passthrough modules (or a mix).  The only
2756specific requirement for these modes is that all network interfaces
2757must be able to reach all destinations for traffic sent over the
2758bonding device (i.e., the network must converge at some point outside
2759the BladeCenter).
2760
2761The active-backup mode has no additional requirements.
2762
2763Link monitoring issues
2764----------------------
2765
2766When an Ethernet Switch Module is in place, only the ARP
2767monitor will reliably detect link loss to an external switch.  This is
2768nothing unusual, but examination of the BladeCenter cabinet would
2769suggest that the "external" network ports are the ethernet ports for
2770the system, when it fact there is a switch between these "external"
2771ports and the devices on the JS20 system itself.  The MII monitor is
2772only able to detect link failures between the ESM and the JS20 system.
2773
2774When a passthrough module is in place, the MII monitor does
2775detect failures to the "external" port, which is then directly
2776connected to the JS20 system.
2777
2778Other concerns
2779--------------
2780
2781The Serial Over LAN (SoL) link is established over the primary
2782ethernet (eth0) only, therefore, any loss of link to eth0 will result
2783in losing your SoL connection.  It will not fail over with other
2784network traffic, as the SoL system is beyond the control of the
2785bonding driver.
2786
2787It may be desirable to disable spanning tree on the switch
2788(either the internal Ethernet Switch Module, or an external switch) to
2789avoid fail-over delay issues when using bonding.
2790
2791
279215. Frequently Asked Questions
2793==============================
2794
27951.  Is it SMP safe?
2796-------------------
2797
2798Yes. The old 2.0.xx channel bonding patch was not SMP safe.
2799The new driver was designed to be SMP safe from the start.
2800
28012.  What type of cards will work with it?
2802-----------------------------------------
2803
2804Any Ethernet type cards (you can even mix cards - a Intel
2805EtherExpress PRO/100 and a 3com 3c905b, for example).  For most modes,
2806devices need not be of the same speed.
2807
2808Starting with version 3.2.1, bonding also supports Infiniband
2809slaves in active-backup mode.
2810
28113.  How many bonding devices can I have?
2812----------------------------------------
2813
2814There is no limit.
2815
28164.  How many slaves can a bonding device have?
2817----------------------------------------------
2818
2819This is limited only by the number of network interfaces Linux
2820supports and/or the number of network cards you can place in your
2821system.
2822
28235.  What happens when a slave link dies?
2824----------------------------------------
2825
2826If link monitoring is enabled, then the failing device will be
2827disabled.  The active-backup mode will fail over to a backup link, and
2828other modes will ignore the failed link.  The link will continue to be
2829monitored, and should it recover, it will rejoin the bond (in whatever
2830manner is appropriate for the mode). See the sections on High
2831Availability and the documentation for each mode for additional
2832information.
2833
2834Link monitoring can be enabled via either the miimon or
2835arp_interval parameters (described in the module parameters section,
2836above).  In general, miimon monitors the carrier state as sensed by
2837the underlying network device, and the arp monitor (arp_interval)
2838monitors connectivity to another host on the local network.
2839
2840If no link monitoring is configured, the bonding driver will
2841be unable to detect link failures, and will assume that all links are
2842always available.  This will likely result in lost packets, and a
2843resulting degradation of performance.  The precise performance loss
2844depends upon the bonding mode and network configuration.
2845
28466.  Can bonding be used for High Availability?
2847----------------------------------------------
2848
2849Yes.  See the section on High Availability for details.
2850
28517.  Which switches/systems does it work with?
2852---------------------------------------------
2853
2854The full answer to this depends upon the desired mode.
2855
2856In the basic balance modes (balance-rr and balance-xor), it
2857works with any system that supports etherchannel (also called
2858trunking).  Most managed switches currently available have such
2859support, and many unmanaged switches as well.
2860
2861The advanced balance modes (balance-tlb and balance-alb) do
2862not have special switch requirements, but do need device drivers that
2863support specific features (described in the appropriate section under
2864module parameters, above).
2865
2866In 802.3ad mode, it works with systems that support IEEE
2867802.3ad Dynamic Link Aggregation.  Most managed and many unmanaged
2868switches currently available support 802.3ad.
2869
2870The active-backup mode should work with any Layer-II switch.
2871
28728.  Where does a bonding device get its MAC address from?
2873---------------------------------------------------------
2874
2875When using slave devices that have fixed MAC addresses, or when
2876the fail_over_mac option is enabled, the bonding device's MAC address is
2877the MAC address of the active slave.
2878
2879For other configurations, if not explicitly configured (with
2880ifconfig or ip link), the MAC address of the bonding device is taken from
2881its first slave device.  This MAC address is then passed to all following
2882slaves and remains persistent (even if the first slave is removed) until
2883the bonding device is brought down or reconfigured.
2884
2885If you wish to change the MAC address, you can set it with
2886ifconfig or ip link::
2887
2888	# ifconfig bond0 hw ether 00:11:22:33:44:55
2889
2890	# ip link set bond0 address 66:77:88:99:aa:bb
2891
2892The MAC address can be also changed by bringing down/up the
2893device and then changing its slaves (or their order)::
2894
2895	# ifconfig bond0 down ; modprobe -r bonding
2896	# ifconfig bond0 .... up
2897	# ifenslave bond0 eth...
2898
2899This method will automatically take the address from the next
2900slave that is added.
2901
2902To restore your slaves' MAC addresses, you need to detach them
2903from the bond (``ifenslave -d bond0 eth0``). The bonding driver will
2904then restore the MAC addresses that the slaves had before they were
2905enslaved.
2906
290716. Resources and Links
2908=======================
2909
2910The latest version of the bonding driver can be found in the latest
2911version of the linux kernel, found on http://kernel.org
2912
2913The latest version of this document can be found in the latest kernel
2914source (named Documentation/networking/bonding.rst).
2915
2916Discussions regarding the development of the bonding driver take place
2917on the main Linux network mailing list, hosted at vger.kernel.org. The list
2918address is:
2919
2920netdev@vger.kernel.org
2921
2922The administrative interface (to subscribe or unsubscribe) can
2923be found at:
2924
2925http://vger.kernel.org/vger-lists.html#netdev
2926