1 2 Linux Ethernet Bonding Driver HOWTO 3 4 Latest update: 27 April 2011 5 6Initial release : Thomas Davis <tadavis at lbl.gov> 7Corrections, HA extensions : 2000/10/03-15 : 8 - Willy Tarreau <willy at meta-x.org> 9 - Constantine Gavrilov <const-g at xpert.com> 10 - Chad N. Tindel <ctindel at ieee dot org> 11 - Janice Girouard <girouard at us dot ibm dot com> 12 - Jay Vosburgh <fubar at us dot ibm dot com> 13 14Reorganized and updated Feb 2005 by Jay Vosburgh 15Added Sysfs information: 2006/04/24 16 - Mitch Williams <mitch.a.williams at intel.com> 17 18Introduction 19============ 20 21 The Linux bonding driver provides a method for aggregating 22multiple network interfaces into a single logical "bonded" interface. 23The behavior of the bonded interfaces depends upon the mode; generally 24speaking, modes provide either hot standby or load balancing services. 25Additionally, link integrity monitoring may be performed. 26 27 The bonding driver originally came from Donald Becker's 28beowulf patches for kernel 2.0. It has changed quite a bit since, and 29the original tools from extreme-linux and beowulf sites will not work 30with this version of the driver. 31 32 For new versions of the driver, updated userspace tools, and 33who to ask for help, please follow the links at the end of this file. 34 35Table of Contents 36================= 37 381. Bonding Driver Installation 39 402. Bonding Driver Options 41 423. Configuring Bonding Devices 433.1 Configuration with Sysconfig Support 443.1.1 Using DHCP with Sysconfig 453.1.2 Configuring Multiple Bonds with Sysconfig 463.2 Configuration with Initscripts Support 473.2.1 Using DHCP with Initscripts 483.2.2 Configuring Multiple Bonds with Initscripts 493.3 Configuring Bonding Manually with Ifenslave 503.3.1 Configuring Multiple Bonds Manually 513.4 Configuring Bonding Manually via Sysfs 523.5 Configuration with Interfaces Support 533.6 Overriding Configuration for Special Cases 54 554. Querying Bonding Configuration 564.1 Bonding Configuration 574.2 Network Configuration 58 595. Switch Configuration 60 616. 802.1q VLAN Support 62 637. Link Monitoring 647.1 ARP Monitor Operation 657.2 Configuring Multiple ARP Targets 667.3 MII Monitor Operation 67 688. Potential Trouble Sources 698.1 Adventures in Routing 708.2 Ethernet Device Renaming 718.3 Painfully Slow Or No Failed Link Detection By Miimon 72 739. SNMP agents 74 7510. Promiscuous mode 76 7711. Configuring Bonding for High Availability 7811.1 High Availability in a Single Switch Topology 7911.2 High Availability in a Multiple Switch Topology 8011.2.1 HA Bonding Mode Selection for Multiple Switch Topology 8111.2.2 HA Link Monitoring for Multiple Switch Topology 82 8312. Configuring Bonding for Maximum Throughput 8412.1 Maximum Throughput in a Single Switch Topology 8512.1.1 MT Bonding Mode Selection for Single Switch Topology 8612.1.2 MT Link Monitoring for Single Switch Topology 8712.2 Maximum Throughput in a Multiple Switch Topology 8812.2.1 MT Bonding Mode Selection for Multiple Switch Topology 8912.2.2 MT Link Monitoring for Multiple Switch Topology 90 9113. Switch Behavior Issues 9213.1 Link Establishment and Failover Delays 9313.2 Duplicated Incoming Packets 94 9514. Hardware Specific Considerations 9614.1 IBM BladeCenter 97 9815. Frequently Asked Questions 99 10016. Resources and Links 101 102 1031. Bonding Driver Installation 104============================== 105 106 Most popular distro kernels ship with the bonding driver 107already available as a module. If your distro does not, or you 108have need to compile bonding from source (e.g., configuring and 109installing a mainline kernel from kernel.org), you'll need to perform 110the following steps: 111 1121.1 Configure and build the kernel with bonding 113----------------------------------------------- 114 115 The current version of the bonding driver is available in the 116drivers/net/bonding subdirectory of the most recent kernel source 117(which is available on http://kernel.org). Most users "rolling their 118own" will want to use the most recent kernel from kernel.org. 119 120 Configure kernel with "make menuconfig" (or "make xconfig" or 121"make config"), then select "Bonding driver support" in the "Network 122device support" section. It is recommended that you configure the 123driver as module since it is currently the only way to pass parameters 124to the driver or configure more than one bonding device. 125 126 Build and install the new kernel and modules. 127 1281.2 Bonding Control Utility 129------------------------------------- 130 131 It is recommended to configure bonding via iproute2 (netlink) 132or sysfs, the old ifenslave control utility is obsolete. 133 1342. Bonding Driver Options 135========================= 136 137 Options for the bonding driver are supplied as parameters to the 138bonding module at load time, or are specified via sysfs. 139 140 Module options may be given as command line arguments to the 141insmod or modprobe command, but are usually specified in either the 142/etc/modrobe.d/*.conf configuration files, or in a distro-specific 143configuration file (some of which are detailed in the next section). 144 145 Details on bonding support for sysfs is provided in the 146"Configuring Bonding Manually via Sysfs" section, below. 147 148 The available bonding driver parameters are listed below. If a 149parameter is not specified the default value is used. When initially 150configuring a bond, it is recommended "tail -f /var/log/messages" be 151run in a separate window to watch for bonding driver error messages. 152 153 It is critical that either the miimon or arp_interval and 154arp_ip_target parameters be specified, otherwise serious network 155degradation will occur during link failures. Very few devices do not 156support at least miimon, so there is really no reason not to use it. 157 158 Options with textual values will accept either the text name 159or, for backwards compatibility, the option value. E.g., 160"mode=802.3ad" and "mode=4" set the same mode. 161 162 The parameters are as follows: 163 164active_slave 165 166 Specifies the new active slave for modes that support it 167 (active-backup, balance-alb and balance-tlb). Possible values 168 are the name of any currently enslaved interface, or an empty 169 string. If a name is given, the slave and its link must be up in order 170 to be selected as the new active slave. If an empty string is 171 specified, the current active slave is cleared, and a new active 172 slave is selected automatically. 173 174 Note that this is only available through the sysfs interface. No module 175 parameter by this name exists. 176 177 The normal value of this option is the name of the currently 178 active slave, or the empty string if there is no active slave or 179 the current mode does not use an active slave. 180 181ad_select 182 183 Specifies the 802.3ad aggregation selection logic to use. The 184 possible values and their effects are: 185 186 stable or 0 187 188 The active aggregator is chosen by largest aggregate 189 bandwidth. 190 191 Reselection of the active aggregator occurs only when all 192 slaves of the active aggregator are down or the active 193 aggregator has no slaves. 194 195 This is the default value. 196 197 bandwidth or 1 198 199 The active aggregator is chosen by largest aggregate 200 bandwidth. Reselection occurs if: 201 202 - A slave is added to or removed from the bond 203 204 - Any slave's link state changes 205 206 - Any slave's 802.3ad association state changes 207 208 - The bond's administrative state changes to up 209 210 count or 2 211 212 The active aggregator is chosen by the largest number of 213 ports (slaves). Reselection occurs as described under the 214 "bandwidth" setting, above. 215 216 The bandwidth and count selection policies permit failover of 217 802.3ad aggregations when partial failure of the active aggregator 218 occurs. This keeps the aggregator with the highest availability 219 (either in bandwidth or in number of ports) active at all times. 220 221 This option was added in bonding version 3.4.0. 222 223all_slaves_active 224 225 Specifies that duplicate frames (received on inactive ports) should be 226 dropped (0) or delivered (1). 227 228 Normally, bonding will drop duplicate frames (received on inactive 229 ports), which is desirable for most users. But there are some times 230 it is nice to allow duplicate frames to be delivered. 231 232 The default value is 0 (drop duplicate frames received on inactive 233 ports). 234 235arp_interval 236 237 Specifies the ARP link monitoring frequency in milliseconds. 238 239 The ARP monitor works by periodically checking the slave 240 devices to determine whether they have sent or received 241 traffic recently (the precise criteria depends upon the 242 bonding mode, and the state of the slave). Regular traffic is 243 generated via ARP probes issued for the addresses specified by 244 the arp_ip_target option. 245 246 This behavior can be modified by the arp_validate option, 247 below. 248 249 If ARP monitoring is used in an etherchannel compatible mode 250 (modes 0 and 2), the switch should be configured in a mode 251 that evenly distributes packets across all links. If the 252 switch is configured to distribute the packets in an XOR 253 fashion, all replies from the ARP targets will be received on 254 the same link which could cause the other team members to 255 fail. ARP monitoring should not be used in conjunction with 256 miimon. A value of 0 disables ARP monitoring. The default 257 value is 0. 258 259arp_ip_target 260 261 Specifies the IP addresses to use as ARP monitoring peers when 262 arp_interval is > 0. These are the targets of the ARP request 263 sent to determine the health of the link to the targets. 264 Specify these values in ddd.ddd.ddd.ddd format. Multiple IP 265 addresses must be separated by a comma. At least one IP 266 address must be given for ARP monitoring to function. The 267 maximum number of targets that can be specified is 16. The 268 default value is no IP addresses. 269 270arp_validate 271 272 Specifies whether or not ARP probes and replies should be 273 validated in any mode that supports arp monitoring, or whether 274 non-ARP traffic should be filtered (disregarded) for link 275 monitoring purposes. 276 277 Possible values are: 278 279 none or 0 280 281 No validation or filtering is performed. 282 283 active or 1 284 285 Validation is performed only for the active slave. 286 287 backup or 2 288 289 Validation is performed only for backup slaves. 290 291 all or 3 292 293 Validation is performed for all slaves. 294 295 filter or 4 296 297 Filtering is applied to all slaves. No validation is 298 performed. 299 300 filter_active or 5 301 302 Filtering is applied to all slaves, validation is performed 303 only for the active slave. 304 305 filter_backup or 6 306 307 Filtering is applied to all slaves, validation is performed 308 only for backup slaves. 309 310 Validation: 311 312 Enabling validation causes the ARP monitor to examine the incoming 313 ARP requests and replies, and only consider a slave to be up if it 314 is receiving the appropriate ARP traffic. 315 316 For an active slave, the validation checks ARP replies to confirm 317 that they were generated by an arp_ip_target. Since backup slaves 318 do not typically receive these replies, the validation performed 319 for backup slaves is on the broadcast ARP request sent out via the 320 active slave. It is possible that some switch or network 321 configurations may result in situations wherein the backup slaves 322 do not receive the ARP requests; in such a situation, validation 323 of backup slaves must be disabled. 324 325 The validation of ARP requests on backup slaves is mainly helping 326 bonding to decide which slaves are more likely to work in case of 327 the active slave failure, it doesn't really guarantee that the 328 backup slave will work if it's selected as the next active slave. 329 330 Validation is useful in network configurations in which multiple 331 bonding hosts are concurrently issuing ARPs to one or more targets 332 beyond a common switch. Should the link between the switch and 333 target fail (but not the switch itself), the probe traffic 334 generated by the multiple bonding instances will fool the standard 335 ARP monitor into considering the links as still up. Use of 336 validation can resolve this, as the ARP monitor will only consider 337 ARP requests and replies associated with its own instance of 338 bonding. 339 340 Filtering: 341 342 Enabling filtering causes the ARP monitor to only use incoming ARP 343 packets for link availability purposes. Arriving packets that are 344 not ARPs are delivered normally, but do not count when determining 345 if a slave is available. 346 347 Filtering operates by only considering the reception of ARP 348 packets (any ARP packet, regardless of source or destination) when 349 determining if a slave has received traffic for link availability 350 purposes. 351 352 Filtering is useful in network configurations in which significant 353 levels of third party broadcast traffic would fool the standard 354 ARP monitor into considering the links as still up. Use of 355 filtering can resolve this, as only ARP traffic is considered for 356 link availability purposes. 357 358 This option was added in bonding version 3.1.0. 359 360arp_all_targets 361 362 Specifies the quantity of arp_ip_targets that must be reachable 363 in order for the ARP monitor to consider a slave as being up. 364 This option affects only active-backup mode for slaves with 365 arp_validation enabled. 366 367 Possible values are: 368 369 any or 0 370 371 consider the slave up only when any of the arp_ip_targets 372 is reachable 373 374 all or 1 375 376 consider the slave up only when all of the arp_ip_targets 377 are reachable 378 379downdelay 380 381 Specifies the time, in milliseconds, to wait before disabling 382 a slave after a link failure has been detected. This option 383 is only valid for the miimon link monitor. The downdelay 384 value should be a multiple of the miimon value; if not, it 385 will be rounded down to the nearest multiple. The default 386 value is 0. 387 388fail_over_mac 389 390 Specifies whether active-backup mode should set all slaves to 391 the same MAC address at enslavement (the traditional 392 behavior), or, when enabled, perform special handling of the 393 bond's MAC address in accordance with the selected policy. 394 395 Possible values are: 396 397 none or 0 398 399 This setting disables fail_over_mac, and causes 400 bonding to set all slaves of an active-backup bond to 401 the same MAC address at enslavement time. This is the 402 default. 403 404 active or 1 405 406 The "active" fail_over_mac policy indicates that the 407 MAC address of the bond should always be the MAC 408 address of the currently active slave. The MAC 409 address of the slaves is not changed; instead, the MAC 410 address of the bond changes during a failover. 411 412 This policy is useful for devices that cannot ever 413 alter their MAC address, or for devices that refuse 414 incoming broadcasts with their own source MAC (which 415 interferes with the ARP monitor). 416 417 The down side of this policy is that every device on 418 the network must be updated via gratuitous ARP, 419 vs. just updating a switch or set of switches (which 420 often takes place for any traffic, not just ARP 421 traffic, if the switch snoops incoming traffic to 422 update its tables) for the traditional method. If the 423 gratuitous ARP is lost, communication may be 424 disrupted. 425 426 When this policy is used in conjunction with the mii 427 monitor, devices which assert link up prior to being 428 able to actually transmit and receive are particularly 429 susceptible to loss of the gratuitous ARP, and an 430 appropriate updelay setting may be required. 431 432 follow or 2 433 434 The "follow" fail_over_mac policy causes the MAC 435 address of the bond to be selected normally (normally 436 the MAC address of the first slave added to the bond). 437 However, the second and subsequent slaves are not set 438 to this MAC address while they are in a backup role; a 439 slave is programmed with the bond's MAC address at 440 failover time (and the formerly active slave receives 441 the newly active slave's MAC address). 442 443 This policy is useful for multiport devices that 444 either become confused or incur a performance penalty 445 when multiple ports are programmed with the same MAC 446 address. 447 448 449 The default policy is none, unless the first slave cannot 450 change its MAC address, in which case the active policy is 451 selected by default. 452 453 This option may be modified via sysfs only when no slaves are 454 present in the bond. 455 456 This option was added in bonding version 3.2.0. The "follow" 457 policy was added in bonding version 3.3.0. 458 459lacp_rate 460 461 Option specifying the rate in which we'll ask our link partner 462 to transmit LACPDU packets in 802.3ad mode. Possible values 463 are: 464 465 slow or 0 466 Request partner to transmit LACPDUs every 30 seconds 467 468 fast or 1 469 Request partner to transmit LACPDUs every 1 second 470 471 The default is slow. 472 473max_bonds 474 475 Specifies the number of bonding devices to create for this 476 instance of the bonding driver. E.g., if max_bonds is 3, and 477 the bonding driver is not already loaded, then bond0, bond1 478 and bond2 will be created. The default value is 1. Specifying 479 a value of 0 will load bonding, but will not create any devices. 480 481miimon 482 483 Specifies the MII link monitoring frequency in milliseconds. 484 This determines how often the link state of each slave is 485 inspected for link failures. A value of zero disables MII 486 link monitoring. A value of 100 is a good starting point. 487 The use_carrier option, below, affects how the link state is 488 determined. See the High Availability section for additional 489 information. The default value is 0. 490 491min_links 492 493 Specifies the minimum number of links that must be active before 494 asserting carrier. It is similar to the Cisco EtherChannel min-links 495 feature. This allows setting the minimum number of member ports that 496 must be up (link-up state) before marking the bond device as up 497 (carrier on). This is useful for situations where higher level services 498 such as clustering want to ensure a minimum number of low bandwidth 499 links are active before switchover. This option only affect 802.3ad 500 mode. 501 502 The default value is 0. This will cause carrier to be asserted (for 503 802.3ad mode) whenever there is an active aggregator, regardless of the 504 number of available links in that aggregator. Note that, because an 505 aggregator cannot be active without at least one available link, 506 setting this option to 0 or to 1 has the exact same effect. 507 508mode 509 510 Specifies one of the bonding policies. The default is 511 balance-rr (round robin). Possible values are: 512 513 balance-rr or 0 514 515 Round-robin policy: Transmit packets in sequential 516 order from the first available slave through the 517 last. This mode provides load balancing and fault 518 tolerance. 519 520 active-backup or 1 521 522 Active-backup policy: Only one slave in the bond is 523 active. A different slave becomes active if, and only 524 if, the active slave fails. The bond's MAC address is 525 externally visible on only one port (network adapter) 526 to avoid confusing the switch. 527 528 In bonding version 2.6.2 or later, when a failover 529 occurs in active-backup mode, bonding will issue one 530 or more gratuitous ARPs on the newly active slave. 531 One gratuitous ARP is issued for the bonding master 532 interface and each VLAN interfaces configured above 533 it, provided that the interface has at least one IP 534 address configured. Gratuitous ARPs issued for VLAN 535 interfaces are tagged with the appropriate VLAN id. 536 537 This mode provides fault tolerance. The primary 538 option, documented below, affects the behavior of this 539 mode. 540 541 balance-xor or 2 542 543 XOR policy: Transmit based on the selected transmit 544 hash policy. The default policy is a simple [(source 545 MAC address XOR'd with destination MAC address XOR 546 packet type ID) modulo slave count]. Alternate transmit 547 policies may be selected via the xmit_hash_policy option, 548 described below. 549 550 This mode provides load balancing and fault tolerance. 551 552 broadcast or 3 553 554 Broadcast policy: transmits everything on all slave 555 interfaces. This mode provides fault tolerance. 556 557 802.3ad or 4 558 559 IEEE 802.3ad Dynamic link aggregation. Creates 560 aggregation groups that share the same speed and 561 duplex settings. Utilizes all slaves in the active 562 aggregator according to the 802.3ad specification. 563 564 Slave selection for outgoing traffic is done according 565 to the transmit hash policy, which may be changed from 566 the default simple XOR policy via the xmit_hash_policy 567 option, documented below. Note that not all transmit 568 policies may be 802.3ad compliant, particularly in 569 regards to the packet mis-ordering requirements of 570 section 43.2.4 of the 802.3ad standard. Differing 571 peer implementations will have varying tolerances for 572 noncompliance. 573 574 Prerequisites: 575 576 1. Ethtool support in the base drivers for retrieving 577 the speed and duplex of each slave. 578 579 2. A switch that supports IEEE 802.3ad Dynamic link 580 aggregation. 581 582 Most switches will require some type of configuration 583 to enable 802.3ad mode. 584 585 balance-tlb or 5 586 587 Adaptive transmit load balancing: channel bonding that 588 does not require any special switch support. 589 590 In tlb_dynamic_lb=1 mode; the outgoing traffic is 591 distributed according to the current load (computed 592 relative to the speed) on each slave. 593 594 In tlb_dynamic_lb=0 mode; the load balancing based on 595 current load is disabled and the load is distributed 596 only using the hash distribution. 597 598 Incoming traffic is received by the current slave. 599 If the receiving slave fails, another slave takes over 600 the MAC address of the failed receiving slave. 601 602 Prerequisite: 603 604 Ethtool support in the base drivers for retrieving the 605 speed of each slave. 606 607 balance-alb or 6 608 609 Adaptive load balancing: includes balance-tlb plus 610 receive load balancing (rlb) for IPV4 traffic, and 611 does not require any special switch support. The 612 receive load balancing is achieved by ARP negotiation. 613 The bonding driver intercepts the ARP Replies sent by 614 the local system on their way out and overwrites the 615 source hardware address with the unique hardware 616 address of one of the slaves in the bond such that 617 different peers use different hardware addresses for 618 the server. 619 620 Receive traffic from connections created by the server 621 is also balanced. When the local system sends an ARP 622 Request the bonding driver copies and saves the peer's 623 IP information from the ARP packet. When the ARP 624 Reply arrives from the peer, its hardware address is 625 retrieved and the bonding driver initiates an ARP 626 reply to this peer assigning it to one of the slaves 627 in the bond. A problematic outcome of using ARP 628 negotiation for balancing is that each time that an 629 ARP request is broadcast it uses the hardware address 630 of the bond. Hence, peers learn the hardware address 631 of the bond and the balancing of receive traffic 632 collapses to the current slave. This is handled by 633 sending updates (ARP Replies) to all the peers with 634 their individually assigned hardware address such that 635 the traffic is redistributed. Receive traffic is also 636 redistributed when a new slave is added to the bond 637 and when an inactive slave is re-activated. The 638 receive load is distributed sequentially (round robin) 639 among the group of highest speed slaves in the bond. 640 641 When a link is reconnected or a new slave joins the 642 bond the receive traffic is redistributed among all 643 active slaves in the bond by initiating ARP Replies 644 with the selected MAC address to each of the 645 clients. The updelay parameter (detailed below) must 646 be set to a value equal or greater than the switch's 647 forwarding delay so that the ARP Replies sent to the 648 peers will not be blocked by the switch. 649 650 Prerequisites: 651 652 1. Ethtool support in the base drivers for retrieving 653 the speed of each slave. 654 655 2. Base driver support for setting the hardware 656 address of a device while it is open. This is 657 required so that there will always be one slave in the 658 team using the bond hardware address (the 659 curr_active_slave) while having a unique hardware 660 address for each slave in the bond. If the 661 curr_active_slave fails its hardware address is 662 swapped with the new curr_active_slave that was 663 chosen. 664 665num_grat_arp 666num_unsol_na 667 668 Specify the number of peer notifications (gratuitous ARPs and 669 unsolicited IPv6 Neighbor Advertisements) to be issued after a 670 failover event. As soon as the link is up on the new slave 671 (possibly immediately) a peer notification is sent on the 672 bonding device and each VLAN sub-device. This is repeated at 673 each link monitor interval (arp_interval or miimon, whichever 674 is active) if the number is greater than 1. 675 676 The valid range is 0 - 255; the default value is 1. These options 677 affect only the active-backup mode. These options were added for 678 bonding versions 3.3.0 and 3.4.0 respectively. 679 680 From Linux 3.0 and bonding version 3.7.1, these notifications 681 are generated by the ipv4 and ipv6 code and the numbers of 682 repetitions cannot be set independently. 683 684packets_per_slave 685 686 Specify the number of packets to transmit through a slave before 687 moving to the next one. When set to 0 then a slave is chosen at 688 random. 689 690 The valid range is 0 - 65535; the default value is 1. This option 691 has effect only in balance-rr mode. 692 693primary 694 695 A string (eth0, eth2, etc) specifying which slave is the 696 primary device. The specified device will always be the 697 active slave while it is available. Only when the primary is 698 off-line will alternate devices be used. This is useful when 699 one slave is preferred over another, e.g., when one slave has 700 higher throughput than another. 701 702 The primary option is only valid for active-backup(1), 703 balance-tlb (5) and balance-alb (6) mode. 704 705primary_reselect 706 707 Specifies the reselection policy for the primary slave. This 708 affects how the primary slave is chosen to become the active slave 709 when failure of the active slave or recovery of the primary slave 710 occurs. This option is designed to prevent flip-flopping between 711 the primary slave and other slaves. Possible values are: 712 713 always or 0 (default) 714 715 The primary slave becomes the active slave whenever it 716 comes back up. 717 718 better or 1 719 720 The primary slave becomes the active slave when it comes 721 back up, if the speed and duplex of the primary slave is 722 better than the speed and duplex of the current active 723 slave. 724 725 failure or 2 726 727 The primary slave becomes the active slave only if the 728 current active slave fails and the primary slave is up. 729 730 The primary_reselect setting is ignored in two cases: 731 732 If no slaves are active, the first slave to recover is 733 made the active slave. 734 735 When initially enslaved, the primary slave is always made 736 the active slave. 737 738 Changing the primary_reselect policy via sysfs will cause an 739 immediate selection of the best active slave according to the new 740 policy. This may or may not result in a change of the active 741 slave, depending upon the circumstances. 742 743 This option was added for bonding version 3.6.0. 744 745tlb_dynamic_lb 746 747 Specifies if dynamic shuffling of flows is enabled in tlb 748 mode. The value has no effect on any other modes. 749 750 The default behavior of tlb mode is to shuffle active flows across 751 slaves based on the load in that interval. This gives nice lb 752 characteristics but can cause packet reordering. If re-ordering is 753 a concern use this variable to disable flow shuffling and rely on 754 load balancing provided solely by the hash distribution. 755 xmit-hash-policy can be used to select the appropriate hashing for 756 the setup. 757 758 The sysfs entry can be used to change the setting per bond device 759 and the initial value is derived from the module parameter. The 760 sysfs entry is allowed to be changed only if the bond device is 761 down. 762 763 The default value is "1" that enables flow shuffling while value "0" 764 disables it. This option was added in bonding driver 3.7.1 765 766 767updelay 768 769 Specifies the time, in milliseconds, to wait before enabling a 770 slave after a link recovery has been detected. This option is 771 only valid for the miimon link monitor. The updelay value 772 should be a multiple of the miimon value; if not, it will be 773 rounded down to the nearest multiple. The default value is 0. 774 775use_carrier 776 777 Specifies whether or not miimon should use MII or ETHTOOL 778 ioctls vs. netif_carrier_ok() to determine the link 779 status. The MII or ETHTOOL ioctls are less efficient and 780 utilize a deprecated calling sequence within the kernel. The 781 netif_carrier_ok() relies on the device driver to maintain its 782 state with netif_carrier_on/off; at this writing, most, but 783 not all, device drivers support this facility. 784 785 If bonding insists that the link is up when it should not be, 786 it may be that your network device driver does not support 787 netif_carrier_on/off. The default state for netif_carrier is 788 "carrier on," so if a driver does not support netif_carrier, 789 it will appear as if the link is always up. In this case, 790 setting use_carrier to 0 will cause bonding to revert to the 791 MII / ETHTOOL ioctl method to determine the link state. 792 793 A value of 1 enables the use of netif_carrier_ok(), a value of 794 0 will use the deprecated MII / ETHTOOL ioctls. The default 795 value is 1. 796 797xmit_hash_policy 798 799 Selects the transmit hash policy to use for slave selection in 800 balance-xor, 802.3ad, and tlb modes. Possible values are: 801 802 layer2 803 804 Uses XOR of hardware MAC addresses and packet type ID 805 field to generate the hash. The formula is 806 807 hash = source MAC XOR destination MAC XOR packet type ID 808 slave number = hash modulo slave count 809 810 This algorithm will place all traffic to a particular 811 network peer on the same slave. 812 813 This algorithm is 802.3ad compliant. 814 815 layer2+3 816 817 This policy uses a combination of layer2 and layer3 818 protocol information to generate the hash. 819 820 Uses XOR of hardware MAC addresses and IP addresses to 821 generate the hash. The formula is 822 823 hash = source MAC XOR destination MAC XOR packet type ID 824 hash = hash XOR source IP XOR destination IP 825 hash = hash XOR (hash RSHIFT 16) 826 hash = hash XOR (hash RSHIFT 8) 827 And then hash is reduced modulo slave count. 828 829 If the protocol is IPv6 then the source and destination 830 addresses are first hashed using ipv6_addr_hash. 831 832 This algorithm will place all traffic to a particular 833 network peer on the same slave. For non-IP traffic, 834 the formula is the same as for the layer2 transmit 835 hash policy. 836 837 This policy is intended to provide a more balanced 838 distribution of traffic than layer2 alone, especially 839 in environments where a layer3 gateway device is 840 required to reach most destinations. 841 842 This algorithm is 802.3ad compliant. 843 844 layer3+4 845 846 This policy uses upper layer protocol information, 847 when available, to generate the hash. This allows for 848 traffic to a particular network peer to span multiple 849 slaves, although a single connection will not span 850 multiple slaves. 851 852 The formula for unfragmented TCP and UDP packets is 853 854 hash = source port, destination port (as in the header) 855 hash = hash XOR source IP XOR destination IP 856 hash = hash XOR (hash RSHIFT 16) 857 hash = hash XOR (hash RSHIFT 8) 858 And then hash is reduced modulo slave count. 859 860 If the protocol is IPv6 then the source and destination 861 addresses are first hashed using ipv6_addr_hash. 862 863 For fragmented TCP or UDP packets and all other IPv4 and 864 IPv6 protocol traffic, the source and destination port 865 information is omitted. For non-IP traffic, the 866 formula is the same as for the layer2 transmit hash 867 policy. 868 869 This algorithm is not fully 802.3ad compliant. A 870 single TCP or UDP conversation containing both 871 fragmented and unfragmented packets will see packets 872 striped across two interfaces. This may result in out 873 of order delivery. Most traffic types will not meet 874 this criteria, as TCP rarely fragments traffic, and 875 most UDP traffic is not involved in extended 876 conversations. Other implementations of 802.3ad may 877 or may not tolerate this noncompliance. 878 879 encap2+3 880 881 This policy uses the same formula as layer2+3 but it 882 relies on skb_flow_dissect to obtain the header fields 883 which might result in the use of inner headers if an 884 encapsulation protocol is used. For example this will 885 improve the performance for tunnel users because the 886 packets will be distributed according to the encapsulated 887 flows. 888 889 encap3+4 890 891 This policy uses the same formula as layer3+4 but it 892 relies on skb_flow_dissect to obtain the header fields 893 which might result in the use of inner headers if an 894 encapsulation protocol is used. For example this will 895 improve the performance for tunnel users because the 896 packets will be distributed according to the encapsulated 897 flows. 898 899 The default value is layer2. This option was added in bonding 900 version 2.6.3. In earlier versions of bonding, this parameter 901 does not exist, and the layer2 policy is the only policy. The 902 layer2+3 value was added for bonding version 3.2.2. 903 904resend_igmp 905 906 Specifies the number of IGMP membership reports to be issued after 907 a failover event. One membership report is issued immediately after 908 the failover, subsequent packets are sent in each 200ms interval. 909 910 The valid range is 0 - 255; the default value is 1. A value of 0 911 prevents the IGMP membership report from being issued in response 912 to the failover event. 913 914 This option is useful for bonding modes balance-rr (0), active-backup 915 (1), balance-tlb (5) and balance-alb (6), in which a failover can 916 switch the IGMP traffic from one slave to another. Therefore a fresh 917 IGMP report must be issued to cause the switch to forward the incoming 918 IGMP traffic over the newly selected slave. 919 920 This option was added for bonding version 3.7.0. 921 922lp_interval 923 924 Specifies the number of seconds between instances where the bonding 925 driver sends learning packets to each slaves peer switch. 926 927 The valid range is 1 - 0x7fffffff; the default value is 1. This Option 928 has effect only in balance-tlb and balance-alb modes. 929 9303. Configuring Bonding Devices 931============================== 932 933 You can configure bonding using either your distro's network 934initialization scripts, or manually using either iproute2 or the 935sysfs interface. Distros generally use one of three packages for the 936network initialization scripts: initscripts, sysconfig or interfaces. 937Recent versions of these packages have support for bonding, while older 938versions do not. 939 940 We will first describe the options for configuring bonding for 941distros using versions of initscripts, sysconfig and interfaces with full 942or partial support for bonding, then provide information on enabling 943bonding without support from the network initialization scripts (i.e., 944older versions of initscripts or sysconfig). 945 946 If you're unsure whether your distro uses sysconfig, 947initscripts or interfaces, or don't know if it's new enough, have no fear. 948Determining this is fairly straightforward. 949 950 First, look for a file called interfaces in /etc/network directory. 951If this file is present in your system, then your system use interfaces. See 952Configuration with Interfaces Support. 953 954 Else, issue the command: 955 956$ rpm -qf /sbin/ifup 957 958 It will respond with a line of text starting with either 959"initscripts" or "sysconfig," followed by some numbers. This is the 960package that provides your network initialization scripts. 961 962 Next, to determine if your installation supports bonding, 963issue the command: 964 965$ grep ifenslave /sbin/ifup 966 967 If this returns any matches, then your initscripts or 968sysconfig has support for bonding. 969 9703.1 Configuration with Sysconfig Support 971---------------------------------------- 972 973 This section applies to distros using a version of sysconfig 974with bonding support, for example, SuSE Linux Enterprise Server 9. 975 976 SuSE SLES 9's networking configuration system does support 977bonding, however, at this writing, the YaST system configuration 978front end does not provide any means to work with bonding devices. 979Bonding devices can be managed by hand, however, as follows. 980 981 First, if they have not already been configured, configure the 982slave devices. On SLES 9, this is most easily done by running the 983yast2 sysconfig configuration utility. The goal is for to create an 984ifcfg-id file for each slave device. The simplest way to accomplish 985this is to configure the devices for DHCP (this is only to get the 986file ifcfg-id file created; see below for some issues with DHCP). The 987name of the configuration file for each device will be of the form: 988 989ifcfg-id-xx:xx:xx:xx:xx:xx 990 991 Where the "xx" portion will be replaced with the digits from 992the device's permanent MAC address. 993 994 Once the set of ifcfg-id-xx:xx:xx:xx:xx:xx files has been 995created, it is necessary to edit the configuration files for the slave 996devices (the MAC addresses correspond to those of the slave devices). 997Before editing, the file will contain multiple lines, and will look 998something like this: 999 1000BOOTPROTO='dhcp'
1001STARTMODE='on' 1002USERCTL='no' 1003UNIQUE='XNzu.WeZGOGF+4wE' 1004_nm_name='bus-pci-0001:61:01.0' 1005 1006 Change the BOOTPROTO and STARTMODE lines to the following: 1007 1008BOOTPROTO='none' 1009STARTMODE='off' 1010 1011 Do not alter the UNIQUE or _nm_name lines. Remove any other 1012lines (USERCTL, etc). 1013 1014 Once the ifcfg-id-xx:xx:xx:xx:xx:xx files have been modified, 1015it's time to create the configuration file for the bonding device 1016itself. This file is named ifcfg-bondX, where X is the number of the 1017bonding device to create, starting at 0. The first such file is 1018ifcfg-bond0, the second is ifcfg-bond1, and so on. The sysconfig 1019network configuration system will correctly start multiple instances 1020of bonding. 1021 1022 The contents of the ifcfg-bondX file is as follows: 1023 1024BOOTPROTO="static" 1025BROADCAST="10.0.2.255" 1026IPADDR="10.0.2.10" 1027NETMASK="255.255.0.0" 1028NETWORK="10.0.2.0" 1029REMOTE_IPADDR="" 1030STARTMODE="onboot" 1031BONDING_MASTER="yes" 1032BONDING_MODULE_OPTS="mode=active-backup miimon=100" 1033BONDING_SLAVE0="eth0" 1034BONDING_SLAVE1="bus-pci-0000:06:08.1" 1035 1036 Replace the sample BROADCAST, IPADDR, NETMASK and NETWORK 1037values with the appropriate values for your network. 1038 1039 The STARTMODE specifies when the device is brought online. 1040The possible values are: 1041 1042 onboot: The device is started at boot time. If you're not 1043 sure, this is probably what you want. 1044 1045 manual: The device is started only when ifup is called 1046 manually. Bonding devices may be configured this 1047 way if you do not wish them to start automatically 1048 at boot for some reason. 1049 1050 hotplug: The device is started by a hotplug event. This is not 1051 a valid choice for a bonding device. 1052 1053 off or ignore: The device configuration is ignored. 1054 1055 The line BONDING_MASTER='yes' indicates that the device is a 1056bonding master device. The only useful value is "yes." 1057 1058 The contents of BONDING_MODULE_OPTS are supplied to the 1059instance of the bonding module for this device. Specify the options 1060for the bonding mode, link monitoring, and so on here. Do not include 1061the max_bonds bonding parameter; this will confuse the configuration 1062system if you have multiple bonding devices. 1063 1064 Finally, supply one BONDING_SLAVEn="slave device" for each 1065slave. where "n" is an increasing value, one for each slave. The 1066"slave device" is either an interface name, e.g., "eth0", or a device 1067specifier for the network device. The interface name is easier to 1068find, but the ethN names are subject to change at boot time if, e.g., 1069a device early in the sequence has failed. The device specifiers 1070(bus-pci-0000:06:08.1 in the example above) specify the physical 1071network device, and will not change unless the device's bus location 1072changes (for example, it is moved from one PCI slot to another). The 1073example above uses one of each type for demonstration purposes; most 1074configurations will choose one or the other for all slave devices. 1075 1076 When all configuration files have been modified or created, 1077networking must be restarted for the configuration changes to take 1078effect. This can be accomplished via the following: 1079 1080# /etc/init.d/network restart 1081 1082 Note that the network control script (/sbin/ifdown) will 1083remove the bonding module as part of the network shutdown processing, 1084so it is not necessary to remove the module by hand if, e.g., the 1085module parameters have changed. 1086 1087 Also, at this writing, YaST/YaST2 will not manage bonding 1088devices (they do not show bonding interfaces on its list of network 1089devices). It is necessary to edit the configuration file by hand to 1090change the bonding configuration. 1091 1092 Additional general options and details of the ifcfg file 1093format can be found in an example ifcfg template file: 1094 1095/etc/sysconfig/network/ifcfg.template 1096 1097 Note that the template does not document the various BONDING_ 1098settings described above, but does describe many of the other options. 1099 11003.1.1 Using DHCP with Sysconfig 1101------------------------------- 1102 1103 Under sysconfig, configuring a device with BOOTPROTO='dhcp' 1104will cause it to query DHCP for its IP address information. At this 1105writing, this does not function for bonding devices; the scripts 1106attempt to obtain the device address from DHCP prior to adding any of 1107the slave devices. Without active slaves, the DHCP requests are not 1108sent to the network. 1109 11103.1.2 Configuring Multiple Bonds with Sysconfig 1111----------------------------------------------- 1112 1113 The sysconfig network initialization system is capable of 1114handling multiple bonding devices. All that is necessary is for each 1115bonding instance to have an appropriately configured ifcfg-bondX file 1116(as described above). Do not specify the "max_bonds" parameter to any 1117instance of bonding, as this will confuse sysconfig. If you require 1118multiple bonding devices with identical parameters, create multiple 1119ifcfg-bondX files. 1120 1121 Because the sysconfig scripts supply the bonding module 1122options in the ifcfg-bondX file, it is not necessary to add them to 1123the system /etc/modules.d/*.conf configuration files. 1124 11253.2 Configuration with Initscripts Support 1126------------------------------------------ 1127 1128 This section applies to distros using a recent version of 1129initscripts with bonding support, for example, Red Hat Enterprise Linux 1130version 3 or later, Fedora, etc. On these systems, the network 1131initialization scripts have knowledge of bonding, and can be configured to 1132control bonding devices. Note that older versions of the initscripts 1133package have lower levels of support for bonding; this will be noted where 1134applicable. 1135 1136 These distros will not automatically load the network adapter 1137driver unless the ethX device is configured with an IP address. 1138Because of this constraint, users must manually configure a 1139network-script file for all physical adapters that will be members of 1140a bondX link. Network script files are located in the directory: 1141 1142/etc/sysconfig/network-scripts 1143 1144 The file name must be prefixed with "ifcfg-eth" and suffixed 1145with the adapter's physical adapter number. For example, the script 1146for eth0 would be named /etc/sysconfig/network-scripts/ifcfg-eth0. 1147Place the following text in the file: 1148 1149DEVICE=eth0 1150USERCTL=no 1151ONBOOT=yes 1152MASTER=bond0 1153SLAVE=yes 1154BOOTPROTO=none 1155 1156 The DEVICE= line will be different for every ethX device and 1157must correspond with the name of the file, i.e., ifcfg-eth1 must have 1158a device line of DEVICE=eth1. The setting of the MASTER= line will 1159also depend on the final bonding interface name chosen for your bond. 1160As with other network devices, these typically start at 0, and go up 1161one for each device, i.e., the first bonding instance is bond0, the 1162second is bond1, and so on. 1163 1164 Next, create a bond network script. The file name for this 1165script will be /etc/sysconfig/network-scripts/ifcfg-bondX where X is 1166the number of the bond. For bond0 the file is named "ifcfg-bond0", 1167for bond1 it is named "ifcfg-bond1", and so on. Within that file, 1168place the following text: 1169 1170DEVICE=bond0 1171IPADDR=192.168.1.1 1172NETMASK=255.255.255.0 1173NETWORK=192.168.1.0 1174BROADCAST=192.168.1.255 1175ONBOOT=yes 1176BOOTPROTO=none 1177USERCTL=no 1178 1179 Be sure to change the networking specific lines (IPADDR, 1180NETMASK, NETWORK and BROADCAST) to match your network configuration. 1181 1182 For later versions of initscripts, such as that found with Fedora 11837 (or later) and Red Hat Enterprise Linux version 5 (or later), it is possible, 1184and, indeed, preferable, to specify the bonding options in the ifcfg-bond0 1185file, e.g. a line of the format: 1186 1187BONDING_OPTS="mode=active-backup arp_interval=60 arp_ip_target=192.168.1.254" 1188 1189 will configure the bond with the specified options. The options 1190specified in BONDING_OPTS are identical to the bonding module parameters 1191except for the arp_ip_target field when using versions of initscripts older 1192than and 8.57 (Fedora 8) and 8.45.19 (Red Hat Enterprise Linux 5.2). When 1193using older versions each target should be included as a separate option and 1194should be preceded by a '+' to indicate it should be added to the list of 1195queried targets, e.g., 1196 1197 arp_ip_target=+192.168.1.1 arp_ip_target=+192.168.1.2 1198 1199 is the proper syntax to specify multiple targets. When specifying 1200options via BONDING_OPTS, it is not necessary to edit /etc/modprobe.d/*.conf. 1201 1202 For even older versions of initscripts that do not support 1203BONDING_OPTS, it is necessary to edit /etc/modprobe.d/*.conf, depending upon 1204your distro) to load the bonding module with your desired options when the 1205bond0 interface is brought up. The following lines in /etc/modprobe.d/*.conf 1206will load the bonding module, and select its options: 1207 1208alias bond0 bonding 1209options bond0 mode=balance-alb miimon=100 1210 1211 Replace the sample parameters with the appropriate set of 1212options for your configuration. 1213 1214 Finally run "/etc/rc.d/init.d/network restart" as root. This 1215will restart the networking subsystem and your bond link should be now 1216up and running. 1217 12183.2.1 Using DHCP with Initscripts 1219--------------------------------- 1220 1221 Recent versions of initscripts (the versions supplied with Fedora 1222Core 3 and Red Hat Enterprise Linux 4, or later versions, are reported to 1223work) have support for assigning IP information to bonding devices via 1224DHCP. 1225 1226 To configure bonding for DHCP, configure it as described 1227above, except replace the line "BOOTPROTO=none" with "BOOTPROTO=dhcp" 1228and add a line consisting of "TYPE=Bonding". Note that the TYPE value 1229is case sensitive. 1230 12313.2.2 Configuring Multiple Bonds with Initscripts 1232------------------------------------------------- 1233 1234 Initscripts packages that are included with Fedora 7 and Red Hat 1235Enterprise Linux 5 support multiple bonding interfaces by simply 1236specifying the appropriate BONDING_OPTS= in ifcfg-bondX where X is the 1237number of the bond. This support requires sysfs support in the kernel, 1238and a bonding driver of version 3.0.0 or later. Other configurations may 1239not support this method for specifying multiple bonding interfaces; for 1240those instances, see the "Configuring Multiple Bonds Manually" section, 1241below. 1242 12433.3 Configuring Bonding Manually with iproute2 1244----------------------------------------------- 1245 1246 This section applies to distros whose network initialization 1247scripts (the sysconfig or initscripts package) do not have specific 1248knowledge of bonding. One such distro is SuSE Linux Enterprise Server 1249version 8. 1250 1251 The general method for these systems is to place the bonding 1252module parameters into a config file in /etc/modprobe.d/ (as 1253appropriate for the installed distro), then add modprobe and/or 1254`ip link` commands to the system's global init script. The name of 1255the global init script differs; for sysconfig, it is 1256/etc/init.d/boot.local and for initscripts it is /etc/rc.d/rc.local. 1257 1258 For example, if you wanted to make a simple bond of two e100 1259devices (presumed to be eth0 and eth1), and have it persist across 1260reboots, edit the appropriate file (/etc/init.d/boot.local or 1261/etc/rc.d/rc.local), and add the following: 1262 1263modprobe bonding mode=balance-alb miimon=100 1264modprobe e100 1265ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up 1266ip link set eth0 master bond0 1267ip link set eth1 master bond0 1268 1269 Replace the example bonding module parameters and bond0 1270network configuration (IP address, netmask, etc) with the appropriate 1271values for your configuration. 1272 1273 Unfortunately, this method will not provide support for the 1274ifup and ifdown scripts on the bond devices. To reload the bonding 1275configuration, it is necessary to run the initialization script, e.g., 1276 1277# /etc/init.d/boot.local 1278 1279 or 1280 1281# /etc/rc.d/rc.local 1282 1283 It may be desirable in such a case to create a separate script 1284which only initializes the bonding configuration, then call that 1285separate script from within boot.local. This allows for bonding to be 1286enabled without re-running the entire global init script. 1287 1288 To shut down the bonding devices, it is necessary to first 1289mark the bonding device itself as being down, then remove the 1290appropriate device driver modules. For our example above, you can do 1291the following: 1292 1293# ifconfig bond0 down 1294# rmmod bonding 1295# rmmod e100 1296 1297 Again, for convenience, it may be desirable to create a script 1298with these commands. 1299 1300 13013.3.1 Configuring Multiple Bonds Manually 1302----------------------------------------- 1303 1304 This section contains information on configuring multiple 1305bonding devices with differing options for those systems whose network 1306initialization scripts lack support for configuring multiple bonds. 1307 1308 If you require multiple bonding devices, but all with the same 1309options, you may wish to use the "max_bonds" module parameter, 1310documented above. 1311 1312 To create multiple bonding devices with differing options, it is 1313preferable to use bonding parameters exported by sysfs, documented in the 1314section below. 1315 1316 For versions of bonding without sysfs support, the only means to 1317provide multiple instances of bonding with differing options is to load 1318the bonding driver multiple times. Note that current versions of the 1319sysconfig network initialization scripts handle this automatically; if 1320your distro uses these scripts, no special action is needed. See the 1321section Configuring Bonding Devices, above, if you're not sure about your 1322network initialization scripts. 1323 1324 To load multiple instances of the module, it is necessary to 1325specify a different name for each instance (the module loading system 1326requires that every loaded module, even multiple instances of the same 1327module, have a unique name). This is accomplished by supplying multiple 1328sets of bonding options in /etc/modprobe.d/*.conf, for example: 1329 1330alias bond0 bonding 1331options bond0 -o bond0 mode=balance-rr miimon=100 1332 1333alias bond1 bonding 1334options bond1 -o bond1 mode=balance-alb miimon=50 1335 1336 will load the bonding module two times. The first instance is 1337named "bond0" and creates the bond0 device in balance-rr mode with an 1338miimon of 100. The second instance is named "bond1" and creates the 1339bond1 device in balance-alb mode with an miimon of 50. 1340 1341 In some circumstances (typically with older distributions), 1342the above does not work, and the second bonding instance never sees 1343its options. In that case, the second options line can be substituted 1344as follows: 1345 1346install bond1 /sbin/modprobe --ignore-install bonding -o bond1 \ 1347 mode=balance-alb miimon=50 1348 1349 This may be repeated any number of times, specifying a new and 1350unique name in place of bond1 for each subsequent instance. 1351 1352 It has been observed that some Red Hat supplied kernels are unable 1353to rename modules at load time (the "-o bond1" part). Attempts to pass 1354that option to modprobe will produce an "Operation not permitted" error. 1355This has been reported on some Fedora Core kernels, and has been seen on 1356RHEL 4 as well. On kernels exhibiting this problem, it will be impossible 1357to configure multiple bonds with differing parameters (as they are older 1358kernels, and also lack sysfs support). 1359 13603.4 Configuring Bonding Manually via Sysfs 1361------------------------------------------ 1362 1363 Starting with version 3.0.0, Channel Bonding may be configured 1364via the sysfs interface. This interface allows dynamic configuration 1365of all bonds in the system without unloading the module. It also 1366allows for adding and removing bonds at runtime. Ifenslave is no 1367longer required, though it is still supported. 1368 1369 Use of the sysfs interface allows you to use multiple bonds 1370with different configurations without having to reload the module. 1371It also allows you to use multiple, differently configured bonds when 1372bonding is compiled into the kernel. 1373 1374 You must have the sysfs filesystem mounted to configure 1375bonding this way. The examples in this document assume that you 1376are using the standard mount point for sysfs, e.g. /sys. If your 1377sysfs filesystem is mounted elsewhere, you will need to adjust the 1378example paths accordingly. 1379 1380Creating and Destroying Bonds 1381----------------------------- 1382To add a new bond foo: 1383# echo +foo > /sys/class/net/bonding_masters 1384 1385To remove an existing bond bar: 1386# echo -bar > /sys/class/net/bonding_masters 1387 1388To show all existing bonds: 1389# cat /sys/class/net/bonding_masters 1390 1391NOTE: due to 4K size limitation of sysfs files, this list may be 1392truncated if you have more than a few hundred bonds. This is unlikely 1393to occur under normal operating conditions. 1394 1395Adding and Removing Slaves 1396-------------------------- 1397 Interfaces may be enslaved to a bond using the file 1398/sys/class/net/<bond>/bonding/slaves. The semantics for this file 1399are the same as for the bonding_masters file. 1400 1401To enslave interface eth0 to bond bond0: 1402# ifconfig bond0 up 1403# echo +eth0 > /sys/class/net/bond0/bonding/slaves 1404 1405To free slave eth0 from bond bond0: 1406# echo -eth0 > /sys/class/net/bond0/bonding/slaves 1407 1408 When an interface is enslaved to a bond, symlinks between the 1409two are created in the sysfs filesystem. In this case, you would get 1410/sys/class/net/bond0/slave_eth0 pointing to /sys/class/net/eth0, and 1411/sys/class/net/eth0/master pointing to /sys/class/net/bond0. 1412 1413 This means that you can tell quickly whether or not an 1414interface is enslaved by looking for the master symlink. Thus: 1415# echo -eth0 > /sys/class/net/eth0/master/bonding/slaves 1416will free eth0 from whatever bond it is enslaved to, regardless of 1417the name of the bond interface. 1418 1419Changing a Bond's Configuration 1420------------------------------- 1421 Each bond may be configured individually by manipulating the 1422files located in /sys/class/net/<bond name>/bonding 1423 1424 The names of these files correspond directly with the command- 1425line parameters described elsewhere in this file, and, with the 1426exception of arp_ip_target, they accept the same values. To see the 1427current setting, simply cat the appropriate file. 1428 1429 A few examples will be given here; for specific usage 1430guidelines for each parameter, see the appropriate section in this 1431document. 1432 1433To configure bond0 for balance-alb mode: 1434# ifconfig bond0 down 1435# echo 6 > /sys/class/net/bond0/bonding/mode 1436 - or - 1437# echo balance-alb > /sys/class/net/bond0/bonding/mode 1438 NOTE: The bond interface must be down before the mode can be 1439changed. 1440 1441To enable MII monitoring on bond0 with a 1 second interval: 1442# echo 1000 > /sys/class/net/bond0/bonding/miimon 1443 NOTE: If ARP monitoring is enabled, it will disabled when MII 1444monitoring is enabled, and vice-versa. 1445 1446To add ARP targets: 1447# echo +192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target 1448# echo +192.168.0.101 > /sys/class/net/bond0/bonding/arp_ip_target 1449 NOTE: up to 16 target addresses may be specified. 1450 1451To remove an ARP target: 1452# echo -192.168.0.100 > /sys/class/net/bond0/bonding/arp_ip_target 1453 1454To configure the interval between learning packet transmits: 1455# echo 12 > /sys/class/net/bond0/bonding/lp_interval 1456 NOTE: the lp_inteval is the number of seconds between instances where 1457the bonding driver sends learning packets to each slaves peer switch. The 1458default interval is 1 second. 1459 1460Example Configuration 1461--------------------- 1462 We begin with the same example that is shown in section 3.3, 1463executed with sysfs, and without using ifenslave. 1464 1465 To make a simple bond of two e100 devices (presumed to be eth0 1466and eth1), and have it persist across reboots, edit the appropriate 1467file (/etc/init.d/boot.local or /etc/rc.d/rc.local), and add the 1468following: 1469 1470modprobe bonding 1471modprobe e100 1472echo balance-alb > /sys/class/net/bond0/bonding/mode 1473ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up 1474echo 100 > /sys/class/net/bond0/bonding/miimon 1475echo +eth0 > /sys/class/net/bond0/bonding/slaves 1476echo +eth1 > /sys/class/net/bond0/bonding/slaves 1477 1478 To add a second bond, with two e1000 interfaces in 1479active-backup mode, using ARP monitoring, add the following lines to 1480your init script: 1481 1482modprobe e1000 1483echo +bond1 > /sys/class/net/bonding_masters 1484echo active-backup > /sys/class/net/bond1/bonding/mode 1485ifconfig bond1 192.168.2.1 netmask 255.255.255.0 up 1486echo +192.168.2.100 /sys/class/net/bond1/bonding/arp_ip_target 1487echo 2000 > /sys/class/net/bond1/bonding/arp_interval 1488echo +eth2 > /sys/class/net/bond1/bonding/slaves 1489echo +eth3 > /sys/class/net/bond1/bonding/slaves 1490 14913.5 Configuration with Interfaces Support 1492----------------------------------------- 1493 1494 This section applies to distros which use /etc/network/interfaces file 1495to describe network interface configuration, most notably Debian and it's 1496derivatives. 1497 1498 The ifup and ifdown commands on Debian don't support bonding out of 1499the box. The ifenslave-2.6 package should be installed to provide bonding 1500support. Once installed, this package will provide bond-* options to be used 1501into /etc/network/interfaces. 1502 1503 Note that ifenslave-2.6 package will load the bonding module and use 1504the ifenslave command when appropriate. 1505 1506Example Configurations 1507---------------------- 1508 1509In /etc/network/interfaces, the following stanza will configure bond0, in 1510active-backup mode, with eth0 and eth1 as slaves. 1511 1512auto bond0 1513iface bond0 inet dhcp 1514 bond-slaves eth0 eth1 1515 bond-mode active-backup 1516 bond-miimon 100 1517 bond-primary eth0 eth1 1518 1519If the above configuration doesn't work, you might have a system using 1520upstart for system startup. This is most notably true for recent 1521Ubuntu versions. The following stanza in /etc/network/interfaces will 1522produce the same result on those systems. 1523 1524auto bond0 1525iface bond0 inet dhcp 1526 bond-slaves none 1527 bond-mode active-backup 1528 bond-miimon 100 1529 1530auto eth0 1531iface eth0 inet manual 1532 bond-master bond0 1533 bond-primary eth0 eth1 1534 1535auto eth1 1536iface eth1 inet manual 1537 bond-master bond0 1538 bond-primary eth0 eth1 1539 1540For a full list of bond-* supported options in /etc/network/interfaces and some 1541more advanced examples tailored to you particular distros, see the files in 1542/usr/share/doc/ifenslave-2.6. 1543 15443.6 Overriding Configuration for Special Cases 1545---------------------------------------------- 1546 1547When using the bonding driver, the physical port which transmits a frame is 1548typically selected by the bonding driver, and is not relevant to the user or 1549system administrator. The output port is simply selected using the policies of 1550the selected bonding mode. On occasion however, it is helpful to direct certain 1551classes of traffic to certain physical interfaces on output to implement 1552slightly more complex policies. For example, to reach a web server over a 1553bonded interface in which eth0 connects to a private network, while eth1 1554connects via a public network, it may be desirous to bias the bond to send said 1555traffic over eth0 first, using eth1 only as a fall back, while all other traffic 1556can safely be sent over either interface. Such configurations may be achieved 1557using the traffic control utilities inherent in linux. 1558 1559By default the bonding driver is multiqueue aware and 16 queues are created 1560when the driver initializes (see Documentation/networking/multiqueue.txt 1561for details). If more or less queues are desired the module parameter 1562tx_queues can be used to change this value. There is no sysfs parameter 1563available as the allocation is done at module init time. 1564 1565The output of the file /proc/net/bonding/bondX has changed so the output Queue 1566ID is now printed for each slave: 1567 1568Bonding Mode: fault-tolerance (active-backup) 1569Primary Slave: None 1570Currently Active Slave: eth0 1571MII Status: up 1572MII Polling Interval (ms): 0 1573Up Delay (ms): 0 1574Down Delay (ms): 0 1575 1576Slave Interface: eth0 1577MII Status: up 1578Link Failure Count: 0 1579Permanent HW addr: 00:1a:a0:12:8f:cb 1580Slave queue ID: 0 1581 1582Slave Interface: eth1 1583MII Status: up 1584Link Failure Count: 0 1585Permanent HW addr: 00:1a:a0:12:8f:cc 1586Slave queue ID: 2 1587 1588The queue_id for a slave can be set using the command: 1589 1590# echo "eth1:2" > /sys/class/net/bond0/bonding/queue_id 1591 1592Any interface that needs a queue_id set should set it with multiple calls 1593like the one above until proper priorities are set for all interfaces. On 1594distributions that allow configuration via initscripts, multiple 'queue_id' 1595arguments can be added to BONDING_OPTS to set all needed slave queues. 1596 1597These queue id's can be used in conjunction with the tc utility to configure 1598a multiqueue qdisc and filters to bias certain traffic to transmit on certain 1599slave devices. For instance, say we wanted, in the above configuration to 1600force all traffic bound to 192.168.1.100 to use eth1 in the bond as its output 1601device. The following commands would accomplish this: 1602 1603# tc qdisc add dev bond0 handle 1 root multiq 1604 1605# tc filter add dev bond0 protocol ip parent 1: prio 1 u32 match ip dst \ 1606 192.168.1.100 action skbedit queue_mapping 2 1607 1608These commands tell the kernel to attach a multiqueue queue discipline to the 1609bond0 interface and filter traffic enqueued to it, such that packets with a dst 1610ip of 192.168.1.100 have their output queue mapping value overwritten to 2. 1611This value is then passed into the driver, causing the normal output path 1612selection policy to be overridden, selecting instead qid 2, which maps to eth1. 1613 1614Note that qid values begin at 1. Qid 0 is reserved to initiate to the driver 1615that normal output policy selection should take place. One benefit to simply 1616leaving the qid for a slave to 0 is the multiqueue awareness in the bonding 1617driver that is now present. This awareness allows tc filters to be placed on 1618slave devices as well as bond devices and the bonding driver will simply act as 1619a pass-through for selecting output queues on the slave device rather than 1620output port selection. 1621 1622This feature first appeared in bonding driver version 3.7.0 and support for 1623output slave selection was limited to round-robin and active-backup modes. 1624 16254 Querying Bonding Configuration 1626================================= 1627 16284.1 Bonding Configuration 1629------------------------- 1630 1631 Each bonding device has a read-only file residing in the 1632/proc/net/bonding directory. The file contents include information 1633about the bonding configuration, options and state of each slave. 1634 1635 For example, the contents of /proc/net/bonding/bond0 after the 1636driver is loaded with parameters of mode=0 and miimon=1000 is 1637generally as follows: 1638 1639 Ethernet Channel Bonding Driver: 2.6.1 (October 29, 2004) 1640 Bonding Mode: load balancing (round-robin) 1641 Currently Active Slave: eth0 1642 MII Status: up 1643 MII Polling Interval (ms): 1000 1644 Up Delay (ms): 0 1645 Down Delay (ms): 0 1646 1647 Slave Interface: eth1 1648 MII Status: up 1649 Link Failure Count: 1 1650 1651 Slave Interface: eth0 1652 MII Status: up 1653 Link Failure Count: 1 1654 1655 The precise format and contents will change depending upon the 1656bonding configuration, state, and version of the bonding driver. 1657 16584.2 Network configuration 1659------------------------- 1660 1661 The network configuration can be inspected using the ifconfig 1662command. Bonding devices will have the MASTER flag set; Bonding slave 1663devices will have the SLAVE flag set. The ifconfig output does not 1664contain information on which slaves are associated with which masters. 1665 1666 In the example below, the bond0 interface is the master 1667(MASTER) while eth0 and eth1 are slaves (SLAVE). Notice all slaves of 1668bond0 have the same MAC address (HWaddr) as bond0 for all modes except 1669TLB and ALB that require a unique MAC address for each slave. 1670 1671# /sbin/ifconfig 1672bond0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4 1673 inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0 1674 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 1675 RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0 1676 TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0 1677 collisions:0 txqueuelen:0 1678 1679eth0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4 1680 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 1681 RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0 1682 TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0 1683 collisions:0 txqueuelen:100 1684 Interrupt:10 Base address:0x1080 1685 1686eth1 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4 1687 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 1688 RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0 1689 TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0 1690 collisions:0 txqueuelen:100 1691 Interrupt:9 Base address:0x1400 1692 16935. Switch Configuration 1694======================= 1695 1696 For this section, "switch" refers to whatever system the 1697bonded devices are directly connected to (i.e., where the other end of 1698the cable plugs into). This may be an actual dedicated switch device, 1699or it may be another regular system (e.g., another computer running 1700Linux), 1701 1702 The active-backup, balance-tlb and balance-alb modes do not 1703require any specific configuration of the switch. 1704 1705 The 802.3ad mode requires that the switch have the appropriate 1706ports configured as an 802.3ad aggregation. The precise method used 1707to configure this varies from switch to switch, but, for example, a 1708Cisco 3550 series switch requires that the appropriate ports first be 1709grouped together in a single etherchannel instance, then that 1710etherchannel is set to mode "lacp" to enable 802.3ad (instead of 1711standard EtherChannel). 1712 1713 The balance-rr, balance-xor and broadcast modes generally 1714require that the switch have the appropriate ports grouped together. 1715The nomenclature for such a group differs between switches, it may be 1716called an "etherchannel" (as in the Cisco example, above), a "trunk 1717group" or some other similar variation. For these modes, each switch 1718will also have its own configuration options for the switch's transmit 1719policy to the bond. Typical choices include XOR of either the MAC or 1720IP addresses. The transmit policy of the two peers does not need to 1721match. For these three modes, the bonding mode really selects a 1722transmit policy for an EtherChannel group; all three will interoperate 1723with another EtherChannel group. 1724 1725 17266. 802.1q VLAN Support 1727====================== 1728 1729 It is possible to configure VLAN devices over a bond interface 1730using the 8021q driver. However, only packets coming from the 8021q 1731driver and passing through bonding will be tagged by default. Self 1732generated packets, for example, bonding's learning packets or ARP 1733packets generated by either ALB mode or the ARP monitor mechanism, are 1734tagged internally by bonding itself. As a result, bonding must 1735"learn" the VLAN IDs configured above it, and use those IDs to tag 1736self generated packets. 1737 1738 For reasons of simplicity, and to support the use of adapters 1739that can do VLAN hardware acceleration offloading, the bonding 1740interface declares itself as fully hardware offloading capable, it gets 1741the add_vid/kill_vid notifications to gather the necessary 1742information, and it propagates those actions to the slaves. In case 1743of mixed adapter types, hardware accelerated tagged packets that 1744should go through an adapter that is not offloading capable are 1745"un-accelerated" by the bonding driver so the VLAN tag sits in the 1746regular location. 1747 1748 VLAN interfaces *must* be added on top of a bonding interface 1749only after enslaving at least one slave. The bonding interface has a 1750hardware address of 00:00:00:00:00:00 until the first slave is added. 1751If the VLAN interface is created prior to the first enslavement, it 1752would pick up the all-zeroes hardware address. Once the first slave 1753is attached to the bond, the bond device itself will pick up the 1754slave's hardware address, which is then available for the VLAN device. 1755 1756 Also, be aware that a similar problem can occur if all slaves 1757are released from a bond that still has one or more VLAN interfaces on 1758top of it. When a new slave is added, the bonding interface will 1759obtain its hardware address from the first slave, which might not 1760match the hardware address of the VLAN interfaces (which was 1761ultimately copied from an earlier slave). 1762 1763 There are two methods to insure that the VLAN device operates 1764with the correct hardware address if all slaves are removed from a 1765bond interface: 1766 1767 1. Remove all VLAN interfaces then recreate them 1768 1769 2. Set the bonding interface's hardware address so that it 1770matches the hardware address of the VLAN interfaces. 1771 1772 Note that changing a VLAN interface's HW address would set the 1773underlying device -- i.e. the bonding interface -- to promiscuous 1774mode, which might not be what you want. 1775 1776 17777. Link Monitoring 1778================== 1779 1780 The bonding driver at present supports two schemes for 1781monitoring a slave device's link state: the ARP monitor and the MII 1782monitor. 1783 1784 At the present time, due to implementation restrictions in the 1785bonding driver itself, it is not possible to enable both ARP and MII 1786monitoring simultaneously. 1787 17887.1 ARP Monitor Operation 1789------------------------- 1790 1791 The ARP monitor operates as its name suggests: it sends ARP 1792queries to one or more designated peer systems on the network, and 1793uses the response as an indication that the link is operating. This 1794gives some assurance that traffic is actually flowing to and from one 1795or more peers on the local network. 1796 1797 The ARP monitor relies on the device driver itself to verify 1798that traffic is flowing. In particular, the driver must keep up to 1799date the last receive time, dev->last_rx, and transmit start time, 1800dev->trans_start. If these are not updated by the driver, then the 1801ARP monitor will immediately fail any slaves using that driver, and 1802those slaves will stay down. If networking monitoring (tcpdump, etc) 1803shows the ARP requests and replies on the network, then it may be that 1804your device driver is not updating last_rx and trans_start. 1805 18067.2 Configuring Multiple ARP Targets 1807------------------------------------ 1808 1809 While ARP monitoring can be done with just one target, it can 1810be useful in a High Availability setup to have several targets to 1811monitor. In the case of just one target, the target itself may go 1812down or have a problem making it unresponsive to ARP requests. Having 1813an additional target (or several) increases the reliability of the ARP 1814monitoring. 1815 1816 Multiple ARP targets must be separated by commas as follows: 1817 1818# example options for ARP monitoring with three targets 1819alias bond0 bonding 1820options bond0 arp_interval=60 arp_ip_target=192.168.0.1,192.168.0.3,192.168.0.9 1821 1822 For just a single target the options would resemble: 1823 1824# example options for ARP monitoring with one target 1825alias bond0 bonding 1826options bond0 arp_interval=60 arp_ip_target=192.168.0.100 1827 1828 18297.3 MII Monitor Operation 1830------------------------- 1831 1832 The MII monitor monitors only the carrier state of the local 1833network interface. It accomplishes this in one of three ways: by 1834depending upon the device driver to maintain its carrier state, by 1835querying the device's MII registers, or by making an ethtool query to 1836the device. 1837 1838 If the use_carrier module parameter is 1 (the default value), 1839then the MII monitor will rely on the driver for carrier state 1840information (via the netif_carrier subsystem). As explained in the 1841use_carrier parameter information, above, if the MII monitor fails to 1842detect carrier loss on the device (e.g., when the cable is physically 1843disconnected), it may be that the driver does not support 1844netif_carrier. 1845 1846 If use_carrier is 0, then the MII monitor will first query the 1847device's (via ioctl) MII registers and check the link state. If that 1848request fails (not just that it returns carrier down), then the MII 1849monitor will make an ethtool ETHOOL_GLINK request to attempt to obtain 1850the same information. If both methods fail (i.e., the driver either 1851does not support or had some error in processing both the MII register 1852and ethtool requests), then the MII monitor will assume the link is 1853up. 1854 18558. Potential Sources of Trouble 1856=============================== 1857 18588.1 Adventures in Routing 1859------------------------- 1860 1861 When bonding is configured, it is important that the slave 1862devices not have routes that supersede routes of the master (or, 1863generally, not have routes at all). For example, suppose the bonding 1864device bond0 has two slaves, eth0 and eth1, and the routing table is 1865as follows: 1866 1867Kernel IP routing table 1868Destination Gateway Genmask Flags MSS Window irtt Iface 186910.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth0 187010.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth1 187188.8.131.52 0.0.0.0 255.255.0.0 U 40 0 0 bond0 1872127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo 1873 1874 This routing configuration will likely still update the 1875receive/transmit times in the driver (needed by the ARP monitor), but 1876may bypass the bonding driver (because outgoing traffic to, in this 1877case, another host on network 10 would use eth0 or eth1 before bond0). 1878 1879 The ARP monitor (and ARP itself) may become confused by this 1880configuration, because ARP requests (generated by the ARP monitor) 1881will be sent on one interface (bond0), but the corresponding reply 1882will arrive on a different interface (eth0). This reply looks to ARP 1883as an unsolicited ARP reply (because ARP matches replies on an 1884interface basis), and is discarded. The MII monitor is not affected 1885by the state of the routing table. 1886 1887 The solution here is simply to insure that slaves do not have 1888routes of their own, and if for some reason they must, those routes do 1889not supersede routes of their master. This should generally be the 1890case, but unusual configurations or errant manual or automatic static 1891route additions may cause trouble. 1892 18938.2 Ethernet Device Renaming 1894---------------------------- 1895 1896 On systems with network configuration scripts that do not 1897associate physical devices directly with network interface names (so 1898that the same physical device always has the same "ethX" name), it may 1899be necessary to add some special logic to config files in 1900/etc/modprobe.d/. 1901 1902 For example, given a modules.conf containing the following: 1903 1904alias bond0 bonding 1905options bond0 mode=some-mode miimon=50 1906alias eth0 tg3 1907alias eth1 tg3 1908alias eth2 e1000 1909alias eth3 e1000 1910 1911 If neither eth0 and eth1 are slaves to bond0, then when the 1912bond0 interface comes up, the devices may end up reordered. This 1913happens because bonding is loaded first, then its slave device's 1914drivers are loaded next. Since no other drivers have been loaded, 1915when the e1000 driver loads, it will receive eth0 and eth1 for its 1916devices, but the bonding configuration tries to enslave eth2 and eth3 1917(which may later be assigned to the tg3 devices). 1918 1919 Adding the following: 1920 1921add above bonding e1000 tg3 1922 1923 causes modprobe to load e1000 then tg3, in that order, when 1924bonding is loaded. This command is fully documented in the 1925modules.conf manual page. 1926 1927 On systems utilizing modprobe an equivalent problem can occur. 1928In this case, the following can be added to config files in 1929/etc/modprobe.d/ as: 1930 1931softdep bonding pre: tg3 e1000 1932 1933 This will load tg3 and e1000 modules before loading the bonding one. 1934Full documentation on this can be found in the modprobe.d and modprobe 1935manual pages. 1936 19378.3. Painfully Slow Or No Failed Link Detection By Miimon 1938--------------------------------------------------------- 1939 1940 By default, bonding enables the use_carrier option, which 1941instructs bonding to trust the driver to maintain carrier state. 1942 1943 As discussed in the options section, above, some drivers do 1944not support the netif_carrier_on/_off link state tracking system. 1945With use_carrier enabled, bonding will always see these links as up, 1946regardless of their actual state. 1947 1948 Additionally, other drivers do support netif_carrier, but do 1949not maintain it in real time, e.g., only polling the link state at 1950some fixed interval. In this case, miimon will detect failures, but 1951only after some long period of time has expired. If it appears that 1952miimon is very slow in detecting link failures, try specifying 1953use_carrier=0 to see if that improves the failure detection time. If 1954it does, then it may be that the driver checks the carrier state at a 1955fixed interval, but does not cache the MII register values (so the 1956use_carrier=0 method of querying the registers directly works). If 1957use_carrier=0 does not improve the failover, then the driver may cache 1958the registers, or the problem may be elsewhere. 1959 1960 Also, remember that miimon only checks for the device's 1961carrier state. It has no way to determine the state of devices on or 1962beyond other ports of a switch, or if a switch is refusing to pass 1963traffic while still maintaining carrier on. 1964 19659. SNMP agents 1966=============== 1967 1968 If running SNMP agents, the bonding driver should be loaded 1969before any network drivers participating in a bond. This requirement 1970is due to the interface index (ipAdEntIfIndex) being associated to 1971the first interface found with a given IP address. That is, there is 1972only one ipAdEntIfIndex for each IP address. For example, if eth0 and 1973eth1 are slaves of bond0 and the driver for eth0 is loaded before the 1974bonding driver, the interface for the IP address will be associated 1975with the eth0 interface. This configuration is shown below, the IP 1976address 192.168.1.1 has an interface index of 2 which indexes to eth0 1977in the ifDescr table (ifDescr.2). 1978 1979 interfaces.ifTable.ifEntry.ifDescr.1 = lo 1980 interfaces.ifTable.ifEntry.ifDescr.2 = eth0 1981 interfaces.ifTable.ifEntry.ifDescr.3 = eth1 1982 interfaces.ifTable.ifEntry.ifDescr.4 = eth2 1983 interfaces.ifTable.ifEntry.ifDescr.5 = eth3 1984 interfaces.ifTable.ifEntry.ifDescr.6 = bond0 1985 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 5 1986 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2 1987 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 4 1988 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1 1989 1990 This problem is avoided by loading the bonding driver before 1991any network drivers participating in a bond. Below is an example of 1992loading the bonding driver first, the IP address 192.168.1.1 is 1993correctly associated with ifDescr.2. 1994 1995 interfaces.ifTable.ifEntry.ifDescr.1 = lo 1996 interfaces.ifTable.ifEntry.ifDescr.2 = bond0 1997 interfaces.ifTable.ifEntry.ifDescr.3 = eth0 1998 interfaces.ifTable.ifEntry.ifDescr.4 = eth1 1999 interfaces.ifTable.ifEntry.ifDescr.5 = eth2 2000 interfaces.ifTable.ifEntry.ifDescr.6 = eth3
2001 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 6 2002 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2 2003 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 5 2004 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1 2005 2006 While some distributions may not report the interface name in 2007ifDescr, the association between the IP address and IfIndex remains 2008and SNMP functions such as Interface_Scan_Next will report that 2009association. 2010 201110. Promiscuous mode 2012==================== 2013 2014 When running network monitoring tools, e.g., tcpdump, it is 2015common to enable promiscuous mode on the device, so that all traffic 2016is seen (instead of seeing only traffic destined for the local host). 2017The bonding driver handles promiscuous mode changes to the bonding 2018master device (e.g., bond0), and propagates the setting to the slave 2019devices. 2020 2021 For the balance-rr, balance-xor, broadcast, and 802.3ad modes, 2022the promiscuous mode setting is propagated to all slaves. 2023 2024 For the active-backup, balance-tlb and balance-alb modes, the 2025promiscuous mode setting is propagated only to the active slave. 2026 2027 For balance-tlb mode, the active slave is the slave currently 2028receiving inbound traffic. 2029 2030 For balance-alb mode, the active slave is the slave used as a 2031"primary." This slave is used for mode-specific control traffic, for 2032sending to peers that are unassigned or if the load is unbalanced. 2033 2034 For the active-backup, balance-tlb and balance-alb modes, when 2035the active slave changes (e.g., due to a link failure), the 2036promiscuous setting will be propagated to the new active slave. 2037 203811. Configuring Bonding for High Availability 2039============================================= 2040 2041 High Availability refers to configurations that provide 2042maximum network availability by having redundant or backup devices, 2043links or switches between the host and the rest of the world. The 2044goal is to provide the maximum availability of network connectivity 2045(i.e., the network always works), even though other configurations 2046could provide higher throughput. 2047 204811.1 High Availability in a Single Switch Topology 2049-------------------------------------------------- 2050 2051 If two hosts (or a host and a single switch) are directly 2052connected via multiple physical links, then there is no availability 2053penalty to optimizing for maximum bandwidth. In this case, there is 2054only one switch (or peer), so if it fails, there is no alternative 2055access to fail over to. Additionally, the bonding load balance modes 2056support link monitoring of their members, so if individual links fail, 2057the load will be rebalanced across the remaining devices. 2058 2059 See Section 12, "Configuring Bonding for Maximum Throughput" 2060for information on configuring bonding with one peer device. 2061 206211.2 High Availability in a Multiple Switch Topology 2063---------------------------------------------------- 2064 2065 With multiple switches, the configuration of bonding and the 2066network changes dramatically. In multiple switch topologies, there is 2067a trade off between network availability and usable bandwidth. 2068 2069 Below is a sample network, configured to maximize the 2070availability of the network: 2071 2072 | | 2073 |port3 port3| 2074 +-----+----+ +-----+----+ 2075 | |port2 ISL port2| | 2076 | switch A +--------------------------+ switch B | 2077 | | | | 2078 +-----+----+ +-----++---+ 2079 |port1 port1| 2080 | +-------+ | 2081 +-------------+ host1 +---------------+ 2082 eth0 +-------+ eth1 2083 2084 In this configuration, there is a link between the two 2085switches (ISL, or inter switch link), and multiple ports connecting to 2086the outside world ("port3" on each switch). There is no technical 2087reason that this could not be extended to a third switch. 2088 208911.2.1 HA Bonding Mode Selection for Multiple Switch Topology 2090------------------------------------------------------------- 2091 2092 In a topology such as the example above, the active-backup and 2093broadcast modes are the only useful bonding modes when optimizing for 2094availability; the other modes require all links to terminate on the 2095same peer for them to behave rationally. 2096 2097active-backup: This is generally the preferred mode, particularly if 2098 the switches have an ISL and play together well. If the 2099 network configuration is such that one switch is specifically 2100 a backup switch (e.g., has lower capacity, higher cost, etc), 2101 then the primary option can be used to insure that the 2102 preferred link is always used when it is available. 2103 2104broadcast: This mode is really a special purpose mode, and is suitable 2105 only for very specific needs. For example, if the two 2106 switches are not connected (no ISL), and the networks beyond 2107 them are totally independent. In this case, if it is 2108 necessary for some specific one-way traffic to reach both 2109 independent networks, then the broadcast mode may be suitable. 2110 211111.2.2 HA Link Monitoring Selection for Multiple Switch Topology 2112---------------------------------------------------------------- 2113 2114 The choice of link monitoring ultimately depends upon your 2115switch. If the switch can reliably fail ports in response to other 2116failures, then either the MII or ARP monitors should work. For 2117example, in the above example, if the "port3" link fails at the remote 2118end, the MII monitor has no direct means to detect this. The ARP 2119monitor could be configured with a target at the remote end of port3, 2120thus detecting that failure without switch support. 2121 2122 In general, however, in a multiple switch topology, the ARP 2123monitor can provide a higher level of reliability in detecting end to 2124end connectivity failures (which may be caused by the failure of any 2125individual component to pass traffic for any reason). Additionally, 2126the ARP monitor should be configured with multiple targets (at least 2127one for each switch in the network). This will insure that, 2128regardless of which switch is active, the ARP monitor has a suitable 2129target to query. 2130 2131 Note, also, that of late many switches now support a functionality 2132generally referred to as "trunk failover." This is a feature of the 2133switch that causes the link state of a particular switch port to be set 2134down (or up) when the state of another switch port goes down (or up). 2135Its purpose is to propagate link failures from logically "exterior" ports 2136to the logically "interior" ports that bonding is able to monitor via 2137miimon. Availability and configuration for trunk failover varies by 2138switch, but this can be a viable alternative to the ARP monitor when using 2139suitable switches. 2140 214112. Configuring Bonding for Maximum Throughput 2142============================================== 2143 214412.1 Maximizing Throughput in a Single Switch Topology 2145------------------------------------------------------ 2146 2147 In a single switch configuration, the best method to maximize 2148throughput depends upon the application and network environment. The 2149various load balancing modes each have strengths and weaknesses in 2150different environments, as detailed below. 2151 2152 For this discussion, we will break down the topologies into 2153two categories. Depending upon the destination of most traffic, we 2154categorize them into either "gatewayed" or "local" configurations. 2155 2156 In a gatewayed configuration, the "switch" is acting primarily 2157as a router, and the majority of traffic passes through this router to 2158other networks. An example would be the following: 2159 2160 2161 +----------+ +----------+ 2162 | |eth0 port1| | to other networks 2163 | Host A +---------------------+ router +-------------------> 2164 | +---------------------+ | Hosts B and C are out 2165 | |eth1 port2| | here somewhere 2166 +----------+ +----------+ 2167 2168 The router may be a dedicated router device, or another host 2169acting as a gateway. For our discussion, the important point is that 2170the majority of traffic from Host A will pass through the router to 2171some other network before reaching its final destination. 2172 2173 In a gatewayed network configuration, although Host A may 2174communicate with many other systems, all of its traffic will be sent 2175and received via one other peer on the local network, the router. 2176 2177 Note that the case of two systems connected directly via 2178multiple physical links is, for purposes of configuring bonding, the 2179same as a gatewayed configuration. In that case, it happens that all 2180traffic is destined for the "gateway" itself, not some other network 2181beyond the gateway. 2182 2183 In a local configuration, the "switch" is acting primarily as 2184a switch, and the majority of traffic passes through this switch to 2185reach other stations on the same network. An example would be the 2186following: 2187 2188 +----------+ +----------+ +--------+ 2189 | |eth0 port1| +-------+ Host B | 2190 | Host A +------------+ switch |port3 +--------+ 2191 | +------------+ | +--------+ 2192 | |eth1 port2| +------------------+ Host C | 2193 +----------+ +----------+port4 +--------+ 2194 2195 2196 Again, the switch may be a dedicated switch device, or another 2197host acting as a gateway. For our discussion, the important point is 2198that the majority of traffic from Host A is destined for other hosts 2199on the same local network (Hosts B and C in the above example). 2200 2201 In summary, in a gatewayed configuration, traffic to and from 2202the bonded device will be to the same MAC level peer on the network 2203(the gateway itself, i.e., the router), regardless of its final 2204destination. In a local configuration, traffic flows directly to and 2205from the final destinations, thus, each destination (Host B, Host C) 2206will be addressed directly by their individual MAC addresses. 2207 2208 This distinction between a gatewayed and a local network 2209configuration is important because many of the load balancing modes 2210available use the MAC addresses of the local network source and 2211destination to make load balancing decisions. The behavior of each 2212mode is described below. 2213 2214 221512.1.1 MT Bonding Mode Selection for Single Switch Topology 2216----------------------------------------------------------- 2217 2218 This configuration is the easiest to set up and to understand, 2219although you will have to decide which bonding mode best suits your 2220needs. The trade offs for each mode are detailed below: 2221 2222balance-rr: This mode is the only mode that will permit a single 2223 TCP/IP connection to stripe traffic across multiple 2224 interfaces. It is therefore the only mode that will allow a 2225 single TCP/IP stream to utilize more than one interface's 2226 worth of throughput. This comes at a cost, however: the 2227 striping generally results in peer systems receiving packets out 2228 of order, causing TCP/IP's congestion control system to kick 2229 in, often by retransmitting segments. 2230 2231 It is possible to adjust TCP/IP's congestion limits by 2232 altering the net.ipv4.tcp_reordering sysctl parameter. The 2233 usual default value is 3. But keep in mind TCP stack is able 2234 to automatically increase this when it detects reorders. 2235 2236 Note that the fraction of packets that will be delivered out of 2237 order is highly variable, and is unlikely to be zero. The level 2238 of reordering depends upon a variety of factors, including the 2239 networking interfaces, the switch, and the topology of the 2240 configuration. Speaking in general terms, higher speed network 2241 cards produce more reordering (due to factors such as packet 2242 coalescing), and a "many to many" topology will reorder at a 2243 higher rate than a "many slow to one fast" configuration. 2244 2245 Many switches do not support any modes that stripe traffic 2246 (instead choosing a port based upon IP or MAC level addresses); 2247 for those devices, traffic for a particular connection flowing 2248 through the switch to a balance-rr bond will not utilize greater 2249 than one interface's worth of bandwidth. 2250 2251 If you are utilizing protocols other than TCP/IP, UDP for 2252 example, and your application can tolerate out of order 2253 delivery, then this mode can allow for single stream datagram 2254 performance that scales near linearly as interfaces are added 2255 to the bond. 2256 2257 This mode requires the switch to have the appropriate ports 2258 configured for "etherchannel" or "trunking." 2259 2260active-backup: There is not much advantage in this network topology to 2261 the active-backup mode, as the inactive backup devices are all 2262 connected to the same peer as the primary. In this case, a 2263 load balancing mode (with link monitoring) will provide the 2264 same level of network availability, but with increased 2265 available bandwidth. On the plus side, active-backup mode 2266 does not require any configuration of the switch, so it may 2267 have value if the hardware available does not support any of 2268 the load balance modes. 2269 2270balance-xor: This mode will limit traffic such that packets destined 2271 for specific peers will always be sent over the same 2272 interface. Since the destination is determined by the MAC 2273 addresses involved, this mode works best in a "local" network 2274 configuration (as described above), with destinations all on 2275 the same local network. This mode is likely to be suboptimal 2276 if all your traffic is passed through a single router (i.e., a 2277 "gatewayed" network configuration, as described above). 2278 2279 As with balance-rr, the switch ports need to be configured for 2280 "etherchannel" or "trunking." 2281 2282broadcast: Like active-backup, there is not much advantage to this 2283 mode in this type of network topology. 2284 2285802.3ad: This mode can be a good choice for this type of network 2286 topology. The 802.3ad mode is an IEEE standard, so all peers 2287 that implement 802.3ad should interoperate well. The 802.3ad 2288 protocol includes automatic configuration of the aggregates, 2289 so minimal manual configuration of the switch is needed 2290 (typically only to designate that some set of devices is 2291 available for 802.3ad). The 802.3ad standard also mandates 2292 that frames be delivered in order (within certain limits), so 2293 in general single connections will not see misordering of 2294 packets. The 802.3ad mode does have some drawbacks: the 2295 standard mandates that all devices in the aggregate operate at 2296 the same speed and duplex. Also, as with all bonding load 2297 balance modes other than balance-rr, no single connection will 2298 be able to utilize more than a single interface's worth of 2299 bandwidth. 2300 2301 Additionally, the linux bonding 802.3ad implementation 2302 distributes traffic by peer (using an XOR of MAC addresses 2303 and packet type ID), so in a "gatewayed" configuration, all 2304 outgoing traffic will generally use the same device. Incoming 2305 traffic may also end up on a single device, but that is 2306 dependent upon the balancing policy of the peer's 8023.ad 2307 implementation. In a "local" configuration, traffic will be 2308 distributed across the devices in the bond. 2309 2310 Finally, the 802.3ad mode mandates the use of the MII monitor, 2311 therefore, the ARP monitor is not available in this mode. 2312 2313balance-tlb: The balance-tlb mode balances outgoing traffic by peer. 2314 Since the balancing is done according to MAC address, in a 2315 "gatewayed" configuration (as described above), this mode will 2316 send all traffic across a single device. However, in a 2317 "local" network configuration, this mode balances multiple 2318 local network peers across devices in a vaguely intelligent 2319 manner (not a simple XOR as in balance-xor or 802.3ad mode), 2320 so that mathematically unlucky MAC addresses (i.e., ones that 2321 XOR to the same value) will not all "bunch up" on a single 2322 interface. 2323 2324 Unlike 802.3ad, interfaces may be of differing speeds, and no 2325 special switch configuration is required. On the down side, 2326 in this mode all incoming traffic arrives over a single 2327 interface, this mode requires certain ethtool support in the 2328 network device driver of the slave interfaces, and the ARP 2329 monitor is not available. 2330 2331balance-alb: This mode is everything that balance-tlb is, and more. 2332 It has all of the features (and restrictions) of balance-tlb, 2333 and will also balance incoming traffic from local network 2334 peers (as described in the Bonding Module Options section, 2335 above). 2336 2337 The only additional down side to this mode is that the network 2338 device driver must support changing the hardware address while 2339 the device is open. 2340 234112.1.2 MT Link Monitoring for Single Switch Topology 2342---------------------------------------------------- 2343 2344 The choice of link monitoring may largely depend upon which 2345mode you choose to use. The more advanced load balancing modes do not 2346support the use of the ARP monitor, and are thus restricted to using 2347the MII monitor (which does not provide as high a level of end to end 2348assurance as the ARP monitor). 2349 235012.2 Maximum Throughput in a Multiple Switch Topology 2351----------------------------------------------------- 2352 2353 Multiple switches may be utilized to optimize for throughput 2354when they are configured in parallel as part of an isolated network 2355between two or more systems, for example: 2356 2357 +-----------+ 2358 | Host A | 2359 +-+---+---+-+ 2360 | | | 2361 +--------+ | +---------+ 2362 | | | 2363 +------+---+ +-----+----+ +-----+----+ 2364 | Switch A | | Switch B | | Switch C | 2365 +------+---+ +-----+----+ +-----+----+ 2366 | | | 2367 +--------+ | +---------+ 2368 | | | 2369 +-+---+---+-+ 2370 | Host B | 2371 +-----------+ 2372 2373 In this configuration, the switches are isolated from one 2374another. One reason to employ a topology such as this is for an 2375isolated network with many hosts (a cluster configured for high 2376performance, for example), using multiple smaller switches can be more 2377cost effective than a single larger switch, e.g., on a network with 24 2378hosts, three 24 port switches can be significantly less expensive than 2379a single 72 port switch. 2380 2381 If access beyond the network is required, an individual host 2382can be equipped with an additional network device connected to an 2383external network; this host then additionally acts as a gateway. 2384 238512.2.1 MT Bonding Mode Selection for Multiple Switch Topology 2386------------------------------------------------------------- 2387 2388 In actual practice, the bonding mode typically employed in 2389configurations of this type is balance-rr. Historically, in this 2390network configuration, the usual caveats about out of order packet 2391delivery are mitigated by the use of network adapters that do not do 2392any kind of packet coalescing (via the use of NAPI, or because the 2393device itself does not generate interrupts until some number of 2394packets has arrived). When employed in this fashion, the balance-rr 2395mode allows individual connections between two hosts to effectively 2396utilize greater than one interface's bandwidth. 2397 239812.2.2 MT Link Monitoring for Multiple Switch Topology 2399------------------------------------------------------ 2400 2401 Again, in actual practice, the MII monitor is most often used 2402in this configuration, as performance is given preference over 2403availability. The ARP monitor will function in this topology, but its 2404advantages over the MII monitor are mitigated by the volume of probes 2405needed as the number of systems involved grows (remember that each 2406host in the network is configured with bonding). 2407 240813. Switch Behavior Issues 2409========================== 2410 241113.1 Link Establishment and Failover Delays 2412------------------------------------------- 2413 2414 Some switches exhibit undesirable behavior with regard to the 2415timing of link up and down reporting by the switch. 2416 2417 First, when a link comes up, some switches may indicate that 2418the link is up (carrier available), but not pass traffic over the 2419interface for some period of time. This delay is typically due to 2420some type of autonegotiation or routing protocol, but may also occur 2421during switch initialization (e.g., during recovery after a switch 2422failure). If you find this to be a problem, specify an appropriate 2423value to the updelay bonding module option to delay the use of the 2424relevant interface(s). 2425 2426 Second, some switches may "bounce" the link state one or more 2427times while a link is changing state. This occurs most commonly while 2428the switch is initializing. Again, an appropriate updelay value may 2429help. 2430 2431 Note that when a bonding interface has no active links, the 2432driver will immediately reuse the first link that goes up, even if the 2433updelay parameter has been specified (the updelay is ignored in this 2434case). If there are slave interfaces waiting for the updelay timeout 2435to expire, the interface that first went into that state will be 2436immediately reused. This reduces down time of the network if the 2437value of updelay has been overestimated, and since this occurs only in 2438cases with no connectivity, there is no additional penalty for 2439ignoring the updelay. 2440 2441 In addition to the concerns about switch timings, if your 2442switches take a long time to go into backup mode, it may be desirable 2443to not activate a backup interface immediately after a link goes down. 2444Failover may be delayed via the downdelay bonding module option. 2445 244613.2 Duplicated Incoming Packets 2447-------------------------------- 2448 2449 NOTE: Starting with version 3.0.2, the bonding driver has logic to 2450suppress duplicate packets, which should largely eliminate this problem. 2451The following description is kept for reference. 2452 2453 It is not uncommon to observe a short burst of duplicated 2454traffic when the bonding device is first used, or after it has been 2455idle for some period of time. This is most easily observed by issuing 2456a "ping" to some other host on the network, and noticing that the 2457output from ping flags duplicates (typically one per slave). 2458 2459 For example, on a bond in active-backup mode with five slaves 2460all connected to one switch, the output may appear as follows: 2461 2462# ping -n 10.0.4.2 2463PING 10.0.4.2 (10.0.4.2) from 10.0.3.10 : 56(84) bytes of data. 246464 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.7 ms 246564 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!) 246664 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!) 246764 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!) 246864 bytes from 10.0.4.2: icmp_seq=1 ttl=64 time=13.8 ms (DUP!) 246964 bytes from 10.0.4.2: icmp_seq=2 ttl=64 time=0.216 ms 247064 bytes from 10.0.4.2: icmp_seq=3 ttl=64 time=0.267 ms 247164 bytes from 10.0.4.2: icmp_seq=4 ttl=64 time=0.222 ms 2472 2473 This is not due to an error in the bonding driver, rather, it 2474is a side effect of how many switches update their MAC forwarding 2475tables. Initially, the switch does not associate the MAC address in 2476the packet with a particular switch port, and so it may send the 2477traffic to all ports until its MAC forwarding table is updated. Since 2478the interfaces attached to the bond may occupy multiple ports on a 2479single switch, when the switch (temporarily) floods the traffic to all 2480ports, the bond device receives multiple copies of the same packet 2481(one per slave device). 2482 2483 The duplicated packet behavior is switch dependent, some 2484switches exhibit this, and some do not. On switches that display this 2485behavior, it can be induced by clearing the MAC forwarding table (on 2486most Cisco switches, the privileged command "clear mac address-table 2487dynamic" will accomplish this). 2488 248914. Hardware Specific Considerations 2490==================================== 2491 2492 This section contains additional information for configuring 2493bonding on specific hardware platforms, or for interfacing bonding 2494with particular switches or other devices. 2495 249614.1 IBM BladeCenter 2497-------------------- 2498 2499 This applies to the JS20 and similar systems. 2500 2501 On the JS20 blades, the bonding driver supports only 2502balance-rr, active-backup, balance-tlb and balance-alb modes. This is 2503largely due to the network topology inside the BladeCenter, detailed 2504below. 2505 2506JS20 network adapter information 2507-------------------------------- 2508 2509 All JS20s come with two Broadcom Gigabit Ethernet ports 2510integrated on the planar (that's "motherboard" in IBM-speak). In the 2511BladeCenter chassis, the eth0 port of all JS20 blades is hard wired to 2512I/O Module #1; similarly, all eth1 ports are wired to I/O Module #2. 2513An add-on Broadcom daughter card can be installed on a JS20 to provide 2514two more Gigabit Ethernet ports. These ports, eth2 and eth3, are 2515wired to I/O Modules 3 and 4, respectively. 2516 2517 Each I/O Module may contain either a switch or a passthrough 2518module (which allows ports to be directly connected to an external 2519switch). Some bonding modes require a specific BladeCenter internal 2520network topology in order to function; these are detailed below. 2521 2522 Additional BladeCenter-specific networking information can be 2523found in two IBM Redbooks (www.ibm.com/redbooks): 2524 2525"IBM eServer BladeCenter Networking Options" 2526"IBM eServer BladeCenter Layer 2-7 Network Switching" 2527 2528BladeCenter networking configuration 2529------------------------------------ 2530 2531 Because a BladeCenter can be configured in a very large number 2532of ways, this discussion will be confined to describing basic 2533configurations. 2534 2535 Normally, Ethernet Switch Modules (ESMs) are used in I/O 2536modules 1 and 2. In this configuration, the eth0 and eth1 ports of a 2537JS20 will be connected to different internal switches (in the 2538respective I/O modules). 2539 2540 A passthrough module (OPM or CPM, optical or copper, 2541passthrough module) connects the I/O module directly to an external 2542switch. By using PMs in I/O module #1 and #2, the eth0 and eth1 2543interfaces of a JS20 can be redirected to the outside world and 2544connected to a common external switch. 2545 2546 Depending upon the mix of ESMs and PMs, the network will 2547appear to bonding as either a single switch topology (all PMs) or as a 2548multiple switch topology (one or more ESMs, zero or more PMs). It is 2549also possible to connect ESMs together, resulting in a configuration 2550much like the example in "High Availability in a Multiple Switch 2551Topology," above. 2552 2553Requirements for specific modes 2554------------------------------- 2555 2556 The balance-rr mode requires the use of passthrough modules 2557for devices in the bond, all connected to an common external switch. 2558That switch must be configured for "etherchannel" or "trunking" on the 2559appropriate ports, as is usual for balance-rr. 2560 2561 The balance-alb and balance-tlb modes will function with 2562either switch modules or passthrough modules (or a mix). The only 2563specific requirement for these modes is that all network interfaces 2564must be able to reach all destinations for traffic sent over the 2565bonding device (i.e., the network must converge at some point outside 2566the BladeCenter). 2567 2568 The active-backup mode has no additional requirements. 2569 2570Link monitoring issues 2571---------------------- 2572 2573 When an Ethernet Switch Module is in place, only the ARP 2574monitor will reliably detect link loss to an external switch. This is 2575nothing unusual, but examination of the BladeCenter cabinet would 2576suggest that the "external" network ports are the ethernet ports for 2577the system, when it fact there is a switch between these "external" 2578ports and the devices on the JS20 system itself. The MII monitor is 2579only able to detect link failures between the ESM and the JS20 system. 2580 2581 When a passthrough module is in place, the MII monitor does 2582detect failures to the "external" port, which is then directly 2583connected to the JS20 system. 2584 2585Other concerns 2586-------------- 2587 2588 The Serial Over LAN (SoL) link is established over the primary 2589ethernet (eth0) only, therefore, any loss of link to eth0 will result 2590in losing your SoL connection. It will not fail over with other 2591network traffic, as the SoL system is beyond the control of the 2592bonding driver. 2593 2594 It may be desirable to disable spanning tree on the switch 2595(either the internal Ethernet Switch Module, or an external switch) to 2596avoid fail-over delay issues when using bonding. 2597 2598 259915. Frequently Asked Questions 2600============================== 2601 26021. Is it SMP safe? 2603 2604 Yes. The old 2.0.xx channel bonding patch was not SMP safe. 2605The new driver was designed to be SMP safe from the start. 2606 26072. What type of cards will work with it? 2608 2609 Any Ethernet type cards (you can even mix cards - a Intel 2610EtherExpress PRO/100 and a 3com 3c905b, for example). For most modes, 2611devices need not be of the same speed. 2612 2613 Starting with version 3.2.1, bonding also supports Infiniband 2614slaves in active-backup mode. 2615 26163. How many bonding devices can I have? 2617 2618 There is no limit. 2619 26204. How many slaves can a bonding device have? 2621 2622 This is limited only by the number of network interfaces Linux 2623supports and/or the number of network cards you can place in your 2624system. 2625 26265. What happens when a slave link dies? 2627 2628 If link monitoring is enabled, then the failing device will be 2629disabled. The active-backup mode will fail over to a backup link, and 2630other modes will ignore the failed link. The link will continue to be 2631monitored, and should it recover, it will rejoin the bond (in whatever 2632manner is appropriate for the mode). See the sections on High 2633Availability and the documentation for each mode for additional 2634information. 2635 2636 Link monitoring can be enabled via either the miimon or 2637arp_interval parameters (described in the module parameters section, 2638above). In general, miimon monitors the carrier state as sensed by 2639the underlying network device, and the arp monitor (arp_interval) 2640monitors connectivity to another host on the local network. 2641 2642 If no link monitoring is configured, the bonding driver will 2643be unable to detect link failures, and will assume that all links are 2644always available. This will likely result in lost packets, and a 2645resulting degradation of performance. The precise performance loss 2646depends upon the bonding mode and network configuration. 2647 26486. Can bonding be used for High Availability? 2649 2650 Yes. See the section on High Availability for details. 2651 26527. Which switches/systems does it work with? 2653 2654 The full answer to this depends upon the desired mode. 2655 2656 In the basic balance modes (balance-rr and balance-xor), it 2657works with any system that supports etherchannel (also called 2658trunking). Most managed switches currently available have such 2659support, and many unmanaged switches as well. 2660 2661 The advanced balance modes (balance-tlb and balance-alb) do 2662not have special switch requirements, but do need device drivers that 2663support specific features (described in the appropriate section under 2664module parameters, above). 2665 2666 In 802.3ad mode, it works with systems that support IEEE 2667802.3ad Dynamic Link Aggregation. Most managed and many unmanaged 2668switches currently available support 802.3ad. 2669 2670 The active-backup mode should work with any Layer-II switch. 2671 26728. Where does a bonding device get its MAC address from? 2673 2674 When using slave devices that have fixed MAC addresses, or when 2675the fail_over_mac option is enabled, the bonding device's MAC address is 2676the MAC address of the active slave. 2677 2678 For other configurations, if not explicitly configured (with 2679ifconfig or ip link), the MAC address of the bonding device is taken from 2680its first slave device. This MAC address is then passed to all following 2681slaves and remains persistent (even if the first slave is removed) until 2682the bonding device is brought down or reconfigured. 2683 2684 If you wish to change the MAC address, you can set it with 2685ifconfig or ip link: 2686 2687# ifconfig bond0 hw ether 00:11:22:33:44:55 2688 2689# ip link set bond0 address 66:77:88:99:aa:bb 2690 2691 The MAC address can be also changed by bringing down/up the 2692device and then changing its slaves (or their order): 2693 2694# ifconfig bond0 down ; modprobe -r bonding 2695# ifconfig bond0 .... up 2696# ifenslave bond0 eth... 2697 2698 This method will automatically take the address from the next 2699slave that is added. 2700 2701 To restore your slaves' MAC addresses, you need to detach them 2702from the bond (`ifenslave -d bond0 eth0'). The bonding driver will 2703then restore the MAC addresses that the slaves had before they were 2704enslaved. 2705 270616. Resources and Links 2707======================= 2708 2709 The latest version of the bonding driver can be found in the latest 2710version of the linux kernel, found on http://kernel.org 2711 2712 The latest version of this document can be found in the latest kernel 2713source (named Documentation/networking/bonding.txt). 2714 2715 Discussions regarding the usage of the bonding driver take place on the 2716bonding-devel mailing list, hosted at sourceforge.net. If you have questions or 2717problems, post them to the list. The list address is: 2718 firstname.lastname@example.org 2720 2721 The administrative interface (to subscribe or unsubscribe) can 2722be found at: 2723 2724https://lists.sourceforge.net/lists/listinfo/bonding-devel 2725 2726 Discussions regarding the development of the bonding driver take place 2727on the main Linux network mailing list, hosted at vger.kernel.org. The list 2728address is: 2729 email@example.com 2731 2732 The administrative interface (to subscribe or unsubscribe) can 2733be found at: 2734 2735http://vger.kernel.org/vger-lists.html#netdev 2736 2737Donald Becker's Ethernet Drivers and diag programs may be found at : 2738 - http://web.archive.org/web/*/http://www.scyld.com/network/ 2739 2740You will also find a lot of information regarding Ethernet, NWay, MII, 2741etc. at www.scyld.com. 2742 2743-- END -- 2744