linux/Documentation/networking/ixgb.txt
<<
>>
Prefs
   1Linux Base Driver for 10 Gigabit Intel(R) Network Connection
   2=============================================================
   3
   4October 9, 2007
   5
   6
   7Contents
   8========
   9
  10- In This Release
  11- Identifying Your Adapter
  12- Building and Installation
  13- Command Line Parameters
  14- Improving Performance
  15- Additional Configurations
  16- Known Issues/Troubleshooting
  17- Support
  18
  19
  20
  21In This Release
  22===============
  23
  24This file describes the ixgb Linux Base Driver for the 10 Gigabit Intel(R)
  25Network Connection.  This driver includes support for Itanium(R)2-based
  26systems.
  27
  28For questions related to hardware requirements, refer to the documentation
  29supplied with your 10 Gigabit adapter.  All hardware requirements listed apply
  30to use with Linux.
  31
  32The following features are available in this kernel:
  33 - Native VLANs
  34 - Channel Bonding (teaming)
  35 - SNMP
  36
  37Channel Bonding documentation can be found in the Linux kernel source:
  38/Documentation/networking/bonding.txt
  39
  40The driver information previously displayed in the /proc filesystem is not
  41supported in this release.  Alternatively, you can use ethtool (version 1.6
  42or later), lspci, and ifconfig to obtain the same information.
  43
  44Instructions on updating ethtool can be found in the section "Additional
  45Configurations" later in this document.
  46
  47
  48Identifying Your Adapter
  49========================
  50
  51The following Intel network adapters are compatible with the drivers in this
  52release:
  53
  54Controller  Adapter Name                 Physical Layer
  55----------  ------------                 --------------
  5682597EX     Intel(R) PRO/10GbE LR/SR/CX4 10G Base-LR (1310 nm optical fiber)
  57            Server Adapters              10G Base-SR (850 nm optical fiber)
  58                                         10G Base-CX4(twin-axial copper cabling)
  59
  60For more information on how to identify your adapter, go to the Adapter &
  61Driver ID Guide at:
  62
  63    http://support.intel.com/support/network/sb/CS-012904.htm
  64
  65
  66Building and Installation
  67=========================
  68
  69select m for "Intel(R) PRO/10GbE support" located at:
  70      Location:
  71        -> Device Drivers
  72          -> Network device support (NETDEVICES [=y])
  73            -> Ethernet (10000 Mbit) (NETDEV_10000 [=y])
  741. make modules && make modules_install
  75
  762. Load the module:
  77
  78    modprobe ixgb <parameter>=<value>
  79
  80   The insmod command can be used if the full
  81   path to the driver module is specified.  For example:
  82
  83     insmod /lib/modules/<KERNEL VERSION>/kernel/drivers/net/ixgb/ixgb.ko
  84
  85   With 2.6 based kernels also make sure that older ixgb drivers are
  86   removed from the kernel, before loading the new module:
  87
  88     rmmod ixgb; modprobe ixgb
  89
  903. Assign an IP address to the interface by entering the following, where
  91   x is the interface number:
  92
  93     ifconfig ethx <IP_address>
  94
  954. Verify that the interface works. Enter the following, where <IP_address>
  96   is the IP address for another machine on the same subnet as the interface
  97   that is being tested:
  98
  99     ping  <IP_address>
 100
 101
 102Command Line Parameters
 103=======================
 104
 105If the driver is built as a module, the  following optional parameters are
 106used by entering them on the command line with the modprobe command using
 107this syntax:
 108
 109     modprobe ixgb [<option>=<VAL1>,<VAL2>,...]
 110
 111For example, with two 10GbE PCI adapters, entering:
 112
 113     modprobe ixgb TxDescriptors=80,128
 114
 115loads the ixgb driver with 80 TX resources for the first adapter and 128 TX
 116resources for the second adapter.
 117
 118The default value for each parameter is generally the recommended setting,
 119unless otherwise noted.
 120
 121FlowControl
 122Valid Range: 0-3 (0=none, 1=Rx only, 2=Tx only, 3=Rx&Tx)
 123Default: Read from the EEPROM
 124         If EEPROM is not detected, default is 1
 125    This parameter controls the automatic generation(Tx) and response(Rx) to
 126    Ethernet PAUSE frames.  There are hardware bugs associated with enabling
 127    Tx flow control so beware.
 128
 129RxDescriptors
 130Valid Range: 64-512
 131Default Value: 512
 132    This value is the number of receive descriptors allocated by the driver.
 133    Increasing this value allows the driver to buffer more incoming packets.
 134    Each descriptor is 16 bytes.  A receive buffer is also allocated for
 135    each descriptor and can be either 2048, 4056, 8192, or 16384 bytes,
 136    depending on the MTU setting.  When the MTU size is 1500 or less, the
 137    receive buffer size is 2048 bytes. When the MTU is greater than 1500 the
 138    receive buffer size will be either 4056, 8192, or 16384 bytes.  The
 139    maximum MTU size is 16114.
 140
 141RxIntDelay
 142Valid Range: 0-65535 (0=off)
 143Default Value: 72
 144    This value delays the generation of receive interrupts in units of
 145    0.8192 microseconds.  Receive interrupt reduction can improve CPU
 146    efficiency if properly tuned for specific network traffic.  Increasing
 147    this value adds extra latency to frame reception and can end up
 148    decreasing the throughput of TCP traffic.  If the system is reporting
 149    dropped receives, this value may be set too high, causing the driver to
 150    run out of available receive descriptors.
 151
 152TxDescriptors
 153Valid Range: 64-4096
 154Default Value: 256
 155    This value is the number of transmit descriptors allocated by the driver.
 156    Increasing this value allows the driver to queue more transmits.  Each
 157    descriptor is 16 bytes.
 158
 159XsumRX
 160Valid Range: 0-1
 161Default Value: 1
 162    A value of '1' indicates that the driver should enable IP checksum
 163    offload for received packets (both UDP and TCP) to the adapter hardware.
 164
 165
 166Improving Performance
 167=====================
 168
 169With the 10 Gigabit server adapters, the default Linux configuration will
 170very likely limit the total available throughput artificially.  There is a set
 171of configuration changes that, when applied together, will increase the ability
 172of Linux to transmit and receive data.  The following enhancements were
 173originally acquired from settings published at http://www.spec.org/web99/ for
 174various submitted results using Linux.
 175
 176NOTE: These changes are only suggestions, and serve as a starting point for
 177      tuning your network performance.
 178
 179The changes are made in three major ways, listed in order of greatest effect:
 180- Use ifconfig to modify the mtu (maximum transmission unit) and the txqueuelen
 181  parameter.
 182- Use sysctl to modify /proc parameters (essentially kernel tuning)
 183- Use setpci to modify the MMRBC field in PCI-X configuration space to increase
 184  transmit burst lengths on the bus.
 185
 186NOTE: setpci modifies the adapter's configuration registers to allow it to read
 187up to 4k bytes at a time (for transmits).  However, for some systems the
 188behavior after modifying this register may be undefined (possibly errors of
 189some kind).  A power-cycle, hard reset or explicitly setting the e6 register
 190back to 22 (setpci -d 8086:1a48 e6.b=22) may be required to get back to a
 191stable configuration.
 192
 193- COPY these lines and paste them into ixgb_perf.sh:
 194#!/bin/bash
 195echo "configuring network performance , edit this file to change the interface
 196or device ID of 10GbE card"
 197# set mmrbc to 4k reads, modify only Intel 10GbE device IDs
 198# replace 1a48 with appropriate 10GbE device's ID installed on the system,
 199# if needed.
 200setpci -d 8086:1a48 e6.b=2e
 201# set the MTU (max transmission unit) - it requires your switch and clients
 202# to change as well.
 203# set the txqueuelen
 204# your ixgb adapter should be loaded as eth1 for this to work, change if needed
 205ifconfig eth1 mtu 9000 txqueuelen 1000 up
 206# call the sysctl utility to modify /proc/sys entries
 207sysctl -p ./sysctl_ixgb.conf
 208- END ixgb_perf.sh
 209
 210- COPY these lines and paste them into sysctl_ixgb.conf:
 211# some of the defaults may be different for your kernel
 212# call this file with sysctl -p <this file>
 213# these are just suggested values that worked well to increase throughput in
 214# several network benchmark tests, your mileage may vary
 215
 216### IPV4 specific settings
 217# turn TCP timestamp support off, default 1, reduces CPU use
 218net.ipv4.tcp_timestamps = 0
 219# turn SACK support off, default on
 220# on systems with a VERY fast bus -> memory interface this is the big gainer
 221net.ipv4.tcp_sack = 0
 222# set min/default/max TCP read buffer, default 4096 87380 174760
 223net.ipv4.tcp_rmem = 10000000 10000000 10000000
 224# set min/pressure/max TCP write buffer, default 4096 16384 131072
 225net.ipv4.tcp_wmem = 10000000 10000000 10000000
 226# set min/pressure/max TCP buffer space, default 31744 32256 32768
 227net.ipv4.tcp_mem = 10000000 10000000 10000000
 228
 229### CORE settings (mostly for socket and UDP effect)
 230# set maximum receive socket buffer size, default 131071
 231net.core.rmem_max = 524287
 232# set maximum send socket buffer size, default 131071
 233net.core.wmem_max = 524287
 234# set default receive socket buffer size, default 65535
 235net.core.rmem_default = 524287
 236# set default send socket buffer size, default 65535
 237net.core.wmem_default = 524287
 238# set maximum amount of option memory buffers, default 10240
 239net.core.optmem_max = 524287
 240# set number of unprocessed input packets before kernel starts dropping them; default 300
 241net.core.netdev_max_backlog = 300000
 242- END sysctl_ixgb.conf
 243
 244Edit the ixgb_perf.sh script if necessary to change eth1 to whatever interface
 245your ixgb driver is using and/or replace '1a48' with appropriate 10GbE device's
 246ID installed on the system.
 247
 248NOTE: Unless these scripts are added to the boot process, these changes will
 249      only last only until the next system reboot.
 250
 251
 252Resolving Slow UDP Traffic
 253--------------------------
 254If your server does not seem to be able to receive UDP traffic as fast as it
 255can receive TCP traffic, it could be because Linux, by default, does not set
 256the network stack buffers as large as they need to be to support high UDP
 257transfer rates.  One way to alleviate this problem is to allow more memory to
 258be used by the IP stack to store incoming data.
 259
 260For instance, use the commands:
 261    sysctl -w net.core.rmem_max=262143
 262and
 263    sysctl -w net.core.rmem_default=262143
 264to increase the read buffer memory max and default to 262143 (256k - 1) from
 265defaults of max=131071 (128k - 1) and default=65535 (64k - 1).  These variables
 266will increase the amount of memory used by the network stack for receives, and
 267can be increased significantly more if necessary for your application.
 268
 269
 270Additional Configurations
 271=========================
 272
 273  Configuring the Driver on Different Distributions
 274  -------------------------------------------------
 275  Configuring a network driver to load properly when the system is started is
 276  distribution dependent. Typically, the configuration process involves adding
 277  an alias line to files in /etc/modprobe.d/ as well as editing other system
 278  startup scripts and/or configuration files.  Many popular Linux distributions
 279  ship with tools to make these changes for you.  To learn the proper way to
 280  configure a network device for your system, refer to your distribution
 281  documentation.  If during this process you are asked for the driver or module
 282  name, the name for the Linux Base Driver for the Intel 10GbE Family of
 283  Adapters is ixgb.
 284
 285  Viewing Link Messages
 286  ---------------------
 287  Link messages will not be displayed to the console if the distribution is
 288  restricting system messages. In order to see network driver link messages on
 289  your console, set dmesg to eight by entering the following:
 290
 291       dmesg -n 8
 292
 293  NOTE: This setting is not saved across reboots.
 294
 295
 296  Jumbo Frames
 297  ------------
 298  The driver supports Jumbo Frames for all adapters. Jumbo Frames support is
 299  enabled by changing the MTU to a value larger than the default of 1500.
 300  The maximum value for the MTU is 16114.  Use the ifconfig command to
 301  increase the MTU size.  For example:
 302
 303        ifconfig ethx mtu 9000 up
 304
 305  The maximum MTU setting for Jumbo Frames is 16114.  This value coincides
 306  with the maximum Jumbo Frames size of 16128.
 307
 308
 309  Ethtool
 310  -------
 311  The driver utilizes the ethtool interface for driver configuration and
 312  diagnostics, as well as displaying statistical information.  The ethtool
 313  version 1.6 or later is required for this functionality.
 314
 315  The latest release of ethtool can be found from
 316  http://ftp.kernel.org/pub/software/network/ethtool/
 317
 318  NOTE: The ethtool version 1.6 only supports a limited set of ethtool options.
 319        Support for a more complete ethtool feature set can be enabled by
 320        upgrading to the latest version.
 321
 322
 323  NAPI
 324  ----
 325
 326  NAPI (Rx polling mode) is supported in the ixgb driver.  NAPI is enabled
 327  or disabled based on the configuration of the kernel.  see CONFIG_IXGB_NAPI
 328
 329  See www.cyberus.ca/~hadi/usenix-paper.tgz for more information on NAPI.
 330
 331
 332Known Issues/Troubleshooting
 333============================
 334
 335  NOTE: After installing the driver, if your Intel Network Connection is not
 336  working, verify in the "In This Release" section of the readme that you have
 337  installed the correct driver.
 338
 339  Intel(R) PRO/10GbE CX4 Server Adapter Cable Interoperability Issue with
 340  Fujitsu XENPAK Module in SmartBits Chassis
 341  ---------------------------------------------------------------------
 342  Excessive CRC errors may be observed if the Intel(R) PRO/10GbE CX4
 343  Server adapter is connected to a Fujitsu XENPAK CX4 module in a SmartBits
 344  chassis using 15 m/24AWG cable assemblies manufactured by Fujitsu or Leoni.
 345  The CRC errors may be received either by the Intel(R) PRO/10GbE CX4
 346  Server adapter or the SmartBits. If this situation occurs using a different
 347  cable assembly may resolve the issue.
 348
 349  CX4 Server Adapter Cable Interoperability Issues with HP Procurve 3400cl
 350  Switch Port
 351  ------------------------------------------------------------------------
 352  Excessive CRC errors may be observed if the Intel(R) PRO/10GbE CX4 Server
 353  adapter is connected to an HP Procurve 3400cl switch port using short cables
 354  (1 m or shorter). If this situation occurs, using a longer cable may resolve
 355  the issue.
 356
 357  Excessive CRC errors may be observed using Fujitsu 24AWG cable assemblies that
 358  Are 10 m or longer or where using a Leoni 15 m/24AWG cable assembly. The CRC
 359  errors may be received either by the CX4 Server adapter or at the switch. If
 360  this situation occurs, using a different cable assembly may resolve the issue.
 361
 362
 363  Jumbo Frames System Requirement
 364  -------------------------------
 365  Memory allocation failures have been observed on Linux systems with 64 MB
 366  of RAM or less that are running Jumbo Frames.  If you are using Jumbo
 367  Frames, your system may require more than the advertised minimum
 368  requirement of 64 MB of system memory.
 369
 370
 371  Performance Degradation with Jumbo Frames
 372  -----------------------------------------
 373  Degradation in throughput performance may be observed in some Jumbo frames
 374  environments.  If this is observed, increasing the application's socket buffer
 375  size and/or increasing the /proc/sys/net/ipv4/tcp_*mem entry values may help.
 376  See the specific application manual and /usr/src/linux*/Documentation/
 377  networking/ip-sysctl.txt for more details.
 378
 379
 380  Allocating Rx Buffers when Using Jumbo Frames
 381  ---------------------------------------------
 382  Allocating Rx buffers when using Jumbo Frames on 2.6.x kernels may fail if
 383  the available memory is heavily fragmented. This issue may be seen with PCI-X
 384  adapters or with packet split disabled. This can be reduced or eliminated
 385  by changing the amount of available memory for receive buffer allocation, by
 386  increasing /proc/sys/vm/min_free_kbytes.
 387
 388
 389  Multiple Interfaces on Same Ethernet Broadcast Network
 390  ------------------------------------------------------
 391  Due to the default ARP behavior on Linux, it is not possible to have
 392  one system on two IP networks in the same Ethernet broadcast domain
 393  (non-partitioned switch) behave as expected.  All Ethernet interfaces
 394  will respond to IP traffic for any IP address assigned to the system.
 395  This results in unbalanced receive traffic.
 396
 397  If you have multiple interfaces in a server, do either of the following:
 398
 399  - Turn on ARP filtering by entering:
 400      echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter
 401
 402  - Install the interfaces in separate broadcast domains - either in
 403    different switches or in a switch partitioned to VLANs.
 404
 405
 406  UDP Stress Test Dropped Packet Issue
 407  --------------------------------------
 408  Under small packets UDP stress test with 10GbE driver, the Linux system
 409  may drop UDP packets due to the fullness of socket buffers. You may want
 410  to change the driver's Flow Control variables to the minimum value for
 411  controlling packet reception.
 412
 413
 414  Tx Hangs Possible Under Stress
 415  ------------------------------
 416  Under stress conditions, if TX hangs occur, turning off TSO
 417  "ethtool -K eth0 tso off" may resolve the problem.
 418
 419
 420Support
 421=======
 422
 423For general information, go to the Intel support website at:
 424
 425    http://support.intel.com
 426
 427or the Intel Wired Networking project hosted by Sourceforge at:
 428
 429    http://sourceforge.net/projects/e1000
 430
 431If an issue is identified with the released source code on the supported
 432kernel with a supported adapter, email the specific information related
 433to the issue to e1000-devel@lists.sf.net
 434
lxr.linux.no kindly hosted by Redpill Linpro AS, provider of Linux consulting and operations services since 1995.