1                          The Linux RapidIO Subsystem
   5The RapidIO standard is a packet-based fabric interconnect standard designed for
   6use in embedded systems. Development of the RapidIO standard is directed by the
   7RapidIO Trade Association (RTA). The current version of the RapidIO specification
   8is publicly available for download from the RTA web-site [1].
  10This document describes the basics of the Linux RapidIO subsystem and provides
  11information on its major components.
  131 Overview
  16Because the RapidIO subsystem follows the Linux device model it is integrated
  17into the kernel similarly to other buses by defining RapidIO-specific device and
  18bus types and registering them within the device model.
  20The Linux RapidIO subsystem is architecture independent and therefore defines
  21architecture-specific interfaces that provide support for common RapidIO
  22subsystem operations.
  242. Core Components
  27A typical RapidIO network is a combination of endpoints and switches.
  28Each of these components is represented in the subsystem by an associated data
  29structure. The core logical components of the RapidIO subsystem are defined
  30in include/linux/rio.h file.
  322.1 Master Port
  34A master port (or mport) is a RapidIO interface controller that is local to the
  35processor executing the Linux code. A master port generates and receives RapidIO
  36packets (transactions). In the RapidIO subsystem each master port is represented
  37by a rio_mport data structure. This structure contains master port specific
  38resources such as mailboxes and doorbells. The rio_mport also includes a unique
  39host device ID that is valid when a master port is configured as an enumerating
  42RapidIO master ports are serviced by subsystem specific mport device drivers
  43that provide functionality defined for this subsystem. To provide a hardware
  44independent interface for RapidIO subsystem operations, rio_mport structure
  45includes rio_ops data structure which contains pointers to hardware specific
  46implementations of RapidIO functions.
  482.2 Device
  50A RapidIO device is any endpoint (other than mport) or switch in the network.
  51All devices are presented in the RapidIO subsystem by corresponding rio_dev data
  52structure. Devices form one global device list and per-network device lists
  53(depending on number of available mports and networks).
  552.3 Switch
  57A RapidIO switch is a special class of device that routes packets between its
  58ports towards their final destination. The packet destination port within a
  59switch is defined by an internal routing table. A switch is presented in the
  60RapidIO subsystem by rio_dev data structure expanded by additional rio_switch
  61data structure, which contains switch specific information such as copy of the
  62routing table and pointers to switch specific functions.
  64The RapidIO subsystem defines the format and initialization method for subsystem
  65specific switch drivers that are designed to provide hardware-specific
  66implementation of common switch management routines.
  682.4 Network
  70A RapidIO network is a combination of interconnected endpoint and switch devices.
  71Each RapidIO network known to the system is represented by corresponding rio_net
  72data structure. This structure includes lists of all devices and local master
  73ports that form the same network. It also contains a pointer to the default
  74master port that is used to communicate with devices within the network.
  763. Subsystem Initialization
  79In order to initialize the RapidIO subsystem, a platform must initialize and
  80register at least one master port within the RapidIO network. To register mport
  81within the subsystem controller driver initialization code calls function
  82rio_register_mport() for each available master port. After all active master
  83ports are registered with a RapidIO subsystem, the rio_init_mports() routine
  84is called to perform enumeration and discovery.
  86In the current PowerPC-based implementation a subsys_initcall() is specified to
  87perform controller initialization and mport registration. At the end it directly
  88calls rio_init_mports() to execute RapidIO enumeration and discovery.
  904. Enumeration and Discovery
  93When rio_init_mports() is called it scans a list of registered master ports and
  94calls an enumeration or discovery routine depending on the configured role of a
  95master port: host or agent.
  97Enumeration is performed by a master port if it is configured as a host port by
  98assigning a host device ID greater than or equal to zero. A host device ID is
  99assigned to a master port through the kernel command line parameter "riohdid=",
 100or can be configured in a platform-specific manner. If the host device ID for
 101a specific master port is set to -1, the discovery process will be performed
 102for it.
 104The enumeration and discovery routines use RapidIO maintenance transactions
 105to access the configuration space of devices.
 107The enumeration process is implemented according to the enumeration algorithm
 108outlined in the RapidIO Interconnect Specification: Annex I [1].
 110The enumeration process traverses the network using a recursive depth-first
 111algorithm. When a new device is found, the enumerator takes ownership of that
 112device by writing into the Host Device ID Lock CSR. It does this to ensure that
 113the enumerator has exclusive right to enumerate the device. If device ownership
 114is successfully acquired, the enumerator allocates a new rio_dev structure and
 115initializes it according to device capabilities.
 117If the device is an endpoint, a unique device ID is assigned to it and its value
 118is written into the device's Base Device ID CSR.
 120If the device is a switch, the enumerator allocates an additional rio_switch
 121structure to store switch specific information. Then the switch's vendor ID and
 122device ID are queried against a table of known RapidIO switches. Each switch
 123table entry contains a pointer to a switch-specific initialization routine that
 124initializes pointers to the rest of switch specific operations, and performs
 125hardware initialization if necessary. A RapidIO switch does not have a unique
 126device ID; it relies on hopcount and routing for device ID of an attached
 127endpoint if access to its configuration registers is required. If a switch (or
 128chain of switches) does not have any endpoint (except enumerator) attached to
 129it, a fake device ID will be assigned to configure a route to that switch.
 130In the case of a chain of switches without endpoint, one fake device ID is used
 131to configure a route through the entire chain and switches are differentiated by
 132their hopcount value.
 134For both endpoints and switches the enumerator writes a unique component tag
 135into device's Component Tag CSR. That unique value is used by the error
 136management notification mechanism to identify a device that is reporting an
 137error management event.
 139Enumeration beyond a switch is completed by iterating over each active egress
 140port of that switch. For each active link, a route to a default device ID
 141(0xFF for 8-bit systems and 0xFFFF for 16-bit systems) is temporarily written
 142into the routing table. The algorithm recurs by calling itself with hopcount + 1
 143and the default device ID in order to access the device on the active port.
 145After the host has completed enumeration of the entire network it releases
 146devices by clearing device ID locks (calls rio_clear_locks()). For each endpoint
 147in the system, it sets the Discovered bit in the Port General Control CSR
 148to indicate that enumeration is completed and agents are allowed to execute
 149passive discovery of the network.
 151The discovery process is performed by agents and is similar to the enumeration
 152process that is described above. However, the discovery process is performed
 153without changes to the existing routing because agents only gather information
 154about RapidIO network structure and are building an internal map of discovered
 155devices. This way each Linux-based component of the RapidIO subsystem has
 156a complete view of the network. The discovery process can be performed
 157simultaneously by several agents. After initializing its RapidIO master port
 158each agent waits for enumeration completion by the host for the configured wait
 159time period. If this wait time period expires before enumeration is completed,
 160an agent skips RapidIO discovery and continues with remaining kernel
 1635. References
 166[1] RapidIO Trade Association. RapidIO Interconnect Specifications.
 168[2] Rapidio TA. Technology Comparisons.
 170[3] RapidIO support for Linux.
 172[4] Matt Porter. RapidIO for Linux. Ottawa Linux Symposium, 2005
 174 kindly hosted by Redpill Linpro AS, provider of Linux consulting and operations services since 1995.