linux/Documentation/media-framework.txt
<<
>>
Prefs
   1Linux kernel media framework
   2============================
   3
   4This document describes the Linux kernel media framework, its data structures,
   5functions and their usage.
   6
   7
   8Introduction
   9------------
  10
  11The media controller API is documented in DocBook format in
  12Documentation/DocBook/media/v4l/media-controller.xml. This document will focus
  13on the kernel-side implementation of the media framework.
  14
  15
  16Abstract media device model
  17---------------------------
  18
  19Discovering a device internal topology, and configuring it at runtime, is one
  20of the goals of the media framework. To achieve this, hardware devices are
  21modelled as an oriented graph of building blocks called entities connected
  22through pads.
  23
  24An entity is a basic media hardware building block. It can correspond to
  25a large variety of logical blocks such as physical hardware devices
  26(CMOS sensor for instance), logical hardware devices (a building block
  27in a System-on-Chip image processing pipeline), DMA channels or physical
  28connectors.
  29
  30A pad is a connection endpoint through which an entity can interact with
  31other entities. Data (not restricted to video) produced by an entity
  32flows from the entity's output to one or more entity inputs. Pads should
  33not be confused with physical pins at chip boundaries.
  34
  35A link is a point-to-point oriented connection between two pads, either
  36on the same entity or on different entities. Data flows from a source
  37pad to a sink pad.
  38
  39
  40Media device
  41------------
  42
  43A media device is represented by a struct media_device instance, defined in
  44include/media/media-device.h. Allocation of the structure is handled by the
  45media device driver, usually by embedding the media_device instance in a
  46larger driver-specific structure.
  47
  48Drivers register media device instances by calling
  49
  50        media_device_register(struct media_device *mdev);
  51
  52The caller is responsible for initializing the media_device structure before
  53registration. The following fields must be set:
  54
  55 - dev must point to the parent device (usually a pci_dev, usb_interface or
  56   platform_device instance).
  57
  58 - model must be filled with the device model name as a NUL-terminated UTF-8
  59   string. The device/model revision must not be stored in this field.
  60
  61The following fields are optional:
  62
  63 - serial is a unique serial number stored as a NUL-terminated ASCII string.
  64   The field is big enough to store a GUID in text form. If the hardware
  65   doesn't provide a unique serial number this field must be left empty.
  66
  67 - bus_info represents the location of the device in the system as a
  68   NUL-terminated ASCII string. For PCI/PCIe devices bus_info must be set to
  69   "PCI:" (or "PCIe:") followed by the value of pci_name(). For USB devices,
  70   the usb_make_path() function must be used. This field is used by
  71   applications to distinguish between otherwise identical devices that don't
  72   provide a serial number.
  73
  74 - hw_revision is the hardware device revision in a driver-specific format.
  75   When possible the revision should be formatted with the KERNEL_VERSION
  76   macro.
  77
  78 - driver_version is formatted with the KERNEL_VERSION macro. The version
  79   minor must be incremented when new features are added to the userspace API
  80   without breaking binary compatibility. The version major must be
  81   incremented when binary compatibility is broken.
  82
  83Upon successful registration a character device named media[0-9]+ is created.
  84The device major and minor numbers are dynamic. The model name is exported as
  85a sysfs attribute.
  86
  87Drivers unregister media device instances by calling
  88
  89        media_device_unregister(struct media_device *mdev);
  90
  91Unregistering a media device that hasn't been registered is *NOT* safe.
  92
  93
  94Entities, pads and links
  95------------------------
  96
  97- Entities
  98
  99Entities are represented by a struct media_entity instance, defined in
 100include/media/media-entity.h. The structure is usually embedded into a
 101higher-level structure, such as a v4l2_subdev or video_device instance,
 102although drivers can allocate entities directly.
 103
 104Drivers initialize entities by calling
 105
 106        media_entity_init(struct media_entity *entity, u16 num_pads,
 107                          struct media_pad *pads, u16 extra_links);
 108
 109The media_entity name, type, flags, revision and group_id fields can be
 110initialized before or after calling media_entity_init. Entities embedded in
 111higher-level standard structures can have some of those fields set by the
 112higher-level framework.
 113
 114As the number of pads is known in advance, the pads array is not allocated
 115dynamically but is managed by the entity driver. Most drivers will embed the
 116pads array in a driver-specific structure, avoiding dynamic allocation.
 117
 118Drivers must set the direction of every pad in the pads array before calling
 119media_entity_init. The function will initialize the other pads fields.
 120
 121Unlike the number of pads, the total number of links isn't always known in
 122advance by the entity driver. As an initial estimate, media_entity_init
 123pre-allocates a number of links equal to the number of pads plus an optional
 124number of extra links. The links array will be reallocated if it grows beyond
 125the initial estimate.
 126
 127Drivers register entities with a media device by calling
 128
 129        media_device_register_entity(struct media_device *mdev,
 130                                     struct media_entity *entity);
 131
 132Entities are identified by a unique positive integer ID. Drivers can provide an
 133ID by filling the media_entity id field prior to registration, or request the
 134media controller framework to assign an ID automatically. Drivers that provide
 135IDs manually must ensure that all IDs are unique. IDs are not guaranteed to be
 136contiguous even when they are all assigned automatically by the framework.
 137
 138Drivers unregister entities by calling
 139
 140        media_device_unregister_entity(struct media_entity *entity);
 141
 142Unregistering an entity will not change the IDs of the other entities, and the
 143ID will never be reused for a newly registered entity.
 144
 145When a media device is unregistered, all its entities are unregistered
 146automatically. No manual entities unregistration is then required.
 147
 148Drivers free resources associated with an entity by calling
 149
 150        media_entity_cleanup(struct media_entity *entity);
 151
 152This function must be called during the cleanup phase after unregistering the
 153entity. Note that the media_entity instance itself must be freed explicitly by
 154the driver if required.
 155
 156Entities have flags that describe the entity capabilities and state.
 157
 158        MEDIA_ENT_FL_DEFAULT indicates the default entity for a given type.
 159        This can be used to report the default audio and video devices or the
 160        default camera sensor.
 161
 162Logical entity groups can be defined by setting the group ID of all member
 163entities to the same non-zero value. An entity group serves no purpose in the
 164kernel, but is reported to userspace during entities enumeration. The group_id
 165field belongs to the media device driver and must not by touched by entity
 166drivers.
 167
 168Media device drivers should define groups if several entities are logically
 169bound together. Example usages include reporting
 170
 171        - ALSA, VBI and video nodes that carry the same media stream
 172        - lens and flash controllers associated with a sensor
 173
 174- Pads
 175
 176Pads are represented by a struct media_pad instance, defined in
 177include/media/media-entity.h. Each entity stores its pads in a pads array
 178managed by the entity driver. Drivers usually embed the array in a
 179driver-specific structure.
 180
 181Pads are identified by their entity and their 0-based index in the pads array.
 182Both information are stored in the media_pad structure, making the media_pad
 183pointer the canonical way to store and pass link references.
 184
 185Pads have flags that describe the pad capabilities and state.
 186
 187        MEDIA_PAD_FL_SINK indicates that the pad supports sinking data.
 188        MEDIA_PAD_FL_SOURCE indicates that the pad supports sourcing data.
 189
 190One and only one of MEDIA_PAD_FL_SINK and MEDIA_PAD_FL_SOURCE must be set for
 191each pad.
 192
 193- Links
 194
 195Links are represented by a struct media_link instance, defined in
 196include/media/media-entity.h. Each entity stores all links originating at or
 197targeting any of its pads in a links array. A given link is thus stored
 198twice, once in the source entity and once in the target entity. The array is
 199pre-allocated and grows dynamically as needed.
 200
 201Drivers create links by calling
 202
 203        media_entity_create_link(struct media_entity *source, u16 source_pad,
 204                                 struct media_entity *sink,   u16 sink_pad,
 205                                 u32 flags);
 206
 207An entry in the link array of each entity is allocated and stores pointers
 208to source and sink pads.
 209
 210Links have flags that describe the link capabilities and state.
 211
 212        MEDIA_LNK_FL_ENABLED indicates that the link is enabled and can be used
 213        to transfer media data. When two or more links target a sink pad, only
 214        one of them can be enabled at a time.
 215        MEDIA_LNK_FL_IMMUTABLE indicates that the link enabled state can't be
 216        modified at runtime. If MEDIA_LNK_FL_IMMUTABLE is set, then
 217        MEDIA_LNK_FL_ENABLED must also be set since an immutable link is always
 218        enabled.
 219
 220
 221Graph traversal
 222---------------
 223
 224The media framework provides APIs to iterate over entities in a graph.
 225
 226To iterate over all entities belonging to a media device, drivers can use the
 227media_device_for_each_entity macro, defined in include/media/media-device.h.
 228
 229        struct media_entity *entity;
 230
 231        media_device_for_each_entity(entity, mdev) {
 232                /* entity will point to each entity in turn */
 233                ...
 234        }
 235
 236Drivers might also need to iterate over all entities in a graph that can be
 237reached only through enabled links starting at a given entity. The media
 238framework provides a depth-first graph traversal API for that purpose.
 239
 240Note that graphs with cycles (whether directed or undirected) are *NOT*
 241supported by the graph traversal API. To prevent infinite loops, the graph
 242traversal code limits the maximum depth to MEDIA_ENTITY_ENUM_MAX_DEPTH,
 243currently defined as 16.
 244
 245Drivers initiate a graph traversal by calling
 246
 247        media_entity_graph_walk_start(struct media_entity_graph *graph,
 248                                      struct media_entity *entity);
 249
 250The graph structure, provided by the caller, is initialized to start graph
 251traversal at the given entity.
 252
 253Drivers can then retrieve the next entity by calling
 254
 255        media_entity_graph_walk_next(struct media_entity_graph *graph);
 256
 257When the graph traversal is complete the function will return NULL.
 258
 259Graph traversal can be interrupted at any moment. No cleanup function call is
 260required and the graph structure can be freed normally.
 261
 262Helper functions can be used to find a link between two given pads, or a pad
 263connected to another pad through an enabled link
 264
 265        media_entity_find_link(struct media_pad *source,
 266                               struct media_pad *sink);
 267
 268        media_entity_remote_pad(struct media_pad *pad);
 269
 270Refer to the kerneldoc documentation for more information.
 271
 272
 273Use count and power handling
 274----------------------------
 275
 276Due to the wide differences between drivers regarding power management needs,
 277the media controller does not implement power management. However, the
 278media_entity structure includes a use_count field that media drivers can use to
 279track the number of users of every entity for power management needs.
 280
 281The use_count field is owned by media drivers and must not be touched by entity
 282drivers. Access to the field must be protected by the media device graph_mutex
 283lock.
 284
 285
 286Links setup
 287-----------
 288
 289Link properties can be modified at runtime by calling
 290
 291        media_entity_setup_link(struct media_link *link, u32 flags);
 292
 293The flags argument contains the requested new link flags.
 294
 295The only configurable property is the ENABLED link flag to enable/disable a
 296link. Links marked with the IMMUTABLE link flag can not be enabled or disabled.
 297
 298When a link is enabled or disabled, the media framework calls the
 299link_setup operation for the two entities at the source and sink of the link,
 300in that order. If the second link_setup call fails, another link_setup call is
 301made on the first entity to restore the original link flags.
 302
 303Media device drivers can be notified of link setup operations by setting the
 304media_device::link_notify pointer to a callback function. If provided, the
 305notification callback will be called before enabling and after disabling
 306links.
 307
 308Entity drivers must implement the link_setup operation if any of their links
 309is non-immutable. The operation must either configure the hardware or store
 310the configuration information to be applied later.
 311
 312Link configuration must not have any side effect on other links. If an enabled
 313link at a sink pad prevents another link at the same pad from being enabled,
 314the link_setup operation must return -EBUSY and can't implicitly disable the
 315first enabled link.
 316
 317
 318Pipelines and media streams
 319---------------------------
 320
 321When starting streaming, drivers must notify all entities in the pipeline to
 322prevent link states from being modified during streaming by calling
 323
 324        media_entity_pipeline_start(struct media_entity *entity,
 325                                    struct media_pipeline *pipe);
 326
 327The function will mark all entities connected to the given entity through
 328enabled links, either directly or indirectly, as streaming.
 329
 330The media_pipeline instance pointed to by the pipe argument will be stored in
 331every entity in the pipeline. Drivers should embed the media_pipeline structure
 332in higher-level pipeline structures and can then access the pipeline through
 333the media_entity pipe field.
 334
 335Calls to media_entity_pipeline_start() can be nested. The pipeline pointer must
 336be identical for all nested calls to the function.
 337
 338media_entity_pipeline_start() may return an error. In that case, it will
 339clean up any of the changes it did by itself.
 340
 341When stopping the stream, drivers must notify the entities with
 342
 343        media_entity_pipeline_stop(struct media_entity *entity);
 344
 345If multiple calls to media_entity_pipeline_start() have been made the same
 346number of media_entity_pipeline_stop() calls are required to stop streaming. The
 347media_entity pipe field is reset to NULL on the last nested stop call.
 348
 349Link configuration will fail with -EBUSY by default if either end of the link is
 350a streaming entity. Links that can be modified while streaming must be marked
 351with the MEDIA_LNK_FL_DYNAMIC flag.
 352
 353If other operations need to be disallowed on streaming entities (such as
 354changing entities configuration parameters) drivers can explicitly check the
 355media_entity stream_count field to find out if an entity is streaming. This
 356operation must be done with the media_device graph_mutex held.
 357
 358
 359Link validation
 360---------------
 361
 362Link validation is performed by media_entity_pipeline_start() for any
 363entity which has sink pads in the pipeline. The
 364media_entity::link_validate() callback is used for that purpose. In
 365link_validate() callback, entity driver should check that the properties of
 366the source pad of the connected entity and its own sink pad match. It is up
 367to the type of the entity (and in the end, the properties of the hardware)
 368what matching actually means.
 369
 370Subsystems should facilitate link validation by providing subsystem specific
 371helper functions to provide easy access for commonly needed information, and
 372in the end provide a way to use driver-specific callbacks.
 373
lxr.linux.no kindly hosted by Redpill Linpro AS, provider of Linux consulting and operations services since 1995.