GStreamer is a cross-platform open source multimedia framework that provides
infrastructure to integrate multiple multimedia components and create pipelines/graphs.
GStreamer graphs are made of two or more plugin elements which are delivered as shared
libraries. The following is a list of commonly performed tasks in the GStreamer
framework:
- Selection of a source GStreamer plugin
- Selection of a processing GStreamer plugin
- Selection of a sink GStreamer plugin
- Creation of a GStreamer graph based on above plugins plus capabilities
- Configuration of properties of above GStreamer plugins
- Control of a GStreamer pipeline/graph
Plugins
The following GStreamer plugin categories are used in this design:
- Source
- mediasrcbin: V4l2 sources such as USB webcam, MIPI single-sensor, MIPI quad-sensor
- multisrc/filesrc: video file source for raw or encoded image/video files
- Sink
- kmssink: KMS display sink for HDMI Tx
- filesink: video file sink for raw or encoded image/video files
- appsink: sink that makes video buffers available to an application such as the display inside jupyter notebooks
- Encode/decode
- jpegenc/dec: jpg image file encode/decode
- vp9enc/dec: vp9 video file encode/decode
- Processing/acceleration
- sdxfilter2d: 2D filter accelerator
- Other
- capsfilter: filters capabilities
- tee: tee element to create a fork in the data flow
- queue: creates separate threads between pipeline elements and adds additional buffering
- perf: measure frames-per-seconds (fps) at an arbitrary point in the pipeline
Capabilities
The pads are the element's interface to the outside world. Data streams from one
element's source pad to another element's sink pad. The specific type of media that
the element can handle is exposed by the pad's capabilities. The following
capabilities are used between the video-source plugin and its peer plugin (either
video-sink or video-processing). These capabilities (also called
capsfilter) are specified while constructing a GStreamer
graph, for
example:
"video/x-raw, width=<width of videosrc>, height=<height of videosrc>, format=YUY2, ramerate=<fps/1>"If multisrc is used as video-source plugin, the
videoparse element is used instead of a
capsfilter to parse the raw video file and transform it to
frames:
"video/x-raw, width=<width of videosrc>, height=<height of videosrc>, format=YUY2, framerate=<fps/1>"Pipeline Control
The GStreamer framework is used to control the GStreamer graph. It provides the
following functionality:
- Start/stop video stream inside a graph
- Get/set controls
- Buffer operations
- Get frames-per-second information
There are four states defined in the GStreamer graph: "NULL", "READY", "PAUSED", and "PLAYING". The "PLAYING" state of a GStreamer graph is used to start the pipeline and the "NULL" state is to stop the pipeline.
Allocators
GStreamer abstracts buffer allocation and pooling. Custom allocators and buffer pools can be implemented to accommodate custom use-cases and constraints. The video source controls buffer allocation, but the sink can propose parameters in the negotiation phase.
The DMABUF framework is used to import and export buffers in a 0-copy fashion
between pipeline elements, which is required for high-performance pipelines, as
shown in the following figure. The v4l2src
element has a property named io-mode which
allows allocation and export of DMABUFs to its peer element. The kmssink element allows import as well as export of
DMABUFs to/from its peer element. The accelerator element xrtbase allows only import of DMABUFs, which means it relies on
DMABUFs being allocated by its peer elements connected to the source and sink
pads.
Figure 1.
DMABUF Sharing Mechanism
Note that DMABUFs are not necessarily physically contiguous depending on the underlying kernel device driver, that is, the UVC v4l2 driver does not allocate CMA memory which results in a data copy if its peer element can only handle contiguous memory.