Performance Improvement from the GStreamer Perspective

Queue Element

In the single threaded GStreamer pipeline, data starvation can occur. Use the queue element to improve the performance of a single threaded pipeline.

Data is queued until one of the limits specified by the “max-size-buffers”, “max-size-bytes” and/or “max-size-time” properties has been reached. Any attempt to push more buffers into the queue blocks the pushing thread until more space becomes available.

The queue element adds a thread boundary to the pipeline, and support for buffering. The queue creates a new thread on the source pad to decouple the processing on sink and source pad.

The default queue size limits are 200 buffers, 10 MB of data, or one second worth of data, whichever is reached first.

Sample pipeline with queue element:

gst-launch-1.0 filesrc location="/media/card/input-file.mp4" !qtdemux name=demux
demux.video_0 ! h265parse ! omxh265dec ! queue max-size-bytes=0 ! 
fpsdisplaysink text-overlay=false 
video-sink="kmssink bus-id=a0070000.v_mix" -v

Quality Of Service (QoS)

Quality of Service in GStreamer is about measuring and adjusting the real-time performance of a pipeline. The real-time performance is always measured relative to the pipeline clock and typically happens in the sinks when they synchronize buffers against the clock.

When buffers arrive late in the sink, that is, when their running-time is smaller than that of the clock, the pipeline has a quality of service problem. These are a few possible reasons:

High CPU load, there is not enough CPU power to handle the stream, causing buffers to arrive late in the sink.
Network problems
Other resource problems such as disk load, and memory bottlenecks

The measurements result in QoS events to adjust the data rate in one or more upstream elements. The following are two possible adjustments.

Short time emergency corrections based on latest observation in the sinks
Long term rate corrections based on trends observed in the sinks

Short Term Correction

Use the timestamp and the jitter value in the QoS event to perform a short-term correction. If the jitter is positive, the previous buffer arrived late, and can be sure that a buffer with a timestamp < timestamp + jitter is also going to be late. Therefore, drop all buffers with a timestamp less than timestamp + jitter.

Long Term Correction

Long term corrections are a bit more difficult to perform. They rely on the value of the proportion in the QoS event. Elements should reduce the amount of resources they consume by the proportion field in the QoS message.

Here are some possible strategies to achieve this:

Permanently drop frames or reduce the CPU or bandwidth requirements of the element.
Switch to lower quality processing or reduce the algorithmic complexity. Care should be taken that this does not introduce disturbing visual or audible glitches.
Switch to a lower quality source to reduce network bandwidth.
Assign more CPU cycles to critical parts of the pipeline. This could, for example, be done by increasing the thread priority.

In all cases, elements should be prepared to go back to their normal processing rate, when the proportion member in the QOS event approaches the ideal proportion.

The Encoder and Decoder plugin also supports the QoS functionality.

In the decoder, QoS is enabled by default and drops frames after decoding is finished, based on the QoS event from downstream.
In the encoder, QoS is disabled by default and drops the input buffer while encoding, if the QoS condition is true, based on the QoS event from downstream.

Sync

In the GStreamer pipeline, the sync flag plays an important role. The sync flag is used for the synchronization of audio/video in the pipeline by checking the timestamp in the sink element. To know the best outcome of the pipeline, disable the sync flag in the sink element of the pipeline, keeping in mind that synchronization cannot be achieved by setting it to false. The sync flag is useful for a record pipeline to dump the data as soon as it receives it at the sink element.

Use the following example with sync=false for a record pipeline:

gst-launch-1.0 v4l2src
device=/dev/video0 io-mode=4 ! video/x-raw, format=NV12,width=3840,height=2160,framerate=60/1 ! omxh265enc qp-
mode=auto  gop-mode=basic
gop-length=60 b-frames=0 target-bitrate=60000 num-slices=8 control- rate=constant
prefetch-buffer=true cpb-size=1000 initial-delay=500 ! queue ! video/x- h265, profile=main,
alignment=au ! mpegtsmux alignment=7 name=mux ! filesink location="/ media/card/test.ts"
sync=false