Capture to Encode
The v4l2src component (producer) programs the Sync IP for the capture to encode use case. The capture driver releases the buffer to the encoder module immediately without filling the data (early call-back) when the pipeline goes into a playing state. The Sync IP between the capture device and VCU encoder is responsible for buffer synchronization.
The API flow is as follows:
- v4l2src programs syncip as per address ranges for the first input buffer and enables the sync IP.
- v4l2src calls VIDIOC_DQBUF and sends empty input buffer to omx encoder using early dequeue mechanism.
- omx encoder receives this empty buffer and starts generating read request after calling OMX_EmptyThisBuffer and sends an event to v4l2src to start the DMA and start filling input buffer.
- SyncIP blocks the omx encoder until v4l2src is done writing data corresponding to read request made by omx encoder.
- Similarly, for consecutive buffers v4l2src programs SyncIP,
submits buffer to omx encoder using VIDIOC_DQBUF and syncip block the omx
encoder until v4l2src has written sufficient data. This maintains the
synchronization between producer (v4l2src) and consumer
(omxh265enc/omxh264enc).
You can find sample Sync IP programming code and API descriptions in the following directories:
gst-plugins-good/sys/v4l2/ext/xlnx-ll/ xlnxsync.h (SyncIp driver ioctl header file xvfbsync.c (SyncIp Application API definition file) xvfbsync.h (SyncIp Application API header file)
Decode to Display
The VCU OMX decoder component has a custom early FillBufferDone callback which releases the output buffer to the next component, the display at the start of decoding, thus allowing concurrent access to the buffer for the decoder and display components and thus reducing the latency. To maintain synchronization between the decoder and display components so that display DMA does not start reading before the decoder has written, the decoder is at least half a frame period ahead of display as per the following API flow.
- Gst-Omx calls FillThisBuffer with the decoder output buffer.
- Gst-Omx call EmptyThisBuffer with the decoder input buffer.
- The OMX decoder generates early FillBufferDone callback and sends out empty buffer to the succeeding video sink kmssink element.
- The kmssink predicts the time when the vertical blanking signal is going to come based upon the previous vertical blanking time stamp.
- If the time to reach vertical blanking signal as per prediction is more than half frame period, then there is no extra wait required and buffer can be submitted directly to DRM driver using drmModeSetPlane.
- If the time to reach vertical blanking signal as per prediction is less than half frame period than kmssink wait for extra time period such that time to reach vertical blanking signal plus the extra time period should match the half frame period.
- This half frame period window give enough time for decoder to write enough data such that it is always ahead of display by at least half frame period and thus making sure that display DMA does not under-run by reading data before the decoder has written.