Debugging Performance Issues - 2024.1 English - PG252

For additional debugging information, see the Debugging Tools page of GStreamer website at https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html.

If the problem is low frame rate or frame dropping, follow these steps to debug the system.

1. Use the fpsdisplaysink element to report frame rate and dropped frame count (refer to the example above)
2. Check the QoS settings of HP ports that interface VCU with PS DDR. Check for outstanding transaction count configuration.
   Note: For VCU traffic, the QoS should be set as Best Effort (BE) and outstanding transaction count should be set to maximum (0xF for AFI ports).
3. Check if SMMU is enabled in the device tree. If SMMU is enabled, disable it because it imparts longer latency in the transaction completion.
4. Check if encoder buffer (prefetch-buffer) is used in the user design. If not, check if encoder buffer impacts the performance. If performance improves with encoder buffer, it might indicate a system bandwidth issue. Use the following sample pipeline to enable the prefetch buffer in design:

   gst-launch-1.0 videotestsrc ! omxh265enc prefetch-buffer=true ! fakesink

5. Try with different encoder/decoder properties to see if the performance drop is related to any of the properties. Avoid B-frames in the pipeline to see if there is any performance improvement. If there is improvement, it might indicate a system bandwidth issue. Reduce the bit rate to see if there is improvement in framer ate. If reducing target-bit rate gives better throughput, it might indicate a system bandwidth issue.
6. Check for CPU usage while the pipeline is running. A higher CPU usage indicates that there could be an impact in interrupt processing time which explains the lower framerate.
7. Try using a queue element between two GStreamer plugins that are in the datapath to check for any performance improvement.
8. Check for DDR bandwidth utilization using DDR APM and VCU APM.
9. Use gst-shark (a GStreamer-based tool) to verify performance and create scheduletime and interlatency plots to understand which element is causing performance drops.
10. Using environment variables, you can increase encoder input and output buffers count for debug or performance tuning purpose. Use the ENC_EXTRA_IP_BUFFERS and ENC_EXTRA_OP_BUFFERS environment variable to provide extra buffers needed on encoder ports. For example, suppose X number of buffers are allocated for encoder input/output by default. To add five more buffers to it, assign ENC_EXTRA_IP_BUFFERS=5. So, new allocated buffers for encoder input is X+5. Similarly, for encoder output buffers, use ENC_EXTRA_OP_BUFFERS. The pipelines are as follows:

    ENC_EXTRA_IP_BUFFERS=5 gst-launch-1.0 -v v4l2src io-mode=4 device=/dev/video0 num-buffers=1000 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=60/1 ! omxh264enc control-rate=constant target-bitrate=50000 prefetch-buffer=TRUE ! video/x-h264, profile=high ! filesink location=test.avc

    ENC_EXTRA_OP_BUFFERS=5 gst-launch-1.0 -v v4l2src io-mode=4 device=/dev/video0 num-buffers=1000 ! video/x-raw, format=NV12, width=3840, height=2160, framerate=60/1 ! omxh264enc control-rate=constant target-bitrate=50000 prefetch-buffer=TRUE ! video/x-h264, profile=high ! filesink location=test.avc