The Vitis Video Analytics SDK (VVAS) is a framework designed for AMD platforms that allows the creation of various AI solutions. It supports input from file, and uses AMD Vitis™ AI to generate insights from pixels for various use cases. VVAS SDK can be the foundation layer for multiple video analytic solutions like understanding traffic and pedestrians in smart city, health and safety monitoring in hospitals, self-checkout, and analytics in retail, detecting component defects at a manufacturing facility and others.
The core SDK comprises of optimized GStreamer plug-ins that employ various accelerators such as video image processing kernels for resizing and color space conversion, and the Neural Processing Unit (NPU) for machine learning.
By running computationally heavy operations on specialized accelerators, VVAS helps create high-performance video analytics and other applications.
For developers, VVAS provides a framework in the form of Infrastructure GStreamer plug-ins and a simplified interface to create custom acceleration libraries to control a custom hardware accelerator. This framework enables users to easily integrate their custom accelerators/kernels into the GStreamer framework. VVAS builds on top of Xilinx Run Time (XRT) and Vitis AI, abstracting these complex interfaces, so developers can create video analytics and other applications without having to learn the intricacies of XRT or Vitis AI.
VVAS SDK supports deployment on embedded edge devices like VEK385.
Fig. 1 VVAS Block Diagram
VVAS Graph Architecture
VVAS is a streamlined graph structure constructed with the help of the GStreamer framework. The following diagram demonstrates a common video analytic pipeline that begins with the input of video data and concludes with the presentation of valuable insights. Each of the constituent blocks is composed of distinct plugins. The application employs several hardware engines, which are depicted at the bottom. By implementing efficient memory management techniques, avoiding unnecessary memory duplication between plugins, and using diverse accelerators, the system attains optimal performance.
Data is read from the local file system.
Optional image pre-processing is done on the input video stream, where the input image can undergo certain adjustments prior to inference. Pre-processing can entail resizing the image, converting its color space, or performing mean subtraction, among other operations. The
vvas_xabrscalerplugin can perform hardware-accelerated resizing, color format conversion, and pre-processing operations such as mean subtraction and normalization on the frame.Following pre-processing, the frame is directed to undergo inference using the
vvas_xinferplugin.To overlay inference results such as bounding boxes, labels, and arrows into the video output, two plugins are used:
vvas_xmetaconvertandvvas_xoverlay. The former plugin comprehends theGstInferenceMetameta-data structure and carries out parsing operations to create new metadata for lines, arrows, text, and other such elements, which can be understood by thevvas_xoverlayplugin. Thevvas_xoverlayplugin is responsible for drawing various components such as bounding boxes, arrows, circles, and text based on the received information.
Why VVAS?
VVAS enables application developers to construct seamless streaming pipelines for AI-based video and image analytics and various other solutions without requiring any comprehension of FPGA or other complex development environments. VVAS comes equipped with a multitude of hardware accelerators for various functionalities, along with highly optimized GStreamer plugins, so you can execute any Neural Network model with a full suite of infrastructure support for an end-to-end use case implementation. This makes it an excellent choice for video analytics tasks. The design is optimized for machine learning (ML) operations on the Neural Processing Unit (NPU) and leverages hardware-accelerated AMD Vitis™ kernels to efficiently handle preprocessing, postprocessing, and inference.
For advanced developers, VVAS offers a user-friendly framework that allows for integration of their own hardware accelerators/kernels into GStreamer-based applications. VVAS also provides support for popular classification models such as resnet50, resnet18.
All of this infrastructure grants the flexibility for swift prototyping and progression to full-fledged production-level solutions, thereby significantly reducing the time-to-market for AMD platform-based solutions.
VVAS Components
This section gives deeper insight into VVAS. VVAS comprises the following core components.
Custom GStreamer Plug-ins
These are highly optimized GStreamer plug-ins developed to provide very specific functionality using optimized Kernels and IPs on AMD Platform. Refer to VVAS Custom Plug-ins for more details about how to use these plug-ins.
Infrastructure GStreamer Plug-ins
These GStreamer plugins are designed as generic infrastructure to assist users in directly integrating their kernels into the GStreamer framework without requiring an in-depth understanding of the framework itself. Refer to VVAS Infrastructure Plug-ins for more details about how to use these plug-ins.
Acceleration S/W Libs
These optimized software libraries for acceleration have been developed to manage the state machine of acceleration kernels/IPs and provide an interface for integration with VVAS generic infrastructure plugins. They can serve as a reference for developing new acceleration software libraries based on the VVAS framework. For more information on VVAS acceleration software libraries and their integration with infrastructure plugins, see the VVAS Infrastructure Plug-ins and Acceleration s/w Libraries section.
Acceleration H/W (Kernels/IPs)
These are highly optimized Kernels being developed by AMD. Details of these are captured in this section.
Reference Applications
Sample applications and the quick-start flow are described in the Applications section.