This section describes the steps for creating an acceleration software library using VVAS APIs for a kernel. As an example, a sample software kernel is used, and this library is for controlling this vvas_xstub kernel.
The vvas_xstub kernel is a software kernel supporting three operation modes.
PASSTHROUGH : pass through the input without any modification
INPLACE : updating the buffer in place to specific value (currently supported with input format rgb and bgr)
TRANSFORM : generates the new buffer at outputs and validates it
Note
The vvas_xstub kernel is primarily used for illustrative purposes. Delving deep into discussions about the `vvas_xstub` kenel falls beyond the scope of this topic.
Each software acceleration library must expose the mandatory core APIs. GStreamer infrastructure plug-ins, as well as applications, call these core APIs to control the kernel. The mandatory core APIs are as follows.
/********* CORE APIs ************/
VvasReturnType xlnx_kernel_init (VVASKernel * handle);
VvasReturnType xlnx_kernel_start (VVASKernel * handle, int start,
VVASFrame * input[MAX_NUM_OBJECT], VVASFrame * output[MAX_NUM_OBJECT]);
VvasReturnType xlnx_kernel_done (VVASKernel * handle);
VvasReturnType xlnx_kernel_deinit (VVASKernel * handle);
All the necessary one-time initialization procedures must be executed within the xlnx_kernel_init API. The subsequent private structure is employed for the purpose of retaining and managing all these initializations. It is left to your discretion to implement your own storage classes for the management of these one-time initializations.
/*
* Structure to store all initializations
*/
typedef struct _vvas_xstub
{
int log_level;
Vvas_XKernelMode kernel_type;
Vvas_XElementMode element_mode;
} VVASXStub;
xlnx_kernel_init is responsible for initializing the kernel parameters and preparing them for further operations. It extracts configuration information from the provided VVASKernel handle, initializes the kernel parameters, and prepares the environment required for the kernel’s functionality.
VvasReturnType
xlnx_kernel_init (VVASKernel * handle)
{
VVASXStub *kpriv;
json_t *val;
char *static_cfg;
/* kernel specification info is required to validate kernel library */
if (handle->kernel_config == NULL) {
LOG_MESSAGE (LOG_LEVEL_ERROR, LOG_LEVEL_WARNING,
"json object for static kernel configuration is NOT available");
return VVAS_RET_ERROR;
}
kpriv = (VVASXStub *) malloc (sizeof (VVASXStub));
if (!kpriv) {
LOG_MESSAGE (LOG_LEVEL_ERROR, LOG_LEVEL_WARNING,
"failed to allocate private handle\n");
return VVAS_RET_ERROR;
}
.
.
.
handle->kernel_priv = (void *) kpriv;
.
.
return VVAS_RET_SUCCESS;
}
Kernel configuration to configure the vvas_xstub kernel:
This json demostrates the inplace mode. Given mode can be changed by replacing ‘inplace’ string to ‘transform’ or ‘passthrough’ for other configurations.
{
"xclbin-location":"/run/media/mmcblk0p1/x_plus_ml.xclbin",
"vvas-library-repo": "/usr/lib",
"element-mode":"inplace",
"kernel" :
{
"library-name":"libvvas_xstub.so",
"config": {
"kernel-lib-type" : "soft-lib",
"element-mode" : "inplace",
"src-stride-align" : 1,
"src-height-align" : 1,
"sink-stride-align" : 1,
"sink-height-align" : 1,
"debug-level" : 4
}
}
}
The following section breaks down the function flow of the acceleration library’s xlnx_kernel_init API.
Allocate memory for the VVASXStub structure (kpriv) to store the kernel parameters. If memory allocation fails, an error message is logged, and the function returns an error code.
kpriv = (VVASXStub *) malloc (sizeof (VVASXStub));
if (!kpriv) {
LOG_MESSAGE (LOG_LEVEL_ERROR, LOG_LEVEL_WARNING,
"failed to allocate private handle\n");
return VVAS_RET_ERROR;
}
Retrieve the debug log level from the kernel configuration. If not provided, set a default warning log level. Otherwise, set the value that is extracted from the configuration.
/*
* Get the kernel configuration from application through JSON File.
*/
val = json_object_get (handle->kernel_config, "debug-level");
if (!val || !json_is_integer (val))
kpriv->log_level = LOG_LEVEL_WARNING;
else
kpriv->log_level = json_integer_value (val);
Memory bank indexes for input and output GstMemory are retrieved from the configuration. If not the indexes are not provided, the default values are used.
/*
* Get and set the memory bank index where to allocate input GstMemory
* by infrastructure plugin
*/
val = json_object_get (handle->kernel_config, "in-mem-bank");
if (!json_is_integer (val)) {
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"in-mem-bank is not set, setting it to %d", STUB_DEFAULT_MEM_BANK);
handle->in_mem_bank = STUB_DEFAULT_MEM_BANK;
} else {
handle->in_mem_bank = json_integer_value (val);
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level, "in-mem-bank is set to %d",
json_integer_value (val));
}
/*
* Get and set the memory bank index where to allocate output GstMemory
* by infrastructure plugin
*/
val = json_object_get (handle->kernel_config, "out-mem-bank");
if (!json_is_integer (val)) {
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"out-mem-bank is not set, setting it to %d", STUB_DEFAULT_MEM_BANK);
handle->out_mem_bank = STUB_DEFAULT_MEM_BANK;
} else {
handle->out_mem_bank = json_integer_value (val);
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level, "out-mem-bank is set to %di",
json_integer_value (val));
}
As per different functionalities of the vvas_xstub kernel, there are various configuration options and parameters that gets parsed from json.
kpriv->kernel_type = get_kernel_lib_type (json_string_value (val));
if (kpriv->kernel_type == VVAS_KERNEL_LIB_TYPE_UNKNOWN) {
LOG_MESSAGE (LOG_LEVEL_ERROR, kpriv->log_level,
"unknown kernel lib type : %s\n", json_string_value (val));
return VVAS_RET_ERROR;
}
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level, "kernel library type : %s\n",
json_string_value (val));
if (kpriv->kernel_type == VVAS_KERNEL_LIB_TYPE_SOFTLIB) {
/* kernel library is for software library */
if (handle->dev_handle) {
LOG_MESSAGE (LOG_LEVEL_WARNING, kpriv->log_level,
"avoid setting xclbin-location in root json");
}
.
.
.
.
LOG_MESSAGE (LOG_LEVEL_DEBUG, kpriv->log_level, "STUB Init Successful");
return VVAS_RET_SUCCESS;
After completing all the necessary initializations in the xlnx_kernel_init API, the next step is to execute the kernel using the xlnx_kernel_start API. This API is called by the infrastructure plug-in for each input frame or buffer received, indicating that the kernel needs to process the provided data.
VvasReturnType
xlnx_kernel_start (VVASKernel * handle, int start,
VVASFrame * input[MAX_NUM_OBJECT], VVASFrame * output[MAX_NUM_OBJECT])
{
VVASXStub *kpriv = (VVASXStub *) handle->kernel_priv;
.
.
.
.
return VVAS_RET_SUCCESS;
}
The following section breaks down the function flow of the acceleration library’s xlnx_kernel_start API.
The API begins by retrieving the kernel’s private data (kpriv) that was initialized during the xlnx_kernel_init API. This data contains various parameters required for the kernel’s execution.
/*
* Get the VVASXStub which was initialized during xlnx_kernel_init api
*/
VVASXStub *kpriv = (VVASXStub *) handle->kernel_priv;
Perform the validation for input frames.
/**************************************************
*********** Input frames validation **************
**************************************************/
while (idx < MAX_NUM_OBJECT) {
if (input[idx]) {
VVASFrame *inframe = input[idx];
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"----- Input frame index %d -----", idx);
ret = validate_vvas_frame (handle, inframe);
if (ret < 0) {
return VVAS_RET_ERROR;
}
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"----------------------------");
idx++;
} else
break;
}
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"number of input frames received : %d", idx);
if (output[0] == NULL) {
LOG_MESSAGE (LOG_LEVEL_ERROR, kpriv->log_level,
"Kernel needs output buffer as it works only in transform mode.");
return VVAS_RET_ERROR;
}
Perform the validation for output frames.
/**************************************************
*********** Output frames validation *************
**************************************************/
idx = 0;
if (output[idx]) {
while (idx < MAX_NUM_OBJECT) {
if (output[idx]) {
VVASFrame *outframe = output[idx];
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"----- Output frame index %d -----", idx);
ret = validate_vvas_frame (handle, outframe);
if (ret < 0) {
return VVAS_RET_ERROR;
}
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"----------------------------");
idx++;
} else
break;
}
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"number of output frames received : %d", idx);
} else {
Handling of output buffer in case of passthrough or inplace mode
} else {
/* no output buffers in case of passthrough/inplace mode */
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"number of output frames received : %d", idx);
idx = 0;
while (idx < MAX_NUM_OBJECT) {
if (input[idx]) {
if (kpriv->element_mode == VVAS_ELEMENT_MODE_IN_PLACE) {
VVASFrame *inframe = input[idx];
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"Modifying the input %d in inplace mode", idx);
modify_input_frame_inplace (handle, inframe);
} else if (kpriv->element_mode == VVAS_ELEMENT_MODE_PASSTHROUGH) {
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"input %d is passthrough", idx);
}
idx++;
} else
break;
}
}
Allocate frames using VVAS MEMORY MANGEMENT APIs. This example demonstrates the allocation with specific memory types and properties.
/*
* Allocate VVAS frames using VVAS MEMORY MANGEMENT APIs
*/
/*
* In this example we are going to allocate frame as VVAS_FRAME_MEMORY
* and VVAS_INTERNAL_MEMORY using VVAS MEMORY MANGEMENT APIs
*/
/* VVAS_FRAME_MEMORY validation */
/* In case there is frequent memory allocation/de-allocation is required,
then we can allocate memory from the gstreamer memory pool.
To allocate memory from memory pool managed by the GStreamer Infrastructure plug-in,
developer must pass mem_type parameter value as VVAS_FRAME_MEMORY */
/*
* Set the frame properties like height, width and format of the frame
* which could be used by infrastructure plugin to create the memory pool
*/
user_frame_props.height = user_height;
user_frame_props.width = user_width;
user_frame_props.fmt = VVAS_VFMT_Y_UV8_420;
user_mem_type = VVAS_FRAME_MEMORY;
user_mem_bank = STUB_DEFAULT_MEM_BANK;
/* If memtype is a VVAS_FRAME_MEMORY size is calculated by gstreamer plugin, so setting it to 0 */
user_buf_size = 0;
user_frame =
vvas_alloc_buffer (handle, user_buf_size, user_mem_type, user_mem_bank,
&user_frame_props);
if (user_frame == NULL) {
LOG_MESSAGE (LOG_LEVEL_ERROR, kpriv->log_level,
"Couldn't allocate vvas frame");
return VVAS_RET_ERROR;
}
if (user_frame->app_priv) {
GstVideoMeta *vmeta;
GstBuffer *buffer = (GstBuffer *) user_frame->app_priv;
int p;
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"Allocated vvas frame from gst memory pool %d", user_frame->n_planes);
vmeta = gst_buffer_get_video_meta (buffer);
if (vmeta == NULL) {
LOG_MESSAGE (LOG_LEVEL_ERROR, kpriv->log_level,
"Could not find vmeta in gst buffer");
return VVAS_RET_ERROR;
}
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"Gst Buffer height %d width %d Planes %d", vmeta->height, vmeta->width,
vmeta->n_planes);
for (p = 0; p < vmeta->n_planes; p++) {
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"stride of plane_%d is %d ", p, vmeta->stride[p]);
}
for (p = 0; p < user_frame->n_planes; p++) {
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"plane %d : bo %p vaddr %p paddr %p size %u", p, user_frame->bo[p],
user_frame->vaddr[p], (void *) user_frame->paddr[p],
user_frame->size[p]);
}
}
vvas_free_buffer (handle, user_frame);
/* VVAS_INTERNAL_MEMORY validation */
user_mem_type = VVAS_INTERNAL_MEMORY;
user_buf_size = user_height * user_width * 1.5;
/* No need to pass VVASFrameProps while using VVAS_INTERNAL_MEMORY */
user_frame =
vvas_alloc_buffer (handle, user_buf_size, user_mem_type, user_mem_bank,
NULL);
if (user_frame == NULL) {
LOG_MESSAGE (LOG_LEVEL_ERROR, kpriv->log_level,
"Couldn't allocate vvas frame");
return VVAS_RET_ERROR;
}
if (user_frame->bo[0]) {
int p = 0;
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"Allocated vvas frame from internal memory");
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"plane %d : bo %p vaddr %p paddr %p size %u", p, user_frame->bo[0],
user_frame->vaddr[0], (void *) user_frame->paddr[0],
user_frame->size[0]);
}
vvas_free_buffer (handle, user_frame);
The function is used to validate the vvas frame data. This function checks various frame properties and accordingly prints the log information and in case of error returns the negative result.
int32_t
validate_vvas_frame (VVASKernel * handle, VVASFrame * frame)
{
int p;
VVASXStub *kpriv = (VVASXStub *) handle->kernel_priv;
if (frame->n_planes > VIDEO_MAX_PLANES) {
LOG_MESSAGE (LOG_LEVEL_ERROR, kpriv->log_level,
"wrong number of planes in frame %u", frame->n_planes);
return -1;
}
for (p = 0; p < frame->n_planes; p++) {
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"plane %d : bo %p vaddr %p paddr %p size %u", p, frame->bo[p],
frame->vaddr[p], (void *) frame->paddr[p], frame->size[p]);
if (kpriv->kernel_type == VVAS_KERNEL_LIB_TYPE_HARDKERNEL
|| kpriv->kernel_type == VVAS_KERNEL_LIB_TYPE_PSKERNEL) {
if (frame->paddr[p] == 0 || frame->paddr[p] == (uint64_t) - 1) {
LOG_MESSAGE (LOG_LEVEL_ERROR, kpriv->log_level,
"not valid physical address %lu", frame->paddr[p]);
return -1;
}
/* vaddr not required in case hardkernel and ps_kernel
* but, if gstplugin assigns this it might have resulted DMA copy in PCIe platforms
*/
if (frame->vaddr[p]) {
LOG_MESSAGE (LOG_LEVEL_WARNING, kpriv->log_level,
"vaddr %p available..possibility of DMA copy and hence performance drop",
frame->vaddr[p]);
}
} else {
if (frame->vaddr[p] == 0 || (uint64_t) frame->vaddr[p] == (uint64_t) - 1) {
LOG_MESSAGE (LOG_LEVEL_ERROR, kpriv->log_level,
"not valid virtual address %p", (void *) frame->paddr[p]);
return -1;
}
if (frame->paddr[p]) {
LOG_MESSAGE (LOG_LEVEL_WARNING, kpriv->log_level,
"paddr %p available.. but not required here as kernel type is softlib",
frame->vaddr[p]);
}
}
}
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"frame props : width %u, height %u, stride %u, fmt %u",
frame->props.width, frame->props.height, frame->props.stride,
frame->props.fmt);
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"meta_data %p, app_priv %p, mem_type %u, n_planes %u", frame->meta_data,
frame->app_priv, frame->mem_type, frame->n_planes);
return 0;
}
The following function is used with inplace mode to update the value of frame data.
void
modify_input_frame_inplace (VVASKernel * handle, VVASFrame * frame)
{
int p;
VVASXStub *kpriv = (VVASXStub *) handle->kernel_priv;
for (p = 0; p < frame->n_planes; p++) {
LOG_MESSAGE (LOG_LEVEL_INFO, kpriv->log_level,
"plane %d : bo %p vaddr %p paddr %p size %u", p, frame->bo[p],
frame->vaddr[p], (void *) frame->paddr[p], frame->size[p]);
if (kpriv->kernel_type == VVAS_KERNEL_LIB_TYPE_SOFTLIB
&& (frame->props.fmt == VVAS_VFMT_RGB8
|| frame->props.fmt == VVAS_VFMT_BGR8)) {
memset (frame->vaddr[p], 255, frame->props.stride * frame->props.height);
}
}
}
To determine whether the kernel has completed processing the ongoing frame or buffer, the infrastructure plug-in or an application can invoke the xlnx_kernel_done API.
VvasReturnType
xlnx_kernel_done (VVASKernel * handle)
{
VVASXStub *kpriv = (VVASXStub *) handle->kernel_priv;
LOG_MESSAGE (LOG_LEVEL_DEBUG, kpriv->log_level, "Kernel Done\n");
return VVAS_RET_SUCCESS;
}
xlnx_kernel_deinit is responsible for releasing any resources allocated during the initialization and execution. It performs necessary clean up tasks ,such as freeing private handles and releasing any internally allocated memory.
VvasReturnType
xlnx_kernel_deinit (VVASKernel * handle)
{
VVASXStub *kpriv = (VVASXStub *) handle->kernel_priv;
if (kpriv) {
LOG_MESSAGE (LOG_LEVEL_DEBUG, kpriv->log_level, "enter");
free (kpriv);
}
return VVAS_RET_SUCCESS;
}
Example pipeline using passthrough mode. Pipeline can be updated to other modes with similar formation changes and the json update.
gst-launch-1.0 filesrc location=input.nv12 ! rawvideoparse width=1920 height=1080 format=nv12 ! queue ! \
vvas_xfilter kernels-config=/etc/vvas/configs/xfilter/softlib_stub.json name=xfilter ! queue ! \
fpsdisplaysink video-sink="filesink location=output_passthrough.nv12" text-overlay=false sync=false -v