Dense Non-Pyramidal LK Optical Flow - 2024.1 English

Vitis Libraries

Release Date
2024-05-30
Version
2024.1 English

Optical flow is the pattern of apparent motion of image objects between two consecutive frames, caused by the movement of object or camera. It is a 2D vector field, where each vector is a displacement vector showing the movement of points from first frame to second.

Optical Flow works on the following assumptions:

  • Pixel intensities of an object do not have too many variations in consecutive frames
  • Neighboring pixels have similar motion
Consider a pixel I(x, y, t) in first frame. (Note that a new dimension, time, is added here. When working with images only, there is no need of time). The pixel moves by distance (dx, dy) in the next frame taken after time dt. Thus, since those pixels are the same and the intensity does not change, the following is true:
image99

Taking the Taylor series approximation on the right-hand side, removing common terms, and dividing by dt gives the following equation:

image100

Where image101, image102, image103 and image104.

The above equation is called the Optical Flow equation, where, fx and fy are the image gradientsand ft is the gradient along time. However, (u, v) is unknown. It is not possible to solve this equation with two unknown variables. Thus, several methods are provided to solve this problem. One method is Lucas-Kanade. Previously it was assumed that all neighboring pixels have similar motion. The Lucas-Kanade method takes a patch around the point, whose size can be defined through the ‘WINDOW_SIZE’ template parameter. Thus, all the points in that patch have the same motion. It is possible to find (fx, fy, ft ) for these points. Thus, the problem now becomes solving ‘WINDOW_SIZE * WINDOW_SIZE’ equations with two unknown variables,which is over-determined. A better solution is obtained with the “least square fit” method. Below is the final solution, which is a problem with two equations and two unknowns:

image105

API Syntax

template< int WINDOW_SIZE, int TYPE, int ROWS, int COLS, int NPC, bool USE_URAM=false, int XFCVDEDPTH_IN_0 = _XFCVDEPTH_DEFAULT, int XFCVDEDPTH_IN_1 = _XFCVDEPTH_DEFAULT, int XFCVDEDPTH_IN_2 = _XFCVDEPTH_DEFAULT, int XFCVDEDPTH_IN_3 = _XFCVDEPTH_DEFAULT>
void DenseNonPyrLKOpticalFlow (xf::cv::Mat<TYPE, ROWS, COLS, NPC, XFCVDEDPTH_IN_0> & frame0, xf::cv::Mat<TYPE, ROWS, COLS, NPC, XFCVDEDPTH_IN_1> & frame1, xf::cv::Mat<XF_32FC1, ROWS, COLS, NPC, XFCVDEDPTH_IN_2> & flowx, xf::cv::Mat<XF_32FC1, ROWS, COLS, NPC, XFCVDEDPTH_IN_3> & flowy)

Parameter Descriptions

The following table describes the template and the function parameters.

Table 629 Table . DenseNonPyrLKOpticalFlow Function Paramete Descriptions
Parameter Description
Type pixel type. The current supported pixel value is XF_8UC1, unsigned 8 bit.
ROWS Maximum number of rows of the input image that the hardware kernel must be built for.
COLS Maximum number of columns of the input image that the hardware kernel must be built for.
NPC Number of pixels to process per cycle. Supported values are XF_NPPC1 (=1) and XF_NPPC2(=2).
WINDOW_SIZE Window size over which optical flow will be computed. This can be any odd positive integer.
USE_URAM Enable to map storage structures to UltraRAM.
XFCVDEDPTH_IN_0 Depth of the input image.
XFCVDEDPTH_IN_1 Depth of the output image.
XFCVDEDPTH_IN_2 Depth of the input image.
XFCVDEDPTH_IN_3 Depth of the output image.
frame0 First input images.
frame1 Second input image. Optical flow is computed between frame0 and frame1.
flowx Horizontal component of the flow vectors. The format of the flow vectors is XF_32FC1 or single precision.
flowy Vertical component of the flow vectors. The format of the flow vectors is XF_32FC1 or single precision.

Resource Utilization

The following table summarizes the resource utilization of DenseNonPyrLKOpticalFlow for a 4K image, as generated in the Vivado HLS 2019.1 version tool for the Xilinx Xczu9eg-ffvb1156-1-i-es1 FPGA at 300 MHz.

Table 630 Table . DenseNonPyrLKOpticalFlow Function Resource Utilization Summary
Operating Mode

Operating Frequency

(MHz)

Utilization Estimate
BRAM_18K DSP_48Es FF LUTs
1 pixel 300 178 42 11984 7730
2 pixel 300 258 82 22747 15126

The following table summarizes the resource utilization of DenseNonPyrLKOpticalFlow for a 4K image, as generated in the Vivado HLS version tool for the Xilinx Xczu7eg-ffvb1156-1 FPGA at 300 MHz with UltraRAM enabled.

Table 631 Table . DenseNonPyrLKOpticalFlow Function Resource Utilization Summary with UltraRAM Eanble
Operating Mode

Operating Frequency

(MHz)

Utilization Estimate
BRAM_18K URAM DSP_48Es FF LUTs
1 pixel 300 0 12 42 11803 7469
2 pixel 300 0 23 80 22124 13800

Performance Estimate

The following table summarizes performance estimates of the DenseNonPyrLKOpticalFlow function for a 4K image, generated using Vivado HLS 2019.1 version tool for the Xilinx Xczu9eg-ffvb1156-1-i-es1 FPGA.

Table 632 Table . DenseNonPyrLKOpticalFlow Function Performance Estimate Summary
Operating Mode

Operating Frequency

(MHz)

Latency Estimate
Max (ms)
1 pixel 300 28.01
2 pixel 300 14.01