Dense Pyramidal LK Optical Flow - 2023.2 English

Optical flow is the pattern of apparent motion of image objects between two consecutive frames, caused by the movement of object or camera. It is a 2D vector field, where each vector is a displacement vector showing the movement of points from first frame to second.

Optical Flow works on the following assumptions:

• Pixel intensities of an object do not have too many variations in consecutive frames
• Neighboring pixels have similar motion

Consider a pixel I(x, y, t) in first frame. (Note that a new dimension, time, is added here. When working with images only, there is no need of time). The pixel moves by distance (dx, dy) in the next frame taken after time dt. Thus, since those pixels are the same and the intensity does not change, the following is true:

Taking the Taylor series approximation on the right-hand side, removing common terms, and dividing by dt gives the following equation:

Where , , and .

The above equation is called the Optical Flow equation, where, fx and fy are the image gradientsand ft is the gradient along time. However, (u, v) is unknown. It is not possible to solve this equation with two unknown variables. Thus, several methods are provided to solve this problem. One method is Lucas-Kanade. Previously it was assumed that all neighboring pixels have similar motion. The Lucas-Kanade method takes a patch around the point, whose size can be defined through the ‘WINDOW_SIZE’ template parameter. Thus, all the points in that patch have the same motion. It is possible to find (fx, fy, ft ) for these points. Thus, the problem now becomes solving ‘WINDOW_SIZE * WINDOW_SIZE’ equations with two unknown variables,which is over-determined. A better solution is obtained with the “least square fit” method. Below is the final solution, which is a problem with two equations and two unknowns:

This solution fails when a large motion is involved and so pyramids are used. Going up in the pyramid, small motions are removed and large motions become small motions and so by applying Lucas-Kanade, the optical flow along with the scale is obtained.

API Syntax

template< int NUM_PYR_LEVELS, int NUM_LINES, int WINSIZE, int FLOW_WIDTH, int FLOW_INT, int TYPE, int ROWS, int COLS, int NPC T, bool USE_URAM=false, int XFCVDEPTH_CURR_IMG = _XFCVDEPTH_DEFAULT, int XFCVDEPTH_NEXT_IMG = _XFCVDEPTH_DEFAULT, int XFCVDEPTH_STREAM_IN = _XFCVDEPTH_DEFAULT, int XFCVDEPTH_STREAM_OUT = _XFCVDEPTH_DEFAUL>
void densePyrOpticalFlow(
xf::cv::Mat<TYPE,ROWS,COLS,NPC, XFCVDEPTH_CURR_IMG> & _current_img,
xf::cv::Mat<TYPE,ROWS,COLS,NPC, XFCVDEPTH_NEXT_IMG> & _next_image,
xf::cv::Mat<XF_32UC1,ROWS,COLS,NPC, XFCVDEPTH_STREAM_IN> & _streamFlowin,
xf::cv::Mat<XF_32UC1,ROWS,COLS,NPC, XFCVDEPTH_STREAM_OUT> & _streamFlowout,
const int level, const unsigned char scale_up_flag, float scale_in, ap_uint<1> init_flag)

Parameter Descriptions

The following table describes the template and the function parameters.

Resource Utilization

The following table summarizes the resource utilization of densePyrOpticalFlow for 1 pixel per cycle implementation, with the optical flow computed for a window size of 11 over an image size of 1920x1080 pixels. The results are after implementation in Vivado HLS 2019.1 for the Xilinx xczu9eg-ffvb1156-2L-e FPGA at 300 MHz.

Resource Utilization with UltraRAM Enable

The following table summarizes the resource utilization of densePyrOpticalFlow for 1 pixel per cycle implementation, with the optical flow computed for a window size of 11 over an image size of 3840X2160 pixels. The results are after implementation in Vivado HLS 2019.1 for the Xilinx xczu7ev-ffvc1156-2 FPGA at 300 MHz with UltraRAM enabled.

Performance Estimate

The following table summarizes performance figures on hardware for the densePyrOpticalFlow function for 5 iterations over 5 pyramid levels scaled down by a factor of two at each level. This has been tested on the zcu102 evaluation board.