Image Sensor Processing (ISP) is a pipeline of image processing functions processing the raw image from the sensor.
This ISP includes following blocks:
- Black level correction: Black level leads to the whitening of image in dark region and perceived loss of overall contrast. The Blacklevelcorrection algorithm corrects the black and white levels of the overall image.
- BPC (Bad pixel correction): An image sensor may have a certain number of defective/bad pixels that may be the result of manufacturing faults or variations in pixel voltage levels based on temperature or exposure. Bad pixel correction module removes defective pixels.
- Gain Control: The Gain control module improves the overall brightness of the image.
- Demosaicing: The demosaic module reconstructs RGB pixels from the input Bayer image (RGGB,BGGR,RGBG,GRGB).
- Auto white balance: The AWB module improves color balance of the image by using image statistics.
- Colorcorrection matrix: corrects color suitable for display or video system.
- Quantization and Dithering: Quantization and Dithering performs the uniform quantization to also reduce higher bit depth to lower bit depths.
- Gamma correction: Gamma correction improves the overall brightness of image.
- Color space conversion: Converting RGB image to YUV422(YUYV) image for HDMI display purpose. RGB2YUYV converts the RGB image into Y channel for every pixel and U and V for alternate pixels.
- Resize: Resolution Conversion is the method used to resize the source image to the size of the destination image. Different types of interpolation techniques can be used in resize function, namely: Nearest-neighbor, Bilinear, and Area interpolation. The type of interpolation can be passed as a template parameter to the API.
Current design example demonstrates how to use ISP functions in a pipeline.
User can dynamically configure the below parameters to the pipeline.
| Parameter | Description |
|---|---|
| common_config | unsigned integer 32-bit value. [31:16] height, [15:0] width height range [2160 to 128] width range [3840 to 128] |
| blc_config_1 | unsigned integer 32-bit value. Black level value Range [65535 to 0] |
| blc_config_2 | unsigned integer. Multiplication factor for blacklevel correction; which is computed as (unsigned int)((maxlevel/(maxlevel-blacklevel)) * Range [65535 to 0] maxlevel for 16 bit = 65535 maxlevel for 8 bit = 255 |
| resize_config | unsigned integer 32-bit value [31:16] new_height, [15:0] new_width new_height range [2160 to 128] new_width range [3840 to 128] |
| gain_control_config_1 | unsigned integer 32-bit value [31:16] bgain, [15:0] rgain bgain range [1024 to 0] rgain range [1024 to 0] |
| gain_control_config_2 | unsigned integer 32-bit value [17 : 16] bformat, [15:0] ggain ggain range [1024 to 0] bformat range [3 to 0] 0 - XF_BAYER_BG 1 - XF_BAYER_GB 2 - XF_BAYER_GR 3 - XF_BAYER_RG |
| awb_config | unsigned integer 32-bit value Threshold value, which is used in gray world white balance method to compute average pixel values below the threshold value. Range [65535 to 0] |
| gamma_lut | unsigned char [256 * 3] Lookup table for gamma values.first 256 will be R, next 256 values + are G gamma and last 256 values are B values |
| ccm_config_1 | signed integer. ccm_matrix Computed as (signed int)(ccm_matrix_float * 1048576) ccm_matrix_float is a 3×3 float matrix |
| ccm_config_2 | signed integer. offsetarray Computed as(signed int)(offsetarray_float * 1048576) offsetarray_float is floating point array of size 3 |
| ltm_config | 32-bit value. [31:16] block_height [15:0] block_width block_height range [32 to (Image-width / 2)] block_width range [32 to (Image-width / 2)] |
| pipeline_config_info | unsigned integer 32-bit value. bit - 0 -> IN_TYPE,0 -Pipeline processing single(Mono)
bit - 4 -> OUT_TYPE, OUT_C_TYPE : (Optional ) bit - [7 : 5] -> OUT_BW_MODE, output frame bit-depth info
bit - [11 : 8] -> NPC, xf::cv::log2<XF_NPPCX>::fvalue bit - [16 : 12] -> NUM_STREAMS(Not configurable
bit - [31 : 17] -> Reserved, |
| max_supported_size | unsigned integer 32-bit value [31:16] MAX_HEIGHT [15:0] MAX_WIDTH MAX_HEIGHT range [2160 to 128] MAX_WIDTH range [3840 to 128] |
| funcs_available | unsigned integer 32-bit value Compile time bypass.(READ REGISTER) set corresponding bit to ‘0’ to prevent the function from compiling. set corresponding bit to ‘1’ to compile the module. bit - 0 -> HDR_MODE(Not available in this example), bit - 1 -> HDR_EN(Not available in this example), bit - 2 -> RGBIR_EN(Not configurable
bit - 4 -> BLC_EN, bit - 5 -> BPC_EN (Not available in this example), bit - 6 -> DEGAMMA_EN(Not available in this example), bit - 7 -> LSC_EN(Not available in this example), bit - 8 -> GAIN_EN, bit - 9 -> DEMOSAIC_EN(Not configurable
bit - 10 -> AWB_EN, bit - 11 -> CCM_EN, bit - 12 -> TM_EN, bit - [14 : 13] -> TM_TYPE, bit - 15 -> GAMMA_EN, bit - 16 -> 3DLUT_EN(Not available in this example), bit - 17 -> CSC_EN(Not configurable
bit - [31 : 24] -> Reserved, |
| funcs_bypassable | unsigned integer 32-bit value.(READ REGISTER) set corresponding bit to ‘1’ to get bypass module at run time feature. set corresponding bit to ‘0’ to not to have bypass module at run time feature. bit - 0 -> ISP_BYPASS_EN(Not available in this
bit - 4 -> BLC_BYPASS_EN, bit - 5 -> BPC_BYPASS_EN(Not available in this
bit - 8 -> GAIN_BYPASS_EN, bit - 9 -> DEMOSAIC_BYPASS_EN(Not configurable
bit - 10 -> AWB_BYPASS_EN, bit - 11 -> CCM_BYPASS_EN, bit - 12 -> TM_BYPASS_EN, bit - [14 : 13] -> Reserved, bit - 15 -> GAMMA_BYPASS_EN, bit - 16 -> 3DLUT_BYPASS_EN(Not available in this
bit - [31 : 24] -> Reserved, |
| funcs_bypass_config | unsigned integer 32-bit value (WRITE REGISTER) bit - 0 -> BYPASS_ISP(Not available in this
bit - 4 -> BYPASS_BLC, bit - 5 -> BYPASS_BPC(Not available in this
bit - 8 -> BYPASS_GAIN, bit - 9 -> BYPASS_DEMOSAIC(Not configurable
bit - 10 -> BYPASS_AWB, bit - 11 -> BYPASS_CCM, bit - 12 -> BYPASS_TM, bit - [14 : 13] -> Reserved, bit - 15 -> BYPASS_GAMMA, bit - 16 -> BYPASS_3DLUT(Not available in this
bit - [31 : 24] -> Reserved, |
User can also use below compile time parameters to the pipeline.
| Parameter | Description |
|---|---|
| XF_HEIGHT | Maximum height of input and output image range [2160 to 128] |
| XF_WIDTH | Maximum width of input and output image (Must be a multiple of NPC) range [2160 to 128] |
| XF_NPPCX | Number of pixels to be processed per cycle; possible options are XF_NPPC1 and XF_NPPC8 for 1, 2, 4, 8 pixel operations respectively. |
| IN_TYPE | Input pixel type,Supported pixel widths are XF_8UC1, XF_10UC1, XF_12UC1, XF_16UC1 |
| XF_GTM_T | Input pixel type,Supported pixel widths are XF_8UC3 |
| OUT_TYPE | output pixel type,Supported pixel widths are XF_8UC1, XF_10UC1, XF_12UC1, XF_16UC1 |
| XF_USE_URAM | Enable to map storage structures to UltraRAM. |
| WB_TYPE | White balance type. Supported types are 0 - Gray world and 1 - simple. |
| SCALE_FACTOR | The SCALE_FACTOR must be power of 2 & less than or equal to 2^(output PIXEL_BITWIDTH) |
| MAX_REPRESENTED_VALUE | The MAX_REPRESENTED_VALUE must be equal to equal to 2^(input PIXEL_BITWIDTH) |
| INTERPOLATION | Type of interpolation, either 0 - XF_INTERPOLATION_NN (nearest neighbor) or 1 - XF_INTERPOLATION_BILINEAR (linear interpolation) |
| XF_NEWHEIGHT | Maximum Height of output image for which the hardware kernel would be built. range [2160 to 128] |
| XF_NEWWIDTH | Maximum Width of output image for which the hardware kernel would be built (must be a multiple of 8). range [2160 to 128] |
| MAXDOWNSCALE | Set to 2 for all 1 pixel modes, and for upscale in x direction. When down scaling in x direction in 8-pixel mode, please set this parameter to the next highest integer value of the down scale factor i.e., if downscaling from 1920 columns to 1280 columns, set to 2. For 1920 to 640, set to 3. |
| MUL_VALUE_WIDTH | Width of multiplication factor (16). |
| FL_POS | Number of fractional bits in multiplication factor(15). |
| USE_DSP | Enables usage of DSP for multiplication. |
| BLOCK_HEIGHT | Max block height the image is divided into. This can be any positive integer greater than or equal to 32 and less than input image height. |
| BLOCK_WIDTH | Max block width the image is divided into. This can be any positive integer greater than or equal to 32 and less than input image width. |
| BIL_T | BILINEAR INTERPOLATE TYPE. XF_32FC3 |
The following example demonstrates the ISP pipeline with above list of functions.
void ISPPipeline_accel(InVideoStrm_t& s_axis_video,
OutVideoStrm_t& m_axis_video,
unsigned int common_config,
#if XF_BLC_EN
unsigned int blc_config_1,
unsigned int blc_config_2,
#endif
unsigned int resize_config,
#if XF_GAIN_EN
unsigned int gain_control_config_1,
unsigned int gain_control_config_2,
#endif
#if XF_AWB_EN
unsigned int awb_config,
#endif
#if XF_GAMMA_EN
unsigned char gamma_lut[256 * 3],
#endif
#if XF_CCM_EN
signed int ccm_config_1[3][3],
signed int ccm_config_2[3],
#endif
#if (XF_TM_TYPE == 0)
uint32_t ltm_config,
#endif
unsigned int& pipeline_config_info,
unsigned int& max_supported_size,
unsigned int& funcs_available,
unsigned int& funcs_bypassable,
unsigned int funcs_bypass_config) {
// clang-format off
#pragma HLS INTERFACE axis port=s_axis_video register
#pragma HLS INTERFACE axis port=m_axis_video register
#pragma HLS INTERFACE s_axilite port=common_config bundle=CTRL offset=0x00010
#pragma HLS INTERFACE s_axilite port=resize_config bundle=CTRL offset=0x00078
#if XF_GAIN_EN
#pragma HLS INTERFACE s_axilite port=gain_control_config_1 bundle=CTRL offset=0x00038
#pragma HLS INTERFACE s_axilite port=gain_control_config_2 bundle=CTRL offset=0x00040
#endif
#if XF_AWB_EN
#pragma HLS INTERFACE s_axilite port=awb_config bundle=CTRL offset=0x00020
#endif
#if XF_GAMMA_EN
#pragma HLS INTERFACE s_axilite port=gamma_lut bundle=CTRL offset=0x02000
#endif
#if XF_CCM_EN
#pragma HLS INTERFACE s_axilite port=ccm_config_1 bundle=CTRL offset=0x04000
#pragma HLS INTERFACE s_axilite port=ccm_config_2 bundle=CTRL offset=0x04100
#endif
#if XF_BLC_EN
#pragma HLS INTERFACE s_axilite port=blc_config_1 bundle=CTRL offset=0x00028
#pragma HLS INTERFACE s_axilite port=blc_config_2 bundle=CTRL offset=0x00030
#endif
#if (XF_TM_TYPE == 0)
#pragma HLS INTERFACE s_axilite port=ltm_config bundle=CTRL offset=0x00058
#endif
#pragma HLS INTERFACE s_axilite port=pipeline_config_info bundle=CTRL offset=0x00080
#pragma HLS INTERFACE mode=ap_none port=pipeline_config_info
#pragma HLS INTERFACE s_axilite port=max_supported_size bundle=CTRL offset=0x00090
#pragma HLS INTERFACE mode=ap_none port=max_supported_size
#pragma HLS INTERFACE s_axilite port=funcs_available bundle=CTRL offset=0x000a0
#pragma HLS INTERFACE mode=ap_none port=funcs_available
#pragma HLS INTERFACE s_axilite port=funcs_bypassable bundle=CTRL offset=0x000b0
#pragma HLS INTERFACE mode=ap_none port=funcs_bypassable
#pragma HLS INTERFACE s_axilite port=funcs_bypass_config bundle=CTRL offset=0x000c0
#pragma HLS INTERFACE s_axilite port=return bundle=CTRL
// clang-format on
// extracting height and width //
uint16_t height, width;
height = (uint16_t)(common_config >> 16);
width = (uint16_t)(common_config);
// extracting new_height and new_width //
uint16_t new_height, new_width;
new_height = (uint16_t)(resize_config >> 16);
new_width = (uint16_t)(resize_config);
#if XF_GAIN_EN
// extracting rgain and bgain //
uint16_t rgain, bgain;
bgain = (uint16_t)(gain_control_config_1 >> 16);
rgain = (uint16_t)(gain_control_config_1);
uint16_t bformat;
bformat = (uint16_t)(gain_control_config_2 >> 16);
uint16_t ggain;
ggain = (uint16_t)(gain_control_config_2);
#endif
#if XF_AWB_EN
// awb specific data //
float inputMin = 0.0f;
float inputMax = (1 << (XF_DTPIXELDEPTH(IN_TYPE, XF_NPPCX))) - 1; // 65535.0f;
float outputMin = 0.0f;
float outputMax = (1 << (XF_DTPIXELDEPTH(IN_TYPE, XF_NPPCX))) - 1; // 65535.0f;
// clang-format off
#pragma HLS ARRAY_PARTITION variable=hist0_awb complete dim=1
#pragma HLS ARRAY_PARTITION variable=hist1_awb complete dim=1
#endif
#if (XF_TM_TYPE == 0)
uint32_t block_height = (uint16_t)(ltm_config >> 16);;
uint32_t block_width= (uint16_t)(ltm_config);
#pragma HLS ARRAY_PARTITION variable=omin dim=1 complete
#pragma HLS ARRAY_PARTITION variable=omin dim=2 cyclic factor=2
#pragma HLS ARRAY_PARTITION variable=omin dim=3 cyclic factor=2
#pragma HLS ARRAY_PARTITION variable=omax dim=1 complete
#pragma HLS ARRAY_PARTITION variable=omax dim=2 cyclic factor=2
#pragma HLS ARRAY_PARTITION variable=omax dim=3 cyclic factor=2
#endif
// clang-format on
max_supported_size = (XF_WIDTH << MAX_WIDTH_INDEX) | (XF_HEIGHT << MAX_HEIGHT_INDEX);
pipeline_config_info = (IN_C_TYPE << IN_C_TYPE_INDEX) | (IN_BW_MODE << IN_BW_MODE_INDEX) |
(OUT_C_TYPE << OUT_C_TYPE_INDEX) | (OUT_BW_MODE << OUT_BW_MODE_INDEX) |
(NPPCX << NPPCX_INDEX) | (NUM_STREAMS << NUM_STREAMS_INDEX);
funcs_available = (XF_BLC_EN << XF_BLC_EN_INDEX) | (XF_GAIN_EN << XF_GAIN_EN_INDEX) |
(XF_AWB_EN << XF_AWB_EN_INDEX) | (XF_CCM_EN << XF_CCM_EN_INDEX) | (XF_TM_EN << XF_TM_EN_INDEX) |
(XF_TM_TYPE << XF_TM_TYPE_INDEX) | (XF_GAMMA_EN << XF_GAMMA_EN_INDEX);
funcs_bypassable = (XF_BLC_BYPASS_EN << XF_BLC_EN_INDEX) | (XF_GAIN_BYPASS_EN << XF_GAIN_EN_INDEX) |
(XF_AWB_BYPASS_EN << XF_AWB_EN_INDEX) | (XF_CCM_BYPASS_EN << XF_CCM_EN_INDEX) |
(XF_GAMMA_BYPASS_EN << XF_GAMMA_EN_INDEX);
if (!flag) {
ISPpipeline(s_axis_video, m_axis_video, height, width,
new_height, new_width,
#if XF_BLC_EN
blc_config_1, blc_config_2,
#endif
#if XF_GAIN_EN
rgain, bgain, ggain, bformat,
#endif
#if XF_AWB_EN
hist0_awb, awb_config, inputMin, inputMax, outputMin, outputMax, hist1_awb,
#endif
#if XF_GAMMA_EN
gamma_lut,
#endif
#if XF_CCM_EN
ccm_config_1, ccm_config_2,
#endif
#if (XF_TM_TYPE == 0)
block_height, block_width, omin[0], omax[0], omin[1], omax[1],
#endif
funcs_bypass_config);
flag = 1;
} else {
ISPpipeline(s_axis_video, m_axis_video, height, width,
new_height, new_width,
#if XF_BLC_EN
blc_config_1, blc_config_2,
#endif
#if XF_GAIN_EN
rgain, bgain, ggain, bformat,
#endif
#if XF_AWB_EN
hist1_awb, awb_config, inputMin, inputMax, outputMin, outputMax, hist0_awb,
#endif
#if XF_GAMMA_EN
gamma_lut,
#endif
#if XF_CCM_EN
ccm_config_1, ccm_config_2,
#endif
#if (XF_TM_TYPE == 0)
block_height, block_width, omin[0], omax[0], omin[1], omax[1],
#endif
funcs_bypass_config);
flag = 0;
}
}