Top-Level Architecture
The Image Signal Processor (ISP) extends PS functionality with multi-media processing and display capabilities. The main function of the ISP is to preprocess RAW image sensor data and make it available for human viewing and/or machine vision processing. The key use cases for ISP are in the automotive systems that require multiple camera interfaces, each of which needs an inline ISP.
Depending on the device selected, there can be three ISP tiles or a single ISP tile. Each is comprised of two ISP instances: the first instance with up to four streaming inputs and the second instance with up to two streaming inputs. Collectively the two instances provide up to five streams per tile, arranged either in a 3+2 or 4+1 configuration, allowing single and multiple stream configurations.
Feature Summary
The following outlines the features and requirements of each ISP instance:
- A maximum pixel rate of 600 megapixel/second (includes blanking for an effective 540 megapixel/second rate aggregate per ISP instance), which is approximately 4096x3072@40fps / 3840x2160@60fps.
- Supports a maximum horizontal or vertical resolution of 4096 pixels.
- Compatible with standard Bayer input (RGGB, GRBG, GBRG, BGGR), monochrome (CCCC), RYYCy, RCCG, RCCC, and RGB-IR (with 4x4 pattern) sensor types.
- Supports input pixel depths in linear (8, 10, 12, 14, 16 bits) and compressed (up to 24 bits) formats. Pixel depth does not impact maximum pixel rate.
- Has an AXI4-Stream input interface that accepts streaming live data from a Mobile Industry Processor Interface (MIPI) Camera Serial Interface-2 (CSI-2).
- Accepts Memory-input data from DMA Read functionality.
- Accepts input test patterns from in-built Test Pattern Generator (TPG).
- Offers video output formats such as YUV 4:2:0, YUV 4:2:2, Y only, 8 or 10 bits per component, and RGB888 supported through AXI4-streaming (for Live out) and AXI4-memory mapped interfaces (for Memory out).
- Has dual output capability per ISP instance enabling primary output and secondary output with separate controls. One input stream can be processed by a single ISP instance (at the same time) for different primary and secondary output streams.
- Has the capability to process RGB-IR image sensor data and provides RGB data on the primary output and IR data on the secondary output.
- Has Functional Safety (FuSa) support that encompasses ASIL-B random and ASIL-D systematic protocols.
- Supports a 48-bit system address space.
- Includes Statistics DMA to write image statistics information generated during the frame processing to memory.
- In Memory Out I/O type, both primary and secondary output DMA supports raster half-DWORD aligned frame buffer format suitable for 10-bit max color depth.
- Digital zoom is supported on both the primary and secondary output.
Modes of Operations and I/O Types
The ISP instance operates in one of two modes with I/O types as follows:
- MCM mode (multiple stream): The ISP MCM (Multi-Context Manager) makes it possible for a single ISP instance to support up to four streams of input with LIMO.
- Non-MCM mode (single stream): supports single stream with LILO, LIMO and MIMO.
The I/O types are:
MIMO (Memory In Memory Out)
In this I/O type, images in the memory serve as input to the ISP instance, undergoing processing within the ISP pipeline, and subsequently, the processed output image data is written to the DDR memory output DMA write channel.
LILO (Live In Live Out)
In this I/O type, both inputs and outputs are active live streams. The image sensor captures the data and forwards to the ISP instance to get the enhanced image quality live output, without being stored in memory.
LIMO (Live In Memory Out)
In this I/O type, single or multiple live inputs coming from image sensors are processed in the ISP instance to get the enhanced image quality outputs, which will be written to the DDR memory using DMA write operation.
The LIMO I/O type is supported in both non-MCM and MCM modes.
Non-MCM Mode
LIMO I/O type operating in non-MCM mode processes a single live input and produces corresponding outputs.
MCM Mode
Up to four incoming live video streams (from four sensor streams) are directed to individual frame buffers in the DDR memory using dedicated DMA write channels. These streams are subsequently fed sequentially to the ISP instance through a single DMA read channel. The ISP processes this aggregated data and writes the image output data to the DDR memory using output DMA write channels.
- ISP instance 0 can accommodate up to 4 streaming inputs, and the ISP instance 1 can accommodate up to 2 streaming inputs.
- Each input can be independent in terms of resolution and frame rate.
- Different video formats are supported for each input.
Refer to the following diagram that illustrates the ISP tile architecture:
Primary and Secondary Outputs
In all operating modes, ISP supports two outputs. Both outputs are derived from the same input stream, but may be independently cropped, scaled, and formatted. In addition, in RGB-IR mode it shall be possible for the secondary output to carry IR-only data, while the primary output carries RGB data.
For more information on the Image Signal Processor, see Versal AI Edge Series Gen 2 Image Signal Processor (ISP) Product Guide (PG432).