Understanding Holoscan Sensor Bridge and TintE ISP Performance in Embedded cameras

Latency refers to the delay between the moment a camera captures a scene and when that scene is displayed on the viewer’s device. In embedded vision systems, latency is a critical factor—as important as high-resolution imaging. Both directly impact the accuracy and responsiveness of decision-making, which is important in time-sensitive applications such as AI-driven security systems, autonomous vehicles, or automated quality inspection systems.

In this blog, you’ll learn how e-con Systems’ embedded camera, combined with Lattice’s Holoscan Sensor Bridge and e-con Systems’ TintE ISP, achieves low-latency transmission on NVIDIA Jetson Orin platforms.

About e-con Systems’ e-CAM85_CUHSB

The e-CAM85_CUHSB camera is integrated with the NVIDIA® Holoscan Sensor Bridge (HSB), which leverages Lattice’s FPGA technology and e-con Systems’ TintE ISP and Holoscan Sensor IP on the NVIDIA Jetson Orin platforms.  This collaborative solution offers broad interface support and high-throughput packetization for AI acceleration on the NVIDIA Jetson AGX Orin platform.

This camera is also designed to be compatible with the upcoming NVIDIA Jetson Thor series, which delivers up to 2070 FP4 TFLOPS for edge AI applications like humanoid robots.

Now, let us first understand what the Lattice HSB board is, and then we will delve into its data processing pipeline—both with NVIDIA’s user’ ISP and with the TintE ISP.

What is a Holoscan Sensor Bridge – And How Does It Work?

The Holoscan Sensor Bridge provides an FPGA-based interface that uses GPU RDMA (Remote Direct Memory Access) and high-speed Ethernet to enable low-latency sensor data processing. The Lattice Holoscan Sensor Bridge board incorporates two powerful Lattice FPGAs, each playing a specialized role: CrossLink-NX FPGA handles the reception of high-bandwidth MIPI camera data, while the other CertusPro-NX FPGA is responsible for transmitting this data at multi-gigabit speeds over 10G SFP+ Ethernet ports.

It can also be adapted to a single FPGA architecture in production to simplify the design and improve efficiency, although the Lattice HSB setup utilizes two FPGAs for flexible development and testing. It’s a ready-to-use, configurable Holoscan Sensor Bridge IP, paired with the Holoscan software to deliver a sensor-agnostic data-to-Ethernet host platform. This IP simplifies and accelerates FPGA design while offering scalability and configurability to adapt to various sensor-to-host applications.

The board is compatible with the Jetson AGX Orin through socket-based Ethernet connections and the Jetson IGX Orin platforms through ROCE. It supports high-speed data transmission via dual 10G SFP+ Ethernet ports for seamless camera integration.

Now, let’s see how Lattice HSB processes data on the Jetson AGX Orin platform using a user’s ISP with the Holoscan SDK.

Holoscan Data Processing Pipeline with user’s ISP

The flowchart below illustrates the processing of RAW image data on Jetson using a user’s ISP with the Holoscan SDK.

Figure 1: Flowchart of Holoscan Data Processing Pipeline with NVIDIA User’s ISP

This process, as depicted, begins with the e-CAM85_CUHSB camera capturing high-resolution RAW Bayer frames at 4K @ 60FPS or 1080p @ 60FPS with 10/12-bit and transmitting data over a 4-lane MIPI-CSI interface and uses I2C for control. The CrossLink-NX FPGA receives this RAW video via MIPI-CSI and converts it into LVDS format for the next stage of processing. This LVDS data is then sent to the CertusPro-NX FPGA, which decodes it into AXI4-Stream format, packetizes the frame data using UDP, and streams it out through dual 10G SFP+ Ethernet ports.

On the receiving end, the NVIDIA Jetson setup, equipped with the Holoscan SDK, captures these packets using either the LinuxReceiverOperator for AGX or the RoceReceiverOperator for IGX. The NVIDIA Jetson AGX Orin or IGX runs multiple image processing operations via:

• ImageProcessorOperator
• BayerDemosaicOperator
• Inferencing & Postprocessing Operator (Optional)
• HolovizVisualizer

Finally, the processed frames are either directly displayed on a GUI via the HolovizVisualizer operator, or passed through an AI inference pipeline  (Inferencing and Postprocessing Operator) for  object detection, and advanced analytics in a live preview setup.

Now, let’s find out how e-con Systems’ TintE ISP—an FPGA-based ISP is used with Lattice HSB

Holoscan Data Processing pipeline using e-con Systems’ TintE ISP on Lattice HSB

The e-CAM85_CUHSB, a high-resolution camera from e-con Systems, integrates with the NVIDIA Holoscan Sensor Bridge, leveraging  Lattice’s FPGA and TintE ISP, an FPGA-based Image Signal Processor for achieving high performance.

The flowchart below illustrates the data processing on the Jetson AGX Orin platform using an e-con Systems’ TintE ISP – with the Holoscan SDK.

Figure 2: Flowchart of Holoscan Data Processing Pipeline with TintE ISP

The e-CAM85_CUHSB camera connected with HSB performs an initial image stream with a Lattice CrossLink-NX FPGA. The data is then passed to the Lattice CertusPro-NX FPGA, including e-con Systems’ TintE ISP.

The TintE ISP performs key ISP functions such as demosaicing, white balance, color correction, and YUV (UYVY 16-bit) conversion. This enables seamless integration with edge AI platforms like the NVIDIA Jetson AGX Orin and IGX.

Let’s see the brief explanation about the imaging processing pipeline with lattice HSB using TintE ISP in the following section.

Camera Input

The high-resolution e-CAM85_CUHSB camera streams RAW10/12-bit Bayer data over a 4-lane MIPI CSI interface. Sensor parameters such as exposure, gain, resolution, and frame rate are configured via the I2C interface, allowing precise control of the image sensor for optimized performance in downstream processing.

CrossLink-NX FPGA

The incoming MIPI stream is received by the CrossLink-NX FPGA, which performs MIPI D-PHY to LVDS conversion. This step adapts the sensor’s native interface for reliable high-speed transmission to the processing core.

CertusPro-NX FPGA with e-con Systems’ TintE ISP

The CertusPro-NX FPGA serves as the core of the Holoscan Sensor Bridge and integrates three critical components: an LVDS Receiver with an AXI4-Stream Formatter, which prepares incoming video streams for further processing, e-con Systems’ TintE ISP, which performs the complete ISP pipeline, including tasks such as demosaicing, white balance, color correction, and YUV conversion, and a UDP Packetizer, which converts image frames into high-speed UDP packets.

The block diagram below shows the CertusPro-NX FPGA with the TintE ISP.

Figure 3: Block Diagram of CertusPro-NX FPGA with TintE ISP

By offloading the ISP workload from the NVIDIA Jetson Platform, the system maximizes GPU availability for real-time AI inference.

Additionally, e-con Systems customized the Holoscan SDK to enable dynamic runtime control of ISP parameters and support high-resolution still image capture in parallel with live video streaming.

Dual 10G Ethernet

Once image frames are processed and packetized, they are streamed over dual 10G SFP+ Ethernet interfaces, ensuring high bandwidth and low latency for multi-sensor or multi-camera deployments.

Host (NVIDIA Jetson AGX Orin or IGX Orin)

On the host side (Jetson AGX Orin or IGX), the Holoscan SDK takes over using GPU acceleration:

• Linux Receiver Operator: Captures end-of-frame UDP packets to memory.
• CSI to Bayer Operator: Converts CSI-2 embedded RAW data to video frames.
• Format Converter Operator: Resizes and normalizes pixel formats.
• Holoviz Operator: Displays RGB888 data on GUI or feeds it into AI inference.

Customized e-con Systems’ TintE ISP for Lattice HSB Integration

The customization enables the following features such as:

1. UYVY Conversion with TintE ISP
   The TintE ISP converts RAW Bayer data directly into UYVY (16-bit) format on the FPGA. This eliminates the need for additional ISP processing on the host platform.

2. Real-Time Image Enhancements
   It supports essential real-time enhancements like Auto White Balance (AWB), gamma correction, denoising, lens shading correction (LSC), and color correction (CC). These operations help optimize image quality before it reaches the Jetson platform.

3. Reduced Load on NVIDIA AGX Orin/IGX Orin
   By handling the complete ISP pipeline on the FPGA, the TintE ISP significantly reduces the processing load on the NVIDIA Jetson AGX or IGX platform. This frees up valuable GPU resources for AI and computer vision tasks.

4. Dynamic ISP Parameter Control via Holoscan SDK
   The TintE ISP fine-tunes ISP parameters such as Contrast, Brightness, Saturation, Sharpness, Auto Exposure (AE), and Auto White Balance (AWB) in real time using the Holoscan SDK.

Let’s now compare its performance with other commonly used camera interfaces.

Interface Comparison: Latency Test Results

This comparison shows that the Holoscan Sensor Bridge consistently delivers lower latency than MIPI and GMSL interfaces, with the lowest latency of 28ms achieved.

Interface	Sensor	Resolution	Frame rate	ISP	Bits per pixel (NVIDIA Host Input)	Latency (ms)
MIPI (Ipex)	IMX715	1920 x 1080	60	LibArgus ISP	RAW 10bits	55ms
GMSL	IMX715	1920 x 1080	60	LibArgus ISP	RAW 12bits	50ms
Holoscan Sensor Bridge	IMX715	1920 x 1080	60	user’s ISP	RAW 12bits	34ms
Holoscan Sensor Bridge	IMX715	1920 x 1080	60	user’s ISP	RAW 10bits	28ms
Holoscan Sensor Bridge	IMX715	1920 x 1080	60	e-con Systems’ TintE ISP	UYVY 16bit	34ms

Note: TintE ISP was tested with Holoscan SDK version 2.0 GA.

Test Setup: How Latency Was Measured Platform: NVIDIA Jetson AGX Orin platform Display Output: DisplayPort to HDMI monitor at 1080p60 Camera: e-con Systems’ Holoscan Camera for NVIDIA Platforms featuring IMX715 sensor (used for all interface tests) Interfaces Tested: Native MIPI-CSI (Ipex connector) GMSL via serializer-deserializer pair Holoscan Sensor Bridge (HSB) via 10G Ethernet (GPU ISP and the TintE ISP) Frame Format: RAW10, RAW12, and UYVY, depending on the interface and ISP Software Stack: Holoscan SDK or LibArgus pipeline, depending on the input

The latency is measured using a latency meter kit by Science Mosaic, which measures capture-to-display latency.

e-con Systems Offers Low-Latency Cameras 

e-con Systems, a leading camera solution provider since 2003, has consistently pushed the boundaries in embedded vision. As an NVIDIA Elite Partner, e-con Systems brings deep expertise in delivering high-performance cameras tailored for the entire NVIDIA Jetson platform, including Jetson AGX Orin, Jetson Orin NX/Nano, Jetson Xavier NX/Nano/TX2 NX, and Jetson AGX Xavier.

Our camera portfolio supports features such as multi-camera synchronization for up to 8 cameras, resolutions up to 20MP, HDR with LFM, and superior low-light performance—catering to a wide range of embedded vision applications.

To select your custom camera, explore the website at Camera Selector.

For expert guidance on camera integration support, please write to camerasolutions@e-consystems.com.