Most discussions about RGB-IR cameras focus on what they capture. The more consequential question for engineers building DMS systems is what happens to that data after it leaves the sensor, specifically, who does the processing, and what it costs the system when that work lands in the wrong place.
e-con Systems’ STURDeCAM57 includes a built-in Image Signal Processor that handles RGB-IR separation, demosaicing, and color reconstruction entirely inside the camera.
For systems running on the NVIDIA AGX Orin platform, it changes the computational economics of the entire imaging pipeline and has direct consequences for the quality of AI inference downstream.
This blog explains why this is important, how it works, and what it means specifically for Driver Monitoring Systems.
The Hidden Cost of RAW RGB-IR Data
An RGB-IR sensor does not natively produce two clean image streams. What it produces is a single RAW frame from a modified color filter array — a 4×4 pixel pattern that mixes red, green, blue, and infrared-sensitive pixels together. The ISP has to untangle that mixture before either stream is usable.
That involves:
- IR contamination removal: IR light leaks into the R, G, and B channels. Without correction, this contamination distorts color output — red signals shift toward yellow, white lights produce halos, and color-dependent AI models receive corrupted input. The ISP must estimate the IR component at each RGB pixel and subtract it using pre-calibrated coefficients.
- Demosaicing: The RAW mixed-pattern data has to be interpolated into full-resolution RGB and IR frames separately. High-quality demosaicing is computationally intensive. Benchmarks on the NVIDIA AGX Orin show that processing RAW16 data through a high-quality demosaic algorithm alone can consume up to 72% of GPU load at 4K resolution.
- Color reconstruction: After IR subtraction and demosaicing, the RGB output still requires white balance, tone mapping, and color correction to produce perceptually accurate imagery suitable for AI inference.
What STURDeCAM57’s Onboard ISP Actually Does
e-con Systems’ STURDeCAM57 performs all three processing steps inside the camera before any data reaches the NVIDIA AGX Orin. The host receives two fully processed, independent streams, comprising a clean RGB frame and a clean IR frame, ready for AI consumption.
This means the NVIDIA platform’s GPU never handles IR contamination math, demosaicing interpolation, or color reconstruction for the camera feed. Those cycles are freed for the tasks that drive actual application value, such as vehicle detection models, license plate recognition, gaze direction estimation, drowsiness classification, and occupancy mapping.
The architecture also eliminates a subtle but important failure mode. When RGB-IR separation is done on the host using software pipelines, the quality of that separation depends on the implementation, the available compute budget at runtime, and whether the pipeline can keep up with the camera’s frame rate under load.
STURDeCAM57’s onboard ISP performs this work in dedicated hardware at a fixed quality level, regardless of what else is running on the host.
Why STURDeCAM57’s Onboard ISP Is Critical for DMS
- Gaze estimation, head pose classification, eye openness detection, and facial landmark tracking all depend on color-accurate RGB frames. IR contamination silently degrades each task.
- IR interference is most acute at night, when contamination levels are highest and model confidence is most at risk of quiet, cumulative erosion.
- The ISP’s IR subtraction algorithm of e-con Systems’ STURDeCAM57 corrects contamination at the source, delivering a clean RGB stream and not a post-processed one competing for GPU headroom.
- Built-in ISP processes highlights and shadows simultaneously, managing extreme contrast ratios common in cabin lighting transitions and direct sunlight exposure.
- Blown highlights or crushed shadows arriving at the inference engine are unrecoverable. on-camera processing guarantees the full luminance range reaches the AI pipeline intact.
Why STURDeCAM57’s Onboard ISP Is Relevant for DMS
- Gaze estimation, eye openness detection, head pose classification, and facial landmark tracking are trained on specific pixel distributions. Contaminated frames silently erode detection confidence.
- IR stream quality is as critical as RGB quality for DMS. Facial IR imaging operates precisely when visible light fails.
- e-con Systems’ STURDeCAM57 streams IR as a fully independent channel, delivering a dedicated IR image of the driver’s face rather than a contaminated color frame requiring downstream interpretation.
- 940nm illumination is invisible to the driver, providing continuous, non-distracting facial imaging regardless of ambient light conditions.
- Global shutter eliminates rolling shutter distortion on fast head turns, blinks, and micro-expressions, a hard prerequisite for reliable landmark detection.
- ISP cleanliness is the final gate. A geometrically accurate but color-corrupted face frame is still a degraded input, regardless of downstream model capability.
Learn More About e-con Systems’ STURDeCAM57
e-con Systems has been designing, developing, and manufacturing embedded vision solutions, such as custom OEM cameras and end-to-end ODM platforms. STURDeCAM57 is our latest 5MP RGB-IR global shutter GMSL camera, targeted at Driver Monitoring Systems.
Use our Camera Selector to explore our full portfolio.
As always, to discuss your specific requirements or speak with a vision expert, write to camerasolutions@e-consystems.com.
FAQs
What processing does STURDeCAM57’s onboard ISP actually perform?
The onboard ISP handles RGB-IR separation, IR contamination removal from the RGB channels, demosaicing, and color reconstruction, all inside the camera. The NVIDIA AGX Orin receives two fully processed, independent streams – a clean RGB frame and a clean IR frame, both ready for AI inference without further host-side processing.
Why does it matter whether RGB-IR processing happens in the camera or on the host?
When processing is done on the host, it consumes GPU cycles that would otherwise be available for AI inference workloads. Benchmarks show that high-quality demosaicing of RAW sensor data alone can consume a significant portion of GPU load on the NVIDIA AGX Orin. Hence, e-con Systems’ STURDeCAM57 frees the host platform to focus entirely on inference, detection, and application logic.
What is IR contamination, and why does it affect DMS performance?
IR light leaks into the RGB channels of an RGB-IR sensor because the color filter does not perfectly block infrared wavelengths. Without correction, this shifts color values — red signals can appear yellow, and bright light sources produce halos. STURDeCAM57’s ISP removes IR contamination using pre-calibrated coefficients before the RGB stream is delivered to the host.
Does the onboard ISP affect how the IR stream is delivered to DMS algorithms?
Yes. The ISP separates the IR data from the mixed sensor output and delivers it as a fully independent channel. DMS algorithms receive a dedicated IR image of the driver’s face (not a color image with IR influence mixed in).
How does on-camera processing simplify NVIDIA AGX Orin integration?
It eliminates the need to build or maintain a custom RGB-IR separation and demosaicing pipeline on the host. Engineers connect the camera’s RGB and IR output streams directly to their inference pipeline without an intermediate processing stage.
Suresh Madhu is the product marketing manager with 16+ years of experience in embedded product design, technical architecture, SOM product design, camera solutions, and product development. He has played an integral part in helping many customers build their products by integrating the right vision technology into them.