The RGB-IR format is a novel image format in the CMOS image sensor technology. The traditional CMOS image sensor usually employ a color filter array (CFA) called as Bayer format. This Bayer format CFA allows each pixel of the CMOS sensor to be excited by a specified wavelength of light. More information about the Bayer format is provided below.
In the RGB-IR format, the CFA has been modified to pass infra-red light only (no visible light) to one of every four-pixel group. In a de-facto Bayer CFA, there will be 50% of Green pixels, 25% of Red pixels and 25% of Blue pixels. In the RGB-IR format, there will be 25% of Green pixels, 25% of Red pixels, 25% of Blue pixels and 25% of IR (only IR) pixels. The pixel format is given below:
The following table summarizes the difference between the Bayer CFA and RGB-IR CFA formats for a quick reference.
Note: In Bayer format even though there are no pixels for Infrared, all the Red, Green and Blue pixels will be sensitive to respective color wavelengths and also for Infrared. Hence in a Bayer format sensors, an Infra-red cut-off filter is placed to avoid the direct influence of infrared over other colors.
In the RGB-IR format, all the pixels are sensitive to Infrared light, except that the IR pixels are sensitive only to Infrared light.
The following two images differentiate the Bayer CFA and the RGB-IR formats.
As it can be seen from the above, the ‘G’ pixel neighboring the ‘R’ pixel has been converted to Infrared pixel.
The RAW Bayer format is not the same as the standard RGB format. The RAW format refers to the raw output from the CMOS image sensor which is based on the Bayer Color Filter Array (CFA). Each pixel of the CMOS image sensor picks only one color (either Red or Green or Blue) of the incident light. The Bayer pattern is one type of color filter array, where 50% of the pixels respond to Green color, 25% of pixels respond to Red and remaining 25% to Blue color. The sensor has a micro-lens array that filters the respective color (Red or Green or Blue) for the respective pixels. Each pixel of the CMOS sensor array responds to one of these primary colors and the other two colors for the same pixel has to be interpolated.
The above picture shows the Bayer filter geometry. For more information please visit http://en.wikipedia.org/wiki/Bayer_filter
The RAW Bayer format contains one color information per pixel. That color information may be Red or Green or Blue, depending on the position of that pixel.
In the standard RGB format, each pixel contains all the three primary color information.
As explained in questions above, the Bayer RGB format contains only one color information per pixel. The other two color information has to be interpolated from the near pixels. There are various interpolation algorithms available and this process of obtaining RGB image from a Bayer pattern image is called as “demosaicing”. There are lots of demosaicing algorithms each claiming its own advantages on image quality, noise, performance, chromatic noise reduction etc.
No, the demosaicing is just one of the processes, but the most important process affecting the image quality. All the processing required creating a RGB image out of Bayer pattern image is collectively known as Image Signal Processing (ISP). The ISP pipeline has multiple processes and demosaicing is one of critical process. Other processes include Color Correction, Dead/Bad Pixel correction, Black Level compensation, Noise reduction, Gamma correction etc. And the ISP can also have Auto Exposure, Auto White Balance, Auto focus etc.
The standard method to get the YUV image from Bayer pattern sensor is to first reconstruct the RGB image and then convert the RGB image to YUV image.
As explained, all the pixels are sensitive to Infrared and the IR pixels are sensitive to IR pixels only. The illustration of spectral characteristics of the OV4682 image sensor is given below. This is not an accurate graph, but can be used for illustrative purposes only. For more accurate spectral characteristics waveform, please contact email@example.com.
OmniVision recommends the following operation to get the RGB information from the RGB-IR image. Subtract the IR pixel value from neighboring R,G and B pixel values and then do the demosaicing operation for the R,G and B pixels to get the remaining colors for these pixels. The IR pixel will then be extrapolated from the neighboring R,G and B pixels. This is one suggestion and customers can play with this format and optimize this.
There are lot of algorithms on demosaicing and extrapolation and customers can use the algorithm that fits the most for their application.
The spectral response curve of the OV4682 sensor is available from OmniVision under NDA. However, e-con has arrived at the OV4682 sensor spectral characteristics as illustrated below.
This is only for the illustration purposes only and is not accurate. However this spectral response curve illustrates the spectral response for R,G, B and IR pixels.
e-CAM40_CUMI4682_MOD Camera Module
No, the e-CAM40_CUMI4682_MOD supports only MIPI CSI-2 interface. In fact, the OV4682 sensor also does not have parallel data interface.
The e-CAM40_CUMI4682_MOD camera module does not have autofocus mechanism. The current version supports only the S-mount (M12) lens holder only. Please contact firstname.lastname@example.org. for support on autofocus for the same module.
The camera comes with the M12 lens holder (sometimes called S-mount). Customers can choose any M12 lens based on their application and use with our camera.
The default lens does not have any optical filter. The lens allows both the visible and IR light.
No, the default lens that we supply does not have IR cut-off filter. It passes all the visible and IR light.
To get both the visible and IR, we recommend using a Dual band-pass filer that allows the visible light and the 850nm IR light. The typical spectrum of such filter will be as below.
The above filter is recommended for use with visible+IR applications.
Yes, we can get full 4MP (2688x1520) IR images from the e-CAM40_CUMI4682 camera. All the 4MP pixels of the sensor are sensitive to Infrared and we can get 4MP IR images.
Yes, the camera can be used only with IR illumination to get the IR image. Or alternatively, you can also use IR-only-pass filter to stop the visible light to get the IR image.
Customers can evaluate the e-CAM40_CUMI4682_MOD with the USB 3.0 interface on PC using our See3CAM_CU40 camera or on the NVIDIA Jetson board using our e-CAM40_CUTK1 kit.
No, at this moment we don’t have any plans to support this camera module on Denebola CX3 RDK.
e-con provides the e-CAMView application with a basic Image Signal Processing filter to capture the RAW video from the See3CAM_CU40 camera and display both the RGB color video and IR video side-by-side. This application is only for the evaluation purposes and customers can develop their own applications using DirectShow or Windows Media Foundation framework provided by Microsoft for Windows platforms. For linux the camera is exposed as V4L2 device. e-con Systems provides QtCAM application compatible with V4L2 to capture images from the See3CAM_CU40.
e-con provides the test application that captures the RAW RGB-IR frames from the camera. This is based on V4L2 API and customers can develop their own applications and implement the image processing software on their own.
The ISP library performs all the image signal processing for the RGB and RGB-IR format inputs. At this time, e-con's ISP library has the support for the Bayer demosaicing and color correction and other modules are being developed. This library can do the Bayer demosaicing and also the RGB-IR demosaicing. This is a proprietary ISP library developed by e-con and the source code for this is not available along with the e-CAM40_CUTK1 camera board.
e-CAMView and DirectShow based Application Development
The e-CAMView application is a simple video viewer application built on DirectShow framework provided by Microsoft. This video viewer application is generic and can work with almost any UVC compliant video source such as integrated webcam etc. The e-CAMView application also supports all our USB UVC camera products.
The Microsoft DirectShow application programming interface (API) is a media-streaming architecture for Microsoft Windows. Using DirectShow, your applications can perform high-quality video and audio playback or capture.
The DirectShow headers, libraries, SDK tools, and samples are available in the Windows SDK.
QtCAM is e-con's open source Linux webcam software with advanced features. This Qt Linux camera software application provides easier user interface for capturing and viewing video from devices supported by Linux UVC driver. It would also work with any V4L2 compatible device.
For more information visit QtCAM Web page.
Yes. You can download the QtCAM Source Code in the following link https://github.com/econsysQtCAM/QtCAM
The building block of DirectShow is a software component called a filter. A filter is a software component that performs some operation on a multimedia stream. For example, DirectShow filters can
1. Read files
2. Get video from a video capture device
3. Decode various stream formats, such as MPEG-1 video
4. Pass data to the graphics or sound card
Filters receive input and produce output. For example, if a filter decodes MPEG-1 video, the input is the MPEG-encoded stream and the output is a series of uncompressed video frames.
When we refer Software ISP filter, this filter receives the RAW Bayer format video and converts the same to YUV422 format video.
In DirectShow, an application performs any task by connecting chains of filters together, so that the output from one filter becomes the input for another. And hence, the DirectShow application is just a place holder where the filters are connected together to perform a particular job(s) and then offers handle to control the operation of these filters.
As mentioned earlier, the e-CAMView employs a basic implementation of Sofware ISP for converting the RGB-IR output of e-CAM40_CUMI4682_MOD to YUV or RGB. This also employs a basic software extrapolation algorithm for the missing Green pixels. This Software ISP filter is a very primitive implementation and contains just the bare minimal modules of the ISP pipeline and this is not optimized for the high quality Image Signal Processing.
No, we don’t provide the source code for the Software ISP filter.
Yes, you can license the source code of the e-CAMView application by executing a Source Code licensing Agreement with e-con Systems and paying an one-time royalty fee. Please contact email@example.com. to get the copy of SLA and the quotation for one-time license fee. Please note that e-con does not provide the source code of the Software ISP filter.
The first step is to download the Windows SDK which also contains the DirectShow SDK. Once you download and install the Windows SDK, then you can go the samples located at [SDK Root]\Samples\Multimedia\DirectShow folder. There are a number of sample applications that are provided by Microsoft as part of the SDK, which are used as a starting point for custom applications.
There are two capture applications that may be of interest for you.
The first sample application is called PlayCap, which is a very basic capture application that displays the video from the first capture device it locates in your system. This sample is available at the following location of your SDK installation:
The secnd capture sample shall be AmCap. This sample application demonstrates the following tasks related to audio/video capture:
1. Capture to a file
2. Live preview
3. Allocation of the capture file
4. Display of device property pages
5. Device enumeration
6. Stream control
This sample is available at the following location of your SDK installation:
Both the samples are available with source code and you can use these samples to build your own application. For more information and help, please refer the SDK documentation or http://msdn.microsoft.com/en-us/library/windows/desktop/dd375454%28v=vs.85%29.aspx and http://msdn.microsoft.com/en-us/library/windows/desktop/dd375849%28v=vs.85%29.aspx
You can still use See3CAM_CU40 along with the associated firmware binary. But instead of using e-CAMView, you can write your own DirectShow application to grab the RAW RGB-IR format video directly from the sensor. You can implement a grabber filter (a type of DirectShow filter) and grab the RAW RGB-IR images and you can do any processing.
e-CAM40_CUMI4682_MOD Camera module customization
Yes, we can customize the PCB, sensor orientation, lens mount or even the PCB shape. These are minimal level changes. All these customizations attract NRE charges. Please contact firstname.lastname@example.org. along with your customization request to get the quote for the same.