How to Use the RGB-D Mapping Feature of e-con Systems’ ToF Camera – DepthVista

When deploying vision applications involving depth sensing, achieving seamless alignment between depth and RGB images is a critical step. e-con Systems’ DepthVista, as a two-board camera solution containing both RGB and ToF depth cameras, helps accomplish exactly this task.

In this blog, you’ll learn step-by-step how to map the depth frame onto the RGB frame with the help of DepthVista SDK.

About e-con Systems’ DepthVista

DepthVista integrates both RGB and ToF depth sensors on a unified solution. As a USB UVC-compliant 3D camera, it is equipped with a SuperSpeed USB 3.2 Gen 1 interface.

• RGB camera: Uses a high-performance 1/2.6″ AR0234CS CMOS sensor from onsemi™ with a global shutter. This sensor is paired with a dedicated color image signal processor for superior image output.
• ToF depth camera: Leverages a specialized 1/4″ CCD sensor alongside a dedicated depth processor for capturing accurate depth information.

Built for plug-and-play usability, DepthVista interfaces directly through a USB Type-C connector—with immediate readiness on Windows and Linux operating systems.

Understanding RGB-D Mapping

e-con Systems’ DepthVista camera streams RGB color data and depth information simultaneously. As the RGB and depth sensors reside on distinct modules, they capture scenes from slightly varying viewpoints. Aligning these perspectives to a unified frame of reference is necessary to interpret combined RGB-D data accurately. DepthVista accomplishes this using several carefully calibrated parameters:

• Camera matrix: Determines intrinsic camera parameters like focal length and optical center (principal point).
• Distortion coefficients: Rectify image distortion effects introduced by camera optics.
• Transformation matrix: Executes spatial alignment, repositioning the depth information into the RGB frame reference.
• Rotation matrix: Corrects angular misalignment between RGB and depth sensor perspectives.

RGB-D Mapping Through DepthVista SDK

DepthVista SDK simplifies the alignment task via its SetRGBDMapping API. Developers can seamlessly enable RGB-D mapping using this straightforward interface. Once activated, the SDK applies the pre-calibrated parameters.

It applies undistortion algorithms on depth images to counteract lens-induced distortion. It also leverages stored calibration details, including camera matrix, distortion coefficients, transformation, and rotation matrices. This helps align the depth frame with the RGB frame.

Enabling RGB-D mapping ensures both streams have a unified resolution and aligned fields of view. It leads to highly reliable and visually coherent datasets for vision tasks.

Using the 3D Point Cloud

Post alignment, e-con Systems’ DepthVista helps generate rich 3D point clouds. The SDK uses intrinsic and extrinsic camera matrices calibrated for HD and VGA resolutions:

• Intrinsic matrix: Defines optical properties such as focal length and the optical center of the RGB sensor.
• Extrinsic matrix: Establishes spatial correlation between RGB and depth sensor modules.

(To dive deeper into camera projection and parameter concepts, check out our Blog: Comprehensive guide to understand camera projection and parameters)

The SDK’s GetDeviceCalibrationParams API conveniently provides access to these parameters.

Once depth and RGB frames are aligned using SetRGBDMapping, the SDK maps the depth frame onto the RGB frame. Each RGB pixel color is accurately projected onto corresponding depth points by using intrinsic camera parameters, forming a detailed 3D model.

The images below illustrate the resulting Depth and RGB point clouds after successful alignment.

Using rgbdMapping.py, the images from the Depth and RGB cameras are transformed into a common camera coordinate system. This aligned data can then be projected into 3D space, generating an RGB-D point cloud for visualization and analysis.

 How to Install DepthVistaSDK in Linux

1. Run the following command to extract the package
   Unzip -X <packageName.zip><Extracted Directory>\linux\Bin\Ubuntu18.04\x64\SDK\DepthVistaSDKInstaller will have an install.sh file.
   Note: For Ubuntu 20.04, the install.sh file will be present in <Extracted Directory>\linux\Bin\Ubuntu20.04\x64\SDK\DepthVistaSDKInstaller.

2. Open the folder containing sh in the terminal.
3. Run the following command to give executable permission for sh file.
   chmod +x install.sh

4. Run the following command to install the DepthVistaSDK.
   sudo ./install.sh

Once installation is successful, the Installation DepthVistaSDK success message appears.

Fig 1: DepthVista SDK Installation Success Message

How to Install DepthVistaSDK in Windows

1. Extract the given package.<Extracted Directory>/Windows/Bin/SDK/Dll/x64 will contain the DepthVistaSDK.dll file.Note: Use x64 dll for x64 Python and x86 dll for X86 Python.

2. Copy the DepthVistaSDK.dll and opencv_world420.dll and place it in the Python Installation folder (root folder).

How to Use the RGBDMapping Python Script

Requirements

• Python version 6 or above
• OpenCV for Python version 2 or above
• Numpy package
• DepthVista SDK

Launching DepthVista Python Script in Linux

1. Open a terminal from the directory where the rgbdMapping.py script is located.

2. Run the following command to run the application.sudo python3 rgbdMapping.py

Launching DepthVista Python Script in Windows

1. Open the command prompt from the directory where the main.py script is located.

2. Run the following command to run the application.python rgbdMapping.py

Selecting the Camera Device

Initially, the command line displays the number of devices connected. You must select the camera device to explore its features.

Figure 2: Application Launch Screen

Once the device is selected, the stream will start with Depth_IR_RGB_HD streaming mode.

Figure 3: Selecting Options

Capturing Frames

Enter 1 in Pick a Relevant Option to capture images.

Figure 4: Capture Frames

The files will be saved with the name as shown below.

• RGB Frame – DepthVista_rgb_yyyy_mm_dd_hrs_min_sec.bmp
• Depth Raw Frame – DepthVista_Raw_yyyy_mm_dd_hrs_min_sec.raw
• Depth Color map – DepthVista_Depth_yyyy_mm_dd_hrs_min_sec.bmp
• 3D ply file – DepthVista_PLY_yyyy_mm_dd_hrs_min_sec.ply

Exiting Application

Enter 0 in Pick a Relevant Option to exit the application.

Workflow of DepthVista’s rgbdMapping.py Script

Let’s explore the workflow of rgbdMapping.py script, which can obtain RGB-D mapped stream using the DepthVista SDK and generate a 3D point cloud.

The below steps outline the workflow implemented in the rgbdMapping.py script:

1. Initialize the DepthVista SDK
   • Use the Initialize API to initialize the DepthVista SDK and prepare it for operation.

2. Open the Device
   • Determine the number of DepthVista devices connected to the system using the GetDeviceCount API.
   • Retrieve device information using the GetDeviceInfo or GetDeviceListInfo API.
   • Use the retrieved device information to open the desired device with the OpenDevice API.
   • The OpenDevice API returns a DeviceHandle, which serves as an input parameter for subsequent API calls.

3. Set a DataMode that has support for both Depth and RGB streaming
   • Configure the data mode using the SetDataMode In this sample python script, Depth_IR_RGB_HD mode is set. We can also use Depth_IR_RGB_VGA mode.

4. Enable RGB-D mapping
   • Enable RGB-D mapping using the SetRGBDMapping

5. Start a thread to Retrieve and Preview Frames
   • Use the GetNextFrame API to retrieve frames from the
   • If GetNextFrame returns success, use the GetFrames API to get the processed and separated frames. We have introduced and used a structure called Frames, which has all the separated frames, such as rgb, ir, raw_depth and depth_colormap.

6. 6. Obtain RGB-D Mapped Frames
      • After enabling RGB-D Mapping, the GetFrames API will return the mapped 2D depth frame and colormap frame. The resolution of the depth_colormap frame and raw_depth frame will match the resolution of the RGB frame. Since we have set Depth_IR_RGB_HD mode in this sample, the resolution of RGB frame, depth_colormap frame and raw_depth frame will be 1280×720. If Depth_IR_RGB_VGA mode is used, the resolution of the RGB frame, the depth_colormap frame and raw_depth frame will be 640×480.

7. The script includes an option to capture the depth_colormap, raw_depth, RGB frames, and the 3D point cloud.

8. Generating 3D point cloud
   • To generate a 3D point cloud, we need 2D raw_depth points, RGB frame and RGB camera’s intrinsic parameters. Get RGB intrinsics and other calibration parameters using GetDeviceCalibrationParams

The logic behind point cloud generation is discussed in detail below.

How to Generate the 3D Point Cloud

The below data is vital for point cloud generation:

• Depth frame which has mapped 2D depth pixels (2D frame with depth data which is already mapped with RGB field of view)
• Corresponding RGB frame
• Focal length and principle point of RGB This can be found in the intrinsic matrix of the RGB sensor.

The following calculation converts the 2D depth coordinates (u, v) into 3D coordinates (x, y, z):

• Z = Depth[u, v] / zoom_factor
  Where u is the 2D x coordinate, v is the 2D y coordinate and zoom_factor is a scaling constant. Here, we consider zoom_factor as 1. So,
  Z = Depth[u, v]
  Z is the depth value of a given pixel at the u,v position.

• X = (u – principle point x) * Z / focal length x
  Where the principle point x is taken from the intrinsic matrix position [0,2] and focal length x is taken from the Intrinsic matrix position [0, 0].

• Y = (v – principle point y) * Z / focal length y
  Where the principle point y is taken from the intrinsic matrix position [1,2] and focal length y is taken from the Intrinsic matrix position [1, 1]

These points are collected to form a point cloud and saved as .ply file.

Troubleshooting When running the application, you may encounter the following error: error while loading shared libraries: libdc1394.so.25: cannot open shared object file: No such file or directory Warning: libdc1394.so.25, needed by /usr/lib/libopencv_world.so.4.2.0, not found Cause: This error occurs when the required libdc1394 library (used for IEEE 1394 camera support and required by OpenCV) is missing from your system. Solution: Run the following command in terminal to install the missing library sudo apt-get install libdc1394-25 Once installed, re-run the application. The error should be resolved.

e-con Systems Offers State-of-the-Art ToF cameras

 Since 2003, e-con Systems has been designing, developing, and manufacturing off-the-shelf and customized OEM camera solutions. Our ToF cameras leverage NIR wavelengths (940nm/850nm), delivering consistent 3D imaging across indoor and outdoor environments.

Each camera includes integrated depth analytics, generating instant real-time 2D and 3D imaging data. Available interfaces include USB, MIPI, and GMSL2. We also offer driver support and SDKs compatibility for NVIDIA Jetson AGX ORIN/AGX Xavier and X86-based systems.

Use our Camera Selector to check out our complete portfolio.

If you need an expert to help select and integrate the perfect camera solution for your embedded vision system, please write to camerasolutions@e-consystems.com.