AN03-The using and setting of RGB sensor in Vzense 3D Time of Flight (ToF) camera

Time of flight has become one of the most popular depth sensing technologies in the market today. 3D ToF cameras are particularly widely used in the field of autonomous mobile robots (AMR) and autonomous guided vehicles (AGV). By transmitting laser light to the target and receiving the time of reflected light, the time of flight camera can accurately calculate the distance between the target and the camera, and at the same time realize obstacle detection, and then realize autonomous navigation. Although ToF cameras only provide depth data of the scene, they do not have the ability to capture color information, which becomes a drawback in some applications that require object recognition. The RGB camera can provide color images, and together with the depth images provided by the ToF camera, it can play an important role in object recognition and visual analysis.

Examples of RGB image application scenarios

Whether it is necessary to have both depth and RGB data in a frame depends on the specific application requirements.

Scenario 1: Specific objects need to be identified

In the case of a smart forklift, if the purpose is only to detect obstacles and not to identify objects, then the ordinary depth-sensing camera, no need to RGB image, can meet the requirements, but if the intelligent forklift is needed to identify goods or pallets, the RGB color image with AI technology will undoubtedly have a more accurate recognition effect, so it is necessary to use RGB to identify the pallet, and then locate the position of the pallet on the ToF depth map through coordinate system conversion. Here are some other vision applications that require combining depth and RGB data in a single frame.
1. AMR (Autonomous mobile robots): such as cleaning robots, delivery robots, service robots, companion robots, etc;
2. AGV (Autonomous Guided Vehicle);
3. Robotic arm;
4. Pedestrians counting system.

Scene 2: Recognize and classify objects in comlicated scenes

For some scenes, when using RGB color images for object recognition or classification, however the RGB image is always very complicated, which requires a lot of computing resource to separate the target object from the background. On the premise that the position and distance of the target object are roughly determined, the depth image output by ToF can be used to first segment the foreground and background by the distance, and then segment the RGB image containing the target object, and finally use the segmented RGB image for AI training and recognition, which can greatly reduce the computational power required for training and recognition.
For example: The original test environment RGB image is shown in the following figure. The white target board in the middle is the target object, but the color of the target object and the background are very similar at this time, so it requires more manpower and computing power to label and recognize.

According to the following depth map, the target object is close to the camera and can be captured by the ToF camera, but the background is far away from the 3D time of flight camera and cannot be captured.

The following figure is the segmented RGB image, which the far away background area on the depth map is segmented on the RGB image, only the close target object is retained, and then the segmented RGB image is used for semantic recognition, which can greatly reduce the computing power consumption of the processor.

Parameter setting of RGB module of Vzense ToF camera

RGB image resolution

The RGB resolution of NYX650/660, DS86/87 can be set to 640×480, 800×600, or 1600×1200. The higher the resolution, the clearer the picture quality of the camera.

Sample code：
BaseSDK/Windows/Samples/Base/NYX650/ColorResolutionChange.
Software API for RGB resolution adjustmenet is as below:

//set the RGB sensor resolution
ScStatus scSetColorResolution(ScDeviceHandle device, int32_t w, int32_t h);

//get the current RGB sensor resolution setting
ScStatus scGetColorResolution(ScDeviceHandle device, int32_t* pW, int32_t* pH);

//get the RGB sensor supported resolution list
ScStatus scGetSupportedResolutionList(ScDeviceHandle device, ScSensorType type, ScResolutionList* pList);

/**
 * @brief    Specifies the type of sensor.
 */
typedef enum
{
    SC_TOF_SENSOR   = 0x01,   //ToF camera.
    SC_COLOR_SENSOR = 0x02,   //RGB camera.
} ScSensorType;

Exposure mode

Exposure time of RGB camera is an important parameter, different exposure time can affect the brightness of RGB image, the longer the exposure time, the higher the brightness of the image.
The RGB camera of all products can be configured to AUTO Exposure (AUTO) or Manual exposure (MANUAL).
Sample code:
BaseSDK/Windows/Samples/Base/NYX650/ColorExposureTimeSetGet.
The exposure mode (auto or manual) can be set or obtained by below API:

//set the exposure control mode, AEC or manual, for RGB sensor, input the sensorType is SC_COLOR_SENSOR,
ScStatus scSetExposureControlMode(ScDeviceHandle device, ScSensorType sensorType, ScExposureControlMode controlMode);

//get the current exposure control mode, AEC or manual, for RGB sensor, input the sensorType is SC_COLOR_SENSOR
ScStatus scGetExposureControlMode(ScDeviceHandle device, ScSensorType sensorType, ScExposureControlMode* pControlMode);

/**
 * @brief    Specifies the type of sensor.
 */
typedef enum
{
    SC_TOF_SENSOR   = 0x01,   //ToF camera.
    SC_COLOR_SENSOR = 0x02,   //Color camera.
} ScSensorType;

/**
 * @brief    Specifies the type of exposure mode.
 */
typedef enum
{
    SC_EXPOSURE_CONTROL_MODE_AUTO   = 0,   //Enter the auto exposure mode.
    SC_EXPOSURE_CONTROL_MODE_MANUAL = 1,   //Enter the manual exposure mode.
} ScExposureControlMode;

At each mode, the maximum exposure time relates to the frame rate conversely, that means the higher frame rate is, the shorter maximum exposure time will be.
User can get the supported maximum exposure time by below software API：

//for RGB sensor, input the sensorType is SC_COLOR_SENSOR, it can get the maximum exposure time of RGB sensor at the current frame rate
ScStatus scGetMaxExposureTime(ScDeviceHandle device, ScSensorType sensorType, int32_t* pMaxExposureTime);

Auto exposure control mode

Under Auto Exposure Control(AEC), the camera detects ambient light in real time and then adjusts its exposure time to automatically adapt to changes in ambient light. Note that AEC is a feedback system, and when the light changes drastically, it often takes several consecutive frames to adjust to the appropriate exposure time.

Vzense’s 3D ToF product can set the maximum exposure time of the RGB camera in automatic exposure mode. It does not represent the actual exposure value, but the maximum amount of automatic exposure that can be obtained. The actual exposure time is still controlled by the automatic exposure algorithm of the RGB camera.

The maximum AEC exposure time of RGB sensor can be set or obtained by below API:

//set the max exposure time at AEC mode, for RGB sensor, input the sensorType is SC_COLOR_SENSOR
ScStatus scSetColorAECMaxExposureTime(ScDeviceHandle device, int32_t exposureTime);

//get the current max exposure time at AEC mode, for RGB sensor, input the sensorType is SC_COLOR_SENSOR
ScStatus scGetColorAECMaxExposureTime(ScDeviceHandle device, int32_t* pExposureTime);

Manual exposure mode

In the manual exposure mode, the camera is set to a fixed exposure time that does not change automatically. The exposure time setting value is the actual exposure time of the RGB camera. The effect of setting different exposure times is shown below:

In manual exposure mode, the image brightness can also be adjusted by gain setting, as shown below:

The real-time exposure time can be set and obtained by below API:

//set the exposure time of manual mode, for RGB, input sensorType is SC_COLOR_SENSOR
ScStatus scSetExposureTime(ScDeviceHandle device, ScSensorType sensorType, int32_t exposureTime);

//get the current exposure time of manual mode, for RGB, input sensorType is SC_COLOR_SENSOR
ScStatus scGetExposureTime(ScDeviceHandle device, ScSensorType sensorType, int32_t* pExposureTime);

/**
 * @brief    Specifies the type of sensor.
 */
typedef enum
{
    SC_TOF_SENSOR   = 0x01,   //ToF camera.
    SC_COLOR_SENSOR = 0x02,   //Color camera.
} ScSensorType;

The real-time color gain can be set and obtained by below API:

//set the gain of color image
ScStatus scSetColorGain(ScDeviceHandle device, float params);

//get the current gain setting of color image
ScStatus scGetColorGain(ScDeviceHandle device, float* pParams);