Table of Contents
One frame procedure of 3D ToF camera
The process from ToF(Time of Flight) exposure to user reading out a frame on the host can be summarized as follows:
- Exposure
- ISP processing
- Transmition
- SDK processing
- Program readout
Exposure and ISP processing is implemented in the camera side, SDK processing and program readout is implemented in the host side. The specific process is shown in the figure below:
Timestamp and frame index
What is timestamp?
When the Vzense 3D Time of Flight(ToF) camera is powered on, the camera system starts a local timer starting from 0. The timer automatically increases regardless of the working state of the camera.
Note: By default, the system time of the camera is not the real time clock which calibrated with the host time. If the user requires to calibrate the local time with the host time, the NTP/PTP method must be configured between the camera and the host. Please refer to application notes “AN05-NTP function settings for Vzense 3D Time of Flight (ToF) camera” and “AN06-PTP function settings for Vzense 3D Time of Flight (ToF) camera“.
At the end of each frame exposure, the camera will allocate each frame a timestamp (t), in 1ms unit. This timestamp will be sent to the host side along with the data for the frame image and is available through the API.
Timestamps can be used as alignment and calibration between the camera and the host system, especially in trigger mode, so that the occurrence time of a certain frame can be known at the application side to avoid uncertain delays in the calculation and transmission process, so as to align with the system time.
Frame index
When an image is transferred to the host, the host SDK assigns a unique frame index to each frame, starting at 0 and increasing in sequence. Frame index is assigned by the host SDK so it doesn’t matter with the exposure, ISP processing and transmission stage. This means, even a frame is lost during exposure, ISP processing and transmission stage, the frame index is still in sequence.
After obtaining one frame, the SDK will temporarily store the frame in the buffer, preparing for the application program reading. If the application program does not read in time, the SDK could overwrites the old frame with the new one, resulting a frame loss. Through the frame index, the user can determine whether frame loss happen caused by host process scheduling or high CPU load.
As shown in the figure above, the SDK received frames 1,2,4 in a row, so we know that a frame 3 in the middle was not read correctly. From the principle of frame index generation, it can be known that if frame loss occurs, there must be a problem with the application program reading on the host side.
Obtaining timestamp and frame index
The timestamp and frame index of the Vzense camera image are in the ScFrame structure.
typedef struct
{
uint32_t frameIndex; //The index of the frame.
ScFrameType frameType; //The type of frame.
ScPixelFormat pixelFormat; //The pixel format used by a frame.
uint8_t* pFrameData; //A buffer containing the frame’s image data.
uint32_t dataLen; //The length of pFrame, in bytes.
uint16_t width; //The width of the frame, in pixels.
uint16_t height; //The height of the frame, in pixels.
uint64_t deviceTimestamp; /*The timestamp(in milliseconds) when the frame be generated
on the device. Frame processing and transfer time are not
included.*/
} ScFrame;The timestamp and frame index can be obtained through the scGetFrame interface. The sample code is as follows:
//depthFrame for example.
if (1 == FrameReady.depth)
{
status = scGetFrame(deviceHandle, SC_DEPTH_FRAME, &depthFrame);
if (depthFrame.pFrameData != NULL)
{
cout << "scGetFrame status:" << status << " "
<< "frameType:" << depthFrame.frameType << " "
<< "deviceTimestamp:" << depthFrame.deviceTimestamp << " "
<< "frameIndex:" << depthFrame.frameIndex << endl;
}
}Frame latency
Latency in one frame
According to the processing procedure of one frame, the delay of the camera is composed of the following parts:
| Part | Description | Factors | |
| Exposure | Camera | The acquisition of one frame image is completed by exposure | 1. Exposure time setting 2. HDR function (enable HDR, time consumption increases) 3. WDR function (enable WDR, time-consuming increase) |
| ISP processing | Camera | Depth map calculation, IR map calculation, RGB image conversion, RGB image coding, image filtering, distortion reduction and so on | 1. RGB image resolution (the higher the resolution, the longer the time consumption); 2. Depth image resolution (the higher the resolution, the longer the time consuming); 3. Filter switch (open filter, time-consuming increase) 4. HDR function (enable HDR, time consumption increases) 5. WDR function (enable WDR, time consumption increase) |
| Transmition | Network device between camera and host | Transfer data from the camera to the host computer | 1. Network device performance (such as network card, network cable, etc.) 2. Host network load |
| SDK processing | Host | Point cloud conversion, image alignment, RGB image decoding, etc | 1. Host performance 2. User application multithreaded design |
| Program readout | Host | Read the image from the SDK buffer | 1. Host performance 2. User application multithreaded design |
Test methods of frame latency
The following part will explain and exemplify how to perform one frame latency. In the below tests, two types of latency are tested: total single-frame latency, and latency excluding exposure time (end of exposure to application program reading).
Test steps:
1. Enable the camera’s NTP or PTP timing function.
Please refer to application notes “AN05-NTP function settings for Vzense 3D Time of Flight (ToF) camera” and “AN06-PTP function settings for Vzense 3D Time of Flight (ToF) camera“.
If you only need to test the total delay of a single frame, you can skip this step.
Note: Some product camera do not support NTP, PTP clock synchronization at the moment.
2. Modify the sample code to add timestamp printing
The routines(“DeviceSWTriggerMode”) are very close to our test methods and only require minor modifications. Since Windows and Linux have different time-getting functions, we present them separately here.
Sample code：
BaseSDK/Windows/Samples/Base/NYX650/DeviceSWTriggerMode.
1) Windows
Use the ftime function to get a converted millisecond UNIX timestamp.
Sample code is as follows:
#include <sys/timeb.h>
/*
omit some code
*/
//Starts capturing the image stream
status = scStartStream(deviceHandle);
if (status != ScStatus::SC_OK)
{
cout << "scStartStream failed status:" << status << endl;
return -1;
}
cout << "Software trigger test begins" << endl;
//delay for start stream ready.
this_thread::sleep_for(chrono::milliseconds(3000));
//1.software trigger.
//2.ReadNextFrame.
//3.GetFrame acoording to Ready flag and Frametype.
//4.sleep 1000/frameRate (ms)
for (int i = 0; i < frameSpace; i++)
{
timeb time_start, time_end;
//call the below api to trigger one frame, then the frame will be sent
// if do not call this function, the frame will not be sent and the below call will return timeout fail
status = scSoftwareTriggerOnce(deviceHandle);
ftime(&time_start); // record the start timestamp
if (status != ScStatus::SC_OK)
{
cout << "scSoftwareTriggerOnce failed status:" <<status<< endl;
continue;
}
//If no image is ready within 1000ms, the function will return ScRetGetFrameReadyTimeOut
status = scGetFrameReady(deviceHandle, 1200, &FrameReady);
if (status != ScStatus::SC_OK)
{
cout << "scGetFrameReady failed status:" << status << endl;
//this_thread::sleep_for(chrono::seconds(1));
continue;
}
//depthFrame for example.
if (1 == FrameReady.depth)
{
status = scGetFrame(deviceHandle, SC_DEPTH_FRAME, &depthFrame);
if (depthFrame.pFrameData != NULL)
{
cout << "get Frame successful,status:" << status << " "
<< "frameTpye:" << depthFrame.frameType << " "
<< "frameIndex:" << depthFrame.frameIndex << endl
ftime(&time_end);// record the end timestamp
uint64_t unix_end = time_end.time * 1000 + time_end.millitm;
uint64_t unix_start = time_start.time * 1000 + time_start.millitm;
uint64_t unix_exposure_end = depthFrame.deviceTimestamp;
cout << "one frame total time delay from trigger: " << unix_end - unix_start << endl;
if ((unix_end - unix_exposure_end) > 2000)
{
cout << "NTP/PTP is not work ！！！" << endl;
}
else
{
cout << "time delay from exposure end: " << unix_end - unix_exposure_end << endl;
}
}
}
this_thread::sleep_for(chrono::milliseconds(1000 / frameRate));
}2) Linux
Use the clock_gettime function to get a converted millisecond UNIX timestamp.
Sample code is as follows:
#include <time.h>
/*
省略部分代码
*/
//Starts capturing the image stream
status = scStartStream(deviceHandle);
if (status != ScStatus::SC_OK)
{
cout << "scStartStream failed status:" << status << endl;
return -1;
}
cout << "Software trigger test begins" << endl;
//delay for start stream ready.
this_thread::sleep_for(chrono::milliseconds(3000));
//1.software trigger.
//2.ReadNextFrame.
//3.GetFrame acoording to Ready flag and Frametype.
//4.sleep 1000/frameRate (ms)
for (int i = 0; i < frameSpace; i++)
{
timespec time_start, time_end;
//call the below api to trigger one frame, then the frame will be sent
// if do not call this function, the frame will not be sent and the below call will return timeout fail
status = scSoftwareTriggerOnce(deviceHandle);
clock_gettime(CLOCK_REALTIME, &time_start); // record the start timestamp
if (status != ScStatus::SC_OK)
{
cout << "scSoftwareTriggerOnce failed status:" <<status<< endl;
continue;
}
//If no image is ready within 1000ms, the function will return ScRetGetFrameReadyTimeOut
status = scGetFrameReady(deviceHandle, 1200, &FrameReady);
if (status != ScStatus::SC_OK)
{
cout << "scGetFrameReady failed status:" << status << endl;
//this_thread::sleep_for(chrono::seconds(1));
continue;
}
//depthFrame for example.
if (1 == FrameReady.depth)
{
status = scGetFrame(deviceHandle, SC_DEPTH_FRAME, &depthFrame);
if (depthFrame.pFrameData != NULL)
{
cout << "get Frame successful,status:" << status << " "
<< "frameTpye:" << depthFrame.frameType << " "
<< "frameIndex:" << depthFrame.frameIndex << endl
clock_gettime(CLOCK_REALTIME, &time_end);// record the end timestamp
uint64_t unix_end = time_end.tv_sec * 1000 + time_end.tv_nsec/1000000;
uint64_t unix_start = time_start.tv_sec * 1000 + time_start.tv_nsec/1000000;
uint64_t unix_exposure_end = depthFrame.deviceTimestamp;
cout << "one frame total time delay from trigger: " << unix_end - unix_start << endl;
if ((unix_end - unix_exposure_end) > 2000)
{
cout << "NTP/PTP is not work ！！！" << endl;
}
else
{
cout << "time delay from exposure end: " << unix_end - unix_exposure_end << endl;
}
}
}
this_thread::sleep_for(chrono::milliseconds(1000 / frameRate));
}The results of the implementation are as follows：
---DeviceSWTriggerMode---
Get device count: 1
serialNumber:GN6501CBCA3310172
ip:192.168.1.101
connectStatus:1
frameRate :15
Software trigger test begins
get Frame successful,status:0 frameTpye:0 frameIndex:1
one frame total time delay from trigger: 85
time delay from exposure end: 66
get Frame successful,status:0 frameTpye:0 frameIndex:2
one frame total time delay from trigger: 83
time delay from exposure end: 64
get Frame successful,status:0 frameTpye:0 frameIndex:3
one frame total time delay from trigger: 77
time delay from exposure end: 59
get Frame successful,status:0 frameTpye:0 frameIndex:4
one frame total time delay from trigger: 77
time delay from exposure end: 59
get Frame successful,status:0 frameTpye:0 frameIndex:5
one frame total time delay from trigger: 76
time delay from exposure end: 57
get Frame successful,status:0 frameTpye:0 frameIndex:6
one frame total time delay from trigger: 80
time delay from exposure end: 60
get Frame successful,status:0 frameTpye:0 frameIndex:7
one frame total time delay from trigger: 77
time delay from exposure end: 59
get Frame successful,status:0 frameTpye:0 frameIndex:8
one frame total time delay from trigger: 77
time delay from exposure end: 58
get Frame successful,status:0 frameTpye:0 frameIndex:9
one frame total time delay from trigger: 78
time delay from exposure end: 59
get Frame successful,status:0 frameTpye:0 frameIndex:10
one frame total time delay from trigger: 79
time delay from exposure end: 61
---end---The above method allows you to test the latency of the current system for several times.
Take the printout as above as an example to learn the two test results for this test.
a) The total latency of a single frame is about 80ms.
b) The latency excluding exposure time is about 60ms.
Vzense ToF camera latency test data
Take NYX650(B22) + ScepterSDK(v1.1.4) as an example, the ToF exposure time is 5ms, RGB resolution is 640×480, Depth2RGB alignment is on, and the delay test data of Depth and RGB image after getting the alignment is as follows:
| Platform | Native processing | Network transmission | SDK Processing | Total Latency |
| RK3588 | 64ms | 11ms | 20ms | 64+11+20 = 95ms |
| Nvidia Xavior NX | 64ms | 10ms | 10ms | 64+10+10 = 84ms |
The latency of DS series products will be a bit higher compared to NYX. Take DS86(B22) + ScepterSDK(v1.1.4) as an example, ToF exposure time XXXms, RGB resolution 640X480, open Depth2RGB alignment, get the aligned Depth and RGB image latency test data is shown in the following table:
| Platform | Native processing | Network transmission | SDK Processing | Total Latency |
| RK3588 | ||||
| Nvidia Xavior NX |
Latency optimization
If, after testing, it is found that the overall delay cannot meet the application requirements, it can be adjusted and optimized from the following perspectives:
1.Image resolution: The larger the image resolution, the larger the amount of data that needs to be processed, so the difference in resolution has the greatest impact on latency. It is recommended to choose a smaller resolution, such as 640×480, if the business allows.
2. System platform: the performance of the system platform directly affects the efficiency and performance of the SDK, so try to choose a platform that meets the requirements, as well as try to balance the resources of other processes on the camera SDK.
3. Point cloud conversion: point cloud conversion involves a large number of float operations, consumption is large, try to reduce the number of point cloud conversion can improve efficiency. For example, we only convert point cloud in ROI(region of interest) area, and don’t convert the whole map.
4. Filter switch: According to the business requirements, you can turn off some of the filter switches to reduce resource consumption and improve latency.
5. Camera model: Vzense camera has different models, different models of cameras because of different principles, different ways of realization, there will be delay differences.
