We can use multiple metrics to evaluate depth quality. It is recommended to use a universal scene compatible with all tests, such as a smooth, flat, matte, uniformly parallel-aligned plane for the depth camera, placed at a controlled and known distance during testing. A white or light-colored board mounted on a rigid frame or structure is a suitable target. The accuracy and precision of depth data (also known as temporal noise or repeatability) are the main characteristics for judging the image quality of a depth camera. For details, please refer to “TN01-Understanding accuracy vs. precision : What are their differences?“. This article introduces the depth quality testing of the NYX series time of flight cameras.
Table of Contents
NYX series products
The Vzense NYX series includes two products, the NYX650 and NYX660, which are based on Pulsed indirect Time of Flight(P-iToF) technology. These cameras feature a high SNR, strong light resistance, high dynamic range, prevention of range alising, and adaptive multi-camera coexistence. Its P-iToF technology paired with global shutter exposure lenses makes it easier to capture fast-moving objects’ trajectories. Its pulse iToF technology paired with global shutter exposure lenses makes it easier to capture fast-moving objects’ trajectories, providing stronger light resistance outdoors and better adaptability to outdoor scenes. The cameras are designed to industrial grade standards, ensuring high reliability and stability. The bidirectional mounting hole design facilitates structural integration and fixation of the buttom camera. The optional waterproof design with an aviation plug supports a high protection level of IP67.
Testing conditions
Target object: An 80cm x 80cm gray calibration board with 40% planar reflectance, mounted on a pole connected to the motion platform. As shown in the figure below:
Camera testing: Randomly select 3 calibrated NYX650 cameras. During calibration, use an 80cm x 80cm gray calibration board with 40% planar reflectance. The imaging performance of the NYX660 is identical to that of the NYX650, so the performance of the NYX650 can represent that of the NYX660.
Camera position: Mounted on a steel fixture, with the lens surface parallel to the calibration board surface. As shown in the figure below:
Camera settings: Default settings, as shown in the figure below, camera preheated for 20 minutes.
Imaging environment: Room temperature of 22°C, indoor lighting at 200lux during the test, with black curtains used on one side to minimize surrounding reflections. As shown in the figure below:
Testing platform: Linear guide motion platform, with a platform accuracy and precision of 1mm.
The motion platform moves from 0.5m to 4.5m, stopping at 0.5m, 1m, 1.5m, 2m, 2.5m, 3m, 3.5m, 4m, and 4.5m. At each step, measure the depth in the 10×10 pixel region of interest (ROI) at the center of the image, and repeat 32 times to obtain the average value.
Testing method: Use our self-developed tools to measure and record depth metrics in real-time. For accuracy depth value assessment, we use the average depth of each pixel in the image set. For precision depth value assessment, we calculate the standard deviation between each pixel in the image set.
Accuracy testing
In order to obtain the most accurate results possible, the 3D time of flight cameras are calibrated and tested in the factory to ensure reliable accuracy. The NYX series cameras are calibrated for a measurement range of 0.3 to 4.5 meters. Based on the test conditions mentioned above, the measurement errors for the three sets of data at different distances are shown in the figure below. According to the actual test results, the measurement errors are generally maintained at millimeter-level accuracy, with an accuracy ＜2%.
Accuracy trend at different distances, as shown in the figure below. The accuracy of the depth of the NYX series cameras decreases as the distance increases.
Precision testing
Precision measures the variation in depth values over a specific number of frames, also known as temporal noise or repeatability. The purpose of precision measuring is to understand the Z-axis precision of each pixel of the depth camera at the same exposure time. It is measured per pixel, then the average or median of the ROI is taken as the precision data of the depth value standard deviation. The measurement errors at different distances are generally maintained at millimeter-level precision, with precision＜ 1% based on actual test results.
Relationship between precision and distance
The depth quality test data trend with an exposure time of 5ms is shown in the figure below. The precision of the depth values of the NYX series cameras decreases as the distance increases.
Relationship between precision and exposure time
To help users understand the factors affecting precison, we tested the precision of different exposure time settings (2ms, 3ms, 4ms, and 5ms) at each distance. The data below indicates that the longer the exposure time set on the camera, the higher precision the camera can get, as long as images are not saturation.
Test results
The table below shows the depth quality test results for three sets of NYX650 cameras based on the above testing conditions.
Camera A test results are as follows:
| Test No. | True value (mm) | Measured value (mm) | Measured accuracy error value (mm) | Accuracy (%) | Measured precision error value (mm) | Precision (%) |
| 1 | 500 | 494 | -6 | -1.20% | 2.4 | 0.48% |
| 2 | 1000 | 992 | -8 | -0.80% | 2.8 | 0.28% |
| 3 | 1500 | 1492 | -8 | -0.53% | 3.7 | 0.24% |
| 4 | 2000 | 1991 | -9 | -0.45% | 5.8 | 0.29% |
| 5 | 2500 | 2490 | -10 | -0.40% | 6.4 | 0.256% |
| 6 | 3000 | 2996 | -4 | -0.13% | 11.5 | 0.38% |
| 7 | 3500 | 3510 | 10 | 0.29% | 18.7 | 0.534% |
| 8 | 4000 | 4011 | 11 | 0.28% | 22.4 | 0.56% |
| 9 | 4500 | 4515 | 15 | 0.33% | 28.1 | 0..62% |
Camera B test results are as follows:
| Test No. | True value (mm) | Measured value (mm) | Measured accuracy error value (mm) | Accuracy (%) | Measured precision error value (mm) | Precision (%) |
| 1 | 500 | 507 | 7 | 1.40% | 2.8 | 0.56% |
| 2 | 1000 | 1005 | 5 | 0.50% | 3.6 | 0.36% |
| 3 | 1500 | 1498 | -2 | -0.13% | 4.8 | 0.32% |
| 4 | 2000 | 1996 | -4 | -0.20% | 6.0 | 0.3% |
| 5 | 2500 | 2509 | 9 | 0.36% | 7.8 | 0.312% |
| 6 | 3000 | 3011 | 11 | 0.37% | 13.5 | 0.45% |
| 7 | 3500 | 3505 | 5 | 0.14% | 16.3 | 0.465% |
| 8 | 4000 | 4010 | 10 | 0.25% | 20.3 | 0.507% |
| 9 | 4500 | 4511 | 11 | 0.24% | 24.9 | 0.553% |
Camera C test results are as follows:
| Test No. | True value (mm) | Measured value (mm) | Measured accuracy error value (mm) | Accuracy (%) | Measured precision error value (mm) | Precision (%) |
| 1 | 500 | 497 | -3 | -0.60% | 2.6 | 0.52% |
| 2 | 1000 | 993 | -7 | -0.70% | 3.5 | 0.35% |
| 3 | 1500 | 1508 | 8 | 0.53% | 5.4 | 0.36% |
| 4 | 2000 | 2010 | 10 | 0.50% | 7.2 | 0.36% |
| 5 | 2500 | 2508 | 8 | 0.32% | 9.6 | 0.38% |
| 6 | 3000 | 2993 | -7 | -0.23% | 15.3 | 0.51% |
| 7 | 3500 | 3495 | -5 | -0.14% | 19.7 | 0.56% |
| 8 | 4000 | 3992 | -8 | -0.20% | 25.2 | 0.63% |
| 9 | 4500 | 4507 | 7 | 0.16% | 29.2 | 0.64% |
