Table of Contents
What is flying pixel?
Flying points are erroneous depth readings that may not always be evident in depth images but become apparent in 3D point clouds. Flying points appear as erroneous pixels in the 3D point cloud generated by Time-of-Flight(ToF) cameras, typically occurring at depth discontinuities in the scene, especially at object boundaries. The light from the foreground and background may merge at the edges, producing intermediate erroneous depth values, resulting in ghost pixels.
Factors contributing to the formation of flying pixel
Factor 1: aperture size
Large Lens aperture will allow a large number of photons to reach the sensor, increasing the brightness of the object, but object boundaries form more artifact and flying pixel. On the contrary, a narrow aperture will limit the amount of light reached by the sensor and will reduce the SNR and image quality. But it can also mitigate the effect of flying pixels by reducing the effect of infrared rays reflected at certain angles.
Factor 2: exposure time
In ideal conditions, each pixel of a ToF camera receives the light path from a single target within a given time frame. In actual scenarios, the relative motion between the camera and the scene causes each pixel to receive different light paths from the target, resulting in inaccurate depth at object boundaries and creating motion artifacts and flying pixels.
Factor 3: pixel size
When measuring an object edge, a single pixel receives reflected light from both foreground and background. This results in the superimposition of depth information from different distances, causing the original sensor data to contain varying depth values.
Factor 4: interference from other light sources
Other light emission at similar wavelengths can interfere with the ToF camera measurements, affect its ability to accurately perceive information, and appear flying pixels.
Different methods to eliminate flying pixels
Median Filter of software-based techniques
The Median Filter analyzes the depth values of neighboring pixels and excludes those that deviate significantly from the median value. This filter is based on a user-defined threshold, excluding pixels if the difference between the median value and the nearest neighbor exceeds the threshold. By mitigating outliers, the Median Filter effectively suppresses the influence of flying pixels on the final depth map. As shown in the diagram below:
Mask ToF of hardware-based techniques
Mask ToF augments the camera pipeline by inserting microlens mask between the sensor pixel and its microlens. This mask selectively blocks incident light paths, allowing each pixel to have its own custom aperture. By reducing stray light paths, Mask ToF effectively reduces flying pixels and improves depth sensing accuracy.
Example of flying pixel filter application on Vzense ScepterGUITool
Users can try the flying pixel filter function on ScepterGUITool, please see below figure:
Detect the object with time of flight camera, turn off the filter, and its point cloud map is shown below：
The flying pixel filter threshold ranges is [1, 16], turn on the flying pixel filter. The larger the threshold value set by the user, the more obvious the phenomenon of flying pixel elimination is.
The below point cloud shows the samples with flying pixel filter threshold value is 5:
The below point cloud shows the samples with flying pixel filter threshold value is 16:
Pros and cons of flying pixel filter
Pros: Eliminate flying pixels at object boundaries.
Cons: The larger threshold value of flying pixel filter may cause data loss on the depth map.
