Speaker:

Flying pixels are false depth readings produced by a Time-of-Flight camera. They appear as stray points in a 3D point cloud, and they don't correspond to any real surface in the scene.

They mostly show up around object edges or depth discontinuities, where a pixel can receive light from multiple distances during the measurement cycle. When the IR reflections from the foreground and background combine, the camera reports a depth value that sits between them.

These values may blend into a 2D depth map, but in a 3D point cloud they stand out as suspended points. They're often referred to as ghost pixels because they have no physical source.

Host:

What leads to flying pixels? Walk us through the key factors.

Speaker:

Several factors inside the optical and sensing pipeline contribute to flying pixels.

Aperture size
A large aperture lets in more light, which improves brightness in low-light scenes. But it also reduces depth-of-field. At edges, this increases the chance that a pixel captures reflections from two surfaces. That overlap introduces mixed phase information, which results in a false depth value.
A smaller aperture reduces the incoming light but limits the spread of angles hitting the sensor, which lowers the chance of mixed reflections.

Exposure or integration time
ToF pixels gather light over a defined integration period. During that window, the camera and the scene must ideally remain still. Any relative movement can cause the pixel to sample reflections from different positions or targets. The camera blends those readings, producing an incorrect depth at boundaries.

Interference from other IR sources
External IR like sunlight, other ToF systems operating at similar wavelengths, or any IR-emitting device can disrupt the signal phase or amplitude. This disturbance creates unstable readings, especially at discontinuities where the signal is already complex.

Pixel size
Each pixel has physical dimensions. When an edge falls within a pixel's footprint, the pixel receives reflected light from both the foreground and background. The camera blends these distances, generating an intermediate value that has no real-world counterpart.
All these factors intersect at edges, which is why flying pixels cluster around boundaries and object outlines.

Host:

When these stray points show up, how do they affect embedded vision systems in practice?

Speaker:

Even a small number of flying pixels can disrupt systems that depend on accurate depth perception.

Take Autonomous Mobile Robots, for example. These robots rely on depth data for obstacle detection, path planning, and environment understanding. A flying pixel may look like an obstacle. The robot may slow down, reroute, or treat an area as blocked, even though the path is free.

These false signals distort the robot's interpretation of the scene. The result can be unnecessary detours or unpredictable decisions.

Filtering may remove a bit of valid data around edges, but doing so protects decision-making. Removing flying pixels improves reliability because the remaining depth points represent real surfaces.

Host:

Let's move into mitigation. What software-based techniques help reduce flying pixels?

Speaker:

Software filtering is the first and most flexible way to reduce flying pixels. Two widely used approaches stand out.

DepthVista helps RPM systems in several ways:

Depth discontinuity filter
This filter checks how different a pixel's depth is compared to its neighbors. If a pixel varies sharply from the surrounding region, it's flagged as unreliable and removed.
This method works well because flying pixels often sit alone or in small clusters at boundaries. Removing them results in cleaner edges and a more stable depth map.

Median filter
A median filter evaluates a block of neighboring pixels, finds the median depth, and measures how far the current pixel deviates from that median. If the difference exceeds a threshold, the pixel is excluded.

This helps suppress outliers that appear as isolated spikes.

Some of the benefits of software filtering are:

• Fast processing
• Strong suppression of unwanted noise
• Cleaner depth with fewer irrelevant values

However, there are challenges too. These include:
• Performance depends on host CPU
• Some complex edges may still produce residual flying pixels

Even with these limitations, software filtering significantly improves depth consistency, especially in scenes with mixed edges.

Host:

Besides software, what hardware-level approaches help reduce this artifact?

Speaker:

Some teams tackle flying pixels by adjusting the optical pipeline directly. One such approach is Mask ToF.

This method inserts a micro lens-level mask between the pixel and its micro lens. The mask selectively blocks certain incident light paths. That means each pixel receives a more controlled subset of the scene instead of a wide range of angles.

Because each pixel gets a custom aperture pattern, stray reflections are reduced. This lowers edge mixing and improves depth accuracy.

The advantages of Mask ToF are:

• Higher depth accuracy
• Reduced boundary artifacts
• Cleaner point clouds

The challenges of Mask ToF are that it requires hardware modification and can be less flexible for rapid adjustments. At times, software filters could outperform it in some conditions.

Ultimately, Mask ToF is powerful when hardware changes are possible, but it's selectively used due to its design impact.

Host:

Are there situations where flying pixels naturally stay minimal?

Speaker:

Yes. Flying pixels mainly arise near depth discontinuities. Given this, scenes with uniform surfaces, such as plain walls, floors, large flat objects, generate very few flying pixels. There are no competing reflections, so the camera returns stable, consistent depth. The edges, corners, and transitions are the ones that need post-processing support.

Host:

Bringing everything together, how should developers approach flying pixel reduction when deploying ToF cameras in real systems?

Speaker

The best approach depends on the system's constraints.

If speed and flexibility matter, software filtering is the strongest path. It's fast, it doesn't alter the optical stack, and it can be tuned for different environments.

If maximum boundary accuracy is required and hardware changes are possible, Mask ToF or similar optical adjustments can bring deeper suppression.

In many deployments, developers combine both. Software filters clean the depth map quickly, while hardware-level improvements reduce the number of artifacts that reach the processing stage.

The important thing is understanding that flying pixels stem from mixed returns. Any strategy that narrows the light path or removes inconsistent depth values helps produce reliable 3D data.

Host

Excellent insights. Before we wrap up, any final thoughts for teams working with ToF?

Speaker

Flying pixels are a predictable artifact in Time-of-Flight imaging. They appear for physical reasons that come from aperture behavior, integration time, interference, and pixel geometry. Once you understand why they occur, they're manageable.
Removing them improves scene interpretation, strengthens navigation logic, and prevents systems from reacting to phantom obstacles. Whether the solution is software, hardware, or a mix of both, the goal remains the same: depth data that reflects the real world as accurately as possible.

Host

That brings us to the end of today's episode of Vision Vitals. Flying pixels may seem like a minor artifact when you first spot them, but as we've discussed, they influence everything from obstacle detection to how a system interprets depth boundaries in real time.

If you'd like help evaluating ToF behavior in your own deployment or want support in improving depth quality for embedded vision, our team is always ready to guide you.

Reach us anytime at www.e-consystems.com.

Thanks again for spending time with us on Vision Vitals.

We appreciate every listener who joins these deep dives, and we look forward to bringing you more insights in the next episode.