How FPGA-Based Frame Grabbers Are Powering Next-Gen Multi-Camera Systems

Modern embedded vision systems depend on multi-camera aggregation for autonomous driving, intelligent surveillance, industrial automation, and humanoid robots. These systems need a higher frame rate and high-quality 4k video data for better understanding of the surroundings and making better decisions on the edge.

However, different applications need multiple fps and resolution needs and switch between resolutions for their own application demands. Image sensors also have different interfaces like MIPI, LVDS, and SLVS-EC, which need to be updated from time to time. Capturing, synchronizing all the cameras, and transmitting them in real time with minimum latency are the major challenges in the application.

This is where the FPGA-based PCIe aggregators shine.

In this blog post, you’ll get expert insights into the architecture, challenges, and design considerations of using modern FPGA platforms to build a PCIe Gen3 video aggregator for 8 GMSL cameras. First, let’s understand how an FPGA can be used as an aggregator.

Why Leverage FPGAs as Aggregators

Imagine you have 8 GMSL cameras, with each transmitting uncompressed or lightly compressed high-definition video data. Gigabit Multimedia Serial Link (GMSL) technology, developed by Analog Devices (formerly Maxim Integrated), enables transmission of high-speed video and control data over a single coaxial cable for long distances.

Read more: What Is GMSL Technology and How Does It Work?

While it solves the connectivity problem at the edge, the real challenge lies in how to funnel all the data into a central processing unit, such as a host CPU, GPU, or SoC.

Moreover, standard I/O interfaces like USB or Ethernet struggle with deterministic latency, scalability, or bandwidth when dealing with 8 simultaneous camera streams. On the other hand, PCIe Gen3, with its high throughput (up to 64 Gbps for x8 lanes) and low-latency direct memory access, is ideal for the task.

There are many solutions available in the market with PCIe aggregator. e-con’s FPGA based solutions stands out due to their major advantages on data handling on real time and other advantages as listed below.

By leveraging an FPGA as an aggregator, you can:

• Interface with multiple deserializer hubs handling GMSL inputs
• Time-multiplex and buffer the incoming data stream for handling multiple cameras
• Transfer the data via PCIe to the host at a faster line rate, with minimal software overhead

FPGAs offer the parallelism, flexibility, and deterministic performance needed for such a system, especially when dealing with real-time video analytics, edge AI pipelines, or safety-focused systems like Advanced Driver Assistance Systems (ADAS).

Overall System Architecture of a PCIe-based Sensor Aggregator 

The hardware for the system is a half-sized PCB, meant to be sleek to fit any host hardware partitions. The block diagram below shows the overall architecture:

The system houses the 2 Quad FAKRA connectors that can receive 4 x 4 lane cameras. The 2 deserializer receive the MIPI data from the GMSL cameras, separate the video stream into 4 MIPI streams for each deserializer, and produce 8 x 4 lane MIPI output. The FPGA receives the 8 camera streams and sends out each camera’s data in time domain multiplexing through PCIe to the host.

Video input block

The PCIe card receives MIPI video input from 8 cameras, each utilizing 4 data lanes, with MIPI lane speeds reaching up to 1.2 Gbps. It is fully compatible with e-con Systems’ GMSL camera lineup, including models such as the AR0821 (RGB), AR0341 (HDR), and Time-of-Flight (ToF) cameras. GMSL technology is particularly well-suited for long-distance, high-speed video transmission.

Explore: Our GMSL Cameras 

e-con Systems also offers support for other camera interfaces on a case-by-case basis, providing a flexible integration path for custom applications. In addition, we deliver comprehensive camera support, including driver development, camera and lens selection, and ISP tuning to meet specific image quality requirements.

Arbiter/scheduler

The arbiter’s primary responsibility is to determine the order and timing with which multiple incoming video streams are granted access to a shared output PCIe path, ensuring low latency and bandwidth efficiency. The arbiter block uses a weighted round robin-based arbitration policy.

The arbiter block architecture is register pipelined and optimized for handling the smoother data pipeline.

DMA engine

A DMA engine in the host is a hardware block that autonomously allocates the memory for video data and shares the memory address with the FPGA. An effective DMA engine ensures low-latency, high-throughput, and non-blocking transfer of video data to host memory without processor intervention. In a video aggregation system, it must handle high-throughput, low-latency, and multi-channel data movement reliably.

The PCIe driver on the FPGA (In the PCIe card) writes the video data straight to the video frame buffers on the Host, and the interrupt will be sent to the processor with the camera ID. This interrupt-based DMA drastically reduces the overall load of the CPU. e-con Systems provides PCIe driver support for AMD, NVIDIA, Intel and other platforms.

The PCIe driver on the PCIe card side handles the descriptor queue, data burst engine, interrupt controller, and DMA address counters.

System performance

The frame grabber board supports a wide range of frame rates and resolutions, offering a total system throughput of ~22 Gbps. The frame grabber can handle up to 4k @20 fps on 8 cameras. The table below highlights the validated resolutions currently available as turnkey solutions:

S.NO	MODE NAME	RESOLUTION	NO. OF CAMERAS	FPS	BANDWIDTH ACHIEVED
1	4k	3840×2160	8	20	21.3 Gbps
2	1080p	1920×1080	8	60	15.9 Gbps
3	720p	1280×720	8	60	7.07 Gbps

The MIPI-based systems create room for virtual channel-based aggregation. The architecture enables the FPGA to handle multiple video streams from different sensors or multiple data types (such as RGB and YUV422) from a single sensor concurrently over shared MIPI lanes. The use of VCs ensures efficient utilization of the available bandwidth and supports scalable multi-sensor input, enabling the system to manage high-throughput imaging up to the FPGA’s and host interface’s maximum capacity.

As a result, the system architecture is highly scalable. It can theoretically support an arbitrary number of camera streams, provided the aggregate data rate does not exceed the maximum bandwidth limit of 22 Gbps. It makes the solution suitable for advanced multi-camera applications such as stereo vision, surround view, or sensor fusion.

The overall system works on the line buffers on the FPGA. It reduces overall latency and introduces a latency of less than 100us from the frame grabber input to the output.

Interface support

The PCIe card supports PCIe Gen3 x4 lane interface as output of the standard PCI x4 edge finger connector. The system supports various hosts like NVIDIA, Intel, and AMD architectures. Eight cameras can be connected through two GMSL pair FAKRA connectors, as each supports 4 GMSL cameras.

Power and thermal considerations

The system works in the passive cooling system. Although the system works on a higher bandwidth of 22 Gbps, it does not need a cooling system. The PCIe card gets the power either from the host or the external power supply. The number of cameras decides the external power supply requirement.

e-con Systems’ FPGA-Based Camera Solutions

Since 2003, e-con Systems has been designing, developing, and manufacturing OEM cameras. Over the years, we have built a strong ecosystem of FPGA-based camera solutions, with experience across platforms from Lattice Semiconductor, Efinix, and Xilinx.

Our camera solutions integrate advanced FPGA platforms from these trusted partners to deliver high-performance imaging for embedded vision. e-con Systems also provides complete development support with custom design services, reference implementations, demo software packages, IP cores, and hardware kits to speed up your project timeline.

Use our Camera Selector to explore e-con Systems’ full portfolio.

For integrating custom cameras into your embedded vision systems, please reach out to us at camerasolutions@e-consystems.com.

FAQs

1. What is the role of an FPGA in a PCIe-based frame grabber?
   An FPGA acts as the core of the frame grabber by aggregating multiple video streams from high-resolution cameras and organizing the data flow so it can be transferred efficiently to the host system. In practical terms, it connects to de-serializers that convert incoming GMSL signals into MIPI streams, buffers these data channels, and then time-multiplexes them.Once the streams are aligned, the FPGA pushes them to the host processor through PCIe with deterministic latency.

2. Why is PCIe preferred over interfaces like USB or Ethernet for multi-camera aggregation?
   While USB and Ethernet are common for camera connectivity, they fall short when eight high-bandwidth camera streams must be handled in parallel. USB has limited scalability and struggles with deterministic latency, while Ethernet adds overhead that can cause jitter and delays in real-time applications.PCIe Gen3, on the other hand, provides direct memory access to the host system with bandwidths up to 64 Gbps on x8 lanes. This direct path eliminates processor bottlenecks and allows seamless data transfers at line rate.

3. How many cameras can this frame grabber support?
   The FPGA-based PCIe frame grabber described here supports up to 8 GMSL cameras operating simultaneously. These cameras can run at validated resolutions such as 4K at 20 fps, 1080p at 60 fps, or 720p at 60 fps, with aggregate bandwidth reaching up to ~22 Gbps.The architecture is scalable because the FPGA works with virtual channels, meaning it can theoretically manage more data types or additional sensors as long as the overall bandwidth ceiling is not crossed.

4. What applications benefit most from FPGA-based video aggregation?
   This is best suited for applications where multiple high-resolution cameras must work together in real time. For instance, in automotive use cases, such as ADAS and autonomous driving, the ability to merge video from 8 cameras helps enable features like lane detection and blind spot monitoring. In industrial automation, FPGA-based aggregation supports robotic arms that need synchronized multi-camera vision for safety and accuracy.

5. Does e-con Systems provide integration support for custom applications?
   Yes, e-con Systems not only offers FPGA-based frame grabbers but also complete support for integration into embedded vision systems. This includes custom camera design services, reference implementations, IP cores, and ISP tuning for application-specific image quality.