On-Board Image Processing With Fpga Acceleration Using Deep Neural Network Inference

The acquisition and processing of earth observation data from satellites is in transition. New Earth observation constellations of small satellites generate large amounts of data that cannot easily be transmitted to the ground. In the near future, the amount of data is expected to significantly grow, as sensor resolution and realizable constellation sizes are increasing much faster than the downlink capacity. The processing power of small satellites which rely on commercial off-the-shelf (COTS) hardware is nowadays capable of processing large amounts of data efficiently on board, despite the limited resources available. Consequently, this paper proposes to exploit the power of current advances in deep-learning-based image processing to extract relevant data from the acquired images, which is then transferred to the ground. This contribution attempts to demonstrate the feasibility of using state-of-the-art machine-learning image analysis, directly aboard a small satellite. Thus, on-board pre-processing can make better use of the available communication capacity by discarding unimportant data and reducing the latency between acquisition and reception of information. This results in a reduction of operating costs and increases satellite autonomy. To demonstrate this concept, a deep convolutional neural network has been trained and deployed to a Zynq UltraScale+ MPSoC-based data processing unit (DPU) that is being developed at the Fraunhofer Ernst-Mach-Institut (EMI) for the use in nanosatellite imaging payloads. This DPU contains a Field Programmable Gate Array (FPGA) that allows the implementation of a hardware accelerator, enabling an efficient and fast inference of neural networks. The convolutional neural network employed is a semantic segmentation model trained on satellite imagery. This model uses a UNET-type architecture to segment satellite images into 12 surface classes (e.g., water, vegetation, clouds). It is trained utilizing publicly available ground truth data from the Sentinel-2 L2A mission. To evaluate the model's performance, it is afterwards tested with data from different satellites with comparable spectra and ground resolution. The proposed approach demonstrates that the employed DPU can apply a complex neural network on the typical data volume of a small earth observation satellite in real-time. Furthermore, it demonstrates that this performance can be achieved despite the limited power and cooling resources available on board a small satellite. The use of deep neural network for onboard pre-processing enables the imminent extraction of information from earth observation data and can help the satellite operator to react to short-term events and to determine which data is worth transmitting to the ground.