In the last decade, the topic of Deep Learning (DL) has been one of the most popular topics of research in mathematics and computer engineering and this recent interest in DL does not come from nowhere. Indeed, DL has revolutionized a great number of Artificial Intelligence (AI) applications (e.g. speech recognition, visual object recognition, object detection, ...). DL models are composed of multiple processing layers able to learn complex non-linear representations of data. One of the main requirements for success in DL is the availability of a large enough dataset to correctly train the DL model. In this aspect, the field of meteorology satisfies the data requirement. Indeed, most meteorological institutes like the Royal Meteorological Institute of Belgium (RMIB) have accumulated decades of weather and climate data from various sources.

In particular, DL could improve the accuracy of precipitation estimation. Accurate precipitation estimates are a very important products of weather institutes for hydrological applications, and as input and verification datasets for Numerical Weather Prediction models and precipitation nowcasting systems. Currently, precipitation intensities and amounts are estimated using both radar and rain gauge data, but the coverage of these measurements can be limited in some areas, such as over the sea. A possible solution to this problem would be to use observations from satellite radiometers. Unfortunately, estimating accurately precipitation from satellite radiometer imagery remains a challenge.

As a solution, we developped a DL method to merge rain gauge measurements with a ground-based radar composite (OPERA) and satellite radiometer imagery (SEVIRI). The proposed convolutional neural network, composed of an encoder–decoder architecture, performs a multiscale analysis of the three input modalities to estimate simultaneously the rainfall probability and the precipitation rate. The training of our model and its performance evaluation were carried out on a dataset spanning 5 years from 2015 to 2019 and covering Belgium, the Netherlands, Germany and the North Sea.

Our results for instantaneous precipitation detection, instantaneous precipitation rate estimation, and for daily rainfall accumulation estimation show that the best accuracy is obtained for the model combining all three modalities. The ablation study, done to compare every possible combination of the three modalities, shows that the combination of rain gauges measurements with radar data allows for a considerable increase in the accuracy of the precipitation estimation, and the addition of satellite imagery provides precipitation estimates where rain gauge and radar coverage are lacking. Our results also show that our multi-modal model significantly improves performance compared to the European radar composite product provided by OPERA.