Learning from Rendering: Realistic and Controllable Extreme Rainy Image Synthesis for Autonomous Driving Simulation
Kaibin Zhou, Kaifeng Huang, Hao Deng, Zelin Tao, Ziniu Liu, Lin Zhang, Shengjie Zhao
The School of Computer Science and Technology, Tongji University, Shanghai, China
Arxiv Preprint: https://arxiv.org/abs/2502.16421
Codes: Learning-from-Rendering Rainy Image Synthesizer, CARLARain
HRI Datasets: Hugging Face , Google Drive , Baidu Cloud(Extraction Code: v9b2)
CARLARain Datasets: Baidu Cloud(Extraction Code: i4zi)
Introduction
This is the website for our paper "Learning from Rendering: Realistic and Controllable Extreme Rainy Image Synthesis for Autonomous Driving Simulation".
 |
 |
Learning from Rendering Rainy Image Synthesizer
To incorporate both controllability and realism into rainy image synthesis, we propose a Learning-from-Rendering rainy image synthesizer, which combines the benefits of realism and controllability. In the rendering stage, we propose a 3D rainy scene rendering pipeline to render realistic high-resolution paired rainy-clean images. In the learning stage, we train a High-resolution Rainy Image Generation Network (HRIGNet) to controllably generate extreme rainy images conditioned on clean images.
CARLARain: Extreme Rainy Street Scene Simulator
To construct a continuous set of well-labeled extreme rainy images, we integrated our proposed synthesizer with the CARLA driving simulator and the CRIGNet, and develop CARLARain. CARLARain is an extreme rainy street scene simulator which can obtain paired extreme rainy-clean images and labels under complex illumination conditions.
High-resolution Rainy Image (HRI) Dataset
In the rendering stage, we create a High-resolution Rainy Image (HRI) dataset in the rendering stage of the proposed rainy image synthesizer. The HRI dataset comprises a total of 3,200 image pairs. Each image pair comprises a clean background image, a depth image, a rain layer mask image, and a rainy image. It contains three scenes: lane, citystreet and japanesestreet, with image resolutions of 2048×1024.
Lane Scene:
Citystreet Scene:
Japanesestreet Scene:
ExtremeRain Dataset
Based on CARLARain, we construct an extreme rainy street scene image dataset, ExtremeRain. This dataset contains 8 different street scenes and 3 illumination conditions: daytime, sunset, night. The rainy scenes feature a rain intensity ranging from 5 mm/h - 100 mm/h, covering extreme rainfalls under complex illumination conditions. The dataset contains comprehensive label information to meet the requirements of multi-task visual perception models, including semantic segmentation, instance segmentation, depth estimation, and object detection.
| Background | Rainy | Depth | Semantic Segmentation | Instance Segmentation |
|---|---|---|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Experiment - Realism And Controllability Of Rainy Image Generation
- Compare with baseline: To evaluate the performance of HRIGNet in high-resolution rainy image generation, we compare it with several baseline image generative models: LDM, DiT and CycleGAN. The figure below illustrates a comparison of rainy image generation results of these methods.
 |
 |
 |
| Background | LDM | DiT |
 |
 |
 |
| Ground truth | CycleGAN | HRIGNet(ours) |
- Controlibility: As shown in the figure below, some rainy images from ExtremeRain are presented. It is possible to control different background scenes, achieve variations in illumination such as daytime, sunset, and night, and control attributes like rain intensity and direction. The controllability of multiple attributes ensures the diversity of the dataset.
Experiment - Semantic Segmentation In Extreme Rainfall
To improve the accuracy of semantic segmentation models in extreme rainy scenes, we conduct augmented training with the ExtremeRain dataset and evaluate several SOTA semantic segmentation models on real datasets. We collect real rainy scene images with different illumination conditions from the Internet, and use them as the test set.
Source Codes and Data
Get the codes: Learning-from-Rendering, CARLARain
Get the HRI dataset and the Blender scene files: Hugging Face , Google Drive , Baidu Cloud(Extraction Code: v9b2)
Get the ExtremeRain Dataset: Baidu Cloud(Extraction Code: i4zi)
Other Resources
Last update: July. 30, 2025