How Can Event Sensors Reveal Tiny Objects Under Extreme Lighting?

Detecting tiny moving objects when light is too bright or too dim is a challenge for traditional cameras. Imagine a surveillance camera under scorching sun that cannot spot a drone flying near a tree. To overcome this limit, researchers from Remote Sensing combined two sensor types: the classic RGB camera, which captures colors and details, and the new event sensor, which records only changes in brightness in real time. The result is an RGB‑Event fusion system that uses two key modules. The first, Temporal Multi‑Scale Attention Fusion (TMAF), analyzes event data at different temporal speeds, highlighting the rapid movements of small objects. The second, Sparse Noisy Gated Attention Fusion (SNGAF), works like a panel of experts: it sparsely selects which experts to use depending on the scene, avoiding noise and improving accuracy. To test the method, the authors created the RGBE‑UAV dataset, with drone videos under over‑ and under‑exposure conditions. Results show that the new approach outperforms existing solutions on both the new dataset and the public DSEC‑MOD set, demonstrating greater robustness in difficult lighting. This research paves the way for more reliable vision systems in security, autonomous driving, and aerial monitoring.

The event‑based sensor, also known as a Dynamic Vision Sensor (DVS), operates asynchronously: each pixel generates an "event" only when it detects an intensity change above a predefined threshold. This approach drastically reduces data flow compared to a traditional frame‑based camera, enabling microsecond‑scale latency and a higher exposure dynamic range. In the proposed model, RGB and event data are first spatially aligned via intrinsic and extrinsic calibration, then fed into two separate network branches. The RGB branch uses a CNN backbone (e.g., ResNet‑50) to extract high‑level features, while the event branch employs a spiking network or a 3‑D CNN to capture the temporal structure of activity spikes. The TMAF module applies multiple temporal windows (e.g., 1 ms, 5 ms, 20 ms) on each event stream, generating attention maps that weight regions with rapid variations more heavily. These maps are then fused with RGB activation maps via concatenation and 1×1 convolutions. The SNGAF, inspired by mixture‑of‑experts, contains N fusion experts (e.g., concatenation, weighted sum, cross‑attention). A gating network, implemented with a sparse softmax, selects the top‑K experts based on contrast and noise metrics, reducing computational load and limiting the influence of noisy events. Training uses a combined loss: cross‑entropy for object classification and an L0 regularization term on the gating to encourage sparsity. The RGBE‑UAV dataset was collected with an RGB camera (30 fps) and a DVS (1 M events/s) synchronized over urban and rural scenarios, including exposure variations from -3 to +3 EV. Evaluation metrics (mAP, IoU) show an average improvement of 7 % over RGB‑only methods and 4 % over previous linear fusion approaches, confirming the effectiveness of multi‑scale attention and sparse gating in handling extreme lighting conditions.