How do self‑driving cars see the road without flickering?

Have you ever wondered how self‑driving cars can understand the road without getting confused by jumping frames? Researchers have devised a system that merges spatial‑temporal Vision Transformers with a federated learning method resistant to attacks. The core is the ST‑HMR module, which stores tiny “prompt” tokens – instructions for the network – in an asymmetrical temporal cross‑attention buffer that looks forward and backward in time, keeping successive frames coherent and flicker‑free. At the same time, the BR‑FL pipeline gathers updates from a few vehicles, discards suspicious ones using a geometric distance filter (Multi‑Krum) and updates only 1.4 % of the parameters, saving bandwidth and time. Evaluated on the Cityscapes Video dataset, the approach achieves a video segmentation mIoU of 83.5 %, a temporal consistency score of 88.5, and reduces depth absolute relative error to 0.085, all while running in real‑time at 38 FPS on edge hardware. Even when up to 30 % of the sensors are Byzantine/malicious, the protocol detects attackers with 98.4 % accuracy, preserves a global mIoU of 81.9 %, and cuts OTA traffic from 1.2 GB to just 3.8 MB per round (over 99 % reduction). In short, efficient prompt caching removes frame‑to‑frame jitter and geometric gradient filtering defends against poisoning, paving the way for a secure, smooth urban perception system for next‑generation edge robotics.

The ST-HMR (Spatiotemporal Hierarchical Mask‑Refinement) relies on an asymmetrical temporal cross‑attention buffer that stores pixel‑level prompt tokens across multiple frames. This enables the model to propagate geometric context information without reprocessing the whole scene at each time step, cutting computational cost compared to a full 3D transformer. The buffer is updated via a gating mechanism that weights new tokens by their similarity to stored ones, ensuring only significant variations are incorporated. On the federated learning side, the BR-FL (Byzantine‑Robust Federated Learning) protocol employs Multi‑Krum, a geometric‑distance‑based filtering technique in gradient space. Each client sends the gradient of the active prompt subspace (about 1.4 % of total parameters). Multi‑Krum selects the K gradients with the smallest average distance to others, discarding outliers that likely correspond to Byzantine attacks. This method provides robustness as long as the fraction of malicious clients stays below 50 %; empirically it achieves a 98.4 % detection rate even with 30 % compromised nodes. Moreover, because only the prompt subspace is communicated, the OTA payload per round drops from roughly 1.2 GB (full‑weight transmission) to a few megabytes, making continuous updates feasible over limited‑bandwidth vehicle fleets. The outcome is an urban perception system that retains high semantic precision (mIoU > 80) and excellent temporal consistency (TC > 88), while edge‑hardware power consumption stays compatible with real‑time operation at 38 FPS. These findings suggest that prompt caching and geometric gradient filtering can become core pillars for distributed, secure, and efficient perception in autonomous driving and urban robotics.