Why More Transformations Don't Always Improve Medical Segmentation?

In recent years, machine learning has learned to study images without needing labels, thanks to "contrastive learning." Basically, the model learns to tell two versions of the same picture apart from two different pictures. To make the task harder – and thus more useful – we use "data augmentation": rotating, changing contrast, adding noise, and so on.

It would seem that more transformations always mean better results, but researchers discovered that for medical images this isn’t true. When segmenting organs or lesions, some modifications can confuse the model, causing it to lose crucial details. The authors tested various combinations of transformations, finding that lighter ones – like small shifts or brightness tweaks – truly improve accuracy, while aggressive versions degrade performance.

The outcome is surprising: the key isn’t the amount of transformations, but their relevance to the task. This opens the door to new training strategies where augmentations are tailored to the image type and clinical goal. In the future, smarter algorithms could automatically decide which transformations to apply, making automatic segmentation more reliable and faster for doctors.

Contrastive learning relies on a loss function called "InfoNCE," which maximizes similarity between two positive views (the same image with different augmentations) and minimizes similarity with negative views (different images). In medical imaging, the intrinsic variability is lower than in natural images: organs and tissues have well‑defined structures and are often acquired with standardized protocols. Consequently, overly aggressive augmentations—such as 90° rotations or extreme geometric distortions—can introduce artifacts not present in clinical reality, altering the topology of the structures to be segmented.

The authors compared a classic augmentation pipeline (random crop, color jitter, Gaussian blur) with a more conservative selection (small random translation, slight contrast adjustment, mild Gaussian noise). Results show that the conservative pipeline raises the IoU (Intersection over Union) by about 3–5% compared to the aggressive version, especially on brain MRI datasets. They also introduced an "augmentation scheduling" strategy: early training stages use stronger transformations to push the model to explore the representation space, while later stages reduce the modifications to fine‑tune details.

Another crucial aspect is the choice of negatives. In medical contexts, it is useful to include as negatives not only images from other patients but also slightly altered versions of the same image, teaching the model to distinguish physiological variations from acquisition artifacts. Finally, techniques like "mixup" or "cutmix"—which combine parts of different images—were evaluated but showed limited benefits, likely because mixing tissues from different organs creates unrealistic patterns. These findings suggest that designing augmentations for contrastive learning in medical imaging requires a delicate balance between diversity and anatomical fidelity.