Skin Lesion Phenotyping via Nested Multi-modal Contrastive Learning

We introduce SLIMP (Skin Lesion Image-Metadata Pre-training) for learningrich representations of skin lesions through a novel nested contrastivelearning approach that captures complex relationships between images andmetadata. Melanoma detection and skin lesion classification based solely onimages, pose significant challenges due to large variations in imagingconditions (lighting, color, resolution, distance, etc.) and lack of clinicaland phenotypical context. Clinicians typically follow a holistic approach forassessing the risk level of the patient and for deciding which lesions may bemalignant and need to be excised, by considering the patient's medical historyas well as the appearance of other lesions of the patient. Inspired by this,SLIMP combines the appearance and the metadata of individual skin lesions withpatient-level metadata relating to their medical record and other clinicallyrelevant information. By fully exploiting all available data modalitiesthroughout the learning process, the proposed pre-training strategy improvesperformance compared to other pre-training strategies on downstream skinlesions classification tasks highlighting the learned representations quality.