Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualization

by   Zi-Hao Qiu, et al.

In this paper, we aim to optimize a contrastive loss with individualized temperatures in a principled and systematic manner for self-supervised learning. The common practice of using a global temperature parameter τ ignores the fact that “not all semantics are created equal", meaning that different anchor data may have different numbers of samples with similar semantics, especially when data exhibits long-tails. First, we propose a new robust contrastive loss inspired by distributionally robust optimization (DRO), providing us an intuition about the effect of τ and a mechanism for automatic temperature individualization. Then, we propose an efficient stochastic algorithm for optimizing the robust contrastive loss with a provable convergence guarantee without using large mini-batch sizes. Theoretical and experimental results show that our algorithm automatically learns a suitable τ for each sample. Specifically, samples with frequent semantics use large temperatures to keep local semantic structures, while samples with rare semantics use small temperatures to induce more separable features. Our method not only outperforms prior strong baselines (e.g., SimCLR, CLIP) on unimodal and bimodal datasets with larger improvements on imbalanced data but also is less sensitive to hyper-parameters. To our best knowledge, this is the first methodical approach to optimizing a contrastive loss with individualized temperatures.


page 9

page 22


Mini-Batch Optimization of Contrastive Loss

Contrastive learning has gained significant attention as a method for se...

Probabilistic Contrastive Loss for Self-Supervised Learning

This paper proposes a probabilistic contrastive loss function for self-s...

Simple and Effective Balance of Contrastive Losses

Contrastive losses have long been a key ingredient of deep metric learni...

Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Data

Most approaches for self-supervised learning (SSL) are optimised on cura...

Model-Aware Contrastive Learning: Towards Escaping Uniformity-Tolerance Dilemma in Training

Instance discrimination contrastive learning (CL) has achieved significa...

Stochastic Optimization of Area Under Precision-Recall Curve for Deep Learning with Provable Convergence

Areas under ROC (AUROC) and precision-recall curves (AUPRC) are common m...

Please sign up or login with your details

Forgot password? Click here to reset