[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85008-en":3,"doc-seo-85008-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85008,13056703019404,"Miles","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","A Theory of Contrastive Learning with Natural Images","Analytical study of why contrastive learning with simple augmentations and stationary natural-image statistics produces representations useful for downstream tasks. The work derives the optimal representation under contrastive loss across a range of basic augmentations and shows the optimum can be realized by a CNN whose first-layer filters are sinusoids, followed by pointwise nonlinearity, global average pooling, and a final linear partial-whitening stage. For more complex augmentations, optimal CNN weights remain sinusoidal, with frequencies and weights computable via a “waterfilling” algorithm from the dataset power spectrum. Experiments confirm that SGD learns sinusoidal first-layer filters and partial whitening.","A Theory of Contrastive Learning with Natural Images  \nAntonio Torralba 1 Yair Weiss 2  \narXiv :2607 .07470v 1 [ cs .CV] 8 Jul 2026  \nAbstract  \nWhy does contrastive learning with simple images and augmentations yield useful representations for downstream tasks? We address this question by analytically computing the optimal representation in terms of the contrastive loss for a range of basic augmentations and any image dataset with stationary statistics. We show that for certain augmentations the optimum can be attained by a CNN whose first layer filters are sinusoids, followed by a pointwise nonlinearity, global average pooling, and a final linear layer that performs partial whitening. We also show that the optimal weights in such CNNs for more complicated augmentations are still sinusoids. The frequencies of the sinusoids and their weights can be computed using a simple “waterfilling” algorithm given the dataset’s expected power spectrum. Experiments with different image datasets and augmentations show that such CNNs trained with SGD empirically learn sinusoids in their first layer and to perform partial whitening.  \nContrastive learning (CL) is a remarkably successful method for learning useful image representations without labeled data. While numerous variants have been suggested, almost all of them follow the recipe suggested by (Chen et al., 2020) . For each training image, an augmentation is applied, and the goal of learning is to find a representation where two augmentations ofthe same image are close, while augmentations of different images are far away. Although conceptually simple, when applied to large scale datasets, these approaches have paved the way for image representations that can then be used to solve a large number of computer vision tasks without requiring additional representation learning (e.g. (Oquab et al., 2024)) .  \nIn this paper, we seek to understand why contrastive learning  \n1 CSAIL, MIT 2 School of Computer Science and Engineering, Hebrew University of Jerusalem. Correspondence to: Yair Weiss \u003C[yair.weiss@mail.huji.ac.il](yair.weiss@mail.huji.ac.il) >.  \nProceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) .  \nOriginal Blur Random crop Hor. flip Color jitter  \nCIFAR10  \nFigure 1. Mysteries of contrastive learning. Columns show simple augmentations that are used in CL. A combination of these low-level augmentations yields state-of-the-art recognition performance. Rows show different image datasets: real images (top), fractal noise (middle) and dead leaves (bottom) . Even simple augmentations with nonrealistic images yield useful features for real images (Baradad et al., 2021) .  \nworks so well when it is applied to natural image datasets. A large number of papers have shown that variants of CL are closely related to spectral methods for unsupervised learning such as Laplacian Eigenmaps (Belkin & Niyogi, 2001) and Spectral Clustering (Ng et al., 2001) (e.g. (Balestriero & LeCun, 2022 ; HaoChen et al., 2021 ; Bansal et al., 2025)) . But as pointed out in (HaoChen & Ma, 2023 ; Saunshi et al., 2022) without additional inductive biases, for finite datasets both CL and spectral methods can lead to trivial solutions that are not useful for any downstream tasks.  \nSpecifically our paper was motivated by two aspects of the success of contrastive learning that we find mysterious: the fact that it works with simple augmentations and the fact that it works with simple images of noise (Baradad et al., 2021) .  \nThe first motivation for our work is illustrated by the columns of figure 1 which show the augmentations that are typically used in state-of-the-art CL. These augmentations are extremely low-level. For example, the default configuration of (Chen et al., 2020) uses only three augmentations: random crop, color jitter and Gaussian blur. These augmentations seem to have nothing to do with object recognition and it is of","cbCailTC8fuW6eDy","https://ap.wps.com/l/cbCailTC8fuW6eDy","pdf",27326387,1,30,"English","en",105,"# Abstract\n# Contrastive learning and spectral connections\n# Motivation: simple augmentations and simple images\n## Low-level augmentations and invariance\n## Learning from non-realistic/noise images\n# Main theoretical results and CNN structure\n## Waterfilling computation from power spectrum\n# Experiments and empirical confirmation","[{\"question\":\"What central question does the paper address about contrastive learning?\",\"answer\":\"Why contrastive learning using simple images and augmentations yields representations that transfer well to downstream tasks.\"},{\"question\":\"Under what conditions does the paper show the optimal representation form?\",\"answer\":\"For stationary-statistics image datasets and a wide range of basic augmentations, the optimal solution can be characterized analytically.\"},{\"question\":\"What structure does the paper claim for an optimal CNN?\",\"answer\":\"It can be achieved by a CNN whose first-layer filters are sinusoids, followed by a pointwise nonlinearity, global average pooling, and a final linear layer performing partial whitening.\"}]",1784200202,76,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"a-theory-of-contrastive-learning-with-natural-images","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/a-theory-of-contrastive-learning-with-natural-images/85008/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What central question does the paper address about contrastive learning?","Question",{"text":74,"@type":75},"Why contrastive learning using simple images and augmentations yields representations that transfer well to downstream tasks.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"Under what conditions does the paper show the optimal representation form?",{"text":79,"@type":75},"For stationary-statistics image datasets and a wide range of basic augmentations, the optimal solution can be characterized analytically.",{"name":81,"@type":72,"acceptedAnswer":82},"What structure does the paper claim for an optimal CNN?",{"text":83,"@type":75},"It can be achieved by a CNN whose first-layer filters are sinusoids, followed by a pointwise nonlinearity, global average pooling, and a final linear layer performing partial whitening.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,121,126,129,133],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":21,"slug":120},"research-report",{"id":122,"doc_module":4,"doc_module_name":45,"category_name":123,"show_sort_weight":124,"slug":125},9,"Religion & Spirituality",20,"religion-spirituality",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":124,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]