[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85788-en":3,"doc-seo-85788-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85788,8796095462418,"Noah","https://ap-avatar.wpscdn.com/avatar/80000253c1241d02b47?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778826106357471780",8,"Research & Report","Local Multimodal Music Alignment from Global Supervision","Local multimodal music understanding depends on learning localized correspondences between modalities, such as mapping performance audio time to positions in a sheet-music image. However, local alignment supervision is costly, while datasets typically provide only global paired supervision (audio-image segments). FuSiLi (Fused Sinkhorn-Localized Similarity) is introduced as a multimodal contrastive similarity score that performs Sinkhorn-based soft alignment on local patches and frames. Experiments fine-tune CLIP/CLAP encoders on sheet-music images and audio, improving frame-level alignment while remaining competitive for cross-modal retrieval.","LOCAL MULTIMODAL MUSIC ALIGNMENT  \nFROM GLOBAL SUPERVISION  \nIrmak Bukey 1 Zachary Novack2 Jongmin Jung3 Dasaem Jeong4 Chris Donahue 1 1 Carnegie Mellon University 2 University of California San Diego 3 Neutune 4 Sogang University  \narXiv :2607 . 10023v 1 [ cs . SD] 10 Jul 2026  \nABSTRACT  \nUnderstanding music requires understanding localized relationships across data modalities, e.g., how time in performance audio maps onto position in a score image. Yet supervision for such local correspondences is difficult to obtain—in practice, we often only have access to coarser global supervision like paired segments of audio and images. To address this gap, we propose FuSiLi (Fused Sinkhorn-Localized Similarity), a similarity score for multimodal contrastive learning operating directly on local image patch and audio frame features via Sinkhornbased soft alignment. We show that FuSiLi (i) effectively learns local relationships, (ii) requires only global supervision, and (iii) retains the global alignment capabilities of conventional contrastive approaches. We fine-tune pretrained CLIP and CLAP encoders on pairs of raw sheet music images and audio using a hybrid contrastive objective combining FuSiLi with conventional global similarity. We evaluate on cross-modal retrieval and frame-level alignment tasks against a range of global and local baselines, showing that our approach outperforms them on local alignment while remaining competitive on retrieval.  \n1. INTRODUCTION  \nMany multimodal music understanding tasks involve localized correspondences between data modalities [1–3] . For example, we may want to understand how performance audio aligns to an image of the corresponding sheet music. Ideally, supervision for this task would involve two types of ground truth information: (1) global supervision pairing the right audio with the right image, and (2) a local alignment specifying a mapping between time in the audio and pixels in the image. If we could collect both forms of supervision at scale, we could learn to predict the local alignment for a new pair of inputs in a fully supervised fashion. However, it is prohibitively expensive to collect local alignments at scale in practice, as it requires significant manual effort from expert annotators [4, 5] . Compared to the expense of collecting local alignments, it is  \n © I. Bukey, Z. Novack, J. Jung, D. Jeong, and C. Donahue. Licensed under a Creative Commons Attribution 4 .0 International License (CC BY 4.0) . Attribution: I. Bukey, Z. Novack, J. Jung, D. Jeong, and C. Donahue,“Local Multimodal Music Alignment from Global Supervision”, in Proc. of the 27th Int. Society for Music Information Retrieval Conf., Abu Dhabi, UAE, 2026 .  \nmuch cheaper to collect global supervision in the form of audio-image pairs that correspond to the same piece. Such supervision scales naturally from numerous sources, as IMSLP provides pairs at the piece-level, while datasets such as YTSV [6] offer pairs on more granular segments (a few measures of music) . This broad availability motivatesour key research question: can we learn local alignments across modalities from global supervision alone?  \nThe dominant approach for leveraging multimodal global supervision at scale is contrastive learning. Models like CLIP [7] and CLAP [8,9] have demonstrated that contrastive objectives applied to globally paired data can yield powerful multimodal representations. However, we show that this approach is not effective at capturing local alignments, achieving only 16% top-1 accuracy on a frame-level alignment task (here, aligning time frames of audio to pixel patches in the score image) . Other specialized approaches attempt to learn localized alignments but carry significant drawbacks: some need explicit local supervision [10], while others are prohibitively expensive for retrieval [6], requiring |Q| × |R| forward passes (query/retrieval sets Q/R), compared to the |Q| + |R| passes for contrastive models.  \nTo address this ","cbCaidM1QY1FPWrJ","https://ap.wps.com/l/cbCaidM1QY1FPWrJ","pdf",1204348,1,9,"English","en",105,"# Abstract\n# Introduction\n# Related Work","[{\"question\":\"What problem does the paper address?\",\"answer\":\"It addresses learning fine-grained local alignments between music modalities, such as aligning audio time frames to pixel patches in sheet-music images, using only cheaper global supervision (paired audio-image segments).\"},{\"question\":\"How does FuSiLi improve over standard contrastive learning?\",\"answer\":\"FuSiLi uses a fuse-then-pool formulation: it computes pairwise similarities between local frame-level features and applies Sinkhorn-derived soft alignment before pooling, enabling local correspondences to emerge from global supervision.\"},{\"question\":\"What models and training setup does FuSiLi use?\",\"answer\":\"The method fine-tunes pretrained CLIP and CLAP encoders on raw sheet-music image and audio pairs using a hybrid contrastive objective that combines FuSiLi with conventional global similarity.\"}]",1784206294,23,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"local-multimodal-music-alignment-from-global-supervision","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/local-multimodal-music-alignment-from-global-supervision/85788/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper address?","Question",{"text":75,"@type":76},"It addresses learning fine-grained local alignments between music modalities, such as aligning audio time frames to pixel patches in sheet-music images, using only cheaper global supervision (paired audio-image segments).","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does FuSiLi improve over standard contrastive learning?",{"text":80,"@type":76},"FuSiLi uses a fuse-then-pool formulation: it computes pairwise similarities between local frame-level features and applies Sinkhorn-derived soft alignment before pooling, enabling local correspondences to emerge from global supervision.",{"name":82,"@type":73,"acceptedAnswer":83},"What models and training setup does FuSiLi use?",{"text":84,"@type":76},"The method fine-tunes pretrained CLIP and CLAP encoders on raw sheet-music image and audio pairs using a hybrid contrastive objective that combines FuSiLi with conventional global similarity.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":21,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]