[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-86154-en":3,"doc-seo-86154-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":11,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},86154,962075114101,"Seraphina","https://ap-avatar.wpscdn.com/avatar/e000253a75eb197efd?x-image-process=image/resize,m_fixed,w_180,h_180&k=1780044092746381165",8,"Research & Report","ANYSYNTH Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance","Zero-shot instrument cloning synthesizes a target MIDI with the acoustic identity of an unseen instrument from only a short reference audio–MIDI pair. Existing approaches compress reference audio into fixed embeddings via pretrained encoders such as CLAP, losing fine-grained timbre cues needed for faithful reconstruction. ANYSYNTH proposes an embedding-free neural synthesizer using in-context flow matching with a DiT conditioned directly on uncompressed reference audio and target MIDI. Experiments show improved audio quality, timbre similarity, and melody adherence, plus prompt-length scaling. Asymmetric Hierarchical CFG decouples MIDI and timbre guidance to improve controllability.","ANYSYNTH: ZERO-SHOT INSTRUMENT CLONING VIA IN-CONTEXT LEARNING AND ASYMMETRIC HIERARCHICAL GUIDANCE  \nChong Jing 1 Junan Zhang 1 Jing Yang2 Yulun Wu2 Fan Fan2 Zhizheng Wu 1  \n1 Chinese University of Hong Kong  \n2 Central Media Technology Institute, Huawei  \n[chongjing@link.cuhk.edu.cn](chongjing@link.cuhk.edu.cn) , [junanzhang@link.cuhk.edu.cn](junanzhang@link.cuhk.edu.cn)  \n[yangjing201@huawei.com](yangjing201@huawei.com) , [wuyulun9@huawei.com](wuyulun9@huawei.com) , [fanfan1@huawei.com](fanfan1@huawei.com)  \n[wuzhizheng@cuhk.edu.cn](wuzhizheng@cuhk.edu.cn)  \narXiv :2607 . 1 1 143v 1 [ cs . SD] 13 Jul 2026  \nABSTRACT  \nZero-shot instrument cloning aims to render an arbitrary [Target MIDI] sequence with the acoustic identity of an unseen instrument given only a short [Reference Audio, Reference MIDI] pair. Existing methods rely on pre-trained embeddings (e.g., CLAP) that compress the reference audio into a fixed-length vector, discarding fine-grained acoustic cues essential for faithful timbre reconstruction. We present ANYSYNTH, an embedding-free neural synthesizer based on in-context flow matching. By conditioning a Diffusion Transformer (DiT) directly on the uncompressed reference audio and target MIDI, our model allows self-attention to dynamically retrieve acoustic details at generation time. Experiments show that ANYSYNTH outperforms embedding-based and auto-regressive baselines in audio quality, timbre similarity, and melody adherence. Notably, the model exhibits prompt-length scaling: longer reference prompts yield steadily better timbre fidelity, a property absent in embedding-based systems. To optimize controllability, we further propose Asymmetric Hierarchical CFG, which structurally decouples MIDI and reference-timbre guidance based on their natural semanticacoustic dependency. This asymmetric formulation avoids gradient conflicts and improves both note accuracy and timbre fidelity, pushing the boundary of expressive, zeroshot instrument cloning. Demo audios are available at [https://anysynth-demo.github.io/](https://anysynth-demo.github.io/)  \n1. INTRODUCTION  \nRecent autoregressive and diffusion-based models have substantially improved music audio generation [1–12] . As synthesis quality increases, the next challenge is fine-grained controllability: rather than generating plausible audio in coarse granularity (e.g., conditioned on text like \"An r&b electric piano solo.\"), musicians require systems capable  \n © C. Jing, J. Zhang, J. Yang, Y. Wu, F. Fan, and Z. Wu. Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0) . Attribution: C. Jing, J. Zhang, J. Yang, Y. Wu, F. Fan, and Z. Wu,“ANYSYNTH: Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance”, in Proc. of the 27th Int. Society for Music Information Retrieval Conf., Abu Dhabi, UAE, 2026 .  \nof rendering a desired note sequence with a specific instrument identity. This motivates the task of zero-shot instrument cloning, where an arbitrary [Target MIDI] must be synthesized using the acoustic identity extracted from a short, aligned [Reference Audio, Reference MIDI] pair [13–16] .  \nExisting state-of-the-art (SOTA) solutions approach this instrument cloning task by disentangling musical \"structure\" (pitch and velocity) from \"timbre\". Whether built upon codec language models (e.g., TokenSynth [15]) or diffusion models (e.g., Control-Transfer-Diffusion [13], FlowSynth [14]), these models share a critical dependency: they heavily rely on a pre-trained semantic encoder such as CLAP or MuLan [17, 18] to extract a fixed-length timbre embedding. While architecturally convenient, this design introduces a severe semantic embedding bottleneck. Retrieval-oriented audio encoders are trained to preserve coarse semantic alignment, inherently discarding the lowlevel acoustic details – such as short transients, nuanced performance artifacts, and specific recording characteristics – required for faithful sound re","cbCaiu3idAsCYWzG","https://ap.wps.com/l/cbCaiu3idAsCYWzG","pdf",2338191,3,1,"English","en",105,"# Abstract\n# Introduction","[{\"question\":\"What problem does zero-shot instrument cloning address?\",\"answer\":\"It aims to synthesize an arbitrary target MIDI sequence using the acoustic identity of an unseen instrument, given only a short reference audio and reference MIDI pair.\"},{\"question\":\"Why do embedding-based methods limit timbre reconstruction quality?\",\"answer\":\"They compress reference audio into fixed-length embeddings using pretrained encoders, which discard fine-grained acoustic details like transients and recording-specific characteristics needed for faithful timbre reconstruction.\"},{\"question\":\"How does ANYSYNTH differ from embedding-based and auto-regressive baselines?\",\"answer\":\"It removes explicit timbre embeddings and performs in-context flow matching with a Diffusion Transformer conditioned directly on uncompressed reference audio and target MIDI. It also improves controllability using Asymmetric Hierarchical CFG to better balance MIDI guidance and reference timbre guidance during generation.\"}]",1784208959,20,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"anysynth-zero-shot-instrument-cloning-via-in-context-learning-and-asymmetric-hierarchical-guidance","",{"@graph":35,"@context":84},[36,52,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,49],{"item":40,"name":41,"@type":42,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":20},"https://docshare.wps.com/document/research-report/",{"item":50,"name":13,"@type":42,"position":51},"https://docshare.wps.com/document/anysynth-zero-shot-instrument-cloning-via-in-context-learning-and-asymmetric-hierarchical-guidance/86154/",4,{"url":50,"name":13,"@type":53,"author":54,"headline":13,"publisher":56,"fileFormat":59,"inLanguage":23,"description":14,"dateModified":60,"datePublished":61,"encodingFormat":59,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":55},"Person",{"url":40,"name":57,"@type":58},"DocShare","Organization","application/pdf","2026-07-26","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":20},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does zero-shot instrument cloning address?","Question",{"text":74,"@type":75},"It aims to synthesize an arbitrary target MIDI sequence using the acoustic identity of an unseen instrument, given only a short reference audio and reference MIDI pair.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"Why do embedding-based methods limit timbre reconstruction quality?",{"text":79,"@type":75},"They compress reference audio into fixed-length embeddings using pretrained encoders, which discard fine-grained acoustic details like transients and recording-specific characteristics needed for faithful timbre reconstruction.",{"name":81,"@type":72,"acceptedAnswer":82},"How does ANYSYNTH differ from embedding-based and auto-regressive baselines?",{"text":83,"@type":75},"It removes explicit timbre embeddings and performs in-context flow matching with a Diffusion Transformer conditioned directly on uncompressed reference audio and target MIDI. It also improves controllability using Asymmetric Hierarchical CFG to better balance MIDI guidance and reference timbre guidance during generation.","https://schema.org",{"og:url":50,"og:type":86,"og:title":13,"og:site_name":57,"og:description":14},"article",{"robots":88,"canonical":50},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,126,129,133],{"id":21,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":51,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":28,"slug":125},9,"Religion & Spirituality","religion-spirituality",{"id":28,"doc_module":4,"doc_module_name":45,"category_name":127,"show_sort_weight":28,"slug":128},"World Cup","world-cup",{"id":130,"doc_module":4,"doc_module_name":45,"category_name":131,"show_sort_weight":130,"slug":132},10,"Lifestyle","lifestyle",{"id":134,"doc_module":4,"doc_module_name":45,"category_name":135,"show_sort_weight":105,"slug":136},19,"General","general"]