[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-121051-en":3,"doc-seo-121051-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},121051,2336464648322,"Aria","https://ap-avatar.wpscdn.com/avatar/2200025388227c56fec?_k=1778556882303663488",8,"Research & Report","Building Sketch-to-Sound Mapping - with Unsupervised Feature Extraction and Interactive Machine Learning","The paper investigates how creators can construct and explore mappings between visual sketches and musical controls through Interactive Machine Learning (IML). Because sketches are high-dimensional, dimensionality reduction is used to extract compact features for the IML model. The authors propose unsupervised learning to encode sketches into lower-dimensional latent representations, which then drive sketch-to-sound mapping construction. A proof-of-concept prototype is built and evaluated with two compositions to analyze controllability, explorability, and their impact on musical expression.","Building Sketch-to-Sound Mapping with Unsupervised Feature Extraction and Interactive Machine Learning  \nShuoyang Zheng  \nCentre for Digital Music Queen Mary University of London  \n[shuoyang.zheng@qmul.ac.uk](shuoyang.zheng@qmul.ac.uk)  \nBleiz M. Del Sette  \nCentre for Digital Music Queen Mary University of London [b.delsette@qmul.ac.uk](b.delsette@qmul.ac.uk)  \nCharalampos Saitis  \nCentre for Digital Music Queen Mary University of London [c.saitis@qmul.ac.uk](c.saitis@qmul.ac.uk)  \nAnna Xambó  \nCentre for Digital Music Queen Mary University of London  \n[a.xambosedo@qmul.ac.uk](a.xambosedo@qmul.ac.uk)  \nNick Bryan-Kinns  \nCreative Computing Institute University of the Arts London [n.bryankinns@arts.ac.uk](n.bryankinns@arts.ac.uk)  \nABSTRACT  \nIn this paper, we explore the interactive construction and exploration of mappings between visual sketches and musical controls. Interactive Machine Learning (IML) allows creators to construct mappings with personalised training examples. However, when it comes to high-dimensional data such as sketches, dimensionality reduction techniques are required to extract features for the IML model. We propose using unsupervised machine learning to encode sketches into lower-dimensional latent representations, which are then used as the source for the IML model to construct sketchto-sound mappings. We build a proof-of-concept prototype and demonstrate it using two compositions. We reflect on the composing processes to discuss the controllability and explorability in mappings built by this approach and how they contribute to the musical expression.  \nAuthor Keywords  \nCross-modal mapping, unsupervised learning, variational autoencoder, sound synthesis control  \nCCS Concepts  \n•Applied computing → Sound and music computing; •Humancentered computing → Graphics input devices; •Computing methodologies → Neural networks;  \n1. INTRODUCTION  \nSketching is an intuitive and natural form of communication, and there has been extensive research on using sketches as sound control interfaces within the NIME community. These works have been used for a variety of applications,  \nLicensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0) . Copyright remains with the author(s) .  \nNIME’24, 4–6 September, Utrecht, The Netherlands.  \nincluding cross-modal control of sound synthesis [17], composition [1, 5], annotation [28], melody generation [13, 22], and graphic sonification [3, 23, 31] . Cross-modal studies have shown that meaningful perceptual associations exist between shapes and sounds [15], but these associations inmost sketch-to-sound applications are pre-determined by instrument makers, and fixed for all musicians who use them [25] . However, a musician may seek to personalise these shape-sound mappings for specific creative goals [12, 20] . Therefore, in this paper we aim to explore the potential of using interactive machine learning to help musicians build personalised sketch-to-sound mappings.  \nInteractive Machine Learning (IML) [7] is commonly used to create small-scale tailored mapping models. In the domain of music, it is often used to build mapping between sensor inputs and sound controls [8] . However, when it comes to complex high-dimensional inputs, such as sketches, feature extraction techniques are required to encode these inputs into lower-dimensional representations to be used asthe source of the IML model.  \nPrevious works [2, 30] tackling image information retrieval have shown unsupervised feature learning’s good capability in extracting representative features from a corpus of unlabeled data. However, with a few exceptions [21, 26], there have not been many works on using this technique to create expressive musical mappings. Therefore, we focus on exploring unsupervised feature learning to build mappings between visual sketches and sound controls. The work presented in this paper is driven by two research questions:  \n1. How can we leverage unsupervised feature learnin","cbCailow3SwoMOG6","https://ap.wps.com/l/cbCailow3SwoMOG6","pdf",1461056,1,7,"English","en",105,"# Introduction\n## Related Work\n## Unsupervised Feature Learning","[{\"question\":\"How does the proposed approach connect visual sketches to musical controls?\",\"answer\":\"Sketches are first encoded into lower-dimensional latent representations using unsupervised feature learning, and those latent features become the input source for an IML model that constructs sketch-to-sound mappings.\"},{\"question\":\"Why are dimensionality reduction techniques necessary for sketch-based IML?\",\"answer\":\"Sketch data is high-dimensional, so feature extraction is required to reduce it into a lower-dimensional form that the IML model can use effectively.\"},{\"question\":\"What evidence is provided to evaluate the method?\",\"answer\":\"The authors build a proof-of-concept prototype and demonstrate it using two compositions, then reflect on how the resulting mappings support controllability and explorability during composing.\"}]","Building Sketch-to-Sound Mapping - with Unsupervised Feature Extraction and Interactive Machine Learning | PDF",1785733499,18,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"building-sketch-to-sound-mapping-with-unsupervised-feature-extraction-and-interactive-machine-learning","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/building-sketch-to-sound-mapping-with-unsupervised-feature-extraction-and-interactive-machine-learning/121051/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"How does the proposed approach connect visual sketches to musical controls?","Question",{"text":75,"@type":76},"Sketches are first encoded into lower-dimensional latent representations using unsupervised feature learning, and those latent features become the input source for an IML model that constructs sketch-to-sound mappings.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why are dimensionality reduction techniques necessary for sketch-based IML?",{"text":80,"@type":76},"Sketch data is high-dimensional, so feature extraction is required to reduce it into a lower-dimensional form that the IML model can use effectively.",{"name":82,"@type":73,"acceptedAnswer":83},"What evidence is provided to evaluate the method?",{"text":84,"@type":76},"The authors build a proof-of-concept prototype and demonstrate it using two compositions, then reflect on how the resulting mappings support controllability and explorability during composing.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":116,"show_sort_weight":117,"slug":118},"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]