[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-121189-en":3,"doc-seo-121189-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},121189,7971461740886,"Theodore","https://ap-avatar.wpscdn.com/davatar_3d24733baf745e90a7e4bdd5f77d97b2",6,"Technology","Building Sketch-to-Sound Mapping with Unsupervised Feature Extraction and Interactive Machine Learning - Summary","Interactive machine learning enables creators to build personalised mappings between inputs and musical controls using their own examples, but sketches are high-dimensional and require feature extraction before mapping. This work proposes using unsupervised learning to encode sketches into compact latent representations, which then serve as the input source for interactive sketch-to-sound mapping. A proof-of-concept prototype is built and evaluated through two composed performance demonstrations, followed by reflection on how controllability and explorability support musical expression.","Building Sketch-to-Sound Mapping with Unsupervised Feature Extraction and Interactive Machine Learning  \nShuoyang Zheng  \nCentre for Digital Music Queen Mary University of London  \n[shuoyang.zheng@qmul.ac.uk](shuoyang.zheng@qmul.ac.uk)  \nBleiz M. Del Sette  \nCentre for Digital Music Queen Mary University of London [b.delsette@qmul.ac.uk](b.delsette@qmul.ac.uk)  \nCharalampos Saitis  \nCentre for Digital Music Queen Mary University of London [c.saitis@qmul.ac.uk](c.saitis@qmul.ac.uk)  \nAnna Xambó  \nCentre for Digital Music Queen Mary University of London  \n[a.xambosedo@qmul.ac.uk](a.xambosedo@qmul.ac.uk)  \nNick Bryan-Kinns  \nCreative Computing Institute University of the Arts London [n.bryankinns@arts.ac.uk](n.bryankinns@arts.ac.uk)  \nABSTRACT  \nIn this paper, we explore the interactive construction and exploration of mappings between visual sketches and musical controls. Interactive Machine Learning (IML) allows creators to construct mappings with personalised training examples. However, when it comes to high-dimensional data such as sketches, dimensionality reduction techniques are required to extract features for the IML model. We propose using unsupervised machine learning to encode sketches into lower-dimensional latent representations, which are then used as the source for the IML model to construct sketchto-sound mappings. We build a proof-of-concept prototype and demonstrate it using two compositions. We reflect on the composing processes to discuss the controllability and explorability in mappings built by this approach and how they contribute to the musical expression.  \nAuthor Keywords  \nCross-modal mapping, unsupervised learning, variational autoencoder, sound synthesis control  \nCCS Concepts  \n•Applied computing → Sound and music computing; •Humancentered computing → Graphics input devices; •Computing methodologies → Neural networks;  \n1. INTRODUCTION  \nSketching is an intuitive and natural form of communication, and there has been extensive research on using sketches as sound control interfaces within the NIME community. These works have been used for a variety of applications,  \nLicensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0) . Copyright remains with the author(s) .  \nNIME’24, 4–6 September, Utrecht, The Netherlands.  \nincluding cross-modal control of sound synthesis [17], composition [1, 5], annotation [28], melody generation [13, 22], and graphic sonification [3, 23, 31] . Cross-modal studies have shown that meaningful perceptual associations exist between shapes and sounds [15], but these associations inmost sketch-to-sound applications are pre-determined by instrument makers, and fixed for all musicians who use them [25] . However, a musician may seek to personalise these shape-sound mappings for specific creative goals [12, 20] . Therefore, in this paper we aim to explore the potential of using interactive machine learning to help musicians build personalised sketch-to-sound mappings.  \nInteractive Machine Learning (IML) [7] is commonly used to create small-scale tailored mapping models. In the domain of music, it is often used to build mapping between sensor inputs and sound controls [8] . However, when it comes to complex high-dimensional inputs, such as sketches, feature extraction techniques are required to encode these inputs into lower-dimensional representations to be used asthe source of the IML model.  \nPrevious works [2, 30] tackling image information retrieval have shown unsupervised feature learning’s good capability in extracting representative features from a corpus of unlabeled data. However, with a few exceptions [21, 26], there have not been many works on using this technique to create expressive musical mappings. Therefore, we focus on exploring unsupervised feature learning to build mappings between visual sketches and sound controls. The work presented in this paper is driven by two research questions:  \n1. How can we leverage unsupervised feature learnin","cbCaiqLbwXHfqPLm","https://ap.wps.com/l/cbCaiqLbwXHfqPLm","pdf",1453368,1,7,"English","en",105,"# Abstract\n# Introduction\n# Related Work\n## Unsupervised Feature Learning","[{\"question\":\"How does the proposed approach build sketch-to-sound mappings?\",\"answer\":\"It encodes sketches into lower-dimensional latent representations using unsupervised learning, then feeds those representations into an interactive machine learning model to construct sketch-to-sound mappings.\"},{\"question\":\"Why is dimensionality reduction needed for sketch inputs in interactive machine learning?\",\"answer\":\"Sketches are high-dimensional, so a feature extraction step is required to represent them in a compact latent space that the interactive model can use.\"},{\"question\":\"What aspects of the mapping are analyzed after the prototype demonstrations?\",\"answer\":\"The work reflects on the composing process to discuss controllability and explorability, and how these properties contribute to musical expression.\"}]","Building Sketch-to-Sound Mapping with Unsupervised Feature Extraction and Interactive Machine Learning - Summary | PDF",1785734275,18,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"building-sketch-to-sound-mapping-with-unsupervised-feature-extraction-and-interactive-machine-learning-summary","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/technology/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/building-sketch-to-sound-mapping-with-unsupervised-feature-extraction-and-interactive-machine-learning-summary/121189/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"How does the proposed approach build sketch-to-sound mappings?","Question",{"text":75,"@type":76},"It encodes sketches into lower-dimensional latent representations using unsupervised learning, then feeds those representations into an interactive machine learning model to construct sketch-to-sound mappings.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why is dimensionality reduction needed for sketch inputs in interactive machine learning?",{"text":80,"@type":76},"Sketches are high-dimensional, so a feature extraction step is required to represent them in a compact latent space that the interactive model can use.",{"name":82,"@type":73,"acceptedAnswer":83},"What aspects of the mapping are analyzed after the prototype demonstrations?",{"text":84,"@type":76},"The work reflects on the composing process to discuss controllability and explorability, and how these properties contribute to musical expression.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,113,117,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":111,"slug":112},50,"technology",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":114,"show_sort_weight":115,"slug":116},"Healthcare",40,"healthcare",{"id":118,"doc_module":4,"doc_module_name":46,"category_name":119,"show_sort_weight":120,"slug":121},8,"Research & Report",30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]