[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-271966-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-271966-en":131},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","semi-supervised-adversarial-text-generation-based-on-seq2seq-models","Semi-supervised Adversarial Text Generation based on Seq2Seq models","","Adversarial training improves robustness, yet generating adversarial perturbations for text is difficult because discrete tokens can break naturalness and fail to preserve semantics, which are crucial for real-world language understanding. This paper presents a semi-supervised approach that uses pseudo-labeled in-domain data to train a seq2seq adversarial generator and combines it with paraphrase detection for quality control. Experiments on an intent classification NLU task show realistic, relevant adversarial samples and up to 70% recovery of affected tail errors, yielding stronger overall robustness.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/semi-supervised-adversarial-text-generation-based-on-seq2seq-models/271966/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/semi-supervised-adversarial-text-generation-based-on-seq2seq-models/271966.png","ImageObject",300,407,{"name":42,"@type":43},"Arica Lee","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-10-31","2026-09-15",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",15,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"Why is adversarial training harder for text than for computer vision?","Question",{"text":63,"@type":64},"Text perturbations are difficult because tokens are discrete, so generated adversarial text may sound unnatural or fail to preserve semantics, harming real-world intent understanding.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does the proposed method generate adversarial text samples?",{"text":68,"@type":64},"It trains a seq2seq model using pseudo-labeled in-domain data to produce adversarial versions of inputs, and uses paraphrase detection as a constraint to keep the generated samples high quality.",{"name":70,"@type":61,"acceptedAnswer":71},"What is the impact of using the generated adversarial samples for training?",{"text":72,"@type":64},"Including these adversarial samples improves robustness of the NLU model and can recover up to 70% of the errors caused by problematic input types, especially improving tail accuracy.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},271966,1789462535,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,120,123,127],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":116,"doc_module":4,"doc_module_name":25,"category_name":117,"show_sort_weight":118,"slug":119},9,"Religion & Spirituality",20,"religion-spirituality",{"id":118,"doc_module":4,"doc_module_name":25,"category_name":121,"show_sort_weight":118,"slug":122},"World Cup","world-cup",{"id":124,"doc_module":4,"doc_module_name":25,"category_name":125,"show_sort_weight":124,"slug":126},10,"Lifestyle","lifestyle",{"id":128,"doc_module":4,"doc_module_name":25,"category_name":129,"show_sort_weight":97,"slug":130},19,"General","general",{"code":4,"msg":82,"data":132},{"doc_id":79,"user_id":133,"nickname":42,"user_avatar":134,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":116,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":80,"read_time":144,"preview_page_count":4},8796096645457,"https://ap-avatar.wpscdn.com/avatar/800003749518d68ffe3?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779345340919836971","Semi-supervised Adversarial Text Generation based on Seq2Seq models  \nHieu Le ∗  \nBoston University, Boston, USA [hle@bu.edu](hle@bu.edu)  \nDieu-Thu Le  \nAmazon Inc., Berlin, Germany [deule@amazon.com](deule@amazon.com)  \nVerena Weber  \nAmazon Inc., Berlin, Germany [wverena@amazon.com](wverena@amazon.com)  \nKay Rottmann  \nAmazon Inc., Berlin, Germany [krrottm@amazon.com](krrottm@amazon.com)  \nChris Church  \nAmazon Inc., Berlin, Germany [cbchurch@amazon.com](cbchurch@amazon.com)  \nMelanie Bradford  \nAmazon Inc., Berlin, Germany [neunerm@amazon.com](neunerm@amazon.com)  \nPeter Chin  \nBoston University, Boston, USA [spchin@bu.edu](spchin@bu.edu)  \nAbstract  \nTo improve deep learning models’ robustness, adversarial training has been frequently used in computer vision with satisfying results. However, adversarial perturbation on text have turned out to be more challenging due to the discrete nature of text. The generated adversarial text might not sound natural or does not preserve semantics, which is the key for real world applications where text classification is based on semantic meaning. In this paper, we describe a new way for generating adversarial samples by using pseudo-labeled in-domain text data to train a seq2seq model for adversarial generation and combine it with paraphrase detection. We showcase the benefit of our approach for a real-world Natural Language Understanding (NLU) task, which maps a user’s request to an intent. Furthermore, we experiment with gradient-based training for the NLU task and try using token importance scores to guide the adversarial text generation. We show that our approach can generate realistic and relevant adversarial samples compared to other state-of-the-art adversarial training methods. Applying adversarial training using these generated samples helps the NLU model to recover up to 70% of these types of errors and makes the model more robust, especially in the tail distribution in a large scale real world application.  \n∗The work is completed during Hieu Le’s internship at Amazon Inc.  \n1 Introduction  \nOver the past years, neural machine learning models have become more popular in a wide range of real world natural language applications. While these models have become very accurate, they are susceptible to errors due to small changes in the input, e.g. adding a stop word. This makes it difficult for a natural language understanding system to recognize all utterances correctly since there are many ways to formulate one request. In fact, while common user commands can be understood very well by the system, a system can react differently due to minor input changes, e.g., article variations, paraphrasing, or adding functional words.  \nNatural Language Understanding (NLU) is at the core of voice assistants and maps a user’s request (also referred to as utterance later) to a specific intent within a certain domain, i.e. PlayMusic intent within the Music domain. In this paper, we identify the weaknesses of an NLU text classification model by finding the types of inputs that have high probability of causing prediction errors and show how to mitigate them. Inspired by recent work in adversarial training and adversarial sample generation (Goodfellow et al., 2015 ; Morris et al., 2020a), we describe how we employ adversarial text perturbation to identify and fix such samples to ensure a stable behavior of the model and thereby significantly boost the tail accuracy in a large scale real world application through adversarial training.  \n264  \nProceedings of EMNLP 2022 Industry Track, pages 264–272 December 9–11, 2020 . ©2022 Association for Computational Linguistics  \nWe show that while common text perturbation methods relying on character manipulation (Ebrahimi et al., 2018 ; Li et al., 2019), word swap (Alzantot et al., 2018 ; Ren et al., 2019) could generate adversarial samples on the original text inputs to trick the model, most of them often generate many irrelevant samples that either do not m","cbCaiidVsAsFkWo3","https://ap.wps.com/l/cbCaiidVsAsFkWo3","pdf",648882,"English","# Abstract\n# Introduction\n# Related work","[{\"question\":\"Why is adversarial training harder for text than for computer vision?\",\"answer\":\"Text perturbations are difficult because tokens are discrete, so generated adversarial text may sound unnatural or fail to preserve semantics, harming real-world intent understanding.\"},{\"question\":\"How does the proposed method generate adversarial text samples?\",\"answer\":\"It trains a seq2seq model using pseudo-labeled in-domain data to produce adversarial versions of inputs, and uses paraphrase detection as a constraint to keep the generated samples high quality.\"},{\"question\":\"What is the impact of using the generated adversarial samples for training?\",\"answer\":\"Including these adversarial samples improves robustness of the NLU model and can recover up to 70% of the errors caused by problematic input types, especially improving tail accuracy.\"}]","Semi-supervised Adversarial Text Generation based on Seq2Seq models | PDF",23]