[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-238886-105":53,"doc-detail-238886-en":127},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":120,"head_meta":122,"extra_data":124,"updated_unix":126},105,"en","dialog-system-using-real-time-crowdsourcing-and-twitter-large-scale-corpus","Dialog System Using Real-Time Crowdsourcing and Twitter Large-Scale Corpus","","A dialog system is proposed that generates responses by retrieving the most similar utterance-reply pair from a large-scale dialog corpus built from Twitter data and real-time crowdsourcing. Instead of complex dialog management, the system directly replies with database utterances whose similarity matches user input. When no adequate response exists, a realtime crowdsourcing framework delegates the query to other users, addressing low-similarity cases through crowd-based handling.",{"@graph":63,"@context":119},[64,80,102],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":11,"@type":70,"position":76},"https://docshare.wps.com/template/presentations/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/dialog-system-using-real-time-crowdsourcing-and-twitter-large-scale-corpus/238886/",4,{"url":78,"name":59,"@type":81,"image":82,"author":87,"headline":59,"publisher":90,"fileFormat":93,"inLanguage":57,"description":61,"dateModified":94,"datePublished":95,"encodingFormat":93,"isAccessibleForFree":96,"interactionStatistic":97},"DigitalDocument",{"url":83,"@type":84,"width":85,"height":86},"https://docshare.wps.com/thumbnails/dialog-system-using-real-time-crowdsourcing-and-twitter-large-scale-corpus/238886.png","ImageObject",442,249,{"name":88,"@type":89},"Caleb Sterling","Person",{"url":68,"name":91,"@type":92},"DocShare","Organization","application/pdf","2026-11-03","2026-09-11",true,{"@type":98,"interactionType":99,"userInteractionCount":101},"InteractionCounter",{"@type":100},"ViewAction",8,{"@type":103,"mainEntity":104},"FAQPage",[105,111,115],{"name":106,"@type":107,"acceptedAnswer":108},"How does the dialog system generate responses?","Question",{"text":109,"@type":110},"For each user input, it searches an utterance-pair database and selects the reply associated with the utterance most similar to the input.","Answer",{"name":112,"@type":107,"acceptedAnswer":113},"What happens when the system cannot find a sufficiently similar response?",{"text":114,"@type":110},"It outsources the response to other users via a real-time crowdsourcing mechanism when the similarity score falls below a threshold.",{"name":116,"@type":107,"acceptedAnswer":117},"How is similarity between user input and database utterances computed?",{"text":118,"@type":110},"Tweets are preprocessed and analyzed for parts of speech, converted into document vectors using term frequency, and compared using the inner product similarity (xTa x b).","https://schema.org",{"og:url":78,"og:type":121,"og:title":59,"og:site_name":91,"og:description":61},"article",{"robots":123,"canonical":78},"index,follow",{"doc_id":125,"site_id":56},238886,1791371507,{"code":4,"msg":5,"data":128},{"doc_id":125,"user_id":129,"nickname":88,"user_avatar":130,"doc_module":9,"category_id":8,"category_name":11,"doc_title":59,"doc_description":61,"doc_content":131,"file_id":132,"file_url":133,"file_type":134,"file_size":135,"view_count":101,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":47,"language":136,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":137,"faqs":138,"seo_title":139,"seo_description":61,"update_tm":140,"read_time":73},962084925290,"https://ap-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45","Dialog System Using Real-Time Crowdsourcing and Twitter Large-Scale Corpus  \nFumihiro Bessho, Tatsuya Harada, Yasuo Kuniyoshi  \nThe University of Tokyo Department of Mechano-Informatics 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan {bessho, harada, [kuniyosh](kuniyosh}@isi.imi.i.u-tokyo.ac.jp)[}](kuniyosh}@isi.imi.i.u-tokyo.ac.jp)[@isi.imi.i.u-tokyo.ac.jp](kuniyosh}@isi.imi.i.u-tokyo.ac.jp)  \nAbstract  \nWe propose a dialog system that creates responses based on a large-scale dialog corpus retrieved from Twitter and real-time crowdsourcing. Instead of using complex dialog management, our system replies with the utterance from the database that is most similar to the user input. We also propose a realtime crowdsourcing framework for handling the case in which there is no adequate response in the database.  \n1 Introduction  \nThere is a lot of language data on the Internet. Twitter offers many APIs to retrieve or search post status data, and this data is frequently used in research, such as in stock market prediction (Bollen et al., 2011), the spread of information through social media (Bakshy and Hofman, 2011), and representations of textual content(Ramage et al., 2010) . Several models for conversation using Twitter data (Ritter et al., 2010; Higashinaka et al., 2011) have been proposed because of the data’s vast size and conversational nature.  \nKelly (2009) previously showed that 37% of English tweets are Conversational, of which 69% are two-length (one status post and a reply) . In our analysis of over 2.5 million tweets, 37.5% of all Japanese tweets are Conversational, which matches Kelly’s data. However, less than 58 .3% of these are twolength tweets.  \nMany chat bots are rule-based, which requires alot of human effort to create or add new rules. For example, A.L.I.C.E (Wallace, 2009), which won the  \nU User Utterance  \nFigure 1: Utterance pair.  \nLoebner Prize three times, creates responses based on a dialog strategy database written in a markup language named AIML. Recently, some other chat bots based on a large-scale dialog corpus have been proposed 1,2 .  \nIn this paper, we propose a novel dialog system (chat bot) that uses real-time crowdsourcing and Twitter large-scale corpus. We evaluate response selection methods based on positive/negative example to judge if each feature could be exploited to judge similarity between uterrances.  \n2 Method  \n2.1 Overview  \nWe create an ”Utterance Pair” database as shown in Figure 1 . Each pair is composed of an utterance (Figure 1, A) and a reply to the utterance (Figure 1, B) . Our approach for creating responses is simple and is illustrated in Figure 2 . For each user input, the system searches the utterance-pair database for the pair of which the tweet (Figure 1, A) is most similar to that input. The reply contained in this pair (Figure 1, B) forms the system’s response to the user’s input.  \n1Jabberwacky: [http://www.jabberwacky.com](http://www.jabberwacky.com)  \n2Cleverbot: [http://www.cleverbot.com](http://www.cleverbot.com)  \n227  \nProceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL) , pages 227–231,  \nSeoul, South Korea, 5-6 July 2012 . 􀀍c2012 Association for Computational Linguistics  \nFigure 2: System overview.  \nIf the system cannot find a post that is sufficiently similar to the user’s input, then it ”outsources” the response to another user.  \nTo build the conversation database, we collected 1.2 million utterance-pairs from the microblogging service, Twitter. We fetched public timeline data using the Streaming API 3 , and then looked for tweets which were written in Japanese 4 and had an in-reply-to field. We followed the replies using the REST API 5.  \nRaw posts from Twitter included mentions (the symbol @ followed by a user name), quotes (written with letters ”RT”), hashtags (a word preceded by the symbol \\#), and URLs. We filtered away this information using regular expressions. Unlike English tweets, in Japanese users","cbCaicHqCbd021Sb","https://ap.wps.com/l/cbCaicHqCbd021Sb","pdf",4341935,"English","# Abstract\n## Introduction\n## Method\n### Overview\n### Real-Time Crowdsourcing\n## Similar Utterance-Pairs Retrieval Method","[{\"question\":\"How does the dialog system generate responses?\",\"answer\":\"For each user input, it searches an utterance-pair database and selects the reply associated with the utterance most similar to the input.\"},{\"question\":\"What happens when the system cannot find a sufficiently similar response?\",\"answer\":\"It outsources the response to other users via a real-time crowdsourcing mechanism when the similarity score falls below a threshold.\"},{\"question\":\"How is similarity between user input and database utterances computed?\",\"answer\":\"Tweets are preprocessed and analyzed for parts of speech, converted into document vectors using term frequency, and compared using the inner product similarity (xTa x b).\"}]","Dialog System Using Real-Time Crowdsourcing and Twitter Large-Scale Corpus | PDF",1789140667]