[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-205060-105":53,"doc-detail-205060-en":127},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":120,"head_meta":122,"extra_data":124,"updated_unix":126},105,"en","a-neural-probabilistic-language-model","A Neural Probabilistic Language Model","","A goal of statistical language modeling is to learn the joint probability of word sequences, a task made intrinsically difficult by the curse of dimensionality: test sequences differ from those seen in training. The work addresses this by learning distributed word representations so each training sentence informs the model about exponentially many semantically related sentences. The probability model is trained jointly with the representations, enabling unseen sequences to receive high probability when composed of words similar to those in known contexts, while leveraging longer contexts.",{"@graph":63,"@context":119},[64,80,102],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":11,"@type":70,"position":76},"https://docshare.wps.com/template/presentations/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/a-neural-probabilistic-language-model/205060/",4,{"url":78,"name":59,"@type":81,"image":82,"author":87,"headline":59,"publisher":90,"fileFormat":93,"inLanguage":57,"description":61,"dateModified":94,"datePublished":95,"encodingFormat":93,"isAccessibleForFree":96,"interactionStatistic":97},"DigitalDocument",{"url":83,"@type":84,"width":85,"height":86},"https://docshare.wps.com/thumbnails/a-neural-probabilistic-language-model/205060.png","ImageObject",442,249,{"name":88,"@type":89},"Sage","Person",{"url":68,"name":91,"@type":92},"DocShare","Organization","application/pdf","2026-10-23","2026-09-05",true,{"@type":98,"interactionType":99,"userInteractionCount":101},"InteractionCounter",{"@type":100},"ViewAction",9,{"@type":103,"mainEntity":104},"FAQPage",[105,111,115],{"name":106,"@type":107,"acceptedAnswer":108},"Why is statistical language modeling difficult?","Question",{"text":109,"@type":110},"It is hard because learning the joint probability of word sequences suffers from the curse of dimensionality: test sequences are likely different from training sequences.","Answer",{"name":112,"@type":107,"acceptedAnswer":113},"How does the proposed model improve generalization over n-gram models?",{"text":114,"@type":110},"It learns distributed word representations so a training sentence influences the model about an exponential number of semantically neighboring sentences, assigning high probability to unseen sequences built from similar words.",{"name":116,"@type":107,"acceptedAnswer":117},"What challenge arises when training large neural language models?",{"text":118,"@type":110},"Training models with millions of parameters within a reasonable time is itself a significant challenge.","https://schema.org",{"og:url":78,"og:type":121,"og:title":59,"og:site_name":91,"og:description":61},"article",{"robots":123,"canonical":78},"index,follow",{"doc_id":125,"site_id":56},205060,1788577351,{"code":4,"msg":5,"data":128},{"doc_id":125,"user_id":129,"nickname":88,"user_avatar":130,"doc_module":9,"category_id":8,"category_name":11,"doc_title":59,"doc_description":61,"doc_content":131,"file_id":132,"file_url":133,"file_type":134,"file_size":135,"view_count":101,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":136,"language":137,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":138,"faqs":139,"seo_title":140,"seo_description":61,"update_tm":126,"read_time":141},687197207057,"https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0","A Neural Probabilistic Language Model  \nYoshua Bengio BENGIOY @IRO . UMONTREAL . CA  \nRéjean Ducharme DUCHARME @IRO . UMONTREAL . CA  \nPascal Vincent VINCENTP @IRO . UMONTREAL . CA  \nChristian Jauvin JAUVINC @IRO . UMONTREAL . CA  \nDépartement d’Informatique et Recherche Opérationnelle Centre de Recherche Mathématiques  \nUniversité de Montréal, Montréal, Québec, Canada  \nEditors: Jaz Kandola, Thomas Hofmann, Tomaso Poggio and John Shawe-Taylor  \nAbstract  \nA goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations.  \nGeneralization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts.  \nKeywords: Statistical language modeling, artificial neural networks, distributed representation, curse of dimensionality  \n1. Introduction  \nA fundamental problem that makes language modeling and other learning problems difficult is the curse of dimensionality. It is particularly obvious in the case when one wants to model the joint distribution between many discrete random variables (such as words in a sentence, or discrete attributes in a data-mining task) . For example, if one wants to model the joint distribution of 10 consecutive words in a natural language with a vocabulary V of size 100,000, there are potentially 100 00010 􀀀 1 = 1050 􀀀 1 free parameters. When modeling continuous variables, we obtain generalization more easily (e.g. with smooth classes of functions like multi-layer neural networks or Gaussian mixture models) because the function to be learned can be expected to have some local smoothness properties. For discrete spaces, the generalization structure is not as obvious: any change of these discrete variables may have a drastic impact on the value of the function to be esti-  \n􀀍c2003 Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin.  \nBENGIO, DUCHARME, VINCENT AND JAUVIN  \nmated, and when the number of values that each discrete variable can take is large, most observed objects are almost maximally far from each other in hamming distance.  \nA useful way to visualize how different learning algorithms generalize, inspired from the view of non-parametric density estimation, is to think of how probability mass that is initially concentrated on the training points (e.g., training sentences) is distributed in a larger volume, usually in some form of neighborhood around the training points. In high dimensions, it is crucial to distribute probability mass where it matters rather than uniformly in all directions around each training point. We will show in this paper that the way in which the approach proposed here generalizes is fundamentally different from the way in which p","cbCaird4jDrcn7io","https://ap.wps.com/l/cbCaird4jDrcn7io","pdf",140095,19,"English","# Introduction\n## Curse of dimensionality in language modeling\n## Distributed word representations and generalization\n## Conditional probability and n-gram approaches\n## Smoothing/back-off for unseen word sequences","[{\"question\":\"Why is statistical language modeling difficult?\",\"answer\":\"It is hard because learning the joint probability of word sequences suffers from the curse of dimensionality: test sequences are likely different from training sequences.\"},{\"question\":\"How does the proposed model improve generalization over n-gram models?\",\"answer\":\"It learns distributed word representations so a training sentence influences the model about an exponential number of semantically neighboring sentences, assigning high probability to unseen sequences built from similar words.\"},{\"question\":\"What challenge arises when training large neural language models?\",\"answer\":\"Training models with millions of parameters within a reasonable time is itself a significant challenge.\"}]","A Neural Probabilistic Language Model | PDF",7]