[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85786-en":3,"doc-seo-85786-105":29,"detail-sidebar-cat-0-en-105":83},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85786,8796095462418,"Noah","https://ap-avatar.wpscdn.com/avatar/80000253c1241d02b47?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778826106357471780",8,"Research & Report","Tokenizing Numerical and Embedding Features for LLM RecSys","Large language models increasingly serve as the backbone of recommender systems, yet most LLM recommenders consume only discrete text tokens while real pipelines depend on continuous numerical features and dense embedding features. This mismatch restricts the model’s ability to leverage fine-grained non-textual signals. The paper introduces a soft-token fusion framework that maps numerical and embedding features into the LLM embedding space. It is instantiated in a shared-parameter two-tower retrieval model with interaction-based fusion, yielding improved retrieval performance on three Amazon benchmarks and outperforming direct concatenation.","arXiv :2607 . 100 16v 1 [ cs .IR] 10 Jul 2026  \nTokenizing Numerical and Embedding Features for LLM RecSys  \nZhe Xu∗ Ankit Peshin Chiyu Zhang Feng Qi Johnson Lui  \nMeta Meta Meta Meta Meta  \nAnil Ramakrishna  \nMeta  \nJustin Johnson  \nMeta  \nCarl Hu  \nMeta  \nKaushik Rangadurai  \nMeta  \nLuke Simon  \nMeta  \nAbstract  \nLarge language models (LLMs) are increasingly used as backbone architectures for recommender systems because of their strong sequence modeling and representation learning capabilities. However, most LLM-based recommenders operate primarily on discrete textual tokens, whereas practical recommendation pipelines also rely on continuous numerical features and dense embedding features produced by upstream feature engineering or pretrained encoders. This mismatch limits the ability of LLM-based models to exploit fine-grained non-textual signals. We propose a soft-token fusion framework that maps numerical and embedding features into the LLM embedding space, allowing heterogeneous recommendation signals to be consumed through the standard token interface. We instantiate the framework in a shared-parameter LLM-based two-tower retrieval model and introduce an interaction-based fusion module that refines embedding and numerical soft tokens before they are inserted into the final LLM input. Experiments on three Amazon recommendation benchmarks show that soft-token fusion improves retrieval performance over LLM-based baselines, and that interaction-based fusion is more effective than direct concatenation of heterogeneous soft tokens.  \n1 Introduction  \nLarge language models (LLMs) have recently emerged as promising backbone models for recommender systems because of their sequence modeling, semantic understanding, and instruction-following capabilities. Recent studies reformulate recommendation as language processing (Geng et al., 2022), align LLMs with recommendation data through instruction tuning (Bao et al., 2023), and represent items with semantic identifiers for generative retrieval (Rajput et al., 2023b) . These developments indicate a broader shift from task-specific recommendation architectures toward LLM-centered models that consume token sequences and produce textual responses, item identifiers, or ranking signals.  \nDespite this progress, a key mismatch remains between the token-based input interface of LLMs and the heterogeneous feature representations used in modern recommender systems. Industrial recommendation pipelines commonly rely on categorical IDs, continuous numerical features, and dense embedding features, which are explicitly modeled by neural recommenders through embedding tables, multilayer perceptrons, feature interaction layers, and two-tower retrieval architectures (Cheng et al., 2016; Guo et al., 2017; Wang et al., 2017; Naumov et al., 2019; Covington et al., 2016; Yi et al., 2019) . Existing LLM-based recommenders often verbalize such signals as text, discretize item representations, or incorporate  \n∗ Contact: Zhe Xu, [zhexu@meta.com](zhexu@meta.com)  \nexternal features only after LLM encoding. These strategies can obscure scale-sensitive numerical information, discard fine-grained continuous signals, or limit the interaction between textual and non-textual features inside the LLM backbone.  \nThis paper studies how to incorporate numerical and embedding features into LLM-based recommender systems through tokenization in the continuous embedding space. Our key idea is to represent non-textual recommendation features as soft tokens. Inspired by continuous prompt learning, prefix tuning, and multimodal LLM adapters (Li & Liang, 2021; Lester et al., 2021; Li et al., 2023a), we map dense embedding features and continuous numerical features into soft token sequences that can be consumed by the backbone LLM together with ordinary textual tokens.  \nWe instantiate this idea in an LLM-based two-tower retrieval model. The user and item towers encode textual prompts, numerical features, and embedding features","cbCaioDiq8ewy5A0","https://ap.wps.com/l/cbCaioDiq8ewy5A0","pdf",278402,1,16,"English","en",105,"# Abstract\n# Introduction\n# Related Work\n## LLM-based Recommender Systems","[{\"question\":\"How is fusion implemented and what benefit does it bring?\",\"answer\":\"An interaction-based fusion module models relationships between embedding-derived and numerical soft tokens before they are inserted into the final LLM input, improving retrieval performance and outperforming direct concatenation on Amazon benchmarks.\"}]",1784206274,40,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":78,"head_meta":80,"extra_data":82,"updated_unix":27},"tokenizing-numerical-and-embedding-features-for-llm-recsys","",{"@graph":35,"@context":77},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/tokenizing-numerical-and-embedding-features-for-llm-recsys/85786/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71],{"name":72,"@type":73,"acceptedAnswer":74},"How is fusion implemented and what benefit does it bring?","Question",{"text":75,"@type":76},"An interaction-based fusion module models relationships between embedding-derived and numerical soft tokens before they are inserted into the final LLM input, improving retrieval performance and outperforming direct concatenation on Amazon benchmarks.","Answer","https://schema.org",{"og:url":51,"og:type":79,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":81,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":84},[85,89,93,97,102,107,111,114,119,122,126],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":86,"show_sort_weight":87,"slug":88},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":90,"show_sort_weight":91,"slug":92},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Exam",70,"exam",{"id":98,"doc_module":4,"doc_module_name":45,"category_name":99,"show_sort_weight":100,"slug":101},5,"Comic",60,"comic",{"id":103,"doc_module":4,"doc_module_name":45,"category_name":104,"show_sort_weight":105,"slug":106},6,"Technology",50,"technology",{"id":108,"doc_module":4,"doc_module_name":45,"category_name":109,"show_sort_weight":28,"slug":110},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":112,"slug":113},30,"research-report",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":45,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":98,"slug":129},19,"General","general"]