[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-327299-105":59,"doc-detail-327299-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","token-geometry-read-online-free","Token Geometry - read online free","","Token Geometry studies how language models should optimize continuous programs represented by discrete token symbols, focusing on the embedding table and the LM-head as a distinct read/write interface. The work reveals that this interface follows gradient geometry different from dense hidden weights, enabling improvements to the Pareto frontier across supervised finetuning, reinforcement learning, and pretraining with minimal optimizer-state cost. Ember is introduced as a lightweight optimizer that matches or improves SOTA while scaling efficiently and simplifying distributed training integration.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/token-geometry-read-online-free/327299/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/token-geometry-read-online-free/327299.png","ImageObject",300,407,{"name":92,"@type":93},"Olivia Brown","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-29","2026-09-21",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",16,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"What is the main focus of Token Geometry?","Question",{"text":113,"@type":114},"It focuses on the embedding table and LM-head parameters, treating them as a “token interface” and studying their distinct gradient geometry for better optimization.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"What is Ember, and what problem does it address?",{"text":118,"@type":114},"Ember is a lightweight optimizer designed specifically for embedding and LM-head matrices, aiming to reduce optimizer-state overhead and improve training efficiency versus Adam-style methods.",{"name":120,"@type":111,"acceptedAnswer":121},"How does the paper characterize the optimization trajectory of tokens?",{"text":122,"@type":114},"It argues the optimization trajectory of tokens can be well described by a simple 1D ray, challenging the idea that token-related updates require navigating a highly nonconvex parameter landscape.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},327299,1790086988,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":145,"read_time":146,"preview_page_count":4},16904993612988,"https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd","arXiv :2607 .0 1455v2 [ cs .LG] 7 Jul 2026  \nToken Geometry  \nKathan Shah  \nAbstract  \nLanguage models learn continuous programs over discrete symbols, with the embedding table and LM-head acting as the read/write interface between them. We show that this interface has gradient geometry distinct from dense hidden weights which can be exploited to improve the Pareto frontier across supervised finetuning, RL, and pretraining, while only utilizing kilobytes of optimizer state. We introduce Ember, a lightweight optimizer for embedding and LM-head matrices that utilizes O (V + D) VRAM, instead of Adam’s O(2VD), and forgoes the need to shard both token table optimizer states. We provide empirical evidence that Ember scales effectively across batch size and parameter count. We show that the optimization trajectory of tokens can be well described by a simple 1D ray, counter to the popular belief that neural net parameters navigate a heavily nonconvex landscape. We provide a principled view on the surprisingly narrow space of optimizers that suffice for Transformer training. Finally, we open-source our distributed Ember implementation that merges cleanly with existing ZeRO/FSDP setups to support further research, available [at](at github.com/katop1234/ember)[ github.com/katop1234/ember](at github.com/katop1234/ember).  \nFigure 1: The Ember optimizer utilizes almost no optimizer state for the embedding table and LM-head, and universally matches or improves upon current SOTA. In the sparse gradients/low-batch regime (Fig 1B) Ember outperforms Adam by a wide margin, and in fine-tuning + RL (Fig 1C,1D), it achieves within-seed parity with Adam while doing better at most recorded checkpoints.  \n1 Introduction  \nEmbedding and LM-head matrices, or “token interface” parameters, are currently optimized with Adam [1] . This is corroborated by a report from the current open source frontier, DeepSeek V4 [14], which only changed the optimizer for linear layers to Muon [15] . Of note, in literature, almost all proposed successors of Adam solely apply to linear layers and steer clear of the embedding table. Therefore improvements are pertinent as Adam requires significant distributed training engineering and doesn’t fully leverage the geometry of token interfaces. In this paper, we introduce Ember, a lightweight optimizer for such Transformer parameters that remedies these and achieves competitive results. We begin with some preliminaries on information geometry:  \nPreprint.  \nThe Fisher metric. Under regularity assumptions, cross-entropy loss possesses a remarkable property: the second derivative for a given parameter is just the square of its first derivative,  \n−Epθ 􀀔 22i log pθ (x)􀀕 = Epθ \" 􀀒 ∂i logpθ (x)􀀓 2\\# .  \nalso known as the Fisher Information metric [8, 9, 10], which near the optimum coincides with the Hessian (Appendix 6.1) . Although the standard optimization baseline is Newton’s method, which conditions each gradient with the inverse of its second derivative, standard backprop is not amenable to returning second derivatives cheaply. Fortunately, the Fisher, or second derivative, can be estimated cheaply using only the first derivative’s information as described above.  \nCanonical Geometry of Cross-Entropy. The Fisher metric is especially elegant because it is not merely useful as an approximation, but it is the exact second-order term of the Kullback–Leibler divergence:  \nDKL (pθ ∥ pθ+dθ) = 12dθ ⊤ F (θ)dθ + O(∥dθ∥3 ) .  \nBy Chentsov’s theorem [6], for the KLD there exist no other canonical local geometry besides the Fisher. Therefore, optimization beyond SGD for cross-entropy loss must utilize this quantity; this is well-established as “natural gradient descent” [7, 2, 11, 12] .  \nThere are a few caveats, which are easily remedied. First, the Fisher, being a second order correction, assumes an exact quadratic loss landscape. Second, it is numerically unstable under batch gradient descent, as at any given training step we only observe e","cbCaibZytYtv91tM","https://ap.wps.com/l/cbCaibZytYtv91tM","pdf",2173165,17,"English","# Abstract\n# 1 Introduction\n## 1.1 Related work\n### 1.1.1 Square-root Fisher metric","[{\"question\":\"What is the main focus of Token Geometry?\",\"answer\":\"It focuses on the embedding table and LM-head parameters, treating them as a “token interface” and studying their distinct gradient geometry for better optimization.\"},{\"question\":\"What is Ember, and what problem does it address?\",\"answer\":\"Ember is a lightweight optimizer designed specifically for embedding and LM-head matrices, aiming to reduce optimizer-state overhead and improve training efficiency versus Adam-style methods.\"},{\"question\":\"How does the paper characterize the optimization trajectory of tokens?\",\"answer\":\"It argues the optimization trajectory of tokens can be well described by a simple 1D ray, challenging the idea that token-related updates require navigating a highly nonconvex parameter landscape.\"}]","Token Geometry - read online free | PDF",1789976934,43]