[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-328192-105":59,"doc-detail-328192-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","riemannian-geometry-for-pre-trained-language-model-embeddings-328192","Riemannian Geometry for Pre-trained Language Model Embeddings","","Understanding the geometric structure of pretrained language model embeddings matters for interpretability and safety. The study tests whether sentence-level classification signals reside in the Riemannian geometry of contextual token embeddings by computing per-token pullback metrics from an encoder’s analytical Jacobian and aggregating them using Fréchet means on the symmetric positive definite manifold via Riemannian Mean Pooling (RMP). On CoLA, CREAK, and RTE, RMP improves over Euclidean mean pooling, while on FEVER-Symmetric it stays at chance. Ablations localize gains to geometric aggregation, with additional benefit from a trained encoder on CREAK.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/riemannian-geometry-for-pre-trained-language-model-embeddings-328192/328192/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/riemannian-geometry-for-pre-trained-language-model-embeddings-328192/328192.png","ImageObject",300,407,{"name":92,"@type":93},"Aurelia","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-11-01","2026-09-21",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",16,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"What is the main goal of Riemannian Mean Pooling (RMP)?","Question",{"text":113,"@type":114},"RMP tests whether sentence-level classification signals are captured by the Riemannian geometry of contextual token embeddings rather than by flat Euclidean aggregation.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"How does RMP construct its geometric representation?",{"text":118,"@type":114},"It extracts per-token pullback metrics from an encoder’s analytical Jacobian, aggregates the metrics on the symmetric positive definite (SPD) manifold using a Fréchet mean, and classifies in the tangent space at the population Riemannian mean.",{"name":120,"@type":111,"acceptedAnswer":121},"What do the results show on FEVER-Symmetric compared with other datasets?",{"text":122,"@type":114},"RMP outperforms Euclidean mean pooling on CoLA, CREAK, and RTE, but it stays at chance on FEVER-Symmetric, which is designed to remove lexical and annotation artifacts.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},328192,1790197955,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":140,"language":141,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":142,"faqs":143,"seo_title":144,"seo_description":67,"update_tm":145,"read_time":146,"preview_page_count":4},1099514068365,"https://ap-avatar.wpscdn.com/avatar/10000253d8d9f28188e?x-image-process=image/resize,m_fixed,w_180,h_180&k=1794309523707217728","Riemannian Geometry for Pre-trained Language Model Embeddings  \nSzczepan Konior 1 Alexandre Quemy2 Przemysław Klocek 1  \nGrégoire Cattan 1 Bartłomiej Sobieski3 ,4  \n1IBM Automation and AI, Krakow, Poland 2Hother, Krakow, Poland  \n3University of Warsaw 4 Centre for Credible AI, Warsaw University of Technology {[name.surname}@ibm.com](name.surname}@ibm.com)  \n[alexandre@hother.io](alexandre@hother.io)  \n[sobieski.bartlomiej.jan@gmail.com](sobieski.bartlomiej.jan@gmail.com)  \narXiv :2607 .07047v 1 [ cs .CL] 8 Jul 2026  \nAbstract  \nUnderstanding the geometric structure of pretrained language model embeddings matters for interpretability and safety. We ask whether sentence-level classification signal lives in the Riemannian geometry of contextual token embeddings, and probe it by extracting pertoken pullback metrics from a learned encoder’s analytical Jacobian and aggregating them with the Fréchet mean on the symmetric positive definite (SPD) manifold; we call this procedure Riemannian Mean Pooling (RMP) . Across three datasets with non-trivial linguistic structure (CoLA, CREAK, RTE), RMP outperforms Euclidean mean pooling, while on FEVER-Symmetric, a benchmark constructed to remove annotation-driven lexical artifacts, the method correctly stays at chance. Ablations show that a randomly initialised encoder combined with Fréchet aggregation already beats Euclidean pooling on two of the three signal-bearing datasets, localising the source of the gain to the geometric aggregation rather than to learned manifold structure; the trained encoder contributes additional signal specifically on CREAK, the most knowledge-heavy of the three signal-bearing datasets.  \n1 Introduction  \nPre-trained language models achieve remarkable success in natural language understanding, yet the geometric structure of their internal representations remains an open challenge. Most analyses treat token embeddings as points in flat Euclidean space, but three lines of evidence motivate departing from that assumption: hierarchical structure in language admits lower-distortion embeddings in negatively curved spaces (Nickel and Kiela, 2017, 2018 ; Sala et al., 2018 ; Park et al., 2025) and is reflected in non-standard inner products on the embedding space (Park et al., 2024 ; Marks and Tegmark, 2024); contextualized  \nrepresentations in BERT, ELMo, and GPT-2 are strongly anisotropic, occupying narrow cones not invariant under Euclidean transformation (Ethayarajh, 2019 ; Kataiwa et al., 2025); and token embedding spaces may fail to form smooth manifolds globally, even if local-manifold-like structure still supports geometric analysis at the neighbourhood level (Robinson et al., 2025) .  \nThis work investigates whether sentence-level classification signal lives in the Riemannian geometry of token embeddings. We address three questions: (1) Do embeddings exhibit local geometric structure accessible via pullback metrics?  \n(2) Does this geometric structure carry signal beyond what flat Euclidean aggregation extracts?  \n(3) Which component of a geometric aggregation pipeline is responsible for any observed gain: the encoder, the metric, or the aggregation?  \nWe evaluate our approach, Riemannian Mean Pooling (RMP), on four diverse NLP tasks: fact verification (FEVER-Symmetric), textual entailment (RTE), grammatical acceptability (CoLA), and commonsense reasoning (CREAK) . FEVERSymmetric is included specifically as a negative control: it was constructed to remove lexical and annotation artifacts (Schuster et al., 2019), and we expect any method relying solely on claim-surface geometry to perform at chance.  \nAny differentiable encoder yields a pullback metric through its Jacobian, but the metric’s geometric interpretability depends on the training objective. We use Intrinsic Green’s Learning (IGL) (Quemy, 2026)1 : a closed-form kernel readout replaces the learned decoder, constraining the encoder to coordinates aligned with the smooth structure of the data manifold rather t","cbCaid075GVeUO8h","https://ap.wps.com/l/cbCaid075GVeUO8h","pdf",892682,15,"English","# Abstract\n# 1 Introduction\n# Contributions\n# 2 Related work","[{\"question\":\"What is the main goal of Riemannian Mean Pooling (RMP)?\",\"answer\":\"RMP tests whether sentence-level classification signals are captured by the Riemannian geometry of contextual token embeddings rather than by flat Euclidean aggregation.\"},{\"question\":\"How does RMP construct its geometric representation?\",\"answer\":\"It extracts per-token pullback metrics from an encoder’s analytical Jacobian, aggregates the metrics on the symmetric positive definite (SPD) manifold using a Fréchet mean, and classifies in the tangent space at the population Riemannian mean.\"},{\"question\":\"What do the results show on FEVER-Symmetric compared with other datasets?\",\"answer\":\"RMP outperforms Euclidean mean pooling on CoLA, CREAK, and RTE, but it stays at chance on FEVER-Symmetric, which is designed to remove lexical and annotation artifacts.\"}]","Riemannian Geometry for Pre-trained Language Model Embeddings | PDF",1789978810,38]