[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82667-en":3,"doc-seo-82667-105":29,"detail-sidebar-cat-0-en-105":83},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82667,1649267921044,"Ava Thompson","https://us-avatar.wpscdn.com/avatar/1800007509477c92dfb?_k=1782875107921204101",8,"Research & Report","WattGPU Predicting Inference Power and Latency on Unseen GPUs and LLMs","Large Language Model (LLM) inference workloads are driving rapidly increasing data center energy consumption, yet operators often lack tools to match specific LLMs to the most efficient GPUs without exhaustive profiling. WattGPU introduces two predictive models for mean GPU power draw and inter-token latency (ITL) that use only publicly available LLM metadata and GPU specifications. Leave-one-GPU-out and leave-one-LLM-out evaluations on 42 open-source LLMs and 8 GPUs show strong generalization to unseen hardware. Power errors are ≤3.4% offline and ≤13.5% on servers, while latency error is ≤8.5% in server mode.","WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs  \nMauricio Fadel Argerich 1, * , Jonathan Fürst2 and Marta Patiño-Martínez1  \n1 Universidad Politécnica de Madrid, C. de los Ciruelos, 28660 Boadilla del Monte, Madrid, Spain 2 Zurich University of Applied Sciences, Gertrudstrasse 15, 8400 Winterthur, Switzerland  \nAbstract  \nLarge Language Model (LLM) inference workloads are a rapidly growing contributor to data center energy consumption. Optimizing these deployments requires matching specific LLMs to the most efficient GPUs, but operators currently lack the tools to do so without exhaustively profiling each combination. While some predictive models exist, they still require profiling data and struggle to generalize to hardware unseen during training. To address this, we introduce WattGPU, featuring two predictive models for mean GPU power draw and Inter-Token Latency (ITL). Our approach leverages only publicly available LLM metadata and GPU specifications, eliminating the need for hardware access or profiling while enabling generalization to unseen NVIDIA server-grade GPUsand LLMs. We evaluate our models using rigorous leave-one-GPU-out and leave-one-LLM-out cross-validation on a dataset of 42 open-source LLMs (0.1B–27B parameters) and 8 GPUs under both offline and server scenarios. The mean power draw model achieves a median absolute percentage error of ≤ 3.4% for offline and ≤ 13.5% for server scenarios on unseen GPUs, while the latency model achieves ≤ 8.5% in server mode, both maintaining strong GPU ranking correlations for server scenarios (Kendall 􀀜 ≥ 0.76) . Compared to standard physically grounded baselines —Load-Scaled Thermal Design Power (TDP) for power draw and roofline for latency— our models reduce median absolute percentage error by approximately 4× on unseen LLM-GPU combinations for server scenarios or approximately 2 × for completely unseen GPUs. WattGPU’s data and code are publicly available at [https://github.com/maufadel/wattgpu](https://github.com/maufadel/wattgpu).  \nKeywords  \nSustainable AI, Large Language Models Energy Prediction, LLM Energy Estimation, LLM inference  \n1. Introduction  \nThe growing use of Large Language Models (LLMs) in user and enterprise applications, combined with the increasing number and quality of open-source models, is driving the proliferation of LLM inference servers in cloud deployments. This trend is further reinforced by user and business demand for data privacy and sovereignty.  \nWhile attention typically focuses on hyperscalers and major AI companies (e.g., OpenAI, AWS, Google) that consume gigawatts of energy to process billions of daily requests, small-to medium-scale deployments are rapidly growing. These smaller deployments are often less optimized, and their energy consumption remains poorly characterized. Consequently, even though their individual energy usage is lower, their aggregate energy footprint is growing alarmingly fast [1] . This inefficiency is particularly pronounced in serving workloads; recent measurements reveal that even within the online deployments of industry leaders, GPUs spend 14–76% of their time and 7–65% of their energy in execution-idle states [2] . This hints at massive potential savings if deployment efficiency can be improved.  \nA critical lever for optimizing energy consumption in LLM inference is optimal GPU selection. GPUs dominate the energy consumption of LLM inference [3], and careful LLM-GPU matching yields substantial efficiency gains. For instance, deploying a medium-sized model like Llama 3.1 8B on an NVIDIA A30 can reduce power draw by up to 43% in low-load scenarios compared to an H100 (a common high-end GPU), as illustrated in Figure 1. Achieving these savings not only reduces operational  \nSuRE’26: Workshop on Sustainability and Resource-Efficiency of Artificial Intelligence, August 17, 2024, Bremen, Germany * Corresponding author.  \n$ [mauricio.fadel@alumnos.upm.es](mauricio.fadel@alumnos.upm.es) (M. ","cbCaigOiOD0Or2Ur","https://ap.wps.com/l/cbCaigOiOD0Or2Ur","pdf",974207,1,12,"English","en",105,"# Introduction\n## Problem: Efficient LLM inference hardware matching\n## Approach: WattGPU predictive models\n## Contributions and evaluation setup","[{\"question\":\"How is the method evaluated for generalization to unseen GPUs and LLMs?\",\"answer\":\"Evaluation uses leave-one-GPU-out and leave-one-LLM-out cross-validation on a dataset of 42 open-source LLMs and 8 GPUs, under both offline and server scenarios.\"}]",1784182167,30,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":78,"head_meta":80,"extra_data":82,"updated_unix":27},"wattgpu-predicting-inference-power-and-latency-on-unseen-gpus-and-llms","",{"@graph":35,"@context":77},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/wattgpu-predicting-inference-power-and-latency-on-unseen-gpus-and-llms/82667/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71],{"name":72,"@type":73,"acceptedAnswer":74},"How is the method evaluated for generalization to unseen GPUs and LLMs?","Question",{"text":75,"@type":76},"Evaluation uses leave-one-GPU-out and leave-one-LLM-out cross-validation on a dataset of 42 open-source LLMs and 8 GPUs, under both offline and server scenarios.","Answer","https://schema.org",{"og:url":51,"og:type":79,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":81,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":84},[85,89,93,97,102,107,112,114,119,122,126],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":86,"show_sort_weight":87,"slug":88},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":90,"show_sort_weight":91,"slug":92},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Exam",70,"exam",{"id":98,"doc_module":4,"doc_module_name":45,"category_name":99,"show_sort_weight":100,"slug":101},5,"Comic",60,"comic",{"id":103,"doc_module":4,"doc_module_name":45,"category_name":104,"show_sort_weight":105,"slug":106},6,"Technology",50,"technology",{"id":108,"doc_module":4,"doc_module_name":45,"category_name":109,"show_sort_weight":110,"slug":111},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":28,"slug":113},"research-report",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},9,"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":45,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":98,"slug":129},19,"General","general"]