[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-81879-en":3,"doc-seo-81879-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},81879,2336464648322,"Aria","https://ap-avatar.wpscdn.com/avatar/2200025388227c56fec?_k=1778556882303663488",6,"Technology","ELiTeFormer An Efficient Transformer for FPGAs","Transformer blocks dominate large language model deployment but face high computational and memory costs. ELiTeFormer (Efficient Linear Ternary Transformer) unifies hybrid linear attention with ultra-low-precision ternary linear projections through FPGA-oriented hardware–algorithm co-design. It reaches 10× model weight compression and 12.8× KV cache compression versus LLaMA 3 while keeping competitive accuracy. A new processing element micro-architecture removes multiplications via bitmasking, avoiding dedicated DSP usage, and delivers simulated and deployed speedup and efficiency gains on FPGA and GPU.","ELiTeFormer: An Efficient Transformer for FPGAs  \nVictor Agostinelli∗ Oregon State University Corvallis, Oregon, USA Pacific Northwest National Laboratory  \nRichland, Washington, USA [agostiniv@oregonstate.edu](agostiniv@oregonstate.edu)  \nNicolas Bohm Agostini Pacific Northwest National  \nLaboratory Richland, Washington, USA [nicolas.agostini@pnnl.gov](nicolas.agostini@pnnl.gov)  \nAntonino Tumeo Pacific Northwest National  \nLaboratory Richland, Washington, USA [antonino.tumeo@pnnl.gov](antonino.tumeo@pnnl.gov)  \narXiv :2607 .03652v 1 [ cs .AR] 4 Jul 2026  \nAbstract  \nTransformer blocks are prevalent in large language model (LLM) but present deployment challenges due to their challenging computational and memory demands. While prior work has typically optimized attention mechanisms or feed-forward networks (FFNs) separately, few hardware (HW) architecture have jointly addressed both components with co-designed hardware acceleration. We present ELiTeFormer (Efficient Linear Ternary Transformer), the first Transformer model architecture that unifies hybrid linear attention with ultra-low-precision (ternary) linear projections, specifically co-designed for field-programmable gate array (FPGA) deployment. ELiTeFormer achieves 10× model weight compression and 12 . 8× key-value (KV) cache compression compared to LLaMA 3, while maintaining competitive accuracy (31.9% on the MMLU benchmark, within 3.0% of BitNet b1.58) . Our key architectural contribution is a novel processing element (PE) micro-architecture that eliminates all multiplications in ternary linear projections through bitmasking operations, significantly reducing resource utilization by completely avoiding dedicated digital signal processing (DSP) blocks. We simulate, synthesize, and deploy ELiTeFormer targeting a Xilinx VCK5000 Versal board using high-level synthesis (HLS) flows. Block-level simulations show 9. 6× speedup for FFN operations and 4.4× speedup for attention compared to standard implementations. End-to-end deployment achieves up to 3. 9× lower latency and 3. 2× better energy efficiency than LLaMA 3 on an NVIDIA A100 graphics processing unit (GPU) at long context lengths. This represents the first FPGA realization combining linear attention with ternary quantization, demonstrating the viability of algorithm-architecture co-design for next-generation LLM acceleration.  \n1 Introduction  \nTransformer-based large language models (LLMs) have seen explosive growth in both industry and academia, with rapid progress in model capabilities and widespread adoption across domains. LLM use spans a diverse range of applications, including question/answering and general conversational artificial intelligence (AI) [33], assisting in image generation [31], code generation and analysis [24], basic sentiment analysis [60], and more. As these models scale in size and complexity, delivering efficient compute and memory solutions has become a significant challenge. This is particularly true in scenarios demanding low latency and energy efficiency, such as real-time inference and edge deployment. Even  \n∗ Currently at Meta.  \nin datacenter settings, reducing the cost and energy footprint of inference at scale remains critical.  \nOn the algorithmic side, researchers have proposed several alternatives to address inefficiencies. These efforts typically focus on optimizing individual components such as attention or feedforward network (FFN) blocks. Efficient attention mechanisms aim to subquadratic scaling with context length, including sparse attention [8, 45, 56] and linear attention [1, 2, 26, 54]. FFN efficiency has been pursued via reduced-precision arithmetic [11, 32, 48] and fine-grained mixture-of-experts approaches [6] . However, these optimizations are often explored in isolation. To the best of our knowledge, no prior work has jointly optimized both the attention and FFN blocks with co-designed hardware architectures tailored to exploit both algorithmic innovations simultane","cbCaibXkMnwQk4Wl","https://ap.wps.com/l/cbCaibXkMnwQk4Wl","pdf",1603501,1,13,"English","en",105,"# Introduction\n## Motivation and deployment challenges\n## Algorithmic efficiency approaches\n## FPGA hardware acceleration context\n## Proposed HW/SW co-design and ELiTeFormer overview","[{\"question\":\"What problem does ELiTeFormer target in Transformer deployment on FPGAs?\",\"answer\":\"Transformer blocks are expensive in compute and memory for deployment. ELiTeFormer targets these bottlenecks by co-designing an architecture that reduces both compute and memory costs specifically for FPGA inference.\"},{\"question\":\"How does ELiTeFormer combine efficient attention and ultra-low-precision projections?\",\"answer\":\"It unifies hybrid linear attention with ternary linear projections, producing an architecture co-designed to make both components efficient for FPGA execution.\"},{\"question\":\"What architectural mechanism removes multiplications in ternary linear projections?\",\"answer\":\"ELiTeFormer introduces a processing element (PE) micro-architecture that eliminates multiplications using bitmasking operations, avoiding dedicated DSP blocks and reducing resource utilization.\"}]",1784176829,33,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"eliteformer-an-efficient-transformer-for-fpgas","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/technology/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/eliteformer-an-efficient-transformer-for-fpgas/81879/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does ELiTeFormer target in Transformer deployment on FPGAs?","Question",{"text":75,"@type":76},"Transformer blocks are expensive in compute and memory for deployment. ELiTeFormer targets these bottlenecks by co-designing an architecture that reduces both compute and memory costs specifically for FPGA inference.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does ELiTeFormer combine efficient attention and ultra-low-precision projections?",{"text":80,"@type":76},"It unifies hybrid linear attention with ternary linear projections, producing an architecture co-designed to make both components efficient for FPGA execution.",{"name":82,"@type":73,"acceptedAnswer":83},"What architectural mechanism removes multiplications in ternary linear projections?",{"text":84,"@type":76},"ELiTeFormer introduces a processing element (PE) micro-architecture that eliminates multiplications using bitmasking operations, avoiding dedicated DSP blocks and reducing resource utilization.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,113,118,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":111,"slug":112},50,"technology",{"id":114,"doc_module":4,"doc_module_name":45,"category_name":115,"show_sort_weight":116,"slug":117},7,"Healthcare",40,"healthcare",{"id":119,"doc_module":4,"doc_module_name":45,"category_name":120,"show_sort_weight":121,"slug":122},8,"Research & Report",30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]