[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84659-en":3,"doc-seo-84659-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84659,4810365810221,"Aurora","https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d",8,"Research & Report","Fine-Grained Computation Offload for Off-the-Shelf Servers in Tens of Lines","Hardware accelerators increasingly define online serving latency, ranging from GPUs and FPGAs to remote services such as hardware security modules, post-quantum KEMs, and inference servers. For fine-grained offloads lasting microseconds to a few milliseconds, conventional blocking and busy-waiting fail because context-switch cost matches the offload and busy-wait wastes cores. The work treats overlap as routing: submit the offload to an executor, suspend the request via the server’s deferred-response primitive, and resume on completion. Experiments across ten production servers show 1.2–5.4× recovery with only small code changes and bounded safety hazards.","arXiv :2607 .02630v 1 [ cs .DC] 2 Jul 2026  \nFine-Grained Computation Offload for Off-the-Shelf Servers in Tens of Lines  \nBojie Li  \nPine AI  \nAbstract  \nHardware accelerators now sit on the critical path of online serving—GPUs, FPGAs, and increasingly remote services such as hardware security modules, post-quantum KEMs, and inference servers. For fine-grained offloads (microseconds to a few milliseconds) the classic responses to the resulting stall both fail: a context switch costs as much as the offload, and a busy-wait burns the core. Overlapping the offload with other requests is the fix, and prior systems obtain it by adding concurrency: an async-framework rewrite, a new runtime or dataplane OS, or a hand-tuned point integration.  \nWe observe that the concurrency already exists: serving concurrent requests is suspending and resuming them, so every server ships the machinery overlap needs. Overlap is then a routing problem, not a rewrite problem: submit the offload to an executor, suspend the request with the server’s own deferred-response primitive, resume it on completion. Across ten off-the-shelf servers spanning every production concurrency model, this recipe takes 22–138 lines added, at most one modified, and recovers 1.2–5.4 × on real hardware; the server’s concurrency model and the offload’s weight predict both numbers in advance, and the win is bounded by device throughput and the server’sown overlap capacity. At the limit, an LD_PRELOAD fiber runtime injects the reroute into an unmodified thread-per-connection binary (17.3 ×) within a characterized envelope. Rerouting suspends run-to-completion atomicity; a measured taxonomy confines the hazard to unlocked shared aggregates, and a transparent page-protection detector guards exactly those, validated on stock Redis.  \nCode: [https://github.com/19PINE-AI/transparent-offload](https://github.com/19PINE-AI/transparent-offload)[ ](https://github.com/19PINE-AI/transparent-offload)Website: [https://01.me/research/transparent-offload](https://01.me/research/transparent-offload)  \nApache Redis  \nGo memcached nginx MariaDB Postgres Node.js Python HAProxy  \n\n|  |  |  |  |  |  |  |  |  | 3.45× |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  |  |  |  |  |  |  | 3.01×\u003Cbr>3.01× 2.93×× | \u003Cbr> |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  | \u003Cbr> 2.74 2.59×\u003Cbr>2.59×\u003Cbr>2.54× |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  event loop\u003Cbr> loop + pool\u003Cbr> thread pool\u003Cbr> proxy\u003Cbr> per-conn DB |  |\n|  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  2 2.10× | .37× |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |  |  |  |  |\n\n0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0  \nspeedup over synchronous offload (×, real GPU AES)  \nFigure 1. The headline result. Rerouting the offload through each server’s own concurrency recovers 2.1–3.5 × across ten servers (real GPU AES, 1 MiB; up to 5.4 × at 8 MiB; the databases pipeline the op intra-query); zero-edit rerouting from outside the binary reaches 17.3 × (§4) . Color marks the concurrency model.  \nSynchronous offload: one request, CPU stalls  \nCPU idle (wasted)  \n\n| recv | pre | \u003Cbr>\u003Cbr>accelerator busy | post | send |\n| --- | --- | --- | --- | --- |\n\nOverlapped offload: CPU serves B, C while A's offload is in flight  \n\n| A:recv | A:pre | \u003Cbr>\u003Cbr>A: offloaded (parked) | A:post |\n| --- | --- | --- | --- |\n\nFigure 2. The problem. A serving request is receive → pre-process → offload → post-process → send. Top: with a synchronous offload, the CPU stalls while the accelerator works. Bottom: overlap fills the gap with other requests’ processing.  \n1 Introduction  \nHardware accelerators are now a standard part of online serving. GPUs, TPUs, and FPGAs accelerate inference, image processing, encry","cbCaij00pyVp053E","https://ap.wps.com/l/cbCaij00pyVp053E","pdf",652725,1,21,"English","en",105,"# Abstract\n# Introduction","[{\"question\":\"Why do classic approaches like blocking or busy-waiting fail for fine-grained offloads?\",\"answer\":\"Blocking incurs context-switch overhead that is comparable to the offload latency, adding delay to the request tail. Busy-waiting hides wakeup delay but consumes a CPU core doing no useful work.\"},{\"question\":\"How does the proposed method achieve overlap without rewriting the entire server stack?\",\"answer\":\"It reuses the server’s existing concurrency mechanisms by treating overlap as routing: offload work is submitted to an executor, the request is suspended using the server’s deferred-response primitive, and the request is resumed when the offload completes.\"},{\"question\":\"What results are reported across real hardware and different server concurrency models?\",\"answer\":\"Across ten off-the-shelf servers, the approach adds only about 22–138 lines (with at most one modification) and recovers roughly 1.2–5.4× performance on real hardware, with predictions based on device throughput and overlap capacity. It can reach up to 17.3× using LD_PRELOAD fiber runtime injection within a characterized envelope.\"}]",1784197533,53,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"fine-grained-computation-offload-for-off-the-shelf-servers-in-tens-of-lines","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/fine-grained-computation-offload-for-off-the-shelf-servers-in-tens-of-lines/84659/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why do classic approaches like blocking or busy-waiting fail for fine-grained offloads?","Question",{"text":75,"@type":76},"Blocking incurs context-switch overhead that is comparable to the offload latency, adding delay to the request tail. Busy-waiting hides wakeup delay but consumes a CPU core doing no useful work.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the proposed method achieve overlap without rewriting the entire server stack?",{"text":80,"@type":76},"It reuses the server’s existing concurrency mechanisms by treating overlap as routing: offload work is submitted to an executor, the request is suspended using the server’s deferred-response primitive, and the request is resumed when the offload completes.",{"name":82,"@type":73,"acceptedAnswer":83},"What results are reported across real hardware and different server concurrency models?",{"text":84,"@type":76},"Across ten off-the-shelf servers, the approach adds only about 22–138 lines (with at most one modification) and recovers roughly 1.2–5.4× performance on real hardware, with predictions based on device throughput and overlap capacity. It can reach up to 17.3× using LD_PRELOAD fiber runtime injection within a characterized envelope.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]