[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85324-en":3,"doc-seo-85324-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85324,1099514068365,"Aurelia","https://ap-avatar.wpscdn.com/avatar/10000253d8d9f28188e?_k=1776742907772140068",8,"Research & Report","Stage-Level Executor Allocation in Apache Spark with Cost–Performance Trade-offs","Stage-level executor allocation for distributed data processing must avoid paying for unnecessary scaling-out while also preventing performance bottlenecks caused by under-allocation. Naive application-level approaches offer predictable cost but are consistently sub-optimal across the many heterogeneous stages in a query, and they rarely support user-defined time-and-cost preferences. A new method is proposed for serverless Apache Spark that trains tree-ensemble models to predict per-stage runtime and cost from allocated resources, then recommends executors per stage. Evaluation on TPC-DS and SQLStorm shows about 50% cost savings with ~16% slowdown and ~40.5% cost reduction with ~29% slowdown.","Stage-Level Executor Allocation in Apache Spark™ with Cost–Performance Trade-offs  \nMiriam Rateike  \n[miriam.rateike@ibm.com](miriam.rateike@ibm.com)[ ](miriam.rateike@ibm.com)IBM, University ofTübingen Nairobi, Kenya  \nMichael Kaufmann  \n[kau@zurich.ibm.com](kau@zurich.ibm.com)[ ](kau@zurich.ibm.com)IBM Zurich, Switzerland  \nIsaac Waweru Wambugu  \n[isaacw@ke.ibm.com](isaacw@ke.ibm.com)[ ](isaacw@ke.ibm.com)IBM Nairobi, Kenya  \nIoana Giurgiu  \n[igi@zurich.ibm.com](igi@zurich.ibm.com)[ ](igi@zurich.ibm.com)IBM Zurich, Switzerland  \nCelia Cintas  \n[celia.cintas@ibm.com](celia.cintas@ibm.com)[ ](celia.cintas@ibm.com)IBM Nairobi, Kenya  \nSkyler Speakman  \n[skyler@ke.ibm.com](skyler@ke.ibm.com)[ ](skyler@ke.ibm.com)IBM Nairobi, Kenya  \narXiv :2607 . 11415v1 [ cs .DC] 13 Jul 2026  \nABSTRACT  \nAllocating executors (i.e. compute resources) to distributed processing systems must balance resource costs of scaling-out unnecessarily against artificial, performance-limiting bottlenecks. Naive approaches may allocate executors at the application level, which have predictable costs and performance but are almost guaranteed to be sub-optimal for each of the thousands of diverse, individual stages executed by the application. Users may also have explicit preferences, such as completing an application within a specific time budget while minimizing cost, that existing solutions usually fail to support. We propose a novel method for determining the number of executors per stage in a serverless Apache Spark™ environment, enabling users to specify their desired cost–performance trade-off. Our approach trains tree-ensemble models to estimate the run times and costs of a stage as a function of allocated resources. These estimates are then used to recommend resources for each stage individually. We evaluate our approach on TPC-DS and SQLStorm benchmarks and compare it against two baselines. Depending on the userdefined trade-off parameter and setup, our approach achieves ∼50% cost savings across 103 TPC-DS queries with only a ∼16% slowdown, and ∼40.5% on 96 SQLStorm queries at a ∼29% slowdown.  \nVLDB Workshop Reference Format:  \nMiriam Rateike, Isaac Waweru Wambugu, Celia Cintas, Michael Kaufmann, Ioana Giurgiu, and Skyler Speakman. Stage-Level Executor Allocation in Apache Spark™ with Cost–Performance Trade-offs. VLDB 2026 Workshop: Applied AI for Database Systems and Applications (AIDB 2026) .  \nVLDB Workshop Artifact Availability:  \nThe source code, data, and/or other artifacts have been made available at [https://github.com/mrateike/pvldb-submission](https://github.com/mrateike/pvldb-submission).  \n1 INTRODUCTION  \nLarge-scale data processing increasingly relies on data-parallel engines that execute applications on shared clusters managed by automated resource managers [4, 11, 26]. As corporate cloud spending  \nThis work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit [https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of](https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of)[ ](https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of)[this license. For any use beyond those covered by this license](this license. For any use beyond those covered by this license), [obtain permission by](obtain permission by)[emailing info@vldb.org. Copyright](emailing info@vldb.org. Copyright) is held by the owner/author(s). Publication rights licensed to the VLDB Endowment.  \nProceedings of the VLDB Endowment. ISSN 2150-8097 .  \ncontinues to rise organizations face increasing pressure to reduce costs without compromising performance [8] . This challenge is amplified by the growing use of agentic systems where speculative querying can dramatically increase overall system workloads [12] . In serverless data processing environments (e.g., serverless Apache Spark™), billing is pay-per-use with fine-grained metering of compute and time. Compared to traditional pre-provisioned clusters where reso","cbCaigLAls8GgUg5","https://ap.wps.com/l/cbCaigLAls8GgUg5","pdf",2331300,1,18,"English","en",105,"# Abstract\n# Introduction","[{\"question\":\"Why do application-level executor allocation methods perform poorly in Apache Spark workloads?\",\"answer\":\"Because each SQL query contains many heterogeneous stages with very different runtime sensitivity and task counts. A fixed executor allocation causes some stages to be overprovisioned (unnecessary cost) while others are under-resourced (performance slowdown).\"},{\"question\":\"What problem does the proposed approach address in serverless Apache Spark?\",\"answer\":\"It performs stage-level optimization of the number of executors per stage, accounting for user preferences and balancing cost versus performance in a pay-per-use billing environment with fine-grained metering.\"},{\"question\":\"How does the method choose executors for each stage?\",\"answer\":\"It trains tree-ensemble models to estimate stage runtime and cost as a function of allocated resources, then uses these estimates to recommend executor counts individually for each stage.\"}]",1784202497,45,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"stage-level-executor-allocation-in-apache-spark-with-costperformance-trade-offs","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/stage-level-executor-allocation-in-apache-spark-with-costperformance-trade-offs/85324/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why do application-level executor allocation methods perform poorly in Apache Spark workloads?","Question",{"text":75,"@type":76},"Because each SQL query contains many heterogeneous stages with very different runtime sensitivity and task counts. A fixed executor allocation causes some stages to be overprovisioned (unnecessary cost) while others are under-resourced (performance slowdown).","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What problem does the proposed approach address in serverless Apache Spark?",{"text":80,"@type":76},"It performs stage-level optimization of the number of executors per stage, accounting for user preferences and balancing cost versus performance in a pay-per-use billing environment with fine-grained metering.",{"name":82,"@type":73,"acceptedAnswer":83},"How does the method choose executors for each stage?",{"text":84,"@type":76},"It trains tree-ensemble models to estimate stage runtime and cost as a function of allocated resources, then uses these estimates to recommend executor counts individually for each stage.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]