[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84471-en":3,"doc-seo-84471-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84471,687197100911,"Himbo","https://ap-avatar.wpscdn.com/avatar/a000239b6f1da00475?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782698725881665579",8,"Research & Report","MFTune An Efficient Multi fidelity Framework for Spark SQL Configuration Tuning","Apache Spark SQL performance tuning faces high barriers due to an extremely large configuration space and the prohibitive cost of evaluating massive workloads. Existing multi-fidelity approaches based on common accelerations such as data reduction or early stopping fail for Spark SQL because they do not preserve performance correlations or expose true bottlenecks. MFTune introduces query-based fidelity partitioning with representative SQL subsets as low-cost proxies, then applies density-based search compression and transfer learning plus a two-phase warm start. Experiments on TPC-H and TPC-DS show clear gains over five state-of-the-art methods.","MFTune: An Efficient Multi-fidelity Framework for Spark SQL  \nConfiguration Tuning  \nBeicheng Xu∗† Peking University Beijing, China [beichengxu@stu.pku.edu.cn](beichengxu@stu.pku.edu.cn)  \nLingching Tung∗† Peking University Beijing, China  \n[lingchingtung@stu.pku.edu.cn](lingchingtung@stu.pku.edu.cn)  \nYuchen Wang† Peking University Beijing, China [wychen@stu.pku.edu.cn](wychen@stu.pku.edu.cn)  \nYupeng Lu† Peking University Beijing, China [xinkelyp@pku.edu.cn](xinkelyp@pku.edu.cn)  \nBin Cui† Peking University Beijing, China [bin.cui@pku.edu.cn](bin.cui@pku.edu.cn)  \nABSTRACT  \nApache Spark SQL is a cornerstone of modern big data analytics. However, optimizing Spark SQL performance is challenging due to its vast configuration space and the prohibitive cost of evaluating massive workloads. Existing tuning methods predominantly rely on full-fidelity evaluations, which are extremely time-consuming, often leading to suboptimal performance within practical budgets. While multi-fidelity optimization offers a potential solution, directly applying standard techniques—such as data volume reduction or early stopping—proves ineffective for Spark SQL as they fail topreserve performance correlations or represent true system bottlenecks. To address these challenges, we propose MFTune, an efficient multi-fidelity framework that introduces a query-based fidelity partitioning strategy, utilizing representative SQL subsetsto provide accurate, low-cost proxies. To navigate the huge search space, MFTune incorporates a density-based optimization mechanism for automated knob and range compression, alongside an adapted transfer learning approach and a two-phase warm start to further accelerate the tuning process. Experimental results on TPC-H and TPC-DS benchmarks demonstrate that MFTune significantly outperforms five state-of-the-art tuning methods, identifying superior configurations within practical time constraints.  \nPVLDB Reference Format:  \nBeicheng Xu, Lingching Tung, Yuchen Wang, Yupeng Lu, and Bin Cui. MFTune: An Efficient Multi-fidelity Framework for Spark SQL  \nConfiguration Tuning. PVLDB, 19(11): XXX-XXX, 2026 .  \ndoi:XX.XX/XXX.XX  \nPVLDB Artifact Availability:  \nThe source code, data, and/or other artifacts have been made available at [https://github.com/PKU-DAIR/MFTune](https://github.com/PKU-DAIR/MFTune).  \n∗ Both authors contributed equally to this research.  \n†All authors are with the School of Computer Science & Key Lab of High Confidence Software Technologies (MOE) & Beijing Key Laboratory of Software and Hardware Cooperative Artificial Intelligence Systems, Peking University, Beijing, China.  \nThis work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit [https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of](https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of)[ ](https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of)[this license. For any use beyond those covered by this license](this license. For any use beyond those covered by this license), [obtain permission by](obtain permission by)[emailing info@vldb.org. Copyright](emailing info@vldb.org. Copyright) is held by the owner/author(s). Publication rights licensed to the VLDB Endowment.  \nProceedings of the VLDB Endowment, Vol. 19, No. 11 ISSN 2150-8097 . doi:XX.XX/XXX.XX  \nCount  \n30  \n20  \n10  \n0  \n\n| 25 |  | MFTune\u003Cbr>MFTune w/o multi-fidelity |\n| --- | --- | --- |\n|  |  |  |\n|  |  |  |\n|  |  |  2 \u003Cbr>\u003Cbr>8\u003Cbr>\u003Cbr>\u003Cbr>5\u003Cbr> 3  \u003Cbr>\u003Cbr>\u003Cbr> |\n|  |  |  |\n|  |  |  |\n|  |  |  |\n\n1k 2k  2k 3k  3k 5k   \n5k 7k >7k  error  \nLatency ( better)  \n(a) Performance distribution.  \nCorr with Full Fidelity  \n1.0  \n0.8  \n0.6  \n0.00 0.25 0.50 0.75 1.00  \nAverage Latency Ratio  \n(b) Fidelity correlation.  \nFigure 1: Analysis of multi-fidelity mechanism on TPC-DS (600GB). (a) Latency distribution of configurations evaluated within 48 hours. (b) Fidelity correlation. We evaluate six proxies: 1) SQL-Ours, selecting SQL su","cbCaiqee9hJiX2x1","https://ap.wps.com/l/cbCaiqee9hJiX2x1","pdf",3382764,1,14,"English","en",105,"# Abstract\n# 1 INTRODUCTION\n## Spark SQL workloads and tuning challenges\n## Configuration space and evaluation cost","[{\"question\":\"Why is tuning Spark SQL configurations difficult in practice?\",\"answer\":\"Spark SQL has a vast configuration space (over 200 knobs) and each evaluation is expensive because workloads can process hundreds of gigabytes to terabytes, making full-fidelity tuning slow and costly.\"},{\"question\":\"What limitation makes standard multi-fidelity techniques ineffective for Spark SQL?\",\"answer\":\"Common accelerations like data volume reduction or early stopping do not preserve correlations with full-fidelity performance and may fail to reflect true system bottlenecks for Spark SQL.\"},{\"question\":\"How does MFTune improve multi-fidelity tuning efficiency?\",\"answer\":\"MFTune uses query-based fidelity partitioning that selects representative SQL subsets as low-cost fidelity proxies, then compresses the search via a density-based optimization strategy and accelerates convergence with transfer learning and a two-phase warm start.\"}]",1784195864,35,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"mftune-an-efficient-multi-fidelity-framework-for-spark-sql-configuration-tuning","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/mftune-an-efficient-multi-fidelity-framework-for-spark-sql-configuration-tuning/84471/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why is tuning Spark SQL configurations difficult in practice?","Question",{"text":75,"@type":76},"Spark SQL has a vast configuration space (over 200 knobs) and each evaluation is expensive because workloads can process hundreds of gigabytes to terabytes, making full-fidelity tuning slow and costly.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What limitation makes standard multi-fidelity techniques ineffective for Spark SQL?",{"text":80,"@type":76},"Common accelerations like data volume reduction or early stopping do not preserve correlations with full-fidelity performance and may fail to reflect true system bottlenecks for Spark SQL.",{"name":82,"@type":73,"acceptedAnswer":83},"How does MFTune improve multi-fidelity tuning efficiency?",{"text":84,"@type":76},"MFTune uses query-based fidelity partitioning that selects representative SQL subsets as low-cost fidelity proxies, then compresses the search via a density-based optimization strategy and accelerates convergence with transfer learning and a two-phase warm start.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]