[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84345-en":3,"doc-seo-84345-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84345,1099514068365,"Aurelia","https://ap-avatar.wpscdn.com/avatar/10000253d8d9f28188e?_k=1776742907772140068",8,"Research & Report","Understanding Axes of Difficulty For Long Context Tasks Via PREDICATELONGBENCH","Large language models show improving long-context performance, yet widely used evaluations often focus on average-case behavior and can become saturated or insufficiently robust. This work introduces PREDICATELONGBENCH, a benchmark that stress-tests long-context reasoning by requiring models to find the longest contiguous subsequence in a long input that satisfies specified predicate constraints, such as lexicographic ordering. The benchmark varies multiple axes of difficulty using synthetic and real-document generation pipelines, revealing performance degradation in frontier models as difficulty scales.","Understanding Axes of Difficulty For Long Context  \nTasks Via PREDICATELONGBENCH  \nSiddhartha Jain  \nNVIDIA  \n[siddjain@nvidia.com](siddjain@nvidia.com)  \nAmeya Velingker  \nNVIDIA  \n[avelingker@nvidia.com](avelingker@nvidia.com)  \narXiv :2607 .08284v 1 [ cs .AI] 9 Jul 2026  \nAbstract  \nLarge language models (LLMs) have demonstrated rapidly improving long-context capabilities, prompting a wave of benchmarks designed to evaluate them. However, existing long-context evaluations—from Needle-in-a-Haystack (NIAH) tests to more recent multi-hop reasoning and summarization tasks—predominantly measure average-case performance, and many are either saturated or lack robustness.  \nNotably absent is a systematic way to probe how models perform as we scale up the difficulty of tasks along various axes. We address this gap by proposing PREDICATELONGBENCH, a benchmark that stress-tests long-context reasoning by asking models to identify the longest contiguous subsequence of words in along input that satisfies given predicates/constraints (e.g., lexicographic ordering), drawn from a broader predicate class. The central innovation of our benchmark is the identification and systematic exploration of multiple different axes of difficulty which test multiple aspects of long context understanding. We provide two complementary generation pipelines—a fully synthetic setup using random word-like strings, and a real-world setup that samples words from natural documents while preserving their distributional properties. We find that frontier models struggle to perform well as we scale up the difficulty of tasks along our axes, demonstrating the utility of our benchmark in understanding the limitations of current long-context capabilities. Furthermore, the tasks in PREDICATELONGBENCH, though challenging, are conceptually simple and do not require LLM-based generations or judges.  \n1 Introduction  \nLarge language models (LLMs) have shown increased long-context performance in recent years. As more and more powerful models capable of processing even longer context lengths emerge, there isan increasing need to have robust benchmarks for evaluating long-context capabilities of a model. One of the earliest markers of long-context performance has been the popular Needle-in-a-Haystack (NIAH) test, which evaluates a model’s ability to find a “needle”(tiny piece of localized information) hidden inside a “haystack”(a large window of context) [Kamradt, 2023, Mohtashami and Jaggi, 2023] . A number of variants of NIAH have been devised to make the task more difficult, e.g., embedding multiple needles, adding distractors to the context, etc. However, NIAH tasks nevertheless have limitations and are generally restricted to locating small bits of highly localized information. The need to measure a broader range of long-context capabilities has spurred the development of other long-context evaluations that incorporate additional task diversity in the form of question answering (QA), multi-hop reasoning, summarization, etc.  \nThere are a number of axes along which researchers have sought to increase the difficulty of longcontext evaluations. The most natural one is context length, as increasing the underlying context length of tasks requires a model to find or reason on information contained in a much larger sea of text. To that effect, many popular long-context benchmarks, e.g., RULER [Hsieh et al., 2024],  \nPreprint.  \nHELMET [Yen et al., 2025], are configurable and adaptable to any desired context length, thus allowing one to test the degradation in performance over an increasing list of context lengths. Another axis is the use of distractors, which are surrounding blocks of text designed to hide the key information in the context window. For example, early NIAH tests, such as those in RULER, have used blocks of irrelevant content (e.g., filler phrases like “The grass is green,” sentences from documents or essays, etc.) to obfuscate the needles (key-value pairs) . Newer ","cbCaijwbV5Uj8YYN","https://ap.wps.com/l/cbCaijwbV5Uj8YYN","pdf",578497,1,31,"English","en",105,"# Abstract\n# Introduction\n## Long-context benchmark limitations\n## Difficulty scaling axes","[{\"question\":\"What gap does PREDICATELONGBENCH address in existing long-context evaluations?\",\"answer\":\"It provides a systematic way to probe how model performance changes when task difficulty scales along multiple axes, rather than measuring mostly average-case behavior with saturated or non-robust benchmarks.\"},{\"question\":\"How does PREDICATELONGBENCH define its core task?\",\"answer\":\"Given a long input, models must identify the longest contiguous subsequence of words that satisfies provided predicates or constraints, such as lexicographic ordering.\"},{\"question\":\"How are the benchmark inputs generated?\",\"answer\":\"It uses two complementary pipelines: a fully synthetic setup with random word-like strings and a real-world setup that samples words from natural documents while preserving distributional properties.\"}]",1784194970,78,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"understanding-axes-of-difficulty-for-long-context-tasks-via-predicatelongbench","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/understanding-axes-of-difficulty-for-long-context-tasks-via-predicatelongbench/84345/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What gap does PREDICATELONGBENCH address in existing long-context evaluations?","Question",{"text":75,"@type":76},"It provides a systematic way to probe how model performance changes when task difficulty scales along multiple axes, rather than measuring mostly average-case behavior with saturated or non-robust benchmarks.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does PREDICATELONGBENCH define its core task?",{"text":80,"@type":76},"Given a long input, models must identify the longest contiguous subsequence of words that satisfies provided predicates or constraints, such as lexicographic ordering.",{"name":82,"@type":73,"acceptedAnswer":83},"How are the benchmark inputs generated?",{"text":84,"@type":76},"It uses two complementary pipelines: a fully synthetic setup with random word-like strings and a real-world setup that samples words from natural documents while preserving distributional properties.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]