[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84935-en":3,"doc-seo-84935-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},84935,687197207639,"Asher","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation","AgentLens provides a production-assessed benchmark for interactive code agents by evaluating the full interaction trajectory rather than reducing each run to a single pass/fail bit. It combines formal verification when objective checks exist with LLM-written trajectory reviews and side-by-side comparisons, producing readable explanations for each score. The benchmark supports diagnosing model behavior, comparing successive agent versions, and detecting product regressions within a nightly evaluation pipeline.","arXiv :2607 .06624v 1 [ cs .AI ] 7 Jul 2026  \nAgentLens: Production-Assessed Trajectory Reviews for Coding  \nAgent Evaluation  \nAndrey Podivilov∗1, Vadim Lomshakov∗ ,†1 , Sergey Savin∗1, Matvei Startsev 1 , Roman Pozharskiy 1 , Maksim Parshin 1 , and Sergey Nikolenko2,3  \n1Explyt  \n2 St. Petersburg Department of the Steklov Institute of Mathematics  \n3 St. Petersburg State University  \nJuly 9, 2026  \nAbstract  \nWe present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit—did the task pass?—but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at [https://github.com/agent-lens/agent-lens-bench](https://github.com/agent-lens/agent-lens-bench).  \n∗ Equal contribution. †Corresponding author: [vadim.lomshakov@explyt.com](vadim.lomshakov@explyt.com)  \n1 Introduction  \nMost coding agent benchmarks ask one question: did the agent finish the task (Jimenez et al., 2024 ; Merrill et al., 2026)? For a research leaderboard, a single pass/fail bit is often enough. But for a coding assistant that developers use all day, most of what matters is not captured by that bit. A run can pass a narrow check and still be unreliable, overconfident, unpleasant to work with, or subtly misaligned with what the user actually asked for. Conversely, a run that fails the final check may have done plenty of useful things on the way there: localized the bug correctly, used its tools safely, recovered gracefully from an error and so on.  \nThe problem is most important for tasks that have no natural binary outcome, such as, e.g., writing project documentation. A single success flag cannot say much about it: the useful question is not whether a file appeared on disk, but whether the document is accurate, appropriately scoped, readable, grounded in the actual codebase, and helpful to the readers.  \nOur proposed benchmark, AgentLens, is built to keep this structure visible, evaluating complete trajectories rather than final states alone. A trajectory is the full record of a session: messages from the (simulated) user, the agent’s replies, its tool calls, file edits, command executions, verification attempts, the final response, and the final state of the repository. AgentLens treats this record as the primary object of evaluation and uses it to measure quality closer to the way a user would perceive it. The benchmark uses textual judge reviews as the main artifacts, while formal verification supplies complementary objective checks wherever such checks exist.  \nWe built AgentLens because the benchmarks we had did not match what we needed to know: some lacked realistic user interaction, some collapsed everything to a binary or final-state outcome, and some used tasks that were either too simple or too far from the day-to-day work of a coding assistant. We wanted something that evaluates production-like interactive work and then explains how agents differ.  \nOur main contributions in this work are:  \n• a production-assessed benchmark for interactive code agents;  \n• a trajectory-level evaluation protocol that combines formal verification with LLM judge reviews;  \n• side-by-side reviews for model comparison, feature evaluation, and regression analysis;  \n• an open-source release of the benchmark code at [https://github.com/agent-lens/agent-lens-bench](","cbCaijwszH8J83tY","https://ap.wps.com/l/cbCaijwszH8J83tY","pdf",411341,4,1,30,"English","en",105,"# Introduction\n# Related Work\n# Methodology and Evaluation Protocol\n# Tasks, Simulated Users, and Metrics\n# Experiments and Model Comparisons\n# Analysis, Reliability, and Bias Checks\n# Limitations and Future Work","[{\"question\":\"What makes AgentLens different from typical coding agent benchmarks?\",\"answer\":\"AgentLens evaluates complete production-like trajectories, including the agent’s interactions and recovery behavior, instead of only measuring whether the final task passed a binary check.\"},{\"question\":\"How does AgentLens assess quality during an evaluation run?\",\"answer\":\"It uses formal verification when objective checks are available and pairs that with LLM-written trajectory reviews and side-by-side comparisons, so each run yields an interpretable rationale for its score.\"},{\"question\":\"How is AgentLens used in an ongoing development setting?\",\"answer\":\"The benchmark is integrated into a nightly evaluation pipeline to diagnose model behavior, compare successive versions of the agent, and catch product regressions over time.\"}]",1784199499,76,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"agentlens-production-assessed-trajectory-reviews-for-coding-agent-evaluation","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/agentlens-production-assessed-trajectory-reviews-for-coding-agent-evaluation/84935/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What makes AgentLens different from typical coding agent benchmarks?","Question",{"text":75,"@type":76},"AgentLens evaluates complete production-like trajectories, including the agent’s interactions and recovery behavior, instead of only measuring whether the final task passed a binary check.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does AgentLens assess quality during an evaluation run?",{"text":80,"@type":76},"It uses formal verification when objective checks are available and pairs that with LLM-written trajectory reviews and side-by-side comparisons, so each run yields an interpretable rationale for its score.",{"name":82,"@type":73,"acceptedAnswer":83},"How is AgentLens used in an ongoing development setting?",{"text":84,"@type":76},"The benchmark is integrated into a nightly evaluation pipeline to diagnose model behavior, compare successive versions of the agent, and catch product regressions over time.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,122,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":22,"slug":121},"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]