[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85929-en":3,"doc-seo-85929-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85929,7971461740886,"Theodore","https://ap-avatar.wpscdn.com/davatar_3d24733baf745e90a7e4bdd5f77d97b2",8,"Research & Report","On the Real-World Generalisability of Optical Flow Models","Real-world deployment of optical flow models is hindered by the lack of reliable per-pixel ground truth, which has led research to rely heavily on synthetic training and domain-specific benchmarks. This work evaluates whether benchmark accuracy actually predicts performance on real-world video. A dedicated real-world benchmark is built with 8,204 frame pairs across TAP-Flow, Slow Flow, and FlowFactor, where FlowFactor isolates confounding factors. Results show strong correlation for illumination and large displacements and weak prediction from Sintel, KITTI, and Spring.","arXiv :2607 . 10470v1 [ cs .CV] 11 Jul 2026  \nOn the Real-World Generalisability of Optical Flow Models  \nPetter Reijalt , Sander Gielisse , Rickard Karlsson , and Jan van Gemert  \nComputer Vision Lab, TU Delft, Delft, the Netherlands  \nAbstract. Real-world deployment of vision models to broadly benefit society is arguably a main research objective. In optical flow, however, the difficulty to obtain the ground truth has focused research mainly on synthetic data and domain-specific benchmarks. Here, we investigate the severity of this mismatch. We study how well modern optical flow estimation models generalise to real-world video and question if accuracy on synthetic benchmark proxies actually predicts accuracy on real-world optical flow. To address this, we build a real-world evaluation benchmark and evaluate the real-world generalisability of a broad set of recent optical flow models using standard checkpoints. Our benchmark contains 8,204 frame pairs across TAP-Flow, Slow Flow, and our own dataset FlowFactor. FlowFactor is a manually annotated real-world benchmark of 1,000 HD frame pairs organised into four confounding factors: large displacements, repetitive textures, occlusions, and lighting variation. Each setting mainly varies only one factor, enabling diagnostic, confounder-specific analysis. Using FlowFactor, we reveal that performance on varying lighting and large displacements correlates most strongly with real-world accuracy, and that improvements on large-motion regimes can trade off against robustness in small-motion, stationary scenes. Our experiments show that progress on Sintel, KITTI and Spring only weakly predicts accuracy on real-world data, highlighting the need for a broad real-world optical flow benchmark. Interestingly, scaling up the amount of training data does not necessarily resolve the gap, calling for new innovative research instead of simply scaling data and compute.  \nCode and datasets are available at [https://github.com/Petter6/real](https://github.com/Petter6/real)world-optical-flow.  \nKeywords: Optical Flow · Real-world Benchmark · Generalisation  \n1 Introduction  \nOptical flow, the apparent 2D motion of pixels between consecutive frames, is a foundation for real-world applications ranging from action recognition and video editing to autonomous driving and robotics [3] . Modern optical flow methods [10, 16, 19, 32, 35, 36] learn from annotated data. Due to the difficulty of obtaining annotated per-pixel ground truth optical flow, the training data is created synthetically, such as in FlyingChairs (C) [10] and FlyingThings3D (T) [26]  \n2 P. Reijalt et al.  \nSynthetic optical flow Real-world optical flow  \nFig. 1: How well does optical flow quality on current benchmarks generaliseto real world optical flow quality? Optical flow generalisability is, amongst others, measured on synthetic benchmarks [16, 32, 40] such as Sintel [7] . In this paper, we investigate if current-day benchmarks like Sintel can accurately measure generalisability to real-world settings. Both optical flow predictions above were made by the things checkpoint of FlowFormerPlusPlus [32] . Whereas the prediction on the synthetic frame is nearly perfect, the real-world example reveals noticeable errors, particularly around the legs and parts of the arms, and a loss of fine detail in the hair. This leads us to ask the question: How well do current-day benchmarks measure generalisability to real-world benchmarks?  \nand their combination (C+T) . For evaluating the generalisation of optical flow it is shown that synthetic pre-training can already yield competitive results [16] on differently distributed datasets such as Sintel [7] and KITTI [28] . A notable gap in evaluating optical flow generalisability is diverse real-world data, as illustrated in Fig. 1. Such real-world data is not used due to hardships of obtaining optical flow ground truth where obtaining pixel-accurate motion for every frame is not trivial, due to occlusions,","cbCaij17ObEWBEdd","https://ap.wps.com/l/cbCaij17ObEWBEdd","pdf",25369132,1,25,"English","en",105,"# Introduction\n## Motivation and problem of benchmark mismatch\n## Real-world benchmark design and FlowFactor\n## Confounding-factor evaluation approach\n# Real-world datasets and evaluation plan","[{\"question\":\"Why does optical flow research rely on synthetic data and benchmarks?\",\"answer\":\"Because obtaining dense, pixel-accurate optical flow ground truth in real scenes is difficult due to occlusions, non-rigid motion, and lighting changes, making manual dense annotation essentially infeasible.\"},{\"question\":\"What is FlowFactor and what confounding factors does it target?\",\"answer\":\"FlowFactor is a manually annotated real-world benchmark of 1,000 HD frame pairs organized into four confounding factors: large displacements, repetitive textures, occlusions, and lighting variation.\"},{\"question\":\"Do synthetic benchmark scores reliably predict real-world optical flow accuracy?\",\"answer\":\"No—progress on Sintel, KITTI, and Spring only weakly predicts real-world accuracy, indicating limited benchmark-to-reality generalisability.\"}]",1784207219,63,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"on-the-real-world-generalisability-of-optical-flow-models","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/on-the-real-world-generalisability-of-optical-flow-models/85929/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why does optical flow research rely on synthetic data and benchmarks?","Question",{"text":75,"@type":76},"Because obtaining dense, pixel-accurate optical flow ground truth in real scenes is difficult due to occlusions, non-rigid motion, and lighting changes, making manual dense annotation essentially infeasible.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What is FlowFactor and what confounding factors does it target?",{"text":80,"@type":76},"FlowFactor is a manually annotated real-world benchmark of 1,000 HD frame pairs organized into four confounding factors: large displacements, repetitive textures, occlusions, and lighting variation.",{"name":82,"@type":73,"acceptedAnswer":83},"Do synthetic benchmark scores reliably predict real-world optical flow accuracy?",{"text":84,"@type":76},"No—progress on Sintel, KITTI, and Spring only weakly predicts real-world accuracy, indicating limited benchmark-to-reality generalisability.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]