[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85645-en":3,"doc-seo-85645-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85645,4398048949847,"Eliana","https://ap-avatar.wpscdn.com/avatar/400002536579ef2da7f?_k=1778318612642679267",8,"Research & Report","Forget, Anticipate and Adapt: Test Time Training for Long Videos","Test Time Training (TTT) enables a model to adapt at inference by solving a self-supervised task and updating weights without test-time labels. For long videos, existing sliding-window TTT becomes computationally infeasible because compute grows with window size, and it redundantly updates even for temporally similar frames. Frame Forgetting Network (FFN) reduces cost by using only three key frames and introduces a mathematical “surprise” metric to adapt the effective window size. Experiments on EpicTours up to 3-hour videos validate dense segmentation, classification, depth generalization, and multi-hour stability.","arXiv :2606 .26515v3 [ cs .CV] 10 Jul 2026  \nForget, Anticipate and Adapt: Test Time Training for Long Videos  \nRajat Modi 1 , Sebastian Noel 1 , Xin Liang 1 , and Yogesh S. Rawat 1  \nInstitute of Artificial Intelligence, University of Central Florida [rajatmodi62@gmail.com](rajatmodi62@gmail.com) , [yogesh@crcv.ucf.edu](yogesh@crcv.ucf.edu)[ ](yogesh@crcv.ucf.edu)[https://rajatmodi62.github.io/2025/07/10/ffn/](https://rajatmodi62.github.io/2025/07/10/ffn/)  \nAbstract. Test Time Training (TTT) is a mechanism in which a model adapts to an incoming test-sample by performing some self-supervised (SSL) task and updating its weights even during inference. This procedure does not require labels at test-time. This paper focuses on TTT for long-videos. A major concern with existing approaches is: 1) they perform TTT updates using a sliding window containing frames in the past, whose compute increases linearly with the size of window. This becomes computationally intractable when the videos are hours long. 2) TTT is performed even when temporally close frames look similar, thereby consuming a lot of compute.  \nWe present the Frame Forgetting Network (FFN) that: 1) operates on only three frames within the sliding window, namely the frame that exits, the current frame and the frame after that. The model still manages to retain temporal context and work for hours long-videos; 2) mathematically define a ‘surprise’ metric: how much ‘new information’ the incoming frame contains with respect to the past seen frame. This facilitates determining how to modify the effective window size during TTT and constitutes the core mechanism of an adaptive windowing algorithm. Additionally, we curate a dataset EpicTours containing up to 3 hour long videos of walking city-tours, whereas earlier datasets on this problem were only 5 min long. We demonstrate FFN’s empirical effectiveness on dense-segmentation, video classification tasks, generalization to depthestimation, and multi-hour long videos. The project page can be found at [https://github.com/rajatmodi62/ffn](https://github.com/rajatmodi62/ffn).  \n1 Introduction  \nTypically, machine learning models only update their weights during training, but freeze them during inference [1, 2] . However, as humans, we possess the ability to continuously learn and adapt to our ever-changing environment. Test Time Training (TTT) shows the promise of bringing a similar adaptive capability to artificial intelligence, allowing a model to continuously learn and update its weights even while testing, entirely without the need for ground-truth labels.  \n2 R. Modi, S. Noel, X. Liang, Y.S. Rawat  \nHowever, applying TTT to video processing presents a unique set of challenges. Existing online distillation methods are based on a student-teacher setup [3], with the teacher usually deployed on a remote server and the student on a local device. However, this is a bottleneck in offline real-world scenarios, such as disaster-prone regions, where server connectivity may be limited.  \nConsequently, the key challenge lies in determining how to perform TTT on videos locally, on-device, and computationally cheaply. This is especially critical for dense computer vision tasks, such as segmentation, where each pixel matters, and should ideally be achieved utilizing a single model.  \nHistorically, there have been two approaches towards adaptation: either (i) train a dedicated video model specifically for the task, or (ii) start from a pre-trained image model and adapt it accordingly. Unfortunately, large-scale pre-training datasets for video remain limited, and efforts to repurpose image models are inherently constrained by the absence of explicit temporal information. Furthermore, existing TTT methods for videos rely on a sliding window approach [4] . A sliding window serves as a fixed-size temporal buffer that holds a defined number of preceding frames and moves forward step by step as the video advances. Because the model re-evaluates all","cbCaituPbrEef0Bl","https://ap.wps.com/l/cbCaituPbrEef0Bl","pdf",26041955,1,29,"English","en",105,"# Introduction\n# Frame Forgetting Network\n## Problem statement\n## Preliminaries on TTT","[{\"question\":\"What problem does TTT aim to solve during inference?\",\"answer\":\"TTT lets a model adapt to incoming test samples during inference by running a self-supervised task and updating its weights. This improves downstream performance without requiring labels at test time.\"},{\"question\":\"Why are existing long-video TTT approaches computationally challenging?\",\"answer\":\"They typically use sliding windows that require compute growing linearly with the window size and reprocess all frames in the window at every timestep. This becomes intractable for hours-long videos and wastes compute when consecutive frames are similar.\"},{\"question\":\"How does FFN reduce computation while maintaining temporal context?\",\"answer\":\"FFN processes only three frames within the sliding window: the frame that exits, the current frame, and the following frame. It also uses a surprise metric to adapt when the window should effectively change during TTT.\"}]",1784205327,73,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"forget-anticipate-and-adapt-test-time-training-for-long-videos","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/forget-anticipate-and-adapt-test-time-training-for-long-videos/85645/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does TTT aim to solve during inference?","Question",{"text":75,"@type":76},"TTT lets a model adapt to incoming test samples during inference by running a self-supervised task and updating its weights. This improves downstream performance without requiring labels at test time.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why are existing long-video TTT approaches computationally challenging?",{"text":80,"@type":76},"They typically use sliding windows that require compute growing linearly with the window size and reprocess all frames in the window at every timestep. This becomes intractable for hours-long videos and wastes compute when consecutive frames are similar.",{"name":82,"@type":73,"acceptedAnswer":83},"How does FFN reduce computation while maintaining temporal context?",{"text":84,"@type":76},"FFN processes only three frames within the sliding window: the frame that exits, the current frame, and the following frame. It also uses a surprise metric to adapt when the window should effectively change during TTT.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]