[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-126631-en":3,"doc-seo-126631-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},126631,549768064622,"Anda","https://ap-avatar.wpscdn.com/davatar_6f874abed73319feea01a86fa6f0fab8",8,"Research & Report","Full Reference Video Quality Assessment for Machine Learning-Based Video Codecs - arXiv 2309.00769","Machine learning-based video codecs advance video compression, but accurate evaluation remains a bottleneck because common metrics from DSP-based codecs do not match human judgments when applied to ML-generated artifacts. The work introduces an MLVC dataset of accurately quality-labeled ML video codec outputs and proposes a full reference video quality assessment (FRVQA) model. The model reaches very high alignment with subjective scores at the model level, and the dataset and FRVQA approach are released to accelerate further research and improve evaluation methods for ML video codecs.","Full Reference Video Quality Assessment for Machine Learning-Based Video  \nCodecs  \narXiv :2309 .00769v1 [ ee ss .IV] 2 Sep 2023  \nAbrar Majeedi University of Wisconsin-Madison  \n[majeedi@wisc.edu](majeedi@wisc.edu)  \nBabak Naderi Microsoft  \n[babaknaderi@microsoft.com](babaknaderi@microsoft.com)  \nYasaman Hosseinkashi Microsoft  \n[yahossei@microsoft.com](yahossei@microsoft.com)  \nJuhee Cho  \nMicrosoft [juhcho@microsoft.com](juhcho@microsoft.com)  \nRuben Alvarez Martinez Microsoft  \n[rubenal@microsoft.com](rubenal@microsoft.com)  \nRoss Cutler  \nMicrosoft [ross.cutler@microsoft.com](ross.cutler@microsoft.com)  \nAbstract  \nMachine learning-based video codecs have made significant progress in the past few years. A critical area in the development of ML-based video codecs is an accurate evaluation metric that does not require an expensive and slow subjective test. We show that existing evaluation metrics that were designed and trained on DSP-based video codecs are not highly correlated to subjective opinion when used with ML video codecs due to the video artifacts being quite different between ML and video codecs. We provide a new dataset of ML video codec videos that have been accurately labeled for quality. We also propose a new full reference video quality assessment (FRVQA) model that achieves a Pearson Correlation Coefficient (PCC) of 0.99 and a Spearman’s Rank Correlation Coefficient (SRCC) of 0 .99 at the model level. We make the dataset and FRVQA model opensource to help accelerate research in ML video codecs, and so that others can further improve the FRVQA model.  \n1. Introduction  \nInternet traffic statistics show that internet video traffic will be 82% of all consumer Internet traffic by 2022, up from 73% in 2017, which is a compound annual growth rate of 34%[2] . This number is only expected to grow, with the ever-increasing popularity of the video social media platforms like TikTok, Reels, and YouTube, and videoconferencing applications like Microsoft Teams and Zoom. Since the physical internet infrastructure is limited and cannot be scaled up fast enough to keep up with the exponential growth of internet traffic, video transmission has the potential to choke up the internet [16] . Video compression technologies enable video to be streamed across the internet at a small fraction of the uncompressed bandwidth,  \nFigure 1 . Full reference video quality assessment. The image on the left is from the original video, while the image on the right is from the compressed version. The scores are in the range of 1-9, where 9 is the best.  \nwhich enables video streaming and video conferencing applications to be possible. These video compression methods involve fast and efficient hand-coded digital signal processing (DSP) algorithms to reduce the size of the video files by over 1000x with an acceptable reduction in quality.  \nIn the late 1980s, the common H.26X video compression standards were introduced, starting with H.261 in 1988 [13], followed by H.262 in 1994 [1], H.263 in 1996 [37], H.264 in 2003 [49], H.265 in 2013 [41], and the latest one H.266 in 2020 [6] . Each one of these DSP codecs roughly doubled the coding efficiency of their predecessors, and the average time between versions for the past three codecs is 8.5 years.  \nMachine learning (ML) is now actively being used to improve the coding efficiency of video codecs. However, the vast majority of ML video codecs [4, 22, 23, 27–29] only evaluate the codec quality using metrics like Peak Signalto-Noise ratio (PSNR) [12] and Multi-Scale Structural Similarity Index Measure (MS-SSIM) [48], which as we show in the paper, are poorly correlated with human subjective quality assessment.  \nThe development and adoption of ML video codecs have  \nfaced a roadblock in terms of perceptual quality evaluation, i.e., measuring how good or bad a human viewer perceives the videos compressed by an ML video codec. ML video codecs have different artifacts than DSP-based video codecs, as sh","cbCaidAq4sHSMM3x","https://ap.wps.com/l/cbCaidAq4sHSMM3x","pdf",6066205,1,14,"English","en",105,"# Introduction\n## Motivation: limits of subjective testing\n## Background: DSP codecs and existing metrics\n## Problem: low correlation with ML video codecs\n## Approach: new dataset and FRVQA model","[{\"question\":\"Why do existing evaluation metrics fail for ML-based video codecs?\",\"answer\":\"Existing metrics designed and trained on DSP-based codecs correlate poorly with subjective opinions for ML video codecs because the artifacts differ substantially between ML and traditional codecs.\"},{\"question\":\"What does the proposed FRVQA model achieve compared with existing objective metrics?\",\"answer\":\"The proposed full reference video quality assessment model achieves a Pearson Correlation Coefficient (PCC) of 0.99 and a Spearman’s Rank Correlation Coefficient (SRCC) of 0.99 at the model level, outperforming state-of-the-art objective metrics.\"},{\"question\":\"What resources are released to support research in ML video codecs?\",\"answer\":\"The authors release a new dataset of ML video codec videos with accurate subjective quality labels and the FRVQA model (including code and trained model) to help others accelerate research and further improvements.\"}]","Full Reference Video Quality Assessment for Machine Learning-Based Video Codecs - arXiv 2309.00769 | PDF",1785933910,35,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"full-reference-video-quality-assessment-for-machine-learning-based-video-codecs-arxiv-230900769","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/full-reference-video-quality-assessment-for-machine-learning-based-video-codecs-arxiv-230900769/126631/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-23","2026-08-05",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"Why do existing evaluation metrics fail for ML-based video codecs?","Question",{"text":76,"@type":77},"Existing metrics designed and trained on DSP-based codecs correlate poorly with subjective opinions for ML video codecs because the artifacts differ substantially between ML and traditional codecs.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"What does the proposed FRVQA model achieve compared with existing objective metrics?",{"text":81,"@type":77},"The proposed full reference video quality assessment model achieves a Pearson Correlation Coefficient (PCC) of 0.99 and a Spearman’s Rank Correlation Coefficient (SRCC) of 0.99 at the model level, outperforming state-of-the-art objective metrics.",{"name":83,"@type":74,"acceptedAnswer":84},"What resources are released to support research in ML video codecs?",{"text":85,"@type":77},"The authors release a new dataset of ML video codec videos with accurate subjective quality labels and the FRVQA model (including code and trained model) to help others accelerate research and further improvements.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":93},[94,98,102,106,111,116,121,124,129,132,136],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":112,"doc_module":4,"doc_module_name":46,"category_name":113,"show_sort_weight":114,"slug":115},6,"Technology",50,"technology",{"id":117,"doc_module":4,"doc_module_name":46,"category_name":118,"show_sort_weight":119,"slug":120},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":122,"slug":123},30,"research-report",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":126,"show_sort_weight":127,"slug":128},9,"Religion & Spirituality",20,"religion-spirituality",{"id":127,"doc_module":4,"doc_module_name":46,"category_name":130,"show_sort_weight":127,"slug":131},"World Cup","world-cup",{"id":133,"doc_module":4,"doc_module_name":46,"category_name":134,"show_sort_weight":133,"slug":135},10,"Lifestyle","lifestyle",{"id":137,"doc_module":4,"doc_module_name":46,"category_name":138,"show_sort_weight":107,"slug":139},19,"General","general"]