[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-117586-en":3,"doc-seo-117586-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},117586,1374391974468,"Eden","https://ap-avatar.wpscdn.com/davatar_29158cc5080c5b710cf443261637dec0",8,"Research & Report","Multiple Descent in the Multiple Random Feature Model","This paper studies the multiple descent phenomenon in multi-component prediction models, focusing on ridge regression in high-dimensional regimes. It first analyzes a double random feature model that concatenates two types of random features, deriving the precise limiting excess risk as sample size, data dimension, and feature dimension grow proportionally. Using these calculations, the paper proves that risk curves for such models can show triple descent, and then validates the results with extensive experiments. The study extends to the multiple random feature model, showing K-feature ensembling yields (K+1)-fold descent.","Multiple Descent in the Multiple Random Feature Model  \nXuran Meng xuranmeng@connect.hku.hk  \nDepartment of Statistics and Actuarial Science The University of Hong Kong  \nJianfeng Yao [jeffyao@cuhk.edu.cn](jeffyao@cuhk.edu.cn)  \nSchool of Data Science  \nThe Chinese University of Hong Kong (Shenzhen)  \nYuan Cao yuancao@hku.hk  \nDepartment of Statistics and Actuarial Science The University of Hong Kong  \nEditor: Daniel Hsu  \nAbstract  \nRecent works have demonstrated a double descent phenomenon in over-parameterized learning. Although this phenomenon has been investigated by recent works, it has not been fully understood in theory. In this paper, we investigate the multiple descent phenomenon in a class of multi-component prediction models. We 􀀌rst consider a \\double random feature model\" (DRFM) concatenating two types of random features, and study the excess risk achieved by the DRFM in ridge regression. We calculate the precise limit of the excess risk under the high dimensional framework where the training sample size, the dimension of data, and the dimension of random features tend to in􀀌nity proportionally. Based on the calculation, we further theoretically demonstrate that the risk curves of DRFMs can exhibit triple descent. We then provide a thorough experimental study to verify our theory.  \nAt last, we extend our study to the \\multiple random feature model\" (MRFM), and show that MRFMs ensembling K types of random features may exhibit (K + 1)-fold descent.  \nOur analysis points out that risk curves with a speci􀀌c number of descent generally exist in learning multi-component prediction models.  \nKeywords: Over-parameterization, excess risk, multiple descent, double random feature model, multiple random feature model  \n1. Introduction  \nModern machine learning models such as deep neural networks are usually highly overparameterized so that they can be trained to exactly 􀀌t the training data. Such overparameterized models have gained immense popularity and achieved state-of-the-art performance in various learning tasks. However, in classical statistical learning theory, overparameterized models are believed to have high excess risks due to over􀀌tting, and hence their success has not been fully explained in theory. This gap between theory and practice has motivated a number of recent works to study the success of over-parameterized models.  \n􀀍c2024 Xuran Meng, Jianfeng Yao and Yuan Cao.  \nLicense: CC-BY 4.0, see [https://creativecommons.org/licenses/by/4.0/](https://creativecommons.org/licenses/by/4.0/. Attribution)[. Attribution](https://creativecommons.org/licenses/by/4.0/. Attribution) requirements are provided  \nat [http://jmlr.org/papers/v25/22-1389.html](http://jmlr.org/papers/v25/22-1389.html).  \nMeng, Yao and Cao  \nRecent works have pointed out a double/multiple descent phenomenon in over-parameterized learning: as the number of parameters in a model increases, the excess risk may increase and decrease multiple times (see Figure 1 for some examples) . The double descent phenomenon was 􀀌rst demonstrated experimentally by Belkin et al. (2019) in random feature models, random forests and neural networks, and then studied theoretically by a series of works under di􀀋erent settings. Speci􀀌cally, Belkin et al. (2020) theoretically demonstrated the double descent shape of the risk curve of the minimum norm predictor in learning linear models and Fourier series models. Wu and Xu (2020); Mel and Ganguli (2021); Hastie et al.(2022) studied the excess risk in linear regression under the setting where the dimension and sample size go to in􀀌nity preserving a 􀀌xed ratio, and showed that the risk decreases with respect to this ratio in the over-parameterized setting. Mei and Montanari (2022); Liao et al. (2020) further studied double descent in random feature models when the sample size, data dimension and the number of random features have 􀀌xed ratios and Adlam et al.(2022) extended the model by adding bias terms. Deng et al. (2022)","cbCaislh1BwT65be","https://ap.wps.com/l/cbCaislh1BwT65be","pdf",2575376,1,49,"English","en",105,"# Introduction\n## Background: Double and multiple descent in over-parameterized learning\n## Multi-component predictors and the random feature framework","[{\"question\":\"What phenomenon does the paper investigate in multi-component prediction models?\",\"answer\":\"The paper investigates the multiple descent phenomenon, where the excess risk changes non-monotonically as model capacity grows, potentially showing multiple dips (double, triple, or higher-fold).\"},{\"question\":\"How is the double random feature model (DRFM) constructed and analyzed?\",\"answer\":\"The DRFM concatenates two types of random features and is studied in ridge regression under a high-dimensional limit where sample size and feature dimensions grow proportionally. The paper derives the precise limiting excess risk and characterizes the resulting risk curve behavior.\"},{\"question\":\"What generalization result does the paper claim for the multiple random feature model (MRFM)?\",\"answer\":\"For the MRFM that ensembles K types of random features, the paper shows that the risk curve can exhibit (K+1)-fold descent, generalizing the earlier double/triple descent behavior.\"}]","Multiple Descent in the Multiple Random Feature Model | PDF",1785677117,123,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"multiple-descent-in-the-multiple-random-feature-model","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/multiple-descent-in-the-multiple-random-feature-model/117586/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What phenomenon does the paper investigate in multi-component prediction models?","Question",{"text":75,"@type":76},"The paper investigates the multiple descent phenomenon, where the excess risk changes non-monotonically as model capacity grows, potentially showing multiple dips (double, triple, or higher-fold).","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How is the double random feature model (DRFM) constructed and analyzed?",{"text":80,"@type":76},"The DRFM concatenates two types of random features and is studied in ridge regression under a high-dimensional limit where sample size and feature dimensions grow proportionally. The paper derives the precise limiting excess risk and characterizes the resulting risk curve behavior.",{"name":82,"@type":73,"acceptedAnswer":83},"What generalization result does the paper claim for the multiple random feature model (MRFM)?",{"text":84,"@type":76},"For the MRFM that ensembles K types of random features, the paper shows that the risk curve can exhibit (K+1)-fold descent, generalizing the earlier double/triple descent behavior.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]