[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-119920-en":3,"doc-seo-119920-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},119920,4810365810221,"Aurora","https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d",8,"Research & Report","Dynamic Management of Distributed Machine Learning Projects - paper","Machine learning requirements have intensified in recent years due to rapidly growing, highly diverse, and fast-changing data, creating pressure on infrastructure and update costs. The document presents CEDEs, a distributed learning system deployed on an Hadoop cluster and designed around blocks, replication, and balancing. CEDEs trains models using data locality, and uses an optimization module to adapt ensemble components as data evolves, enabling continuous model evolution.","Dynamic Management of Distributed Machine Learning Projects  \nFilipe Oliveira and André Alves and Hugo Moço and José Monteiro and Óscar Oliveira and Davide Carneiro and Paulo Novais  \nAbstract Given the new requirements of Machine Learning problems in the last years, especially in what concerns the volume, diversity and speed of data, new approaches are needed to deal with the associated challenges. In this paper we describe CEDEs-a distributed learning system that runs on top of an Hadoop cluster and takes advantage of blocks, replication and balancing. CEDEs trains models ina distributed manner following the principle of data locality, and is able to change parts of the model through an optimization module, thus allowing a model to evolve over time as the data changes. This paper describes its generic architecture, details the implementation of the first modules, and provides a first validation.  \n1 Introduction  \nDespite all the recent advances in the field of Machine Learning (ML) and related supporting areas, many new challenges keep emerging [1] . These stem mostly from the volume and diversity of data in current ML problems, as well as from the need to deliver services in real-time, which led to the emergence of streaming data, streaming analytics [2] and streaming ML [3] .  \nML applications nowadays require such a volume that datasets must be distributed across clusters. Consequently, ML algorithms need to learn in a distributed manner, from multiple sources of data. Moreover, these data change over time, leading to the need for models tobe updated or fully retrained, which has an increasingly significant cost on the organizations’ infrastructure.  \nFilipe Oliveira, André Alves, Hugo Moço, José Monteiro, Óscar Oliveira, Davide Carneiro CIICESI,ESTG, Politécnico do Porto, Portugal, e-mail: {fvol,afba,hmsm,jmgm,oao,dcarneiro}@[estg.ipp.pt](estg.ipp.pt)  \nPaulo Novais  \nAlgoritmi Center/Department of Informatics, University of Minho, Portugal e-mail: [pjon@di.uminho.pt](pjon@di.uminho.pt)  \n2 Authors Suppressed Due to Excessive Length  \nIn this paper we describe the initial work being developed in the context of the CEDEs project - Continuously Evolving Distributed Ensembles. CEDEs aims to build an environment for the distributed training of ML models, in which the models can evolve over time as data change, in a cost-effective way. Therefore, not only does it address the issue of learning from large datasets, but also that of learning continuously from streaming data.  \nCEDEs takes advantage of existing block-based distributed file systems, such asthe Hadoop Distributed File System (HDFS) [4], to parallelize and distribute learning tasks, following the principle of data locality. That is, the computation is moved to where the data reside, rather than the other way around [5] . Moreover, instead of more traditional models, CEDEs uses Ensembles [6]: a base model is trained for a block of a dataset, where the data resides. Then, Ensembles are built in real time by combining available models according to criteria such as their performance or the state of the nodes where the base models reside. This is done by the optimization module that adapts the Ensemble as data changes.  \nThe system also makes use of two other mechanisms: replication and balancing. Replication allows to automatically create replicas of blocks so that the same block is available in multiple locations simultaneously. This increases the use of storage space, but makes it easier for the optimization module to find suitable nodes for carrying out tasks. Balancing, on the other hand, distributes the blocks evenly across the cluster, so that the load when executing tasks is appropriately distributed.  \nFinally, CEDEs also stores the base models themselves in the HDFS, which are then replicated. This means that, not only, the Ensemble itself is distributed, which speeds up predictions, but also that, thank to replication, multiple nodes will have the same base models a","cbCaiu4llmqcsDR7","https://ap.wps.com/l/cbCaiu4llmqcsDR7","pdf",574239,1,10,"English","en",105,"# Introduction\n## Motivation and challenges\n# CEDEs project overview\n## Continuously Evolving Distributed Ensembles\n# System mechanisms\n## Replication and balancing\n## Distributed ensemble predictions\n# Conceptualization\n## Data model","[{\"question\":\"What problem does CEDEs address in distributed machine learning?\",\"answer\":\"CEDEs targets the challenges created by large, diverse, and streaming datasets, where models must evolve over time without prohibitive retraining costs.\"},{\"question\":\"How does CEDEs achieve distributed training?\",\"answer\":\"It runs on an Hadoop cluster using block-based storage (e.g., HDFS) and follows data locality by moving computation to where dataset blocks reside.\"},{\"question\":\"What are replication and balancing used for in CEDEs?\",\"answer\":\"Replication creates block replicas across multiple locations to increase flexibility for selecting nodes, while balancing spreads blocks evenly across the cluster to distribute execution load appropriately.\"}]","Dynamic Management of Distributed Machine Learning Projects - paper | PDF",1785727002,25,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"dynamic-management-of-distributed-machine-learning-projects-paper","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/dynamic-management-of-distributed-machine-learning-projects-paper/119920/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-03",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does CEDEs address in distributed machine learning?","Question",{"text":75,"@type":76},"CEDEs targets the challenges created by large, diverse, and streaming datasets, where models must evolve over time without prohibitive retraining costs.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does CEDEs achieve distributed training?",{"text":80,"@type":76},"It runs on an Hadoop cluster using block-based storage (e.g., HDFS) and follows data locality by moving computation to where dataset blocks reside.",{"name":82,"@type":73,"acceptedAnswer":83},"What are replication and balancing used for in CEDEs?",{"text":84,"@type":76},"Replication creates block replicas across multiple locations to increase flexibility for selecting nodes, while balancing spreads blocks evenly across the cluster to distribute execution load appropriately.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,134],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":21,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":21,"slug":133},"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]