[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84553-en":3,"doc-seo-84553-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84553,34359740700684,"Finn","https://ap-avatar.wpscdn.com/avatar/1f400023980c374ae676?_k=1777273430885731487",8,"Research & Report","Does Your ViT Still Need U-Net for Segmentation","Medical image segmentation is dominated by U-Net-style encoder–decoder architectures. Vision Transformers (ViTs) use self-attention to model long-range dependencies, yet early ViT segmentation methods often kept U-Net decoders because pretrained ViT features were not strong enough for dense prediction. Large-scale pretraining has since improved ViT representation capability, motivating two questions about whether U-Net is still needed and how to build an encoder-only framework. The work introduces EoSeg, a query-based encoder-only design with multi-level query modeling and learnable block fusion, validated on seven datasets across CT, MRI, histopathology, endoscopy, and dermoscopy.","Does Your ViT Still Need U-Net for Segmentation?  \nXin Li 1 , Wenhui Zhu 1 , Xuanzhao Dong 1 , Xiwen Chen2 , Yanxi Chen 1 Yujian Xiong 1 , Hao Wang2 , Oana M. Dumitrascu3 , Yalin Wang 1  \n1Arizona State University, Tempe, AZ, USA  \n2 Clemson University, Clemson, SC, USA  \n3Mayo Clinic, Scottsdale, AZ, USA  \narXiv :2607 .00223v1 [ cs .CV] 30 Jun 2026  \nAbstract  \nMedical image segmentation is dominated by U-Net-style encoder–decoder architectures. Vision Transformers (ViTs) overcome the limited receptive field of convolutional networks through self-attention, enabling modeling of longrange dependencies. Early ViT-based segmentation methods typically retained U-Net-style decoders because pretrained ViT representations were insufficient to support accurate dense prediction. Recent advances in large-scale pretraining have redefined the representation capability of ViTs, reducing the reliance on UNet-style decoder architectures in modern vision models. This prompts two questions: Is the U-Net paradigm still necessary for medical image segmentation? If not, how should an encoder-only segmentation framework be designed? Motivated by these questions, we explore key architectural choices for encoderonly medical image segmentation based on modern ViT backbones and establish a query-based encoder-only design with multi-level query modeling and learnable block fusion, realized in Encoder-only Segmentation (EoSeg). Extensive experiments across seven benchmark datasets spanning CT, MRI, histopathology, endoscopy, and dermoscopy validate the effectiveness of the proposed design across diverse medical imaging modalities, including mDice scores of 85 .50% on Synapse, 91.73% on ACDC, and 93.27% on GlaS. The results demonstrate that a U-Net-style decoder is no longer necessary for medical image segmentation with modern ViT backbones and further show that EoSeg provides an effective encoder-only design. Code is available at: [https:](https:)// [github.com/Retinal-Research/EoSeg](github.com/Retinal-Research/EoSeg).  \n1. Introduction  \nMedical image segmentation is a fundamental task in medical image analysis, providing the basis for a wide range of clinical applications such as organ delineation, lesion assessment, treatment planning, and disease monitoring [2,  \nFigure 1 . Evolution of medical image segmentation architectures. As representation capability evolves from local convolutional features to modern pretrained ViT backbones, segmentation frameworks transition from encoder–decoder architectures toward encoder-only designs.  \n25, 33, 43] . Over the past decade, medical segmentation frameworks have undergone several major architectural shifts, as illustrated in Fig. 1. Early fully convolutional networks (FCNs) [34] performed segmentation through pixelwise prediction using a simple prediction head. However, the downsampling operations often resulted in the loss of fine spatial details, making accurate delineation of anatomical structures challenging [33] . To address this limitation, U-Net introduced an encoder–decoder architecture with skip connections that allow high-resolution features tobe reused during mask generation [33] . This design substantially improved segmentation quality and quickly became the dominant paradigm in medical image segmentation [7, 11, 17, 19–21, 27, 33, 38, 43] . Despite its success, the standard-convolution is still constrained by the limited receptive fields, making it difficult to capture global contextual information [10] .  \nVision Transformers (ViTs) overcome the limited receptive field of convolutional networks through self-attention, enabling effective modeling of long-range dependencies.  \nThis capability quickly led to the adoption of ViTs in medical image segmentation. Early methods such as TransUNet [3] replaced the convolutional encoder with a ViT while retaining the U-Net decoder, whereas SwinUNet [2] further extended this paradigm with a pure Transformer architecture. Numerous subsequent variants [9, ","cbCaihyro8e2TY73","https://ap.wps.com/l/cbCaihyro8e2TY73","pdf",43071161,1,12,"English","en",105,"# Introduction\n## Architectural evolution in medical segmentation\n## ViT-based segmentation and decoder reliance\n## Motivation for encoder-only segmentation\n## Proposed approach (EoSeg)","[{\"question\":\"Why have U-Net-style encoder–decoder architectures dominated medical image segmentation?\",\"answer\":\"U-Net introduced skip connections that reuse high-resolution features for mask generation, significantly improving segmentation quality and becoming a dominant paradigm in medical imaging.\"},{\"question\":\"How do modern Vision Transformers change the need for U-Net-style decoders?\",\"answer\":\"Self-attention in ViTs enables modeling of long-range dependencies, and large-scale pretraining strengthens ViT representations, reducing reliance on decoder architectures for accurate dense prediction.\"},{\"question\":\"What is EoSeg and what design choices does it introduce?\",\"answer\":\"EoSeg is a query-based encoder-only segmentation framework that uses multi-level query modeling and learnable block fusion to perform effective segmentation without a U-Net-style decoder.\"}]",1784196676,30,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"does-your-vit-still-need-u-net-for-segmentation","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/does-your-vit-still-need-u-net-for-segmentation/84553/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why have U-Net-style encoder–decoder architectures dominated medical image segmentation?","Question",{"text":75,"@type":76},"U-Net introduced skip connections that reuse high-resolution features for mask generation, significantly improving segmentation quality and becoming a dominant paradigm in medical imaging.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How do modern Vision Transformers change the need for U-Net-style decoders?",{"text":80,"@type":76},"Self-attention in ViTs enables modeling of long-range dependencies, and large-scale pretraining strengthens ViT representations, reducing reliance on decoder architectures for accurate dense prediction.",{"name":82,"@type":73,"acceptedAnswer":83},"What is EoSeg and what design choices does it introduce?",{"text":84,"@type":76},"EoSeg is a query-based encoder-only segmentation framework that uses multi-level query modeling and learnable block fusion to perform effective segmentation without a U-Net-style decoder.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":28,"slug":121},"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]