[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85059-en":3,"doc-seo-85059-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},85059,1099514067415,"Rowan","https://ap-avatar.wpscdn.com/avatar/100002539d78ffe74a7?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779092875211072502",8,"Research & Report","On the Limitations of Non-GPU AI Accelerators for Large-Model Inference","Non-GPU AI accelerators are increasingly used for large-model inference, yet the real migration cost from CUDA-based ecosystems remains underdocumented. This field study deploys two demanding workloads on a 16-device Huawei Ascend 910 system using CANN and vLLM-Ascend. It covers an MoE LLM-as-a-judge safety/evaluation pipeline and a multimodal medical vision–language benchmarking workload. Integration required source-level patches, correctness-focused disabling of throughput features, and operational scaffolding, leading to eight platform-level limitation classes.","arXiv :2607 .082 15v 1 [ cs .DC] 9 Jul 2026  \nOn the Limitations of Non-GPU AI Accelerators for Large-Model Inference: A Field Study of MoE and Multimodal Serving on Huawei Ascend  \nA Preprint  \nZheng Yu∗  \nSchool of Data Science  \nThe Chinese University of Hong Kong, Shenzhen  \nShenzhen, Guangdong, China  \nJuly 10, 2026  \nAbstract  \nNon-GPU AI accelerators are increasingly adopted as alternatives to general-purpose GPUs for large-model inference, motivated by supply, cost, and availability considerations. Yet the practical cost of migrating a demanding workload from the mature CUDA ecosystem to such an accelerator is poorly documented. We report a field study of deploying two concrete, demanding inference workloads on a 16-device Huawei Ascend 910 system using the CANN software stack and the vLLM-Ascend inference engine. The first is an LLM-as-a-judge valuealignment and safety evaluation pipeline: a weight-quantized (W8A8) Mixture-of-Experts (MoE) judge model, DeepSeek-V4-Flash (∼300 GB), scores the responses of twenty frontier LLMs (e.g., GPT-5.1, Gemini 3 Pro, Claude Haiku 4.5, DeepSeek-V3.2, Qwen3-Max, GLM- 4.6) at a scale of tens of thousands of prompts each. The second is a multimodal medical vision–language benchmarking workload: DeepSeek-V4-Flash-Vision (∼540 GB in bf16), a frozen MoE language decoder fused with a Qwen3.5 vision tower through a trained merger/bridge, evaluated on the MMMU and MMMU-Pro benchmarks. Bringing both workloadsto a serviceable state required twelve source-level patches to the vendor inference plugin, the deliberate disabling of several throughput features to preserve numerical correctness, and operational scaffolding to absorb recurring low-level device faults. We organize our observations into eight classes of platform-level limitation — software-stack maturity and operator/feature-coverage gaps, fragile multi-axis parallelism, kernel-level (aicore/vectorcore) numerical faults, immature graph compilation, unstable advanced features, performance and scalability ceilings, weak operational observability, and ecosystem fragmentation  \n— and, for each, give the symptom, evidence, and likely root cause. We quantify the integration effort, the observed concurrency behavior, and the end-to-end benchmark quality that confirms both workloads were served correctly, and we distill a set of general, vendoragnostic strategies for adopting alternative accelerators. Our aim is a reproducible reference for teams evaluating or operating this class of accelerator.  \nKeywords AI accelerators · NPU · large language models · inference serving · mixture-of-experts · vLLM · Huawei Ascend · machine learning systems  \n∗ Correspondence: [zheng.yu3@mail.mcgill.ca](zheng.yu3@mail.mcgill.ca).  \nContents  \n1 Introduction 3  \n2 Related Work 3  \n3 Background 4  \n3.1 Ascend DaVinci, CANN, HCCL, and torch_npu ......................... 4  \n3.2 vLLM-Ascend as a vendor plugin .................................. 4  \n3.3 Contrast with the CUDA ecosystem ................................ 4  \n4 Deployment Setup and Methodology 4  \n4.1 Case study A: an LLM-as-a-judge value-alignment service .................... 4  \n4.2 Case study B: a multimodal medical vision–language model ................... 5  \n5 Limitations 6  \n5.1 Software-stack maturity and operator/feature coverage gaps ................... 6  \n5.2 Fragile multi-axis parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7  \n5.3 Kernel-level numerical and stability faults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8  \n5.4 Immature graph compilation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9  \n5.5 Advanced-feature gaps ........................................ 9  \n5.6 Performance and scalability ceilings ................................. 9  \n5.7 Operational reliability and observability .............................. 10  \n5.8 Ecosystem fragmentation and portability tax ........................... 11  \n6 Disc","cbCaicZwp7RqS6vb","https://ap.wps.com/l/cbCaicZwp7RqS6vb","pdf",443796,2,1,14,"English","en",105,"# Introduction\n# Related Work\n# Background\n## Ascend DaVinci, CANN, HCCL, and torch_npu\n## vLLM-Ascend as a vendor plugin\n## Contrast with the CUDA ecosystem\n# Deployment Setup and Methodology\n## Case study A: LLM-as-a-judge value-alignment service\n## Case study B: multimodal medical vision–language model\n# Limitations\n## Software-stack maturity and operator/feature coverage gaps\n## Fragile multi-axis parallelism\n## Kernel-level numerical and stability faults\n## Immature graph compilation\n## Advanced-feature gaps\n## Performance and scalability ceilings\n## Operational reliability and observability\n## Ecosystem fragmentation and portability tax\n# Recommendations and General Strategies\n## Tactics for this stack\n## General strategies for adopting alternative accelerators\n# Conclusion","[{\"question\":\"What problem does the field study address?\",\"answer\":\"It examines the practical effort and hidden failure modes when migrating large-model inference workloads from the CUDA ecosystem to non-GPU AI accelerators. The focus is on what costs are not well captured in vendor documentation.\"},{\"question\":\"Which hardware and software stack are used for the deployments?\",\"answer\":\"The study runs workloads on a 16-device Huawei Ascend 910 system, using the CANN software stack and the vLLM-Ascend inference engine. The document also describes relevant components such as Ascend DaVinci, torch_npu, and the Ascend plugin.\"},{\"question\":\"What kinds of platform limitations are reported?\",\"answer\":\"The findings are organized into eight classes, including software-stack maturity/operator coverage gaps, fragile multi-axis parallelism, kernel-level numerical/stability faults, immature graph compilation, advanced-feature gaps, performance/scalability ceilings, weak observability, and ecosystem fragmentation/portability tax.\"}]",1784200709,35,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"on-the-limitations-of-non-gpu-ai-accelerators-for-large-model-inference","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/on-the-limitations-of-non-gpu-ai-accelerators-for-large-model-inference/85059/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the field study address?","Question",{"text":75,"@type":76},"It examines the practical effort and hidden failure modes when migrating large-model inference workloads from the CUDA ecosystem to non-GPU AI accelerators. The focus is on what costs are not well captured in vendor documentation.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Which hardware and software stack are used for the deployments?",{"text":80,"@type":76},"The study runs workloads on a 16-device Huawei Ascend 910 system, using the CANN software stack and the vLLM-Ascend inference engine. The document also describes relevant components such as Ascend DaVinci, torch_npu, and the Ascend plugin.",{"name":82,"@type":73,"acceptedAnswer":83},"What kinds of platform limitations are reported?",{"text":84,"@type":76},"The findings are organized into eight classes, including software-stack maturity/operator coverage gaps, fragile multi-axis parallelism, kernel-level numerical/stability faults, immature graph compilation, advanced-feature gaps, performance/scalability ceilings, weak observability, and ecosystem fragmentation/portability tax.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]