Next generation Co-Packaged Optics Technology to Train & Run Generative AI Models in Data Centers and other computing applications

Q: Why does data center traffic drive demand for next-generation optics and networking equipment?

Data-center traffic is increasingly concentrated within data centers, which accelerates the need for higher port density. Traditional copper links suffer signal degradation over distance, and conventional pluggable optics bandwidth grows too slowly compared with traffic needs.

Q: What problem does Co-Packaged Optics (CPO) target compared with conventional approaches?

CPO increases interconnection bandwidth density and energy efficiency by dramatically shortening electrical link length. It does so via advanced packaging and cooptimization of electronics and photonics, reducing I/O power and enabling high-density optical connectors.

Q: How does distributed training affect performance when training large language models?

Distributed training requires device-to-device communication to transfer, reduce, aggregate, or exchange information. Communication can become the critical path, limiting throughput scaling as device count increases and leaving compute resources idle.

Nov 2, 2026

Uploaded by Fahsai

Next generation Co-Packaged Optics (CPO) addresses the data-center bottleneck of limited electrical link length and insufficient optics bandwidth scaling as generative AI grows. The approach shortens intra-package signaling distances through co-packaging and cooptimizing electronics with photonics, improving energy efficiency and increasing interconnection density. The paper motivates CPO by contrasting rapid compute scaling with slower I/O bandwidth growth and evaluates CPO benefits for training large language models, including distributed training communication effects and measured throughput impacts across tensor-parallel degrees.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages13

Reading Time33 min

AccessFree to download

Introduction

Training Large Language Models

Next generation Co-Packaged Optics Technology to Train & Run Generative AI Models in Data Centers and other computing applications John Knickerbocker4, Jean Benoit Heroux1, Griselda Bonilla1, Hsiang Hsu3, Neng Liu3, Adrian Paz Ramos1, Francois Arguin1, Yan Tribodeau1, Badr Terjani1, Mark Schultz4, Raghu Kiran Ganti3, Linsong Chu3, Chinami Marushima2, Yoichi Taira2, Sayuri Kohara2, Akihiro Horibe2, Hiroyuki Mori2 and Hidetoshi Numata2 1 IBM Bromont, 23 [boul. de](boul. de) l’aeroport, Bromont, QC J2L 1A3, Canada 2IBM Japan, 7-7 Shin-Kawasaki, Saiwai-ku, Kawasaki 212-0032, Japan 3IBM Albany, 257 Fuller Road, Albany, NY 12203 4IBM T. J. Watson Research Center, 1101 Kitchawan Rd, Yorktown Heights, NY 10598 Introduction The need to manage large amounts of data quickly and efﬁciently is boosting the demand for high-speed data transfer in data centers. The emergence of Generative AI has further fueled the demand for high-speed data transfer such that nearly threefourths of the data center trafﬁc resides within data centers [1]. The growth in trafﬁc accelerates the need for next-generation networking equipment to support higher port density. However, traditional copper cables used for data transfer are limited by signal degradation over long distances. This, in turn, drives the requirements for large-scale deployment of high-speed optics to connect the various layers of the networking equipment. Today, data centers rely heavily on optics, but not for the short to midlength (< 2m) interconnections. The conventional pluggable optics bandwidth increases at a much slower rate than that of data center trafﬁc and the gap between application requirements and the capability of conventional pluggable optics keeps increasing, a trend that is unsustainable. Co-packaged optics (CPO) is a disruptive approach to increasing the interconnection bandwidth density and energy efﬁciency by dramatically shortening the electrical link length through advanced packaging and cooptimization of electronics and photonics. Advances in compute performance have beneﬁtted from Moore’s law scaling and have seen up to 60000x in performance in the past 20 years, as shown in Figure 1. However, the I/O bandwidth has increased by a mere 30x in that same timeframe. The electrical signal rate increase needs to signiﬁcantly advance to enable the signals to enter/exit and, in addition, there is the concomitant challenge to further move the electrical signals across a card to the front panel of the router or switch, depending on the application. To solve this challenge, the industry will come to rely on the integration of silicon and optics by co-packaging optical engines along with the main compute chip on a common substrate. This reduces input/output (I/O) power by limiting electrical signaling to intra-package distances, reduces cost by increasing channel count per optical sub-assembly and by eliminating discrete transceiver packaging as well as high-speed PCB traces, and enables high-density optical faceplate connectors. Figure 1 Scaling of the bandwidth (BW) of different generations of interconnectionsand memory, as well as the hardware (HW) Peak FLOPS. Note the slow increase in interconnect bandwidth compared to HW FLOPS. Training Large Language Models Generative AI is transforming society, largely enabled by more efﬁcient hardware, larger datasets, and algorithmic advances, including exponential model size growth that further beneﬁts from more efﬁcient hardware and larger datasets. However, models are scaling rapidly (1000×) and are imposing signiﬁcant computational demands with increasing model sizes. Developers have thereby relied on distributed training, where multiple accelerators (e.g., GPUs) harness their collective memory capacities and compute capabilities to collaboratively train a large language model (LLM) . Distributed training techniques have associated communication between devices to transfer, reduce, aggregate, or exchange infor

FAQs

Why does data center traffic drive demand for next-generation optics and networking equipment?

Data-center traffic is increasingly concentrated within data centers, which accelerates the need for higher port density. Traditional copper links suffer signal degradation over distance, and conventional pluggable optics bandwidth grows too slowly compared with traffic needs.

What problem does Co-Packaged Optics (CPO) target compared with conventional approaches?

CPO increases interconnection bandwidth density and energy efficiency by dramatically shortening electrical link length. It does so via advanced packaging and cooptimization of electronics and photonics, reducing I/O power and enabling high-density optical connectors.

How does distributed training affect performance when training large language models?

Distributed training requires device-to-device communication to transfer, reduce, aggregate, or exchange information. Communication can become the critical path, limiting throughput scaling as device count increases and leaving compute resources idle.

Nov 2, 2026

Uploaded by Fahsai