Tunneling Intelligence: Scaling Data Movement for Trillion-Parameter AI

About the Session

As AI models scale to and beyond the trillion-parameter threshold, data movement becomes a defining challenge. Training models like PaLM and Gopher requires moving over 750 TB of gradient data per iteration across massive distributed clusters. Yet, the performance of AI pipelines often collapses under flat tunneling architectures—incurring up to 37% latency overhead, 43% bandwidth waste, and 38% signaling loss. This session introduces Hierarchical Advanced Tunneling Architecture (HATA)—a four-tier framework (Core, Distribution, Access, Virtual Overlay) designed for high-throughput, low-latency, and fault-tolerant data flows in hyperscale AI infrastructure. Tested across 16,384-node clusters, HATA achieved a 78.3% reduction in signaling overhead, 62.4% shorter routing paths, and a 42.5% boost in throughput. It integrates seamlessly with large-scale training platforms and data pipelines by leveraging control-plane offload, dynamic hierarchy depth, and intelligent caching (87.3% hit rate using 256 GB). Attendees will learn how HATA redefines AI data plane architecture—enabling faster, more reliable model training, real-time inference, and improved energy efficiency across GPU/DPU-based systems.

Speaker

Harish Kumar Chencharla Raghavendra

Software Engineer