AI Datacenter EVPN-VXLAN Fabric

Configure a 2-spine / 4-leaf CLOS fabric from scratch via the in-browser console. Learn shared-AS spines, per-leaf ASNs, ECMP with multipath-relax, peer-groups, fast BGP timers — and why each choice matters for AI workloads.

• eBGP CLOS underlay design
• Shared-AS spines vs per-leaf ASNs
• ECMP with bestpath as-path multipath-relax
• Peer-groups for operational scale
• Fast BGP timers (3s/9s) for training-job continuity

Stretch a single L2 segment (VNI 10100, subnet 192.168.100.0/24) across all four leaves on top of the working underlay. Bring up Linux bridge + VXLAN devices, activate the BGP L2VPN-EVPN address family, watch Type-2 MAC routes traverse the fabric, and confirm overlay traffic actually flows leaf-to-leaf.

• BGP L2VPN-EVPN address family + advertise-all-vni
• Why spines need next-hop-unchanged in shared-AS CLOS EVPN
• Pre-allocated VTEP loopbacks as the data-plane endpoint
• Type-2 MAC routes vs Type-3 inclusive-multicast routes
• Linux kernel VXLAN + bridge setup that FRR discovers
• How an overlay rides on top of an underlay — proven by ping

Bring the eight GPU workers onto the stretched L2 segment you built in Lab 2. Attach each leaf's worker ports as VLAN 1000 access ports, move every worker's eth1 onto 192.168.100.0/24, and run a real Gloo AllReduce — first 2-rank across two leaves, then full 8-rank across the fabric.

• VLAN access ports — bridging worker veths into a VXLAN-backed L2 segment
• Why an AI fabric prefers one flat L2 over per-leaf /31 underlay for the GPU plane
• Gloo rendezvous + ring AllReduce as the canonical CPU collective
• How a single AllReduce step maps to N² TCP flows over VXLAN tunnels
• Why dropping the leaf-side L3 IP on a worker port is what makes the access-port story work
• Reading effective AllReduce bandwidth as a fabric-health signal

Bring up a gNMI streaming-telemetry pipeline (gnmic → Prometheus → Grafana) over your working fabric. Watch real-time per-link Mbps fill in as you run AllReduces — see ECMP load-spread across both spines, see worker ports light up, build operator intuition for what a healthy AI training fabric looks like in motion.

• gNMI streaming telemetry vs SNMP polling — the AI-DC default
• OpenConfig interface counters as the canonical wire-rate signal
• Why a virtual SONiC fabric needs a kernel-netdev side channel (and what that means for real hardware)
• Reading per-link Mbps to confirm AllReduce phase activity
• ECMP load-spread across spines, visible as nearly-equal traffic on parallel paths
• Why Linux multipath default breaks VXLAN ECMP — and the one sysctl that fixes it
• Where Prometheus + Grafana fit in an AI-fabric operator stack

Walk the math behind multi-pod AI fabrics: when does a 2-tier CLOS run out of room, what does the third tier look like in BGP, and why do hyperscalers schedule training jobs to live inside a pod when they can? Conceptual lab — no super spines are deployed; you inspect your existing fabric to anchor each point.

• Switch radix → pod ceiling — why 32-port silicon caps a pod at ~1024 GPUs
• When hyperscalers add a 3rd tier (super spines) — multi-pod scale, fault-domain isolation, multi-tenant separation
• The shape of 3-tier BGP — shared-AS super-spine tier, eBGP loop prevention without allowas-in, per-pod-pair ECMP across the new layer
• Why super spines transit underlay-only in the simple case — and what changes if you bring EVPN to that tier
• What in-pod vs cross-pod traffic means for AllReduce — and why job schedulers care about pod placement