Lab Guide 7 — Super Spines: beyond a single-pod CLOS

In Lab 1 you built a 2-tier CLOS. In Labs 2–3 you stretched a single L2 segment across it and ran a real AllReduce on top. In Lab 4 you watched the per-link Mbps fill in as the collective ran.

Everything you've built so far fits inside a single pod. Two spines, four leaves, eight workers — one rectangle of bandwidth. That rectangle has a hard ceiling: the radix of the switches it's built from. Once you run out of leaf-facing ports on the spines, the only way to grow is up — add a third tier above the spines.

That third tier is the super spine. This lab walks through what it is, when hyperscalers actually reach for it, what its BGP plane looks like, and why it matters for AI training fabrics specifically. Then you'll inspect your existing 2-tier fabric to ground the math.

What this lab is — and isn't. Lab 7 is a conceptual lab. There are no super-spine containers deployed in this platform; the existing 2-spine / 4-leaf / 8-worker pod is the surface you'll discuss against. You'll read, run a few show commands to anchor the math, and click through three inspection checkpoints. There's nothing to type and break. The "Solve ✓" button is a no-op re-apply of the same baseline. See notes/decisions.md ADR-013 for the rationale.

By the end of this lab you'll be able to answer, on a whiteboard:

1. Why does any AI fabric ever need more than a 2-tier CLOS?
2. At what scale does a 2-tier CLOS stop being enough?
3. What does the third tier look like — physically, in BGP, and in failure semantics?
4. What changes for the workload (NCCL/Gloo collectives) when traffic crosses pods vs. stays inside one?

What you'll learn

Concepts, not CLI flags:

• Radix and the pod ceiling. Why "32-port switches → ~1024 GPUs/pod" isn't a coincidence — it's the arithmetic of CLOS.
• Why a 3rd tier exists. Multi-pod scaling, fault-domain isolation, blast-radius reduction, multi-tenant separation.
• The shape of 3-tier BGP. Adding another eBGP hop, what the AS numbering pattern looks like, why super spines are usually a shared-AS tier (same teaching choice as your spines today — ADR-002).
• Why super spines transit underlay-only by default. The EVPN routes still flow end-to-end, but the super-spine tier doesn't need to participate in EVPN signaling for a pure cross-pod underlay design — it's a routing-only relay.
• What it means for collectives. When does an AllReduce stay in-pod and when does it cross the super spine? Why hyperscalers schedule training jobs to minimize cross-pod traffic.
• What failure looks like at this scale. A spine failure inside a pod converges over the pod's leaves; a super-spine failure spans pods. The blast radius is deliberately layered.

💡 Why this matters in AI DCs. Every public AI fabric paper (Meta RSC, Microsoft GPU pods, Google's TPU networks, AWS HyperPod) eventually shows the same shape: tier-1 leaves, tier-2 spines, tier-3 super spines (sometimes called aggregation or core). The first time you see a 3-tier diagram you should be asking "at what scale did they need this?" — that's the question this lab answers.

Teaching philosophy

Labs 1–3 were build labs — you typed config, the fabric came up, ping worked. Lab 4 was an observe lab — you watched the dashboard fill. Lab 7 is a reason lab — most of your time is reading and thinking, not typing. Where you do open a console, you're inspecting state that already exists to anchor a concept (show bgp summary to count the current spine fan-out; show ip route ... json to count today's ECMP paths). Each inline Check ▸ widget passes against the healthy fabric — clicking it is the moment the guide says "you've now observed the thing I just explained."

This is deliberate. Network design at hyperscale is mostly whiteboarding — the actual vtysh commands are the small last step. A lab that's all typing would mis-teach the topic.

Prerequisites

• Labs 1–4 complete (or at least Lab 4's fabric state). Lab 7 leaves the fabric exactly where Lab 4 did: workers on 192.168.100.0/24, EVPN-VXLAN overlay live, all 8 BGP sessions Established. If anything is sick, click Start ▶ to re-apply _overlay_workers.
• Open the Lab 4 dashboard in another tab — when we talk about "per-pod ECMP" it'll be useful to flip over and look at real link rates.
• Keep ../topology.md handy for the existing IP / link / ASN scheme.

The current 2-tier addressing scheme (what the math is anchored to)

Before we walk into a 3rd tier, anchor on what you already have:

This is 1 pod. Spine1 burns all 4 fabric-facing ports on leaves; the pod can't grow without bigger spines or another tier. That's the constraint Lab 7 unpacks.

The workflow loop

For each section of the exercise:

1. Read the conceptual content. The scale math is the load-bearing part.
2. Run the indicated inspection command on the indicated console (spine1 or leaf1). Confirm what the guide claimed.
3. Click Check ▸ to record the observation.

At the end, Submit ✓ runs all the inspection checks plus the 56-pair worker ping mesh — a regression guard that the conceptual walk didn't perturb anything.

Persistence note

This lab doesn't change fabric state. There's nothing to persist or roll back. Start ▶ and Solve ✓ both re-apply the same _overlay_workers baseline; Reset ↺ is identical. You can click any of them at any time without losing anything.

Where to go

• lab7-exercise.md — the guided walkthrough. Start here.
• lab7-solution.md — radix-math worksheet answers + reference FRR config blocks a real super-spine tier would use (for when you want to see the actual BGP shape, not just discuss it).

Reference material to keep open in another tab:

• ../topology.md — current 2-tier IP / link / ASN scheme.
• ../../notes/decisions.md — ADR-002 (shared-AS spines, why the same choice would apply at the super-spine tier), ADR-013 (why this lab is conceptual, not deployed).
• Lab 4 Grafana — useful for "per-pod ECMP" discussion.

Exercise — Super spines: when one pod isn't enough

Read lab7-overview.md first if you haven't.

Scenario

You're operating a healthy AI training pod: 2 spines, 4 leaves, 8 GPUs. AllReduce runs cleanly, the Grafana dashboard fills, your boss is happy. Then product comes back: "we need to add another 1000 GPUs by Q3."

Can your current pod absorb that? Almost certainly not — and the reason is switch radix. This lab walks the math, then walks what you'd build instead. You won't deploy anything new; the existing fabric stays exactly where Lab 4 left it. You'll inspect it to anchor each conceptual point.

Get to the starting line

Click Start lab ▶ in the top bar. The orchestrator re-applies the _overlay_workers baseline (same as Lab 4's solved state) — workers on 192.168.100.0/24, EVPN-VXLAN live, all BGP sessions Established. Within ~30 seconds the lab status pill flips to In progress.

Open consoles for spine1 and leaf1 via the Topology button — you'll alternate between them.

Step 1: The radix wall

A CLOS fabric's size is bounded by the radix of its switches — how many ports each one has. The pod ceiling falls out of one equation:

Max workers per pod = (leaf-facing ports per spine) × (worker-facing ports per leaf)

Walk it for your current fabric. On the leaf1 console:

docker exec leaf1 ip -br link show | grep -E '^eth[1-4]' | head -10

You'll see four fabric/worker-facing veths:

• eth1 → spine1 (fabric-side)
• eth2 → spine2 (fabric-side)
• eth3 → gpu1 (worker-side)
• eth4 → gpu2 (worker-side)

So each leaf has 2 worker ports. Now on spine1:

docker exec spine1 ip -br link show | grep -E '^eth[1-4]' | head -10

Same shape: eth1..eth4 → leaf1, leaf2, leaf3, leaf4. 4 leaf-facing ports per spine.

Plug into the equation:

max workers per pod = 4 (leaf-facing on spine) × 2 (worker-facing on leaf) = 8 workers

That's the size of this pod. You're already at the ceiling.

💡 Why this matters in AI DCs. Real production switches are 32-port to 128-port (with some 256-port silicon in newer designs). The same equation scales: 32-port spines × 32-port leaves = 1024 workers per pod. Above that — and any meaningful AI training cluster is above that — you can't add another rack of GPUs without either replacing every switch (expensive, ops-painful) or adding a third tier. This is the moment the super spine enters the diagram.

Sanity-check that you started from a healthy 2-tier fabric:

Step 2: Read the current spine fan-out

Your spine is the radix-bounded thing. Watch what it currently sees. On the spine1 console:

vtysh -c "show bgp ipv4 unicast summary"

Look at the Neighbor / V / AS / State columns. Expected:

Neighbor    V    AS      ... State/PfxRcd
10.1.1.1    4    65101   ... <int>   (leaf1)
10.1.1.3    4    65102   ... <int>   (leaf2)
10.1.1.5    4    65103   ... <int>   (leaf3)
10.1.1.7    4    65104   ... <int>   (leaf4)

Four leaf sessions, all Established (PfxRcd is an int, not "Active" or "Idle"). spine1 has no more fabric-facing capacity in this image — you'd need to either (a) renumber onto a switch with more ports or (b) add another spine alongside it.

💡 Why ipv4 unicast and not just show bgp summary? Without an address-family qualifier, FRR prints every AF — IPv4 unicast AND L2VPN-EVPN. On this fabric every neighbor exists in both AFs, so the bare show bgp summary lists each neighbor twice. Scoping to ipv4 unicast is what makes "4 rows = 4 leaves" line up.

(a) is the "scale up" path — bigger silicon, more expensive per port, eventually hits its own ceiling. (b) is the "scale out" path — but to wire a 5th leaf into the pod, you'd need a 5th port on every existing spine too, and you're back to (a).

The way out is to add a tier above the spines. The spines stop being the top — they become the middle. Above them, a new layer of "super spines" connects multiple pods. Each pod retains its own 2-tier internal CLOS; super spines stitch pods into one fabric.

💡 Why this matters in AI DCs. Real GPU training jobs map onto pods. A small job (a few hundred GPUs) fits in one pod — its collectives ride the existing 2-tier CLOS, no cross-pod traffic. A large job (thousands of GPUs) spans pods — collectives cross the super-spine layer. The scheduler that places jobs is acutely aware of this distinction; the super spine is the fabric primitive that makes "multi-pod jobs" possible without making them invisible.

Step 3: ECMP today, and what a 3rd tier extends it to

Inside your current pod, leaf-to-leaf traffic ECMPs across both spines. Confirm it. On the leaf1 console:

vtysh -c "show ip route 10.0.10.3"

You'll see leaf3's VTEP loopback reachable via two nexthops:

B>* 10.0.10.3/32 [20/0] via 10.1.1.0, eth1, weight 1, ...
  *                    via 10.1.2.0, eth2, weight 1, ...

Two ECMP paths — one through spine1, one through spine2. Per-flow load-balancing across both. This is the per-pod ECMP that Lab 4's Grafana dashboard makes visible when you ran the 8-rank AllReduce: traffic from each leaf splits across eth1 (spine1) and eth2 (spine2).

Now picture the 3-tier extension. In a multi-pod build:

                  +----- supersp1 -----+----- supersp2 -----+
                  |        |           |        |           |
              +---+--------+--+    +---+--------+--+        ...  more pods
              |  pod-1 spines |    |  pod-2 spines |
              +---------------+    +---------------+
              |  pod-1 leaves |    |  pod-2 leaves |
              +---------------+    +---------------+
              | pod-1 workers |    | pod-2 workers |
              +---------------+    +---------------+

Traffic from a pod-1 worker to a pod-2 worker:

• worker → leaf (in pod-1)
• leaf → spine (pod-1) — 2 ECMP paths within the pod (this is what you just saw)
• spine → super spine — N ECMP paths to N super spines
• super spine → spine (pod-2)
• spine → leaf → worker (in pod-2)

The ECMP picture you have today (2 paths per source) is a per-pod view. The 3rd tier adds another ECMP axis on top — per-pod-pair ECMP across the super-spine layer. Same load-balancing principle, one tier up.

💡 Why this matters in AI DCs. AllReduce ring algorithms produce N² TCP flows (every rank talks to every other rank). With per-flow ECMP at both tiers, those flows spread across every available physical path — that's how a single AllReduce step can saturate dozens of links simultaneously. Without per-flow ECMP (or with broken hashing — see CLAUDE.md pitfall #18), the collective serializes onto a single path and bandwidth craters. Real AI DCs treat ECMP hashing as a tier-0 correctness concern, not a perf nice-to-have.

Step 4: What the FRR config for a super spine would look like

If you were going to deploy supersp1 and supersp2 containers above the current spines, this is the BGP config they'd run. (You're not going to — this is reading, not typing. But it's worth seeing the actual shape; it's a clean extension of patterns you already used in Labs 1 and 2.)

supersp1 (proposed AS 64999):

frr defaults datacenter
hostname supersp1
log syslog informational
service integrated-vtysh-config
!
interface lo
 ip address 10.0.0.101/32
!
interface eth1
 description to_spine1
 ip address 10.3.1.0/31
!
interface eth2
 description to_spine2
 ip address 10.3.1.2/31
!
router bgp 64999
 bgp router-id 10.0.0.101
 bgp bestpath as-path multipath-relax
 no bgp default ipv4-unicast
 neighbor SPINES peer-group
 neighbor SPINES advertisement-interval 0
 neighbor SPINES timers 3 9
 neighbor 10.3.1.1 remote-as 65000
 neighbor 10.3.1.1 peer-group SPINES
 neighbor 10.3.1.1 description spine1
 neighbor 10.3.1.3 remote-as 65000
 neighbor 10.3.1.3 peer-group SPINES
 neighbor 10.3.1.3 description spine2
 !
 address-family ipv4 unicast
  network 10.0.0.101/32
  maximum-paths 64
  neighbor SPINES activate
  neighbor SPINES soft-reconfiguration inbound
 exit-address-family
!
line vty
!

Spine1 would gain a matching block — two more interface stanzas (eth5, eth6) and a SUPERSPINES peer-group with the two super-spine neighbors.

Notable design choices in that block:

• AS 64999 is shared between supersp1 and supersp2 — same teaching choice as your two spines today (ADR-002). Loops are still prevented by standard own-AS rejection.
• No address-family l2vpn evpn on the super spines. The super spine is a routing-only relay between pods. EVPN routes can still ride through it (the underlay carries them as IPv4 unicast next-hops), but the super spine doesn't participate in EVPN signaling. Production designs differ — some put route reflectors at this tier — but for our "scale out the underlay" thesis, EVPN-off is the simpler, more honest shape.
• 10.3.0.0/16 /31 block for supersp↔spine links. The lab's 10.2.x.x block is taken by worker /31s (workers/entrypoint.sh and orchestrator/api/labruns.py); 10.3 is the next clean choice.

The full set of config blocks (spine1, spine2, supersp1, supersp2) lives in lab7-solution.md Appendix A as reference material.

💡 Why this matters in AI DCs. A real super-spine deployment is not architecturally interesting — the BGP shape is just "another tier of the same pattern you already understand." What's interesting is the operational story: bringing a new tier up without disrupting in-pod traffic, planning the failure modes (a super-spine outage is multi-pod; an in-pod-spine outage is single-pod), and scheduling jobs to land on pods rather than spanning them. Most of those concerns aren't in the BGP config — they're in the orchestrator above it.

Step 5: Submit ✓

Click Submit ✓ in the top bar. The orchestrator runs:

1. The three inspection checks you've already clicked through, all over again (they should still pass — the fabric hasn't changed).
2. The full 56-pair worker ping mesh across the overlay (8 sources × 7 destinations). This is the regression guard: a conceptual lab shouldn't perturb fabric state. If the mesh still pings 56/56, you've completed Lab 7 with zero collateral damage.

If everything passes, the lab stamps as Passed, the completion screen appears, and the CTA for Lab 8 ("Inject Failure During AllReduce") lights up — though Lab 8 itself is coming-soon for now.

Stuck?

Because this lab doesn't change fabric state, all four buttons are mostly equivalent here. Start, Reset, and Solve all re-apply _overlay_workers. Submit runs the checks but changes nothing.

Where to go next

• lab7-solution.md — radix-math worksheet answers + reference FRR config blocks (the full would-be _super_spine/ state)
• ../topology.md — current 2-tier IP / link / ASN reference
• ../../notes/decisions.md — ADR-002 (why the super-spine tier would also be shared-AS), ADR-013 (why Lab 7 teaches super spines conceptually rather than deploying them)
• Lab 8 — Inject Failure During AllReduce — coming-soon. Failure injection at the spine tier is the natural follow-on to "what does each tier of the CLOS protect against?"