Lab Guide 6 — Build an SRv6 uSID transport across your fabric

In Labs 1–4 you built an IPv4 CLOS underlay, stretched an EVPN-VXLAN overlay across it, put GPUs on that overlay, and watched real AllReduce traffic light up a telemetry dashboard. Then in Lab 5 you stood up a dual-stack IPv6 underlay alongside it — a numbered IPv6 BGP session per spine link (fc00:<spine>:<leaf>::/127), advertising each leaf's uSID locator /48 with ECMP via both spines. Everything so far rides one transport: VXLAN-over-IPv4-UDP.

This lab introduces a second, modern transport — SRv6 (Segment Routing over IPv6) — and lays it down additively, alongside what you already have. Your IPv4 underlay and your EVPN-VXLAN overlay keep running untouched. On top of them you'll dual-stack the fabric with IPv6, define micro-SID (uSID) locators, and build an SRv6 transport that carries leaf-to-leaf traffic and — the payoff — load-spreads per-flow across both spines using the IPv6 flow label, exactly the way VXLAN did in Lab 4, but with a completely different encapsulation.

By the end, a packet from a host behind sleaf1 to a host behind sleaf3 will be wrapped in an SRv6 uSID, ECMP'd across sspine1 and sspine2 on a per-flow basis, decapsulated by sleaf3, and delivered — and you'll watch the spread fill in on the same Grafana dashboard from Lab 4.

Note: this lab is leaf-and-spine, additive. The IPv6 underlay (numbered /127 spine links + locator reachability) is the one you built in Lab 5 — you arrive here with it up and ECMP'd, so you can focus entirely on the SRv6 layer: locators, endpoints, and headend steering. The IPv4 underlay, the VXLAN overlay, and the GPU workers on 192.168.100.0/24 are all still there and still working the whole time.

SRv6 in one minute

Classic Segment Routing puts an ordered list of "segments" (instructions) into the packet header and lets each hop pop the next one. SRv6 encodes those segments as IPv6 addresses — so the segment list is just an IPv6 routing header (SRH), and every router in between is a plain IPv6 router that forwards on the outer destination address. No new data plane, no LDP, no MPLS label distribution. The fabric you already have forwards SRv6 the moment you turn on IPv6.

uSID (micro-SID) is the compressed flavor. Instead of a full 128-bit IPv6 address per segment, a uSID is a short (here, 16-bit) "micro-instruction" packed into a uSID container — a single IPv6 address that can hold a sequence of them. For a one-hop path like ours, the whole instruction rides in the outer IPv6 destination address with no SRH at all (this is H.Encaps.Red — "reduced"). That's why SRv6 uSID has tiny header overhead and is what hyperscalers actually deploy.

Three roles, three places in the fabric:

The addressing scheme

The IPv6 underlay you built in Lab 5 already carries the locators; the SRv6 SIDs and service prefixes below are what this lab layers on. Keep it handy:

The IPv6 underlay runs a numbered IPv6 BGP session per spine link — a /127 on each link (fc00:<spine>:<leaf>::/127, spine end ::0, leaf end ::1), sitting alongside the IPv4 session on the same link. The locators below are advertised over those sessions; the locator's RIB nexthops resolve to the spines' link-locals (fe80::%ethN), which is normal for IPv6.

fcbb:bb00::/32 is the conventional SRv6 uSID block (the same shape Cisco/Arista docs use). Every leaf's /48 locator is advertised into IPv6 BGP with maximum-paths 64, so each leaf learns every other leaf's locator via both spines — that two-way ECMP (which you stood up in Lab 5) is what makes per-flow load-spreading possible.

What you'll learn

Concepts, not CLI flags:

• SRv6 vs MPLS-SR vs VXLAN — why encoding segments as IPv6 addresses means the underlay needs no new data plane, and how that compares to the VXLAN overlay you built
• uSID / micro-segments — the compressed SID format, and what block-len 32 node-len 16 actually carves up in a 128-bit address
• Locator → endpoint → headend — the three pieces, which live in FRR (control plane) and which live in the Linux kernel (data plane)
• End.DT6 — "decapsulate and do an IPv6 table lookup," the endpoint behavior that terminates a uSID path
• H.Encaps.Red — reduced encapsulation: a single uSID in the outer DA, no SRH, minimal overhead
• Per-flow ECMP via the IPv6 flow label — how Linux derives the outer flow label from the inner flow so distinct flows hash to different spines (the SRv6 twin of Lab 4's VXLAN UDP-source-port trick)
• The one gotcha that will bite you — why an SRv6 endpoint must bind to a real device and silently does nothing on lo

💡 Why two CLIs again? Like Lab 2, you'll use both surfaces. The SRv6 locator is a control-plane object, so it lives in vtysh (FRR). The endpoint and headend are kernel data-plane state, so they're plain ip -6 route commands in the shell — the same division of labor Lab 2/3 used (vtysh for BGP, config/ip for the VXLAN kernel devices). FRR 10.4 takes the locator config but doesn't program the kernel endpoint itself, so you do that hop directly.

Teaching philosophy

Lab 2 built a service (an overlay you could sell a tenant). Lab 6 builds an alternative transport for that service — and does it without tearing anything down. The whole lab is a lesson in additive evolution: real fabrics don't get rebuilt, they get dual-stacked and migrated one capability at a time. You'll feel that directly — VXLAN keeps pinging the entire time you stand SRv6 up next to it.

For each step you'll:

1. Define the control plane — the uSID locator in vtysh (show segment-routing srv6 locator confirms it).
2. Program the data plane — the End.DT6 endpoint and the H.Encaps headend with ip -6 route (verify with ip -6 route show and the seg6local packet counter).
3. Send a packet — a ping that rides a uSID from leaf to leaf, decapsulated at the far end.
4. Watch it spread — drive several flows and see them split across both spines on the embedded Grafana dashboard.

Each step has a 💡 Why this matters in AI DCs callout connecting the mechanic to real fabrics — traffic engineering, multi-tenancy, the end-to-end-vs-fabric encapsulation debate, and why per-flow entropy is non-negotiable for collective performance.

Prerequisites

• Labs 1–5 understood, and ideally Lab 5 was the last lab you solved (so the dual-stack IPv6 underlay is built). When you click Start lab ▶ for Lab 6, the orchestrator applies _srv6_skeleton: your full IPv4 + EVPN-VXLAN fabric plus the numbered IPv6 dual-stack underlay you built in Lab 5 (per-link /127 IPv6 BGP sessions + ECMP'd locators). If anything looks sick, click Start ▶ again to re-apply it.
• Keep the addressing table above open in another tab. You'll reference the locator and service prefixes constantly.
• The embedded Telemetry tab (the Lab 4 Grafana dashboard) is back — you'll use it in the final step to watch the ECMP spread.

Where to go next

• lab6-exercise.md — the guided, command-by-command walkthrough. Start here.
• lab6-solution.md — the copy-pasteable answer key, a vtysh/ip ↔ config mapping, and a common-mistakes table (including the dev lo trap).

Exercise — Build an SRv6 uSID transport, one piece at a time

Read lab6-overview.md first if you haven't — it has the addressing table you'll reference throughout.

Scenario

Your fabric is humming: IPv4 underlay, EVPN-VXLAN overlay, eight GPUs on 192.168.100.0/24, telemetry live. Nothing here is going away. The architecture team wants to evaluate SRv6 as a future transport — encode forwarding instructions as IPv6 addresses, carry traffic in micro-SIDs, and get per-flow ECMP "for free" from the IPv6 flow label — without ripping out VXLAN to try it.

So you'll dual-stack and migrate additively. You arrive with the IPv6 underlay you built in Lab 5 already up: a numbered IPv6 BGP session per spine link (fc00:<spine>:<leaf>::/127), with each leaf's uSID locator /48 advertised and ECMP'd via both spines. The seg6 kernel switches (seg6_enabled, IPv6 forwarding, IPv6 multipath hashing, flow-label derivation) are already on. Your job is the SRv6 layer itself: locators, endpoints, headend — built on the four leaves while VXLAN keeps pinging the whole time.

You'll build sleaf1 and sleaf3 fully first (they're the two ends of the demo path), then replicate onto sleaf2 and sleaf4, then send a uSID packet across the fabric and watch it load-spread.

Why an AI Data Center cares about SRv6

VXLAN is a great overlay, but it's an overlay — opaque to the fabric, a fixed UDP tunnel between two VTEPs. SRv6 puts the steering in the packet's IPv6 header, which buys an AI DC three things:

• Traffic engineering — you can pin a flow down a specific path (a low-latency rail, a drained spine) by listing segments, without per-flow state in the fabric.
• One protocol, end to end — the same SRv6 can run on the GPU server's NIC, the leaf, and the WAN, so a tenant's traffic keeps its policy across domains.
• Native per-flow ECMP — because the steering rides in IPv6 and Linux derives the outer flow label from the inner flow, collective traffic spreads across every spine path automatically. That last property is the one we'll prove.

This is the direction hyperscale fabrics are moving — uSID specifically, because its header overhead is tiny enough to live alongside RDMA.

Step 1: Confirm the Lab 5 underlay is still healthy

You built the dual-stack underlay in Lab 5; here you just sanity-check it before layering SRv6 on top. Open the Topology tab and click sleaf1 (or hit + in the terminal pane and pick sleaf1). A console opens.

vtysh -c "show ip bgp summary" | head -12
vtysh -c "show bgp ipv6 unicast summary" | head -12
vtysh -c "show ipv6 route fcbb:bb00:3::/48"

You want three things to be true:

• IPv4 BGP is Established to both spines (the underlay from Lab 1 — still untouched).
• IPv6 BGP is Established to both spines. The IPv6 underlay is numbered, so the neighbor column shows the spine's /127 address (fc00:1:1::0 via sspine1, fc00:2:1::0 via sspine2) sitting alongside the IPv4 session on the same link. That's expected.
• sleaf3's locator fcbb:bb00:3::/48 is in the RIB via two nexthops — one through each spine. The locator's RIB nexthops resolve to the spines' link-locals (fe80::%ethN), which is normal for IPv6 even on a numbered underlay:

sleaf1's view of sleaf3's locator (RIB nexthops resolve to link-locals):
  fcbb:bb00:3::/48
     via fe80::%eth1  (sspine1)   <-- ECMP path 1
     via fe80::%eth2  (sspine2)   <-- ECMP path 2

That two-way ECMP is the whole reason uSID traffic will load-spread. You built it in Lab 5; here you just confirm it's still up.

💡 Why this matters in AI DCs: dual-stacking is how real fabrics adopt SRv6 — you light up IPv6 + a locator block alongside the production IPv4/VXLAN and migrate services one at a time. The fabric never goes down for the transport swap; the two coexist for as long as the migration takes (often years). Lab 5 stood up that IPv6 plane; this lab puts the first service on it.

Step 2: Define the uSID locator (the control plane)

A locator is a leaf's block of SRv6 address space — its /48 — plus the rule for carving SIDs out of it. It's a control-plane object, so it lives in FRR. Drop into vtysh on sleaf1:

vtysh
conf t
segment-routing
 srv6
  locators
   locator MAIN
    prefix fcbb:bb00:1::/48 block-len 32 node-len 16
    behavior usid
end

sleaf1  locator MAIN
  fcbb:bb00:1::/48
  |__ block 32 bits __|__ node 16 __|__ function 16 __|
     fcbb:bb00          0001            (per-SID)
     "which fabric"     "which leaf"    "which behavior"
  behavior: uSID   <-- compressed micro-segments

Confirm it landed, then leave vtysh:

end
do show segment-routing srv6 locator
exit

You should see MAIN ... fcbb:bb00:1::/48 ... Up, behavior uSID.

Now repeat on sleaf2, sleaf3, sleaf4 — open a console on each and run the same block with its own locator prefix:

When all four leaves report their locator Up, run the check.

💡 Why this matters in AI DCs: the locator is a leaf's identity in SRv6 — one summarizable /48 that the whole fabric routes toward, no matter how many individual SIDs (endpoints, VPNs, policies) the leaf later carves from it. That summarization is why SRv6 scales to fabrics with hundreds of thousands of endpoints without exploding the routing table — the spines only ever carry one /48 per leaf, exactly what you saw in Step 1.

Step 3: Program the endpoint — make the leaf decapsulate (End.DT6)

The locator told the control plane "this /48 is mine." Now you program the data plane: an endpoint SID the leaf will match, decapsulate, and deliver from. The endpoint behavior is a kernel seg6local route. Back in the sleaf1 shell (type exit if you're still in vtysh):

ip link add srv6end type dummy
ip link set srv6end up
sysctl -w net.ipv6.conf.srv6end.seg6_enabled=1
ip -6 route replace fcbb:bb00:1:fe00:: encap seg6local action End.DT6 table 255 dev srv6end

sleaf1  endpoint
   fcbb:bb00:1:fe00::   --(arrives wrapped)-->  End.DT6
                                                  |__ strip outer IPv6/uSID
                                                  |__ look inner up in table 255 (local) = deliver here
                                                  |__ deliver to fd00:100:1::x  (local)
   bound to:  srv6end  (a dummy device -- NOT lo!)

Verify it attached — the packet counter is your ground truth:

ip -6 -s route show fcbb:bb00:1:fe00::

You should see encap seg6local action End.DT6 ... dev srv6end (and packets 0 for now — it'll climb in Step 5).

⚠️ The trap that will cost you an hour if you hit it: an End.DT6 route on dev lo is silently accepted and does nothing — ip route show prints a plain route with no encap, and any uSID packet that arrives gets an ICMP "destination unreachable." seg6local endpoints must bind to a real device. That's why we created the srv6end dummy. If your endpoint check fails, this is the first thing to look at.

Repeat on sleaf2, sleaf3, sleaf4 — same four commands, each with its own endpoint SID (fcbb:bb00:2:fe00::, fcbb:bb00:3:fe00::, fcbb:bb00:4:fe00::).

💡 Why this matters in AI DCs: End.DT6 is "decapsulate and look the inner packet up in a table" — and that table is the hook for multi-tenancy. Point different endpoint SIDs at different VRF tables and one fabric carries many isolated tenants' GPU pods over a shared SRv6 core, each only reaching its own hosts. It's the SRv6 equivalent of an EVPN L3VNI, done with one kernel route.

Step 4: Program the headend — steer flows into a uSID (H.Encaps.Red)

The endpoint lets a leaf receive uSID traffic. The headend lets it send: a route that says "for traffic to that remote service prefix, wrap it in the remote leaf's uSID." On sleaf1, install a headend for each other leaf's service prefix:

ip -6 route replace fd00:100:2::/64 encap seg6 mode encap.red segs fcbb:bb00:2:fe00:: dev eth1
ip -6 route replace fd00:100:3::/64 encap seg6 mode encap.red segs fcbb:bb00:3:fe00:: dev eth1
ip -6 route replace fd00:100:4::/64 encap seg6 mode encap.red segs fcbb:bb00:4:fe00:: dev eth1

a flow sleaf1 -> fd00:100:3::5  gets wrapped (H.Encaps.Red, no SRH):

 inner:  [ IPv6 | fd00:100:1::1 -> fd00:100:3::5 ]            the real packet
                         |
                         v   sleaf1 headend
 outer:  [ IPv6 | sleaf1 -> fcbb:bb00:3:fe00:: | flowlabel=hash(inner) ][ inner ]
                          ^^^^^^^^^^^^^^^^^^^^   ^^^^^^^^^^^^^^^^^^^^^^
                          sleaf3's uSID in the    derived from the inner flow
                          outer DA (no SRH!)     => different per flow => ECMP

mode encap.red is reduced encap: a single uSID rides in the outer destination address with no separate routing header — minimum overhead.

Now do the same on sleaf2, sleaf3, sleaf4, each steering the other three service prefixes into the matching uSID. (sleaf3 needs a headend for fd00:100:1::/64 so its replies to sleaf1 ride SRv6 back — the path is symmetric.) The check inspects sleaf1:

💡 Why this matters in AI DCs: the headend is where traffic engineering lives. Today you steer with a single uSID = shortest path. List more segments and you pin a flow down a specific route — around a hot spine, onto a dedicated low-jitter rail for a latency-sensitive all-to-all, or through a firewall for a cross-tenant flow — all by editing one headend route, with zero new state in the fabric core. That's the SRv6 superpower VXLAN can't match.

Step 5: Send a packet across the uSID transport

Everything's in place: sleaf1 can wrap, the spines forward IPv6, sleaf3 decapsulates. Send a flow from a host behind sleaf1 to a host behind sleaf3:

ping6 -c3 -I fd00:100:1::1 fd00:100:3::1

  sleaf1 (headend)                         sleaf3 (End.DT6)
  fd00:100:1::1  --wrap--> fcbb:bb00:3:fe00:: --decap--> fd00:100:3::1
                      \                       /
                       sspine1  -- or --  sspine2     (ECMP picks one per flow)

It should reply. To prove sleaf3 actually decapsulated (rather than some other path), check its endpoint counter — it climbed:

# on sleaf3:
ip -6 -s route show fcbb:bb00:3:fe00::

packets is now non-zero: that's sleaf3 popping your uSID and delivering the inner packet locally.

💡 Why this matters in AI DCs: you just moved real traffic over a transport the fabric core has no per-flow state for — the spines are plain IPv6 routers. That statelessness is why SRv6 scales: adding a tenant, a policy, or a path is an edit at the edge (the leaf headend), never a touch to the thousands of switches in the middle. Collective traffic for a new training job lands on the fabric without anyone reconfiguring a spine.

Step 6: Watch it spread across both spines, then Submit

A single ping is one flow — it picks one spine and stays there. The payoff is many flows spreading. Generate a burst of distinct flows from sleaf1 to sleaf3 (each destination address is a different flow → a different flow label → a different ECMP decision):

# on sleaf1 — 16 parallel flows to distinct hosts behind sleaf3:
for i in $(seq 1 16); do
  ping6 -q -i 0.2 -c 200 -I fd00:100:1::1 fd00:100:3::$i &
done

Open the Telemetry tab (the Grafana dashboard from Lab 4). Watch the sspine1 and sspine2 ingress/egress toward sleaf3 — both light up, splitting the flows between them (a hash spreads 16 flows, so expect a rough — not perfectly even — split; the more flows, the closer to 50/50):

            uSID traffic sleaf1 -> sleaf3, 16 flows
                       /              \
                 sspine1                sspine2
           some flows            the rest        <- per-flow ECMP on the flow label
                       \              /
                       sleaf3 (End.DT6)

That's the same load-spreading you saw for VXLAN in Lab 4 — but the entropy is now the IPv6 flow label the kernel derived from each inner flow, not a VXLAN UDP source port. Two transports, same essential property, because both put per-flow entropy where the fabric's ECMP hash can see it.

Let the pings finish (or kill %1 %2 ...), then click Submit ✓. The finale verifies three things at once:

1. uSID traffic can spread — sleaf1 still reaches sleaf3's locator via 2 ECMP nexthops.
2. The SRv6 path works end-to-end (sleaf1 → sleaf3 over uSID).
3. Regression — the full 56-pair worker overlay mesh still pings. Your additive SRv6 work didn't disturb VXLAN one bit.

💡 Why this matters in AI DCs: per-flow ECMP is not a nicety — a collective is N² flows between GPUs, and if they all pin to one spine you've halved your bisection bandwidth and your AllReduce stalls. Whether the transport is VXLAN or SRv6, the rule is the same: get per-flow entropy into a field the fabric hashes on. You've now built that property twice, two different ways — which is exactly the operator intuition this lab exists to give you.

You're done

You stood up a complete SRv6 uSID transport — locators, endpoints, headend — additively, alongside a live VXLAN fabric that never skipped a beat, and proved it load-spreads per-flow across both spines. If everything passed, the lab stamps Passed and the CTA for Lab 7 — Super Spines lights up.

Stuck? Want to restart?

Reset re-applies _srv6_skeleton — it tears down any srv6end device and seg6 routes you added and puts you back at the numbered dual-stack underlay from Lab 5 (IPv4 + VXLAN + IPv6, no SRv6 layer). Your IPv4/VXLAN fabric is never touched by Reset.

Where to go from here

• lab6-solution.md — the full copy-pasteable config for all four leaves, a vtysh/ip ↔ behavior mapping, and a common-mistakes table
• ../../notes/decisions.md — ADR-014 (why this lab is additive dual-stack with leaves as endpoints, and the dev lo spike finding)
• Lab 7 — Super Spines — Beyond a Single-Pod CLOS — the conceptual follow-on on multi-pod scale