Lab Guide 5 — IPv6 Underlay for SRv6: Dual-Stack BGP + ECMP

In Labs 1–4 you built an IPv4 CLOS underlay, stretched an EVPN-VXLAN overlay across it, put GPUs on that overlay, and watched real AllReduce traffic light up a telemetry dashboard. Everything you've built so far rides one address family: IPv4 unicast in the underlay, L2VPN-EVPN for the overlay.

This lab adds a second underlay — IPv6 — alongside the IPv4 one, without disturbing it at all. Your IPv4 fabric and your EVPN-VXLAN overlay keep running untouched the entire time. On top of them you'll bring up a second, numbered BGP session per fabric link — an IPv6 session running right next to the IPv4 one you built in Lab 1. Same physical links, same leaf↔spine pairs, but a separate IPv6 BGP relationship carrying IPv6 routes. This is dual-stacking: two parallel control planes on one fabric.

Each leaf advertises one thing into this new IPv6 underlay: its uSID locator — a /48 that means "this is leaf N; any SRv6 micro-instruction starting with my locator belongs to me." And because the fabric has two spines, you'll make sure each leaf learns every other leaf's locator via both spines, so traffic can spread per-flow across the two of them.

By the end of this lab the fabric is dual-stacked: IPv4 + VXLAN exactly as you left it, plus a clean IPv6 underlay where all four leaf locators (fcbb:bb00:1::/48 … fcbb:bb00:4::/48) are reachable from anywhere via two-way ECMP. That underlay is the foundation the next lab builds an SRv6 uSID transport on top of.

Note: this lab is underlay only — no overlay change, no dataplane encapsulation yet. You are building the IPv6 control plane (BGP sessions + locator reachability). The SRv6 endpoints and headend that actually wrap packets in a uSID are Lab 6. Here you're laying the road; Lab 6 drives on it.

A second, numbered BGP session in one minute

You already have one eBGP session on every leaf↔spine link: the IPv4 underlay from Lab 1, with a /31 on each end and a neighbor <ipv4> remote-as 65000 line. This lab adds a second eBGP session on the same link — this time over IPv6.

It works exactly like the IPv4 session you already know, just in the IPv6 address family:

• Each link end has an IPv6 /127 (the IPv6 analogue of the IPv4 /31) — already pre-set for you. The spine end is ::0, the leaf end is ::1.
• On each leaf you add neighbor <spine-ipv6> remote-as 65000 for both spines, activate them in address-family ipv6 unicast, and advertise your locator.

You might reasonably expect to use BGP unnumbered here — peer over the link-local with neighbor ethN interface. That's a great pattern on a fresh fabric, but it collides with the IPv4 underlay session already running on this link: a single unnumbered interface session can't coexist cleanly with the numbered IPv4 session on the same link in this build. So the fabric runs a separate, numbered IPv6 session instead — clean, predictable, and independent of the IPv4 one. (See the sidebar in the solution's common-mistakes section.)

The IPv6 link addresses are tiny and pre-allocated for you (fc00:<spine>:<leaf>::/127); the only IPv6 prefixes that carry real meaning in this lab are the locators, which you advertise.

What you'll learn

Concepts, not CLI flags:

• Dual-stacking a live fabric — running a second, numbered IPv6 BGP session per link alongside the existing IPv4 underlay session, in its own peer-group, without touching IPv4 or VXLAN
• IPv6 /127 fabric links — the IPv6 analogue of the IPv4 /31, why a /127 is the right per-link size, and how the spine ::0 / leaf ::1 ends pair up
• The uSID locator — what a fcbb:bb00:<leaf>::/48 is, why a /48 is the right size (SRv6 will slice micro-SIDs out of the lower bits later), and why it's a summarizable per-leaf identity
• Originating an anchor prefix — why advertising a /48 you don't have a real route to needs both a blackhole static and no bgp network import-check (both pre-set for you) plus a network statement (you add)
• maximum-paths and ECMP — how installing both spine paths to every locator is the non-negotiable foundation for per-flow load-spreading of collective traffic
• Reading IPv6 RIB output — why the locator's nexthops render as fe80::...%ethN link-locals even though the BGP session is numbered, so it doesn't surprise you

💡 Why two address families, one fabric? You already run IPv4 unicast and L2VPN-EVPN on the spine↔leaf links. This lab adds a third, completely separate BGP relationship over the same physical links — a numbered IPv6 session in its own peer-group. The IPv4 sessions (and your whole VXLAN overlay) are untouched; you're stacking a parallel control plane next to them, the way a real fabric dual-stacks to IPv6/SRv6 one capability at a time.

Teaching philosophy

Lab 1 was "build the routing protocol." Lab 2 built a service. Lab 5 builds a second underlay for a future transport — and it's a lesson in additive evolution: real fabrics don't get rebuilt, they get dual-stacked one capability at a time. You'll feel that directly — the IPv4 fabric and the VXLAN overlay keep working the entire time you stand IPv6 up next to them.

For each step you'll:

1. Inspect what's there — the handed-over IPv4 BGP and a working overlay ping, so you know the baseline is healthy before you touch anything.
2. Look at the link — see the pre-set IPv6 /127 already on the interface; your links are addressed, you just add BGP on top.
3. Build the control plane — drop into vtysh, add the two numbered IPv6 spine neighbors on each leaf, watch the sessions reach Established.
4. Advertise the locator — add the network fcbb:bb00:<n>::/48 on each leaf and watch all four locators populate the fabric-wide IPv6 table.
5. Make it ECMP — maximum-paths 64, then confirm every remote locator is reachable via two spine nexthops.

Each step has a 💡 Why this matters in AI DCs callout connecting the mechanic to real fabrics — additively dual-stacking a live fabric, the locator as a summarizable identity, and why per-flow ECMP is the bedrock of collective performance.

Prerequisites

• Labs 1–4 complete (or understood). This IPv6 underlay rides alongside the IPv4 underlay + VXLAN overlay you built there. When you click Start lab ▶ for Lab 5, the orchestrator applies _srv6_underlay_skeleton: your full IPv4 + EVPN-VXLAN fabric (workers on 192.168.100.0/24) plus the IPv6 link plumbing pre-set — interfaces dual-stacked with their /127 link addresses, the locator's blackhole anchor, and no bgp network import-check on the leaves. The IPv6 BGP itself is absent on every switch — both the spine and leaf sides are your build. If anything looks sick, click Start ▶ again to re-apply it.
• Keep the addressing table below open in another tab. You'll reference the locators (and the two spine /127 ends you peer with) constantly.

The addressing scheme

fcbb:bb00:<leaf>::/48 where <leaf> = leaf_id makes locators trivial to read: fcbb:bb00:3::/48 is always sleaf3, no matter which switch you're sitting on. Likewise fc00:<spine>:<leaf>::/127 reads as "the link between spine <spine> and leaf <leaf>."

What's pre-set vs what you build

The orchestrator hands you a fabric with the IPv6 link addressing already in place — the tedious per-link /127s — but no IPv6 BGP anywhere. The BGP is the lesson, so it's all yours, on both the spines and the leaves.

So your entire build lives in router bgp 6510<n> on the four leaves — three logical pieces (the two numbered spine neighbors, the locator network, max-paths) — and the spines need nothing from you.

The workflow loop

For each leaf:

1. Open the leaf's console from the Topology button.
2. Drop into vtysh, add the SPINES6 peer-group and the two neighbor fc00:<spine>:<n>::0 numbered peers (one per spine).
3. In address-family ipv6 unicast, add network fcbb:bb00:<n>::/48, maximum-paths 64, and neighbor SPINES6 activate.

Then verify on a leaf:

1. ip -6 addr show dev eth1 → the pre-set fc00:1:<n>::1/127 global plus the kernel fe80::/64 link-local — your link is already addressed.
2. vtysh -c "show bgp ipv6 unicast summary" → 2 sessions Established (the two spines, shown by their fc00:...::0 addresses).
3. vtysh -c "show bgp ipv6 unicast" → all four fcbb:bb00:1::/48 … fcbb:bb00:4::/48 locators present.
4. vtysh -c "show ipv6 route fcbb:bb00:3::/48" → 2 nexthops (one per spine), each shown as fe80::...%ethN — ECMP proof.
5. Inline Check ▸ after each verification step; Submit ✓ when done.

💡 Verify the neighbor token actually landed. FRR's boot parser (vtysh -b) silently drops tokens it doesn't understand — a typo'd neighbor fc00:... line just vanishes, no error. After you enter the neighbors, always confirm with vtysh -c "show running-config" | grep "neighbor fc00". If the grep is empty (or missing a spine), the line didn't land — re-enter it.

Persistence note

Same caveats as the earlier labs. Your vtysh edits go to the running FRR daemon, not written back to disk (config save isn't run by the lab). If a switch container restarts, your work is lost — but the orchestrator never restarts switch containers, so you can walk away.

For controlled state changes:

• Reset — re-applies _srv6_underlay_skeleton: it wipes the IPv6 BGP you added on the spines and leaves (leaving only the pre-set /127 links + no bgp network import-check + blackhole anchor) and leaves the IPv4 underlay + VXLAN overlay completely intact. Use when you've made a mess and want a clean IPv6 slate.
• Solve — applies the full Lab-5 answer: the complete numbered IPv6 underlay on all four leaves. Your run is flagged "solved" on the completion screen. That solved state (_srv6_skeleton) is exactly Lab 6's starting point.

Neither Reset nor Solve ever touches your IPv4 fabric or the VXLAN overlay — this lab is purely additive.

Where to go

• lab5-exercise.md — the guided, command-by-command walkthrough. Start here.
• lab5-solution.md — the copy-pasteable per-leaf answer key, a vtysh ↔ frr.conf mapping, and a common-mistakes table (tuned to the numbered IPv6 failure modes).
• lab6-overview.md — Lab 6 — SRv6 uSID Transport, which builds the SRv6 dataplane on top of exactly the underlay you stand up here.

Reference material to keep open in another tab:

• ../topology.md — full fabric reference (the IPv4 side; the IPv6 /127s mirror the same spine↔leaf link layout).
• ../../notes/decisions.md — ADR-001/002 (shared-AS spines), and the SRv6 ADR that frames why the underlay is shaped this way.

Exercise — Build an IPv6 underlay for SRv6, one piece at a time

Scenario

The fabric you built in Labs 1–4 is up and healthy: an IPv4 CLOS underlay, an EVPN-VXLAN overlay, GPU workers on 192.168.100.0/24, and a telemetry dashboard. None of that is going to change in this lab.

What you're going to add is a second underlay — IPv6 — built as a separate, numbered BGP session per link, running alongside the IPv4 underlay session you built in Lab 1. Same physical leaf↔spine links, but a parallel IPv6 BGP relationship. The link ends are already addressed with IPv6 /127s (pre-set for you); you add the BGP on top. Each leaf advertises exactly one prefix into this new underlay — its uSID locator fcbb:bb00:<leaf>::/48 — and you'll make sure every leaf learns every other leaf's locator via both spines, so traffic can spread per-flow across them.

You'll build the IPv6 BGP on every switch — both spines and all four leaves — exactly as you did the IPv4 underlay in Lab 1. Only the link /127s are pre-set; the BGP is all yours. In vtysh:

• On each spine: add a LEAVES6 peer-group with the four numbered leaf neighbors (neighbor fc00:<spine>:<leaf>::1) and activate them in the IPv6 address-family. The spine owns no locator — it just relays the leaves' locators between each other.
• On each leaf: add the two numbered spine neighbors (neighbor fc00:1:<n>::0, neighbor fc00:2:<n>::0), advertise the locator (network fcbb:bb00:<n>::/48), and enable ECMP (maximum-paths 64).

Then verify, and you're done — this is the underlay Lab 6 lays SRv6 on top of.

Why an AI Data Center needs this

Two reasons this exact shape matters at scale:

• Additive dual-stacking — IPv6 alongside a live IPv4 fabric. A hyperscale fabric is never rebuilt to add a capability; it's dual-stacked one address family at a time while production training traffic keeps flowing. Bringing up a second, numbered IPv6 BGP session next to the existing IPv4 underlay — in its own peer-group, touching neither IPv4 nor VXLAN — is exactly how operators roll out an IPv6/SRv6 transport without a maintenance window.
• Per-flow ECMP is the bedrock of collective performance. An AllReduce is N² flows between GPUs. If those flows pin to one spine, you've halved your bisection bandwidth and your training stalls. Installing both spine paths to every locator (maximum-paths) is the non-negotiable foundation — Lab 6's SRv6 transport will ride exactly this ECMP.

Step 1: Confirm the baseline is healthy before you touch anything

You're about to dual-stack a live fabric. The cardinal rule of additive work: prove the thing you're not changing is healthy first, so if something breaks later you know it was your change.

Click Topology → sleaf1. New terminal tab.

The IPv4 underlay is up

vtysh -c "show bgp ipv4 unicast summary"

Expected: two neighbors (sspine1, sspine2), both Established, each with a positive PfxRcd — the IPv4 underlay you built in Lab 1, exactly as you left it.

The VXLAN overlay still pings

docker exec sgpu1 ping -c 2 -W 2 192.168.100.3

(Or from the sgpu1 console: ping -c 2 -W 2 192.168.100.3.) Expected: success — sgpu1 reaching sgpu3 across the EVPN-VXLAN overlay, riding the IPv4 underlay. This ping keeps working for the entire lab — that's the whole point of additive dual-stacking.

💡 Why this matters in AI DCs: real fabrics are never greenfield when you change them — there's always live training traffic on the wire. The discipline of "snapshot the baseline, change additively, prove the baseline survived" is how operators dual-stack a production fabric to IPv6/SRv6 without a maintenance window. You'll re-verify this same overlay ping at the end (and the Submit finale runs the full worker mesh) precisely to prove your IPv6 work disturbed nothing.

Step 2: Look at a link — it's already addressed with an IPv6 /127

Before configuring anything, understand the link you're going to peer over. On sleaf1:

ip -6 addr show dev eth1

Expected — the pre-set global /127 (toward sspine1) and the kernel link-local:

2: eth1@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
    inet6 fc00:1:1::1/127 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::a8c1:abff:fe00:1/64 scope link
       valid_lft forever preferred_lft forever

Your IPv6 link is already addressed. fc00:1:1::1/127 is sleaf1's end of the sleaf1↔sspine1 link; the spine end is fc00:1:1::0. That's the address you'll point your neighbor line at. (eth1 is already carrying an IPv4 /31 for the Lab-1 underlay too — this link is now dual-stacked.) The fe80:: link-local is the kernel's automatic per-link address; you'll see it again as the nexthop for learned locators, but you peer over the global /127.

Do the same on eth2 (the link to sspine2) and you'll see fc00:2:1::1/127 — sleaf1's end toward sspine2, whose end is fc00:2:1::0.

💡 Why this matters in AI DCs: the IPv6 /127 is the direct analogue of the IPv4 /31 you used in Lab 1 — a two-address subnet, one per end, the smallest sane point-to-point prefix. Pre-addressing the links and then layering a second BGP session on top is the literal mechanic of dual-stacking a production fabric: the IPv4 underlay never moves, you just add an IPv6 relationship beside it. The links being already numbered is why your only job here is BGP.

Step 3: Build the numbered IPv6 BGP on all six switches

Now bring up the IPv6 sessions — on the spines first, then the leaves. Until both ends of a link are configured, nothing comes up; that's expected.

On the spines: add the four leaf neighbors

Open sspine1's console, enter vtysh, and add a LEAVES6 peer-group with the four numbered leaf neighbors:

configure terminal
router bgp 65000
 neighbor LEAVES6 peer-group
 neighbor LEAVES6 advertisement-interval 0
 neighbor LEAVES6 timers 3 9
 neighbor fc00:1:1::1 remote-as 65101
 neighbor fc00:1:1::1 peer-group LEAVES6
 neighbor fc00:1:1::1 description sleaf1-v6
 neighbor fc00:1:2::1 remote-as 65102
 neighbor fc00:1:2::1 peer-group LEAVES6
 neighbor fc00:1:2::1 description sleaf2-v6
 neighbor fc00:1:3::1 remote-as 65103
 neighbor fc00:1:3::1 peer-group LEAVES6
 neighbor fc00:1:3::1 description sleaf3-v6
 neighbor fc00:1:4::1 remote-as 65104
 neighbor fc00:1:4::1 peer-group LEAVES6
 neighbor fc00:1:4::1 description sleaf4-v6
 address-family ipv6 unicast
  maximum-paths 64
  neighbor LEAVES6 activate
 exit-address-family
end

Read the key lines:

• neighbor fc00:1:1::1 remote-as 65101 — each leaf is in its own AS (65101–65104), so the spine names the remote AS per neighbor (unlike the leaf side, where one remote-as 65000 covers both shared-AS spines). fc00:1:1::1 is sleaf1's end of the link; the spine's own end is fc00:1:1::0.
• neighbor LEAVES6 activate inside address-family ipv6 unicast — without it the sessions come up but carry no IPv6 routes.

Then do the same on sspine2, changing only the middle digit of every neighbor address from :1: to :2: (sspine2's links are fc00:2:<leaf>::1 on the leaf end). router bgp 65000 is identical — both spines share the AS.

💡 Verify the tokens landed — vtysh -b silently drops tokens it doesn't understand:

vtysh -c "show running-config" | grep "neighbor fc00"

You should see all four neighbor fc00:<spine>:<leaf>::1 ... lines. An empty grep means a typo or a dropped line — go back into conf t and re-enter it.

On the leaves: add the two spine neighbors

Open sleaf1's console, enter vtysh, and add the SPINES6 peer-group plus the two numbered spine neighbors:

configure terminal
router bgp 65101
 neighbor SPINES6 peer-group
 neighbor SPINES6 remote-as 65000
 neighbor SPINES6 advertisement-interval 0
 neighbor SPINES6 timers 3 9
 neighbor fc00:1:1::0 peer-group SPINES6
 neighbor fc00:1:1::0 description sspine1-v6
 neighbor fc00:2:1::0 peer-group SPINES6
 neighbor fc00:2:1::0 description sspine2-v6
 address-family ipv6 unicast
  neighbor SPINES6 activate
  neighbor SPINES6 soft-reconfiguration inbound
 exit-address-family
end

Read the key lines:

• neighbor SPINES6 remote-as 65000 — both spines share AS 65000 (same shared-AS-spine design as your IPv4 underlay), so one remote-as on the peer-group covers both.
• neighbor fc00:1:1::0 peer-group SPINES6 — the numbered IPv6 neighbor. fc00:1:1::0 is sspine1's end of sleaf1's eth1 link; fc00:2:1::0 is sspine2's end on eth2. Note you point at the spine's ::0, not your own ::1.
• neighbor SPINES6 activate inside address-family ipv6 unicast — without this the session comes up but carries no IPv6 routes. (Same activate discipline as every other AF in this fabric.)

💡 Verify the neighbor token landed — vtysh -b silently drops tokens it doesn't understand. Immediately after entering the block:

vtysh -c "show running-config" | grep "neighbor fc00"

You should see all four neighbor fc00:1:1::0 ... / neighbor fc00:2:1::0 ... lines. If the grep is empty (or a spine is missing), the line didn't take (a typo, or it got dropped) — go back into conf t and re-enter it. Never assume a config landed; confirm it.

Now repeat the same shape on sleaf2, sleaf3, sleaf4, changing the router bgp <ASN> line and the <n> digit in the neighbor addresses to that leaf's number. For example sleaf3 (AS 65103) peers fc00:1:3::0 (sspine1) and fc00:2:3::0 (sspine2). The structure is byte-for-byte identical leaf to leaf; only the leaf digit moves.

Verify the sessions came up

On sleaf1:

vtysh -c "show bgp ipv6 unicast summary"

Expected — two neighbors, both Established, named by their numbered IPv6 addresses (the spine ::0 ends):

Neighbor        V         AS   MsgRcvd   MsgSent ...  Up/Down State/PfxRcd ...
fc00:1:1::0     4      65000        ...       ... ... 00:00:30            1 sspine1-v6
fc00:2:1::0     4      65000        ...       ... ... 00:00:30            1 sspine2-v6

On a spine, the same command shows four Established (one per leaf). The State/PfxRcd column is an integer (a route count), not Active/Idle — that's a healthy session.

💡 Heads-up on the nexthop rendering you'll see next. The neighbors show as their numbered fc00:...::0 addresses (this is a numbered session). But in the RIB (Step 5) the learned-locator nexthops render as fe80::...%ethN — FRR resolves a remote prefix's nexthop to the directly-connected peer's link-local, even on a numbered session. That's not a bug or a half-formed session; it's normal IPv6 nexthop resolution. The %ethN suffix is the scope (which interface the link-local lives on), since a fe80:: is only meaningful per-link.

Once all four leaves show 2 Established (and the spines show 4):

💡 Why this matters in AI DCs: you just brought up a full IPv6 BGP mesh across the fabric next to the live IPv4 one — a second control plane on the same wires, in its own peer-group, with the IPv4 underlay and VXLAN overlay never blinking. That's the literal operational model for adding IPv6/SRv6 to a production AI fabric: dual-stack additively, prove the old plane survived, then build the new transport on the new plane.

Step 4: Advertise each leaf's uSID locator

The sessions are up but nobody's advertising anything yet — show bgp ipv6 unicast on a leaf will be empty. Time to give each leaf an identity in the IPv6 underlay: its uSID locator fcbb:bb00:<leaf>::/48.

On sleaf1, in vtysh:

configure terminal
router bgp 65101
 address-family ipv6 unicast
  network fcbb:bb00:1::/48
 exit-address-family
end

That single network statement is sleaf1 announcing "fcbb:bb00:1::/48 is me." Repeat on the other leaves with their own digit:

💡 Why does advertising a /48 you can't actually reach work? A BGP network statement normally only advertises a prefix if there's a matching route in the RIB (the "import check"). But a locator isn't a real reachable network — it's an anchor, a summarizable identity. Two pre-set pieces make it advertisable:

• ipv6 route fcbb:bb00:<n>::/48 blackhole (pre-set) — installs a local route for the locator whose action is "drop." That gives the network statement something to match, and it means any uSID for this leaf that doesn't hit a more-specific endpoint route gets dropped rather than looped.
• no bgp network import-check (pre-set) — tells BGP "advertise this network even though its only backing route is a blackhole." Without it, FRR marks the prefix invalid and never sends it.

Both are already in your config — you only add the network line. (If you ever find your locator not advertising, check these two weren't removed — see the solution's common-mistakes table.)

Verify all four locators are in the fabric

On sleaf1:

vtysh -c "show bgp ipv6 unicast"

Expected — all four locators present: your own (fcbb:bb00:1::/48) and the three remote ones, each learned via both spines (you'll see two paths per remote, *> best and *= equal). Note the nexthops are link-locals:

   Network              Next Hop            Metric LocPrf Weight Path
*> fcbb:bb00:1::/48     ::                       0         32768 i
*> fcbb:bb00:2::/48     fe80::...%eth1                          0 65000 65102 i
*=                      fe80::...%eth2                          0 65000 65102 i
*> fcbb:bb00:3::/48     fe80::...%eth1                          0 65000 65103 i
*=                      fe80::...%eth2                          0 65000 65103 i
*> fcbb:bb00:4::/48     fe80::...%eth1                          0 65000 65104 i
*=                      fe80::...%eth2                          0 65000 65104 i

Two paths for every remote locator — one via each spine (%eth1 = sspine1, %eth2 = sspine2). Your own locator (fcbb:bb00:1::/48) shows nexthop :: because it's locally originated. The nexthops show as link-locals (not the fc00:... peer address) because FRR resolves them to the connected peer's fe80:: — normal even on a numbered session.

💡 Why this matters in AI DCs: the /48 locator is a summarizable identity, and that's deliberate. In Lab 6, SRv6 will slice the lower bits of this /48 into micro-SIDs — short instructions like "decapsulate here" or "go via this leaf." Because the whole leaf is one /48, the fabric only carries one route per leaf no matter how many SIDs that leaf defines internally — the route table scales with the number of leaves, not the number of services or flows. That summarization is what keeps SRv6 fabrics' RIBs small at hyperscale.

Step 5: Make every locator reachable via both spines (per-flow ECMP)

Look closely at the previous step: BGP learned two paths to each remote locator (*> and *=), but by default BGP only installs one into the forwarding table. Confirm that's the current state on sleaf1:

vtysh -c "show ipv6 route fcbb:bb00:3::/48"

If you haven't added max-paths yet, you'll see a single nexthop. You want two. Add maximum-paths 64 on each leaf:

configure terminal
router bgp 65101
 address-family ipv6 unicast
  maximum-paths 64
 exit-address-family
end

Repeat on sleaf2, sleaf3, sleaf4 (only the router bgp <ASN> line changes). 64 is generous headroom — with two spines you'll install two paths today, but the same config scales to a fabric with dozens of spines untouched.

Verify two-way ECMP

On sleaf1, re-check the route to sleaf3's locator:

vtysh -c "show ipv6 route fcbb:bb00:3::/48"

Expected — now two nexthops, each a link-local on a different uplink:

B>* fcbb:bb00:3::/48 [20/0] via fe80::...%eth1, eth1, weight 1, 00:01:12
  *                        via fe80::...%eth2, eth2, weight 1, 00:01:12

Two via fe80::...%ethN lines — one out eth1 (toward sspine1), one out eth2 (toward sspine2). Even though you peered over the numbered fc00:...::0 addresses, the installed nexthops resolve to the peers' link-locals — that's expected, not a sign anything's wrong. This is the ECMP that lets a future SRv6 flow hash to either spine. Spot-check sleaf2's and sleaf4's locators too — all remote locators should show two nexthops.

            sleaf1 wants fcbb:bb00:3::/48 (sleaf3)
                       /              \
              via %eth1                via %eth2
                  |                        |
                sspine1                  sspine2          <- both installed:
                  |                        |               per-flow ECMP
                   \                      /
                          sleaf3 (fcbb:bb00:3::/48)

💡 Why this matters in AI DCs: this is the single most important property for collective traffic. An AllReduce is thousands of simultaneous flows; if they all pinned to sspine1 you'd be using half your fabric and your training step would stall on the congested path. maximum-paths installs both spines so the flows can spread. You've now built the reachability for that spread — Lab 6 adds the entropy (an IPv6 flow label derived per-flow) that actually makes distinct flows hash to different spines. Reachability here, entropy there; you need both, and this lab lands the first half.

Step 6: Submit

You've built the complete IPv6 underlay: numbered IPv6 BGP on every leaf, all four locators advertised, every remote locator reachable via two spines — and the IPv4 fabric + VXLAN overlay never skipped a beat.

Before you submit, re-prove the baseline survived:

docker exec sgpu1 ping -c 2 -W 2 192.168.100.3

Still succeeds — your additive IPv6 work didn't disturb VXLAN.

Now click Submit ✓ in the top bar. The orchestrator will:

1. Re-run every step-level checkpoint (sessions Established, locators advertised, two-way ECMP).
2. Regression-check the IPv4/VXLAN fabric — the full worker overlay mesh still pings.
3. Show you a per-check pass/fail card below this guide.

If everything passes, the lab stamps Passed and the completion screen appears.

💡 Why this matters in AI DCs: the finale checks both the new capability and the old one in one shot — exactly how a real change window closes. "Did the new thing come up?" and "did the old thing survive?" are equally important questions, and a dual-stack migration that breaks the production overlay is a failed migration even if the new underlay is perfect.

You're done — next up: SRv6 uSID transport

You stood up a clean IPv6 underlay the way real SRv6 fabrics begin — a second, numbered IPv6 BGP session per link alongside the live IPv4 one, each leaf advertising a summarizable uSID locator, every locator reachable via two-way ECMP — all of it additive, alongside a live VXLAN fabric that never blinked.

That underlay is the road. In Lab 6 — SRv6 uSID Transport you'll drive on it: define the SRv6 locators in FRR, program End.DT6 endpoints and an H.Encaps headend in the kernel, and send a packet that's wrapped in a uSID, ECMP'd across both spines on the IPv6 flow label, and decapsulated at the far leaf. The locators and ECMP you just built are exactly what Lab 6 consumes — when you click Solve here (or Start ▶ on Lab 6), this solved state is Lab 6's starting point.

Stuck? Want to restart?

Reset re-applies _srv6_underlay_skeleton — it wipes the IPv6 BGP you added on the spines and leaves and puts you back at the starting point (IPv4 + VXLAN + pre-set IPv6 /127 links, no IPv6 BGP anywhere). Your IPv4/VXLAN fabric is never touched by Reset.

Where to go from here

• lab5-solution.md — the full copy-pasteable per-leaf config, a vtysh ↔ frr.conf mapping, and a common-mistakes table (tuned to numbered IPv6 failure modes)
• lab6-overview.md — Lab 6 — SRv6 uSID Transport, built on exactly this underlay
• ../../notes/decisions.md — the ADRs behind shared-AS spines and the SRv6 design