fd00:: Is Not Your 192.168.0.0/16 (short version)

fd00:: Is Not Your 192.168.0.0/16 (short version)

Code: github.com/mabels/mseg-tester · Full write-up

Setup

Our home network runs on two independent uplinks (Vodafone cable and Starlink), with no BGP. That rules out a real, ISP-delegated IPv6 prefix — it belongs to one uplink and breaks on failover. So, like most home networks, we numbered everything with Unique Local Addresses (fc00::/7, RFC 4193) and NAT66 it out whichever uplink is up, the same way you’d use 192.168.0.0/16 with IPv4 NAT. Segment 128 got fd00:192:168:128::/64 — memorable, echoing the IPv4 range right next to it.

Problem

IPv6 on that segment was flapping intermittently; IPv4 on the same segment was rock solid the whole time.

2: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:b4:83:13 brd ff:ff:ff:ff:ff:ff
    inet 192.168.128.200/24 metric 100 brd 192.168.128.255 scope global dynamic ens19
       valid_lft 86350sec preferred_lft 86350sec
    inet6 fd06:d22c:9920:1ae:be24:11ff:feb4:8313/64 scope global deprecated dynamic mngtmpaddr noprefixroute
       valid_lft 1688sec preferred_lft 0sec
    inet6 fd00:192:168:128:be24:11ff:feb4:8313/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:feb4:8313/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

Both inet6 addresses are scope global — nothing marks one as ours and the other as a stranger’s. The routing table shows why this isn’t a routing bug:

fd00:192:168:128::/64 dev ens19 proto ra metric 100 pref medium
fd06:d22c:9920:1ae::/64 dev ens19 proto ra metric 100 expires 1669sec pref medium
fe80::/64 dev ens19 proto kernel metric 256 pref medium
default nhid 1278424908 via fe80::200:c0ff:fea8:8001 dev ens19 proto ra metric 100 expires 65448sec pref medium

Exactly one default route, and it’s the real gateway. The fd06 prefix is just an ordinary, honestly-advertised on-link route sitting next to the real one — routing was never confused about how to reach either. And conntrack on the NAT box shows exactly what happens when the wrong one gets picked as a source address:

tcp      6 108 SYN_SENT src=fd06:d22c:9920:1ae:be24:11ff:feb4:8313 dst=2606:4700:4700::1111 sport=50694 dport=443 [UNREPLIED] src=2606:4700:4700::1111 dst=fd06:d22c:9920:1ae:be24:11ff:feb4:8313 sport=443 dport=50694 mark=0 use=1

A SYN left the house with fd06:... as its source — [UNREPLIED] because nothing on the NAT box knows to masquerade a prefix it was never told about.

The kernel randomly chooses which of the two addresses to use as source for a given outbound connection, but always sends it via the one correct default route — so the same host, hitting the same destination, works one moment and hangs the next. That’s the flapping mseg-tester was catching: not a routing problem, an address-selection coin flip.

Solution

Fixing it needs a real, globally unique IPv6 block instead of a hand-picked fd00 prefix — free /56+ allocations are available from Hurricane Electric and ROUTE64, no BGP or ISP involvement required, NAT66’d outbound exactly like ULA is today.

Don’t use fd00::/8 the way you’d use 192.168.0.0/16. It looks like the same kind of address space. It isn’t.

Full write-up, with the actual ip addr/ip -6 route evidence and the theories that got killed along the way: fd00:: Is Not Your 192.168.0.0/16.