It seems their transit via First National Bank’s own network went down. FNB, in turn, buys transit via IS and MTN Business. FNBConnect’s transit via Telkom SA (SAIX) is still working though, so they are reachable from MTN Business (and the rest of the Internet) via SAIX.

Spot the odd one out...

In the graph on the right I’ve used green to indicate the links and ASNs we know for sure are getting the FNBCONNECT route. Orange indicates the links and ASNs of unknown status, while red shows what is definitely not working.

We at least know that their transit to IS via FNB worked when I first blogged about FNBCONNECT:

* 41.183.0.0/16 168.209.255.8 0 3741 17148 37028 i

So which is more likely, SAIX screwed up their outbound filters on their peering links to IS but not to MTN Business, or IS screwed up their inbound peering filters and its only showing now because their preferred route was via their customer?

This is possibly the third time I’ve noticed a peering problem involving IS, where no problem existed between IS’ peer and another peer (like SAIX or MTN Business).

The end result? IS is routing like we’re back in 1996!:

local-route-server>traceroute 41.183.0.0

1 ar2-rba-tnr-gi0-3-11.ip.isnet.net (196.34.7.195) [AS 3741] 0 msec 4 msec 4 msec
2 core1b-rba-gi1-0-5.ip.isnet.net (196.26.0.181) [AS 3741] 4 msec 4 msec 4 msec
3 mi-za-rba-p5-gi0-1-101.ip.isnet.net (168.209.164.49) [AS 3741] [MPLS: Label 2594 Exp 1] 172 msec 172 msec 176 msec
4 mi-uk-dock-p3-po2-0.ip.isnet.net (168.209.224.65) [AS 3741] [MPLS: Label 2859 Exp 1] 172 msec
5 core1a-dock-gi1-0-0-101.ip.isnet.net (168.209.164.0) [AS 3741] 184 msec 176 msec
6 core1b-dock-gi0-0-2.ip.isnet.net (168.209.246.1) [AS 3741] 176 msec * 172 msec
7 gi8-13.mpd01.lon02.atlas.cogentco.com (149.6.148.1) 180 msec 176 msec 172 msec
8 te4-2.ccr01.lon01.atlas.cogentco.com (130.117.1.201) 172 msec
9 vl3493.mpd01.lon01.atlas.cogentco.com (130.117.2.17) 172 msec
te3-1.mpd01.lon01.atlas.cogentco.com (130.117.3.225) 176 msec
vl3493.mpd01.lon01.atlas.cogentco.com (130.117.2.17) 172 msec
10 te1-2.ccr01.lon05.atlas.cogentco.com (130.117.49.94) 172 msec 176 msec
11 149.6.2.194 184 msec 180 msec 180 msec
12 rrba-ip-esr-1-ge-6-0-0.telkom-ipnet.co.za (196.43.11.166) [AS 5713] 184 msec 184 msec 184 msec
13 first-national-bank-gw.telkom-ipnet.co.za (196.25.207.178) [AS 5713] 188 msec 188 msec 188 msec
=== snip ===


Update:
Dear Readers

I’ve decided that if I’m going to moan, complain, and accuse ISPs of FAILure on this blog, then I should at least follow my accusations up, and provide constructive post-mortem commentary (where possible). So here goes:

IS->FNBConnect traffic is flowing via FNB again. Also see the comment from Nick Treasure (IS).

Regards,
Simeon.

* 41.203.16.0/22 168.209.255.8 0 3741 2905 65419 i
* 41.203.20.0/23 168.209.255.8 0 3741 2905 65419 i
* 41.203.22.0/23 168.209.255.8 0 3741 2905 65419 i
* 41.203.24.0/21 168.209.255.8 0 3741 2905 65419 i
* 41.204.216.0/22 168.209.255.8 0 3741 2905 65419 i
* 41.204.220.0/23 168.209.255.8 0 3741 2905 65419 i
* 196.22.132.0/22 168.209.255.8 0 3741 2905 65419 i
* 196.22.136.0/21 168.209.255.8 0 3741 2905 65419 i
* 196.30.125.0 168.209.255.8 0 3741 2905 65419 i

It looks like Hetzner is announcing these to MTN Business (previously Verizon Business SA) using an autonomous system number from the private range (64512 to 65535). That is perfectly reasonable as long as MTN Business strips the private ASN when they announce these routes on the Internet.

Doing this is easy (neighbor x.x.x.x remove-private-AS), and as far as I can tell, they get it right on their links to SAIX and Verizon Business Europe/US, but as you can see, not on their link to Internet Solutions.