Local Loop

The /16 route being announced by First National Bank’s new IPConnect ADSL-based consumer ISP, FNBCONNECT (AS37028) disappeared from the Internet Solutions (AS3741) local routing table somewhere between last week Wednesday (2009/08/12) and Friday (2009/08/14).

It seems their transit via First National Bank’s own network went down. FNB, in turn, buys transit via IS and MTN Business. FNBConnect’s transit via Telkom SA (SAIX) is still working though, so they are reachable from MTN Business (and the rest of the Internet) via SAIX.

It’s hard to tell who is to blame even when you can check the routes on route servers for each provider’s network. The reason why is because BGP routes could be filtered on either end of a link. This is of course convenient for the providers, as they can (and do!) always blame the other party. When it gets fixed, you never hear who did the fixing. It is however fun to try and infer who screwed up their filters.

In the graph on the right I’ve used green to indicate the links and ASNs we know for sure are getting the FNBCONNECT route. Orange indicates the links and ASNs of unknown status, while red shows what is definitely not working.

We at least know that their transit to IS via FNB worked when I first blogged about FNBCONNECT:

* 41.183.0.0/16 168.209.255.8 0 3741 17148 37028 i

So which is more likely, SAIX screwed up their outbound filters on their peering links to IS but not to MTN Business, or IS screwed up their inbound peering filters and its only showing now because their preferred route was via their customer?

This is possibly the third time I’ve noticed a peering problem involving IS, where no problem existed between IS’ peer and another peer (like SAIX or MTN Business).

The end result? IS is routing like we’re back in 1996!:

local-route-server>traceroute 41.183.0.0

Type escape sequence to abort.
Tracing the route to 41.183.0.0

1 ar2-rba-tnr-gi0-3-11.ip.isnet.net (196.34.7.195) [AS 3741] 0 msec 4 msec 4 msec
2 core1b-rba-gi1-0-5.ip.isnet.net (196.26.0.181) [AS 3741] 4 msec 4 msec 4 msec
3 mi-za-rba-p5-gi0-1-101.ip.isnet.net (168.209.164.49) [AS 3741] [MPLS: Label 2594 Exp 1] 172 msec 172 msec 176 msec
4 mi-uk-dock-p3-po2-0.ip.isnet.net (168.209.224.65) [AS 3741] [MPLS: Label 2859 Exp 1] 172 msec
5 core1a-dock-gi1-0-0-101.ip.isnet.net (168.209.164.0) [AS 3741] 184 msec 176 msec
6 core1b-dock-gi0-0-2.ip.isnet.net (168.209.246.1) [AS 3741] 176 msec * 172 msec
7 gi8-13.mpd01.lon02.atlas.cogentco.com (149.6.148.1) 180 msec 176 msec 172 msec
8 te4-2.ccr01.lon01.atlas.cogentco.com (130.117.1.201) 172 msec
9 vl3493.mpd01.lon01.atlas.cogentco.com (130.117.2.17) 172 msec
te3-1.mpd01.lon01.atlas.cogentco.com (130.117.3.225) 176 msec
vl3493.mpd01.lon01.atlas.cogentco.com (130.117.2.17) 172 msec
10 te1-2.ccr01.lon05.atlas.cogentco.com (130.117.49.94) 172 msec 176 msec
11 149.6.2.194 184 msec 180 msec 180 msec
12 rrba-ip-esr-1-ge-6-0-0.telkom-ipnet.co.za (196.43.11.166) [AS 5713] 184 msec 184 msec 184 msec
13 first-national-bank-gw.telkom-ipnet.co.za (196.25.207.178) [AS 5713] 188 msec 188 msec 188 msec
=== snip ===


Update:
Dear Readers

I’ve decided that if I’m going to moan, complain, and accuse ISPs of FAILure on this blog, then I should at least follow my accusations up, and provide constructive post-mortem commentary (where possible). So here goes:

IS->FNBConnect traffic is flowing via FNB again. Also see the comment from Nick Treasure (IS).

Regards,
Simeon.