ISP Fail: FNBConnect and Internet Solutions. Who screwed up?

Simeon Miteff — Mon, 17 Aug 2009 09:56:35 +0000

The /16 route being announced by First National Bank’s new IPConnect ADSL-based consumer ISP, FNBCONNECT (AS37028) disappeared from the Internet Solutions (AS3741) local routing table somewhere between last week Wednesday (2009/08/12) and Friday (2009/08/14).

It seems their transit via First National Bank’s own network went down. FNB, in turn, buys transit via IS and MTN Business. FNBConnect’s transit via Telkom SA (SAIX) is still working though, so they are reachable from MTN Business (and the rest of the Internet) via SAIX.

Spot the odd one out...

It’s hard to tell who is to blame even when you can check the routes on route servers for each provider’s network. The reason why is because BGP routes could be filtered on either end of a link. This is of course convenient for the providers, as they can (and do!) always blame the other party. When it gets fixed, you never hear who did the fixing. It is however fun to try and infer who screwed up their filters.

In the graph on the right I’ve used green to indicate the links and ASNs we know for sure are getting the FNBCONNECT route. Orange indicates the links and ASNs of unknown status, while red shows what is definitely not working.

We at least know that their transit to IS via FNB worked when I first blogged about FNBCONNECT:

* 41.183.0.0/16 168.209.255.8 0 3741 17148 37028 i

So which is more likely, SAIX screwed up their outbound filters on their peering links to IS but not to MTN Business, or IS screwed up their inbound peering filters and its only showing now because their preferred route was via their customer?

This is possibly the third time I’ve noticed a peering problem involving IS, where no problem existed between IS’ peer and another peer (like SAIX or MTN Business).

The end result? IS is routing like we’re back in 1996!:
local-route-server>traceroute 41.183.0.0


Type escape sequence to abort.

Tracing the route to 41.183.0.0

1 ar2-rba-tnr-gi0-3-11.ip.isnet.net (196.34.7.195) [AS 3741] 0 msec 4 msec 4 msec 2 core1b-rba-gi1-0-5.ip.isnet.net (196.26.0.181) [AS 3741] 4 msec 4 msec 4 msec 3 mi-za-rba-p5-gi0-1-101.ip.isnet.net (168.209.164.49) [AS 3741] [MPLS: Label 2594 Exp 1] 172 msec 172 msec 176 msec 4 mi-uk-dock-p3-po2-0.ip.isnet.net (168.209.224.65) [AS 3741] [MPLS: Label 2859 Exp 1] 172 msec 5 core1a-dock-gi1-0-0-101.ip.isnet.net (168.209.164.0) [AS 3741] 184 msec 176 msec 6 core1b-dock-gi0-0-2.ip.isnet.net (168.209.246.1) [AS 3741] 176 msec * 172 msec 7 gi8-13.mpd01.lon02.atlas.cogentco.com (149.6.148.1) 180 msec 176 msec 172 msec 8 te4-2.ccr01.lon01.atlas.cogentco.com (130.117.1.201) 172 msec 9 vl3493.mpd01.lon01.atlas.cogentco.com (130.117.2.17) 172 msec te3-1.mpd01.lon01.atlas.cogentco.com (130.117.3.225) 176 msec vl3493.mpd01.lon01.atlas.cogentco.com (130.117.2.17) 172 msec 10 te1-2.ccr01.lon05.atlas.cogentco.com (130.117.49.94) 172 msec 176 msec 11 149.6.2.194 184 msec 180 msec 180 msec 12 rrba-ip-esr-1-ge-6-0-0.telkom-ipnet.co.za (196.43.11.166) [AS 5713] 184 msec 184 msec 184 msec 13 first-national-bank-gw.telkom-ipnet.co.za (196.25.207.178) [AS 5713] 188 msec 188 msec 188 msec === snip ===

Update:
Dear Readers

I’ve decided that if I’m going to moan, complain, and accuse ISPs of FAILure on this blog, then I should at least follow my accusations up, and provide constructive post-mortem commentary (where possible). So here goes:

IS->FNBConnect traffic is flowing via FNB again. Also see the comment from Nick Treasure (IS).

Regards,
Simeon.

Post Altech-victory BGP table: New kids on the block

Simeon Miteff — Mon, 11 May 2009 11:17:36 +0000

I compiled my initial list of local ASNs when I started this blog in October last year. Since then we’ve had Altech’s self-provisioning court victory which lead to the potential for many new telecoms operators to compete with the likes of Telkom and Neotel.

Since I’ve just updated the table using a fresh copy of IS’s “local” routing table (minus their African and Malaysian peers), we can check if opening the licensing floodgates has lead to many new South African networks popping up in the BGP table. There are four new ASNs, so the short answer is “not really”. The list follows:

AS35405 (Macquarie Bank)
This is an end user, so not due to regulatory changes. As you can see, they are dual homed (like most of the Banks are), but they’re running their second link in backup-only mode by announcing sub-prefixes on their primary link (similar to my setup for UCT). Datapro is their primary provider and Neotel is backup:
* 87.236.68.0/24 168.209.255.8 0 3741 36937 35405 ? * 87.236.68.0/23 168.209.255.8 0 3741 11845 35405 ? * 87.236.69.0/24 168.209.255.8 0 3741 36937 35405 ?

AS37028 (FNBConnect)
This is First National Bank’s new IPConnect-based ADSL service provider and VOIP operator. If I recall correctly, they would need at least an IECS license to do this, but they probably got the happy meal (IECNS+IECS) anyway. They transit through the bank’s original AS:
* 41.183.0.0/16 168.209.255.8 0 3741 17148 37028 i

AS37049 (South African Digital Villages (Pty) Ltd)
This one is a mystery so far. The name sounds like someone who wires up housing or office complexes and sells them bandwidth. They transit through SAIX:
* 41.222.136.0/21 168.209.255.8 0 3741 5713 37049 i

AS37079 (SMMT Online)
Finally, the winner of the Gauteng Online tender to provide connected computer labs for the province’s schools. They transit through Neotel:
* 41.154.0.0/16 168.209.255.8 0 3741 36937 37079 i

Local Loop » fnbconnect

ISP Fail: FNBConnect and Internet Solutions. Who screwed up?

Post Altech-victory BGP table: New kids on the block