Two comments:

1. MWEB already peers with MTN. Here are the two hops between their networks¹:

   6 g-0-3-vic-jinx-2.mweb.co.za (196.22.163.1)
7 mtnns-2.jinx.net.za (198.32.142.31)

   I think what MWEB probably means by "transit" is that MTN charges them for peering. If MTN agreed to MWEB's ultimatum, they would still be peering at JINX. So if you are an MWEB customer who likes to surf Mybroadband.co.za², you should demand a discount to compensate for the de-peering.

2. "We do not believe in paying for transit or selling transit. Rather, we have invited all of them to peer with us."
   -- MyBB quoting MWEB ISP CEO Derek Hershaw

   What makes buying transit in London different from Johannesburg?

1. traceroute from MWEB uncapped ADSL in Pretoria – thanks Ed
2. for example - MyBB is hosted at Hetzner, on the MTN network

So, back in the nineties, Telkom SA decided to call their ISP division “South African Internet Exchange”. That is also the last time SAIX updated their web site, but I digress. I think it’s possible that Telkom chose this name because from their PSTN perspective, an ISP looked like a telephone exchange for the Interwebs.

SAIX doesn’t pretend to be an IX (in fact, they don’t participate in any peering points in South Africa) so I’m willing to attribute their confusing name to misunderstanding.

However, the recently launched Africa Independent Network Exchange’s skilled professionals with many years of experience in the ISP industry should know that when you offer consumer broadband and hosting, you’re most definitely an ISP, so what’s with the name?

To be clear, their “peering service” is either IP transit (ISP business), or layer 2 circuits (telco business).

It seems their transit via First National Bank’s own network went down. FNB, in turn, buys transit via IS and MTN Business. FNBConnect’s transit via Telkom SA (SAIX) is still working though, so they are reachable from MTN Business (and the rest of the Internet) via SAIX.

Spot the odd one out...

In the graph on the right I’ve used green to indicate the links and ASNs we know for sure are getting the FNBCONNECT route. Orange indicates the links and ASNs of unknown status, while red shows what is definitely not working.

We at least know that their transit to IS via FNB worked when I first blogged about FNBCONNECT:

* 41.183.0.0/16 168.209.255.8 0 3741 17148 37028 i

So which is more likely, SAIX screwed up their outbound filters on their peering links to IS but not to MTN Business, or IS screwed up their inbound peering filters and its only showing now because their preferred route was via their customer?

This is possibly the third time I’ve noticed a peering problem involving IS, where no problem existed between IS’ peer and another peer (like SAIX or MTN Business).

The end result? IS is routing like we’re back in 1996!:

local-route-server>traceroute 41.183.0.0

1 ar2-rba-tnr-gi0-3-11.ip.isnet.net (196.34.7.195) [AS 3741] 0 msec 4 msec 4 msec
2 core1b-rba-gi1-0-5.ip.isnet.net (196.26.0.181) [AS 3741] 4 msec 4 msec 4 msec
3 mi-za-rba-p5-gi0-1-101.ip.isnet.net (168.209.164.49) [AS 3741] [MPLS: Label 2594 Exp 1] 172 msec 172 msec 176 msec
4 mi-uk-dock-p3-po2-0.ip.isnet.net (168.209.224.65) [AS 3741] [MPLS: Label 2859 Exp 1] 172 msec
5 core1a-dock-gi1-0-0-101.ip.isnet.net (168.209.164.0) [AS 3741] 184 msec 176 msec
6 core1b-dock-gi0-0-2.ip.isnet.net (168.209.246.1) [AS 3741] 176 msec * 172 msec
7 gi8-13.mpd01.lon02.atlas.cogentco.com (149.6.148.1) 180 msec 176 msec 172 msec
8 te4-2.ccr01.lon01.atlas.cogentco.com (130.117.1.201) 172 msec
9 vl3493.mpd01.lon01.atlas.cogentco.com (130.117.2.17) 172 msec
te3-1.mpd01.lon01.atlas.cogentco.com (130.117.3.225) 176 msec
vl3493.mpd01.lon01.atlas.cogentco.com (130.117.2.17) 172 msec
10 te1-2.ccr01.lon05.atlas.cogentco.com (130.117.49.94) 172 msec 176 msec
11 149.6.2.194 184 msec 180 msec 180 msec
12 rrba-ip-esr-1-ge-6-0-0.telkom-ipnet.co.za (196.43.11.166) [AS 5713] 184 msec 184 msec 184 msec
13 first-national-bank-gw.telkom-ipnet.co.za (196.25.207.178) [AS 5713] 188 msec 188 msec 188 msec
=== snip ===


Update:
Dear Readers

I’ve decided that if I’m going to moan, complain, and accuse ISPs of FAILure on this blog, then I should at least follow my accusations up, and provide constructive post-mortem commentary (where possible). So here goes:

IS->FNBConnect traffic is flowing via FNB again. Also see the comment from Nick Treasure (IS).

Regards,
Simeon.

* 41.203.16.0/22 168.209.255.8 0 3741 2905 65419 i
* 41.203.20.0/23 168.209.255.8 0 3741 2905 65419 i
* 41.203.22.0/23 168.209.255.8 0 3741 2905 65419 i
* 41.203.24.0/21 168.209.255.8 0 3741 2905 65419 i
* 41.204.216.0/22 168.209.255.8 0 3741 2905 65419 i
* 41.204.220.0/23 168.209.255.8 0 3741 2905 65419 i
* 196.22.132.0/22 168.209.255.8 0 3741 2905 65419 i
* 196.22.136.0/21 168.209.255.8 0 3741 2905 65419 i
* 196.30.125.0 168.209.255.8 0 3741 2905 65419 i

It looks like Hetzner is announcing these to MTN Business (previously Verizon Business SA) using an autonomous system number from the private range (64512 to 65535). That is perfectly reasonable as long as MTN Business strips the private ASN when they announce these routes on the Internet.

Doing this is easy (neighbor x.x.x.x remove-private-AS), and as far as I can tell, they get it right on their links to SAIX and Verizon Business Europe/US, but as you can see, not on their link to Internet Solutions.

Because the primary link was both fast (by South African standards) and very expensive, I had to figure out a way to provision the backup link on a budget that was non-existent by comparison. What I came up with was to get Amobia to install a point-to-point link from UCT to their sister company Frogfoot Networks, who agreed to sell IP transit based on per-gigabyte traffic volume, instead of bandwidth. With this deal, UCT could minimize costs by only using the backup link during a primary link failure. The IT managers were very supportive, and so this pet project of mine materialized within a few months.

I leaned three things due to the backup-only nature of the setup that (in my view) isn’t immediately obvious from the standard CC[NP|IE]-curriculum’s version of BGP multi-homing:

1. Influencing inbound path selection:
   The usual mechanisms for influencing BGP inbound path selection (applicable to separate upstreams, so MED doesn’t count) is to perpend your own ASN a number of times on the path you want less likely to be selected. Apart from the difficulty of getting the length right, the other problem with this approach is that there is no guarantee that someone else’s routing policy won’t override your carefully adjusted AS-path length. In fact, most network operators will apply a policy where they adjust the local preference of routes so that customer routes are used before peering routes, and peering routes are used before transit routes.

   This is usually not a train smash for a multi homed end-user site that wants to balance load over their Internet links. In fact, they might specifically want BGP to select the best path, both inbound and outbound. However, since I wanted the backup link to only be used during a failure, I chose to short-circuit all of the BGP route selection by advertising more specific routes (two /17s instead of one /16) via the primary link.

2. What happens during a failure:
   This scheme works well when the link between the customer and the primary ISP fails: the /17s disappear from the global BGP tables, and the remaining /16 being advertised via the backup ISP is selected by everyone (including the primary ISP).

   If the primary ISP experiences a partial failure, for example, if their international link goes down, but the national peering keeps working, then the backup ISP still sees the /17s coming from the customer via the primary ISP, but traffic directed from international networks to the customer arrives at the backup ISP via the /16 route.

   What happens now? Does the backup ISP send this traffic via the /17 routes? It turns out in the UCT setup, Verizon Business South Africa discards that traffic. Perhaps they do some filtering to prevent transit between their upstreams and their peers?

   If you work for Verizon and you understand why this happens, please drop me a comment.

   A possible work-around could be to monitor reliable international beacon prefixes (or just the total number of prefixes) on the primary link, and withdraw the /17s if they disappear.

3. Adding peering to the mix:
   You may have noticed that I mentioned that Frogfoot Networks is the backup ISP, but I’m talking about Verizon above. This is because the Friendly Frogs agreed that they would not bill UCT for peering (traffic between UCT and them and their customers). To make the route selection work, I worked with Warwick at Frogfoot to implement a system of route-maps that allows UCT to mark routes for transit or peering. UCT advertises both /17s and the /16 to Frogfoot, but marks the /17s with the “Don’t announce to your upstream” community.