* 41.203.16.0/22 168.209.255.8 0 3741 2905 65419 i
* 41.203.20.0/23 168.209.255.8 0 3741 2905 65419 i
* 41.203.22.0/23 168.209.255.8 0 3741 2905 65419 i
* 41.203.24.0/21 168.209.255.8 0 3741 2905 65419 i
* 41.204.216.0/22 168.209.255.8 0 3741 2905 65419 i
* 41.204.220.0/23 168.209.255.8 0 3741 2905 65419 i
* 196.22.132.0/22 168.209.255.8 0 3741 2905 65419 i
* 196.22.136.0/21 168.209.255.8 0 3741 2905 65419 i
* 196.30.125.0 168.209.255.8 0 3741 2905 65419 i

It looks like Hetzner is announcing these to MTN Business (previously Verizon Business SA) using an autonomous system number from the private range (64512 to 65535). That is perfectly reasonable as long as MTN Business strips the private ASN when they announce these routes on the Internet.

Doing this is easy (neighbor x.x.x.x remove-private-AS), and as far as I can tell, they get it right on their links to SAIX and Verizon Business Europe/US, but as you can see, not on their link to Internet Solutions.

Because the primary link was both fast (by South African standards) and very expensive, I had to figure out a way to provision the backup link on a budget that was non-existent by comparison. What I came up with was to get Amobia to install a point-to-point link from UCT to their sister company Frogfoot Networks, who agreed to sell IP transit based on per-gigabyte traffic volume, instead of bandwidth. With this deal, UCT could minimize costs by only using the backup link during a primary link failure. The IT managers were very supportive, and so this pet project of mine materialized within a few months.

I leaned three things due to the backup-only nature of the setup that (in my view) isn’t immediately obvious from the standard CC[NP|IE]-curriculum’s version of BGP multi-homing:

1. Influencing inbound path selection:
   The usual mechanisms for influencing BGP inbound path selection (applicable to separate upstreams, so MED doesn’t count) is to perpend your own ASN a number of times on the path you want less likely to be selected. Apart from the difficulty of getting the length right, the other problem with this approach is that there is no guarantee that someone else’s routing policy won’t override your carefully adjusted AS-path length. In fact, most network operators will apply a policy where they adjust the local preference of routes so that customer routes are used before peering routes, and peering routes are used before transit routes.

   This is usually not a train smash for a multi homed end-user site that wants to balance load over their Internet links. In fact, they might specifically want BGP to select the best path, both inbound and outbound. However, since I wanted the backup link to only be used during a failure, I chose to short-circuit all of the BGP route selection by advertising more specific routes (two /17s instead of one /16) via the primary link.

2. What happens during a failure:
   This scheme works well when the link between the customer and the primary ISP fails: the /17s disappear from the global BGP tables, and the remaining /16 being advertised via the backup ISP is selected by everyone (including the primary ISP).

   If the primary ISP experiences a partial failure, for example, if their international link goes down, but the national peering keeps working, then the backup ISP still sees the /17s coming from the customer via the primary ISP, but traffic directed from international networks to the customer arrives at the backup ISP via the /16 route.

   What happens now? Does the backup ISP send this traffic via the /17 routes? It turns out in the UCT setup, Verizon Business South Africa discards that traffic. Perhaps they do some filtering to prevent transit between their upstreams and their peers?

   If you work for Verizon and you understand why this happens, please drop me a comment.

   A possible work-around could be to monitor reliable international beacon prefixes (or just the total number of prefixes) on the primary link, and withdraw the /17s if they disappear.

3. Adding peering to the mix:
   You may have noticed that I mentioned that Frogfoot Networks is the backup ISP, but I’m talking about Verizon above. This is because the Friendly Frogs agreed that they would not bill UCT for peering (traffic between UCT and them and their customers). To make the route selection work, I worked with Warwick at Frogfoot to implement a system of route-maps that allows UCT to mark routes for transit or peering. UCT advertises both /17s and the /16 to Frogfoot, but marks the /17s with the “Don’t announce to your upstream” community.