Massive Route leak causes Internet Slowdown

Earlier Yesterday a massive route leak initiated by Telekom Malaysia (AS4788) caused significant network problems for the global routing system. Primarily affected was Level3 (AS3549 – formerly known as Global Crossing) and their customers. Below are some of the details as we know them now.

Starting at 08:43 UTC today June 12th, AS4788 Telekom Malaysia started to announce about 179,000 of prefixes to Level3 (AS3549, the Global crossing AS), whom in turn accepted these and propagated them to their peers and customers. Since Telekom Malaysia had inserted itself between these thousands of prefixes and Level3 it was now responsible for delivering these packets to the intended destinations.

This event resulted in significant packet loss and Internet slow down in all parts of the world. The Level3 network in particular suffered from severe service degradation between the Asia pacific region and the rest of their network. The graph below for example shows the packet loss as measured by OpenDNS between London over Level3 and Hong Kong. The same loss patterns were visible from other Level3 locations globally to for example Singapore, Hong Kong and Sydney.

Packet loss London to Hong Kong over Level3

At the same time the round trip time between these destination went up significantly as can be seen in the graph below.

Round trip time London to Hong Kong over Level3

Time line

By just looking at the number of BGP messages that BGPmon processed over time as can been seen in the graph below, there’s a clear start where all of a sudden the number of BGP updates increased. When we look closer at the data it becomes clear that this increase in BGP messages starts at 08:43 UTC and aligns exactly with the start of the leak and the start of the packet loss issues. At around 10:40 we slowly observed improvements and at around 11:15 UTC things started to clear up.

BGP update messages

Let’s look at an Example
An example affected prefix is 31.13.67.0/24 which is one of the Facebook prefixes. The AS path looked like this

1103 286 3549 4788 32934

If we look at this path we see that AS32934, Facebook, is the originator of the prefix. Facebook peers with 4788 and announced it to its peer Telekom Malaysia (AS4788) which in turn announced it to Level3 (AS3549) which announced it to all of its peers and customers, essentially giving it transit and causing a major routing leak.

Because Telekom Malaysia did this for about 176,000 prefixes they essentially signalled to the world that they could provide connectivity for all these prefixes and as a result attracted significantly more traffic than normally. All this traffic had to be squeezed through their interconnects with Level3. As a result all this traffic was now being routed via Level3 and Telekom Malaysia was likely to hit capacity issues, which then resulted in the severe packet loss issues as users reported on Twitter and as we’ve shown with the data above.

The 176,000 leaked prefixes are likely all Telekom Malaysia’s customer prefixes combined with routes they learned from peers. This would explain another curious increase in the number of routes Level3 announced during the leak time frame.

The graph below shows the number of prefixes announced by Level3 to its customers. Normally level3 announces ~534,000 prefixes on a full BGP feed. These are essentially all the IP networks on the Internet today. Interestingly during the leak an additional 10,000 prefixes were now being observed. One explanation for this could be that these are more specific prefixes announced by peers of Telekom Malaysia to Telekom Malaysia and are normally supposed to stay regional and not visible via transit.

Number of prefixes on Level3 full IPv4 BGP table

Since Level3 was now announcing many more prefixes than normally, it would have hit Max prefix limits on BGP session with its peers. These peering sessions with other large tier1 networks carry a significant portion of the worlds Internet and the shutdown of these session would cause traffic to shift around even more and exacerbate the performance problems as well as causing even more BGP churn.

So in conclusion, what we saw this morning was a major BGP leak of 176,000 prefixes by Telekom Malaysia to Level3. Level3 erroneously accepted these prefixes and announced these to their peers and customers. Starting at 8:39 and lasting for about 2 hours traffic was being redirected toward Telekom Malaysia, which in many cases would have been a longer route and also caused Telekom Malaysia to be overwhelmed with traffic. As a result significant portions of traffic were dropped, latency increased and users world wide experienced a slower Internet service.

Things to remember in BGP

 

1. BGP uses TCP port 179 for transport. Router with the higher BGP router-id initiates BGP session from a random port.
 
2. The interface from which the BGP router ID is taken does not have to be running BGP. Any valid IP address can be used as BGP router-id, even an address that is not locally configured on the router.
 
3. The BGP router-id must be the same as the OSPF router-id for redistributing the routes from OSPF to BGP or vice versa.
 
4. If the 'network …' command is configured with the 'mask' option under the BGP process, then an exact match (network/mask) must exist in the IP routing table in order to advertise this route into BGP regardless of 'auto-summary' / 'no auto-summary' command. But the 'network …' command configured without the 'mask' assumes the default classful mask and if 'auto-summary' is configured then BGP will advertise a classful network only if any subnets of the classful network exist in the IP routing table. Again if the 'network …' command is configured without the 'mask' option and if 'no auto-summary' is configured, then that router must have the exact classful network in the IP routing table in order to advertise it in BGP.
 
5. To accept and attempt BGP connections to the external peers residing on networks that are not directly connected, we need to use either 'neighbor ebgp-multihop …' or 'neighbor ttl-security …' command. These two commands are mutually exclusive. We can use another command 'neighbor disable-connected-check' to accomplish the same task if the BGP neighbor is one-hop away.
 
6. The synchronization rule states that an iBGP learned prefix cannot be considered best unless there is a matching IGP route for that BGP prefix. BGP only advertises what it considers the best path. This issue can be resolved (1) by redistributing BGP routes into the IGP, (2) by creating a full-mesh of IBGP routers and disabling the synchronization, or (3) by creating a GRE tunnel. When BGP is synchronizing with OSPF, the router ID must match in both protocols in order to make it work.
 
7. When a prefix is received from an eBGP neighbor, it is advertised to both eBGP & iBGP neighbors. When a prefix is received from an iBGP neighbor, it is advertised ONLY to eBGP neighbors and not to any iBGP neighbors. To advertise iBGP leaned routes to other iBGP peers requires the use of route-reflectors or confederations or a full-mesh of iBGP peers.
 
8. While sending BGP updates, EBGP peers modify the next-hop value to its own IP address. But iBGP peers do not modify it.
 
9. The ‘default-information originate’ command, however, requires explicit redistribution of the route 0.0.0.0. . Default routes can be injected into BGP in one of three ways: (1) using the 'network …' command (default route must exist in the local routing table), (2) using the 'default-information originate' command (a redistribution statement must also be configured to redistribute the default route from the local routing table to the BGP table), and (3) using the 'neighbor … default-originate [route-map route-map-name]' command (this method does not even check for the existence of a default route in the IP routing table). The 'default-information originate' command should not be configured with the 'neighbor … default-originate' command on the same router.
 
10. 'weight' and 'local-preference' are set inbound and they affect outbound traffic. But 'as-path' and 'med' are set outbound and they affect inbound traffic.
 
11. The weights assigned with the 'set weight …’ route-map command overrides the weights assigned using the 'neighbor… weight …' command.
 
12. Origin code 'i' is default on the BGP routes advertised by 'network ...', 'aggregate-address ...' (if all subnet has 'i'), and 'neighbor … default-originate' commands. And origin code '?' is default on the BGP routes advertised by 'redistribute ...', 'aggregate-address ...' (if any single subnet has '?', but can be changed using ‘attribute-map’ option), 'default-information originate', and 'bgp inject-map ...' commands.
 
13. When BGP originates a route with the ‘network …’ command, MED is copied from the metric of the original route.
 
14. BGP MED values are not passed beyond the receiving (neighbor) AS.
 
15. Enabling the ‘bgp deterministic-med’ command ensures the comparison of the MED variable when choosing routes advertised by different peers in the same autonomous system. Enabling the ‘bgp always-compare-med’ command ensures the comparison of the MED for paths from neighbors in different autonomous systems.
 
16. The default behavior of BGP routers that run Cisco IOS software is to treat routes without the MED attribute as having a MED of 0, making the route that lacks the MED variable the most preferred. The 'bgp bestpath med missing-as-worst' command can be configured to treat the route that missing MED as the least preferred one.
 
17. ‘bgp bestpath as-path ignore’ is a hidden command in Cisco IOS which allows BGP to not consider the AS path during best path route selection.
 
18. There are two ways to create an aggregate address under BGP. The first is to create a static route to null interface in the routing table for the aggregate address and then advertise it with the ‘network …’ command. The second way is to use the ‘aggregate-address …’ command.
 
19. By default when aggregation is configured in BGP, the 'atomic-aggregate' attribute is attached to the aggregate address if the 'as-set' argument is not used in the 'aggregate-address …' command. The 'as-set' argument reveals the AS numbers which can prevent a routing loop, and once 'as-set' is configured along with the 'aggregate-address …' command, the 'atomic-aggregate' attribute is automatically removed.
 
20. A router reflector and its clients are known collectively as a cluster. If the cluster contains a single route reflector, the cluster ID is the router ID of the route reflector. If the cluster contains multiple route reflectors, each RR must be manually configured with a cluster ID.
 
21. A client router in a route reflection cluster can peer with external neighbors, but the only internal neighbor it can peer with is a route reflector in its cluster or other clients in the cluster. Clients cannot peer with routers outside of their own cluster. However, the RR itself can peer with both internal and external neighbors outside of the cluster and can reflect their routes to its clients.
 
22. In case of route reflection, (1) routes from EBGP are advertised to EBGP, client, non-client (2) routes from client are advertised to EBGP, client, non-client (3) routes from non-client are advertised to EBGP, client.
 
23. When the 'no bgp client-to-client reflection' command is configured the RR does not reflect routes from one client to another. It does, however, continue to reflect routes from clients to peers outside of the cluster, and from peers outside of the cluster to clients.
 
24. Standard and extended BGP communities are removed from the reflected routes unless the 'neighbor ... send-community [both]' is configured on the route reflector. The link bandwidth community is removed from reflected route if the route-reflector performs IBGP multipath load-sharing for that route.
 
25. The “neighbor … nexthop-self” on router reflectors only affects the next hop of eBGP learned routes because the next hop of reflected routes should not be changed. To avoid a common configuration error for reflected routes, the “set ip next-hop” command should not be used in a route map to BGP route reflector clients.
 
26. Unlike route reflector environments in which only the route reflector itself has to support route reflection, all routers within a confederation must support the confederation functionality. 
 
27. EBGP routes external to the confederation are preferred over EBGP routes to member autonomous systems, which are preferred over iBGP routes.
 
28. AS_PATH types are AS_SEQUENCE, AS_CONFED_SEQUENCE, AS_SET, and AS_CONFED_SET. AS_SEQUENCE is an ordered set of AS numbers, and AS_SET is an unordered set of AS numbers. AS_CONFED_SEQUENCE and AS_CONFED_SET are the same as AS_SEQUENCE and AS_SET but are used only within BGP confederations.
 
29. When 'bgp bestpath med confed' command is configured, the router picks the confederation-internal path with the lowest MED and ignores the path with the external AS number.
 
30. BGP private autonomous system numbers are from 64,512 to 65,535
 
31. BGP prefixes can be filtered using (1) 'distribute-list', (2) 'prefix-list', (3) 'filter-list', (4) 'policy-list', (5) community/extended community lists, (6) 'route-map' .
 
32. For BGP, the ‘distance …’ command sets the administrative distance of the External BGP (eBGP) route. This command only affects the routing table and not the BGP table.
 
33. The 'network … backdoor' command has the same effect as the 'network …' command. The EBGP route is treated as a local BGP route, and the administrative distance is changed to 200. The difference is that the address specified by the network backdoor command is not advertised to EBGP peers.
 
34. iBGP routes are not redistributed into an IGP unless you use "bgp redistribute-internal" command under BGP routing process.
 
35. 'bgp inject-map ... exist-map ...' command injects prefixes in the local BGP RIB when a valid parent route exists. Only prefixes that are equal to or more specific than the aggregate route (existing prefix) can be injected. exist-map (route-map) must contain a 'match ip address prefix-list ...' command statement to specify the aggregate prefix and a 'match ip route-source prefix-list ...' command statement to specify the route source. If the parent route is a default route, we can inject any route out of it.
 
36. A BGP neighbor cannot be configured to work with both peer groups and peer templates. BGP peer templates and BGP peer groups are mutually exclusive.
 
37. Peer session template can inherit only one session template directly, but peer policy template can inherit multiple policy templates.
 
38. When the maximum number (as set by the ‘neighbor … maximum-prefix ...’ command) of prefixes are reached, the string "PfxRcd" appears in the entry, the neighbor goes to shutdown state, and the connection becomes idle.
 
39. No penalty is applied to a BGP peer reset when route dampening is enabled. Although the reset withdraws the route, no penalty is applied in this instance.
 
40. In case of iBGP multipath load sharing, when multiple iBGP paths installed in a routing table, a route reflector will advertise only one of the paths (one next hop).
 
41. For multiple paths to the same destination to be considered as multipaths, all attributes including weight, local preference, autonomous system path (entire attribute and not just length), origin code, MED, and IGP distance must be same. But if 'bgp bestpath as-path multipath-relax' command is configured, the AS paths still have to be the same length, but don't have to be identical.
 
42. Though BGP Multipath allows the installation of multiple BGP paths (for load sharing purpose) into the IP routing table for the same prefix, it does not affect the bestpath selection. A router still designates one of the paths as the best path and advertises this best path to its neighbors.
 
43. 'neighbor … dmzlink-bw' command can be used with eBGP and iBGP multipath features to enable unequal cost load balancing over multiple links. BGP can originate the link bandwidth community only for directly connected links to eBGP neighbors.
 
44. The 'bgp update-delay ...' command is used to tune the maximum time the software will wait after the first neighbor is established until it starts calculating best paths and sending out advertisements.
 
45. The routers configured with the “neighbor … local-as …” command prepend local-AS in inbound EBGP updates and prepend both actual AS number and local-AS number in outbound EBGP updates.
 
46. The “neighbor … local-as …” command is valid only if the peer is a true eBGP peer. It does not work for two peers in different sub-ASs in a confederation.
 
47. In a route-map, a continue clause can be executed, without a successful match, if a route map entry does not contain a match clause. But if a match clause exists, the continue clause is executed only if a match occurs. If no successful matches occur, the continue clause is ignored. The continue statement proceeds to the specified route map entry only after configured set actions (if any) are performed.
 
48. When multiple values are configured in the same community list statement, a logical AND condition is created. All community values must match to satisfy an AND condition. When multiple values are configured in separate community list statements, a logical OR condition is created. The first list that matches a condition is processed.
 
49. While redistributing OSPF into BGP, by default only OSPF intra-area and inter-area routes are redistributed into BGP.
 
50. When a BGP router with synchronization enabled has also a OSPF route (redistributed from BGP) for a iBGP-learned route, then the OSPF ASBR router-id must match the originating BGP router-id in order to synchronize BGP route with OSPF route.
 
51. An “update group” is a group of peers with a common outbound policy which will be converged as if they are in a peer-group.

 

Rib-failure BGP

During the times of my CCIE preperation, most of the time whenever I am practicing BGP, I used to get into RIB Failure scenarios and I used to have a tough time to fix them. During the time of my practice, I have completely understood the situtations when the RIB Failures can happen and how to fix them.

Below is the article that i wanted to share on how to troubleshoot RIB Failures in BGP.


Sometimes when configuring BGP you’ll come accross routes that show rib-failure. What exactly does this mean?

Have a look at this output:

R3#sh ip bgp
   Network          Next Hop            Metric LocPrf Weight Path
r> 172.16.220.0/24  172.16.220.1        0             0 3 i
*> 192.68.0.0/16    172.16.220.1        0             0 3 {2,1} i
*> 192.68.10.0      172.16.220.1                      0 3 2 i

172.16.220.0/24 is showing up as r> – but what exactly is going on? There is a command you can use to see what’s happened: show ip bgp rib-failure

R3#sh ip bgp rib-failure
Network            Next Hop                      RIB-failure   RIB-NH Matches
172.16.220.0/24    172.16.220.1        Higher admin distance              n/a

Here it’s telling me that the BGP could not be injected into the routing table as there is already a route with a higher administrative distance there. This is proved with the ip routing table:

R3#sh ip route 172.16.220.0
Routing entry for 172.16.220.0/24
  Known via "connected", distance 0, metric 0 (connected, via interface)
  Routing Descriptor Blocks:
  * directly connected, via FastEthernet0/1
      Route metric is 0, traffic share count is 1

Essentially a RIB-failure is a note letting you know that the route is in BGP, but it has not been injected into the IP routing table even though it is a valid and best route.

BGP Confederations – How, What and Why.

During your BGP studies, you’ll come across BGP confederations a couple of times. There are a few things that are easy to miss, and I’d like to clear it up here.

This will be both theory and practical, and I’ll be using this topology to explain things:

Our company, AS 65535 has a multitude of routers running BGP in our core.

N.B: R2 and R4 will NOT be running BGP at all. We are connected to 2 ISP’s – AS100 and AS200. We are also running OSPF internally inside the organisation.

BGP confederations allow your BGP deployment to scale quite nicely internally. Remember the rule of BGP split horizon – i.e. a BGP router learning a route from an iBGP peer will not advertise that to another iBGP peer. Confederations can help with this, as each intra-confederation connection is actually a special eBGP peering and not a regular iBGP peer.

BGP confederations can also help with splitting up your IGP domains. IGP’s like EIGRP or OSPF cannot scale to gigantic routing table sizes. IGP’s also put more emphasis on convergence speed as opposed to stability like BGP. I know the topology I have is no-where near big enough, but it does allow me to show you how it splits these IGP domains. I am going to run OSPF in both Sub-AS 10 and 30, as well as EIGRP in 10, 20 and 30 so I can seperate the OSPF portion out completely. I am going to be running OSPF in area 0 in Sub-AS 10 as well as in 30, but these will be completely independent of each other.

Each router has a loopback which will be advertised. R1 is 1.1.1.1, R4 is 4.4.4.4 and so on. All iBGP and intra-confederation peers will be peered using the loopback IP addresses.

Configuring:

The ISP itself will have a normal BGP config, nothing special needs to be done. You do need to ensure you are configuring a peer with AS 65535. ISP1 and ISP2 do not know anything about the fact that we are running a confederation.

R1 config:

R1#
router bgp 100
no synchronization
bgp log-neighbor-changes
network 1.1.1.1 mask 255.255.255.255
neighbor 192.168.1.8 remote-as 65535
no auto-summary

R8′s config is like so. The BGP process must be configured under the Sub-AS number. In this case AS 10. The peer connectino between ISP1 and our company will NOT come up until I tell R8 that it should identify itself to ISP1 as being in AS 65535. As soon as the confederation identifier is in place, the peer connection will come up. BGP confederation peers just tells the router itself which AS’s are intra-confederation peers. If you do not add this then the router will assume any AS different to the one it’s using itself will be a full eBGP peer.

R1#
router bgp 10
 no synchronization
 bgp log-neighbor-changes
 bgp confederation identifier 65535
 bgp confederation peers 20 30
 network 8.8.8.8 mask 255.255.255.255
 neighbor 9.9.9.9 remote-as 10
 neighbor 9.9.9.9 update-source Loopback0
 neighbor 192.168.1.1 remote-as 100
 no auto-summary
!
router ospf 1
 log-adjacency-changes
 network 8.8.8.8 0.0.0.0 area 0
 network 10.1.1.16 0.0.0.3 area 0
 network 192.168.1.0 0.0.0.255 area 0

I’ve also added the next hop addresses into OSPF so I don’t need to use next-hop-self.
To do a quick check on the peer connection, have a look here:

R1#sh ip bgp sum

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
192.168.1.8     4 65535      17      17        4    0    0 00:10:06        1

The peer is up, and as far as R1 is concerned, R8 is in AS 65535
R2 is simply running OSPF and nothing else:

R2#
router ospf 1
 log-adjacency-changes
 network 2.2.2.2 0.0.0.0 area 0
 network 10.1.1.16 0.0.0.3 area 0
 network 10.1.1.20 0.0.0.3 area 0

Router 9 is running BGP, OSPF and EIGRP – I wouldn’t do this in the real world. It’s simply to prove a point later. It’s also peered with AS20, a Sub-AS. There is an important thing to note here. Basically iBGP sessions do not need the ‘ebgp-multihop’ command. iBGP peers do NOT have to be directly connected. When Sub-AS’s connect to each other they DO need it though otherwise the peer will simply not come up. You can see that the peer config to R8 does not have it while the peer config to R10 does have it. This is the config:

R9#
router bgp 10
 no synchronization
 bgp log-neighbor-changes
 bgp confederation identifier 65535
 bgp confederation peers 20 30
 network 9.9.9.9 mask 255.255.255.255
 neighbor 8.8.8.8 remote-as 10
 neighbor 8.8.8.8 update-source Loopback0
 neighbor 10.10.10.10 remote-as 20
 neighbor 10.10.10.10 ebgp-multihop 2
 neighbor 10.10.10.10 update-source Loopback0
 no auto-summary
!
router ospf 1
 log-adjacency-changes
 network 9.9.9.9 0.0.0.0 area 0
 network 10.1.1.20 0.0.0.3 area 0
 network 10.1.1.96 0.0.0.3 area 0
!
router eigrp 1
 network 9.9.9.9 0.0.0.0
 network 10.1.1.96 0.0.0.3
 no auto-summary

Router10 is peered with 2 other Sub-AS’s. It’s also running EIGRP:

#R10
router bgp 20
 no synchronization
 bgp log-neighbor-changes
 bgp confederation identifier 65535
 bgp confederation peers 10 30
 network 10.10.10.10 mask 255.255.255.255
 neighbor 3.3.3.3 remote-as 30
 neighbor 3.3.3.3 ebgp-multihop 2
 neighbor 3.3.3.3 update-source Loopback0
 neighbor 9.9.9.9 remote-as 10
 neighbor 9.9.9.9 ebgp-multihop 2
 neighbor 9.9.9.9 update-source Loopback0
 no auto-summary
!
router eigrp 1
 network 10.1.1.36 0.0.0.3
 network 10.1.1.96 0.0.0.3
 network 10.10.10.10 0.0.0.0
 no auto-summary

R3, R4, R11 and R12 are more of the same of what’s just been done. I’ll post just the configs here.

#R3
R3#sh run | begin eigrp
router eigrp 1
 network 3.3.3.3 0.0.0.0
 network 10.1.1.36 0.0.0.3
 auto-summary
!
router ospf 1
 log-adjacency-changes
 network 3.3.3.3 0.0.0.0 area 0
 network 10.1.1.36 0.0.0.3 area 0
 network 10.1.1.44 0.0.0.3 area 0
!
router bgp 30
 no synchronization
 bgp log-neighbor-changes
 bgp confederation identifier 65535
 bgp confederation peers 10 20
 neighbor 10.10.10.10 remote-as 20
 neighbor 10.10.10.10 ebgp-multihop 2
 neighbor 10.10.10.10 update-source Loopback0
 neighbor 11.11.11.11 remote-as 30
 neighbor 11.11.11.11 update-source Loopback0
 no auto-summary

R4#
router ospf 1
 log-adjacency-changes
 network 4.4.4.4 0.0.0.0 area 0
 network 10.1.1.44 0.0.0.3 area 0
 network 10.1.1.52 0.0.0.3 area 0

#R11
router ospf 1
 log-adjacency-changes
 network 10.1.1.52 0.0.0.3 area 0
 network 11.11.11.11 0.0.0.0 area 0
 network 172.20.1.0 0.0.0.255 area 0
!
router bgp 30
 no synchronization
 bgp log-neighbor-changes
 bgp confederation identifier 65535
 bgp confederation peers 10 20
 network 11.11.11.11 mask 255.255.255.255
 neighbor 3.3.3.3 remote-as 30
 neighbor 3.3.3.3 update-source Loopback0
 neighbor 172.20.1.12 remote-as 200
 no auto-summary

#R12
router bgp 200
 no synchronization
 bgp log-neighbor-changes
 network 12.12.12.12 mask 255.255.255.255
 neighbor 172.20.1.11 remote-as 65535
 no auto-summary

Now there are a couple things we need to note about these special BGP peerings. Usually, the next-hop address will change when an update is given to an eBGP peer. If we check R10′s BGP table though, we can see that the next-hop addresses have NOT changed: (192.168.1.1 is R1′s IP address; 172.20.1.12 is R12′s)

R10#sh ip bgp
BGP table version is 8, local router ID is 10.10.10.10
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*  1.1.1.1/32       192.168.1.1              0    100      0 (10) 100 i
*  8.8.8.8/32       8.8.8.8                  0    100      0 (10) i
r> 9.9.9.9/32       9.9.9.9                  0    100      0 (10) i
*> 10.10.10.10/32   0.0.0.0                  0         32768 i
*  11.11.11.11/32   11.11.11.11              0    100      0 (30) i
*  12.12.12.12/32   172.20.1.12              0    100      0 (30) 200 i

That means updates to confederation peers will have the next-hop stay the same. You need to ensure that those next hop addresses are known by all confederation peers otherwise you’ll get what I have above, most have no valid route.

If we check the BGP table on R3, we see the following:

R3#sh ip bgp
BGP table version is 20, local router ID is 3.3.3.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
r> 9.9.9.9/32       9.9.9.9                  0    100      0 (20 10) i
r> 10.10.10.10/32   10.10.10.10              0    100      0 (20) i
r>i11.11.11.11/32   11.11.11.11              0    100      0 i
*>i12.12.12.12/32   172.20.1.12              0    100      0 200 i

R3 can see that the IP 9.9.9.9 came through AS 20 and 10, even though all routers are in the same major AS.

The last thing I’d like to point out is the split of the IGP (OSPF in this case) Both Sub-AS 10 and 30 are running OSPF area 0. We can see how many times the SPF algorithm has run in each:

R9#sh ip ospf
 Routing Process "ospf 1" with ID 9.9.9.9
 
        SPF algorithm executed 3 times

R3#sh ip ospf
 Routing Process "ospf 1" with ID 3.3.3.3
 
        SPF algorithm executed 7 times

Let’s force the algorithm to run again by adding another loopback on Router9 and advertising it into OSPF:

R9#conf t
Enter configuration commands, one per line.  End with CNTL/Z.

R9(config)#int lo2
R9(config-if)#ip address 99.99.99.99 255.255.255.255

R9(config-if)#router ospf 1
R9(config-router)#network 99.99.99.99 0.0.0.0 area 0

If we now check the SPF algorithm again in Both Sub-AS’s:

R9#sh ip ospf
 Routing Process "ospf 1" with ID 9.9.9.9
 
        SPF algorithm last executed 00:00:56.144 ago
        SPF algorithm executed 4 times

R3#sh ip ospf
 Routing Process "ospf 1" with ID 3.3.3.3
 
        SPF algorithm last executed 00:27:18.572 ago
        SPF algorithm executed 7 times

We can see in Sub-AS 10 the SPF algorithm ran 56 seconds ago. In Sub-AS 30 however, it has not forced the algorithm to run again, proving that these IGP domains are completely separate from each other.
So that’s the basics of Confederations. They can be very useful for a number of reasons. Just be sure to remember how exactly they operate. Any questions, feel free to ask

Hidden Treasures of BGP - Part 1

Types of BGP Tables

Till now we all believe that BGP is having only a single routing table where it used to store the routes and process for the best path calculation. But we all are mistaken here actually BGP maintains three table one for storing incoming routes from neighbours, one for sending the routes to neighbours and one for installing the routes where you actually find the routes with next-hop address. The tables are given below:-

a) Adj-RIB-in
b) Adj-RIB-out
c) Loc-RIB

Adj-RIB-in stores the unprocessed information received from its peers. Here the best path selection occurs as per BGP attributes and after conformation path is entered into the local bgp table i.e Loc-RIB. From the local RIB table it conform the next-hop address if it reachable by IGP then the route is entered into the main routing table.

BGP at 18 - Lessons in Protocol Design

Meet Mr.Yakov Rekhter (Inventor of RFC 1918), walks us though his experience on how he has build BGP - Simply Awesome. Njoi