Sometimes in the field of networking, you are forced into a sub-optimal scenario. In this case, I am dealing with something I call link-layer abstraction.

Normally, an Ethernet link interconnects two devices, and when either of them disconnect, the link goes down on both sides. This can be seen when you link a computer into a switch; unplug the wire from the computer, and the switch knows straight away. You can be confident that there is no other equipment involved in this link; it’s a true point-to-point piece of wire.

There are certain situations where this doesn’t, or can’t happen. As a result one side of the link may assume the far end is still connected. This is what I mean by link-layer abstraction; there is no guarantee of an end-to-end signal to signify both devices are connected with each other.

Some typical examples of that cause link-layer abstraction are:

• VPLS
• Ethernet-over-SDH
• Use of intermediate switches to overcome 100m distance limits on cat5e

In all of the above examples, you can only establish a true physical link to the provider equipment. This could be the near end mux at either side of the link in the case of Ethernet over SDH.

Firewalls don’t appreciate this

Of course, all of this is bad for failover systems that rely on link-state. Juniper firewalls that run NSRP are usually configured to rely on link-state to determine a cluster failover. This would work just great if the mux near your firewall had a problem; the firewall would detect your link as being down and failover to the standby node. However, should the same happen at the far end of the link, your firewall would still be sitting there assuming all is well, as the your end of the link going to the local mux would still be up.

One obvious solution would be to set up IP tracking on the interface, but there are situations where this isn’t possible. For example, setting a track-IP is impossible on a /30 link-net that is typical when you have a BGP feed (because you also need a ‘manage-ip’ on the network to send the pings from).

In my case, I’m dealing with two SDH-over-Ethernet links which suffer from this problem. I am also running BGP routing to the provider from my Juniper firewall cluster. The diagram below shows how I plan to overcome the issues:

Running with standard NSRP, I have two firewalls. The lines shown in blue represent the primary /30 link to provider router 1, and the red represent the backup /30 link to provider router 2.

Each of the links terminate on a switch, using a private VLAN to link to both firewalls simultaneously. Remember, only one of these firewalls will be active at any time.

And the BGP?

With this topology, I’m able to run a BGP session from the active firewall to both provider routers, giving maximum redundancy with the setup. Although I have abstracted the links further myself by running them through switches, the situation is no worse than it was already.

Running both BGP sessions on the same firewall isn’t much of a problem. To do this, you need to configure a loopback group, and make the physical interfaces part of it. Here’s a config snippet showing this:

set interface loopback.1 ip 1.2.3.0/26

set interface ethernet0/1 ip 2.2.2.2/30

set interface ethernet0/2 ip 3.2.2.2/30

set interface "ethernet0/1" loopback-group "loopback.1"

set interface "ethernet0/2" loopback-group "loopback.1"

The loopback interface has the IP of the block that’s being announced via BGP. The physical interfaces are numbered with their /30 IP addresses. Then they are bound together using a loopback group. Just add the necessary BGP config to set up the sessions and away you go.

Using a loopback group is a great way of running a dynamic routing protocol with NSRP. If you configure your VPNs, DIPs and other NAT to use IPs within the loopback network range, then your failover will be totally seamless. This applies equally to a link-net failure and a firewall device failure.

Although failure detection can no longer rely on link-state, it’s better than no failover at all. However, this does mean relying on the BGP hold timer expiry to stop traffic from being black-holed for a while. To get over this, the BGP hold timer is set to 6 seconds, which implies a hello timer of 2 seconds (hold-timer/3).

Other tricks

It’s also possible to employ a neat trick on Cisco switches called Link-State Tracking. This tutorial goes into some detail about this feature. The idea would be to speed up failover time in the case of a local mux failure. The ‘upstream’ port would be configured as the link to the mux, and the downlink ports the links to the firewalls. If the local mux has a problem, the firewall links are downed straight away and the BGP session can drop immediately without waiting for hold-timer expiry.

]]> http://netsack.wordpress.com/2009/09/23/dealing-with-nsrp-bgp-and-link-layer-shenanigans/feed/ 0 netsack Redundant BGP using NSRP in a link-layer abstraction scenario