Eric Stewart: Running Off At The Mouth

I will admit on a regular basis that I don’t know a lot about the things I talk about.  I also am the type of individual that learns by doing, particularly after Googling about something I’ve encountered.  At my new job (one that I’ve been wanting for, oh, most of my employed life), Google is occasionally replaced by my boss, who is pretty much the smartest person I know (and he’s also nice enough not to hold your relative ignorance against you).  So, when I start spouting off stuff about “IPv6 router advertisements”, “Cisco ‘monitor session’ configurations”, and the like, I’m willing to admit that I might be talking out of my ass a bit.  But throughout my life, I’ve been right more often than I’ve been wrong when I do this.

One of the first tasks given to me at this new job (at my old employer, so it was kind of like coming home after a year away to really focus on the stuff I wanted to do for the rest of my employed life) was to replace a couple of what we call “monitor” servers.  They have two primary tasks:

1. Poll devices via MRTG and collect data,
2. Have high speed (10Gb) interfaces that simply accept packets from connected equipment for the purposes of packet capture, allowing a network engineer to diagnose and troubleshoot issues

For #2 up there, the interfaces in question usually have no IP (either v4 or v6).  They are simply connected to ports on a router that’s been configured as a “monitor session” port, where an engineer can direct certain traffic to.  The newly installed systems run a fairly recent release of CentOS 5.x, which is supposed to be essentially equivalent to Red Hat Enterprise Linux 5.x (RHEL).  However, if you run RHEL and are viewing this document, YMMV.

So, these monitor interfaces accept packets (actually, Ethernet frames), and these frames may even be for networks that the system itself is not on.  Or perhaps they are.  It doesn’t really matter.

Problems rear their ugly head, though, when these monitor interfaces pick up IPv6 router advertisements (RA).  If not configured to ignore these, the kernel may just think it should pay attention to the RA’s it sees on this interface.  When this happens, the system starts to think, “Hey, maybe I should use that interface instead of that other one to send my IPv6 data out.”  IPv6 packets then become roaches, and the system becomes a roach motel: Packets come in, but they don’t come out.

Assuming you have some other connectivity to the system (either through a console, or IPv4 is working fine), you can take a look at:

route -A inet6

and you’ll see that certain networks aren’t routing through the “main” interface of the device, but rather are routing through your monitor interface.

Here’s how you fix that – and here’s what to watch out for.

/etc/sysctl.conf: Your Friend, if you configure it right

You can see what sysctl parameters are configured with:

sysctl -a

You can apply changes to /etc/sysctl.conf with:

sysctl -p

For the sake of argument, we’ll say that you don’t use automatic configuration of IPv6 on your network.  For that, you’d want:

# disable ipv6 autoconf
net.ipv6.conf.default.autoconf = 0
net.ipv6.conf.all.autoconf = 0

In the config (I’ll go over my understanding of the differences between “default” and “all” later, as it becomes really relevant, and the reason why I bothered to write this post).

Next in our example, your monitor interface.  Let’s just arbitrarily say the interface is eth2.  While there might be other ways to do it, it is my understanding that you’ll want to include the following lines in your config:

net.ipv6.conf.eth2.accept_ra_rtr_pref = 0
net.ipv6.conf.eth2.accept_ra_pinfo = 0
net.ipv6.conf.eth2.accept_ra_defrtr = 0

These will stop eth2 from paying any attention to RA’s (on CentOS 5.x; CentOS 4.x is a different creature, and does not have these options available, and in my experience so far, hasn’t exhibited the problem).  This assumes that after you’ve edited your /etc/sysctl.conf that you’ve applied (sysctl -p) the configuration.  EDIT It’s important to note that the interfaces still receive the packets/frames (and will be seen via a tcpdump or wireshark capture); the kernel just doesn’t process them. /EDIT

all vs default

Here’s the thing: the settings for a specific interface (and any “all” configuration options) only apply when they are applied.  That is to say, they are not applied on the fly.  They are applied when /etc/init.d/network is started or restarted, but let’s say you take your monitor interface and bounce it:

ifdown eth2
ifup eth2

Bring it down, bring it back up.  What happens?

It turns out that the interface is considered “new”, and only the “default” configuration is applied, not the custom configurations for the interface that might be in /etc/sysctl.conf.  To clarify, when “sysctl -p” is run:

• “all” and device-specific configuration options apply to interfaces that are currently up;
• “default” configuration options apply to interfaces that appear in the future.

When I figured this out, I applied the following logic and sent an email to all of the engineers in my group:

• Assuming that, if you’ve got to bounce the “main” interface for some reason, you’re going to make sure you do it right and do an “/etc/init.d/network restart”,
• But if you just need to bounce the monitor interface, you might just do an “ifdown/ifup”,

I included the following lines in our /etc/sysctl.conf for the effected systems:

net.ipv6.conf.default.accept_ra_rtr_pref = 0
net.ipv6.conf.default.accept_ra_pinfo = 0
net.ipv6.conf.default.accept_ra_defrtr = 0

By default, the kernel uses the following definitions (visible per a “sysctl -a”):

net.ipv6.conf.all.accept_ra_rtr_pref = 1
net.ipv6.conf.all.accept_ra_pinfo = 1
net.ipv6.conf.all.accept_ra_defrtr = 1

So, our “main” interface(s), which have no other special configuration (other than the disabling of IPv6 autoconfiguration through the initial “autoconf” options above) should always accept RA’s (unless some fool accidentally “ifdown/ifup”s it/them).  EDIT Our systems’ “main” interface is a bonded one, and the above config/assumptions avoid having to potentially put in entries for the bond interface as well as all of its slaves. /EDIT  Everything else initially “up” during a normal boot sequence, or if “sysctl -p” is run, will be configured per its custom configuration (otherwise, they would be configured per any “all” configuration options).  However, if a monitor port is bounced, it will get the “default” options, and if a new one is brought up (we usually have them up on boot, but should we have to plug in a new one, it might not be “up” until “ifup” is used), it won’t bring the system’s IPv6 connectivity to a halt.

Hopefully someone else will find this useful and won’t find it too confusing.