Eric Stewart: Running Off At The Mouth

After spending the previous day attempting to change the post layout in two new racks that had some really efficient locktite (resulting in me leaving a little bit of my blood on said racks and unable to wear my wedding ring for a few days), I woke up bright (actually it was 4am and still dark out) and early to replace Place-Of-Employment’s primary DNS server.  This is not the DNS server we have for internal use – for that, we have a cluster of machines running behind a VIP (two clusters, and two VIPs, actually).  No, this was the primary server for serving up POE.edu’s DNS info.

Mother-Of-DNS was a very old machine, way out of warranty, but at least she was running CentOS 5.x, so moving to a new system with CentOS 6.x was mostly hitch-free.  One of the thing MOD also does is provide NTP syncing.  While the DNS was relatively easy, Nagios was nice enough to tell me that, while he was having no problem checking her offset via IPv4, IPv6 wasn’t working for some reason.

I go through the usual suspects:

• ip6tables looks right to me.
• netstat seems to report that she’s listening fine on all the right ports, interfaces, and protocols
• the ntp.conf was a copy over, so I checked it, and it looked fine …

But I noticed, when comparing MOD to her secondary (Slave-To-MOD, who had been upgraded last year to new hardware and CentOS 6.x), there were some differences.  Just a line or two that appeared to be IPv6 specific (it’s the second line below):

restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

I added the line in, restart ntpd, but no go – Nagios still isn’t getting a response over IPv6.

And lo and behold, Nagios didn’t know that it was supposed to be looking at STM’s ntpd via IPv6 – it was only configured for IPv4 monitoring.

I get around to doing a packet capture, and this is what I see:

[root@MOD tiv]# tshark -i bond0 port 123 and host (nagios-v6)
Running as user "root" and group "root". This could be dangerous.
Capturing on bond0
  0.000000 (nagios-v6) -> (MOD-v6) NTP 74 NTP Version 2, control
 22.306226 (nagios-v6) -> (MOD-v6) NTP 110 NTP Version 4, client
 22.306284 (MOD-v6) -> (nagios-v6) NTP 110 NTP Version 4, server
 88.309187 (nagios-v6) -> (MOD-v6) NTP 110 NTP Version 4, client
 88.309262 (MOD-v6) -> (nagios-v6) NTP 110 NTP Version 4, server
120.592323 (nagios-v6) -> (MOD-v6) NTP 74 NTP Version 2, control
154.310881 (nagios-v6) -> (MOD-v6) NTP 110 NTP Version 4, client
154.310923 (MOD-v6) -> (nagios-v6) NTP 110 NTP Version 4, server

What we have here is that the Version 2 lines are Nagios attempting to probe the box, and MOD not sending any traffic back.  But we also see normal NTP sync traffic between Nagios and MOD.  Hm.  Interesting, that.  Thing is, there’s a red herring here: the assumption someone might jump to is that there’s a version issue: check_ntp_peer apparently sends “Version 2” messages, and ntpd can be configured to ignore old version messages.  But that’s not it (at least not NTP version; I think there might be some bugginess in how NTP handles things between IPv4 and IPv6, because, when I let you know what the problem is, it’s still curious that the issue impacted IPv6 but not IPv4 queries).

It turns out that the problem was in the ACL line in ntp.conf (from ntp.conf(5)):

noquery
		     Deny ntpq(8) and ntpdc(8) queries.	 Time service is not
		     affected.

So, adding an additional line (the bottom one of the six below):

# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1
restrict -6 (nagios-v6) kod nomodify notrap nopeer

Fixed the issue, while still keeping the other restrictions in place.  If you feel confident enough, you can just eliminate all the stuff from “kod” on and open NTP to the address in question.