Eric Stewart: Running Off At The Mouth

So we use a Citrix Netscaler (actually, four of them) at work to provide clustering/failover/GSLB services where I currently work.  My most recent project involved a tftp server cluster.  We use tftp for a lot of things: VOIP phone configuration, switch OS upgrades, switch configuration uploading/downloading, etc.  When it comes to methods of clustering (particularly with tftp), you might see a reference to something called “Direct Server Return (DSR)”.

This post addresses the server side of the concept of having multiple servers behind a single IP address (I don’t have enough knowledge about the Netscaler to include more than a couple of conceptual notes).  With most services (which are usually TCP based), this isn’t complicated, and the cluster managers handle that perfectly well.  However, tftp is UDP based, and that’s just the start of the issues.

See, my boss wanted a specific set of behaviors from the load balancer and servers.  The following, while doable, was unacceptable to him:

bigrouter#conf n
Host or network configuration file [host]? 
This command has been replaced by the command:
 'copy <url> system:/running-config'
Address or name of remote host [(previously_used_ip)]? (virtual_ip}
Source filename [config.txt]? 
Configure using tftp://(virtual_ip)/config.txt? [confirm]
Loading config.txt from (real_server_ip) (via VlanXXXX): !
[OK - 19271 bytes]

What he would have preferred was:

bigrouter#conf n 
Host or network configuration file [host]? 
This command has been replaced by the command:
 'copy <url> system:/running-config'
Address or name of remote host [(previously_used_ip)]?(virtual_ip)
Source filename [config.txt]? 
Configure using tftp://(virtual_ip)/config.txt? [confirm]
Loading config.txt from (virtual_ip) (via VlanXXXX): !
[OK - 30946 bytes]

A minor difference, and not the easiest thing to arrange.  Also, he wanted this behavior for both IPv4 and IPv6, which made it a bit more complicated.

First, and most important, the Netscaler (or other cluster manager) needs to be told not to change the destination IP address of the packet when sending it to the server; just the MAC address (assuming Layer 2 adjacency).

Next, you need to have the virtual IP addresses actually on the server itself.  This can get a little tricky:

• For IPv4, you make a secondary loopback (/etc/sysconfig/network-scripts/ifcfg-lo:0) with:

  DEVICE=lo:0
  IPADDR=(virtual_ip)
  NETMASK=255.255.255.255
  ONBOOT=yes
  NAME=loopback

• For IPv6, you add a secondary IP to the main loopback by editing /etc/sysconfig/network-scripts/ifcfg-lo and adding:

  IPV6INIT=yes
  IPV6ADDR_SECONDARIES=(IPv6:Virtual::IP)/128

And then restart your networking (double check your “ifconfig” to make sure everything works properly).  This ensures that your server will accept packets coming into it’s connected interface that have the destination address of the virtual IPs.

Next, you need to change what IP addresses your tftp service listens to.  This wasn’t well documented and I ended up taking two or three shots in the dark before getting a solution that worked.

If you look at your /etc/xinetd.d/tftp configuration file, it should look something like:

# default: off
# description: The tftp server serves files using the trivial file transfer \
# protocol. The tftp protocol is often used to boot diskless \
# workstations, download configuration files to network-aware printers, \
# and to start the installation process for some operating systems.
service tftp
{
   socket_type = dgram
   protocol = udp
   wait = yes
   user = root
   server = /usr/sbin/in.tftpd
   server_args = -vvv -s /var/lib/tftpboot
   disable = no
   per_source = 11
   cps = 100 2
#  flags = IPv4
}

By default xinetd will launch a tftp session for every interface that gets a packet on.  That would work fine if the system passed traffic coming into the connected interface (like eth0 or em1) to the loopback, but it doesn’t.  It just takes the traffic at that interface and starts tftp there.  The trick is to use the “bind” configuration option and specify an address for xinetd to bind to for tftp.

But wait … you have two addresses you want to use!  You can’t put bind in there twice, and you can’t give bind more than one argument.  So what do you do?

The clue for me was the commented out “flags” argument.  So what worked for me was specifying the “service” stanza twice, like so:

# default: off
# description: The tftp server serves files using the trivial file transfer \
# protocol. The tftp protocol is often used to boot diskless \
# workstations, download configuration files to network-aware printers, \
# and to start the installation process for some operating systems.
service tftp
{
    socket_type = dgram
    protocol = udp
    wait = yes
    user = root
    server = /usr/sbin/in.tftpd
    server_args = -vvv -s /var/lib/tftpboot
    disable = no
    per_source = 11
    cps = 100 2
    bind = (virtual_ip)
    flags = IPv4
}
service tftp
{
    socket_type = dgram
    protocol = udp
    wait = yes
    user = root
    server = /usr/sbin/in.tftpd
    server_args = -vvv -s /var/lib/tftpboot
    disable = no
    per_source = 11
    cps = 100 2
    bind = (IPv6:Virtual::IP)
   flags = IPv6
}

And there you have it.  tftp connections work as desired (assuming a properly configured cluster manager), always reporting the virtual IP instead of the server’s real one.  Remember that these changes need to be made to all cluster members.  Oddly enough, tftp checks from Nagios to the real IPs still seem to work.  I would also guess that if you had multiple IPs (I mean more than two) this would still be a possible way to specify them separately.  xinetd will see them as separate services.

Running into issues?  Well, check your ACLs, if you have any.  Check iptables/ip6tables on the servers.  Ensure that routes exist for the virtual addresses.  These were some of the things I ran into while working on this.