We have an on call rotation at $JOB where, for a week, a given engineer is responsible for responding to after hours phone calls as well as tickets and ACL requests. During a recent turn of mine with the football, I got an ACL request that didn’t make sense, and after working it through with the DB Admin that made the request, I figured I’d write this. It’s sad that it’s probably mostly networking people that read the blog, because they already understand everything this will explain. It’s the DBAs, Developers, and other IT support personnel that could benefit from what I go over here.
I don’t know if this will become a full blown series, but the Network Engineers frequently get requests that result in the realization that people often don’t understand simple networking concepts, and thus blame the network for the issue, when the network is working just fine. In this case, I’m going to talk about DNS: the Domain Name System, and why simple misunderstandings (or misconfigurations) can cause things not to work as expected … when the network (and even the DNS servers) are working just fine.
The exchange regarding the situation went something like this:
DBA: I need an ACL to open MS file share ports between 10.0.0.20 and 10.0.0.21.
Me: That’s not needed; those systems are on the same subnet and the firewall doesn’t get involved.
DBA: But I can’t reach one from the other.
Me: It’s not the firewall that’s the issue.
DBA: It says it can’t connect … oh wait, I can connect if I use the IP, but not the name.
Me: The FQDN or just the hostname?
How Things Actually Work
Behind the scenes, when you feed an app (like when you try to map a drive or ssh or ftp or even go to a web page) a host name, it takes that name and asks DNS (assuming it’s properly configured), “I want to go to this host name. What’s the address?” If DNS can find it, it responds with that address. DNS isn’t very forgiving. It wants an FQDN address that has to match exactly with something it knows about (or knows who to ask about other domains) before it will give you an address. I could go into great detail about how DNS works across the Internet (because your DNS server only knows about its own domain, and will ask the rest of the DNS systems about all the others), but it’s not necessary for the meat of this post. Just be aware that when you feed an application a name, it’s going to have to look up the IP from somewhere …
What’s an FQDN?
“FQDN” is shorthand for “Fully Qualified Domain Name.” Servers often have a “short” hostname – examples: “www”, “mypc”, db-server1″. Those are all fine and dandy, and on a properly configured server/workstation, you might even be able to reach those servers with just the short hostname. However, DNS doesn’t know about the names like that.
DNS always wants to see an FQDN. Something like “www.example.com”, where “example.com” is considered the “domain name”. This is further complicated by the fact that Windows PCs within an Active Directory Domain often add the domain name automatically before attempting to ask DNS for the IP address. This makes things like mapping a drive deceptively simple: there’s a lot going on in the background that allows “\\file-serv1\mount” to actually work … not the least of which is your PC asking DNS where “file-serv1.your-domain.com” is, even though you didn’t include the “your-domain.com” part.
But what if, for whatever reason, “file-serv1” isn’t in “your-domain.com”? The thing is, it’s possible to give your system configuration options to tell it to search a list of domains. If the server administrator that puts your servers together remembered to do it on one system but not another, it’s very possible you’ll run into cases where using the short name works on one server, but not the other.
How Do You Detect This Issue?
Well, assuming you know the FQDN or the IP, try either one of those. If they work … then it’s not the network.
What’s The Best Way To Avoid The Issue?
Frankly, just use the FQDN. Get used to typing it. In this case, they may have resorted to putting entries in “etc/hosts” (which on Windows systems is buried in a system directory, but on Linux boxes is actually /etc/hosts), but that’s ugly and if you decided to change the IP of the target server at some point, you have to remember to change it in all the etc/hosts files on every box that needs to know about it, rather than just making sure your DNS admin changed the DNS records appropriately.
If this stops someone from blaming the network for the inability to reach a server just once, then the time spent writing this post was worth it.