The Penguin's Practical Network Troubleshooting Guide - page 2

Start with Cable Testing

Suppose you have a remote Web server that is not responding. I know this is horridly obvious, but sometimes people forget that the first step in network troubleshooting is always to confirm connectivity, and for this we have our little friend ping. There is more to ping than you may realize, so let's take a closer look. First, always make sure you are connected to the network. I've been bitten by this more than once. Next, ping localhost:

$ ping -c4 -a localhost

-c4 sends four ICMP ECHO_REQUESTs, and -a makes it ping audibly. Then ping the IP of the box you're trying to connect to. Then ping the hostname. With three simple commands you have confirmed that your NIC is up, and tested both connectivity and DNS.

ping messages give some clues as to where the problem lies. This example shows that the hostname resolves, and there is a route to the host, but ping is receiving no responses of any kind:

$ ping -c10 somename.com

PING somename.com (1.2.3.4) 56(84) bytes of data.

--- somename.com ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9999ms

You can try pinging the IP to see if it's a DNS problem:

$ ping -c10 1.2.3.4
PING 1.2.3.4 (1.2.3.4) 56(84) bytes of data.

--- 1.2.3.4 ping statistics ---

10 packets transmitted, 0 received, 100% packet loss, time 8999ms

Nope, it's not DNS. Chances are the entire remote network is offline, because you should at least get a "Destination Host Unreachable" message from the network's border router. But it could be a problem anywhere between you and the remote machine. Trying to pinpoint an Internet trouble spot is difficult and frustrating. In the olden days traceroute was a good tool for this, but in these here modern times a lot of network admins program their routers to not respond to traceroute packets. ping is often blocked as well.

A good alternative is to use tcptraceroute. tcptraceroute sends TCP packets instead of UDP datagrams or ICMP ECHO requests like traceroute, so it's unlikely they'll be blocked. And a nice bonus is tcptraceroute traverses NAT firewalls. Use it like this:

$ tcptraceroute somename.com
Selected device wan, address 11.22.33.44, port 32783 for outgoing packets
Tracing the path to somename.com (1.2.3.4) on TCP port 80 (www), 30 hops max
 1  12.160.224.21  18.383 ms  15.855 ms  14.915 ms
 2  router.foo.net (12.160.224.22)  16.884 ms  15.412 ms  14.670 ms
 3  12.17.165.97  15.942 ms  16.928 ms  14.914 ms

 4  12.118.134.5  45.727 ms  44.255 ms  43.988 ms
 5  12.123.44.2  57.315 ms  55.858 ms  63.676 ms
 6  tbr1-p012501.st6wa.ip.att.net (12.122.12.157)  56.307 ms  60.762 ms  54.591 ms
 7  12.127.6.73  60.220 ms  59.547 ms  52.862 ms
 8  POS2-0.BR1.SEA1.ALTER.NET (204.255.174.241)  51.870 ms * 51.498 ms
 9  0.so-4-2-0.XL1.SEA1.ALTER.NET (152.63.105.82)  55.560 ms  52.386 ms  55.570 ms

10  0.so-7-0-0.XL1.DCA6.ALTER.NET (152.63.41.129)  117.896 ms  115.958 ms  121.841 ms
11  0.so-6-0-0.WR1.IAD6.ALTER.NET (152.63.39.113)  119.130 ms  120.162 ms  136.860 ms
12  so-1-0-0.ur1.iad6.web.wcom.net (157.130.59.78)  117.899 ms  126.721 ms  118.613 ms
13  206.112.64.6  120.843 ms  117.494 ms  116.865 ms
14  * * *
15  uu-3-166.hostdomains.com (22.33.44.55) [open]  120.232 ms  145.716 ms  176.254 ms

This shows that tcptraceroute can trace the remote server all the way to its origin, which is the fictional hostdomains.com, a Web hosting service. So now we know the pipeline is open end-to-end, but the somename.com server is not reachable, and we know which service provider to nag to fix it. If it were your own remote location, this would tell you that the problem is at your remote site.

If it's possible, set up a direct dial-in connection to any remote server that you are running. This will let you find out quickly if it's up or not, and also to perform troubleshooting tests from the other direction.