traceroute - a very useful troubleshooting tool which reveals the bottlenecks on the Internet.

January 17, 2007
I am sure anyone who is at the least Internet savvy, will be aware that to move data from one point say A to another point B across the Internet, it has to pass through a number of intermediary points say C, D,E.... But what many won't know is that your data is not transferred in one piece when it is sent over the net, rather, it is split into chunks of say 1500 bytes each, then each chunk is enclosed in what is known as a packet which contain some additional data such as the destination IP address and port number apart from some other details which provide the unique identity to the packet and finally it is sent across the net.

While the packets travel the path from point A to point B, each packet may take a different path depending upon diverse factors and eventually they are merged together in the same order at the receiving end to provide the document you sent in the first place.

The intermediate gateways through which the packets pass through before they reach the final destination are known as hops. So for data to travel from point A to point B on the net, it has to go through a number of hops.

Linux & Unix being network operating systems have a number of powerful tools which aid the network administrator to find out a wealth of data about their network and the Internet. One such tool is the ubiquitous traceroute.

The tool traceroute is available in all Unix and Linux distributions and is used to find out the potential bottlenecks in between your computer and a remote computer across the net. The usage of this tool is quite simple and is as follows:
# traceroute <domain or IP address>
Usually you have to be root to run this tool as it resides in the /usr/sbin directory. But if you use the full path, then you can run this tool as a normal user as follows:
$ /usr/sbin/traceroute <domain or IP address>

For example, this is the output I received when I ran a trace on the www.yahoo.com domain from my machine.
$/usr/sbin/traceroute www.yahoo.com

traceroute to www.yahoo.com (69.147.114.210), 30 hops max, 40 byte packets
1 10.2.71.1 (10.2.71.1) 21.965 ms 22.035 ms 22.111 ms
2 (ISP) (ISP gateway) 22.510 ms 25.716 ms 26.073 ms
3 61.246.224.209 (61.246.224.209) 69.212 ms 59.778 ms 63.334 ms
4 59.145.6.1 (59.145.6.1) 65.632 ms 64.750 ms 64.868 ms
5 59.145.11.69 (59.145.11.69) 63.562 ms 64.219 ms 63.742 ms
6 203.208.143.241 (203.208.143.241) 318.632 ms 307.733 ms 316.650 ms
7 203.208.149.25 (203.208.149.25) 317.534 ms 308.116 ms 307.507 ms
8 203.208.186.10 (203.208.186.10) 245.835 ms 247.878 ms 248.862 ms
9 so-1-1-0.pat1.dce.yahoo.com (216.115.101.129) 286.774 ms 289.702 ms so-1-1-0.pat2.dce.yahoo.com (216.115.101.131) 326.470 ms
10 ge-2-1-0-p141.msr1.re1.yahoo.com (216.115.108.19) 324.044 ms 324.497 ms 326.011 ms
11 ge-1-32.bas-a1.re3.yahoo.com (66.196.112.35) 333.479 ms 333.019 ms ge-1-41.bas-a2.re3.yahoo.com (66.196.112.201) 292.967 ms
12 * * *
13 * * *
14 * * *
15 * * *
.
. //Truncated for brevity
.
29 * * *
30 * * *
As you can see from the output spewed by traceroute, it defaults to a maximum of 30 hops. The first line of the output gives the IP address of the yahoo.com domain which is 69.147.114.210, the maximum number of hops traceroute will keep track of the packets before it reaches the destination and the size of the packets which is 40 bytes.

The next 30 or so lines show the IP address or domain name of the gateway servers through which the packets pass through as well as the time in milli-seconds of the ICMP TIME_EXCEEDED response from each gateway along the path to the host. traceroute program utilizes the IP protocol's time to live (TTL) field. By default, it starts with a TTL value of 1 but this value can be changed with the -f option.

Now lets take a closer look at the output of traceroute to the yahoo.com domain as shown in the listing above. As you can see, the second hop is always to ones ISP's gateway as shown by the address (I have removed the address of my ISP's gateway). On the same line, followed by the IP address, there are three time values in milli seconds. There are three values because traceroute by default sends simultaneously, 3 packets of 40 bytes each. And the three time values are the time taken to send the packets and receive a ICMP TIME_EXCEEDED response from the gateway. Put another way, these three values are the round trip times of the packets. So for the three packets to reach my ISP's gateway, and get an echo back, it takes 22.510 milli seconds, 25.716 ms and 26.073 ms respectively as is displayed by the values of the 2nd hop.

Lets look at the 5th and 6th hop in the output above. If you compare the times, you will find a drastic increase in the times. If it is 63.562 ms for the 5th hop, it is 318.632 ms for the 6th hop. This is because up till the fifth hop, the gateway servers were within the Indian sub-continent itself. Where as the gateway of the 6th hop is in Singapore and so it takes that much more time to get a reply. Generally, smaller numbers mean better connections.

Check out the 11th hop. It shows two domains with one domain for the first two packets and a different domain for the third packet.

And from 12th hop onwards I get a series of time outs as shown by the asterisks. So my trace of the www.yahoo.com domain resulted in a series of time outs and did not complete. The problems could be one of the following:
  • The network connection between the server on the 11th hop and that on 12th hop is broken.
  • The server on the 12th hop is down.
  • Or there is some problem with the way in which the server on the 12th hop has been setup.
To make sure, I did a ping of the www.yahoo.com domain and as expected, I received 100% packet loss as shown by the ping output below.
$ ping -c 2 www.yahoo.com
PING www.yahoo-ht2.akadns.net (69.147.114.210) 56(84) bytes of data.

--- www.yahoo-ht2.akadns.net ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1009ms
Usually this means I will not be able to access the concerned domain. But in yahoo.com's case, I was able to access the domain without any problem as in all probability, their website is mirrored across a number of servers spread across the world. So if one server is down, the query is re-routed to the next nearest server.

traceroute is a very useful tool to pin-point where the error occurs on the internet. It can also be used to test the responsiveness of a domain or server. For example, If your route to a server is very long (takes over 25 hops), performance is going to suffer. A long route can be due to less-than-optimal configuration within some network along the way.

Similarly, if you see in a trace output, a large jump in latency (delay) from one hop to the next, that could indicate a problem. It could be a saturated (overused) network link; a slow network link; an overloaded router; or some other problem at that hop. It can also indicate a long hop, such as a cross-country link or one that crosses an ocean (compare the timing of the 5th and 6th hop in the yahoo.com trace output above).

9 comments:

  • The reason you were not able to traceroute all the way to yahoo.com is because people block ICMP traffic in their firewalls, while permitting TCP80 (HTTP). This means that ping won't work but firefox will.

    Maxim.

  • That's why you can run traceroute with the -p option. You can then specify port number.

  • 1.) yahoo.com is pingable for me (in the US) just fine. yahoo is probably blocking pings/ICMP traffic from your network.

    2.) "dig yahoo.com" returns two ip addresses for me - 66.94.234.13 and 216.109.112.135. It would have been a good idea to try both.

  • Take a look at the 'mtr' utility too. Works in console as well, is somehow graphic and interactive and has more options. Basic usage is the same - 'mtr [ip_or_hostname]'.

  • Kagehi

    Actually, no. ICMP packets, while directed at ports, are never the less *not* the same as normal TCP/IP packets. If the router/firewall is blocking them right, then changing the port *won't* have any effect on if it works (or at least shouldn't, since control packets are "control packets", not "data packets"). Now, its almost standard for Linux systems to be using TCP/IP to do it. In both cases you are waiting for some sort of message. ICMP can be specifically asked to give some information that TCP/IP "can't", which may make the trace "slightly" faster. I think you could say, "Hello C, I want to send something to Z, tell me how long it took for the packet to be received by D.", where the path is A->B->C->D->E->etc. With TCP/IP all you can do is keep lengthening the time out, then wait to see if the returning data says it got some place the last one didn't. In principle, its the same result, but the method is very different.

    That said, if you are not using Linux, you are screwed with a internet router, because most have firewalls, and most block ICMP by default. Tracert for Windows, up to XP Pro, has "no" ability to change ports and uses ICMP, which is blocked. In fact, since ICMP is a control protocol, I am not sure its even "possible" when using it to set a different port, since only the routers know what to do with them and all should be using the same port for that operation, unlike TCP/IP, which, as I said, is a *completely different* protocol. Heck, even Windows Firewall has no specific "Block port X, which is used by ICMP." It specifically has entirely separate settings "just for" ICMP and what its allowed to do, if it can be received or sent, by how to where, etc.

    Wish I could find a "free" TCP/IP based one for Windows though. I have to help some people out once in a while running Windows and its impossible to tell the difference between the modem disconnecting from the ISP and a firewall blocking the ICMP based trace... :(

  • Kiri

    Try tcptraceroute which can be found here:

    http://michael.toren.net/code/tcptraceroute/

    Kiri

  • Frits

    look for hping2. you can do a traceroute mode, specify protocol and port number

  • JeffSoh

    hping doesn't run on Windows. Never been ported.

  • Daniel Torres Sandi

    Maybe Cartoreso network mapper would be useful (create map of large networks)

    http://tips-linux.net/en/content/cartoreso-network-mapper