August 20th, 2009 12:49 pm
Posted by Douglas Eadline
Tags: linux, monitoring, multi-core, top
The top monitoring tool can be used in very simple ways to generate useful information
Top is one of the most used and feature full Linux monitoring utilities. Many users, however, are not aware of the many useful ways top can be employed. On the contrary, many users simple run the top command and watch the display. To finish they usually enter q or CTRL-C when they are done. There is much more that top can do. An example of a top screen is shown below in Figure One.
Figure One: Default top screen
The top display shows some general statistics for the machine then in the lower window shows the running processes. The default sorting of the process is by %CPU. You can also sort by memory or time usage by entering M and T respectively. You can move back to the %CPU view by entering P. Of course you can get further help by pressing h or consulting the man page.
The first thing most people don't know is you can vary the update delay using the -d argument. For instance, the standard delay between updates is 3 seconds. If you want to watch a remote system and not generate a huge amount of traffic or constant screen updating, you can enter a delay value. For instance;
$ top -d 30
will update every 30 seconds. If you need an update in the middle of the delay, just hit the space key. In a similar way, you can have top run once and record its output by entering the following;
$ top -b -n 1 >top.out
The -b is for batch mode (no input) and the -n is for the number of iterations. This command might be useful for getting a quick "snapshot" of a remote node using ssh or rsh. You can also look at individual processes or users. For instance;
$ top -p 21864
will show you only process 21864 and
$ top - u deadline
will show only deadline's jobs. This type of command can be useful if for example you want to see what your job is doing on a particular node. For instance, if I wanted to know what my jobs were doing on node024, then I could enter;
$ ssh deadline@node024 top -b -n 1 -u deadline
The resulting output would be something like:
top - 12:51:57 up 47 days, 14:30, 0 users, load average: 2.07, 2.02, 2.00
Tasks: 66 total, 3 running, 63 sleeping, 0 stopped, 0 zombie
Cpu(s): 76.5% us, 0.7% sy, 0.0% ni, 22.6% id, 0.0% wa, 0.1% hi, 0.1% si
Mem: 2056056k total, 538728k used, 1517328k free, 40644k buffers
Swap: 1959920k total, 0k used, 1959920k free, 369464k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6561 deadline 18 0 44396 4516 800 R 97 0.2 25970:31 mds
6562 deadline 25 0 44396 4496 800 R 95 0.2 25970:27 mds
6556 deadline 16 0 5324 1040 884 S 0 0.1 0:00.00 bash
6558 deadline 17 0 5324 1040 884 S 0 0.1 0:00.00 bash
There are some other simple top commands that can be useful. Since almost all processors are multi-core these days, it is helpful to see all the cores in the display. For instance, in Figure One there are four processes running on a quad-core processor. If you want to see the individual cores, press 1. As an example, the display in Figure Two is the same as Figure One, but now showing the cores.
Figure Two: Top showing all cores
Another piece of information users often want to know is are my processes moving from core to core. You can monitor this with top by entering f to change the display fields. This panel allows you to toggle the possible display fields that top uses. In this case, we will turn on the "Last used cpu" field which indicates the last core on which the process was running. Figure Three is an example of this display for the same processes running in Figures Two above.
Figure Three: Top with "last cpu" column "P"
As you can imagine there are many more things top can display. We cannot go into all the details here. Consulting the man page will get you started on more top options. Just remember, you probably just learned enough here to impress your friends with some fancy top mojo!
JOIN THE CONVERSATION
You must be a Registered Member in order to comment on Cluster Connection posts.
Members enjoy the ability to take an active role in the conversations that are shaping the HPC community. Members can participate in forum discussions and post comments to a wide range of HPC-related topics. Share your challenges, insights and ideas right now.
Login Register Now