Troubleshooting Hung Systems: Spiked CPU

First off I wanted to thank Filip Hanik. He illustrated this technique in a presentation at SpringOne this year in New Orleans. I happened to actually give a similar presentation to my fellow engineers at my current employer.

The goal here is to figure out using the top command in linux combined with thread dumps to figure out which thread and java code is consuming CPU resources. Below is a step by step walk through on how to accomplish this.

Step 1

Execute top command to determine the process id (PID) for the JBoss server. It will be listed under “java”

Step 2

Run the top command to show only the threads associated with the java process.
Execute top –H –p PID
(In this example the PID is 10937)

Step 3

Execute kill -3
to obtain a stack trace (In this example the PID is 10937)
Note: You can also utilize the dump stack capability in the JBoss JMX Console under the ServerInfo MBean

Step 4

Take the PID of the thread you want to correlate and convert it to hex using calc on Windows.

Step 5

Open the log file and do a search for the hex representation (native ID) of the thread ID. In this example it will be “2ae1”

This effectively tells you the line of code that is responsible for consuming the CPU resources.

Advertisement