Tracking down the process responsible for high disk I/O at irregular intervals can be challenging, especially if these peaks occur sporadically. Usually commands such as iostat, vmstat, and sar help to report disk I/O performance. But to find the offending process which causing high disk I/O we can use some specific tool. Let's discuss it here.
Check disk I/O per process
The following commands help to determine which process using I/O utilization.
- pidstat: It is used for monitoring individual tasks currently being managed by the Linux kernel. We can use it to check tasks' IO usage as well.
- iotop: It watches I/O usage information output by the Linux kernel (requires 2.6.20 or later) and displays a table of current I/O usage by processes or threads on the system.
- atop: The program atop is an interactive monitor to view the load on a Linux system. It shows the occupation of the most critical hardware resources (from a performance point of view) on the system level, i.e. CPU, memory, disk, and network.
- /proc/pid/io: You can get the IO stats by process from proc. The information is available under /proc/[PID]/io.
Note: If required you may also use strace command to identify files that a process is actively reading or writing.
Let's look into each command in detail.
Using pidstat
Before using pidstat, we must install the sysstat package in the system first. In Ubuntu system, do the following:
$ sudo apt install sysstatThen, run the following command to count the IO usage of the processes:
$ sudo pidstat -dl 5 5
14:11:25 UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
14:11:30 0 330 0.00 1.60 0.00 0 jbd2/sda1-8The above command means to count the current IO usage by process, with an interval of 5 seconds, and a total of 5 statistics. You should use the ‘-d’ argument to report IO statistics here.
The returned results said, the command “jbd2/sda1-8” run by UID 0 (root) with PID 330, reads 0 kilobytes from disk per second, writes 1.6 kilobytes to disk per second, with zero IO delay.
The output columns include the following information:
UID: The real user identification number of the task being monitored.USER: The name of the real user owning the task being monitored.PID: The identification number of the task being monitored.kB_rd/s: Number of kilobytes the task has caused to be read from disk per second.kB_wr/s: Number of kilobytes the task has caused, or shall cause to be written to disk per second.kB_ccwr/s: Number of kilobytes whose writing to disk has been canceled by the task. This may occur when the task truncates some dirty pagecache. In this case, some IO which another task has been accounted for will not be happening.Iodelay: Block I/O delay of the task being monitored, measured in clock ticks. This metric includes the delays spent waiting for sync block I/O completion and for swapin block I/O completion.Command: The command name of the task.
You can check only the IO usage of the specified process, using the ‘-p’ parameter to set the process ID:
$ sudo pidstat -dl -p 330 5 5
14:16:46 UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command
14:16:51 0 330 0.00 3.20 0.00 0 jbd2/sda1-8Alternatively, you can use keywords to specify specific commands, using the ‘-C’ parameter to set the keywords:
$ sudo pidstat -dl -C "jbd2" 5 5 14:19:00 UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command14:19:05 0 330 0.00 6.40 0.00 0 jbd2/sda1-8
Using iotop
Before using iotop, we should install it first. In Ubuntu system, just run:
$ sudo apt install iotopThen run the command iotop:
$ sudo iotopYou will see an interactive screen which pirints the system IO statistics like follows.
Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
Current DISK READ: 0.00 B/s | Current DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
1 be/4 root 0.00 B/s 0.00 B/s ?unavailable? systemd --system --deserialize 41
2 be/4 root 0.00 B/s 0.00 B/s ?unavailable? [kthreadd]
3 be/0 root 0.00 B/s 0.00 B/s ?unavailable? [rcu_gp]
4 be/0 root 0.00 B/s 0.00 B/s ?unavailable? [rcu_par_gp]As you see above, the four columns below are important for the overall IO stats of the system.
- Total DISK READ
- Total DISK WRITE
- Current DISK READ
- Current DISK WRITE
But how to watch the IO stats for a separated process? Using the ‘-p’ argument.
$ sudo iotop -p 330Here is the output:
Total DISK READ: 0.00 B/s | Total DISK WRITE: 19.95 K/s
Current DISK READ: 0.00 B/s | Current DISK WRITE: 31.93 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
330 be/3 root 0.00 B/s 19.95 K/s ?unavailable? [jbd2/sda1-8]You will see only the IO usage by process ID 330 here.
Using atop
Before using atop command you should install it first. In Ubuntu linux, just run:
$ sudo apt install atopatop can make a lot of statistics for different system resources like CPU, memory, disk, network etc. Here we just run the statistics for disk only.
$ sudo atop -dThe results are printed on an interactive screen as follows.
PID TID RDDSK WRDSK WCANCL DSK CMD 1/1
710066 - 0B 164.0K 0B 82% systemd-journa
330 - 0B 28.0K 0B 14% jbd2/sda1-8
710085 - 0B 4.0K 0B 2% sshd
710071 - 0B 4.0K 0B 2% rsyslogd
710149 - 0B 0B 0B 0% mysqld
712779 - 0B 0B 0B 0% atop
712793 - - - - 0% <sshd>
710074 - 0B 0B 0B 0% containerd
710065 - 0B 0B 0B 0% multipathd
710058 - 0B 0B 0B 0% irqbalance
712607 - 0B 0B 0B 0% kworker/u256:0
712792 - - - - 0% <sshd>
712791 - - - - 0% <sshd>
586597 - 0B 0B 0B 0% apache2For the output above, the key columns include:
- RDDSK: The size of data transferred during disk reads.
- WRDSK: The size of data transferred during disk writes.
- WCANCL: The size of data initially written, but later withdrawn
- DSK: The percentage of Disk occupied.
- CMD: The name of the process.
And, you can run atop with ‘-p’ argument to show accumulated process-info per program (i.e. same name).
$ sudo atop -d -pThe result shows:
NPROCS SYSCPU USRCPU VSIZE RSIZE PSIZE LOCKSZ SWAPSZ RDDSK WRDSK RNET SNET DSK CMD 1/1
1 0.00s 0.00s 78.7M 14.9M 0B 0.0K 0B 0B 148.0K 0 0 79% systemd-journa
1 0.00s 0.00s 0B 0B 0B 0.0K 0B 0B 36.0K 0 0 19% jbd2/sda1-8
1 0.00s 0.00s 217.2M 4.0M 0B 0.0K 0B 0B 4.0K 0 0 2% rsyslogd
1 0.01s 0.07s 0B 0B 0B 0.0K 0B 0B 0B 0 0 0% mailops.pl
1 0.03s 0.04s 1.7G 432.5M 0B 0.0K 0B 0B 0B 0 0 0% mysqld
1 0.03s 0.00s 13.0M 12.9M 0B 13.0M 0B 0B 0B 0 0 0% atop
1 0.00s 0.01s 1.2G 40.1M 0B 0.0K 0B 0B 0B 0 0 0% containerd
1 0.00s 0.01s 11.6M 5.5M 0B 0.0K 0B 0B 0B 0 0 0% sudo
1 0.00s 0.00s 1.4G 23.6M 0B 0.0K 8.6M 0B 0B 0 0 0% dockerdFrom /proc/pid/io file
The last way, you can get the IO stats by process from proc. The information is available under /proc/[PID]/io.
The information is stored like this:
$ sudo cat /proc/330/io
rchar: 0
wchar: 0
syscr: 0
syscw: 0
read_bytes: 0
write_bytes: 5651173376
cancelled_write_bytes: 0The definitions of the fields are:
rchar: number of bytes the process read, using any read-like system call (from files, pipes, tty...).wchar: number of bytes the process wrote using any write-like system call.syscr: number of read-like system call invocations that the process performed.syscw: number of write-like system call invocations that the process performed.read_bytes: number of bytes the process directly reads from disk.write_bytes: number of bytes the process originally dirtied in the page-cache (assuming they will go to disk later).cancelled_write_bytes: number of bytes the process "un-dirtied" - e.g. using an "ftruncate" call that truncates pages from the page-cache.
About The Author
Yonghua Peng
Yonghua Peng is a tech lead focusing on cloud computing, big data and DevOps. He have 15 years of experience in Linux development and operations. He is also the author of three technical books.
Comments