Every so often, my Linode goes into a state of apparent frantic I\/O. Page loads slow down a bit, and I get regular email alerts indicating a potential problem:<\/p>\n
Subject: Linode Alert - disk io rate\n\nYour Linode, linode90147, has exceeded the notification threshold (800) for disk io rate by averaging 2146.05 for the last 2 hours. The dashboard for this Linode is located at: ...<\/pre>\nThis is the first time this happened since I switched entirely to nginx. My first test was to install iostat\/sar, to see what is going on.<\/p>\n
\napt-get install sysstat\n<\/pre>\nThe initial output of iostat looks like this:<\/p>\n
\navg-cpu: %user %nice %system %iowait %steal %idle\n 0.75 0.28 0.19 1.21 0.01 97.56\n\nDevice: tps Blk_read\/s Blk_wrtn\/s Blk_read Blk_wrtn\nxvda 12.53 168.70 73.94 250258986 109688664\nxvdb 23.49 127.81 86.97 189603528 129015512\n<\/pre>\nThis shows point in time output for the read\/write rate, which doesn’t look nearly as high as Linode is reporting. You can do a continuous reporting by doing the following:<\/p>\n
\niostat -d 2\n<\/pre>\nThis showed the read\/write rates running anywhere from 0 to 5500 blocks\/second, about 2.8 MB\/s (512 bytes\/block). Some points to note: xvda is Xen Virtual Disk. Watching the usage for a while, both disks about simultaneously, but most of the writes are xvdb, which may indicate loading a lot of data from disk into memory (swap) space.<\/p>\n
To find out which process(es) are doing the disk use, I ran the following:<\/p>\n
\npidstat -d 2 300\n<\/pre>\nThis takes 300 I\/O samples at two second intervals (i.e. for ten minutes). It prints out each sample and an average summary. Running this, I got the following output:<\/p>\n
\nAverage: PID kB_rd\/s kB_wr\/s kB_ccwr\/s Command\nAverage: 996 0.01 2.31 0.00 kjournald\nAverage: 1930 2.43 0.07 0.00 rsyslogd\nAverage: 1958 0.10 0.00 0.00 atd\nAverage: 1959 109.13 14.20 0.00 cron\nAverage: 1971 0.26 0.00 0.00 memcached\nAverage: 1978 47.54 0.48 0.00 mysqld\nAverage: 2045 26.32 0.11 0.03 munin-node\nAverage: 2131 10.21 0.01 0.00 sendmail-mta\nAverage: 2234 2.62 0.01 0.00 ntpd\nAverage: 2397 10.71 0.00 0.00 fail2ban-server\nAverage: 13689 0.29 0.00 0.00 pidstat\nAverage: 14427 0.23 0.00 0.00 cron\nAverage: 14428 0.18 0.00 0.00 sh\nAverage: 14431 0.01 0.00 0.00 munin-cron\nAverage: 14432 9.95 0.01 0.00 munin-update\nAverage: 14433 6.95 5.61 0.00 munin-update\nAverage: 14434 10.14 0.04 0.01 munin-node\nAverage: 14813 0.04 0.00 0.00 vmstat\nAverage: 14814 0.15 0.00 0.00 vmstat\nAverage: 15685 12.96 0.00 0.00 php-cgi\nAverage: 15686 13.79 0.00 0.00 php-cgi\nAverage: 15687 20.47 0.59 0.15 php-cgi\nAverage: 15688 15.01 0.00 0.00 php-cgi\nAverage: 15689 33.72 0.00 0.00 php-cgi\nAverage: 15690 15.53 0.00 0.00 php-cgi\nAverage: 15691 9.60 0.00 0.00 php-cgi\nAverage: 15692 19.13 0.00 0.00 php-cgi\nAverage: 15693 12.64 0.00 0.00 php-cgi\nAverage: 15694 15.76 0.00 0.00 php-cgi\nAverage: 15695 16.30 0.01 0.00 php-cgi\nAverage: 15696 18.60 0.00 0.00 php-cgi\nAverage: 15697 12.65 0.00 0.00 php-cgi\nAverage: 16334 30.81 0.72 0.15 php-cgi\nAverage: 16338 14.99 0.00 0.00 php-cgi\nAverage: 21209 1.21 0.00 0.00 sshd\nAverage: 31579 1.03 0.86 0.79 nginx\nAverage: 31580 0.67 0.03 0.00 nginx\nAverage: 31581 1.11 0.07 0.00 nginx\nAverage: 31582 1.82 0.17 0.00 nginx\n<\/pre>\nThere are ares to research as an avenue for research- php, mysql, and cron. I know I added some jobs, so I tested that first. To see all available cron jobs:<\/p>\n
\n for user in $(cut -f1 -d: \/etc\/passwd); do crontab -u $user -l; done\n<\/pre>\nFrom this output, I removed two obsolete hourly tasks I had created. For good measure, I also decreased the frequency of man-db lookups from daily to monthly, removed apache2 cleanup (no longer used) and popularity-contest. Everything remaining appears to be important to system maintenance. The following is a second performance log, after this runs. Very little has changed.<\/p>\n
\nAverage: 1 1.22 0.02 0.01 init\nAverage: 996 0.03 3.51 0.00 kjournald\nAverage: 1930 2.74 0.26 0.00 rsyslogd\nAverage: 1959 131.20 14.24 0.00 cron\nAverage: 1971 0.01 0.00 0.00 memcached\nAverage: 1978 78.06 0.63 0.11 mysqld\nAverage: 2045 27.10 0.09 0.03 munin-node\nAverage: 2131 23.92 0.03 0.00 sendmail-mta\nAverage: 2234 2.77 0.00 0.00 ntpd\nAverage: 2397 13.69 0.00 0.00 fail2ban-server\nAverage: 15685 24.56 0.01 0.00 php-cgi\nAverage: 15686 17.49 0.00 0.00 php-cgi\nAverage: 15687 40.37 0.00 0.00 php-cgi\nAverage: 15688 21.09 0.01 0.00 php-cgi\nAverage: 15689 29.78 0.00 0.00 php-cgi\nAverage: 15690 10.94 0.00 0.00 php-cgi\nAverage: 15691 23.94 0.00 0.00 php-cgi\nAverage: 15692 16.95 0.01 0.00 php-cgi\nAverage: 15693 8.02 0.01 0.00 php-cgi\nAverage: 15694 7.47 0.01 0.00 php-cgi\nAverage: 15695 23.33 0.00 0.00 php-cgi\nAverage: 15696 8.47 0.01 0.00 php-cgi\nAverage: 15697 13.05 0.01 0.00 php-cgi\nAverage: 16334 13.13 0.01 0.00 php-cgi\nAverage: 16338 11.27 0.00 0.00 php-cgi\nAverage: 21209 0.60 0.00 0.00 sshd\nAverage: 22998 0.41 0.00 0.00 pidstat\nAverage: 31579 0.51 0.03 0.00 nginx\nAverage: 31580 1.37 3.08 1.43 nginx\nAverage: 31581 1.32 1.55 1.44 nginx\nAverage: 31582 2.55 1.77 0.00 nginx\n<\/pre>\nThe php work is a little lower, but likely not enough to be significant. Next up: PHP. I had APC working when I was running Apache, but perhaps it’s not working now, with Nginx as the primary server. <\/p>\n
I rebuilt APC from scratch, in case there was a newer version. The lynchpin of this was discovering multiple php.ini files on the VPS. The instructions for building APC are as follows:<\/p>\n
\nwget http:\/\/pecl.php.net\/package\/APC\ntar -xzf APC-3.1.9.tgz\ncd APC-3.1.9\nphpize\n.\/configure --enable-apc --enable-apc-mmap --with-apxs --with-php-config=\/etc\/php5\/cgi\/php.ini\nmake\nmake test\nmake install\n\nvi \/etc\/php5\/cgi\/php.ini\n<\/pre>\nAdd this line at the end:<\/p>\n
\nextension=apc.so\n<\/pre>\nThen restart phpd\/php-cgi. E.g. if you installed nginx\/fast_cgi as an init.d service, do something like this:<\/p>\n
\n \/etc\/init.d\/php-fastcgi restart\n<\/pre>\n