8

I have an app running on rackspace cloud (ubuntu 10.04). For some reason, over the last 3 nights, the memory has been spiking and then leveling off for about 7 hours before eventually falling off. It seems to start at midnight and last through 7 am (ignore the times on the chart -- they're off by 4 hours).

alt text

Looking through Munin chart, there doesn't seem to be any increased activity in mysql, and the ethernet traffic doesn't seem to be spiking at all during that time (which would seem to negate something like a bot or crawler).

Which tools are available for linux to troubleshoot something like this?

tshepang
  • 65,642
jerhinesmith
  • 181

3 Answers3

3

I'm not entirely sure it's going to be a perfect fit, but the process accounting package (mostly named psacct or acct) is able to keep record of who (which account) runs what (what program). This might help you figure out what is running at the time you specified.

wzzrd
  • 3,720
  • This might actually be a good idea. psacct usually is used to record, report or even limit via quota of CPU usage or clock ticks. It was the old way of making sure the customer/user of the UNIX server was paying the usage he/she/it did. One note on this though, psacct turned on to at least log everything is a massive disk-hogger in disk space usage. – reiche Aug 14 '10 at 11:16
1

In general I don't know what tool could help you, except logging ps regularly and analysing later the processes activities.

However, I can make a guess about the spikes around 6:00am on your graph, it could be the Ubuntu default daily cron jobs. On my machine /etc/crontab, daily.d is configured to run at 6:25am. But you said we should ignore the time, so the obvious question is, do you have cron jobs?

elmarco
  • 1,434
  • 1
  • 14
  • 13
  • That was my first thought as well. However, my daily cron jobs run at 1:00 pm local time. There are a few jobs being run by the app, but they either start at 6:00 pm or 2:00 am. – jerhinesmith Aug 12 '10 at 17:10
  • 2
    If you're going down the regular ps route, the "HowTo: Profile Memory in a Linux System" HOWTO goes into a bit more detail: http://mail.nl.linux.org/linux-mm/2003-03/msg00077.html – Geoff Childs Aug 12 '10 at 18:23
1

My default answer to such questions would be sar (System Activity Reporter) from the sysstat package.

But as far as I know, sar doesn't collect an equivalent to the output of ps. So perhaps the combination of sar and elmarcos answer (regularly capture the output of ps) would help.

EDIT:

Steve D has mentioned pidstat in this question. This seems more suited for your needs.

Community
  • 1
Christian
  • 1,591