Having a system for collecting performance statistics can be extremely useful. In the past, I've used Munin for this, and it has been invaluable in analyzing bottlenecks and various other issues. I was recently made aware of collectd, which seems very similar to Munin.
What monitoring applications are available and should be considered (other than Munin and Collectd), and how do you choose which one to use?
Munin is a data collector and visualizer (grapher) tool. It is easy to set up. Easy to use, but it uses too much resources and does not scale well. The default collection interval is 5min and it is not easy to change that, because it will overload your machine and some plugin has problems if you do that anyway. The plugins are executed (forked) every time when data collection occurs which is expensive. It has networked setup. You have to set up a local server and node even if you use one machine.
Collectd is a data collector tool only. You can choose 3rd party soultions to graph the collected data, but it does not work out of the box. It has many plugins, mostly written as a C module which gets started once when you start the daemon. You can change the collection interval and you can get fine grained statistics. It can collect data locally or via the network.
My favorite performance monitoring analysis tool is SGI's open source Performance CoPilot (PCP). For a single system, it might be overkill - for a set of enterprise systems, it's fantastic. PCP offers historical data, a networked configuration, and an alarm system which is incomparable in open source (or just about anywhere else).
My opinion is that zabbix rules. It's powerful and user friendly. You don't need any plugin or third party in order to make it work.
You can find lots of other suggestions in this post at server fault.
