3

How can i setup monitoring to my system services? Either using automated script which scan every moment, if httpd, mysqld, and my custom daemon is running or not, if not running it will automatically on the fly restart it?

Any idea?

For example:

*Day 1:* System is running in Rail way where no support can be 24/7 available, Day 1 was fine. 
*Day 2:* System in the middle of the Rail way crashed cause httpd and mysqld for some reason not running the service

How can it be automated so that the service httpd remain running and service mysqld remain running?

5 Answers5

4

There are various tools to do this (of which, other than daemontools and perp, I don't have much experience with):

The one we have come to like at my workplace is perp, which was the best featured for our infrastructure. Some of these tools only do what you want as a subset of their total functionality, so they may not be suitable for your use case.

Chris Down
  • 125,559
  • 25
  • 270
  • 266
  • "spurred" or "spawned", not "spurred", surely? And having more tools in a toolkit doesn't necessarily make the toolkit unsuitable. It's what the individual tools do. – JdeBP Jan 04 '15 at 17:41
  • @JdeBP It's a typo for "spawned", thanks for letting me know to correct it! :-) – Chris Down Jan 04 '15 at 19:04
2

As mentioned in other answers, Dan Bernstein's daemontools began a whole family of toolsets that share the same raw mechanisms:

Under pretty much any of them, one writes a run program that runs/is the dæmon, and a service manager or supervisor process simply monitors it as a forked child process using the normal Unix and Linux mechanism. For Apache and MySQL, you are treading where a lot of people have trodden before, and there are a lot of examples of how to run these servers under daemontools-family service management. Here are only some:

Chris Down suggests that the larger toolsets may be unsuitable. This isn't really the case. Whilst all of these toolsets are coherent and self-consistent, none of them demand that one use any of the tools other than the ones needed in any particular situation. One can also mix and match. One can use Laurent Bercot's execlineb and all of its utilities under perp, or my nosh script interpreter and all of its utilities under runit; just as one can equally use Gerrit Pape's chpst under my service-manager.

Equally, you could run Apache and MySQL from systemd (if you are Linux-only) or launchd (if you are using MacOS 10). A launchd configuration file is rather complex and cumbersome, by comparison to the other systems mentioned. systemd unit files, however, are on the same order of simplicity as run scripts:

There are a fair few home-brewed mysqld.service and httpd.service service units to be found on the World Wide Web, in various people's collections of home-brewed handy systemd service units.

All of these provide the basic substrate of starting a dæmon at bootstrap, stopping and starting it under administrator/automated control whilst the system is running, and automatically restarting it in various failure cases. Xion345 makes the mistake of conflating this with monit. As you can see in xyr answer, the substrate there for monitoring and controlling is dæmon System 5 rc. It could equally well have been systemd or nosh. (In fact, if Xion345's example had used the service command instead of trying to run init.d scripts directly, which isn't a good idea for this and several other reasons, it would have worked pretty much as-is with systemd, upstart, or nosh.)

Where monit belongs is in the layer above. monit uses the start/stop/supervision substrate, and monitors the actual service provided, which one does in addition to supervising the dæmon process with a dæmon supervisor. In this layer one finds related tools such as nagios. (One can fairly easily plumb nagios into a daemontools-family-monitored dæmon, and have it check process state and uptimes via the daemontools API. My nosh toolset even comes with one basic nagios plug-in for doing this.)

JdeBP
  • 68,745
1

In addition to what Chris Down wrote, I would also recommend monit. It can notably check if a port if open (eg 80) and restart the appropriate service (eg httpd) if this port is closed. See this example for sshd :

check process sshd with pidfile /var/run/sshd.pid
start program  "/etc/init.d/sshd start"
stop program  "/etc/init.d/sshd stop"
if failed port 22 protocol ssh then restart
if 5 restarts within 5 cycles then timeout

Monit uses a different approach than perp or daemontools : it does not only ensure a process is running but rather checks if a port is port open or if a file exists (could be a UNIX socket). It could be easier to set up and a little bit less intrusive (you don't have to make sure that monit properly interacts with your init system) than daemontools or perp. It can also be configured to send emails if it constantly fails to restart a service.

Xion345
  • 749
  • Actually, you do have to make sure that it properly interacts with your init system. Those start program and stop program lines in your example are where it interacts with your init system, and, ironically, they will be problematic with init systems other than System 5 rc. (There's no guarantee that a native systemd SSH service will be startable or stoppable in this way, for just one example.) – JdeBP Jan 04 '15 at 17:39
1

Puppet allows you to define which services should be running on your system.

Puppet is IT automation software that helps system administrators manage infrastructure throughout its lifecycle, from provisioning and configuration to patch management and compliance. Using Puppet, you can easily automate repetitive tasks, quickly deploy critical applications, and proactively manage change, scaling from 10s of servers to 1000s, on-premise or in the cloud.

For example :

service { 'apache2':
    ensure => running,
    enable => true,
    require => Package['apache2'],
    subscribe => File['/etc/apache2/httpd.conf'],
}

This configuration (called a manifest in the context of Puppet) will make sure the apache2 service is running, that it starts up at boot-time, that it doesn't try to manage the service unless the apache2 package is installed, and that it restarts if /etc/apache2/httpd.conf is changed.

With Puppet you can not only manage the service processes, but also their dependencies and configuration files.

jcharaoui
  • 346
  • 1
  • 10
0

There's also God.

God is an easy to configure, easy to extend monitoring framework written in Ruby.

Keeping your server processes and tasks running should be a simple part of your deployment process. God aims to be the simplest, most powerful monitoring application available.

Write a simple server, simple.rb:

loop do
  puts 'Hello'
  sleep 1
end

Now create a script, simple.god, that watches the daemon:

God.watch do |w|
  w.name = "simple"
  w.start = "ruby /full/path/to/simple.rb"
  w.keepalive
end

Now to start the monitoring script:

god -c path/to/simple.god -D

God can watch more than just ruby apps, it could watch your httpd or mysqld and called the corresponding /etc/init./d/... script as needed.

slm
  • 369,824