197

It appears systemd is the hot new init system on the block, same as Upstart was a few years ago. What are the pros/cons for each? Also, how does each compare to other init systems?

tshepang
  • 65,642
  • 4
    @keith iirc openrc simply uses SysV, the advantage is a well designed collection of startup scripts that use common components and are portable (meaning work on any shell) It's a good cleanup, but not really a new initd – xenoterracide Jan 21 '11 at 19:26
  • @xeno It does, but you can't really tell. there are no rcX.d or [KS] symlinks at all. Actually sysv init itself is pretty flexible, and the runlevels are not really used in the usual way. – Keith Jan 22 '11 at 20:15
  • Although the author of this blog is against systemd, I suggest giving this a read. It goes over the pros and cons of systemd and BSD init. https://www.textplain.net/blog/2015/problems-with-systemd-and-why-i-like-bsd-init/ – Peschke Jun 03 '16 at 15:48
  • 1
    Please go through 2016 update http://unix.stackexchange.com/a/287282/49091 too. – igauravsehrawat Feb 09 '17 at 12:15
  • Any purported advantages of systemd will not in a 100 years offset the cost already incurred to the world to implement this. Every minute or hour or day spent by a Unix admin to deal with this rubbish must already add up to billions and for what real benefits other than a couple of bells and whistles ? – Waslap Dec 14 '18 at 09:17

6 Answers6

106

2016 Update

Most answers here are five years old so it's time for some updates.

Ubuntu used to use upstart by default but they abandoned it last year in favor of systemd - see:

Because of that there is a nice article Systemd for Upstart Users on Ubuntu wiki - very detailed comparison between upstart and systemd and a transition guide from upstart to systemd.

(Note that according to the Ubuntu wiki you can still run upstart on current versions of Ubuntu by default by installing the upstart-sysv and running sudo update-initramfs -u but considering the scope of the systemd project I don't know how it works in practice, or whether or not systemd is possible to uninstall.)

Most of the info in the Commands and Scripts sections below is adapted from some of the examples used in that article (that is conveniently licensed just like Stack Exchange user contributions under the Creative Commons Attribution-ShareAlike 3.0 License).

Here is a quick comparison of common commands and simple scripts, see sections below for detailed explanation. This answer is comparing the old behavior of Upstart-based systems with the new behavior of systemd-based systems, as asked in the question, but note that the commands tagged as "Upstart" are not necessarily Upstart-specific - they are often commands that are common to every non-systemd Linux and Unix system.

Commands

Running su:

  • upstart: su
  • systemd: machinectl shell

(see "su command replacement" section below)

Running screen:

  • upstart: screen
  • systemd: systemd-run --user --scope screen

(see "Unexpected killing of background processes" section below)

Running tmux:

  • upstart: tmux
  • systemd: systemd-run --user --scope tmux

(see "Unexpected killing of background processes" section below)

Starting job foo:

  • upstart: start foo
  • systemd: systemctl start foo

Stopping job foo:

  • upstart: stop foo
  • systemd: systemctl stop foo

Restarting job foo:

  • upstart: restart foo
  • systemd: systemctl restart foo

Listing jobs:

  • upstart: initctl list
  • systemd: systemctl status

Checking configuration of job foo:

  • upstart: init-checkconf /etc/init/foo.conf
  • systemd: systemd-analyze verify /lib/systemd/system/foo.service

Listing job's environement variables:

  • upstart: initctl list-env
  • systemd: systemctl show-environment

Setting job's environment variable:

  • upstart: initctl set-env foo=bar
  • systemd: systemctl set-environment foo=bar

Removing job's environment variable:

  • upstart: initctl unset-env foo
  • systemd: systemctl unset-environment foo

Logs

In upstart, the logs are normal text files in the /var/log/upstart directory, so you can process them as usual:

cat /var/log/upstart/foo.log
tail -f /var/log/upstart/foo.log

In systemd logs are stored in an internal binary format (not as text files) so you need to use journalctl command to access them:

sudo journalctl -u foo
sudo journalctl -u foo -f

Scripts

Example upstart script written in /etc/init/foo.conf:

description "Job that runs the foo daemon"
start on runlevel [2345]
stop on runlevel [016]
env statedir=/var/cache/foo
pre-start exec mkdir -p $statedir
exec /usr/bin/foo-daemon --arg1 "hello world" --statedir $statedir

Example systemd script written in /lib/systemd/system/foo.service:

[Unit]
Description=Job that runs the foo daemon
Documentation=man:foo(1)
[Service]
Type=forking
Environment=statedir=/var/cache/foo
ExecStartPre=/usr/bin/mkdir -p ${statedir}
ExecStart=/usr/bin/foo-daemon --arg1 "hello world" --statedir ${statedir}
[Install]
WantedBy=multi-user.target

su command replacement

A su command replacement was merged into systemd in pull request #1022:

because, according to Lennart Poettering, "su is really a broken concept".

He explains that "you can use su and sudo as before, but don't expect that it will work in full".

The official way to achieve a su-like behavior is now:

machinectl shell

It has been further explained by Lennart Poettering in the discussion to issue #825:

"Well, there have been long discussions about this, but the problem is that what su is supposed to do is very unclear. [...] Long story short: su is really a broken concept. It will given you kind of a shell, and it’s fine to use it for that, but it’s not a full login, and shouldn’t be mistaken for one." - Lennart Poettering

See also:

Unexpected killing of background processes

Commands like:

no longer work as expected. For example, nohup is a POSIX command to make sure that the process keeps running after you log out from your session. It no longer works on systemd. Also programs like screen and tmux need to be invoked in a special way or otherwise the processes that you run with them will get killed (while not getting those processes killed is usually the main reason of running screen or tmux in the first place).

This is not a mistake, it is a deliberate decision, so it is not likely to get fixed in the future. This is what Lennart Poettering has said about this issue:

In my view it was actually quite strange of UNIX that it by default let arbitrary user code stay around unrestricted after logout. It has been discussed for ages now among many OS people, that this should possible but certainly not be the default, but nobody dared so far to flip the switch to turn it from a default to an option. Not cleaning up user sessions after logout is not only ugly and somewhat hackish but also a security problem. systemd 230 now finally flipped the switch and finally by default cleans everything up correctly when the user logs out.

For more info see:

High-level startup concept

In a way systemd works backwards - in upstart jobs start as soon as they can and in systemd jobs start when they have to. At the end of the day the same jobs can be started by both systems and in pretty much the same order, but you think about it looking from an opposite direction so to speak.

Here is how Systemd for Upstart Users explains it:

Upstart's model for starting processes (jobs) is "greedy event-based", i. e. all available jobs whose startup events happen are started as early as possible. During boot, upstart synthesizes some initial events like startup or rcS as the "tree root", the early services start on those, and later services start when the former are running. A new job merely needs to install its configuration file into /etc/init/ to become active.

systemd's model for starting processes (units) is "lazy dependency-based", i. e. a unit will only start if and when some other starting unit depends on it. During boot, systemd starts a "root unit" (default.target, can be overridden in grub), which then transitively expands and starts its dependencies. A new unit needs to add itself as a dependency of a unit of the boot sequence (commonly multi-user.target) in order to become active.

Usage in distributions

Now some recent data according to Wikipedia:

Distributions using upstart by default:

Distributions using systemd by default:

(See Wikipedia for up to date info)

Distributions using neither Upstart nor systemd:

Controversy

In the past A fork of Debian has been proposed to avoid systemd. The Devuan GNU+Linux was created - a fork of Debian without systemd (thanks to fpmurphy1 for pointing it out in the comments).

For more info about this controversy, see:

As many of you might know already, the Init GR Debian vote promoted by Ian Jackson wasn't useful to protect Debian's legacy and its users from the systemd avalanche.

This situation prospects a lock in systemd dependencies which is de-facto threatening freedom of development and has serious consequences for Debian, its upstream and its downstream.

The CTTE managed to swap a dependency and gain us time over a subtle install of systemd over sysvinit, but even this process was exhausting and full of drama. Ultimately, a week ago, Ian Jackson resigned. [...]

I am resigning from the Technical Committee with immediate effect.

While it is important that the views of the 30-40% of the project who agree with me should continue to be represented on the TC, I myself am clearly too controversial a figure at this point to do so. I should step aside to try to reduce the extent to which conversations about the project's governance are personalised. [...]

Devuan was born out of a controversy over the decision to use as the default init system for Debian. The official Debian position on systemd is full of claims that others have debunked. Interested readers can continue discussing this hot topic in The systemd controversy. However we encourage you to keep your head cool and your voice civil. At Devuan we’re more interested in programming them wrong than looking back. [...]

Some websites and articles dedicated to the systemd controversy has been created:

There is a lot of interesting discussion on Hacker News:

Similar tendencies in other distros can be observed as well:

Philosophy

upstart follows the Unix philosophy of DOTADIW - "Do One Thing and Do It Well." It is a replacement for the traditional init daemon. It doesn't do anything other than starting and stopping services. Other tasks are delegated to other specialized subsystems.

systemd does much more than that. In addition to starting and stopping services it also manages passwords, logins, terminals, power management, factory resets, log processing, file system mount points, networking and much more - see the NEWS file for some of the features.

Plans of expansion

According to A Perspective for systemd What Has Been Achieved, and What Lies Ahead presentation by Lennart Poettering in 2014 at GNOME.asia, here are the main objectives of systemd, areas that were already covered and those that were still in progress:

systemd objectives:

Our objectives

  • Turning Linux from a bag of bits into a competitive General Purpose Operating System.

  • Building the Internet’s Next Generation OS Unifying pointless differences between distributions

  • Bringing innovation back to the core OS

  • Desktop, Server, Container, Embedded, Mobile, Cloud, Cluster, . . . These areas are closer together than you might think

  • Reducing administrator complexity, reliability without supervision

  • Everything introspectable

  • Auto discovery, plug and play is key

  • We fix things where they are broken, never tape over them

Areas already covered:

What we already cover:

init system, journal logging, login management, device management, temporary and volatile file management, binary format registration, backlight save/restore, rfkill save/restore, bootchart, readahead, encrypted storage setup, EFI/GPT partition discovery, virtual machine/container registration, minimal container management, hostname management, locale management, time management, random seed management, sysctl variable management, console managment, . . .

Work in progress:

What we are working on:

  • network management
  • systemd-networkd
  • Local DNS cache, mDNS responder, LLMNR responder, DNSSEC verification
  • IPC in the kernel
  • kdbus, sd-bus
  • Time synchronisation with NTP
  • systemd-timesyncd
  • More integration with containers
  • Sandboxing of Services
  • Sandboxing of Apps
  • OS Image format
  • Container image format
  • App image format
  • GPT with auto-discovery
  • Stateless systems, instantiatable systems, factory reset
  • /usr is the OS
  • /etc is (optional) configuration
  • /var is (optional) state
  • Atomic node initialisation and updates
  • Integration with the cloud
  • Service management across nodes
  • Verifiable OS images
  • All the way to the firmware
  • Boot Loading

Scope of this answer

As fpmurphy1 noted in the comments, "It should be pointed out that systemd has expanded its scope of work over the years far beyond simply that of system startup."

I tried to include most of the relevant info here. Here I am comparing the common features of Upstart and systemd when used as init systems as asked in the question and I only mention features of systemd that go beyond the scope of an init system because those cannot be compared to Startup, but their presence is important to understand the difference between those two projects. The relevant documentation should be checked for more info.

More info

More info can be found at:

Extras

The LinOxide Team has created a Systemd vs SysV Init Linux Cheatsheet.

mayhair
  • 5
rsp
  • 3,416
  • 5
    ... and such a fork of Debian has occurred. Devuan GNU+Linux is a fork of Debian without systemd. – fpmurphy Jun 02 '16 at 22:30
  • 3
    It should be pointed out that systemd has expanded its scope of work over the years far beyond simply that of system startup. – fpmurphy Jun 02 '16 at 22:38
  • @fpmurphy1 Thanks for your comments - I updated my answer and added the info that you provided. – rsp Jun 02 '16 at 23:21
  • 2
    Outstanding post and extremely helpful to Cent guy. Thank you for taking the time sir!!! – origin1tech Dec 14 '16 at 06:58
  • 2
    I definitely don't appreciate the less intuitive and more verbose systemctl commands. There was an elegance to service myservice stop/start/restart/status. – Ron Smith Feb 02 '17 at 21:02
  • 4
    @ronsmith service <foo> start/stop/restart/status still work just fine. Like most unix software, systemd provides command compatibility to well known defaults. – Shadur-don't-feed-the-AI Mar 01 '17 at 09:08
  • 2
    Very good answer. One point though: the BSD operating systems are not Linux distributions: they are different Unix operating systems with their own kernel. – Giorgio Aug 27 '17 at 06:37
  • 1
    Also, to the list of non-systemd distributions, you can add arch-openrc and manjaro-openrc, which just merged into artix. parabola Linux can also be run with openrc. – Giorgio Aug 27 '17 at 06:39
  • I'm dizzy from so much scrolling, but thank you. – stevec Nov 21 '22 at 08:17
70

Both upstart and systemd are attempts to solve some of the problems with the limitations of the traditional SysV init system. For example, some services need to start after other services (for example, you can't mount NFS filesystems until the network is running), but the only way in SysV to handle that is to set the links in the rc#.d directory such that one is before the other. Add to that, you might need to re-number everything later when dependencies are added or changed. Upstart and Systemd have more intelligent settings for defining requirements. Also, there's the issue with the fact that everything is a shell script of some sort, and not everyone writes the best init scripts. That also impacts the speed of the startup.

Some of the advantages of systemd I can see:

  • Every process started gets its own cgroup or a particular cgroup.
  • Pre-creation of sockets and file handles for services, similar to how xinetd does for it's services, allowing dependent services to start faster. For example, systemd will hold open the filehandle for /dev/log for syslog, and subsequent services that send to /dev/log will have their messages buffered until syslogd is ready to take over.
  • Fewer processes run to actually start a service. This means you aren't writing a shell script to start up your service. This can be a speed improvement, and (IMO) something easier to set up in the first place.

One disadvantage I know of is that to take advantage of systemd's socket/FH preallocation, many daemons will have to be patched to have the FH passed to them by systemd.

tshepang
  • 65,642
jsbillings
  • 24,406
  • 1
    Can you explain the last sentence a bit. – tshepang Jan 20 '11 at 17:44
  • 13
    PulseAudio is a much-maligned sound system (http://www.pulseaudio.org/), originally written by Lennart Poettering, the author of systemd. I was mostly making a joke here, because I know several people who like to complain about pulseaudio and I'm sure they'd complain about systemd too. Honestly, I haven't had a problem with either systemd or pulseaudio. – jsbillings Jan 20 '11 at 17:48
  • @jsbillings the system daemon patch isn't that much of a disadvantage, since daemons that don't do it will still work – xenoterracide Jan 21 '11 at 01:50
  • 1
    That said, the potential advantages are very exciting. If the project keeps the level of activity it's had for the last few months, the problems will get ironed out and the whole project is going to be a big step forward from traditional init. It's just still rough around the edges. – mattdm Feb 24 '11 at 15:26
  • 4
    Makes one almost pine for the abundant fiords of Plan9... everything's a file. – dhchdhd May 11 '11 at 09:29
  • The author is worth noting, and definitely effected my opinion. – J. M. Becker Feb 04 '12 at 03:35
  • 1
    The systemd fallacy -- "systemd is wrong on so many levels I hardly know where to start. Perhaps its single most important design fault is that it was conceived with a blatant disregard for servers." – MountainX May 27 '12 at 20:41
  • 4
    To be honest, pulseaudio was a solution to a non-existant problem. There's nothing PA can do that ALSA can't, and I've heard PLENTY of people having issues with PA, over and over again. – WhyNotHugo Jul 23 '12 at 04:37
  • 3
    Two systemd disadvantages you missed: (1) all startup scripts need to be rewritten. (2) there's way less compatibility with non-linux OSs (like BSDs, for example). – WhyNotHugo Jul 23 '12 at 04:38
  • 8
    Just great. Please take a look at http://0pointer.de/blog/projects/the-biggest-myths. I witnessed systemd's growth, and can attest that much of the criticisms given here are completely groundless. At the link you will find a blow by blow, reasoned rebuttal. – vonbrand Jan 27 '13 at 22:45
  • 3
    ... by Lennart Poettering. This is like asking Rupert Murdoch what's wrong with Fox News. – L0j1k Oct 20 '14 at 03:29
29

Saw systemd mentioned on Arch General ML today. So read up on it. The H Online as ever is a great source for Linux Technology and is where I found my place to start researching Systemd as SysV Init and Upstart alternative. However the H Online article (in this case) isn't a very useful read, the real use behind it is it gives links to the useful reads.

The real answer is in the announcement of systemd. Which gives some crucial points of what's wrong with SysV initd, and what new systems need to do

  • To start less.
  • And to start more in parallel.

Its major plan to do this seems to be to start services only as they're needed, and to start a socket for that service, so that the service that needs it can connect to the created socket long before the daemon is fully online. Apparently a socket will retain a small amount of buffered data meaning that no data will be lost during the lag, it will be handled as soon as the daemon is online.

Another part of the plan seems to be to not serialize filesystems, but instead mount those on demand as well, that way you're not waiting on your /home/, etc (not to be confused with /etc) to mount, and/or fsck when you could be starting daemons as / and /var/ etc, are already mounted. It said it was going to use autofs to this end.

It also has the goal of creating .desktop style init descriptors as a replacement for scripts. This will prevent tons of slow sh processes and even more forks of processes from things like sed and grep that are often used in shell scripts.

They also plan not to start some services until they are asked for, and perhaps even shut them off if they are no longer needed, bluetooth module, and daemon are only needed when you're using a bluetooth device for example. Another example given is the ssh daemon. This is the kind of thing that inetd is capable of. Personally I'm not sure I like this, as it might mean latency when I do need them, and in the case of ssh I think it means a possible security vulnerability, if my inetd were compromised the whole system would be. However, I've been informed that using this to breach this system is infeasible and that if I want to I can disable this feature per service and in other ways.

Another feature is apparently going to be the capability to start based on time events, either at a regularly scheduled interval or at a certain time. This is similar to what crond and atd do now. Though I was told it will not support user "cron". Personally this sounds like the most pointless thing. I think this was written/thought up by people who don't work in multiuser environments, there isn't much purpose to user cron if you're the only user on the system, other than not running as root. I work on multiuser systems daily, and the rule is always run user scripts as the user. But maybe I don't have the foresight they do, and it will in no way make it so that I can't run crond or atd, so it doesn't hurt anyone but the developers I suppose.

The big disadvantage of systemd is that some daemons will have to be modified in order to take full advantage of it. They'll work now, but they'd work better if they were written specifically for its socket model.

It seems for the most part the systemd's peoples problem with upstart is the event system, and that they believe it to not make sense or be unnecessary. Perhaps their words put it best.

Or to put it simpler: the fact that the user just started D-Bus is in no way an indication that NetworkManager should be started too (but this is what Upstart would do). It's right the other way round: when the user asks for NetworkManager, that is definitely an indication that D-Bus should be started too (which is certainly what most users would expect, right?).
A good init system should start only what is needed, and that on-demand. Either lazily or parallelized and in advance. However it should not start more than necessary, particularly not everything installed that could use that service.

As I've already said this is discussed much more comprehensively in the announcement of systemd.

cjm
  • 27,160
xenoterracide
  • 59,188
  • 74
  • 187
  • 252
  • sorry but the announcement is like a book. I have to read and grok, before I can really include more here. – xenoterracide Jan 20 '11 at 15:56
  • 2
    How is this better than @John's answer? Is it a placeholder? A promo of H Online? – tshepang Jan 20 '11 at 17:46
  • 1
    @tshepang well it actually links to the announcement of system d and the h online stuff has links to that and other interesting links. it's a long tedious read. I might add more once I've groked it... this is not a simple subject. when I wrote this I figured you might want to start reading sooner than later. but feel free to mod me down. certainly @jsbillings answer is decent, and better than mine so far. but not as good as reading the announcement itself – xenoterracide Jan 20 '11 at 18:03
  • @tshepang I will probably add more tomorrow, after bed. h online stuff was just me being a good journalist and citing my sources. – xenoterracide Jan 20 '11 at 18:05
  • @tshepang. I've updated my answer. pretty sure I'm done unless the helpful people on irc://frenode.net/systemd decide they want to offer up a correction of some kind. – xenoterracide Jan 21 '11 at 19:20
  • Thanks for summarizing the info. Do you mind asking Lennart the rationale of what led you to think Personally this sounds like the most pointless thing...? It appears he answers all questions on that blog post (which is still getting comments many months later!). – tshepang Jan 21 '11 at 19:35
  • @tshepang I tried to get the rational for the time based event system out of them last night, but it wasn't anything that made any sense to me. They talked about using it for things like updatedb and logrotate and other things in /etc/cron.d/daily, which sounds good for a desktop system, because I bet most desktop systems use only that. Though I suggested that all those might be better with some kind of fsnotify event handler anyways, than even running on time based. – xenoterracide Jan 21 '11 at 19:38
  • Were you talking with Lennart? Maybe re-ask on the post so that the info is more public? – tshepang Jan 21 '11 at 19:41
  • @tshepang I'm not sure what his irc handle is, so I can't say who I was talking to, though I have logs. feel free to ask anyone else anything you want. I'm simply giving my opinions on a couple of things, which could merely display my ignorance more than an actual problem – xenoterracide Jan 21 '11 at 19:43
  • @tshepang all I can say is event timings seem like an unnecessary feature that is opposite of one program to do one thing well. Perhaps comes from too much launchd love, and that network socket listening sounds like another attack surface to me. But if you really want more of "why" you'll have to read the origin, or ask them yourself. – xenoterracide Jan 21 '11 at 20:07
  • As I understand it, at some point the plan is to completely provide all of the capabilities of atd and crond, although not necessarily in a compatible way. There is also a plan to make it work as a session manager on the user level, so possibly that would work for per-user crontab. I dunno. – mattdm Feb 25 '11 at 01:15
11

Well one thing most of you forgot is the organisation of processes in cgroups.

So if systemd started a thing, it will put this thing in its own cgroup and there is no (unpriviledged) mean for the process to escape that cgroup. Here's the consequences of that:

  • An administrator of a big system with many users has efficient new ways to identify malicious users/processes.
  • The priorities for CPU-scheduling can be determined better as done by the "Wonder autocgroup patch".
tshepang
  • 65,642
8

For a very detailed look at systemd, starting with the first design drafts (and a detailed critique of existing init systems, including upstart, and how systemd proposes to fix them), go to its home page. Over time, there have been several articles on startup published in LWN. Just be advised that any mention of systemd (or pulseaudio) there triggers neverending flamewars.

IMVHO (and as a Fedora user) I'm very happy with it. Something in this line was long overdue to handle the complexity of current Linux systems. Fedora used upstart for a while, but it never got out of the stage of being a fancy replacement for sysvinit, running mostly unchanged init scripts. Its promise of simplifying boot configuration comes at the cost of again manually setting up interdependencies, and that just doesn't work. systemd figures dependecies out by itself (or just allows starting stuff without regard of dependencies, they sort themselves out). Another big advantage (some say it is a severe disadvantage) is that it exploits Linux-specific features to the hilt (notably cgroups allow isolating a daemon and all its descendants, so it is easy to monitor, limit the resources, or kill them as a group; there are many others).

vonbrand
  • 18,253
3

Journaling - Systemd is literally like WinSXS folder when it comes to logging stuff, it creates copies of copies unless you manually delete or reduce the file size it will keep eating away at your drive. I call it boot loader cookies.

Bert
  • 31
  • 1
  • wrong. those aren't copies and it has configurable limit https://www.freedesktop.org/software/systemd/man/journald.conf.html – pal Feb 06 '19 at 01:43