17

Earlier today when building something, I decided to run make as 

$ make -j

perhaps out of habit with other programs such as cabal where -j defaults to a reasonable limit.

About 20 seconds later, my entire desktop grinds to a halt. I look for various signs of activity. No fans spin up. The HDD light is a solid green, but I hear no disk activity. Hmmmmm. After 10 minutes of silence, I finally see a response to the first keypress I made ages ago, and I also begin to hear the all too familiar sound of disk thrashing. 20 minutes later of slowly trying to wade my way into a terminal on this unresponsive machine, I caved and used REISUB.

At first, I figured an unrelated desktop application must have been the culprit, because I have long since had memory limits placed on interactive bash sessions to prevent me from putting myself into exactly this sort of situation! But /var/log/syslog tells a different story; the OOM killer left behind some ps dumps which are suspiciously packed with c++ and cc1plus processes!

Here is a frequency analysis of one of those dumps:

Command          Number of appearances
'sh'                    322
'c++'                   321
'cc1plus'               321
'chrome'                 27
'make'                   27
'bash'                    3
all else combined       120

So I check the man page for GNU make: (emphasis added)

-j [jobs], --jobs[=jobs] Specifies the number of jobs (commands) to run simultaneously. If there is more than one -j option, the last one is effective. If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously.

I'm reluctant to see if I can reproduce the issue (Doctor, it hurts when I do this...), but the results of investigation so far seem to be a home run: Clearly, make -j and the hundreds of resulting processes must have been the cause of the hang and the disk thrashing. That said, searching the internet, I can't find much warning against it. Am I jumping to conclusions?

Is make -j as dangerous as it would seem to me? If so, why on earth is it there, and what can be done to idiot-proof it?

Exp HP
  • 271
  • Note: On the last question, I am considering adding an alias make="make -j4" to eliminate the need for adding -jN arguments, but I still haven't thought through all the possible consequences... – Exp HP Oct 15 '16 at 20:55
  • 1
    For global limits please use make's own facilities and do export MAKEFLAGS="-j 4". For further investigations I advice reading before typing . P.S.: + 2 has served me well for almost two decades now. Back in the day, Linus did a final make -j128 before releasing a kernel. – Bananguin Sep 19 '17 at 18:26

2 Answers2

7

There are tools that allows you to shoot yourself in the foot in many imaginative ways. This is so that you can use your imagination to solve problems without being restricted by what someone else thinks is "sane".

Running make -j on a small project is perfectly reasonable. In other projects, using -j without an argument will seriously cripple system responsiveness. In some projects, using parallel builds, even with -j2, will break the build completely (files created by a parallel make process are not there in time for another etc.).

I would personally avoid aliasing make to make -j4 (as you say you are considering, in comments). I find that it's better to explicitly tell the machine what to do, so that I know what it will do. In a few days, I will have forgotten about that alias, and wonder why the four projects I'm building in separate terminals are making my system unresponsive.

As for "dangerous"... The word means different things in different contexts. Yes, it's "dangerous" because it may make the system unresponsive. Yes, it is "dangerous" because it may well crash the build process halfway through the build. But no, it's not "dangerous" in the sense that it'll reformat your hard drive or start deleting random files.

So, how to idiot-proof it?

Here's a surefire step-by-step guide:

  1. Learn to use your tools.

Notice, too, that the -j flag to BSD make does require an argument, and that this flag is non-standard (the Unix POSIX standard doesn't mention it).

Kusalananda
  • 333,661
5

You can also limit make using -l:

-l [load], --load-average[=load] Specifies that no new jobs (commands) should be started if there are others jobs running and the load average is at least load (a floating-point number). With no argument, removes a previous load limit.

But note that it does not seem to help to run it like this: make -j -l4.

Too many jobs are started before the load average raises over the limit (based on my experience). Thus a combination can work, e.g. make -j8 -l4.

Zitrax
  • 274