Argument list too long error with makefile

Question

In a makefile, I have

@echo "$(IGNORE_DIRS) $(CLEAN_FILES) $(CLEAN_DIRS) $(REALCLEAN_FILES)" | tr ' ' '\n' >> $@

The problem is that $(CLEAN_FILES) is quite large, so when I run make, I get

make: execvp: /bin/sh: Argument list too long

I'm on Xubuntu 18.10.

Edit: I should provide a little more context. What I am working on is a make rule (I'm using GNU make) to automatically generate the .hgignore file. Here is the make rule in its entirety:

.hgignore : .hgignore_extra
    @echo "Making $@"
    @rm -f $@
    @echo "# Automatically generated by Make. Edit .hgignore_extra instead." > $@
    @tail -n +2 $< >> $@
    @echo "" >> $@
    @echo "# The following files come from the Makefile." >> $@
    @echo "syntax: glob" >> $@
    @echo "$(IGNORE_DIRS) $(CLEAN_FILES) $(CLEAN_DIRS) $(REALCLEAN_FILES)" | tr ' ' '\n' >> $@
    @chmod a-w $@
.PHONY : .hgignore

Edit 2: At @mosvy 's suggestion, I have also tried

.hgignore : .hgignore_extra
    @echo "Making $@"
    @rm -f $@
    @echo "# Automatically generated by Make. Edit .hgignore_extra instead." > $@
    @tail -n +2 $< >> $@
    @echo "" >> $@
    @echo "# The following files come from the Makefile." >> $@
    @echo "syntax: glob" >> $@
    $(file >$@) $(foreach V,$(IGNORE_DIRS) $(CLEAN_FILES) $(CLEAN_DIRS) $(REALCLEAN_FILES),$(file >>$@,$V))
    @true
    @chmod a-w $@
.PHONY : .hgignore

Running make .hgignore with this, I no longer get the "Argument list too long" error, but the generated .hgignore file only contains output up to the syntax: glob line, and then nothing after that.

Answer 1

As @schily has already explained, this is not a shell problem, and cannot be worked around with xargs, quoting, splitting into more echo's with ;, etc. All the text from a make action is passed as argument/s to a single execve(2), and it can't be longer than the maximum size allowed by the operating system.

If you're using GNU make (the default on linux), you can use its file and foreach functions:

TEST = $(shell yes foobar | sed 200000q)

/tmp/junk:
        $(file >$@) $(foreach V,$(TEST),$(file >>$@,$V))
        @true

.PHONY: /tmp/junk

This will print all words from $(TEST) separated by newlines into the file named in $@. It's based on a similar example from make's manual.

Your Makefile could probably be reworked into something more manageable, that doesn't require fancy GNU features, but it's hard to tell how from the snippets you posted.

Update:

For the exact snippet from the question, something like this could do:

.hgignore : .hgignore_extra
    $(info Making $@)
    $(file >$@.new)
    $(file >>$@.new,# Automatically generated by Make. Edit .hgignore_extra instead.)
    $(shell tail -n 2 $< >>$@.new)
    $(file >>$@.new,)
    $(file >>$@.new,# The following files come from the Makefile.)
    $(file >>$@.new,syntax: glob)
    $(foreach L, $(IGNORE_DIRS) $(CLEAN_FILES) $(CLEAN_DIRS) $(REALCLEAN_FILES), $(file >>$@.new,$L))
    @mv -f $@.new $@
    @chmod a-w $@
.PHONY : .hgignore

I've changed it a little, so it first writes into .hgignore.new, and if everything goes well, only then move .hgignore.new to .hgignore. You'll have to change back the indenting spaces to tabs, because this dumb interface is mangling whitespaces.

Answer 2

On UNIX the following rule applies:

The following data forms the initial stack of a process:

• The sum of strlen() of all environment strings + final nul character per string
• The sum of strlen() of all argument strings + final nul character per string
• The environment array: n+1 environment strings * sizeof char *
• The argv array: n+1 argument strings * sizeof char *
• A few additional numbers

All this data must not exceed ARG_MAX.

On a historical UNIX, the value for ARG_MAX was 10240 or 20480 bytes.

SunOS-4.0 (published in December 1987) raised that limit to 1MB

Solaris-7.0 (published in 1997) introduced 64 Bit support and in order to avoid a practically smaller limit on 64 Bit systems (caused by bigger env and argvarrays as a result from a larger char *), ARG_MAX was raised to 2MB for 64 bit programs.

BTW: Modern POSIX compliant OS include support for the getconf program and getconf ARG_MAX prints the actual value. On a 64 Bit Linux, this returns 2MB so Linux on the first view seems to adopt the SunOS enhancements....

Now let us look at make:

The make program calls commands from Makefiles via:

sh -ce command

where command is a single argument that is the expanded string you see in an action line from a Makefile.

SunPro Make introduced an optimization in the early 1990s:

• If a command line does not contain shell meta characters, make itself tokenizes the command line and calls the command via: execv() in order to avoid the overhead from a shell call.

Later, gmake and smake adopted this optimization.

smake introduced another optimization in 2012:

• If a shell command is introduced by a simple echo command that ends in a ; and if the following command line does not contain shell meta characters, the echo is inlined to smake and the command after the ; is executed via execv() in order to reduce the overhead for modern build systems that typically use @ to suppress make command echos and rather use simplified echo calls to make the make output easier to understand (see e.g. the Schily Makefile system introduced in February 1993).

None of these make rules apply to your makecommand line since your command line contains shell meta characters. So your whole command is called via:

sh -ce command

where command is a single string with the total size caused by your makefile.

Now it seems that the Linux kernel is neither UNIX nor POSIX compliant and enforces an additional limitation that never existed on UNIX. This additional limitation seems to be based on the maximum length of a single string.

If this is really true, this would disqualify Linux as it would prevent Linux from being useful with larger projects managed by make.

Did you think about making a bug report to the Linux kernel folks?

Answer 3

The ur-question in such cases is why you are doing this in the first place? It looks like you are trying to create some sort of .ignore file containing all the files which are presumably either program output or globs. In the former case I would advocate passing the program output directly to tr (if strictly necessary) and then to the file without passing through an intermediary variable. If you're using globs then you can use a very simple for loop:

for file in *
do
    echo "$file" >> out.txt
done

(Escape if necessary in your Makefile.)

In very general terms:

• Use pipes and redirects rather than variables for any large chunks of data. They are really fast (since they can be handled as fast as every program in the pipeline can process input).
• Use xargs when you really have to.
• Avoid useless echos and cats.

Answer 4

See this answer, in particular this quote:

one argument must not be longer than MAX_ARG_STRLEN (131072). This might become relevant if you generate a long call like "sh -c 'generated with long arguments'".

On the other hand, the limit on total number of arguments to pass is quite high, so maybe instead just pass them as separate arguments, since the end result of the echo will be the same?

This just removes the "s from your original command:

@echo $(IGNORE_DIRS) $(CLEAN_FILES) $(CLEAN_DIRS) $(REALCLEAN_FILES) | tr ' ' '\n' >> $@

Hopefully this is enough to keep the arguments within the limits for number of arguments (which is quite high) and length of each argument (which will no longer be an issue, since each argument is now short.)

---

UPDATE: Removing the quotes won't really fix this completely, since make runs each command in a shell, using the equivalent of sh -c '...' where the whole command becomes a single argument, so it will still be bound to the limit length for an individual argument, which is 131,072 bytes on Linux on x86 platform.