How do I diagnose a failing nix build?

Question

How do I diagnose a nix build failing?

Currently I see output as:

nix build -v
warning: dumping very large path (> 256 MiB); this may run out of memory
building '/nix/store/fdrm6kbm68vld3bhfjizv684ck725lyf-blog.drv'...
builder for '/nix/store/fdrm6kbm68vld3bhfjizv684ck725lyf-blog.drv' failed with exit code 1; last 5 log lines:
  unpacking sources
  unpacking source archive /nix/store/s7r5vlvp49ad6a9d5hqhsiaxw691iyhf-Blog
  source root is Blog
  patching sources
  configuring
[0 built (1 failed), 0.0 MiB DL]
error: build of '/nix/store/fdrm6kbm68vld3bhfjizv684ck725lyf-blog.drv' failed

I'd expect to find some logs / errors of why it failed to build?

Following from https://stackoverflow.com/a/47264375/1663462:

I've tried adding build-cache-failures = true; to the default.nix however I still see no output from:

nix-store --read-log

And nix-store --query-failed-paths results in:

error: no operation specified
Try 'nix-store --help' for more information.

milahu · Answer 1 · 2024-03-21T09:52:55.480

see also:

nix-build-debug - debug failing nix-build in interactive bash shell
Building a stdenv package in nix-shell in the nixpkgs docs
Nix-build-phases: run nix build phases interactively in the nixos discourse
Development environment with nix-shell in the nixos wiki
runPhase function in nixpkgs
- eval "${!curPhase:-$curPhase}"
- cd "${sourceRoot:-.}"
genericBuild function in nixpkgs - calls runPhase $curPhase

to debug the build process of a package:

instead of nix-build

nix-build -E 'with import <nixpkgs> {}; callPackage ./. {}'

run nix-shell to enter the build environment

nix-shell -E 'with import <nixpkgs> {}; callPackage ./. {}'

and then run the build phases

# run build in tempdir
cd $(mktemp -d)
dont install to /nix/store
for n in $outputs; do eval export $n=$PWD/result-$n; done
https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh
phases="${prePhases[*]:-} unpackPhase patchPhase ${preConfigurePhases[*]:-} 

    configurePhase ${preBuildPhases[*]:-} buildPhase checkPhase 

    ${preInstallPhases[*]:-} installPhase ${preFixupPhases[*]:-} fixupPhase installCheckPhase 

    ${preDistPhases[*]:-} distPhase ${postPhases[*]:-}";
echo writing nix-build-phases.txt
for curPhase in $phases; do
    echo "eval &quot;${$curPhase:-$curPhase}&quot;"
    if [ "$curPhase" = unpackPhase ]; then
        echo '[ -n "${sourceRoot:-}" ] && chmod +x "${sourceRoot}"'
        echo 'cd "${sourceRoot:-.}"'
    fi
done >nix-build-phases.txt
echo "please manually run the build phases in nix-build-phases.txt"
echo
cat nix-build-phases.txt

example nix-build-phases.txt

eval "${unpackPhase:-unpackPhase}"
[ -n "${sourceRoot:-}" ] && chmod +x "${sourceRoot}"
cd "${sourceRoot:-.}"
eval "${patchPhase:-patchPhase}"
eval "${updateAutotoolsGnuConfigScriptsPhase:-updateAutotoolsGnuConfigScriptsPhase}"
eval "${configurePhase:-configurePhase}"
eval "${buildPhase:-buildPhase}"
eval "${checkPhase:-checkPhase}"
eval "${installPhase:-installPhase}"
eval "${fixupPhase:-fixupPhase}"
eval "${installCheckPhase:-installCheckPhase}"
eval "${distPhase:-distPhase}"

in this build environment, you can

find the cause of build errors (what build phase is failing?)
modify the build commands, for example redefine the buildPhase: run type buildPhase to print the old phase, copy the old phase, modify the phase, paste the new phase
run the modified build commands. this can be faster than a full rebuild

note: builders can execute different build phases. the actual build phases are stored in variables like prePhases or preConfigurePhases

problem: the build script runs with set -e so it will exit on the first error, but that would exit the interative shell. this is a limitation of the bash shell, so ideally, we would run the build script in a bash debugger, to stop execution on the first error, modify commands, step back and repeat execution of commands, and to allow inspection of state.

# run build in tempdir
cd $(mktemp -d)
dont install to /nix/store
for n in $outputs; do eval export $n=$PWD/result-$n; done
https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh
phases="${prePhases[*]:-} unpackPhase patchPhase ${preConfigurePhases[*]:-} 

    configurePhase ${preBuildPhases[*]:-} buildPhase checkPhase 

    ${preInstallPhases[*]:-} installPhase ${preFixupPhases[*]:-} fixupPhase installCheckPhase 

    ${preDistPhases[*]:-} distPhase ${postPhases[*]:-}";
test
#patchPhase="echo test error in patchPhase; false"
donePhases=()
missingPhases=($phases)
for curPhase in $phases; do
    # no. genericBuild does not "return 1" on error
    #phases="$curPhase" genericBuild || break
    echo
    echo "# running $curPhase"
    # FIXME this also traps suspend (Ctrl-Z)
    trap '
        echo "FIXME error in $curPhase"
        echo "done phases:" ${donePhases[@]}
        echo "missing phases:" ${missingPhases[@]}
        echo "hit Enter to close the nix-shell"
        read
    ' EXIT
    set -e # exit on error
    set -x # trace
    eval "${!curPhase:-$curPhase}"
    set +x
    set +e
    if [ "$curPhase" = unpackPhase ]; then
        [ -n "${sourceRoot:-}" ] && chmod +x "${sourceRoot}"
        cd "${sourceRoot:-.}"
    fi
    donePhases+=($curPhase)
    missingPhases=(${missingPhases[@]:1})
done

Sorry I'm a few years late, but yes this finally does make sense now though I think it's missing some context to highlight that these commands or phases relate to building a derivation. It would be useful to know that these commands are likely exactly the ones being run indirectly via nix-build and hence you get the full output with no truncation or aggregation. — Chris Stryczynski, Jul 28 '23 at 11:15
nix-build can execute different build phases, for example in python3.pkgs.buildPythonPackage. the actual build phases are stored in the variable $phases, so in the nix-shell, see echo $phases. docs: Building a stdenv package in nix-shell, issues: docs: fix nix-shell commands, docs: Building a stdenv package in nix-shell — milahu, Jul 28 '23 at 11:54
Using this approach when running buildPhase results in: no Makefile, doing nothing. Other commands seem to fail with mkdir: cannot create directory ‘/nix/store/kk9irdv2ppca45fzwwz7422vsmbpgga1-HaskellNixCabalStarter-0.1.0.0’: Read-only file system. — Chris Stryczynski, Jan 14 '24 at 16:26
fixed my code to get the actual phases, including extra phases like cmakeConfigurePhase. the Read-only file system error is fixed by using different output paths. another problem is that nix-build runs the build script with set -e to stop on the first error, but this would exit the interactive bash shell... see also nixos discourse — milahu, Jan 15 '24 at 13:21
for the chromium package eval "${unpackPhase:-unpackPhase}" fails with variable $src or $srcs should point to the source but phases="unpackPhase" genericBuild works... so probably genericBuild is better as a generic phase runner — milahu, Feb 05 '24 at 12:26

score 5 · Accepted Answer · edited Oct 10 '23 at 12:45

5

Reading log should work. Did you specify the derivation? nix-store --read-log /nix/store/fdrm6kbm68vld3bhfjizv684ck725lyf-blog.drv

edited Oct 10 '23 at 12:45

Chris Stryczynski

5,818

answered Feb 08 '19 at 18:26

Vladimír Čunát

1,296

4

If by that you mean nix-store --read-log /nix/store/fdrm6kbm68vld3bhfjizv684ck725lyf-blog.drv then yes this seems to add additional info. – Chris Stryczynski Feb 08 '19 at 18:36
4

Sometimes it's also useful to build with --keep-failed to retain the temporary directory of the build. – Vladimír Čunát Feb 08 '19 at 19:19
Is this the same functionality that is implemented here https://github.com/NixOS/nix/pull/2810 with a new --print-build-logs? – Chris Stryczynski Jun 20 '20 at 10:07
What I posted reads logs of past builds. What you link appears to be "only" about showing logs of the builds that are currently happening. – Vladimír Čunát Jun 22 '20 at 16:06
nix log /nix/store/fdrm6kbm68vld3bhfjizv684ck725lyf-blog.drv – David 天宇 Wong Jul 12 '22 at 02:09

Chris Stryczynski · Answer 3 · 2024-01-14T16:29:53.973

3

There is a --debug option as well as a --print-build-logs option (on nix build but not nix-build) that can provide additional info.

Regarding the specific issue - it might be this bug: https://github.com/NixOS/nix/issues/2176

More information

https://releases.nixos.org/nix/nix-0.16/build/565033/download/1/manual/index.html#id494024

edited Jan 14 '24 at 16:29

answered Feb 05 '19 at 00:06

Chris Stryczynski

5,818

1

From the link: At the beginning of each phase, the set of all shell variables is written to the file env-vars at the top-level build directory. This is useful for debugging: it allows you to recreate the environment in which a build was performed. For instance, if a build fails, then assuming you used the -K flag, you can go to the output directory and “switch” to the environment of the builder – Ben Creasy Sep 23 '19 at 01:28
1

thank you! --debug is exactly what I was looking for :) – hraban Jun 23 '22 at 07:40

Robert K. Bell · Answer 4 · 2023-12-03T08:51:11.527

Adding breakpointHook to nativeBuildInputs of the failing derivation is also a powerful technique ― it allows you to connect with cntr and explore the environment as it was at the point that the derivation's build failed.

Thanks to jamesbrock on the NixOS Discourse, the short of it is:

Add the breakpointHook package to the nativeBuildInputs of a mkDerivation.

nativeBuildInputs = [ breakpointHook ];

Install cntr

nix-env -i cntr

Run your Nix build command, and when it fails, the breakpointHook will output a message like this:

build failed in installPhase with exit code 1
To attach install cntr and run the following command as root:
cntr attach -t command cntr-/nix/store/6vwxqrwq5h1fd3nw4mc61wgk7rppn2qw-jupyterlab-extended

So we run that command as root. The root user doesn’t have a PATH to the cntr command which we installed in our Nix profile, so give it the full path to our single-user-installed cntr.

sudo /home/$USER/.nix-profile/bin/cntr attach -t command cntr-/nix/store/6vwxqrwq5h1fd3nw4mc61wgk7rppn2qw-jupyterlab-extended

And now we’re in the sleeping Nix container.
Note that paths like $out and $TMPDIR are available at this stage along with your usual tools, however, they're under the current directory (/var/lib/cntr), instead of /.
(You can use them like cd $(echo $out | cut -c2-).)

From here, run cntr exec to fully load the build environment.

/home/$USER/.nix-profile/bin/cntr exec

The $TMPDIR directory is where Nix has created the temporary directory for the build 10.

cd $TMPDIR

Now we are fully inside the context of the build, and any commands which we enter in the shell will be as if we had entered that command on a line in the phase (for example, buildPhase) of our derivation when the derivation failed.

Mentioned here in the NixOS Discourse:

I want to also add a few details regarding breakpointHook. It is a really powerful debug hook by @Mic92.

It is enough to put pkgs.breakpointHook into your buildInputs to get it working. In the case of a failure there is an exact “connection” command printed out to the console. You can use this command (cntr) to attach a shell into the build environment. To use it you have to be able to run sudo along with cntr (so you should install cntr package before ;-)).

You can find more interesting details in the @Mic92 NixCon presentation: Jörg 'Mic92' Thalheim - About Nix sandboxes and breakpoints (NixCon 2018)

Here's a link to the relevant section of the manual: https://nixos.org/manual/nixpkgs/stable/#breakpointhook

How do I diagnose a failing nix build?

4 Answers4

dont install to /nix/store

https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh

dont install to /nix/store

https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh

test