Map over file tree, preserving structure

Question

I have a file tree that looks like:

$ tree src
src
├── bible
│   ├── index.md
│   └── README.md
├── index.md
└── other.md

I want to render every Markdown file within this file tree to HTML via pandoc(1) -- preserving structure.

This new file tree should be rooted at out, like so:

$ tree out
out
├── bible
│   ├── index.html
│   └── README.html
├── index.html
└── other.html

Ideally, I'd like to do this via make(1). This is what I have so far:

SRC_DIR=src
OUT_DIR=out
.PHONY: all
all: $(SRC_DIR)/.md
    find . -name ".md" -exec pandoc '{}' -o '{}'.html ;
    find -name ".html" -exec bash -c 'mv {} $(OUT_DIR)/dirname {}/basename {} .md.html.html' ;
    # mv $(SRC_DIR)/.html $(OUT_DIR)
    firefox $(OUT_DIR)/index.html
.PHONY: clean
clean:
    rm $(OUT_DIR)/*.html

This fails:

$ make
find . -name "*.md" -exec pandoc '{}' -o '{}'.html \;
find . -name "*.html" -exec bash -c 'mv {} `basename {} .md.html`.html' \;
mv src/*.html out
mv: cannot stat 'src/*.html': No such file or directory
make: *** [Makefile:8: all] Error 1

Kusalananda · Answer 1 · 2021-07-16T06:43:28.287

For the pathname of a single one of your .md files in $pathname, somewhere in or below src:

name=$(basename "$pathname" .md)
destdir=out/$( dirname "${pathname#src/}" )
mkdir -p "$destdir" && pandoc -o "$destdir/$name.html" "$pathname"

Here, basename "$pathname" .md would be the file's name without the .md filename suffix and without any directory paths (e.g. README for src/bible/README.md), ${pathname#src/} would be the the pathname of the file without the initial src/ directory name and $destdir would be set to the destination directory pathname (src/ exchanged for out/, no final filename component, e.g. out/bible for src/bible/README.md). At the end, we let pandoc write to $destdir/$name.html (if the creation of the destination directory succeeded).

You can run this for all .md files in a directory structure:

find src -type f -name '*.md' -exec sh -c '
    for pathname do
        name=$(basename "$pathname" .md)
        destdir=out/$( dirname "${pathname#src/}" )
        mkdir -p "$destdir" && pandoc -o "$destdir/$name.html" "$pathname"
    done' {} +

This is the same set of commands in a loop. We let find feed the loop with pathnames found under src (see also Understanding the -exec option of `find`).

Testing:

$ tree -F
.
`-- src/
    |-- bible/
    |   |-- README.md
    |   `-- index.md
    |-- index.md
    `-- other.md
2 directories, 4 files

(command is being run here)

$ tree -F
.
|-- out/
|   |-- bible/
|   |   `-- README.html
|   |-- index.html
|   `-- other.html
`-- src/
    |-- bible/
    |   |-- README.md
    |   `-- index.md
    |-- index.md
    `-- other.md
4 directories, 7 files

If you want to use your SRC_DIR and OUT_DIR Makefile variables:

find $(SRC_DIR) -type f -name '*.md' -exec sh -c '
    srcdir=${1%/}; outdir=$2; shift 2
    for pathname do
        name=$(basename "$pathname" .md)
        destdir=$outdir/$( dirname "${pathname#$srcdir/}" )
        mkdir -p "$destdir" && pandoc -o "$destdir/$name.html" "$pathname"
    done' $(SRC_DIR) $(OUT_DIR) {} +

That is, pass the src and out names on the sh -c script's command line and pick them out inside the in-line script.

I'm not 100% sure about how the quoting in a Makefile works, and you may want to escape the newlines in the code above, alternatively create a separate little script for doing this, and then call that from the Makefile.

guest_7 · Answer 2 · 2021-07-16T16:19:31.273

Despite the similarities in their syntax, makefiles and shell scripts are as different as chalk from cheese.

You are writing a makefile in the spirit of a shell script. In make, we have to think in terms of relationships and dependencies. The shell comes into the picture only at the last stage as rules where dependencies are satisfied. These are called recipes in make parlance.

###### 
SHELL := /bin/sh
SRC_DIR := src
OUT_DIR := out
recursive glob
*/ = $(foreach i,$(strip 

  $(wildcard $(1:=/))),$(strip 

  $(call $0,$i,$2) 

  $(filter $(subst ,%,$2),$i)))
MD_FILES = $(call */,$(SRC_DIR),*.md)
HTML_FILES = $(subst $(SRC_DIR)/,$(OUT_DIR)/,$(MD_FILES:.md=.html))
.PHONY: all
all: $(HTML_FILES)
.PHONY: make_dir
make_dir: $(OUT_DIR)/
$(OUT_DIR)/%.html: $(SRC_DIR)/%.md |make_dir
  pandoc $< -o $@
$(OUT_DIR)/: $(SRC_DIR)/
  rsync -avz -f"+ /" -f"- " $< $@
.PHONY: clean
clean:
  rm -f -- $(HTML_FILES)

Map over file tree, preserving structure

2 Answers2

recursive glob