What is the fastest way to create a list of directories specified in a file?

Question

I have a text file, "foo.txt", that specifies a directory in each line:

data/bar/foo
data/bar/foo/chum
data/bar/chum/foo
...

There could be millions of directories and subdirectories What is the quickest way to create all the directories in bulk, using a terminal command ?

By quickest, I mean quickest to create all the directories. Since there are millions of directories there are many write operations.

I am using ubuntu 12.04.

EDIT: Keep in mind, the list may not fit in memory, since there are MILLIONS of lines, each representing a directory.

EDIT: My file has 4.5 million lines, each representing a directory, composed of alphanumeric characters, the path separator "/" , and possibly "../"

When I ran xargs -d '\n' mkdir -p < foo.txt after a while it kept printing errors until i did ctrl + c:

mkdir: cannot create directory `../myData/data/a/m/e/d': No space left on device

But running df -h gives the following output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda        48G   20G   28G  42% /
devtmpfs        2.0G  4.0K  2.0G   1% /dev
none            401M  164K  401M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm

free -m

 total       used       free     shared    buffers     cached
Mem:          4002       3743        258          0       2870         13
-/+ buffers/cache:        859       3143
Swap:          255         26        229

EDIT: df -i

Filesystem      Inodes   IUsed  IFree IUse% Mounted on
/dev/xvda      2872640 1878464 994176   66% /
devtmpfs        512053    1388 510665    1% /dev
none            512347     775 511572    1% /run
none            512347       1 512346    1% /run/lock
none            512347       1 512346    1% /run/shm

df -T

Filesystem     Type     1K-blocks     Used Available Use% Mounted on
/dev/xvda      ext4      49315312 11447636  37350680  24% /
devtmpfs       devtmpfs   2048212        4   2048208   1% /dev
none           tmpfs       409880      164    409716   1% /run
none           tmpfs         5120        0      5120   0% /run/lock
none           tmpfs      2049388        0   2049388   0% /run/shm

EDIT: I increased the number of inodes, and reduced the depth of my directories, and it seemed to work. It took 2m16seconds this time round.

Answer 1

With GNU xargs:

xargs -d '\n' mkdir -p -- < foo.txt

xargs will run as few mkdir commands as possible.

With standard syntax:

(export LC_ALL=C
 sed 's/[[:blank:]"\'\'']/\\&/g' < foo.txt | xargs mkdir -p --)

Where it's not efficient is that mkdir -p a/b/c will attempt some mkdir("a") and possibly stat("a") and chdir("a") and same for "a/b" even if "a/b" existed beforehand.

If your foo.txt has:

a
a/b
a/b/c

in that order, that is, if for each path, there have been a line for each of the path components before, then you can omit the -p and it will be significantly more efficient. Or alternatively:

perl -lne 'mkdir $_ or warn "$_: $!\n"' < foo.txt

Which avoids invoking a (many) mkdir command altogether.

Answer 2

I know we will get lot of answers for this question.But still you can TRY this :) :D

while read -r line; do mkdir -p "$line" ; done < file.txt

Answer 3

Try this one-liner:

for i in $(cat foo.txt) ; do mkdir -p $i ; done

This will create the directory/directory-tree in the current working directory. Not in bulk (as in all-the-directories-getting-created-simulataneously), but the creation will be done one after another.