3

Suppose I do a grep that gives me a lot of results. I want to use the output of grep and pass it to a command that takes a filename as it's first argument. For example - 

myCommand filename.txt

I would like to do something like this - 

grep "xyz" | myCommand

However, since myCommand doesn't read from stdin, this piping does not work. I don't want to create a temporary file on disk with the results of grep. Is it possible by any other means to create a "file", say in memory and let myCommand use that?

Gilles 'SO- stop being evil'
  • 829,060
CodeBlue
  • 1,117
  • 8
  • 15
  • 22
  • Possibly related. I asked a similar question here once. It's okay to create the temporary files. You'll find some good practice on how to do this in the answers linked. – rahmu Jul 24 '13 at 14:14

2 Answers2

6

A simple solution is to use Bash's support for process substitution, like this:

myCommand <(grep "xyz" somefile)

This will connect the output of the grep command to a file descriptor and then pass /dev/fd/NNN (where NNN is the fd number) to your command as the input file (for example, your command might actually get run as myCommand /dev/fd/63). This solution does not create a temporary file.

You can also create a named pipe and use that to transfer data around:

$ mkfifo /tmp/myfifo
$ grep "xyz somefile > /tmp/myfifo &
$ myCommand /tmp/myfifo

But really, the first solution is simpler.

Stephane Chazelas comments:

zsh also has myCommand =(grep xyz somefile) which uses a regular temporary file (for cases where myCommand can't accept a pipe).

This is important because some commands expect to be able to perform random access on their input file, which won't work with a pipe. Using this syntax you get the benefits of temporary files without needing to worry about cleaning them up yourself.

larsks
  • 34,737
  • Note that although process substitution is great here, it's not supported by all shells or on all operating systems. The fifo approach should work just about everywhere. – Mat Jul 24 '13 at 14:19
  • myCommand <(grep "xyz" somefile) didn't work on Ubuntu for me. Now I'm trying the second approach. – CodeBlue Jul 24 '13 at 14:28
  • @CodeBlue, note that the first solution does, as I indicated, require Bash. If you're using a different shell there might be a different syntax (or there might not be an equivalent feature). – larsks Jul 24 '13 at 14:31
  • 2
    process substitution originated in ksh93 and is also available in zsh and bash. zsh also has myCommand =(grep xyz somefile) which uses a regular temporary file (for cases where myCommand can't accept a pipe). – Stéphane Chazelas Jul 24 '13 at 14:33
  • Stephane: good to know, I've added this to the answer. – larsks Jul 24 '13 at 14:37
  • @larsks when I do echo $0 on my Ubuntu terminal, I get /bin/bash. So it is indeed using the bash shell, but it still doesn't work. – CodeBlue Jul 24 '13 at 14:46
  • Update: on Ubuntu, the first solution works for some commands but not for others. – CodeBlue Jul 24 '13 at 16:27
2

Most distros now include a RAM disk for this exact purpose. You can write files to it like normal, it's typically called /dev/shm. You can check if you have it like so:

$ mount |grep shm
tmpfs on /dev/shm type tmpfs (rw)

If you have it you can write to it just like any other filesystem:

$ echo "hello world" > /dev/shm/somefile.txt

And see it using normal tools:

$ ls -l /dev/shm/
total 680
-r-------- 1 saml saml 67108904 Jul 22 20:55 pulse-shm-3981461026
-r-------- 1 saml saml 67108904 Jul 18 18:06 pulse-shm-799251885
-rw-rw-r-- 1 saml saml       12 Jul 24 10:59 somefile.txt

However this consumes no actual diskspace, it's using RAM.

slm
  • 369,824