output to file, then use file for input

Question

Is there a shorter way of writing this? Basically output a command to a file, then use the file as input for the next command. I also want to keep the file to view afterwards.

cmd1 > verylong.txt; cmd2 < verylong.txt

I know I can do

cmd1 | tee verylong.txt | cmd2

But since I expect "verylong.txt" to be a huge file, I thought it would be less efficient to use pipe since that would hold the entire file in memory. Whereas if I use file input then it would process it one line at a time. (Or is my assumption wrong?)

It would be great if I could do something elegant like

cmd1 > verylong.txt > cmd2

Answer 1

As far as I know, cmd1 | tee verylong.txt | cmd2 will not hold the whole file in memory. In fact, if cmd2 was to wait too long before consuming its input, cmd1 might block on a write call and unblock only when cmd2 starts reading again.

The reason for that is that there is a buffer for the pipe, and that buffer, by default, is limited to a certain reasonable size.

Of course, the story might be different if cmd2 is sort (or something alike) where the entire input must be read before the command is able to write its output. In that case, the entire file content might be held in cmd2 memory, but that is independent of whether a pipe or an intermediary file was used for the input of that command.

Answer 2

The already given answer is correct. But if your goal is to selectively read your verylongfile.txt w/ cmd2, sed might be another option.

cmd1 | sed -e 'w verylongfile.txt' -e '/notinteresting/d' | cmd2

sed will write all of its input to the outfile, but only the bits that do not match the /notinteresting/ address to the pipe. Or you might negate the action with /interesting/!d which would write only the lines that match the interesting address to the pipe.

If this is not your goal, use tee instead, though - it is a more efficient tool for writing the whole of its input to both the outfile and the pipe.

Answer 3

There is a clever trick with tee and subshells:

cat source.lst | tee >(doSomething.sh) >(somethingElse.sh) | somethingFinal.sh

I've done this before

pv -perl source.list | tee >(doSomething.sh) >(somethingElse.sh) | md5sum

pv will give you a progress bar, an ETA, and a running line total. Then source.lst will be fed to doSomething.sh and somethingElse.sh (and on different CPUs!) Finally we'll get an md5sum of that hugefile, just for academic purposes.

Answer 4

Whats wrong with simple two line batch file? Like:

Cmd1 >filespec
Cmd2 <filespec

Or

cmd1 >filespec
cmd2 filespec

either way, the file is left in mass storage.