0

I'm trying to convert thousands of video files from one format (vob) to another (mkv). Each source video is segmented into < 8 sequentially numbered 1GB *.vob files. Sadly, the video converting program can only accept a single concatenated vob file, which is easily created via:

cat *.vob > full_movie.vob

(where full_movie.vob is typically 6-7 GB in size)

The video format converting program is of the form:

video_convert -i full_movie.vob  many_arguments -o full_movie.mkv

Because these files are all on networked drives, the network traffic is the big bottleneck. I'd like somehow to avoid the intermediate step of creating and afterwards destroying all the temporary full_movie.vob files (even if they're stored locally on my laptop SSD, which I don't want to wear out by writing >100TB to it).

Is there any way to take the output of the cat *.vob command and keep it in RAM (vs stored as a file) and somehow shove that into the video_convert program argument? If I have to put the temporary file on the server then I'll have doubled the network traffic and time.

This is likely a square peg in a round hole thing...

muru
  • 72,889
MickM
  • 1
  • Process substitution (e.g., https://unix.stackexchange.com/a/64011/70524) could help. – muru Jul 06 '21 at 03:52

1 Answers1

0

Does your video_convert program have an option to process stdin rather than a file? If so, then just pipe the output of cat *.vob into it. BTW, many programs allow you to use - as the filename to indicate stdin, or default to using stdin if you don't provide a filename. Or you can use /dev/stdin. e.g.

cat *.vob | video_convert many_arguments -o full_movie.mkv
cat *.vob | video_convert -i -  many_arguments -o full_movie.mkv
cat *.vob | video_convert -i /dev/stdin  many_arguments -o full_movie.mkv

Note that using stdin will not work if video_convert needs to seek to specific locations within the input. It will only work if it can process a continuous stream of data. Same applies to using Process Subsitution or Named Pipes aka fifos as mentioned in @muru's comments - they aren't seekable either.

cas
  • 78,579
  • Sadly, the video_convert program needs to interrogate aspects of the full_movie.vob file before converting so it looks like piping is out.

    Is there perhaps a way to set up a RAM disk on the local computer and write the temporary file to that instead? That may be an OS specific thing, in which case I should add that I'm on macOS 11.4 (Big Sur).

    Is there a way to direct the cat command output to a variable and then trick the video_convert program to use that instead?

    Just fishing here...

     – MickM Jul 06 '21 at 11:08
  • Hmmm - I just read elsewhere:

    /tmp is configure in most distros to use a tmpfs filesystem which is in RAM. When you write a file in /tmp it goes directly to your RAM which makes is a good answer for semi-resilient files that have to be accessed quickly and rewritten many times.

    How can I definitively confirm that this is a RAM "disk" vs something using my regular SSD drive?

     – MickM Jul 06 '21 at 11:32
  • Hey cas - I down voted your answer because, although useful, it didn't solve my issue. I'm new to this forum and I noticed my added comments above didn't return the topic back to the top of the list so I'm trying to change it to 0 Answers vs 1 Answer to see if that helps. – MickM Jul 06 '21 at 11:57