Can we merge binary files without any copy operation by handling filesystem?

Question

I would like to merge efficiently binary files using a shell command and I quickly found classical ways like this one :

cat file1 file2 > file3

Not bad but :

It's slow. IO access are slow.
It needs extra space. I don't want to copy the files. Just concatenate them.

File systems are great to handle file fragmentation. Can't we just merge files using this mecanism?

Related: A virtual file containing the concatenation of other files — Stéphane Chazelas, Feb 19 '16 at 14:08
I agree. And bordline duplicat with this one too : link. But this is not exactly the same. My challenge is "without any copy" by handling filesystem. — zz64, Feb 19 '16 at 14:36

score 11 · Answer 1 · answered Feb 19 '16 at 13:48

11

You can do this:

cat file2 file3 [...] filen >> file1

This will concatenate file2, file3,..., filen to the end of file1 in-place. The >> operator tells the shell to write to the end of the file.

You want to leverage the file system to "handle file fragmentation". Unfortunately, there isn't any general way to do this. It's because "file systems" are much more general than file systems on disks — for example, you have NFS, FUSE, and many other mechanisms which let you expose any kind of resource (not just block devices such as hard disks) as a file system hierarchy. Even for block-device-based file systems, there isn't a standard mechanism to do this, and I don't know any implementation-specific ones either.

answered Feb 19 '16 at 13:48

ThePiercingPrince

476

Thanks for your answer. I'm investigating virtual file systems! – zz64 Feb 19 '16 at 14:44
The only "easy" solution I have found to my challenge is a virtual file that encapsulate my files. I can't do an operation on my hard disk in the name abstraction. So I put even more abstraction to manage this ! :P Ironic isn't it ? – zz64 Feb 19 '16 at 16:00
The first part of your answer is about copying files. I suggest to remove it and to keep only the last part related to manipulating the file system. It would then be my best answer! – zz64 Feb 22 '16 at 10:28
that solution swaps bytes (little big endian) for me... – orange Jul 21 '20 at 06:39

elig · Answer 2 · 2021-06-22T19:26:01.027

1

Put your filenames in order in a text file called list.txt separated by new lines. Then run this in bash:

while read line; do echo -n . ; dd if="$line" of=out status=none conv=notrunc oflag=append; done < list.txt

This will create the concatenated file 'out' in the current directory.

edited Jun 22 '21 at 19:26

answered May 26 '20 at 01:11

elig

560

2

Great script but there is a typo, instead of do, it should have been dd
while read line; do echo -n . ; dd if="$line" of=out status=none conv=notrunc oflag=append; done < list.txt
– Jitesh Pramodray Jun 19 '21 at 23:02
@BirdOfPrey Thank you I fixed it – elig Jul 02 '21 at 16:39

score -1 · Answer 3 · answered Mar 20 '18 at 19:00

-1

    dd if=firstfile.raw > completedfile.raw
    dd if=nfile.raw    >> completedfile.raw
    dd if=lastfile.raw >> completedfile.raw

answered Mar 20 '18 at 19:00

Steve

19
1

2

Uhhh, ooooh, what an epic explanation... – Mar 20 '18 at 19:42
This worked for me. – Scottie H May 01 '23 at 22:41

Can we merge binary files without any copy operation by handling filesystem?

3 Answers3