Best way to remove bytes from the start of a file?

Question

Today I had to remove the first 1131 bytes from an 800MB mixed text / binary file, a filtered subversion dump I'm hacking for a new repository. What's the best way to do this?

To start with I tried

dd bs=1 skip=1131 if=filtered.dump of=trimmed.dump

but after the skip this copies the remainder of the file a byte at a time, i.e. very slowly. In the end I worked out I needed 405 bytes to round this up to three blocks of 512 which I could skip

dd if=/dev/zero of=405zeros bs=1 count=405
cat 405zeros filtered.dump | dd bs=512 skip=3 of=trimmed.dump

which completed fairly quickly but there must have been a simpler / better way? Is there another tool I've forgotten about?

Answer 1

You can switch bs and skip options:

dd bs=1131 skip=1 if=filtered.dump of=trimmed.dump

This way the operation can benefit from a greater block.

Otherwise, you could try with tail (although it's not safe to use it with binary files):

tail -c +1132 filtered.dump >trimmed.dump

Finally, you may use 3 dd instances to write something like this:

dd if=filtered.dump bs=512k | { dd bs=1131 count=1 of=/dev/null; dd bs=512k of=trimmed.dump; }

where the first dd prints its standard output filtered.dump; the second one just reads 1131 bytes and throws them away; then, the last one reads from its standard input the remaining bytes of filtered.dump and write them to trimmed.dump.

Answer 2

Not sure when skip_bytes was added, but to skip the first 11 bytes you have:

# echo {123456789}-abcdefgh- | 
                              dd bs=4096 skip=11 iflag=skip_bytes
-abcdefgh-
0+1 records in
0+1 records out
11 bytes (11 B) copied, 6.963e-05 s, 158 kB/s

Where iflag=skip_bytes tells dd to interpret the value for the skip option as bytes instead of blocks, making it straightforward.

Answer 3

You can use a sub-shell and two dd calls like this:

$ ( dd bs=1131 count=1 of=dev_null && dd bs=4K of=out.mp3 ) < 100827_MR029_LobbyControl.mp3
1+0 records in
1+0 records out
1131 bytes (1.1 kB) copied, 7.9691e-05 s, 14.2 MB/s
22433+1 records in
22433+1 records out
91886130 bytes (92 MB) copied, 0.329823 s, 279 MB/s
$ ls -l *
-rw------- 1 max users 91887261 2011-02-03 22:59 100827_MR029_LobbyControl.mp3
-rw-r--r-- 1 max users     1131 2011-02-03 23:04 dev_null
-rw-r--r-- 1 max users 91886130 2011-02-03 23:04 out.mp3
$ cat dev_null out.mp3 > orig
$ cmp 100827_MR029_LobbyControl.mp3 orig

Answer 4

If the filesystem and the Linux kernel support it then you could try fallocate if you want to make the changes inplace: in the best case there is no data IO at all:

$ fallocate <magic> -o 0 -l 1131 inplace.dump

where <magic> depends on the filesystem, Linux version, and the file type (FALLOC_FL_COLLAPSE_RANGE or FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE could be used internally).

Answer 5

You should use count=0 - that's a simple lseek() whenever possible.

Like this:

{  dd bs=1131 skip=1 count=0; cat; } <filtered.dump >trimmed.dump

dd will lseek() the input file descriptor to an 1131 byte offset, and then cat will simply copy whatever remains to output.

Answer 6

Yet another way to remove leading bytes from a file (without using dd at all) is to use xxd and sed or tail respectively.

bytes=$((1131*2))

xxd -p -c 256 filtered.dump | tr -d '\n' | sed "s/^.\{0,${bytes}\}//" | xxd -r -p > trimmed.dump

bytes=$((bytes + 1)) 
xxd -p -c 256 filtered.dump | tr -d '\n' | tail -c +${bytes} | xxd -r -p > trimmed.dump

Answer 7

For GNU dd(coreutils) 9.1, you can directly use B suffix to skip bytes rather than blocks, which replaces the old way iflag=skip_bytes.

From the man page:

skip=N (or iseek=N) skip N ibs-sized input blocks

N and BYTES may be followed by the following multiplicative suffixes: c=1, w=2, b=512, kB=1000, K=1024, MB=1000*1000, M=1024*1024, xM=M, GB=1000*1000*1000, G=1024*1024*1024, and so on for T, P, E, Z, Y. Binary prefixes can be used, too: KiB=K, MiB=M, and so on. If N ends in B, it counts bytes not blocks.

Answer 8

@maxschlepzig asks for a online liner. Here is one in perl. It takes 2 argument: From byte and length. The input file must be given by '<' and the output will be on stdout:

perl -e 'sysseek(STDIN,shift,0) || die; $left = shift;
     while($read = sysread(STDIN,$buf, ($left > 32768 ? 32768 : $left))){
        $left -= $read; syswrite(STDOUT,$buf);
     }' 12345678901 19876543212 < bigfile > outfile

If length is larger than the file, the rest of the file will be copied.

On my system this delivers 3.5 GB/s.