21

I have a 1 TB file. I would like to read from byte 12345678901 to byte 19876543212 and put that on standard output on a machine with 100 MB RAM.

I can easily write a perl script that does this. sysread delivers 700 MB/s (which is fine), but syswrite only delivers 30 MB/s. I would like something more efficient, preferably something that is installed every Unix system and that can deliver in the order of 1 GB/s.

My first idea is:

dd if=1tb skip=12345678901 bs=1 count=$((19876543212-12345678901))

But that is not efficient.

Edit:

I have no idea how I measured syswrite wrong. This delivers 3.5 GB/s:

perl -e 'sysseek(STDIN,shift,0) || die; $left = shift; \
         while($read = sysread(STDIN,$buf, ($left > 32768 ? 32768 : $left))){ \
            $left -= $read; syswrite(STDOUT,$buf);
         }' 12345678901 $((19876543212-12345678901)) < bigfile

and avoids the yes | dd bs=1024k count=10 | wc nightmare.

Ole Tange
  • 35,514

4 Answers4

23

This is slow because of the small block size. Using a recent GNU dd (coreutils v8.16 +), the simplest way is to use the skip_bytes and count_bytes options:

in_file=1tb

start=12345678901
end=19876543212
block_size=4096


copy_size=$(( $end - $start ))


dd if="$in_file" iflag=skip_bytes,count_bytes,fullblock bs="$block_size" 

  skip="$start" count="$copy_size"

Update

fullblock option added above as per @Gilles answer. At first I thought that it might be implied by count_bytes, but this is not the case.

The issues mentioned are a potential problem below, if dds read/write calls are interrupted for any reason then data will be lost. This is not likely in most cases (odds are reduced somewhat since we are reading from a file and not a pipe).

Using a dd without the skip_bytes and count_bytes options is more difficult:

in_file=1tb

start=12345678901
end=19876543212
block_size=4096


copy_full_size=$(( $end - $start ))
copy1_size=$(( $block_size - ($start % $block_size) ))
copy2_start=$(( $start + $copy1_size ))
copy2_skip=$(( $copy2_start / $block_size ))
copy2_blocks=$(( ($end - $copy2_start) / $block_size ))
copy3_start=$(( ($copy2_skip + $copy2_blocks) * $block_size ))
copy3_size=$(( $end - $copy3_start ))


{
  dd if="$in_file" bs=1 skip="$start" count="$copy1_size"
  dd if="$in_file" bs="$block_size" skip="$copy2_skip" count="$copy2_blocks"
  dd if="$in_file" bs=1 skip="$copy3_start" count="$copy3_size"
}

You could also experiment with different block sizes, but the gains won't be very dramatic. See - Is there a way to determine the optimal value for the bs parameter to dd?

Community
  • 1
Graeme
  • 34,027
8

bs=1 tells dd to read and write one byte at a time. There is an overhead for each read and write call, which makes this slow. Use a larger block size for decent performance.

When you copy a whole file, at least under Linux, I've found that cp and cat are faster than dd, even if you specify a large block size.

To copy only part of a file, you can pipe tail into head. This requires GNU coreutils or some other implementation that has head -c to copy a specified number of bytes (tail -c is in POSIX but head -c isn't). A quick benchmark on Linux shows this to be slower than dd, presumably because of the pipe.

tail -c $((2345678901+1)) | head -c $((19876543212-2345678901))

The problem with dd is that it is not reliable: it can copy partial data. As far as I know, dd is safe when reading and writing to a regular file — see When is dd suitable for copying data? (or, when are read() and write() partial) — but only as long as it isn't interrupted by a signal. With GNU coreutils, you can use the fullblock flag, but this is not portable.

Another problem with dd is that it can be hard to find a block count that works, because both the number of skipped bytes and the number of transferred bytes count need to be a multiple of the block size. You can use multiple calls to dd: one to copy the first partial block, one to copy the bulk of aligned blocks and one to copy the last partial block — see Graeme's answer for a shell snippet. But don't forget that you when you run the script, unless you're using the fullblock flag, you need to pray that dd will copy all the data. dd returns a nonzero status if a copy is partial, so it's easy to detect the error, but there's no practical way to repair it.

POSIX has nothing better to offer at the shell level. My advice would be to write a small special-purpose C program (depending on exactly what you implement, you can call it dd_done_right or tail_head or mini-busybox).

Community
  • 1
Gilles 'SO- stop being evil'
  • 829,060
4

With dd:

dd if=1tb skip=12345678901 count=$((19876543212-12345678901)) bs=1M iflags=skip_bytes,count_bytes

Alternatively with losetup:

losetup --find --show --offset 12345678901 --sizelimit $((19876543212-12345678901))

And then dd, cat, ... the loop device.

frostschutz
  • 48,978
0

This is how you do you can do this:

   i=$(((t=19876543212)-(h=12345678901)))
   { dd count=0 skip=1 bs="$h"
     dd count="$((i/(b=64*1024)-1))" bs="$b"
     dd count=1 bs="$((i%b))"
   } <infile >outfile

That's all that is really necessary - it doesn't require much more. In the first place dd count=0 skip=1 bs=$block_size1 will lseek() over regular file input practically instantaneously. There is no chance of missed data or whatever other untruths are told about it, you can just seek directly to your desired start position. Because the file descriptor is owned by the shell and the dd's are merely inheriting it, they will affect its cursor position and so you can just take it in steps. It really is very simple - and there is no standard tool better suited to the task than dd.

That uses a 64k blocksize which is often ideal. Contrary to popular belief, larger blocksizes do not make dd work faster. On the other hand, tiny buffers are no good either. dd needs to synchronize its time in system calls so that it need not wait on copying data into memory and out again, but also so that it need not wait on system calls. So you want it to take enough time that the next read() doesn't have to wait on the last, but not so much that you're buffering in larger sizes than is necessary.

So the first dd skips to the start position. That takes zero time. You could call any other program you liked at that point to read its stdin and it would begin reading directly at your desired byte offset. I call another dd to read ((interval / blocksize) -1) count blocks to stdout.

The last thing that is necessary is to copy out the modulus (if any) of the previous division operation. And that's that.

Don't believe it, by the way, when people state facts on their face without evidence. Yes, it is possible for dd to do a short read (though such things are not possible when reading from a healthy block device - thus the name). Such things are only possible if you do not correctly buffer a dd stream which is read from other than a block device. For example:

cat data | dd bs="$num"    ### incorrect
cat data | dd ibs="$PIPE_MAX" obs="$buf_size"   ### correct

In both cases dd copies all of the data. In the first case it is possible (though unlikely with cat) that some of the output blocks which dd copies out will bit equal "$num" bytes because dd is spec'd only to buffer anything at all when the buffer is specifically requested on its command-line. bs= represents a maximum block-size because the purpose of dd is real-time i/o.

In the second example I explicitly specify the output blocksize and dd buffers reads until complete writes can be made. That doesn't affect count= which is based on input blocks, but for that you just need another dd. Any misinformation which is given you otherwise should be disregarded.

mikeserv
  • 58,310