11

I'm currently having trouble with dd invoked with a sparse file as input (if) and a file as output (of) with conv=sparse. dd seems to be using one core of the CPU (Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz 4 cores + 4 Intel Hyperthreads) only (100 % of 1 core), so I've been wondering whether it's possible to parallelize dd. I've been

  • looking into info dd and man dd and there seems to built-in function in the version of corutils 8.23
  • checking sgp_dd from sg3-utils package (without understanding whether it suits my needs), but it doesn't seem to be able to handle sparse files
  • dcfldd doesn't seems to have parallelization capabilities

AFAIK

  • an enhanced version/fork with internal handling of program parts in multiple threads (avoid context changes killing I/O performance) is preferred over
  • a solution with GNU parallel running locally is preferred over
  • a custom (possibly untested) code sniplet

How to avoid CPU being the bottleneck of an I/O intensive operation? I'd like to run the command on Ubuntu 14.04 with Linux 3.13 and handle sparse file disk images with it on any filesystem supporting sparse file (at least the solution shouldn't be bound to one specific file system).

Background: I'm trying to create a copy of 11TB sparse file (containing about 2TB data) on a zfs (zfsonlinux 0.6.4 unstable version, possibly buggy and the cause for the CPU bottleneck (eventually slow hole search)). That shouldn't change anything for the question of how to parallelize dd (in a very generic way).

Kalle Richter
  • 2,192
  • I don't see what you could gain from this, as this operation is I/O bound except in extreme cases. In my opinion the best option would be a program that's sparse aware, e.g something like xfs_copy. Its man page mentions: "However, if the file is created on an XFS filesystem, the file consumes roughly the amount of space actually used in the source filesystem by the filesystem and the XFS log. The space saving is because xfs_copy seeks over free blocks instead of copying them and the XFS filesystem supports sparse files efficiently.". – Cristian Ciupitu Oct 10 '14 at 20:40
  • @mikeserv I don't understand your comment... – Kalle Richter Oct 10 '14 at 20:42
  • @CristianCiupitu Well in my case the CPU is the bottleneck - don't ask me why, because I don't know. Your answer made me realize that the solution should support multiple filesystems (able to handle sparse files) (edited) – Kalle Richter Oct 10 '14 at 20:44
  • What CPU and filesystem do you have? How big is the file (length & blocks)? – Cristian Ciupitu Oct 10 '14 at 20:47
  • 4
    dd hogs the CPU by default due to small blocksize. make it larger, like bs=1M. – frostschutz Oct 10 '14 at 23:49
  • @CristianCiupitu CPU-bound filesystem copy isn't as rare as you suggest. Besides faster SSDs, the file might actually be in the system's disk cache. I'm looking at a cp task taking 100% CPU on my server right now, I think because I have a 20GiB file that Ubuntu's disk caching seems to have loaded into RAM. – sudo Sep 08 '17 at 02:05

2 Answers2

7

Tested in Bash:

INFILE=in
seq 0 1000 $((`stat --format %s $INFILE` /100000 )) |
  parallel -k dd if=$INFILE bs=100000 skip={} conv=sparse seek={} count=1000 of=out

You probably need to adjust 1000.

Ole Tange
  • 35,514
3

One custom, untested code sniplet coming up:

dd if=oldf conv=sparse bs=1k                 count=3000000000                 of=newf &
dd if=oldf conv=sparse bs=1k skip=3000000000 count=3000000000 seek=3000000000 of=newf &
dd if=oldf conv=sparse bs=1k skip=6000000000 count=3000000000 seek=6000000000 of=newf &
dd if=oldf conv=sparse bs=1k skip=9000000000 count=3000000000 seek=9000000000 of=newf &
wait

This should logically partition the file into four 3TB chunks and process them in parallel.  (skip= skips over input blocks; seek= seeks over output blocks.)  The fourth command will, of course, read up to the end of the old file, so the count= parameter isn't strictly necessary.

G-Man Says 'Reinstate Monica'
  • 22,870