22

Is there a multi-threaded cp command on Linux?

I know how to do this on Windows, but I don't know how this is approached in a Linux environment.

Braiam
  • 35,991
leeand00
  • 4,615

3 Answers3

35

As Celada mentioned, there would be no point to using multiple threads of execution since a copy operation doesn't really use the cpu. As ryekayo mentioned, you can run multiple instances of cp so that you end up with multiple concurrent IO streams, but even this is typically counter-productive. If you are copying files from one location to another on the same disk, trying to do more than one at a time will result in the disk wasting time seeking back and forth between each file, which will slow things down. The only time it is really beneficial to copy multiple files at once is if you are, for instance, copying several files from several different slow, removable disks onto your fast hard disk, or vice versa.

psusi
  • 17,303
  • 3
    Disk seeks are only relevant for non-SSD disks. – Thorbjørn Ravn Andersen Oct 31 '14 at 19:34
  • 2
    @ThorbjørnRavnAndersen: but random seeks tend to interact badly with caches (in particular, read-ahead caches) for any kind of medium. – Matteo Italia Oct 31 '14 at 23:59
  • @ThorbjørnRavnAndersen, the severe penalty for seeks on HDD is almost none on SSD, yet the point remains that there is no benefit to trying to read or write to/from multiple parts of a single disk at the same time. – psusi Nov 01 '14 at 02:18
  • @MatteoItalia if the IO-channel is saturated there is nothing caches can improve. – Thorbjørn Ravn Andersen Nov 01 '14 at 09:49
  • @psusi The argument was that the disk wasted time seeking, not that the disk could not serve data any faster. – Thorbjørn Ravn Andersen Nov 01 '14 at 09:51
  • 1
    Not all setups are one SSD-disk. For example, right now I am waiting on a 1 hour copy with AWS EFS, which uses multiple disks and has high latency. – Paul Draper Dec 29 '16 at 03:39
  • 1
    Not true, the reading from the source and the writing to the destination should be done in parallel, but it isn't so. Yes, the gnu fileutils requires a few tuning. There are other, not so common cases as well, as parallel copy would be profitable, for example on network drives or on raid/lvm. – peterh Feb 23 '17 at 23:45
  • @peterh, it isn't good on raid either for the same reason it isn't on a single drive: you're just making multiple drives seek their heads back and forth. Network drives are not going to benefit either unless your drive and network connection are both faster than at least one of the server drives and this isn't likely to be the case. – psusi Feb 24 '17 at 00:01
  • @psusi There are many raid personalities, for example in linearly ordered raid devices it is not an issue, furthermore it is also not an issue if the raid has a bigger block structure as the blocks of the worker threads of the cp tools. Furthermore, the actual disk access block order is controlled not by the cp, but by the disk layer (ok, our most pathologic ext4 writes out the write cache in every 5 seconds by default setting...). In the case of the reading is there the trick of the readahead, although it is far not so effective. – peterh Feb 24 '17 at 08:52
  • @psusi But, the focus of my comment was this: the current copy tool 1. reads a block from the input, 2. THEN writes this block to the output. While it reads, the write operation stalls, and while it writes, the read operation stalls. At least its reading and writing should be done on two different threads, this is my point. – peterh Feb 24 '17 at 08:54
  • @peterh, cp can do one byte at a time and it doesn't matter ( much ). Read ahead and write behind make sure that the individual read and write calls do not stall, at least until there is plenty of data in the write behind cache to keep the disk(s) busy, at which point the kernel starts letting the write calls block for a bit to avoid filling all of ram with dirty pages. – psusi Feb 25 '17 at 01:34
  • 3
    on amazon you can get up to 10 times the sequential read speed when using multi threaded access to the same seqeuential file. So the answer isn'T really accurate anymore – John Jan 21 '19 at 20:33
  • I develop recording software that runs on embedded Linux devices. I need to archive my internal media to external thumb drives. I used to use .NET, and having read/write threads increased performance by around %40. I'd like a similar approach, but with cp/native. – Paul Knopf Feb 21 '19 at 13:12
  • This answer is pretty misinformative; there are many cases where parallel copy have different aggregate performance / timings - often drastically so (easily 10x throughput) - nas/raids are very common, as are pcie based memory devices are just a few environments I've observed this. Sometimes for one reason or another this is true of any tech with sockets in the loop as well. It also does not suggest an original solution to the problem; just says try what others suggest even though it'd be counterproductive. – Jason Newton Feb 10 '22 at 00:03
  • Normal raids won't benefit from it either. I suppose if you are using JBOD/linear mode and get lucky and happen to have some of the files on one underlying disk, and some files on another, then you might see some improvement, but that's unusual and unlikely. – psusi Feb 24 '22 at 19:31
  • People routinely mount cloud storage locally, because VFS/fuse rules, and "dedicated" cloud storage utilities suck (e.g. can't handle funny filenames). But fuse drivers will typically go through a full handshake with the cloud on every fopen(), and without parallelization 90% of batch start-to-end time is spent in the handshakes. Parallelizing cp parallelizes those. It's not about data transfer AT ALL. – Szczepan Hołyszewski Jul 26 '22 at 11:23
19

Well, I believe you could use gnu parallel to accomplish your task. 

 seq 70 | parallel -j70 cp filename

You could see a detailed explanation on using gnu parallel from my other answer here.

I just tested the above command in my system and I could see that 70 copies of files are being made.

Community
  • 1
Ramesh
  • 39,297
  • Even easier way than what I got. – ryekayo Oct 31 '14 at 15:15
  • 3
    And that copies filename 70 times? – Szczepan Hołyszewski Dec 28 '21 at 08:56
  • This is the way, especially when copying large/wide file trees (cp -R). I'm copying many, many files between two disks, and this allows my disk read go from 15 MB/s to 150 MB/s. For example, ls lotsofdirs | parallel -j32 --progress cp -R lotsofdirs/{} /mnt/somedisk/lotsofdirs/{} (make sure to mkdir -p /mnt/somedisk/lotsofdirs first) – allidoiswin Jan 02 '23 at 12:49
-1

The closest thing to a multithreaded process is the & which runs commands in the background.

So to use this command, you would do something like:

cp file location &
terdon
  • 242,166
ryekayo
  • 4,763
  • I want to be able to specify the number of parallel threads used. – leeand00 Oct 31 '14 at 15:09
  • You can probably stick this in a function and take in a parameter to accept ints to specify the number of times you want the command run..That'll be a bit of coding on your end though – ryekayo Oct 31 '14 at 15:10
  • 3
    Running a command in the background has nothing to do, at all, with multithreading. – joker Oct 11 '19 at 20:26