200

When moving large directories using mv, is there a way to view the progress (%)? The cp command on gentoo had a -g switch that showed the progress.

Lesmana
  • 27,439
letronje
  • 2,195

14 Answers14

191

There's a new tool called progress that can find any descriptor related to a running command and show progress and speed: available here

progress -w

outputs the stats for all running cp,mv etc. operations

Eric Renouf
  • 18,431
naugtur
  • 2,403
  • 11
    WOW, I did not expect this to be so awesome! I simply typed progress after installing and it presented me with progress for the mv command I was currently waiting for. Thanks! – jredd Sep 30 '15 at 00:59
  • 5
    Thanks! When moving stuff as a root it seems you have to use sudo (same user I guess) - you can also use watch to say what is happening every 2 seconds: watch progress -w – Wilf May 30 '16 at 20:13
  • 22
    Awesome program, but when moving e.g. a folder with many files it only shows the progress of the current file, not the whole folder in total. Instead of watch progress -w you can also simply use progress -m – mxmlnkn Nov 13 '16 at 15:38
  • 11
    IMHO, this is the most unix-way to solve the problem. Instead of patching coreutils or using tools which are not designated for specific purposes (like rsync for mv) or writing overkilling scripts, just use that tool written for the subject purpose with love. – Grief Dec 28 '16 at 01:54
  • The best solution, intuitive. When using rsync I always catch myself on checking man rsync.. – Konrad Sep 04 '20 at 20:20
107

I don't like the idea to overwrite binaries from coreutil when there are simpler solutions, so here are mine:

rsync: Rsync copies files and has a -P switch for a progress bar. So if you have rsync installed, you could use a simple alias in your shells dotfile:

alias cp='rsync -aP'

The downside is, that rsync is a little bit slower than cp, but you should measure this with time and decide for your self, I can live with it :-)

Shell Script: A shell script can also create the progress bar. I found this a while ago on the net and I don't remember the source:

#!/bin/sh
cp_p()
{
   strace -q -ewrite cp -- "${1}" "${2}" 2>&1 \
      | awk '{
        count += $NF
            if (count % 10 == 0) {
               percent = count / total_size * 100
               printf "%3d%% [", percent
               for (i=0;i<=percent;i++)
                  printf "="
               printf ">"
               for (i=percent;i<100;i++)
                  printf " "
               printf "]\r"
            }
         }
         END { print "" }' total_size=$(stat -c '%s' "${1}") count=0
}

This will look like:

% cp_p /home/echox/foo.dat /home/echox/bar.dat
66% [===============================>                      ]

bar:

‘bar’ - ‘cat’ with ASCII progress bar

bar is a small shell script to display a process bar for all kind of operations (cp, tar, etc.). You can find examples on the project homepage.

Its also written for the bourne shell, so it will run nearby everywhere.

echox
  • 18,103
  • 4
    bar doesn't wrap mv yet. – Tobu Jan 02 '11 at 23:04
  • 8
    Credit for the shell script – Lamnk Jun 26 '11 at 16:42
  • 2
    Note that rsync doesn't replace mv (the OP is asking about) completely, especially when it comes to fast copies on copy-on-write filesystems like btrfs. – Kalle Richter May 29 '16 at 14:40
  • Does it work for mv ? – alper Dec 01 '20 at 11:20
  • I have "nmv" aliased locally to rsync -a --stats --progress --remove-source-files. There are different flags you may want to set to control local caching, intermediate compression, etc., depending on your setup. – djvs Oct 30 '22 at 04:13
  • I support your answer, but I'm wondering if aliasing something as fundamental as 'cp' could have some extremely rare unintended consequence down the road. Like I see these people trying to "unshadow" aliases https://unix.stackexchange.com/a/39357/8234 and I'm wondering if I use your solution, will other applications mistakenly use rsync instead of cp? – PJ Brunet Mar 18 '23 at 18:42
59

You can build a patched cp and mv which then both support the -g switch to show progress. There are instructions and patches at this page. However: The page instructs you to do

$ sudo cp src/cp /usr/bin/cp
$ sudo cp src/mv /usr/bin/mv

which overwrites the original cp and mv. This has two disadvantages: Firstly, if an updated coreutils package arrives at your system, they are overwritten. Secondly, if the patched version has a problem, they might break scripts relying on standard cp and mv. I would rather do something like this:

$ sudo cp src/cp /usr/local/bin/cpg
$ sudo cp src/mv /usr/local/bin/mvg

which copies the files to /usr/local/bin which is intended for user compiled programs and gives them a different name. So when you want a progress bar, you say mvg -g bigfile /mnt/backup and use mv normally.

Also you can do alias mvg="/usr/local/mvg -g" then you only need to say mvg bigfile /mnt/backup and directly get the progress bar.

fschmitt
  • 8,790
46

You can use pipe viewer command pv to show progress bar:

pv /original/file > /new/file

I often use this to copy a big file over a mounted network filesystem (combine with gzip and tar). The drawback is that you can only copy one file and not directory. And you must give the new file a name, you can not just give destination directory like cp does. However copying is not pv's purpose. It is a powerful tool and do much more than just copy file. See the homepage for more examples of pv.

A much better option is to use rsync -aP. If you want to mv instead, append the flag --remove-source-files. Add this to your .bashrc if you want to use the commands frequently:

alias rscp='rsync -aP'
alias rsmv='rsync -aP --remove-source-files'

The downside here is rsync only shows progress, not a progress bar.

Lamnk
  • 1,510
  • 1
  • 11
  • 10
  • 6
    tar cf - source-dir | pv | tar xf - -C out-dir will give you throughput of a whole directory. If you know the size and pass -s to pv it can give you progress as well. – arantius Feb 10 '18 at 17:34
10

If your goal is to move/copy a directory with progress bar, but avoiding non-terminal GUI, mc (Midnight Commander) is a good choice.

8

My solution is to use rsync. It can copy directories, remove the source files after a successful copy (thus "moving" them) and display progress reports, among many other features (most notably syncing partially copied directories and working over the network.)

Here is how I would move a directory with rsync, with progress report:

rsync -aP --remove-source-files $src $dst && rm -r $src

Make sure $src does not end in a backslash, otherwise you will get a different meaning than that of GNU cp.

The rm -r at the end is needed because --remove-source-files only removes the source files, after each one is successfully copied over, not the directories. The && makes sure to run it only if rsync completes successfully. If you are paranoid, you can replace it with some kind of recursive rmdir, such as find $src -d -exec rmdir {} +

The only glitch is that the -P option shows progress for each file, not for the entire copy operation. This might be useless if you are trying to move a directory with many tiny files. The latest development version of rsync has an alternate flag --info=progress2 that shows progress for the entire operation.

Tobia
  • 669
4

This does not give you a progress bar, but hit CTRL-T and you will send a SIGINFO to mv, which will report which file it's working on and how far along in the process it is. Pretty handy to have this for large file transfers, possibly not as useful for a transfer with many small files.

Quick example:

mv 135/61 /Volumes/Jules/135/
# hit C-t
load: 1.08  cmd: cp 35470 uninterruptible 0.00u 0.04s
135/61/1/1-s2.0-S006738-main.pdf -> /Volumes/Jules/135/61/1/1-s2.0-S006738-main.pdf  68%
Charles
  • 80
4
mv -v sourcedir targetdir | pv -l -s filecount > logfile

You will need pv (pipe viewer): http://www.ivarch.com/programs/pv.shtml

mv -v will print one line per moved file and pv -l will report progress based on line count.

Use the following command to get file count:

find sourcedir | wc -l

This command will show progress information based on number of files copied. This works best if there are many smallish files. If there are only a few files which are huge then you will not have much fun.


For more alternatives, including methods which can report progress based on size, see here: progress information via pv for directory copy

Methods described there all copy the files instead of moving. but unless you are moving on the same filesystem a move is practically a copy followed by a delete.

Lesmana
  • 27,439
3

Invariably I end up searching for this answer after I start the 'mv' command, at which point I don't want to start it again. If you're in this situation a very simple way to monitor the progress (or to see if it's far enough along that it's not worth killing) is to run:

du -hs /directory/you're/moving/data/to

This will just print the size of the directory so you can see how much data has been copied, or you can run it a couple of times to see how fast the data is being transfered.

qazmonk
  • 31
  • if you're moving a lot of data the du command itself will also take quite some time to compute the size at dest – axolotl Apr 15 '22 at 22:28
2

To avoid coreutil patching to add progress bar to mv and cp, gcp (Goffi's CoPier) is a new alternative: http://wiki.goffi.org/wiki/Gcp/en

To use the same way as cp with options -r for recursivity and -v for verbosity.

It uses a file queue when multiple commands are invoked concurrently. It also allows to record copied file names in a log, see Source saving man page section for details.

Here is a good tutorial to get started with gcp: http://mylinuxbook.com/gcp-advanced-command-line-file-copier-inspired-by-cp/

  • Is there gmv like gcp for cp? – CodyChan Sep 01 '16 at 06:07
  • I have found no equivalent. By the way, there is no need for progress bar when running "mv" on a single file system... it is quite fast. If move is done across two different file systems, it is implemented as cp and then remove source if succeed, or remove target if failed. A simple shell script based on gcp may do it properly (with a first check to know if source and target are really on different file systems) – Yves Martin Sep 01 '16 at 06:23
2

On my Mint I like to use gcp on this or the curl-solution (which is really nice)

gcp SOURCE DESTINATION

You might need to install it sudo apt-get install gcp

And a very nice and excellent task for the progress bar with info about

  • % Total
  • % Received
  • % Xferd
  • Average DL
  • Speed Upload
  • Time left
  • Time spent
  • Time total
  • Current Speed

in a nice readable format is:

curl -o DESTINATION FILE://source

Also, you might need to install it first sudo apt-get install curl

2

I use the following script saved with rsmv name:

#!/usr/bin/env bash

rsync -aP --remove-source-files "$@" && rm -rf "${@:1:$#-1}"

It handles more then one source and filenames with whitespaces:

$ rsmv source1 source2 source\ 3 "source 4" dest
mixel
  • 121
  • Doesn't --remove-source-files accomplish the removal? Why is the && rm ... there? – Jeff Schaller Dec 17 '19 at 12:59
  • This is otherwise very similar to existing answers: https://unix.stackexchange.com/a/184943/117549 and https://unix.stackexchange.com/a/15656/117549 – Jeff Schaller Dec 17 '19 at 13:00
  • --remove-source-files removes only files. So if source arguments contains directories then they will not be deleted. rm ... handles deletion of such directories. – mixel Dec 17 '19 at 13:03
  • Yep, it's improved version that is closer to mv, because it handles many source arguments. – mixel Dec 17 '19 at 13:04
  • Be very very careful if you have symbolic links and are moving an entire dir transfer (i.e. rsync --remove-source-files A/* B/). If B/ is a symlink to A/ you will lose everything in A/ whereas with mv you would not. It happens to me once every 12 months or so and I keep having to use extundelete to undo the mistake. – Sridhar Sarnobat Aug 24 '22 at 21:36
1

First off: I never copy large files without using ionice, unless I know that I will not want to use the computer for half an hour or more.

Second: all my partitions are jouranled so intrapartition copying takes not time. If it is a long copy I do a du -sm on the files and df -m | grep copy_to_partition. Then if curious how much more time it will take I do the df again and see how much of the files was copied.

muru
  • 72,889
HandyGandy
  • 2,209
  • 2
    Would you care to share your ionice tips ? what priority would you give to mv cpu-wise and disk i/o wise ? Here are my settings but the CPU load is still high even after renicing : https://imgur.com/a/DLMJplz – ychaouche Feb 17 '19 at 11:55
1

On mac you can open Activity Monitor and go to the Disk tab, look for the process 'mv' in the list and look at 'Bytes Written'. That shows how far through the transfer is.

Not an exact solution but it does put your mind at ease to know your transfer is working and you can see the approximate speed.

83euh
  • 11
  • 1