Merging folders with mv?

Question

If I use mv to move a folder called "folder" to a directory that already contains "folder" will they merge or will it be replaced?

score 185 · Accepted Answer · edited Nov 18 '22 at 23:03

185

mv cannot merge or overwrite directories, it will fail with the message "mv: cannot move 'a' to 'b': Directory not empty", even when you're using the --force option.

You can work around this using other tools (like rsync, find, or even cp), but you need to carefully consider the implications:

rsync can merge the contents of one directory into another (ideally with the --remove-source-files¹ option to safely delete only those source files that were transferred successfully, and with the usual permission/ownership/time preservation option -a if you wish)
… but this is a full copy operation, and can therefore be very disk-intensive.
You can use find to sequentially recreate the source directory structure at the target, then individually move the actual files
… but this has to recurse through the source multiple times and can encounter race conditions (new directories being created at the source during the multi-step process)
cp can create hard links (simply put, additional pointers to the same existing file), which creates a result very similar to a merging mv (and is very IO-efficient since only pointers are created and no actual data has to be copied)
… but this again suffers from a possible race condition (new files at the source being deleted even though they weren't copied in the previous step)
You can combine rsync's --link-dest=DIR option (to create hardlinks instead of copying file contents, where possible) and --remove-source-files to get a semantic very similar to a regular mv.
For this, --link-dest needs to be given an absolute path to the source directory (or a relative path from the destination to the source).
… but this is using --link-dest in an unintended way (which may or may not cause complications), requires knowing (or determining) the absolute path to the source (as an argument to --link-dest), and again leaves an empty directory structure to be cleaned up as per ¹. (Note: This won't work anymore as of rsync version 3.2.6)

Which of these workarounds (if any) is appropriate will very much depend on your specific use case.
As always, think before you execute any of these commands, and have backups.

_{1: Note that rsync --remove-source-files won't delete any directories, so you will have to do something like find -depth -type d -empty -delete afterwards to get rid of the empty source directory tree.}

edited Nov 18 '22 at 23:03

Jody Frankowski

3

answered May 03 '14 at 18:47

n.st

8,128

1

It sounds like you've tried just one implementation of mv. This answer would be better with a broader truth. Linux, BSD and "real" Unix, or a reference from POSIX or SUS. – Warren Young May 03 '14 at 19:29
@WarrenYoung You're right, I only tried the mv implementation used by Debian - the emphasis being on tried, since the manpage doesn't mention this behavior... – n.st May 03 '14 at 23:47
45

The disadvantage of rsync is that it actually copies the data, rather than just changing the hard link, which is potentially resource intensive if you're dealing with a lot of data. – Jonathan Mayer Sep 13 '14 at 18:53
rsync can delete stuff with --delete. – ki9 Nov 09 '16 at 18:16
9

@Keith Note that --delete only deletes files in the destination directory that don't exist in the source directory. – n.st Nov 09 '16 at 18:18
2

@JonathanMayer rsync as several hard link related features. For example you can just preserve hard links with the -H function or you can hardlink files in the destination using --link-dest. See the man page before using them, though. – allo Apr 18 '19 at 08:17
2

@allo --link-dest is a great idea! Judging by the man page, it's not originally intended for operating on the source directory, but it worked flawlessly (and extremely efficiently) in a quick test. I'll recommend it for now, but I'll appreciate any feedback regarding complications. – n.st Apr 18 '19 at 20:32
@n.st I know it for creating incremental backups (look for "time maschine like backups"), in which you can use it to hardlink unchanged files from the previous backup to the new one. I am not sure, if it has any special intention or is just a feature that can be used in many different ways. – allo Apr 19 '19 at 21:04
1

If I'm not expecting duplicated files in the merge, it's useful to add --ignore-existing to the rsync call, ensuring that existing files will be left alone (and then the source directories won't be empty, so you'll be aware of the situation). – SpinUp __ A Davis Aug 26 '19 at 01:30
Another good option for rsync is --dry-run which lets you check what would happen without actually changing anything. – jeremy_rutman Apr 27 '21 at 04:43
@n.st Can you explain the cp race condition? Why would there be a race condition if you're only deleting after the hard link "copy" has been made already? – localhost Feb 15 '24 at 23:06
@localhost If new files are added at the source after you have started cp (or more specifically, after cp has listed the files it should copy), the subsequent rm will delete them even though they weren't copied. – n.st Feb 28 '24 at 20:09

fazie · Answer 2 · 2014-05-03T19:11:01.883

131

rsync -av /source/ /destination/
(after checking)
rm -rf /source/

edited May 03 '14 at 19:11

answered May 03 '14 at 18:58

fazie

2,417

Will this remove the source files like in the comment by n.st? – Dominique May 03 '14 at 19:01
6

No, I would prefer to make it in two steps for safety reasons. Merged and removed source is unreversible. Additon step in n.st anwer is also needed (to remove directories). – fazie May 03 '14 at 19:10
4

--remove-source-files has the advantage of only removing files that were transferred successfully, so you can use find to remove empty directories and will be left with everything that wasn't transferred without having to check rsyncs output. – n.st May 03 '14 at 23:51
8

But it is not moving actually - the speed impact is huge, if big files are involved. – Alex Mar 15 '16 at 15:12
But you cannot perform pure move and merging actually. – fazie Mar 15 '16 at 19:58

palswim · Answer 3 · 2015-02-12T07:31:32.357

109

You can use the -l option of the cp command, which creates hard links of files on the same filesystem instead of full-data copies. The following command copies the folder source/folder to a parent folder (destination) which already contains a directory with the name folder.

cp -rl source/folder destination
rm -r source/folder

You may also want to use the -P (--no-dereference - do not de-reference symbolic links) or -a (--archive - preserve all metadata, also includes -P option), depending on your needs.

edited Feb 12 '15 at 07:31

answered Dec 09 '14 at 21:32

palswim

5,167

Hard links for what excuse ? – rautamiekka Feb 13 '16 at 17:54
11

@rautamiekka: I assume you are asking the reason for using hard links. If you don't know what hard links are and why you should use them, then you probably shouldn't take this route. However, creating hard links does not do a full copy, so this operation would take orders of magnitude less time than a full copy. And, you would use hard links rather than soft links so that you can delete the source files and still have the correct data instead of pointers to invalid paths. And cp rather than rsync since every system has cp and everyone has familiarity with it. – palswim Feb 16 '16 at 18:24
18

This solution's brilliance may be looked over for not being the accepted answer. It is an elegant solution. You get the merge ability of cp with the operation time of mv. – theherk Dec 09 '16 at 14:41
1

I prefer this solution because there are very few things that can go wrong. – Bell Apr 08 '17 at 21:18
7

if you know that you don't need to move files that already exist on the destination you also want to add -n – ndemou May 08 '17 at 22:32
Although this is an elegant solution, it won't work across filesystems. – Ruslan Jun 27 '17 at 07:58
3

@Ruslan: This is true, but you can't move without a copy across filesystems with any method. Even mv /fs1/file /fs2/ (across filesystems) will perform a copy and then a delete. – palswim Jun 29 '17 at 19:37
2

Right, but while mv will work (provided the target dir doesn't yet exist) even if not "efficiently" or whatever you call it, cp -rl will fail. – Ruslan Jun 29 '17 at 19:47
Nice idea, but does not work on all common operating system/filesystems. On Ubuntu + LVM + XFS I get: cp -rl source/folder destination cp: cannot create hard link 'destination/folder/file1' to 'source/folder/file1': File exists
and the file file1 is left unchanged. cp -pr does work, but as you say, it does mean another copy of files is created which may be undesirable.
– Ján Lalinský May 24 '18 at 15:27
Note that the "-l" option is not in POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/cp.html – kelvin Sep 02 '19 at 20:49
Note that, depending on your OS, hard-linking may not work when you do not own the source file, regardless of permissions. cp --link will then give Operation not permitted. mv does not have this limitation. See https://unix.stackexchange.com/questions/209309/hard-link-permissions-behavior-different-between-centos-6-and-centos-7/209452#209452 and https://unix.stackexchange.com/questions/233275/hard-link-creation-permissions – Rainer Blome Oct 20 '20 at 18:07
Use gcp on macOS. – HappyFace Jan 06 '21 at 15:09
1

This even works on Windows (via wsl) when using NTFS! – Michel de Ruiter Feb 03 '21 at 16:06
1

Keep in mind that you will need the --force flag, if during the merge there are files that already exist in the destination (and clash with a files being copied), and you want them to be overwritten. (pretty obvious, but might save a few seconds) Also be careful of the "destination" path -- it should not include the name of the folder being copied, unless you want destination/my-folder/my-folder. – Venryx Sep 13 '21 at 19:52
@palswim cp's -a/--archive option is unnecessary here - the (one and only) inode contains the metadata, hence 2 hard links must already have the same metadata. HTH. – jaimet Oct 21 '23 at 10:46

Jonathan Mayer · Answer 4 · 2023-03-05T19:40:27.827

30

I'd recommend these four steps:

cd "${SOURCE}"; 
find . -type d -exec mkdir -p "${DEST}/\{}" \; 
find . -type f -exec mv \{} "${DEST}/\{}" \; 
find . -type d -empty -delete

or better yet, here's a script that implements semantics similar to mv:

#!/bin/bash
DEST="${@:${#@}}"
for SRC in "${@:1:$((${#@} -1))}"; do   (
    cd "$SRC";
    find . -type d -exec mkdir -p "${DEST}"/{} ;
    find . -type f -exec mv {} "${DEST}"/{} ;
    find . -type d -empty -delete
) done

The quotes around the SRC and DEST variables will preserve whitespace in path names.

edited Mar 05 '23 at 19:40

answered Sep 15 '14 at 05:33

Jonathan Mayer

401

Args are SOURCE, DEST – schuess Jan 04 '16 at 14:32
This looks pretty useful. I'm tempted to use it for cleaning up my hard drive. Can any other experts comment on it before I entrust a bunch of backups to this script? :-) – LarsH Jun 29 '16 at 03:16
BTW, if you were wanting to do the equivalent of rsync -u (only update if newer), mv (in some versions at least) can also take the -u option. However in that case you might want to delete non-empty source directories as well as empty ones, to cover the cases where files in the source tree are not newer. @schuess: It looks like there can be multiple SOURCE arguments, if you need that. – LarsH Jun 29 '16 at 03:26
1

This doesn't handle whitespace well. I gave it a try with some directories with spaces in them and ended up with an infinite series of nested directories. – rofer Sep 30 '17 at 16:01
1

This needs quotes around ${@:1:$((${#@} -1))} in order to handle whitespace properly. – DLAN Sep 18 '20 at 22:25
Nice and concise. If new directories and files can appear any time in $SRC, there's a race condition here. The mv phase will fail when it encounters a directory that did not exist when the mkdir phase ran. – Rainer Blome Oct 20 '20 at 17:56
You might want to add another line with -type l for symlinks. – logicor Jun 08 '21 at 01:13
This should be voted the best answer since it performs a REAL move so no copying (hard disk or network intensive operation) is needed. – Pielo Nov 11 '22 at 13:04
It also accepts multiple arguments, like src1 src2 src3 dest - it presumes the last argument is the destination. I saved this as merge in ~/bin and it just works. Thank you so much @jonathan-mayer – JBRWilkinson Feb 11 '23 at 15:15

score 16 · Answer 5 · edited Apr 26 '16 at 23:44

16

Here is a way that will merge the directories. It is much faster than rsync since it just renames the files instead of copying them and then deleting them.

cd source; find -type f -print0 | xargs -0 -n 1 -I {} mv '{}' 'dest/{}'

edited Apr 26 '16 at 23:44

mxmlnkn

556
5
11

answered Sep 05 '14 at 04:48

Jewel

171

That's interesting but only vaguely relevant to the topic and not even remotely what the user asked about. – Shadur-don't-feed-the-AI Sep 05 '14 at 05:14
20

Actually, Jewel's code does precisely what the user asked for, with the exception of creating missing directories. Perhaps you should look again? – Jonathan Mayer Sep 13 '14 at 18:44
4

I would add to use "-print0" in find and "-0" in xargs because there are files that have spaces in the names. Also, there is a small problem, if a name contains parenthesis they are not going to be moved. – markuz Aug 18 '15 at 16:33
3

This is much faster than rsync for a small number of files, but it forks a new process for every file, thus the performance is abysmal with a large number of small files. @palswim's answer does not suffer from this problem. – b0fh Mar 01 '16 at 14:43
2

The command will fail, if in dest is already a directory with the same name as in source. And the files will be moved to a dest, which is in source. The command does nothing more than mv source/* source/dest/. – ceving Aug 25 '16 at 13:38
@markuz, can you say why files with parentheses won't be moved? Will -print0 fix this? – Tom Hale Jan 15 '17 at 10:29
@TomHale I believe the problem is related to some sort of string expansion, although I had this issue with rsync and never tried with "mv" – markuz May 23 '17 at 15:19

score 14 · Answer 6 · edited Oct 21 '20 at 00:02

Use `mv` with `find`. You can do this in one pass.

cd "$SRC"
find -type d -exec mkdir -vp "$DST"/{} \; -or -exec mv -nv {} "$DST"/{} \;

… where $SRC and $DST are the source and destination directories, respectively.

Explanation

-type d tests if the item is a directory. If it is a directory, we proceed to the next action or test: -exec ….
In -exec … {} \;, the {} is replaced with the path to the current item, relative to the current working directory. The \; indicates the end of this -exec … command.
In mkdir -pv …, -pv is equivalent to -p -v. The -p means to create all intermediate directories, as needed, and not raise an error if the directory already exists. The -v means --verbose and just tells it to print a message for each directory created, so you can see what it is doing. "$DST"/{} will be expanded to the destination directory, including all needed quotes.
The -or is the interesting part, which allows us to do this in one pass. With the find command, every test (e.g., -type d) or action (e.g., -exec …) result in a status of true or false, depending on if the test passed or action succeeded. Tests and actions can be connected using -and, -or, -not, -true, -false, and $ … $. When you add multiple tests and/or actions without an explicit boolean operator, they are implicitly AND'd together. Thus, the above command is equivalent to this: find $ -type d -and -exec mkdir -vp "$DST"/{} \; $ -or -exec mv -nv {} "$DST"/{} \;. Thus, if -type d passes, then it goes on to the next action (-exec …). If not, then that first branch of the -or is false, and it goes to the second branch, which covers anything that is not a directory (e.g., files).
In mv -nv {} "$DST"/{}, -nv is equivalent to -n -v. The -n tells it to not overwrite any files in the destination directory.. The -v tells it to report a message for every file moved, so you can see what it is doing.
Directories will be created before their files are moved. find uses breadth-first traversal by default.
The {} does NOT need to be enclosed in quotes, even if the item it stands for includes spaces.
Empty directories at the source will remain.

Example

If you wanted to copy /usr/local into /usr, you could enter it like this.

cd /usr/local
find -type d -exec mkdir -vp ../{} \; -or -exec mv -nv {} ../{} \;

It would result in commands like this:

mkdir -pv .././bin
mv -nv ./bin/pip .././bin/pip
mv -nv ./bin/pip3 .././bin/pip3
mv -nv ./bin/python3 .././bin/python3
mv -nv ./bin/python3 .././bin/python3
mv -nv ./bin/xxhsum .././bin/xxhsum
mkdir -pv .././etc
mkdir -pv .././include
mv -nv ./include/xxh3.h .././include/xxh3.h
mv -nv ./include/xxhash.h .././include/xxhash.h

… and so on

How to preview

To see what commands will be run, add echo before each command, right after the -exec, like this:

cd "$SRC"
find -type d -exec echo mkdir -vp "$DST"/{} \; -or -exec echo mv -nv {} "$DST"/{} \;
                   ‾‾‾‾                               ‾‾‾‾

It's quite difficult to believe you actually went ahead and merged /usr/local into /usr as the transcript shows, but maybe you had a throwaway install :) — usretc, Dec 22 '20 at 12:25
This is a brilliant solution, but why couldn't you do mv -nv for directories too? — Rucent88, Aug 22 '21 at 18:38
@Rucent88: for the same reason that, in general, one can't use mv alone to merge one directory into another. Specifically mv X Y fails if Y is an existing non-empty directory. — kjo, Nov 24 '22 at 16:17
You might want to add find -type d -exec rmdir -p {} \; 2>/dev/null to remove the empty source directories after the move, and a last find to check that nothing is left. — tricasse, Jul 22 '23 at 14:03
If the first -exec can cause the -or to be evaluated if it fails, then I guess this relies either on mkdir -p never failing, or the subsequent mv being harmless if it does? — mwfearnley, Nov 15 '23 at 16:49

gjs · Answer 7 · 2019-09-03T12:11:39.307

6

For the purest copies, I use the tar (-)B blockread copy method.

example, from within source path ('cd' there if necessary):

tar cBf - <sourcefolder> | (cd /your/target/folder ; tar xBf -)

this creates an exact copy of the source tree, WITH the owner and permissions intact. And if the target folder exists, the data will be merged. Only files that are already existing will be overwritten.

Example:

 $ cd /data1/home
 $ tar cBf - jdoe | (cd /data2/home ; tar xBf -)

When the copy action is successful, you can remove the source (rm -rf <source>). Of course this is not an exact move: the data will be copied, until you remove the source.

As option you can be verbose (display on screen the file being copied), with -v: tar cBvf -

c: create
B: read full block (for pipe read)
v: verbose
f: file to write
x: extract
-: stdout/stdin

sourcefolder can also be * (for anything in current folder)

edited Sep 03 '19 at 12:11

answered Jun 26 '17 at 14:49

gjs

61

Specifying f - to tar is usually unnecessary - the default is to read from stdin/write to stdout. – muru Jun 27 '17 at 08:11
2

@muru not always. It's safest to specify -f - when scripting. – Chris Davies Nov 18 '19 at 07:44
1

@roaima ah yes, it's a GNU thing. Got bit when I had to work with BSD tar – muru Nov 18 '19 at 07:56

asheeshr · Answer 8 · 2014-05-04T07:30:06.570

5

One way to accomplish this would be to use:

mv folder/* directory/folder/
rmdir folder

As long as no two files have the same name in folder and directory/folder, you will achieve the same result i.e. merging.

edited May 04 '14 at 07:30

answered May 04 '14 at 03:39

asheeshr

1,937

3

How exactly does rm folder work? – Giacomo1968 May 04 '14 at 04:28
5

@JakeGould Not at all. :) – n.st May 04 '14 at 06:58
rm folder -fR always works for me – Octopus Dec 10 '15 at 02:43
6

Be aware that this will not work for hidden files – b0fh Mar 01 '16 at 14:20

score 4 · Answer 9 · edited Jun 08 '21 at 00:20

Quick Python solution that only walks the source file tree once

Since I could not find a satisfactory pre-existing solution, I decided to make a quick Python script to achieve it.

In particular, this method is efficient because it only walks the source file tree once bottom up.

It will also allow you to quickly tweak things like file overwrite handling to your liking.

Usage:

move-merge-dirs src/ dest/

will move all contents of src/* into dest/ and src/ will disappear.

move-merge-dirs

#!/usr/bin/env python3
import argparse
import os
def move_merge_dirs(source_root, dest_root):
    for path, dirs, files in os.walk(source_root, topdown=False):
        dest_dir = os.path.join(
            dest_root,
            os.path.relpath(path, source_root)
        )
        if not os.path.exists(dest_dir):
            os.makedirs(dest_dir)
        for filename in files:
            os.rename(
                os.path.join(path, filename),
                os.path.join(dest_dir, filename)
            )
        for dirname in dirs:
            os.rmdir(os.path.join(path, dirname))
    os.rmdir(source_root)
if name == 'main':
    parser = argparse.ArgumentParser(
        description='Move merge src/* into dest. Overwrite existing files.'
    )
    parser.add_argument('src_dir')
    parser.add_argument('dest_dir')
    args = parser.parse_args()
    move_merge_dirs(args.src_dir, args.dest_dir)

GitHub upstream.

See also: https://stackoverflow.com/questions/22588225/how-do-you-merge-two-directories-or-move-with-replace-from-the-windows-command

Tested on Python 3.7, Ubuntu 18.04.

Note that os.rename will clobber files that already exist in the destination. "If both are files, dst it will be replaced silently if the user has permission." — endolith, Jun 08 '21 at 00:03

xer0x · Answer 10 · 2020-01-18T04:47:31.310

1

Here is a script that worked for me. I prefer mv over rsync, so I use Jewel and Jonathan Mayer's solutions.

#!/bin/bash

# usage source1 .. sourceN dest

length=$(($#-1))
sources=${@:1:$length}
DEST=$(readlink -f ${!#})
for SRC in "$sources"; do
    pushd "$SRC";
    find . -type d -exec mkdir -p "${DEST}/{}" \;
    find . -type f -exec mv {} "${DEST}/{}" \;
    find . -type d -empty -delete
    popd
done

edited Jan 18 '20 at 04:47

answered Nov 29 '14 at 20:13

xer0x

121

This solution does not properly escape path names, be careful. – user12439 Jun 07 '15 at 05:15
@user12439, I'll update the solution if you show me which part to fix. – xer0x Jun 08 '15 at 22:53

TalESid · Answer 11 · 2019-04-18T03:09:28.017

This is the command for moving files & folders to other destination:

$ mv /source/path/folder /target/destination/

Remember: mv command will not work if the folder is b̲e̲i̲n̲g m̲e̲r̲ge̲d̲ (i.e. another folder with the same name already exist in the destination) and the d̲e̲s̲t̲i̲n̲a̲t̲i̲o̲n̲ o̲n̲e̲ i̲s̲ n̲o̲t̲ e̲m̲pt̲y.

mv: cannot move '/source/path/folder' to '/target/destination/folder': Directory not empty

If the destination folder is empty, the above command will work fine.

So, in order to merge both folders in any case,
Either do it in 2 commands:

$ cp -rf /source/path/folder /target/destination/
$ rm -rf /source/path/folder

Or combine both as a one-time command:

$ cp -rf /source/path/folder /target/destination/ && rm -rf /source/path/folder

mv = move
cp = copy
rm = remove

-r for directory (folder)
-f force execution

cp is slower if src and dest are on same disk – Akhil Jul 05 '22 at 17:04 — Akhil, Jul 05 '22 at 17:04

score 1 · Answer 12 · answered Oct 20 '21 at 16:16

The above answers are good, but doing this process made me nervous. I wanted to share a test script that demonstrates what the rsync method will actually do.

    reset_rsync_test_local_move(){
        LOCAL_DPATH=$HOME/tmp/rsync-test/local
        MOVE_TEST_ROOT=$LOCAL_DPATH/rsync_move_test
        # Setup home data
        echo "LOCAL_DPATH = $LOCAL_DPATH"
        if [ -d "$LOCAL_DPATH" ]; then
            rm -rf $LOCAL_DPATH
        fi
        mkdir -p $LOCAL_DPATH
        mkdir -p $MOVE_TEST_ROOT
        # Pretend that we accidently botched a move and have a repo inside of a repo
        # so the goal is merge all files from repo/repo into repo
        mkdir -p $MOVE_TEST_ROOT/repo/
        mkdir -p $MOVE_TEST_ROOT/repo/primes/
        mkdir -p $MOVE_TEST_ROOT/repo/perfect/
        mkdir -p $MOVE_TEST_ROOT/repo/nat/
    mkdir -p $MOVE_TEST_ROOT/repo/repo
    mkdir -p $MOVE_TEST_ROOT/repo/repo/primes/
    mkdir -p $MOVE_TEST_ROOT/repo/repo/perfect/
    mkdir -p $MOVE_TEST_ROOT/repo/repo/nat/

    # Some of the primes ended up in the correct and the botched repo
    touch $MOVE_TEST_ROOT/repo/primes/prime02
    touch $MOVE_TEST_ROOT/repo/primes/prime05
    touch $MOVE_TEST_ROOT/repo/primes/prime13
    touch $MOVE_TEST_ROOT/repo/repo/primes/prime03
    touch $MOVE_TEST_ROOT/repo/repo/primes/prime11
    # For prime7, lets say there is a conflict in the data contained in the file
    echo &quot;correct data&quot; &gt; $MOVE_TEST_ROOT/repo/primes/prime07
    echo &quot;botched data&quot; &gt; $MOVE_TEST_ROOT/repo/repo/primes/prime07

    # All of the perfects ended up in the botched repo
    touch $MOVE_TEST_ROOT/repo/repo/perfect/perfect006
    touch $MOVE_TEST_ROOT/repo/repo/perfect/perfect028
    touch $MOVE_TEST_ROOT/repo/repo/perfect/perfect496

    # The naturals have some symlinks, so we need to be careful there
    touch $MOVE_TEST_ROOT/repo/nat/nat04
    touch $MOVE_TEST_ROOT/repo/nat/nat06

    # basedir nats
    touch $MOVE_TEST_ROOT/repo/nat/nat01
    ln -s $MOVE_TEST_ROOT/repo/primes/prime02 $MOVE_TEST_ROOT/repo/nat/nat02
    (cd $MOVE_TEST_ROOT/repo/nat/ &amp;&amp; ln -s ../primes/prime05 nat05)
    ln -s $MOVE_TEST_ROOT/repo/primes/prime11 $MOVE_TEST_ROOT/repo/nat/nat11

    # Botched nats
    touch  $MOVE_TEST_ROOT/repo/repo/nat/nat08
    ln -s $MOVE_TEST_ROOT/repo/primes/prime07  $MOVE_TEST_ROOT/repo/repo/nat/nat07 
    (cd $MOVE_TEST_ROOT/repo/repo/nat/ &amp;&amp; ln -s  ../primes/prime03 nat03)
    ln -s $MOVE_TEST_ROOT/repo/repo/primes/prime11 $MOVE_TEST_ROOT/repo/repo/nat/nat11

    tree $MOVE_TEST_ROOT
}

test_rsync_merge_folders(){
    __doc__=&quot;
    source ~/misc/tests/bash/test_rsync.sh
    &quot;
    reset_rsync_test_local_move

    # Does not work
    #mv $MOVE_TEST_ROOT/repo/repo/* $MOVE_TEST_ROOT/repo
    rsync -avrRP $MOVE_TEST_ROOT/repo/./repo $MOVE_TEST_ROOT

    tree $MOVE_TEST_ROOT

    # Check the content of prime7 to see if it was overwritten or not
    # Ans: the data is not overwritten, only disjoint files are merged in
    cat $MOVE_TEST_ROOT/repo/primes/prime07

    # Remove the botched repo
    rm -rf $MOVE_TEST_ROOT/repo/repo

    # Note that the broken (nat11) link is overwritten
    tree $MOVE_TEST_ROOT

}

The above script will create a test directory with a "correct" and a "botched" repo. Effectively we are supposed to have a repo with folders for natural, prime, and perfect numbers, but something went wrong and some of the data exists in the correct location, but we accidently made a subfolder repo/repo that contains part of the data. The goal is we want to merge everything from ./repo/repo into ./repo

The initial directory structure looks like this:

/home/joncrall/tmp/rsync-test/local/rsync_move_test
└── repo
    ├── nat
    │   ├── nat01
    │   ├── nat02 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime02
    │   ├── nat04
    │   ├── nat05 -> ../primes/prime05
    │   ├── nat06
    │   └── nat11 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime11
    ├── perfect
    ├── primes
    │   ├── prime02
    │   ├── prime05
    │   ├── prime07
    │   └── prime13
    └── repo
        ├── nat
        │   ├── nat03 -> ../primes/prime03
        │   ├── nat07 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime07
        │   ├── nat08
        │   └── nat11 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/repo/primes/prime11
        ├── perfect
        │   ├── perfect006
        │   ├── perfect028
        │   └── perfect496
        └── primes
            ├── prime03
            ├── prime07
            └── prime11

Note I threw in some relative and absolute symlinks to test how it works with those.

After executing:

rsync -avrRP $MOVE_TEST_ROOT/repo/./repo $MOVE_TEST_ROOT

We get:

/home/joncrall/tmp/rsync-test/local/rsync_move_test
└── repo
    ├── nat
    │   ├── nat01
    │   ├── nat02 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime02
    │   ├── nat03 -> ../primes/prime03
    │   ├── nat04
    │   ├── nat05 -> ../primes/prime05
    │   ├── nat06
    │   ├── nat07 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime07
    │   ├── nat08
    │   └── nat11 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/repo/primes/prime11
    ├── perfect
    │   ├── perfect006
    │   ├── perfect028
    │   └── perfect496
    ├── primes
    │   ├── prime02
    │   ├── prime03
    │   ├── prime05
    │   ├── prime07
    │   ├── prime11
    │   └── prime13
    └── repo
        ├── nat
        │   ├── nat03 -> ../primes/prime03
        │   ├── nat07 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime07
        │   ├── nat08
        │   └── nat11 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/repo/primes/prime11
        ├── perfect
        │   ├── perfect006
        │   ├── perfect028
        │   └── perfect496
        └── primes
            ├── prime03
            ├── prime07
            └── prime11

where everything is almost correctly moved. The only issue is that nat11 in the "correct" repo was a broken symlink, so that was overwritten with data from the "botched" subrepo. Other files are not overwritten, only disjoint data is merged.

Removing the botched subdir gives us:

/home/joncrall/tmp/rsync-test/local/rsync_move_test
└── repo
    ├── nat
    │   ├── nat01
    │   ├── nat02 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime02
    │   ├── nat03 -> ../primes/prime03
    │   ├── nat04
    │   ├── nat05 -> ../primes/prime05
    │   ├── nat06
    │   ├── nat07 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/primes/prime07
    │   ├── nat08
    │   └── nat11 -> /home/joncrall/tmp/rsync-test/local/rsync_move_test/repo/repo/primes/prime11
    ├── perfect
    │   ├── perfect006
    │   ├── perfect028
    │   └── perfect496
    └── primes
        ├── prime02
        ├── prime03
        ├── prime05
        ├── prime07
        ├── prime11
        └── prime13

So, the rsync method mostly works, just be careful because any symlinks that are not resolved might be overwritten.

simon · Answer 13 · 2016-02-26T13:55:55.687

It is not a good idea to use commands like cp or rsync. For large files, it will take a long time. mv is much faster since it only update the inodes without copying the files physically. A better option is to use the file manager of your operating system. For Opensuse, there is a file manager called Konquerer. It can move files without actually copying them. It has "cut and paste" function like in Windows. Just select all the sub-directories in directory A. Right click and "move into" directory B which may contain sub-directories with the same names. It will merge them. There are also options whether you want to overwrite or rename files with the same name.

OP asks what happens when mv is used. – don_crissti Feb 26 '16 at 13:49 — don_crissti, Feb 26 '16 at 13:49

score 0 · Answer 14 · answered Nov 18 '19 at 14:04

No sense copying folders that are empty - YMMV

#!/bin/bash

# usage source1 .. sourceN dest

length=$(($#-1))
sources=${@:1:$length}
DEST=$(readlink -f ${!#})
for SRC in $sources; do
    pushd "$SRC";
    # Only one scan - we only need folders with files
    find . -type f | while read FILE ; do
        DIRNAME=`dirname "$FILE"`
        # Create the lowest level directory at once
        if [ ! -d "$DEST/$DIRNAME" ] ; then
            mkdir -v "$DEST/$DIRNAME"
        fi
        mv -v "$FILE" "$DEST/$FILE"
    done
    # Remove the directories no longer needed
    find . -type -d | sort -r | xargs -i rmdir "{}"
    popd
done

find not executed multiple times
mkdir -p is executed even after finding directories sequentially

score 0 · Answer 15 · answered Jun 26 '21 at 19:48

Just wanted to share my solution to a similar issue. I've got a mess. Multiple copies of a directory where each copy has edits to files, and basically just need to merge it back together:

without losing newer changes.
without losing files by ones that may have been corrupted.
without unnecessary copy operations.
retain as much about files as possible (extended attributes, ACLs, etc)
also, to reduce disk usage by hard linking files that have been copied several times, have totally different filenames, and may exist in any number of directories.

That last operation will be anther step where I build a list of duplicate files in those directories based on inode/size/md5sum comparisons, and then decide whether to hard-link or simply delete duplicates (how to control which one to save I have yet to decide).

However, I plan to complete my first operations with the following:

# hard-link over any files that don't exist in the destination while removing them from source
rsync -livrHAX --remove-source-files --ignore-existing --link-dest=../src/ src/ dst/
move over existing files that are newer and remove them from source, keeping backup of ones that were replaced
(after verifying during test drills that inodes of moved files are the same, I conclude that this doesn't slow-copy files, but YMMV or check rsync source for your OS/arch/filesystem)
rsync -buvaiHAX --remove-source-files --suffix=".bak_older" src/ dst/
move over the rest of the files that are older than the ones in the destination, remove them from source, and retain backups of ones that were replaced
rsync -bvaiHAX --remove-source-files --suffix=".bak_newer" src/ dst/
remove empty directories recursively
find src -type d -exec rmdir -p "{}" ; 2>/dev/null
src/ should hopefully now no longer exist
check for averted clobbers against older files to manually verify that the replacements are acceptable
find dst -name '*.bak_older'
check for averted clobbers again newer files to manually verify that the replacements aren't out-dated (in terms of whatever is important to you)
find dst -name '*.bak_older'

I'll report back to this post if I have any major updates to my procedures, but this in effect feels like a quick & safe "directory merge" operation

sir__finley · Answer 16 · 2020-01-30T08:34:19.363

-2

You can merge a and b with:

shopt -s dotglob
mv a/* b

Before mv:

.
├── a
│   ├── c
│   │   └── x
│   └── .x
└── b
    ├── y
    └── .y

After mv:

.
├── a
└── b
    ├── c
    │   └── x
    ├── .x
    ├── y
    └── .y

dotglob allows you to move hidden dot files like .x

Use rmdir to remove empty directory.

edited Jan 30 '20 at 08:34

answered Jan 30 '20 at 08:00

sir__finley

1

1

What happens if folder a contains a subfolder y? – AdminBee Jan 30 '20 at 08:16
It cause error, and nothing changes. – sir__finley Jan 30 '20 at 08:30

Merging folders with mv?

16 Answers16

Use `mv` with `find`. You can do this in one pass.

Explanation

Example

How to preview

No sense copying folders that are empty - YMMV

move over existing files that are newer and remove them from source, keeping backup of ones that were replaced

(after verifying during test drills that inodes of moved files are the same, I conclude that this doesn't slow-copy files, but YMMV or check rsync source for your OS/arch/filesystem)

move over the rest of the files that are older than the ones in the destination, remove them from source, and retain backups of ones that were replaced

remove empty directories recursively

src/ should hopefully now no longer exist

check for averted clobbers against older files to manually verify that the replacements are acceptable

check for averted clobbers again newer files to manually verify that the replacements aren't out-dated (in terms of whatever is important to you)

Linked

Related

Merging folders with mv?

16 Answers16

Use mv with find. You can do this in one pass.

Explanation

Example

How to preview

No sense copying folders that are empty - YMMV

move over existing files that are newer and remove them from source, keeping backup of ones that were replaced

(after verifying during test drills that inodes of moved files are the same, I conclude that this doesn't slow-copy files, but YMMV or check rsync source for your OS/arch/filesystem)

move over the rest of the files that are older than the ones in the destination, remove them from source, and retain backups of ones that were replaced

remove empty directories recursively

src/ should hopefully now no longer exist

check for averted clobbers against older files to manually verify that the replacements are acceptable

check for averted clobbers again newer files to manually verify that the replacements aren't out-dated (in terms of whatever is important to you)

Linked

Related

Use `mv` with `find`. You can do this in one pass.