Use a wildcard in directory path

Question

I'm trying to write a bash script where part of a directory is a long, unknown string. I know the first part of the directory name. How do I combine the known string and the wildcard in the path? As of now, I have:

ID=$1;

IMP="adam@ocelot.cs.edu:/data/"$ID"/EER_DATA/$ID"*"/ImpostorScores.txt"

but this passes a literal *.

bash does this by default e.g. for FOO=/*/passwd. Is the path local (so an immediate glob expansion can happen) or is it remote? — thrig, May 19 '16 at 22:56
Globbing doesn't understand hosts. This will look for a directory named 'adam@ocelot.cs.edu:in the current directory and so on. The glob won't find a match and thefore remain as an asterisk. What's would you do with the$IMP` variable if if did expand across hosts? — Petr Skocik, May 19 '16 at 22:56
In regards to the first question, it's a remote path. Ultimately I'll be using this in an scp command — Adam_G, May 19 '16 at 22:59
Globbing a remote path would require either that the filesystem be exported (NFS, FUSE-over-SSH, etc) or that a SSH is done over to the system and a glob run there. — thrig, May 19 '16 at 23:42
There are two $ID in your string for $IMP. Is that as you need it? — , May 21 '16 at 04:27

score 2 · Answer 1 · answered May 19 '16 at 23:59

2

There are several options,

Use NFS or FUSE-over-SSH or something to expose the remote filesystem locally, then let bash apply the default FOO=/*/passwd glob on that exported filesystem path. (ZSH has a ${~spec} glob substitution parameter expansion, otherwise see your shell's manual.)
SSH over to the remote system, and do the glob there. ssh host 'echo /*/passwd'
Pass the escaped (or otherwise defanged via backslash or single quotes) glob through to the remote system, scp host:/\*/passwd .

Some of these options are prone to race conditions should the filename change between when the glob is done and any subsequent steps involving that (possibly changed) filename.

answered May 19 '16 at 23:59

thrig

34,938

I'm not familiar with 1. For #2, can you provide more detail? I understand how to login to my server via ssh, but I've never used it in a shell script on my local machine. For 3, I've tried escaping and using single quotes, but it's still passing "*" – Adam_G May 20 '16 at 00:37
@thrig #2 is resolve the full path first then assign it to a variable. dest=$(ssh host 'echo /data/$ID/EER_DATA/$ID*'). Then use that variable, scp filename host:$dest/filename – vjangus May 20 '16 at 01:51

score 0 · Answer 2 · answered May 20 '16 at 01:39

0

You can use tar over ssh,

tar cf - filename | ssh adam@ocelot.cs.edu "cd /data/$ID/EER_DATA/$ID*; tar xf -"

Make sure you have setup ssh-agent so it won't prompt you with the password. I don't think you can use wildcard dir with scp, not even working with a regular cp.

I've tested this on bash and zsh.

answered May 20 '16 at 01:39

vjangus

101

I'm trying to copy a file, not a directory. This returns the error "Not a directory". Which part should I edit? – Adam_G May 20 '16 at 19:31

score 0 · Answer 3 · 2016-05-21T09:52:28.567

The right side of a "Variable Assignment" is considered quoted (no splitting or globing):

LESS=+'/A variable may be assigned' man bash

A variable may be assigned to by a statement of the form:
name=[value]
Word splitting is not performed, ... . Pathname expansion is not performed.

Therefore:

$ var=*
$ echo "$var"
*

To get an asterisk to expand, you need to use it un-quoted:

var="$(echo *)"                ### Expand to the list of files on pwd.

In your case:

$ ID=one
$ IMP="$( echo "adam@ocelot.cs.edu:/data/$ID/EER_DATA/"*"/ImpostorScores.txt")"
$ echo "$IMP"
adam@ocelot.cs.edu:/data/one/EER_DATA/something/ImpostorScores.txt

That something will be expanded if the whole path match, that is, a tree like this actually exists:

$ tree adam@ocelot.cs.edu\:/
adam@ocelot.cs.edu:/
└── data
    └── one
        └── EER_DATA
            └── something
                └── ImpostorScores.txt

Note: I removed the second $ID from your source, easy to re-place.

Use a wildcard in directory path

3 Answers3