Counting Files in Remote location using sftp in shell script

Question

Getting Error with the below command to check on the Remote directory a specific type of files. The requirement is to get the count of the specific files.

file_exists=$(sftp $FTP_UNAME@$FTP_HOST ls *$FILE_ID*.txt | wc -l)

Is this not properly written? Please help...

Below is the Usage Tip - 
usage: sftp [-1246aCfpqrv] [-B buffer_size] [-b batchfile] [-c cipher]
          [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]
          [-o ssh_option] [-P port] [-R num_requests] [-S program]
          [-s subsystem | sftp_server] host
       sftp [user@]host[:file ...]
       sftp [user@]host[:dir[/]]
       sftp -b batchfile [user@]host

Stéphane Chazelas · Answer 1 · 2024-02-06T18:21:19.610

sftp is hard to use reliably in scripts. You could use some SFTP module of some programming languages instead. For instance with perl's Net::SFTP::Foreign:

perl -MNet::SFTP::Foreign -le '
  ($host, $dir, $id) = @ARGV;
  $s = Net::SFTP::Foreign->new($host, autodie => 1, fs_encoding => latin1);
  print scalar @{
    $s->ls($dir, names_only => 1, wanted => qr/\Q$id\E.*\.txt\z/s)
  }' -- "$FTP_UNAME@$FTP_HOST" . "$FILE_ID"

Would report the number of files whose name contains $FILE_ID and ends in .txt in your landing directory whatever $FILE_ID may contain (even things like spaces or ; which would be a problem for sftp) and whatever the file names there contain (even newline characters which would be a problem for wc -l).

(latin1 encoding to not try to do UTF-8 decoding and work with arbitrary byte values in file paths).

To get the list of files in a shell array, with zsh:

files=( ${(0)"$(
  perl -MNet::SFTP::Foreign -l0 -e '
  ($host, $dir, $id) = @ARGV;
  $s = Net::SFTP::Foreign->new($host, autodie => 1, fs_encoding => latin1);
  print for @{
    $s->ls($dir, names_only => 1, wanted => qr/\Q$id\E.*\.txt\z/s)
  }' -- "$FTP_UNAME@$FTP_HOST" . "$FILE_ID")"}
)

With bash 4.4+:

readarray -td '' files < <(
  perl -MNet::SFTP::Foreign -l0 -e '
  ($host, $dir, $id) = @ARGV;
  $s = Net::SFTP::Foreign->new($host, autodie => 1, fs_encoding => latin1);
  print for @{
    $s->ls($dir, names_only => 1, wanted => qr/\Q$id\E.*\.txt\z/s)
  }' -- "$FTP_UNAME@$FTP_HOST" . "$FILE_ID")

Though you lose the exit status of perl.

While this is the more correct solution, it depends on knowledge of a programming language, which is not everyone, especially not a beginner, would have. In my experience the overwhelming majority of filenames don't contain newlines. The worst offenders in filenames are most commonly spaces and special symbols, like (), [] etc. So the choice is between absolute (?) correctness with knowledge of a programming language, versus using the shell with minimal quoting and some file naming discipline. — Vilinkameni, Feb 06 '24 at 17:49

Vilinkameni · Answer 2 · 2024-02-06T18:30:25.053

As stated in the manual, sftp(1) will enter interactive mode unless -b (batch mode) parameter is given. To pipe commands to sftp, specify the special filename of -, for example:

printf "ls\n" | sftp -b- server.example.com

Your command includes unquoted wildcards, which are interpreted by the shell during pathname expansion, applied to the local current directory (which is probably not desired), so that argument needs to be quoted. Further, you need to pass the parameter -1 to sftp's command ls if you want to count the number of lines it outputs.

Lastly, lines of output of the command ls include the line with the interactive prompt and the command itself, so the count needs to be decreased by one. There are multiple ways this can be achieved, for example using arithmetic expansion in a construct like

printf "%d\n" $(( $( ... ) - 1))

Addendum: As @Stéphane Chazelas stated, this solution will have problems with filenames containing newlines (which admittedly is highly unlikely, but theoretically possible), because of the way the OP is counting files, using wc; there's an entire question with answers on why not to parse the output of ls (the utility), and what to do instead, with similar conclusions as Stéphane's. Additionally, wc doesn't distinguish between directories and files - it counts lines, containing directories along with files. Likewise, the patern passed as argument to sftp's command ls won't work if it contains spaces. It can be quoted-within-the-quote, but it adds a layer of complexity:

printf "ls -1 *\"$FILE_ID\"*.txt\n" | ...

The idea behind my answer is more educational - to answer the question "Is this not properly written?" than provide a solution which should work under all circumstances.

Counting Files in Remote location using sftp in shell script

2 Answers2