how to extract just a specific part of a file name in bash

Question

I have the following list of files:

lab-1:/tmp/tmp.NMFaDO# ls                                                                                              
webserver.123.net.1970-01-01.csv          webserver.123.net.2016-10-01.csv          webserver.123.net.2017-10-01.csv                
webserver.123.net.2015-11-01.csv          webserver.123.net.2016-11-01.csv          webserver.123.net.2017-11-01.csv                
webserver.123.net.2015-12-01.csv          webserver.123.net.2016-12-01.csv          webserver.123.net.2017-12-01.csv                
webserver.123.net.2016-01-01.csv          webserver.123.net.2017-01-01.csv          webserver.123.net.2018-01-01.csv                
webserver.123.net.2016-02-01.csv          webserver.123.net.2017-02-01.csv          webserver.123.net.2018-02-01.csv                
webserver.123.net.2016-03-01.csv          webserver.123.net.2017-03-01.csv          webserver.123.net.current.csv                   
webserver.123.net.2016-04-01.csv          webserver.123.net.2017-04-01.csv          webserver.123.net.db.done                       
webserver.123.net.2016-05-01.csv          webserver.123.net.2017-05-01.csv          webserver.123.net.records_to_delete.txt         
webserver.123.net.2016-06-01.csv          webserver.123.net.2017-06-01.csv          sip2.ci2.lab.jwm2.net.2018-02-01.csv               
webserver.123.net.2016-07-01.csv          webserver.123.net.2017-07-01.csv          sip2.ci2.lab.jwm2.net.db.done                      
webserver.123.net.2016-08-01.csv          webserver.123.net.2017-08-01.csv          sip2.ci2.lab.jwm2.net.records_to_delete.txt        
webserver.123.net.2016-09-01.csv          webserver.123.net.2017-09-01.csv

Inside a bash script, I need to loop through this list and extract just the date portion of the file name. I managed to figure out how to do it from inside the
folder that contains the files using this comman:

lab-1:/tmp/tmp.NMFaDO# ls webserver.123.net*.csv | sed 's/webserver.123.net.\(.*\)\.csv/\1/'                     
1970-01-01                                                                                                                                   
2015-11-01                                                                                                                                   
2015-12-01                                                                                                                                   
2016-01-01                                                                                                                                   
2016-02-01                                                                                                                                   
2016-03-01                                                                                                                                   
2016-04-01                                                                                                                                   
2016-05-01                                                                                                                                   
2016-06-01                                                                                                                                   
2016-07-01                                                                                                                                   
2016-08-01                                                                                                                                   
2016-09-01                                                                                                                                   
2016-10-01                                                                                                                                   
2016-11-01                                                                                                                                   
2016-12-01                                                                                                                                   
2017-01-01                                                                                                                                   
2017-02-01                                                                                                                                   
2017-03-01                                                                                                                                   
2017-04-01                                                                                                                                   
2017-05-01                                                                                                                                   
2017-06-01                                                                                                                                   
2017-07-01                                                                                                                                   
2017-08-01                                                                                                                                   
2017-09-01                                                                                                                         
2017-10-01                    
2017-11-01                   
2017-12-01               
2018-01-01              
2018-02-01  
current

but I can't seem to get the logic to work inside my bash script:

dir=/tmp/tmp.NMFaDO
server=webserver.123.net
for csvfile in $dir/$server.*.csv
do 
  echo $csvfile | sed 's/$dir\/$server.\(.*\)\.csv/\1/'
done

It's currently printing out the full path of the name like this:

/tmp/tmp.NMFaDO/webserver.123.net.1970-01-01.csv
/tmp/tmp.NMFaDO/webserver.123.net.2015-11-01.csv

etc.

Can someone tell me what I'm missing in my bash script? thanks.

See How can I use variables when doing a sed? - however, simple replacements such as these can be done in the shell without resorting to sed — steeldriver, Mar 02 '18 at 18:45

score 6 · Answer 1 · answered Mar 02 '18 at 18:58

With the file name format like that, you can remove the extension, and then just remove everything up to the last dot.

With parameter expansion:

for f in ./webserver.123.*; do
    d=${f%.csv}          # remove trailing .csv
    d=${d##*.}           # remove all up to (now) last .
    echo "filename: $f date: $d"
done

or with sed:

for f in ./webserver.123.*; do
    d=$(echo "$f" | sed -e 's/\.csv$//' -e 's/.*\.//');
    echo "filename: $f date: $d";
done

Gilles Quénot · Accepted Answer · 2018-03-02T18:55:30.430

4

Take care of which quotes you use: so

dir=/tmp/tmp.NMFaDO
server=webserver.123.net
for csvfile in $dir/$server.*.csv
do 
  echo "$csvfile" | sed "s|$dir/$server.\(.*\)\.csv|\1|"
done

"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[@]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words

edited Mar 02 '18 at 18:55

answered Mar 02 '18 at 18:45

Gilles Quénot

33,867

Hi. that's giving me an error that says "sed: bad option in substitution expression" – dot Mar 02 '18 at 18:49
@patrix all across the board? – dot Mar 02 '18 at 18:53
Changed delimiter to |, check my edited post – Gilles Quénot Mar 02 '18 at 18:55

how to extract just a specific part of a file name in bash

2 Answers2