Carriage returns are prevent sed reading all my files

Question

I have a long list of folders, inside every folder I have the file "log.dat". I wanted to read the value of a specific line inside this file. I saved the folders names in a file "All.txt", in the parent folder, then I ran this code:

#!/bin/bash
in=/a/b/c   #path to the file All.txt  
for i in $(cat $in/all.txt); do
sed -n '/price/p' ${in}/${i}/log.dat

done

This code was able to read almost all the files but for some reason, which I can't figure it out!! It was output the following error message:

/log.dat: No such file or directory #(then the path to this file)

If I run the command sed -n '/price/p' log.dat inside the folder that output the error message. I get an answer!

PS: The content of the file ALL.txt ( which is created by running the command ls>All.txt in the parent folder) is as follow:

STR-548-021-01
STR-548-021-02
STR-548-022-01
STR-548-022-02
STR-548-023-01
STR-548-023-02
.
.
.
.
.

I tried to use the command "awk" instead of "sed" as follow:

awk '{ price }' ${in}/${i}/log.dat

And I got the same error message.

Can these carriage returns happen in Linux as an output for the command ls>All.txt?

@don_crissti. Thanks! Kindly, How can I use "xargs" instead? Also, This command worked very well for almost all my files. Why it is unable to read some of them. They all identical — , May 01 '16 at 23:11
I got the same error message stating that there is no file or directory — , May 01 '16 at 23:34
find ./ -iname log.dat -print0 | sort -z | xargs -0 -L 1 sed -n '/price/p' | tee result.dat xargs use, like that as example — MolbOrg, May 01 '16 at 23:36
@ MolbOrg. Thanks!. I got with xargs, the same error message — , May 01 '16 at 23:44
try the same, but find ./ -iname log.dat -type f -readable -print0 — MolbOrg, May 01 '16 at 23:49

score 1 · Accepted Answer · edited Apr 13 '17 at 12:36

1

There are carriage returns at the end of $i, because there are carriage returns at the end of the lines in all.txt. It was probably produced on Windows: Windows uses the two-character sequence CR,LF to mark a line break, whereas Unix (and most of the rest of the world) uses just LF (linefeed, synonym of newline in the Unix world), so Unix sees a line with a CR at the end.

Either convert your file to Unix line endings, or strip off the CR:

for i in $(cat $in/all.txt); do
  i=${i%$'\r'}
  sed -n '/price/p' ${in}/${i}/log.dat
done

This does nothing if there's no CR at the end of a line. Explanation: $'\r' is a carriage return, and ${i%…} is the value of i minus the specified suffix

Note that your script expects blank-separated input, not one file per line, and the input is a list of wildcard patterns, not file names. See Why does my shell script choke on whitespace or other special characters? . This makes no difference if there are no spaces or wildcards in the input.

edited Apr 13 '17 at 12:36

Community

1

answered May 01 '16 at 23:58

Gilles 'SO- stop being evil'

829,060

Thanks! The file All.txt was created by running the command ls > All.txt in the parent folder . – May 02 '16 at 00:02
@goro I doubt it: that wouldn't insert CR. Are you running under Windows? In what environment? – Gilles 'SO- stop being evil' May 02 '16 at 00:05
No I am a Linux user. But adding i=${i%$'\r'} to the code worked well – May 02 '16 at 00:06
Gilless. Thanks a lot for this info. You are great! Kindly could you please explain to me what " i=${i%$'\r'} " is standing for? – May 02 '16 at 00:08
1

@goro i=${i%foo} sets i to its current value, minus foo at the end. See http://www.gnu.org/software/bash/manual/bash.html#Shell-Parameter-Expansion – Gilles 'SO- stop being evil' May 02 '16 at 00:10
You are awesome Thanks a lot!! I keep learning from you :) – May 02 '16 at 00:11
You can also use programs unix2dos and dos2unix for CR inserting and CR removal, respectively. – Peschke May 02 '16 at 14:16

score 0 · Answer 2 · answered May 01 '16 at 23:35

0

Your error message indicates that sed is getting an argument of /log.dat - neither $i or $in are set to any value. I think you have a blank line in your All.txt file.

/log.dat: No such file or directory #(then the path to this file)

answered May 01 '16 at 23:35

Quentin Fennessy

504

You are right. it is something related to the file ALL.txt. But it is also something that I was unable to understand!!! The folders names are sorted bellow each other and no blanck spaces between them. I did back slash for the foloowing line to the folder name that output the error then I pressed enter and every thing went fine. Why ? – May 01 '16 at 23:52
No, if the variables were unset, that would be //log.dat. They're set — but the last one ends with a CR which causes the end of the file name to overwrite the beginning of the line. – Gilles 'SO- stop being evil' May 01 '16 at 23:53

Carriage returns are prevent sed reading all my files

2 Answers2