10

We use the following script:

more test.sh
#!/bin/bash

while read -r line
do

echo $line

done < /tmp/file

This is the file:

kafka-broker,log.retention.hours,12
kafka-broker,default.replication.factor,2
fefolp-defaults,fefolp.history.fs.cleaner.interval,1d
fefolp-defaults,fefolp.history.fs.cleaner.maxAge,2d
fefolp-env,fefolp_daemon_memory,10000
blo-site,blo.nodemanager.localizer.cache.target-size-mb,10240
blo-site,blo.nodemanager.localizer.cache.cleanup.interval-ms,300000
ams-env,metrics_collector_heapsize,512
fefolp,hbase_master_heapsize,1408
fefolp,hbase_regionserver_heapsize,512
fefolp,hbase_master_xmn_size,192
core-site,blolp.proxyuser.ambari.hosts,*
core-site,Hadoop.proxyuser.root.groups,*
core-site,Hadoop.proxyuser.root.hosts,*
blo-site,blo.scheduler.minimum-allocation-mb,1024
blolp-env,fefolp_heapsize,4096

Remark - after the last line - there are no space!

But the script prints only these lines (except the last line):

./test.sh
kafka-broker,log.retention.hours,12
kafka-broker,default.replication.factor,2
fefolp-defaults,fefolp.history.fs.cleaner.interval,1d
fefolp-defaults,fefolp.history.fs.cleaner.maxAge,2d
fefolp-env,fefolp_daemon_memory,10000
blo-site,blo.nodemanager.localizer.cache.target-size-mb,140
blo-site,blo.nodemanager.localizer.cache.cleanup.interval-ms,300
ams-env,metrics_collector_heapsize,51
fefolp,hbase_master_heapsize,1408
fefolp,hbase_regionserver_heapsize,542
fefolp,hbase_master_xmn_size,19
core-site,blolp.proxyuser.ambari.hosts,*
core-site,Hadoop.proxyuser.root.groups,*
core-site,Hadoop.proxyuser.root.hosts,*
blo-site,blo.scheduler.minimum-allocation-mb,1024

Why does this happen?

Peter Mortensen
  • 1,025
yael
  • 13,106

3 Answers3

18

Your input text contains an incomplete line as its last line. The last line is not terminated by a newline.

while IFS= read -r line || [ -n "$line" ]; do
    printf '%s\n' "$line"
done <file

The above loop will read unmodified lines¹ (without stripping whitespaces or interpreting backslashed control sequences) from the file called file and print them to standard output.

When an incomplete line is read, read will fail, but $line will still contain data. The extra -n test will detect this so that the loop body is allowed to output the incomplete line. In the iteration after that, read will fail again and $line will be an empty string, thus terminating the loop.

¹ assuming they don't contain NUL characters in shells other than zsh and assuming they don't contain sequences of bytes not forming part of valid characters in the yash shell, both of which shouldn't happen if the input is valid text, though that missing line delimiter on the last line already makes it invalid text.

Stéphane Chazelas
  • 544,893
Kusalananda
  • 333,661
  • the script is only example of the problem , actually I set the echo in the script to show this issue , but what we need is how to set the last line with "\n" with sed for example, since the the - file is already exists – yael Nov 18 '18 at 12:26
  • so if need to add "\n on the last line , then need sed/perl to add the "" for this solution – yael Nov 18 '18 at 12:30
  • @yael You say in the question that you use a loop like the one you are showing. I gave a solution for how to fix that loop. It does not matter what the body of the loop looks like, $line will be each individual line of the file, including the last incomplete one. If you don't use a loop like this, what do you use? – Kusalananda Nov 18 '18 at 12:37
  • ok , so maybe I will post another question for how to add "\n" on the last line – yael Nov 18 '18 at 12:38
  • 1
    @yael How to add a newline to the end of a file? – Kusalananda Nov 18 '18 at 12:40
  • thank you so much my friend I am really appreciate it – yael Nov 18 '18 at 13:08
  • @Kusalananda Could you please also answer this one with a nearly identical answer? https://superuser.com/questions/1500525/reason-for-behind-read – bot47 Nov 09 '19 at 13:14
-1

Input to the while read loop must be done like this :

cat /tmp/file | { cat ; echo ; } | while read -r line; do echo $line;  done

This will process all lines, including the last one.

Source : My open source project https://sourceforge.net/projects/command-output-to-html-table/

Nathan S.R.
  • 33
  • Just do { cat /tmp/file; echo; } if you're going to do that: https://unix.stackexchange.com/a/418081/70524 – muru Oct 12 '21 at 03:25
  • We could use any outputting program instead of cat /tmp/file hence the given structure / syntax. – Nathan S.R. Oct 12 '21 at 03:53
  • Yes, you could use any program: { foo ; echo; } | ... – muru Oct 12 '21 at 04:16
  • 1
    This would add an extra empty line to any file whose last line is properly terminated by a newline character. – Kusalananda Feb 23 '23 at 08:45
  • @Kusalananda But the question is "How to make the 'while' loop recognize the last line?" and my answer solves the problem ! – Nathan S.R. Feb 28 '23 at 13:47
  • 1
    @NathanSR No, because if you are trying to read the last line from a file that has a properly terminated last line already, this would give you an empty line as the result, not the last line of the file. Yes, you would get the last line from your compound command, but that would not be the last line from foo if foo properly terminates the last line. – Kusalananda Feb 28 '23 at 13:56
-1

Following Your input text contains an incomplete line as its last line. The last line is not terminated by a newline. Leave the loop as is and simply make sure there is a return for the variable line.

while read -r i
do
#
line="$(echo "$i"| sed 's/$/@/'| tr '\n' '@'|sed 's/@@/@/' |tr '@' '\n')"
#
echo $line
#
done < /tmp/file

You can change the @ to anything you like.

Edgar Magallon
  • 4,820
Paul L
  • 1
  • This does not work because the read call will fail upon reading the incomplete line and the body of the if-statement will therefore not be executed for that data. Additionally, this would replace any occurrence of @ in the data with a newline, and may also expand \t and \n and other backslash-sequences depending on what shell options are set. It will also not retain tabs or proper spacing (tabs will be changed to spaces, and multiple consecutive spaces will be combined), and will apply filename globbing on the text (due to the unquoted use of $line with your secord echo). – Kusalananda Feb 23 '23 at 08:42