Why are spaces echoing as newlines?

Question

I've got this JSON data that looks like this:

[
  {
    "loginId": "7638749",
    "customerprofileDetails": {
      "securityQuestions": [
        "What is your favorite sports team?",
        "What is your favorite song?",
        "Who is your favorite artist?"
      ]
    }
  }
]

I have a script where I pass in that file as the first argument. The script looks like this:

for json in `cat $PWD/$1 | jq -cr '.[]'` ; do
    echo "$json"
done

For some reason, when I run this script, it outputs this:

> fileFromJson.sh tmp2.json
{"loginId":"7638749","customerprofileDetails":{"securityQuestions":["What
is
your
favorite
sports
team?","What
is
your
favorite
song?","Who
is
your
favorite
artist?"]}}

Why are those newlines there and what's causing this problem? When I just run cat $PWD/tmp2.json | jq -cr '.[]', it prints out on one line. I'm using MacOS if that matters.

Background reading: https://unix.stackexchange.com/questions/131766/why-does-my-shell-script-choke-on-whitespace-or-other-special-characters . I'm not sure that this can be considered a duplicate: what were you trying to do with this for loop? The input is a single piece of JSON, what is there to iterate on? — Gilles 'SO- stop being evil', Jul 30 '18 at 18:20
@Gilles the real input has multiple elements. I need to create a file with the loginId value and use the customerprofileDetails as its content. — Daniel Kaplan, Jul 30 '18 at 18:25

Gilles 'SO- stop being evil' · Accepted Answer · 2018-07-30T18:44:04.867

The unquoted command substitution `cat $PWD/$1 | jq -cr '.[]'` does the following:

Run the command cat $PWD/$1 | jq -cr '.[]'. Which by the way is an overly complicated way of writing jq -cr '.[]' <"$1", except that this variant doesn't break if $1 contains certain special characters or is an absolute path.
Break the resulting string into whitespace-separated pieces. Your question is not very clear, but you seem to expect newline-separated pieces — that's not what an unquoted substitution does.
Treat each piece as a wildcard pattern and, if it matches one or more file name, replace that piece by the list of matching file names.

Steps 2 and 3 are colloquially known as the “split+glob operator” and this is almost never what you want. Always use double quotes around variable and command substitutions unless you know exactly why you need to leave out the quotes.

To process a file line by line in the shell, use a while read loop:

jq -cr '.[]' <"$1" |
  while IFS= read -r line; do
    printf '%s\n' "$line"
  done

But you may be better off piping into some other tool. Shells aren't good at processing large amounts of string data by themselves.

@DanielKaplan See https://unix.stackexchange.com/questions/65803/why-is-printf-better-than-echo — Kusalananda, Jul 30 '18 at 18:39
@dan I think that echo would work here because this is JSON in bash. echo may interpret its first argument as an option if it starts with - and some shells interpret backslash escapes. — Gilles 'SO- stop being evil', Jul 30 '18 at 18:47

confetti · Answer 2 · 2018-07-30T18:24:25.733

0

By default, echo always outputs a trailing newline.

You can pass the -n argument to echo to prevent it from doing so.

From man echo:

-n do not output the trailing newline

Edit:

The "issue" here is not in your cat $PWD/$1 | jq -cr '.[]' command, it's in the echo line itself, or more precisely, the for loop. With the for loop, you are splitting what you're passing to it (the output of the jq command) word by word.

So essentially, you're passing every single string that is separated by a whitespace one-by-one to the echo command, which will by default add a trailing newline everytime it is executed.

edited Jul 30 '18 at 18:24

answered Jul 30 '18 at 18:19

confetti

1,964

That worked, but I don't understand why. Doesn't jq -c convert the json to a single line already? Besides, there were never newlines between those words originally. Why are there trailing newlines between spaces? – Daniel Kaplan Jul 30 '18 at 18:21
I'll edit my answer. – confetti Jul 30 '18 at 18:21
True (for bash, not for all other shells), but not relevant. Changing echo "$json" to echo -n "$json" will remove the newlines but will not print the original. Changing toecho -n "$json "will _not_ print the original either. Consider the input["hello * world"]`… – Gilles 'SO- stop being evil' Jul 30 '18 at 18:22
I have edited my answer. – confetti Jul 30 '18 at 18:24

Why are spaces echoing as newlines?

2 Answers2