15

I have a JSON fragment.

The following does not work:

VALUE=<<PERSON
{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "jim@gmail.com"  
}
PERSON
echo -n "$VALUE" | python -m json.tool

The result is:

No JSON object could be decoded

Doing the same with jq, i. e.

echo -n "$VALUE" | jq '.'

There is no output.

There is the same behavior for the following:

VALUE=<<PERSON
'{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "jim@gmail.com"  
}'
PERSON
echo -n "$VALUE" | python -m json.tool

Response:

No JSON object could be decoded

But the following works:

VALUE='{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "jim@gmail.com"
}'
echo -n "$VALUE" | jq '.'
echo -n "$VALUE" | python -m json.tool
Jim
  • 1,391
  • 5
    I don't know what bash is doing, but there's a trailing comma after the email string in your first two but not on the third, which would make the first couple illegal JSON – Nick T Apr 11 '18 at 14:53
  • @NickT you should make that an answer as I think that is precisely the problem. – rrauenza Apr 11 '18 at 18:20
  • If that's the (sole) answer it should probably be closed as "can't be reproduced (a typo)". However, it looks like Kusa's and terdon's answer mention the assignment + redirection is totally broken so you get an empty string, so there are two problems, both of which would give the same "No JSON..." error. It's very good practice to bisect problems by checking your assumptions in the middle: a simple echo $VALUE without ... | jq would be informative. – Nick T Apr 11 '18 at 18:27
  • @NickT:That was a copy/paste issue. Sorry for the confusion – Jim Apr 11 '18 at 21:00

4 Answers4

22
VALUE=<<PERSON
some data
PERSON

echo "$VALUE"

No output.

A here-document is a redirection, you can't redirect into a variable.

When the command line is parsed, redirections are handled in a separate step from variable assignments. Your command is therefore equivalent to (note the space)

VALUE= <<PERSON
some data
PERSON

That is, it assigns an empty string to your variable, then redirects standard input from the here-string into the command (but there is no command, so nothing happens).

Note that

<<PERSON
some data
PERSON

is valid, as is

<somefile

It's just that there is no command whose standard input stream can be set to contain the data, so it's just lost.

This would work though:

VALUE=$(cat <<PERSON
some data
PERSON
)

Here, the command that receives the here-document is cat, and it copies it to its standard output. This is then what is assigned to the variable by means of the command substitution.

In your case, you could instead use

python -m json.tool <<END_JSON
JSON data here
END_JSON

without taking the extra step of storing the data in a variable.

It may also be worth while to look into tools like jo to create the JSON data with the correct encoding:

For example:

jo type=account customer_id=1234 customer_email=jim@gmail.com random_data="some^Wdata"

... where ^W is a literal Ctrl+W character, would output

{"type":"account","customer_id":1234,"customer_email":"jim@gmail.com","random_data":"some\u0017data"}

So the command in the question could be written

jo type=account customer_id=1234 customer_email=jim@gmail.com |
python -m json.tool
Kusalananda
  • 333,661
  • 2
    You could also just do PERSON=" followed by a newline and the multi-line data, then another " at the end. – R.. GitHub STOP HELPING ICE Apr 11 '18 at 14:14
  • 1
    @R.. Yes, but a here-document allows you to bypass the quoting rules of the shell. It is therefore often safer to use a here-document instead of a quoted string for multi-line data, especially if the data contains single or double quotes (or both). – Kusalananda Apr 11 '18 at 14:20
  • 2
    @R.. Given it's JSON we're talking about, it might be better to use single quotes to not have to escape the double quotes of each property name. PERSON='. That's unless the OP wants to interpolate variables later. – JoL Apr 11 '18 at 15:39
  • (backslash)(newline) seems to vanish in a here document, even if you quote/escape the delimiter word. That might be desirable, but is there any way to disable it? – Scott - Слава Україні Apr 11 '18 at 18:17
  • @Scott If that question hasn't been asked on this site before, it would be an excellent question in its own right. – Kusalananda Apr 11 '18 at 18:24
  • @Scott Here (there are probably duplicates of this one to): https://unix.stackexchange.com/questions/399488/keep-backslash-and-linebreak-with-eof – Kusalananda Apr 11 '18 at 18:36
  • @Scott If you’re inside command substitution too, this question may be relevant - there is a Bash bug. – Michael Homer Apr 11 '18 at 19:20
  • @MichaelHomer: Thank you!   I spent the past two hours experimenting, and I was about to come back and post a comment that bash has a bug, and then I saw your comment.   BTW, I look at Chet Ramey’s so-called “explanation” (quoted by Kevin) and all I see is “Yes, there is a bug.  ’Twas brillig, and the slithy toves did gyre and gimble in the wabe …”; i.e., an acknowledgement that there is a bug, and then gibberish.   Does it make sense to you?  … (Cont’d) – Scott - Слава Україні Apr 11 '18 at 20:48
  • (Cont’d) …  P.S.  I see that I saw Kevin’s post last year (and, in fact, I commented on it), but I cannot remember whether I understood it then. – Scott - Слава Україні Apr 11 '18 at 20:48
  • @Scott I guess you can't see them, but there's a series of very similar deleted answers from the same user. It doesn't really answer the question and it frankly should have been deleted too, but Chet's interpretation is the same as mine. The patch went in so some current or future Bash version will behave correctly. It's just an unpleasant corner case for situations like in this answer at the moment. – Michael Homer Apr 11 '18 at 22:37
  • Just because you don't understand some words, doesn't mean those words are gibberish. – u1686_grawity Apr 12 '18 at 05:26
12

Because the variable isn't being set by your heredoc:

$ VALUE=<<PERSON  
> {    
>   "type": "account",  
>   "customer_id": "1234",  
>   "customer_email": "jim@gmail.com",  
> }  
> PERSON
$ echo "$VALUE" 

$

If you want to use a heredoc to assign a value to a variable, you need something like:

$ read -d '' -r VALUE <<PERSON  
{    
  "type": "account",  
  "customer_id": "1234",  
  "customer_email": "jim@gmail.com",  
}   
PERSON
terdon
  • 242,166
  • 1
    Why are you wrapping the JSON data in single quotes?  It doesn’t really look like the OP wants them to be part of his input string.  Aside from that, +1 for cutting down on the homeless cat population.  As with Kusalananda’s answer, you might want to suggest << \PERSON to protect against $s in the input and backslashes at the ends of lines. – Scott - Слава Україні Apr 11 '18 at 18:17
  • @Scott um, because I just blindly copied the text from the OP. Thanks – terdon Apr 11 '18 at 18:29
  • 3
    This is the right answer. $(cat <<EOF ... EOF) is a weird construct: running a subshell and then sending a heredoc to cat just so it sends it to STDOUT and then assigning the result of that subshell to a variable? I wish people would think about what they're saying about their thought processes. Assigning a heredoc to a variable via read, by comparison, is sane. – Rich Apr 11 '18 at 19:18
  • I wouldn’t say that $(cat << EOF … (data) … EOF ) is weird. It’s awkward and convoluted, but so is read -d … << EOF — especially read -d '' << EOF. I appreciate terdon’s answer because it uses only builtins, no programs. But, more importantly, the $(cat << EOF … (data) … EOF ) fails if any lines end with \ (backslash) — see the comments under Kusalananda’s answer. – Scott - Слава Україні Apr 12 '18 at 01:53
5

It is because the way you have defined a here-doc to use with a JSON is wrong. You need to use it as

VALUE=$(cat <<EOF
{  
  "type": "account",  
  "customer_id": "1234",  
  "customer_email": "jim@gmail.com",  
}
EOF
)

and doing printf "$VALUE" should dump the JSON as expected.

Inian
  • 12,807
3

Heredocs and variables don't mix well or at least not in this way. You can either…

Pass the heredoc as the standard input of an application

python -m json.tool <<PERSON  
{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "jim@gmail.com",
}
PERSON

or…

Store multi-line text in a shell variable

VALUE='{
  "type": "account",
  "customer_id": "1234",
  "customer_email": "jim@gmail.com",
}'

I used single quotes to avoid the need to escape the inner double quotes. Of course you can also use double quotes, e. g. if you need to expand parameters:

VALUE="{
  \"type\": \"account\",
  \"customer_id\": ${ID},
  \"customer_email\": \"${EMAIL}\",
}"

Then you can use the variable value later on.

echo -n "$VALUE" | python -m json.tool
David Foerster
  • 1,575