12

I'm writing some scripts for testing an API. I'm interested in the JSON response as well as the HTTP status code of the request. I definitely want to pretty-print the JSON response in order to make it easier to read.

I'm using curl to perform the request and want to use python -m json.tool to pretty-print the json result.

Curl has a nice option -w that can be used to surface information about the request, like %{http_code}. Unfortunately, that information prints to stdout and confuses python -m json.tool. It seems it isn't possible to configure it to ignore trailing non-json data.

When I do

curl \
'--silent' \
'--insecure' \
'-L' \
'-w' \
'\n%{http_code}\n' \
'--user' \
<REDACTED> \
'-X' \
'GET' \
'--' \
'https://somecompany.com/some_api_endpoint' \
| python -m json.tool

I get

$ bash call_api_endpoint_script.sh 
Extra data: line 2 column 1 - line 3 column 1 (char 203 - 207)
Exit 1

Is there a way to configure curl to write the status code to a file? The -w option in the man page doesn't seem to mention the possibility of redirecting this information elsewhere.

Yuri Astrakhan
  • 107
Greg Nisbet
  • 3,076
  • Is something like PycURL not an option? That should let you interface directly with the response data provided by libcurl. – thrig Feb 06 '17 at 19:22

5 Answers5

9
$ curl -s -k -w '%{stderr}%{http_code}\n%{stdout}\n' \
  https://run.mocky.io/v3/0e98ba3e-335e-421b-b762-884d2bf613ba |\
  tee /dev/stderr | jq -r '.name'
200
{
    "name": "Grape"
}
Grape

stderr From this point on, the -w, --write-out output will be written to standard error. (Added in 7.63.0)

stdout From this point on, the -w, --write-out output will be written to standard output. This is the default, but can be used to switch back after switching to stderr. (Added in 7.63.0)

deFreitas
  • 241
3

I found a way around this issue by using -o to redirect just the content to a temporary file, leaving just the status code in the curl's output.

I can then read the contents of the temporary file from python -m json.tool and pretty print them.

For example,

content=$(mktemp)

curl \
...
-w \
'HTTP_STATUS_CODE: %{http_code}\n' \
...
-o \
"$content" \
-- \
'https://somecompany.com/some_api_endpoint'

<"$content" python -m json.tool
Greg Nisbet
  • 3,076
1

If you use jq instead of python -m json.tool, you’ll find it parses JSON even with trailing text:

$ echo '{ "foo": "bar" } text' | ./jq
{
  "foo": "bar"
}
parse error: Invalid literal at line 2, column 0

Of course if you want to ignore the error message you can just redirect that to /dev/null.

Also note that with jq if you want to use its output in a pipe or redirect it to a file, I think you need to give it a “filter” argument:

echo '{ "foo": "bar" } trailing text' | jq . > OUT.json

. is just the simplest possible filter. It means “take the input and produce it unchanged as output”.

sideshowbarker
  • 140
0

I definitely want to pretty-print the JSON response in order to make it easier to read.

That's exactly what https://httpie.org/ is for. sending GET, POST, PUT, DELETE, PATCH etc to API and pretty print JSON answer.

As for the curl part of your question, I have no clue, sorry.

exore
  • 261
0

Whilst the write-out from -w only goes to stdout the output from -o can be redirected to any file. In particular you could direct it to stderr with -o /dev/stderr. If you now swap over stderr and stdout (with 3>&1- 1>&2- 2>&3-) you'll be in a position to see your HTTP return code (through stderr) whilst piping your output to your desired program.

In other words, something along the lines of:

curl ... -w 'HTTP_STATUS_CODE: %{http_code}\n' -o /dev/stderr ... | 3>&1- 1>&2- 2>&3- | python -m json.tool
borrible
  • 101