Keep escaped forward slashes in JSON string values with jq

Question

According to the JSON specification, forward slashes don't have to be escaped with a backslash but they can be.
I have a JSON file which has all forward slashes in string values escaped for compatibly reasons (but not inside the keys):

{
  "proto://some/path": "\/\/some\/path"
}

However, jq automatically removes these backslashes:

$ echo '{"proto://some/path":"\/\/some\/path"}' | jq -c .
{"proto://some/path":"//some/path"}

I need the output to be {"proto://some/path":"\/\/some\/path"}

How can I tell jq to NOT change any string values and keep these backslashes?
Alternatively, is there a way to re-add these backslashes only to the values after it went through jq?

Answer 1

I'd be surprised if you could. jq decodes its input, does its operations and encodes the resulting object as json. Upon encoding and outputting, it outputs those strings encoded in JSON and the information that there were \s in front of /s has long been lost. Same would happen if those /s where initially written \u002f. You'll find that jq also reformats 1.0 as 1, 1e2 as 100, INF as 1.7976931348623157e+308, etc for the same reason.

However, JSON is a relatively easy file format to process reliably by hand using for instance perl regexps.

To add back the \s before each / in all strings except those that are object keys, you could do:

jq... |
  perl -0777 -pe '
    s{"(?:\\.|.)*?"(\s*:)?}{
      $1 ? $& : $& =~ s{/}{\\/}gr
    }ge'

Which should work correctly even if you have strings embedding "s and \s (like {"key": "//\"//\\"}).

As an alternative to jq, you could use the JSON::PP perl module which can be told to escape slashes (though would be in all strings though):

$ json_pp -json_opt escape_slash < your-file
{"proto:\/\/some\/path":"\/\/some\/path"}

If you're already familiar with perl, the learning curve would be less steep than having to learn the jq syntax.

In any case while the JSON format allows /s to be escaped as \/ (or \u002f like for any character) those don't need to. From what I read online, that's allowed so one can embed a JSON string containing </ in an HTML <script> tag by writing it "<\/whatever". That's why some JSON encoders encode those \/ as / as that makes it more portable. But if that JSON is not meant to be embedded as is in HTML, it probably doesn't matter. And if it were, you'd likely want to have that encoding everywhere including in object keys.

Answer 2

I'm assuming that some process that isn't aware of the need for encoding strings in JSON has inserted the literal string they think they need to store, without encoding it. Since the backslashes are escaping characters that don't need escaping, and since no backslashes are escaping the literal backslashes themselves, they will appear to "disappear" when jq is used to extract and decode the string or when it processes the document for other reasons.

In short, the forward slashes don't have to be escaped (escaping them is effectively a no-op), but backslashes need to be if you want to keep them as literal backslashes.

The following will change each / into \\/ (which is how you write \/ in a JSON string) in all string values in your document, recursively. Remember, when the jq expression is processing the data, the jq parser has already removed the backslashes.

jq 'walk(if type == "string" then gsub("/"; "\\/") else . end)' file

For the given example document, this would generate

{
  "proto://some/path": "\\/\\/some\\/path"
}

Extracting and decoding the encoded string value from the modified document would give you \/\/some\/path:

$ jq 'walk(if type == "string" then gsub("/"; "\\/") else . end)' file | jq -r '."proto://some/path"'
\/\/some\/path

You would arrive at the same JSON document if you created it from scratch like so:

$ jq -n --arg 'proto://some/path' '\/\/some\/path' '$ARGS.named'
{
  "proto://some/path": "\\/\\/some\\/path"
}