3

I need to produce JSON configuration files with echo and tee called from my Python script.

By trial-and-error I've found out that I have to use single quotes. Yet I don't understand all the behaviour that I came across when using Python's run(). The following code prints my questions:

#!/usr/bin/env python3

from subprocess import run


conf_file="""{
"alt-speed-down": 50,
}"""
print("Question 1. It does not work with double quotes. Why?")
run(f"""echo "{conf_file}" """, shell=True)
print("It works with single quotes.")
run(f"""echo '{conf_file}'""", shell=True)
conf_file="""{
"alt-speed-down": 50,
}"""
print("""Question 2. It does not work with double quotes, even when I escape the quotes.
Whereas when I type in my shell: 
echo ""This is a quoted string."" 
it works. Why? 
""")
run(f"""echo "{conf_file}" """, shell=True)
print("""Question 3. It works with single quotes, even with escaped quotes. 
whearas when I type in my shell:
echo '"this is quoted"' 
I get the backslashes printed. Why aren't
the backslashes printed when called with Python's run()?""")
run(f"""echo '{conf_file}'""", shell=True)

I use Bash as my shell. Why does escaping double quotes differ when done from my Bash shell compared to Python's run. Am I not accessing my Bash shell with specifying shell=True in run()?

P.S. I know that generating JSON with json module is a way to do this, but in my case it is mostly copying already existing JSON from my backed up configuration files. I want to avoid reading such JSON files into a string in my script - the script is meant to be run on the newly reinstalled OS where such backups won't be initially available. That is why I need to have many string variables in my Python string that store such JSON configuration files

John Smith
  • 753
  • 2
    I'm sure there are easier ways to generate JSON from Python. The helpful people at StackOverflow may possibly help you with that part. Is your current question basically why two different languages have different quoting rules? It's also unclear why you need to involve Python here at all if all you're doing is shelling out to bash. – Kusalananda Feb 22 '23 at 10:54
  • Yeah, python comes with a json module, which generates working JSON. Also, using tee from a language that can do whatever you want with strings sounds like you really want to spend a tiny bit more time learning Python if you choose to use it. – Marcus Müller Feb 22 '23 at 11:07
  • @Kusalananda I used to create JSON configuration files with Bash scripts in the past. I chose Python because of the ease that """ gives me when defining multiline strings. Thanks to this I do not have to escape many quotes in my multiline Bash commands or JSON files. Also, Python being statically scoped helps a bit when my configuration scripts get really lengthy. – John Smith Feb 22 '23 at 11:27
  • Why are you calling run() just to do an echo? Python has a print function built-in! – jwodder Feb 22 '23 at 19:16
  • @jwodder In my script I use run for this: run(f"""doas systemctl stop transmission-daemon && doas sed -i '/^/d' /var/lib/transmission-daemon/info/settings.json && echo '{conf_file}'|doas tee /var/lib/transmission-daemon/info/settings.json && doas sed -i '/^/d' /etc/transmission-daemon/settings.json && echo '{conf_file}'|doas tee /etc/transmission-daemon/settings.json""", shell=True) but I didn't want to clutter the question with something that is not string manipulation and quoting. – John Smith Feb 22 '23 at 20:41
  • If your only reason for choosing python is multi-line strings then I suggest you use perl (or maybe awk) instead. Both perl and awk fit in better for writing command-line tools for text processing, and both have no difficulty with multi-line strings or with executing external programs like bash does. perl also has a JSON module for parsing/modifying/generating json. Also much of what you learn in perl, awk, or sed (and grep to a lesser extent) helps with learning the others - not so much with python, it's too different. Or just use bash - nothing in your script above requires python. – cas Feb 23 '23 at 00:35

1 Answers1

4

About the quotes, leaving aside the newlines, this:

conf_file="""{ "alt-speed-down": 50, }"""

assigns the string { "alt-speed-down": 50, } to the variable. Then when you run run(f"""echo "{conf_file}" """, shell=True), the shell sees the string

echo "{ "alt-speed-down": 50, }"

which is different from the one with single quotes:

echo '{ "alt-speed-down": 50, }'

conf_file="""{ \"alt-speed-down\": 50, }"""

Here, the backslashes escape the double-quotes, and are removed by Python, so this is equivalent to the first one. Escaping the quotes isn't necessary here, but would be if you had "{ \"alt-speed-down\": 50, }" instead.

If you want to have the backslashes intact in the Python string, you need to use r'' strings, e.g. r'{ \"alt-speed-down\": 50, }' (or the same with double-quotes, r"{ \"alt-speed-down\": 50, }" actually works too, and the backslashes aren't removed, even though they're required to not end the quoted string.)

In the shell, backslashes aren't processed within single quotes, so

echo '\"this is quoted\"'

passes to echo the string \"this is quoted\". Though some implementations of echo would process escapes like \n, regardless of what happens in the shell command line processing.

Whereas with

run(f"""echo '{conf_file}'""", shell=True)

you have no backslashes in sight.

In short, the quoting rules are different between the shells and Python.

See:

Like mentioned in the comments, there's likely better ways of producing JSON (or YAML, or whatever) from Python than manually printing strings. E.g. the json module:

>>> import json
>>> obj = {}
>>> obj["alt-speed-down"] = 50
>>> json.dumps(obj)
'{"alt-speed-down": 50}'
ilkkachu
  • 138,973
  • 2
    Also note that { "alt-speed-down": 50, } is invalid JSON (but valid YAML). They should really be using a library for these sort of things, if they want to do it in Python. In short, this should not be an exercise in string manipulation and quoting, but in data representation and transformation. – Kusalananda Feb 22 '23 at 11:57
  • 1
    @Kusalananda, oh yes indeed. Added a note on that too... – ilkkachu Feb 22 '23 at 13:34
  • Thank you for the answer. It really explains the inner workings of Bash and Python. I've modified my original question in response to your note on json module. – John Smith Feb 22 '23 at 20:50
  • @Kusalananda it's valid JSON5. But yes, agreed that string-pasting is not the way to go :) – hobbs Feb 22 '23 at 21:37
  • 1
    @JohnSmith, yeah, I thought you might know what you're doing, but sometimes the posts that come here make the opposite impression... So people get in the habit of guiding in the choice of tools too, to try to help in the big picture. Of course that might misfire in some cases, but better safe than sorry. Simplifying the situation to focus on the actual question is useful, but it's sometimes hard to know which parts are real and which ones simplified. Like the run("echo ...") there, I expected it's just a placeholder, but it still looks wrong, so I'm not surprised someone asked about it. – ilkkachu Feb 23 '23 at 08:22
  • You are right that the quoting rules are different between the shells and Python and on second thoughts I think that it is simpler to just with open("file_name", 'w') as f: and then write f.write(conf_file) instead of calling echo and tee with Python's run(). – John Smith Feb 23 '23 at 08:36
  • 1
    @JohnSmith, Basically, yeah. Then again, if you need to run doas, like you had in another comment, that might be easier through a shell. But you might consider writing a temporary file from the Python script, and then shelling out doas cat < tmpfile > finalfile. Or doas mv tmpfile finalfile. Passing the data itself through echo is a bit awkward, because you need to be careful with quoting on the shell command line (What if the data has a single quote? Or both single and double quotes?). – ilkkachu Feb 23 '23 at 09:01
  • (Except that doas cat > finalfile doesn't work, so make it doas tee finalfile < tmpfile > /dev/null. The tempfile being the point anyway.) – ilkkachu Feb 23 '23 at 13:52