7

I had always thought that shells parse whole scripts, constructing an AST, and then execute that AST from memory. However, I just read a comment by Stéphane Chazelas, and tested executing this script, edit-while-executing.sh:

#!/bin/bash

echo start
sleep 10

and then while it was sleeping running:

$ echo "echo end" >> edit-while-executing.sh

and it worked to cause it to print "end" at the end.

However, when trying to modify this:

#!/bin/bash

while true; do
  echo yes
done

by doing:

$ printf "%s" "no " | dd of=edit-while-executing.sh conv=notrunc seek=35 bs=1

It didn't work, and kept printing "yes".

I also wondered if other non-shell interpreters also worked like this, and tried the equivalent of the first script with python, but it didn't work. Though, maybe python is not an interpreter anymore and it's more of a JIT compiler.

So to reiterate my question, is this a behaviour ubiquitous to shells and limited to them or also present in other interpreters (those not regarded as shells)? Also how does this work such that could I do the first modification but not the second?

JoL
  • 4,735
  • The shell had, in your looping example, already parsed the complete compound command (the for loop), so the change to the shell script's file had no effect. – Kusalananda Jun 13 '18 at 16:04
  • 1
    If shells had to read their whole input before starting to do anything, you wouldn't be able to use them interactively. shells are primarily intended to be run interactively. scripts are secondary. – Stéphane Chazelas Jun 13 '18 at 16:08
  • @StéphaneChazelas I hadn't thought that maybe python did something differently to run interactively. – JoL Jun 13 '18 at 16:09
  • 1
    Shells in some cases do parse scripts completely before doing anything. This applies e.g. to the dot command, so . myscript will first parse myscript as one compound statement and then execute it. This is e.g. a reason why aliases defined in such a script are not active inside the script already. – schily Jun 13 '18 at 16:11
  • Related: How to read the whole shell script before executing it? – Stéphane Chazelas Jun 13 '18 at 16:11
  • 1
    @schily, that very much depends on the shell. That's not the case with ash, pdksh or zsh for instance. – Stéphane Chazelas Jun 13 '18 at 16:12
  • @StéphaneChazelas I had thought that it would be quite curious for this behaviour to be ubiquitous in shells but not other interpreters. The choice might be due to that difference in use case, that shells are used primarily interactively but not other interpreters. – JoL Jun 13 '18 at 16:15
  • Since the Bourne Shell, ksh88 and ksh93 do it, there is a big indication and since we mention this in the POSIX standard in the current alias documentation, it is even commonly accepted. – schily Jun 13 '18 at 16:15
  • BTW: this first parse the whole thing and then execute the compound statement is also true for e.g. eval and commands that are executed for command substitution. You may verify this by checking for the NLFLG as parameter to the parser that tells the parser to treat newlines as semicolons. – schily Jun 13 '18 at 16:21
  • @schily - the spec also requires \<newline> processing occur first in every parsing case, and so eval - which must reparse - must correctly handle backslash escaped newlines. if you amd your peers are devolving that capability in the interest of compiled in hacks and special sauce, stop. – mikeserv Jan 22 '19 at 02:54

3 Answers3

3

This feature is present in other interpreters that offer what is called a read eval print loop. LISP is a pretty old language with such a feature, and Common LISP has a read function that will read in here the expression (+ 2 2) which can then be passed to eval for evaluation (though in real code you may not want to do it this way for various security reasons):

% sbcl
* (defparameter sexp (read))
(+ 2 2)

SEXP
* (print (eval sexp))

4
4

we can also define our own very simple REPL without much in the way of features or debugging or pretty much anything else, but this does show the REPL parts:

* (defun yarepl () (loop (print (eval (read))) (force-output) (fresh-line)))

YAREPL
* (yarepl)
(* 4 2)

8
(print "hi")

"hi"
"hi"

Basically like it says on the nameplate data is read in, evaluated, printed, and then (assuming nothing crashed and there is still electricity or something powering the device) it loops back to the read No need to build an AST up in advance. (SBCL needs the force-output and fresh-line additions for display reasons, other Common LISP implementations may or may not.)

Other things with REPL include TCL ("a shell bitten by a radioactive LISP") which includes graphics stuff with Tk

% wish
wish> set msg "hello"
hello
wish> pack [label .msg -textvariable msg]
wish> wm geometry . 500x500
wish> exit

Or FORTH here to define a function f>c to do temperature conversion (the " ok" are added by gforth):

% gforth
Gforth 0.7.3, Copyright (C) 1995-2008 Free Software Foundation, Inc.
Gforth comes with ABSOLUTELY NO WARRANTY; for details type `license'
Type `bye' to exit
: f>c ( f -- c ) 32 - 5 9 */ cr . cr ;  ok
-40 f>c
-40
 ok
100 f>c
37
 ok
bye
thrig
  • 34,938
  • Also in tcl when invoked as tclsh < file / expect < file (not tclsh file / expect file). – Stéphane Chazelas Jun 13 '18 at 21:14
  • @StéphaneChazelas I'm wondering now if LISP was the first or if anything had a repl before it... but that's probably more a retro computing question – thrig Jun 13 '18 at 22:04
2

So, this runs indefinitely in Bash/dash/ksh/zsh (or at least until your disk fills up):

#!/bin/sh
s=$0
foo() { echo "hello"; echo "foo" >> $s; sleep .1; }
foo

The thing to note, is that only stuff appended added to the script file after the last line the shell has read matters. The shells don't go back to re-read the earlier parts, which they even couldn't do, if the input was a pipe.

The similar construct doesn't work in Perl, it reads the whole file in before running.

#!/usr/bin/perl -l    
open $fd, ">>", $0;
sub foo { print "hello"; print $fd 'foo;' }
foo;

We can see that it does so also when given input through a pipe. This gives a syntax error (and only that) after 1 second:

$ (echo 'printf "hello\n";' ; sleep 1 ; echo 'if' ) | perl

While the same script piped to e.g. Bash, prints hello, and then throws the syntax error one second later.

Python appears similar to Perl with piped input, even though the interpreter runs a read-eval-print loop when interactive.

In addition to reading the input script line-by-line, at least Bash and dash process arguments to eval one line at a time:

$ cat evaltest.sh
var='echo hello
fi'
eval "$var"
$ bash evaltest.sh
hello
evaltest.sh: eval: line 4: syntax error near unexpected token `fi'
evaltest.sh: eval: line 4: `fi'

Zsh and ksh give the error immediately.

Similarly for sourced scripts, this time Zsh also runs line-by-line, as do Bash and dash:

$ cat sourceme.sh
echo hello
fi
$ zsh -c '. ./sourceme.sh'
hello
./sourceme.sh:2: parse error near `fi'
ilkkachu
  • 138,973
  • "the shells don't go back to re-read the earlier parts" -- consider however perl -e 'if(fork()){exec qw/sh/}else{while(1){sleep 1;sysseek STDIN,0,0}}' < foo.sh – thrig Jun 13 '18 at 22:52
  • @roaima, ah yes, of course. my bad. – ilkkachu Jun 14 '18 at 07:18
  • 1
    @thrig, argh! That's not really the shell "going back", but I do appreciate the awfulness of the idea. – ilkkachu Jun 14 '18 at 07:23
1

At least one shell, fish, doesn't exhibit this behaviour (but then fish is unusual in other ways):

% for sh in zsh mksh fish dash bash tcsh; do echo 'sleep 5' > foo.$sh; $sh foo.$sh & sleep 1; echo 'echo $0' >> foo.$sh; fg; done
[2] 7381
[2]  - 7381 running    $sh foo.$sh
foo.zsh
[2] 7385
[2]  - 7385 running    $sh foo.$sh
foo.mksh
[2] 7387
[2]  - 7387 running    $sh foo.$sh
[2] 7390
[2]  - 7390 running    $sh foo.$sh
foo.dash
[2] 7393
[2]  - 7393 running    $sh foo.$sh
foo.bash
[2] 7415
[2]  - 7415 running    $sh foo.$sh
foo.tcsh

(A previous version of this answer had mistaken observations of Python and Ruby.)

muru
  • 72,889
  • It doesn't work with me with the same python version. I think the difference is that I add the statement seconds after and you do it nearly immediately. It's probably added the statement before python has finished parsing the file. – JoL Jun 13 '18 at 16:20