Ubuntu-latest GitHub runner cannot Connect to localhost API

Question

What specifically needs to be changed in the below in order for a program running on an Ubuntu-latest GitHub runner to successfully connect to a Twisted web server running on localhost of the same GitHub runner?

The same code works on a Windows laptop, with only minor changes like using where twistd on Windows versus which twistd on Ubuntu as shown below.

SOME RELEVANT CODE:

The code to start the Twisted web server on localhost is in a script called twistdStartup.py that reads as follows:

import subprocess
import re
import os
escape_chars = re.compile(r'\x1B[[0-?][ -/][@-~]')
def runShellCommand(commandToRun, numLines=999):
  proc = subprocess.Popen( commandToRun,cwd=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
  while True:
    line = proc.stdout.readline()
    if line:
      if numLines == 1:
        return escape_chars.sub('', line.decode('utf-8').rstrip('\r|\n'))
      else:
        print(escape_chars.sub('', line.decode('utf-8').rstrip('\r|\n')))
    else:
      break
print("About to create venv.")
runShellCommand("python3 -m venv venv")
print("About to activate venv.")
runShellCommand("source venv/bin/activate")
print("About to install flask.")
runShellCommand("pip install Flask")
print("About to install requests.")
runShellCommand("pip install requests")
print("About to install twisted.")
runShellCommand("pip install Twisted")
##Set environ variable for the API
os.environ['PYTHONPATH'] = '.'
print("Done updating PYTHONPATH.  About to start server.")
twistdLocation = runShellCommand("which twistd", 1)
startTwistdCommand = twistdLocation+" web --wsgi myAPI.app  &>/dev/null & "
print("startTwistdCommand is: ", startTwistdCommand)
subprocess.call(startTwistdCommand, shell=True)
print("startTwistdCommand should be running in the background now.")

The code in a calling program named startTheAPI.py that invokes the above twistdStartup.py is:

myScript = 'twistdStartup.py'
print("About to start Twisted.")
subprocess.Popen(["python", myScript], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
print("Just finished starting Twisted.")

The logs produced by that step in the GitHub Action job are as follows:

About to start Twisted.
Just finished starting Twisted.

The results of running the curl command on the same Ubuntu-latest GitHub runner in the next step of the same job after waiting 30 seconds for startup are as follows:

$ curl http://localhost:1234/api/endpoint/
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to localhost port 1234 after 1 ms: Connection refused

The contents of a callTheAPI.py program that runs the curl command would look like:

import subprocess
import re
import os
escape_chars = re.compile(r'\x1B[[0-?][ -/][@-~]')
def runShellCommand(commandToRun, numLines=999):
  proc = subprocess.Popen( commandToRun,cwd=None, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
  while True:
    line = proc.stdout.readline()
    if line:
      if numLines == 1:
        return escape_chars.sub('', line.decode('utf-8').rstrip('\r|\n'))
      else:
        print(escape_chars.sub('', line.decode('utf-8').rstrip('\r|\n')))
    else:
      break
print("About to call API.")

runShellCommand("curl http://localhost:1234/api/endpoint/")
print("Done calling API.")

SUMMARY:

The twistdStartup.py script is running, but is failing to provide any output to the logs despite all the print() commands. The curl command fails to connect to the correctly-stated http://localhost:1234/api/endpoint/

Answer 1

Regarding:

The curl command fails to connect to the correctly-stated http://localhost:1234/api/endpoint/

The main problem here is that every command you run with runShellCommand("somecommand") is executed in a new shell so all changes such as functions,variables, environment variables, etc. disappear after the commands is executed.

For example, try running this simple script in your twistdStartup.py:

print("Running echo $PATH")
runShellCommand("echo $PATH")
print("Running PATH=/")
runShellCommand("PATH=/")
print("Running echo $PATH")
runShellCommand("echo $PATH")

The output (in my case):

Running echo $PATH
/home/edgar/.local/bin:/home/edgar/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/go/bin
Running PATH=/
Running echo $PATH
/home/edgar/.local/bin:/home/edgar/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/go/bin

As you can see above the assignment Running PATH=/ is ignored when I run again runShellCommand("echo $PATH").

The possible solution (untested with all your commands) here is that you wrap all your runShellCommand method calls in a single method call (or convert the code to a shell script), like this:

Section twistdStartup.py:

runShellCommand(
"""
set -e
echo About to create venv.
python3 -m venv venv
echo About to activate venv.
. venv/bin/activate
echo About to install flask.
pip install Flask
echo About to install requests.
pip install requests
echo About to install twisted.
pip install Twisted
export PYTHONPATH='.'
echo Done updating PYTHONPATH.  About to start server.
twistdLocation=$(which twistd)
echo "Running ${twistdLocation} web --wsgi customControllerAPI.app  &>/dev/null &"
( 
$twistdLocation web --wsgi myAPI.py  >/dev/null 2>&1
)&
echo startTwistdCommand should be running in the background now.
""")

Testing on GitHub Actions I noticed that source venv/bin/activate caused a fail because source was not valid (possibly the default shell for Ubuntu in GitHub is dash).
Instead of using source instead you have to use . (which is far better): . venv/bin/activate.
Because of the error above the command: which twistd was not working correctly because the venv was not sourced. Thus the command: $twistdLocation web --wsgi customControllerAPI.app &>/dev/null will fail and the Flask API will never run (for that reason you get the messahe: Failed to connect to localhost port 1234 after 1 ms: Connection refused)

Regarding:

but is failing to provide any output to the logs despite all the print() commands

By looking your file api_server.py I see you are calling the python script setup.py:

apiServer = subprocess.Popen(["python", "setup.py"], stdout=subprocess.DEVNULL, cwd=str(path))

Here you are not getting the output the command python setup.py, so I suggest you remove that line and add these ones:

    apiServer = subprocess.Popen(["python", "setup.py"], cwd=str(path),stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    while True:
        line = apiServer.stdout.readline()
    if line:
        print(self.escape_chars.sub('', line.decode('utf-8').rstrip('\r|\n')))
    else:
      break

---

1. You may want to change the line twistdLocation=$(which twistd) to twistdLocation=$(command -v twistd). See Why not use "which"? What to use then?
2. Also I suggest you add the line set -e in the first line of the script. That command is used to abort the execution of the following commands if some error has been thrown (so in your case if some dependencies fails to install itself then the scripts stops).
3. About ($twistdLocation web --wsgi myAPI.py >/dev/null 2>&1 )& I used that in order to prevent to the python subprocess from waiting to read output from that command since this one was causing to the server a "infinity" stuck).
4. If you are interested in the logs (stdout and stderr) of the command: $twistdLocation ... you should redirect the output to a file, like this:

( 
$twistdLocationn web --wsgi myAPI.py  >/tmp/twistd.logs 2>&1
)&

Also you will have to edit your github action to include the command which cat the content of /tmp/twistd.logs, like this:

steps:
  - uses: actions/checkout@v3
  - shell: bash
    name: Start the localhost Server
    run: |
      echo "Current working directory is: "
      echo "$PWD"
      python code/commandStartServer.py
      echo "Checking twistd logs"
      cat /tmp/twistd.logs

---

So in these files you have to have the following code:

api_server.py

import os
import re
class api_server:
def init(self):

    pass
escape_chars = re.compile(r'\x1B[[0-?][ -/][@-~]')
def startServer(self):
    path = os.getcwd()+"/api/"
    print(path)
    print("About to start .")
    import subprocess
    #The version of command on next line runs the server in the background.  Comment it out and replace with the one below it if you want to see output.
    apiServer = subprocess.Popen(["python", "setup.py"], cwd=str(path),stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    while True:
        line = apiServer.stdout.readline()
    if line:
        print(self.escape_chars.sub('', line.decode('utf-8').rstrip('\r|\n')))
    else:
      break

#The second version of the command below will print output to the console, but will also halt your program because it runs the server in the foreground.
#apiServer = subprocess.Popen(["python", "setup.py"], cwd=str(path))
print("Just finished starting .")


def destroyServer(self):
    ... # your current code

setup.py

import subprocess
import re
import os
escape_chars = re.compile(r'\x1B[[0-?][ -/][@-~]')
def runShellCommand(commandToRun, numLines=999):
  proc = subprocess.Popen( commandToRun,cwd=None,  shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  while True:
    line = proc.stdout.readline()
    if line:
      if numLines == 1:
        return escape_chars.sub('', line.decode('utf-8').rstrip('\r|\n'))
      else:
        print(escape_chars.sub('', line.decode('utf-8').rstrip('\r|\n')))
    else:
      break
runShellCommand(
"""
set -e
echo About to create venv.
python3 -m venv venv
echo About to activate venv.
. venv/bin/activate
echo About to install flask.
pip install Flask
echo About to install requests.
pip install requests
echo About to install twisted.
pip install Twisted
export PYTHONPATH='.'
echo Done updating PYTHONPATH: $PYTHONPATH.  About to start server.
twistdLocation=$(which twistd)
echo "Running ${twistdLocation} web --wsgi customControllerAPI.app  &>/dev/null &"
( 
$twistdLocation web --wsgi myAPI.py  >/dev/null 2>&1
)&
echo startTwistdCommand should be running in the background now.
""")

localhost-api.yml

name: localhost-api
on:
  push:
    branches:
      - main
jobs:
  start-and-then-call-localhost-api:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - shell: bash
        name: Start the localhost Server
        run: |
          echo "Current working directory is: "
          echo "$PWD"
          python code/commandStartServer.py
          echo "Checking twistd logs"
          cat /tmp/twistd.logs
      - shell: bash
        name: Call the localhost API
        run: |
          echo "Current working directory is: "
          echo "$PWD"
          python code/commandCallAPI.py
      - shell: bash
        name: Destroy the localhost Server
        run: |
          echo "Current working directory is: "
          echo "$PWD"
          python code/commandDestroyServer.py