1

Due to me having to re-authenticate with Kerberos I need to run my long lived processes in tmux. I wrote a script that I thought would work but it didn't. The issue is that after I run the tmux session, instead of running the following commands inside of tmux it puts me into a tmux session and only once I detach does it run the rest of the commands -- but it runs them outside the tmux session while I want it to run inside of the tmux session.

This is my current attempt:

# - get a job id for this tmux session
export SLURM_JOBID=$(python -c "import random;print(random.randint(0, 1_000_000))")
echo SLURM_JOBID = $SLURM_JOBID
export OUT_FILE=$PWD/main.sh.o$SLURM_JOBID
export ERR_FILE=$PWD/main.sh.e$SLURM_JOBID

- CAREFUL, if a job is already running it could do damage to it, rm reauth process, qian doesn't do it so skip it


top -u brando9


pkill -9 reauth -u brando9


- start tmux,


https://unix.stackexchange.com/questions/724877/custom-kerberos-tmux-doesnt-let-me-name-my-sessions-help-forcing-it
tmux new -s $SLURM_JOBID


/afs/cs/software/bin/krbtmux new -s $SLURM_JOBID


cat /afs/cs/software/bin/krbtmux


- reauth


/afs/cs/software/bin/reauth


echo $SU_PASSWORD | /afs/cs/software/bin/reauth


echo 'Secret' | /afs/cs/software/bin/reauth


echo 'totally secret password' | kinit user@DOMAIN.EDU


to see reauth running


top -u brando9


- expt python script


python expts.py &
python -u ~/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/experiment_mains/main_sl_with_ddp.py --manual_loads_name sl_hdb1_5cnn_adam_cl_filter_size --filter_size 4 > $OUT_FILE 2> $ERR_FILE &
export JOB_PID=$!
echo JOB_PID = $JOB_PID


- Echo tmux id, should be jobid


tmux display-message -p '#S'
echo SLURM_JOBID = $SLURM_JOBID


- detach current tmux session


tmux detach &


- wait for pid from python to be done, if done kill this tmux sess


wait $JOB_PID
tmux ls
tmux kill-session -t $SLURM_JOBID


check it was killed


tmux ls


- Done


echo "Done with bash script (experiment or dispatched daemon experiments). "

As a side note, once the python script is running and been dispatched with &, what I want is to kill the tmux session when that job is done. I tried that at the end but I suspect it won't work. I think it won't work because once it runs tmux detach & it will go out of the tmux session and run the wait ... command outside of tmux and block my main terminal. Instead what I want is to run that wait command inside of tmux (to not block me) and once the python script is done to kill the tmux session entirely. But that is just to add context to the final part of my script.

related: How does one authenticate with a command that requires your password in linux?

Charlie Parker
  • 1,567

1 Answers1

4

Tmux does not magically alter the flow of your script

after I run the tmux session, instead of running the following commands inside of tmux it puts me into a tmux session and only once I detach does it run the rest of the commands

A shell script is not a set of directives for the terminal to type something. It's a set of directives for a shell to execute something. So when one of the directives is

tmux new   # i.e. tmux new-session

then the shell executes tmux and sits in the background until this very command exits; only then it moves on to the next command. The command starts a tmux client (that may start a tmux server in the background if needed). A shell started by the tmux client has nothing to do with the shell interpreting the script; it certainly does not read commands from the script.

Similarly ssh user@server, if run in a script, ultimately starts a new (remote) shell that does not "take over" the execution of the script. Even simpler example is cat. If you run sole cat in a shell script, it won't consume the rest of the script. The interpreting shell will wait for the cat, ssh or tmux to exit before continuing the script. From the point of view of the shell the fact that tmux or ssh starts a new shell is irrelevant.

In your case, during tmux new the interpreting shell waits and only after you detach (or otherwise terminate or stop the tmux client) it executes the rest of the script.

Running something inside tmux

it runs them outside the tmux session while I want it to run inside of the tmux session

To run something inside tmux, tell a tmux client what to run:

tmux new-session -d 'shell code here; I mean the job you want'

Thanks to -d the client will start a new session (dispatch the job), but it won't attach the new session to the terminal. The client won't put you into the session, it will exit almost immediately, this will make the shell interpreting the script continue.

If you want to run more jobs inside the session, remember new-window and split-window tmux commands can take shell code as an argument too.

Of course the rest of the script shall not contain the shell code being the job. Things you want to run in tmux you pass to tmux …. Things you want to run by the shell interpreting the script you write directly as lines of the script; this includes tmux …. My point is the shell code being the job you want to run in tmux should be an argument to tmux, not something that the interpreting shell would run directly.

Terminating the tmux session

what I want is to kill the tmux session when that job is done

With the default settings* this happens:

  • a pane in tmux is destroyed just after the process assigned to it exits;
  • a window in tmux is destroyed when it no longer contains any pane (note: a window not split to multiple panes contains exactly one pane);
  • a session in tmux is destroyed when it no longer contains any window;
  • a tmux server is destroyed when it no longer contains any session.

This means most likely you don't need to worry. If you start your job in tmux in the right way (described above) and if you don't create additional panes or windows in the session, then the session will exit automatically when the job is done. Compare this other answer of mine.

Further insight

See this question. It's from a user who (like you) used to think that commands in a script will get to a shell in tmux after certain tmux command. My answer there may give you some further insight.

Example

Run the following script:

#!/bin/sh
tmux new-session -d -s myloop 'while sleep 1; do date; done'
echo "Job dispatched. The script will now terminate."

It will dispatch a job and terminate immediately. Check tmux ls. Wait few seconds, then tmux attach -t myloop and you will see the job has been running since and still is. When attached, terminate the loop (with Ctrl+c); the session (and possibly the tmux server) will terminate automatically*, check tmux ls again to confirm.

*Options that alter the behavior: remain-on-exit, exit-empty.

Kamil Maciorowski
  • 21,864