15

I have been using Emacs with ESS and tramp to load remote files and run remote R processes on HPC computers. With ssh keys, tramp makes this easy with C-x C-f to find file at /ssh:myserver:/path/to/file followed by M-x R to launch an R session.

However, on HPC computers that use a job scheduling system (e.g. Sun Grid Engine or Torque) there are restrictions on how much memory and CPU can be used on the head node, so I have to launch an interactive session using qsub -I.

Is it possible to configure tramp, ssh, and/or Emacs to launch an interactive session on a slave node (using qsub -I or equivalent) after an ssh connection is made?

David LeBauer
  • 433
  • 2
  • 10
  • I'm not entirely sure what the limitations are? – PythonNut Jun 04 '15 at 18:13
  • 1
    @PythonNut I can ssh to the head node using tramp, but do not know how to then connect to the slave node (which is not directly accessible via ssh). – David LeBauer Jun 04 '15 at 18:29
  • It does not seem to be possible to tell ESS to launch a different shell first, but with `tramp-remote-shell` set to `qrsh` or `qlogin` you might be able to get an interactive cluster session. –  Jun 04 '15 at 18:53

3 Answers3

7

It sounds like you should extend tramp-methods, adding a new method which is similar to the sudo method but uses qsub -I instead. You would then be able to use a multi-hop tramp path to first connect to the head node and then to connect to a compute node.

Try this:

(add-to-list 'tramp-methods
  '("qsub"
    (tramp-login-program        "qsub")
    (tramp-login-args           (("-I"))) ; other options here?
    ;; Local $SHELL could be a nasty one, like zsh or fish.  Let's override it.
    (tramp-login-env            (("SHELL") ("/bin/sh")))
    (tramp-remote-shell         "/bin/sh")
    (tramp-remote-shell-args    ("-c"))
    (tramp-connection-timeout   10)))

You would then use a tramp path like /ssh:myserver|qsub:myserver:/path/to/file.

You may need to specify other options to pass to qsub; I've not used it so I can't help you there. You'll also want to look over the other options you can specify on a tramp method; there are a couple of dozen listed in tramp.el.

db48x
  • 15,741
  • 1
  • 19
  • 23
  • Would it be possible to request that a command be run first thing after connecting with `qsub`? I would like to enable a package by calling `module load ...`, a single line command. Thanks! – Lino Ferreira May 12 '22 at 16:25
6

I have been struggling to do the same. I found a different route that allowed me to do basically the same thing.

You can open a shell buffer by M-x shell and from there connect to the login node, and then connect to the interactive session by qsub -I. Once you're in the interactive session, start an R session by typing the command R. There, you can do M-x ess-remote. This will prompt a mini-buffer asking which program you want to run (R, S+, Stata, etc.) Once you select R, then you can use the R session on the remote shell just as you use ESS in the local machine.

I personally open Rscripts saved in the remote node by tramping into it (in a separate window) and work with the R session connected by the ess-remote as described above. This way, the R session can directly interact with the Rscripts that I am working with. (For example, source('code.R') command in R will be able to read the 'code.R' in the HPC cluster, which I might have just edited. If I was editing Rscripts in my local machine instead, the ess-remote session of R wouldn't be able to read them unless I uploaded them to the remote node each time.)

Joonha Park
  • 61
  • 1
  • 3
1

I am using a cluster with pjsub rather than qsub, but I have had success with the following. First, create an expect script called, say, inter_R.sh:

#!/usr/bin/expect -- 
set cmdargs [lrange $argv 0 end]
set timeout 130
spawn -nottyinit pjsub --interact -g groupname -L rscgrp=interactive --sparam wait-time=120 -X --X11 
expect -ex "username/ #$ "
send "stty -echo\n"
send "echo /usr/bin/R $cmdargs \n"
send "exec /usr/bin/R $cmdargs \n"
interact

Then in emacs customize, set "Inferior Ess R Program" to be /home/username/inter_R.sh. Now, use tramp to open a connection to a login node on the cluster (say, a directory listing of /home/username). Now do M-x R to start R in that tramp buffer. This will run inter_R.sh on the login node, which will start an interactive job on a compute node and run R on that compute node.

An important argument to pjsub is the -X, which passes all the environment variables through. Without this the shell prompt on the compute node is different than what emacs is expecting because $PS1 isn't passed through. Hopefully qsub would have some equivalent.

Eric Weese
  • 11
  • 1