Async execution in org babel

Question

Is there a good general customization of org-babel to run asynchronously? Recently I plan to use MATLAB via org-babel, but I would like it in an async manner, since some calculations do take time.

I do not wish to customize ob-matlab only. This is because I think it should be done in the level of framework instead of an application. In other words, a same modification should enable async feature for other language extensions e.g. R language.

Does anyone have a good solution? So far I have tried async.el as well as deferred.el to modify org-babel-execute-safely-maybe that can be found in ob-core.el at the moment.

Answer 1

For future references and out-of-box libraries. I will accept this one for the answer because these are more recent.

ob-async

Here's a library using async.el https://github.com/linktohack/ob-async

org-babel-eval-in-repl

My other solution (available in melpa): https://github.com/diadochos/org-babel-eval-in-repl

Answer 2

I have so far discovered that spawning a new Emacs process is a solution.

Here is what I have done.

1. Add a function to start an external emacs process.

init.el

(defvar my/async-emacs-repl-org-babel-init-file "~/.emacs.d/org-babel-async-init" "File to load on executing async babel evaluation.")

(defun my/async-emacs-repl--start (process-name init-file)
  "Start a new Emacs process as a REPL server."
  (async-shell-command (concat
                        "TERM=vt200 emacs --batch -nw"
                        " --eval '(load \"" init-file "\")'"
                        " --eval '(while t (print (eval (read))))'"
                        )
                       process-name))

(defun my/async-emacs-repl--org-babel--start-server ()
  "Starts an Emacs process for async org-babel execution."
  (my/async-emacs-repl--start "*org-babel-async*" my/async-emacs-repl-org-babel-init-file))

(defun my/async-emacs-repl--org-babel--start-if-not-exists ()
  "Starts an Emacs process if the process does not exist."
  (if (not (get-buffer-process "*org-babel-async*")) (my/async-emacs-repl--org-babel--start-server)))

(defun my/async-emacs-repl--org-babel--execute--build-command (file-name line-number)
  "Build the command for executing `org-babel-execute-src-block'."
  (concat
   "(progn"
   " (find-file \"" file-name "\")"
   " (revert-buffer t t)"
   " (goto-line " (number-to-string line-number) ")"
   " (org-babel-execute-src-block t)"
   " (save-buffer)"
   ")"
   "\n"))

(defun my/async-emacs-repl--org-babel--execute (process-name file-name line-number)
  "Sends the command to the server to run the code-block the cursor is at."
  (process-send-string
   process-name
   (my/async-emacs-repl--org-babel--execute--build-command file-name line-number)))

(defun my/async-emacs-repl-org-babel-do-execute ()
  "Run org babel execution at point."
  (my/async-emacs-repl--org-babel--execute "*org-babel-async*" (buffer-file-name) (line-number-at-pos)))

(defun my/async-emacs-repl-org-babel-execute ()
  "Run by the user. Executes command. Starts buffer if not exists."
  (interactive)
  (save-buffer)
  (my/async-emacs-repl--org-babel--start-if-not-exists)
  (my/async-emacs-repl-org-babel-do-execute))

2. Add a config file to load in the new emacs process.

The function above starts emacs in the --batch mode. Thus the normal init.el will not be loaded.

Instead, we want to create a shorter configuration file (to load paths and so on).

The path to our new config file is stored in async-emacs-repl-org-babel-init-file in the snippet above.

org-babel-async-init.el

;; 1
(package-initialize)

;; 2
(setq org-confirm-babel-evaluate nil)

;; 3
(let ((my/org-babel-evaluated-languages
       '(emacs-lisp
         ditaa
         python
         ruby
         C
         matlab
         clojure
         sh
         dot
         plantuml)))
  (org-babel-do-load-languages
   'org-babel-load-languages
   (mapcar (lambda (lang)
             (cons lang t))
           my/org-babel-evaluated-languages)))

Here we ...

1. Add package paths.
2. Tell org-mode to not ask whether to execute code block.
3. Tell org-babel which languages are necessary.

Footnote 1: Without this setting, the evaluation will fail with "No org-babel-execute function for $lang!"

Footnote 2: Of course you can load the normal init.el instead of creating a new config file, if you wish. Do that by adding (setq org-babel-async-init-file "~/.emacs.d/init") to your init.el. But I think creating a configuration file for this task is more straightforward.

3. Additionally...

Add to init.el

;; This will stop the new process buffer from getting focus.
(setq display-buffer-alist (append display-buffer-alist '(("*org-babel-async*" display-buffer-no-window))))

;; This will automatically show the result section.
(global-auto-revert-mode 1)

Add to org-babel-async-init.el

;; This will skip the "Save anyway?" confirmation of automatically saving the file when you also edited the buffer from Emacs while an asynchronous process is running.
(defun advice:verify-visited-file-modtime (orig-func &rest args) t)
(advice-add 'verify-visited-file-modtime :around 'advice:verify-visited-file-modtime)

;; This will skip the "Select coding system" prompt that appears when the result is inserted. This may vary among environments.
(setq coding-system-for-write 'utf-8)

;; This will skip the "changed on disk; really edit the buffer?" checking.
(defun ask-user-about-supersession-threat (fn) "blatantly ignore files that changed on disk")

Add to org-babel-async-init.el (you may not need these. These are for MATLAB)

;; This will set MATLAB cli path.
(setq-default matlab-shell-command "/Applications/MATLAB_R2016a.app/bin/matlab")
;; The MATLAB cli path can be obtained by running `fullfile(matlabroot, 'bin')` in your MATLAB.

;; This will stop MATLAB from showing the splash (the MATLAB logo) at the beginning.
(setq-default matlab-shell-command-switches '("-nodesktop" "-nosplash"))

Add to org-babel-async-init.el (you may not need these. These are for Julia, R and other languages that use ESS.)

;; This will enable :session header in Julia and other languages that use ESS (Emacs speaks statistics).
(load "/path/to/ess-site")
;; This will suppress ESS from prompting for session directory.
(setq ess-ask-for-ess-directory nil)

4. Usage

(After the setup above.)

1. Move cursor to the code snippet you want to execute.
2. Run M-x my/async-emacs-repl-org-babel-execute (instead of doing C-c C-c). This will start an external Emacs process as a REPL server if needed, and then execute the source block you are at.

Acknowledgments

I have learned the idea of starting an emacs process for org-babel evaluation from this post. I would like to thank the author.

Comments for customization

The idea here is simple. Start a new emacs process as a REPL for Elisp, do find-file to the same .org file we are editing, goto-line to the same cursor point, run org-babel-execute-src-block, save-buffer. Stop exiting until the user stops the process (Otherwise, graphs would disappear immediately after they are shown). One can naturally think about extending this by:

• Using org-mode's C-c C-c instead of running functions by hand / setting a new keybind (which can be achieved by advices).
• Conditionally switching process name according to :session variable and the language
• Conditionally switching init files based on the language.

In fact, the success of this approach seems to me to be showing a general way of developing async features in Emacs. Creating a "commands" layer, add scripts to do tasks, and have a framework for starting and reusing emacs processes. Just like Symfony framework of PHP (PHP doesn't have threads) has Command features.

Edit history

Refactored code (2016-04-02). Solution now reuses an Emacs process (2016-04-02). Solution now simplified and has only one interactive command to run (2016-04-02. Added configuration (2016-04-12).