How can I nest one syntax table in another?

Question

I've written a simple mode for handling JSON. It uses the derived machinery to re-use most of json-mode's code. However one addition is you can insert elisp into the JSON text which is evaluated at JSON submission time. For example an excerpt of the json looks like this:

{
    "parameters": {
        "IRC_USER": "stsquad",
        "PUB_KEY": `(format "\"%s\"" (s-trim (shell-command-to-string "cat ~/.ssh/id_rsa.pub")))`
    }
}

Currently the syntax highlighting of this text is broken as the JSON syntax hightlighter get's thrown by the elisp. I'd like to set-up a nested syntax table so the elisp is properly recognised as elisp when inside the escape characters (I've chosen ` in this case). I understand you can join char-tables (which syntax-tables are built from) with something like:

(defvar lava-mode-syntax-table
  (let ((json-table (copy-syntax-table json-mode-syntax-table))
        (elisp-table (copy-syntax-table lisp-mode-syntax-table)))
    (set-char-table-parent elisp-table json-table)
    (modify-syntax-entry ?` "(`" json-table)
    (modify-syntax-entry ?` ")`" json-table)
    json-table)
  "LAVA Mode syntax table.
This is a combination of json-mode-syntax-table with an escape into
  lisp-mode-syntax table for the embedded elisp.")

But I don't understand how I can modify the syntax table to start using the child (elisp) syntax table while between the escape characters?

Answer 1

Ok, let's get some basics straight.

Nesting syntax tables is possible

Syntax tables don't have to be global to the entire buffer. You can apply them as text properties to specific regions. This means you can indeed apply the elisp syntax table only to regions surrounded by backticks.

How do you do that?

Here's one way you can do that. This method does it immediately before font-lock runs through the buffer, so it should specifically prevent your font-locking issues.

(defun endless/set-syntax-then-fontify (beg end loudly)
  "Apply elisp syntax table to relevant regions before calling font-lock."
  (save-match-data
    (save-excursion
      (save-restriction
        (widen)
        (narrow-to-region beg end)
        (while (search-forward "`" nil 'noerror)
          ;; Using `end' here excludes the `, I don't know which syntax you
          ;; want to apply to that.
          (let ((left (match-end 0)))
            (when (search-forward "`" nil 'noerror)
              (add-text-properties
               left (match-beginning 0)
               (list 'syntax-table emacs-lisp-mode-syntax-table))))))))
  (font-lock-default-fontify-region beg end loudly))

In your major-mode definition, you'll need to add:

(set (make-local-variable 'font-lock-fontify-region-function)
     #'endless/set-syntax-then-fontify)

Syntax tables are not the same as syntax highlighting

The syntax highlighter (the font-lock system) uses Syntax Tables as part of its information, so the solution above should prevent the highlighter from going nuts.

However, that is only part of the data, if you also want the text in backticks to be colored exactly as you would see in an elisp buffer, you'll have to extend the function above to do that specifically.

(defun endless/set-syntax-then-fontify (beg end loudly)
  "Apply elisp syntax table to relevant regions before calling font-lock."
  (save-match-data
    (save-excursion
      (save-restriction
        (widen)
        (narrow-to-region beg end)
        (while (search-forward "`" nil 'noerror)
          ;; Using `end' here excludes the `, I don't know which syntax you
          ;; want to apply to that.
          (let ((left (match-end 0)))
            (when (search-forward "`" nil 'noerror)
              (add-text-properties
               left (match-beginning 0)
               (list 'syntax-table emacs-lisp-mode-syntax-table))))))))
  (font-lock-default-fontify-region beg end loudly)
  ;; Do some specific elisp fontifying here
  (save-match-data
    (save-excursion
      (save-restriction
        (widen)
        (narrow-to-region beg end)
        (while (search-forward "`" nil 'noerror)
          (let ((left (match-end 0)))
            (when (search-forward "`" nil 'noerror)
              ;; Call some function to fontify elisp between `left' and (match-beginning 0)
              )))))))

Answer 2

You don't want to nest one syntax table (which is a vector structure) inside another, you want to set up a buffer where, depending on the position, one syntax table would be used instead of the other.

The other answer describes how to do this using the syntax-table text property. Here's how to do it using one of the "multiple major mode" packages, mmm-mode. It will use the primary mode's everything at the top level of the buffer, and the submode' syntax table, font-lock rules, keymap, etc in the "subregions".

(require 'mmm-auto)
(setq mmm-global-mode 'auto)

(mmm-add-classes
 '((eljson :submode emacs-lisp-mode
           :front ": *\\(`\\)" :back "`"
           :front-match 1
           :face mmm-code-submode-face)))

(mmm-add-mode-ext-class 'json-mode "\\.el\\.json\\'" 'eljson)

This assumes that your mixed-mode files are named *.el.json. Adjust as appropriate.

Now, install mmm-mode, evaluate the above and (only then) open one of the files in question.