1

This is closely related to a previous question I asked here: SMIE Basic Identation and Block Example

However, it is sufficiently different (and already big enough) that I might as well ask it as a separate question. Those answers may still be useful for others anyways.

Anyways, given the (corrected) SMIE grammar and rules from the previous question:

(defvar smie-sample-grammar
  (smie-prec2->grammar
   (smie-bnf->prec2
    `((insts (insts ";" insts) (inst))
      (inst ("AttributeBegin" inst "AttributeEnd")))
    '((assoc ";"))))
  "Sample BNF grammar for `smie'.")

(defun smie-sample-rules (kind token)
  "Perform indentation of KIND on TOKEN using the `smie' engine."
  (princ (list kind token))
  (pcase (list kind token)
    (`(:elem basic) smie-sample-indent-offset)
    (`(:elem arg) 0)
    (`(:after "AttributeEnd") (smie-rule-parent))))

Our toy major-mode from the previous question now works. However, if we extend it slightly, e.g., by adding arguments to each instruction that can trail multiple lines, they are indented as expected in almost all cases:

test a
     b
     c;
test a;
test a
     b;
AttributeBegin
    test c
         d;
    AttributeBegin
        test c
             d;
    AttributeEnd
    test c
    d;  # <- Not lining up as expected.
    test c
         d;
AttributeEnd

Only the one instruction marked above is unexpectedly indented (and others that trail directly after an "AttributeEnd" statement). I tried to use smie-edebug to debug the indentation of that particular line, but for some reason SMIE doesn't even refer to the rule-set in that case.

Interestingly, adding a semi-colon after "AttributeEnd" and before the next statement causes the indentation the line up-correctly.

Is there any particular reason that statement should behave that way and is there something that can be added to the configuration for this to behave as intended?

I've updated the gist if anyone wants to take a look at the code.

Xaldew
  • 1,181
  • 9
  • 20

3 Answers3

1

In

    AttributeBegin
        test c
             d;
    AttributeEnd
    test c
    d;

The Att..Begin...Att..End is just a (parenthesized) "expression", like test, c, and d because it's not separated from them with any keyword (which is why the same problem doesn't show up when you add a semi-colon between them). So SMIE considers the above as 4 expressions next to each other, just like:

    foo
    test c
    d;

test is taken to be the first arg to "foo" and is aligned with foo because of your rule (`(:elem arg) 0) and then all subsequent args (c and d) are aligned with the first.

Stefan
  • 26,154
  • 3
  • 46
  • 84
  • That explains it pretty well, but is there anything that can be done for it to work as I intended? – Xaldew Jul 29 '19 at 09:20
  • I faced a similar problem in [sml-mode](http://elpa.gnu.org/packages/sml-mode.html) and managed to work around it with an ad-hoc indentation rule, but the way it presents itself here is different and the best I can think of is to change the lexer so that it returns `";"` when reading the newline that follows `AttributeEnd`. – Stefan Jul 29 '19 at 13:29
  • That's something like the answer below right? – Xaldew Jul 29 '19 at 14:19
1

As suggested by Stefan in his comment, it seems like the best thing to do here is to modify the lexer(s) to return a virtual separator after "AttributeEnd" statements:

(defun smie-sample-forward-token ()
  "Go forwards to the next SMIE token."
  (let ((start-pos (point)))
    (forward-comment (point-max))
    (cond
     ((and (> (point) start-pos)           ; Emit virtual statement separator.
           (looking-back "AttributeEnd[ \t\n]+" nil))
      ";")
     (t
      (buffer-substring-no-properties
       (point)
       (progn (if (zerop
                   (skip-syntax-forward "."))
                  (skip-syntax-forward "w_'"))
              (point)))))))

(defun smie-sample-backward-token ()
  "Go backwards to the previous SMIE token."
  (let ((start-pos (point)))
    (forward-comment (- (point)))
    (cond
     ((and (< (point) start-pos)         ; Emit virtual statement separator.
           (looking-back "AttributeEnd" nil))
      ";")
     (t
      (buffer-substring-no-properties
       (point)
       (progn (if (zerop
                   (skip-syntax-backward "."))
                  (skip-syntax-backward "w_'"))
              (point)))))))

(define-derived-mode smie-sample-mode prog-mode "SMIE"
  "Major mode for trying out SMIE samples."
  ;; ...
  (smie-setup smie-sample-grammar #'smie-sample-rules
              :forward-token #'smie-sample-forward-token
              :backward-token #'smie-sample-backward-token))

I was actually hoping to avoid this kind of solution, as it felt a bit complex for such a "simple" example. This however, actually solves all the cases outlined in the original question.

Xaldew
  • 1,181
  • 9
  • 20
  • 1
    That's indeed what I was suggesting. It's "complex" and hackish, but otherwise there's no "keywords" to which to attach *separately* the `AttributeBegin...AttributeEnd` expression and the `test c d`. It's always possible to deal with it in the `smie-rule-function` instead, but it tends to be much more messy in the long run. – Stefan Jul 29 '19 at 17:37
1

You can also parse the empty string behind AttributeEnd as ;. Point behind AttributeEnd is the corresponding parser state. The only tricky detail is that you must flee from that position when you have parsed ; in forward direction and you may only parse the arrival at AttributeEnd as ; when you are parsing in backward direction.

Minimal working example:

(require 'smie)

;; We are using SMIE's default lexer.
;; A token is:
;; 1. any sequence of characters that have word or symbol syntax
;; 2. any sequence of characters that have punctuation syntax

(defvar smie-sample-grammar nil
  "Sample BNF grammar for `smie'.")

(setq smie-sample-grammar
  (smie-prec2->grammar
   (smie-bnf->prec2
    `((insts (insts ";" insts) (inst))
      (inst ("AttributeBegin" insts "AttributeEnd")))
    '((assoc ";")))))

(defun smie-sample-rules (kind token)
  "Perform indentation of KIND on TOKEN using the `smie' engine."
  (pcase (list kind token)
    ('(:after "AttributeEnd")
     (smie-rule-parent))
    ('(:elem arg) 1)))

(defun smie-sample-backward-token ()
  "Imitate `smie-default-backward-token.
Additionally send a semi-colon right after AttributeEnd."
  (if
      (and ;; first arrival at AttributeEnd
       (null (looking-back "\\_<AttributeEnd" (line-beginning-position)))
       (progn (forward-comment -1)
          (looking-back "\\_<AttributeEnd" (line-beginning-position))))
      ";"
    (smie-default-backward-token)))

(defun smie-sample-forward-token ()
  "Imitate `smie-default-forward-token.
Additionally send a semi-colon right after AttributeEnd."
  (if (looking-back "\\_<AttributeEnd")
      (prog1 ";"
    (forward-comment (point-max)))
    (smie-default-forward-token)))

(define-derived-mode smie-sample-mode prog-mode "ExSMIE"
  "Usage example for the Simple Minded Indentation Engine."
  :syntax-table nil
  (modify-syntax-entry ?\# "<")
  (modify-syntax-entry ?\n ">")
  (setq comment-start "#"
    comment-end "")
  (smie-setup smie-sample-grammar #'smie-sample-rules
          :backward-token #'smie-sample-backward-token
          :forward-token #'smie-sample-forward-token)
  (font-lock-add-keywords nil '(("AttributeBegin" . font-lock-keyword-face)
                ("AttributeEnd" . font-lock-keyword-face)))
  (font-lock-mode)
  (font-lock-ensure (point-min) (point-max)))

Emacs version: GNU Emacs 26.2 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.22.30) of 2019-04-12

Test:

test a
     b
     c;
test a;
test a
     b;
AttributeBegin
    test c
         d;
    AttributeBegin
        test c
             d;
        second a;
        third b;
        one c
            d;
    AttributeEnd
    test c
         d;  # <- Not lining up as expected.
    test c
         d;
AttributeEnd
Tobias
  • 32,569
  • 1
  • 34
  • 75