How to auto calculate the begin and end lines when including source files in org-mode?

Question

I have the below in my documentation:

#+INCLUDE: "code/basic.sv" :src systemverilog :lines "14-117"

Here line 14 is where I have class basic extends .. and line 116 is where I have endclass.

Is there a way to auto insert the numbers 14 and 117 (=116+1) so that I don't have to manually update them every time I modify the code/basic.sv?

No. That was an example. I am thinking of a solution where I can provide regex for begin and end lines.. Something would evaluate a function `org-include-src(FILE, LANGUAGE, REGEX_BEGIN, REGEX_END)` — Kaushal Modi, Sep 24 '14 at 07:42
One way is, placing some sort of unique markers (begin end) in the the included file and find them with a function which would be hooked to `org-export-before-processing-hook` to preprocess for the line numbers. Another way is just send a feature request mail to org mailing list :) — kindahero, Sep 25 '14 at 13:45

score 8 · Accepted Answer · answered Sep 25 '14 at 22:59

Here is another option. This one is let's you customize the regular expressions on a per-include basis. It should fit better with some workflows as you're not limited to extension-based definitions.

To Use

Do something like the following in your org-file. (The :lines keyword is optional)

#+INCLUDE: "code/my-class.sv" :src systemverilog :range-begin "^class" :range-end "^endclass" :lines "14-80"

The function will visit "my-class.sv" and search for those two regexps, and then it will update the :lines keyword according with the match result.

If :range-begin is missing, the range will be "-80".
If :range-end is missing, the range will be "14-".

The Code

(add-hook 'before-save-hook #'endless/update-includes)

(defun endless/update-includes (&rest ignore)
  "Update the line numbers of #+INCLUDE:s in current buffer.
Only looks at INCLUDEs that have either :range-begin or :range-end.
This function does nothing if not in org-mode, so you can safely
add it to `before-save-hook'."
  (interactive)
  (when (derived-mode-p 'org-mode)
    (save-excursion
      (goto-char (point-min))
      (while (search-forward-regexp
              "^\\s-*#\\+INCLUDE: *\"\\([^\"]+\\)\".*:range-\\(begin\\|end\\)"
              nil 'noerror)
        (let* ((file (expand-file-name (match-string-no-properties 1)))
               lines begin end)
          (forward-line 0)
          (when (looking-at "^.*:range-begin *\"\\([^\"]+\\)\"")
            (setq begin (match-string-no-properties 1)))
          (when (looking-at "^.*:range-end *\"\\([^\"]+\\)\"")
            (setq end (match-string-no-properties 1)))
          (setq lines (endless/decide-line-range file begin end))
          (when lines
            (if (looking-at ".*:lines *\"\\([-0-9]+\\)\"")
                (replace-match lines :fixedcase :literal nil 1)
              (goto-char (line-end-position))
              (insert " :lines \"" lines "\""))))))))

(defun endless/decide-line-range (file begin end)
  "Visit FILE and decide which lines to include.
BEGIN and END are regexps which define the line range to use."
  (let (l r)
    (save-match-data
      (with-temp-buffer
        (insert-file file)
        (goto-char (point-min))
        (if (null begin)
            (setq l "")
          (search-forward-regexp begin)
          (setq l (line-number-at-pos (match-beginning 0))))
        (if (null end)
            (setq r "")
          (search-forward-regexp end)
          (setq r (1+ (line-number-at-pos (match-end 0)))))
        (format "%s-%s" l r)))))

This is great! Now I can use this to export multiple snippets from the same file. Snippet 1: `#+INCLUDE: "code/basic.sv" :src systemverilog :range-begin "// Example 1" :range-end "// End of Example 1"`. Snippet 2: `#+INCLUDE: "code/basic.sv" :src systemverilog :range-begin "// Example 2" :range-end "// End of Example 2"`. The execution is flawless! Thanks for implementing this **this** quick! — Kaushal Modi, Sep 26 '14 at 01:38

Malabarba · Answer 2 · 2014-09-25T17:17:07.750

The best way I can think of is to update these numbers immediately before exporting or before evaluating.

The Updater

This is the function that goes through the buffer. You can bind it to a key, or add it to a hook. The following code updates the lines whenever you save the file, but if your use case is different, just find out which hook you need! (org-mode is full of hooks)

(add-hook 'before-save-hook #'endless/update-includes)

(defun endless/update-includes (&rest ignore)
  "Update the line numbers of all #+INCLUDE:s in current buffer.
Only looks at INCLUDEs that already have a line number listed!
This function does nothing if not in org-mode, so you can safely
add it to `before-save-hook'."
  (interactive)
  (when (derived-mode-p 'org-mode)
    (save-excursion
      (goto-char (point-min))
      (while (search-forward-regexp
              "^\\s-*#\\+INCLUDE: *\"\\([^\"]+\\)\".*:lines *\"\\([-0-9]+\\)\""
              nil 'noerror)
        (let* ((file (expand-file-name (match-string-no-properties 1)))
               (lines (endless/decide-line-range file)))
          (when lines
            (replace-match lines :fixedcase :literal nil 2)))))))

The Regexps

This is where you define the regexps which will be used as the first and last lines to be included. You can give a list of regexps for each file extension.

(defcustom endless/extension-regexp-map 
  '(("sv" ("^class\\b" . "^endclass\\b") ("^enum\\b" . "^endenum\\b")))
  "Alist of regexps to use for each file extension.
Each item should be
    (EXTENSION (REGEXP-BEGIN . REGEXP-END) (REGEXP-BEGIN . REGEXP-END))
See `endless/decide-line-range' for more information."
  :type '(repeat (cons string (repeat (cons regexp regexp)))))

The background worker

This is the guy that does most of the work.

(defun endless/decide-line-range (file)
  "Visit FILE and decide which lines to include.
The FILE's extension is used to get a list of cons cells from
`endless/extension-regexp-map'. Each cons cell is a pair of
regexps, which determine the beginning and end of region to be
included. The first one which matches is used."
  (let ((regexps (cdr-safe (assoc (file-name-extension file)
                                  endless/extension-regexp-map)))
        it l r)
    (when regexps
      (save-match-data
        (with-temp-buffer
          (insert-file file)
          (while regexps
            (goto-char (point-min))
            (setq it (pop regexps))
            (when (search-forward-regexp (car it) nil 'noerror)
              (setq l (line-number-at-pos (match-beginning 0)))
              (when (search-forward-regexp (cdr it) nil 'noerror)
                (setq regexps nil
                      r (line-number-at-pos (match-end 0))))))
          (when r (format "%s-%s" l (+ r 1))))))))

If I may suggest, edebug the two functions and then invoke the first one with M-x. That should be very informative. :-) — Malabarba, Sep 25 '14 at 14:33
The function by itself runs fine. But the hook needs to pass an argument to the function it is calling. From the docs for `org-export-before-processing-hook`, `Every function in this hook will be called with one argument: the back-end currently used, as a symbol`. As we are not passing any argument, we get the error `run-hook-with-args: Wrong number of arguments`. Now I am not sure what argument to add to `endless/update-includes`... `(&optional dummy)` ? — Kaushal Modi, Sep 25 '14 at 14:55
@kaushalmodi oops, my bad. I've updated the answer. You can use thing you wrote as well. — Malabarba, Sep 25 '14 at 15:00
OK.. adding `(&optional dummy)` actually worked! But an interesting side-effect of calling the function via hook. If I call the function using `M-x`, it modifies the `.org` file with the updated line numbers. But if I simply export to html and allow the hook to call the function, the updated line numbers are reflected only in the exported file, NOT in the `.org` file. — Kaushal Modi, Sep 25 '14 at 15:00
@kaushalmodi Yes, that's how org hooks work. You can add it to before-save-hook instead. — Malabarba, Sep 25 '14 at 15:22
Thanks! I have also added this function call to `before-save-hook`. Works great! I have also modified your solution with a suggested change for #+INCLUDE regex and a fix to set the "r" line number correctly. — Kaushal Modi, Sep 25 '14 at 16:05
From http://orgmode.org/manual/Include-files.html, "You can also include a portion of a file by specifying a lines range using the :lines parameter. The line at the upper end of the range will not be included." Example: `#+INCLUDE: "~/.emacs" :lines "5-10"` <-- This includes lines 5 to 10, 10 excluded. In your function, `r` is the line containing `endclass` (for example). So if we set `:lines "l-r"`, the `rth` line won't be exported. So the exported formats won't show the line with `endclass`. So you need to increment `r` by 1. — Kaushal Modi, Sep 25 '14 at 16:23