Org mode refiling gobbles a newline and absorbs the next heading

Question

I often have this problem in Org mode: I write an item with org-capture, refile it, and days later I see a result like this:

*** TODO some task
some notes** StackExchange

I believe that the problem is that org capture inserts the text into the buffer, and if the item does not end with a newline, it gobbles the delimiter to the next heading and both get refiled together.

I already added two newlines to my capture template to minimize this problem. My capture template is:

(setq org-capture-templates
      '(("t" "Todo" entry (file+headline "~/org/gtd.org" "Reminders")
     "* TODO %?\n\n"))

If I try a new item with C-c c t and type something, I see it appear in the main buffer. If I delete the newline at the end, the newline is deleted in the buffer too and gobbles the next line. Deleting further characters in the capture buffer does not delete them from the main buffer.

I think I could add a hook such as (add-hook 'org-capture-before-finalize-hook ...) but I'm not sure how to include a newline and how to do it before refiling, to avoid the next item being refiled with the captured item.

How can I fix this problem?

Update: Following @Aquaactress's suggestions, I tried two methods. Both fail and the versions may be important:

Org mode version 9.1.9 (release_9.1.9-65-g5e4542 @ /Applications/Emacs.app/Contents/Resources/lisp/org/)
GNU Emacs 26.3 (build 1, x86_64-apple-darwin18.2.0, NS appkit-1671.20 Version 10.14.3 (Build 18D109)) of 2019-09-02

In the first method, I added :empty-lines in the org-capture template:

(setq org-capture-before-finalize-hook nil)

(setq org-capture-templates
      '(("t" "Todo" entry (file+headline "~/test.org" "heading 1")
     "* TODO %?\n" :empty-lines 1)))

The result in screenshots is below: I add a capture item, delete newlines from the end of the org-capture buffer, and refile. The heading next to the refiled item is gobbled:

In the second method, I added a function that inserts new lines at the beginning and end of the org capture buffer (for exposition, I also add hello world! to show that these insertions go at the beginning and end of the whole org buffer, not the org-capture buffer):

(setq org-capture-before-finalize-hook nil)

(defun +org|insert-newlines ()
  (beginning-of-buffer) (insert "\nhello world!\n")
  (end-of-buffer) (insert "\nhello world!\n"))

(add-hook 'org-capture-before-finalize-hook #'+org|insert-newlines)

(setq org-capture-templates
      '(("t" "Todo" entry (file+headline "~/test.org" "heading 1")
     "* TODO %?\n" :empty-lines 1)))

The result in screenshots is below: I add a capture item, delete newlines from the end of the org-capture buffer, and refile. The heading next to the refiled item is gobbled and the lines inserted go at the beginning and end of the buffer, instead of the org-capture buffer.

Answer 1

As you've noticed org capture by default disregards newlines when placing headlines. To remedy is a setting you can add to capture templates called :empty-lines. Here's what the manual says about it.

`:empty-lines'
   Set this to the number of lines to insert before and after the
   new item.  Default 0, only common other value is 1.

Here is an example of it's usage in my own capture template.

(setq org-capture-templates
      `(("t" "emacs" plain (file ,(concat org-directory "todo.org"))
         "* TODO %?\n" :empty-lines 1)))

Answer 2

I was able to make this work with a slight modification to @Aquaactres's comments under their answer.

I added my function to org-capture-prepare-finalize-hook and not org-capture-before-finalize

The function I added to the hook:

(defun my-org-capture-newlines-at-end ()
  (goto-char (point-max))
  (insert "\n\n"))

For example, if you're using use-package, it would look like:

:hook
(org-capture-prepare-finalize . my-org-capture-newlines-at-end)

Note, this works for me with :prepend t :empty-lines 1

Answer 3

workaround for refiling (first problem)

Below I detail how Org 9.3.3 fixes the issue and how to install it from source. But I have had more complicated problems from installing that version (one that I asked about, and another that I mentioned in a comment and was clearly a bug so did not ask about), so I used a workaround that disables deleting characters and killing lines in an org-capture buffer:

(defun my-org-is-buffer-org-capture()
  "Checks if the current buffer is an org-capture buffer."
  (string-match-p (regexp-quote "CAPTURE-") (buffer-name)))

(defun my-org-delete-char ()
  "Wrapper for org-delete-char, except in org-capture where it's disabled."
  (interactive)
  (if (my-org-is-buffer-org-capture)
      (message "Delete character is disabled in capture buffer")
    (org-delete-char 1)))

(defun my-org-kill-line (&optional arg)
  "Wrapper for kill-visual-line, except in org-capture where it's disabled."
  (interactive)
  (if (my-org-is-buffer-org-capture)
      (message "Kill line is disabled in capture buffer")
    (kill-visual-line arg)))

;; Disable killing lines and deleting forward characters in org capture buffers
(define-key org-mode-map (kbd "C-d") 'my-org-delete-char)
(define-key org-mode-map (kbd "C-k") 'my-org-kill-line)

bug fix for refiling (first problem)

This behavior was indeed a bug, fixed by Bastien and announced in the org-mode mailing list:

I made a change in Org maint branch that could somehow impact the issue you reported. Can you test and see if it does?

As of version 9.3.3 (release_9.3.3-206-gbeae40 on git), the first problem I reported is fixed.

adding lines at beginning and end of buffer (second problem)

The second problem with the addition of text at the beginning and end of the org buffer instead of the org-capture buffer is still present, but I'm not sure if it is a bug or if (beginning-of-buffer) is not meant to refer to the org-capture buffer in the orga-capture-before-finalize hook. And anyway, I would note use that feature except to fix the first problem.

installing the bug fix

To load the latest version from the online repository, which is 9.3.3 or higher, run this in a shell:

cd ~/src/  # or any other location
git clone https://code.orgmode.org/bzg/org-mode.git
cd org-mode/
make autoloads

Then run this Emacs-LISP code before any other org code in your initialization files:

(add-to-list 'load-path "~/src/org-mode/lisp")
(require 'org-loaddefs)

See this answer for details on why you need the second line and in that order to avoid a mixed installation of built-in and latest versions:

The main idea is that you need to make sure that Org mode is set up, using the new bits that you downloaded only. Emacs comes with a version of Org mode bundled up with it, but that is inevitably older than what is available from the Org mode git repo. It is important to avoid loading any piece from the bundled up version or else you might end up with a "mixed" installation (there are many hits for that on the Org mode mailing list).

As I mention in a comment, and @Tobias has verified in another comment, you do not need to byte-compile the new version - you only need to make autoloads. That creates an org-loaddefs.el file, so that all the common entry points into org will, when first invoked, load the appropriate file first. It is important (and contrary to what the OP states in his comment) to set the load-path before loading the org-loaddefs.el file: otherwise, you may end up with the dreaded "mixed" install I mentioned above.

(I borrowed the installation instructions from this answer, and copied them from AskUbuntu and Emacs.)

Answer 4

What actually happens when you capture is that org mode opens the target file, inserts a new item, and narrows your view to that item. So, in effect, you edit the target file.

What works for me is to combine :emtpy-lines 1 with :unnarrowed t. This starts me off with extra lines, and shows me the effects of my edits, which keeps me from accidentally deleting the empty lines.

The disadvantage is I may get distracted by other items that are already in te buffer.

Obviously, this whole mess could be avoided by not using :prepend t in the first place, but that is just not what I want.

Answer 5

I had a similar issue and the solution for me was to create an advice like this:

    (defun my/org--capture-refile (orig-fun &rest args)
      (let ((org-reverse-note-order (org-capture-get :prepend 'local)))
        (apply orig-fun args)))

    (advice-add #'org-capture-refile :around 'my/org--capture-refile)

so if you issues is related to :prepend (like I had) with the advice "org-reverse-note-order" is temporarily overridden to respect :prepend from the org-capture-template

Answer 6

Following the answer by coffeepenbit, I realized the side-effect that other hooks would run from the new position of point (and thus relative to the next heading), I also added a verification to avoid multiple newlines, and here is the code that works for me:

;; Configure capture for Org mode. Extra newlines at the end prevent gobbling
;; of the next item in the same line as this one
(setq org-capture-templates
      '(("t" "Todo" entry (file+headline "~/org/gtd.org" "reminders")
     "* TODO %?\n" :empty-lines 1)
))

;; Add newline to avoid gobbling of refiling items.
(defun my-org-capture-avoid-gobbling ()
  (save-excursion
    (goto-char (- (point-max) 1))
    (unless (looking-at "
")
      (goto-char (point-max))
      (insert "\n!!Gobbling avoided!!\n"))))

;; Some other function to run before refiling, e.g. adding created and scheduled dates.
(defun my-org-insert-created-and-scheduled-date(&rest ignore)
  (when (and (org-get-todo-state)
             (not (org-entry-get nil "CREATED")))
    (org-entry-put nil "CREATED" (format-time-string "[%Y-%m-%d %a]")))
  (when (and (org-get-todo-state)
             (not (org-entry-get nil "SCHEDULED")))
    (org-entry-put nil "SCHEDULED"
           (format-time-string "<%Y-%m-%d %a>"
                       (time-add nil (* 14 86400)))))
  )

;; Insert created date in capture.
(add-hook 'org-capture-prepare-finalize-hook #'my-org-insert-created-and-scheduled-date)

;; Add newlines at capture items: add this one last so it's the first to run and
;; the other hooks run on the clean heading instead of the gobbled heading.
(add-hook 'org-capture-prepare-finalize-hook #'my-org-capture-avoid-gobbling)