17

I want to perform a variant of anchored font lock matching. I have function definitions that start with a list of names, and I want those names to be highlighted inside the function body.

I have created a function that does this and registered it as a jit-lock function with jit-lock-register, however, performance is pretty poor and scrolling lags in larger files.

  • How can I measure performance? If I just time calling my function on a large file (with float-time before and after or with elp) I get wildly varying performance, it takes anything from 0.65 to 12 seconds. Is there a recommended way to benchmark font lock performance?
  • Is there any difference in performance between an anchored matcher defined in font-lock-keywords and adding a function via jit-lock-register?

Edit: It seems that the variability in performance is related to garbage collection, invocations of my jit-lock function get successively slower with each invocation until garbage collection is run, at which point they get fast again.

Joakim Hårsman
  • 719
  • 5
  • 11
  • For the first item, try the profiler. – Malabarba Dec 14 '14 at 02:14
  • I can (and have) used the profiler too see what parts of my code that takes time, but since performance is so inconsistent it is hard to tell whether any changes I make are an improvement or not. – Joakim Hårsman Dec 15 '14 at 05:20
  • Do you have some code we can test? That might help us a lot. – PythonNut Feb 25 '15 at 05:33
  • 2
    Although not about profiling or micro-optimizations, per se: I've found the [font-lock-studio](http://melpa.org/#/font-lock-studio) package to be another helpful tool for understanding font-lock performance. It can help the same way as any other interactive stepping debugger can help -- you might discover the execution paths are not what you expect, and that's the main performance issue. – Greg Hendershott Oct 16 '15 at 15:41
  • Thanks for the tip about font-lock-studio, it's awesome! Doesn't help with jit-lock-functions though, but sure does with everything else. – Joakim Hårsman Oct 16 '15 at 19:28

1 Answers1

9

It turns out that the wildly varying performance was related to garbage collection. Each call to the function would get slower until a garbage collection was run. With stock emacs, gc was run every couple of seconds, but I had a line in my init.el to improve startup time that set gc-cons-threshold to 20 MB, and that meant gc was run much more infrequently, causing benchmarks to report slower and slower timing until a gc was run after a couple of minutes, then times would plummet and be fast again.

After reverting to the default gc-cons-threshhold, benchmarking became easier.

I then profiled for memory with the built in profiler (M-x profiler-start), and discovered that calls to syntax-ppss caused the most allocations, so after some optimization to call syntax-ppss less often I achieved acceptable performance.

Using jit-lock-mode (adding a function via jit-lock-register) seems to be the easiest way to get multi line font locking to work reliably, so that was the method I chose.

Edit: After discovering that performance was still not good enough in very large buffers I spent a lot of time optimizing cpu use and allocation, measuring the performance improvements with the built in Emacs profiler (M-x profiler-start). However, Emacs would still stutter and hang when scrolling quickly through very large buffers. Removing the jit-lock function I registered with jit-lock-register would remove the stuttering and hangs, but profiling showed the jit-lock function to complete in around 8 ms which should be fast enough for smooth scrolling. Removing the call to jit-lock-registerand instead using a regular font-lock-keywords matcher solved the issue.

TLDR: Doing this was slow and would stutter:

(defun my-font-lock-function (start end)
"Set faces for font-lock between START and END.")

(jit-lock-register 'my-font-lock-function)

Doing this was fast and would not stutter:

(defun my-font-lock-function (start end)
"Set faces for font-lock between START and END.")

(defun my-font-lock-matcher (limit)
    (my-font-lock-function (point) limit)
   nil)

(setq font-lock-defaults
  (list 
     ...
    ;; Note that the face specified here doesn't matter since
    ;; my-font-lock-matcher always returns nil and sets the face on
    ;; its own.
    `(my-font-lock-matcher (1 font-lock-keyword-face nil))))
Joakim Hårsman
  • 719
  • 5
  • 11
  • Could you share the code you used? Your solution might help others looking to achieve the same thing. – Manuel Uberti Oct 16 '15 at 07:24
  • I didn't really use any specific code, I just called syntax-ppss less. You can check out the code in question here: https://bitbucket.org/harsman/dyalog-mode/src/faba98970c7198bbbf34a4049756815d35669030/dyalog-mode.el?at=default&fileviewer=file-view-default Look for `dyalog-fontify-locals`. – Joakim Hårsman Oct 16 '15 at 19:18
  • I guess `dyalog-fontify-locals-matcher` should be `my-font-lock-matcher` and one of the `end` should be `limit`. Anyway, really interesting discovery! – Lindydancer Dec 31 '16 at 11:49
  • @Lindydancer: Yes thank you. Fixed. – Joakim Hårsman Dec 31 '16 at 22:26
  • 1
    Re: `gc-cons-threshold`, if you're messing with internal values purely to improve start-up time, I suggest you use `emacs-startup-hook` to restore them afterwards. – phils Jan 15 '17 at 02:02
  • 1
    I also noticed that large gc thresholds cause stuttering, but I have no idea why that is happening. My system has plenty of free RAM laying around ... – HappyFace May 12 '21 at 18:26
  • HappyFace: That's a bit like saying "my neighbours throw garbage over the fence every few seconds, but my back yard is really spacious, so why does cleaning my yard take a long time if I haven't done it for ages?" -- it takes more time because there's more to collect. If you don't let so much garbage pile up, then collecting it won't take so long. – phils Apr 29 '22 at 10:26