33

How do I measure the performance of my elisp code? What tools / external packages are available for me to measure time taken?

In addition to total time, can I see a profile that shows time taken per-function? Can I profile memory usage too?

Wilfred Hughes
  • 6,890
  • 2
  • 29
  • 59
  • 1
    The question is too broad. What kind of performance? Where? When? "***Emacs performance***" can mean anything and everything. – Drew Oct 01 '14 at 15:48
  • @Drew Many other programming languages have a set of benchmarks (e.g. Python: http://speed.pypy.org/, JS: Sunspider etc), and I was hoping that there was an equivalent for the elisp interpreter. – Wilfred Hughes Oct 01 '14 at 16:00
  • Benchmarking such as that provided by function `benchmark` and the profiler does not measure **Emacs** performance. It measures the performance evaluating particular expressions. It is helpful in comparing performances *within* Emacs. To measure the performance of Emacs itself you would need to compare it to the performance of something other than Emacs. And that is where the breadth of Emacs comes into play. You could measure Emacs vs XYZ for this or that, but to measure Emacs performance as a whole you would need umpteen such comparisons. – Drew Oct 01 '14 at 16:13
  • Maybe you meant "*How do I measure performance in Emacs*"? – Drew Oct 01 '14 at 16:15
  • I'm interested in answering the question 'is Emacs 24.4 faster than 24.3?' or 'how much faster would Guile-Emacs be than the GNU Emacs 24.3?'. I'm not sure how to proceed, as the answers here are interesting (and a good start), but not what I had in mind. – Wilfred Hughes Oct 01 '14 at 16:57
  • That's what I mean. Comparing two Emacs versions is too broad (IMO). You could reasonably try to compare two versions for some specific operation. For that, you will need, I think, to compare processor and memory use or clock time or some such. And for that you might need tools that are outside Emacs. Same thing for trying to compare such things across platforms. I don't have an answer for you, and I think the question is too broad. But maybe you will get some help for what you need. – Drew Oct 01 '14 at 17:01
  • Why not just benchmark a set of common operations separately on Emacs 24.4 and 24.3 and then compare which took the longest? – Malabarba Oct 01 '14 at 18:59
  • I admit I think I misread your question. I read it as _“How do I benchmark Emacs Lisp?”_. On a separate note, the question at the end _“Are there any benchmarks of Emacs performance?”_ sounds quite different from _“How do I benchmark Emacs?”_. – Malabarba Oct 01 '14 at 19:04
  • 2
    OK, I've opened http://emacs.stackexchange.com/q/655/304 to be about benchmarking Emacs, and reworded this question be about benchmarking/profiling elisp programs. – Wilfred Hughes Oct 03 '14 at 10:03

4 Answers4

35

Benchmark

The most straightforward options is the built-in benchmark package. Its usage is remarkably simple:

(benchmark 100 (form (to be evaluated)))

It’s autoloaded, so you don’t even need to require it.

Profiling

Benchmark is good at overall tests, but if you’re having performance problems it doesn’t tell you which functions are causing the problem. For that, you have the (also built-in) profiler.

  1. Start it with M-x profiler-start.
  2. Do some time consuming operations.
  3. Get the report with M-x profiler-report.

You should be taken to a buffer with a navigatable tree of function calls.
Profiler screenshot

Malabarba
  • 22,878
  • 6
  • 78
  • 163
  • `benchmark` function doesn't seem to work: when I do inside an opened `.c` file `(benchmark 100 (c-font-lock-fontify-region 0 17355))`, I keep getting `void-function jit-lock-bounds`. – Hi-Angel Mar 06 '19 at 22:38
  • 1
    FTR: as an alternative to `benchmark` there are functions `benchmark-run` and `benchmark-run-compiled`. For me the main difference was that both functions actually work *(see the prev. comment)* :Ь – Hi-Angel Mar 08 '19 at 16:44
16

In addition to @Malabara's answer, I tend to use a custom-made with-timer macro to permanently instrument various parts of my code (e.g my init.el file).

The difference is that while benchmark allows to study the performance of a specific bit of code that you instrument, with-timer always gives you the time spent in each instrumented part of the code (without much overhead for sufficiently large parts), which gives you the input to know which part should be investigated further.

(defmacro with-timer (title &rest forms)
  "Run the given FORMS, counting the elapsed time.
A message including the given TITLE and the corresponding elapsed
time is displayed."
  (declare (indent 1))
  (let ((nowvar (make-symbol "now"))
        (body   `(progn ,@forms)))
    `(let ((,nowvar (current-time)))
       (message "%s..." ,title)
       (prog1 ,body
         (let ((elapsed
                (float-time (time-subtract (current-time) ,nowvar))))
           (message "%s... done (%.3fs)" ,title elapsed))))))

Example use: 

(with-timer "Doing things"
  (form (to (be evaluated))))

yielding the following output in the *Messages* buffer:

Doing things... done (0.047s)

I should mention that this is heavily inspired by Jon Wiegley's use-package-with-elapsed-timer macro in his excellent use-package extension.

François Févotte
  • 5,917
  • 1
  • 24
  • 37
  • If you're measuring init.el, you will probably be interested in [the emacs startup profiler](https://github.com/jschaf/esup). – Wilfred Hughes Oct 01 '14 at 09:58
  • Macros are awesome. This deserves more votes. – Malabarba Oct 01 '14 at 12:58
  • 2
    Emacs records the total init time. You can show it with the command `emacs-init-time`. – Joe Oct 01 '14 at 13:21
  • 1
    @WilfredHughes yes, I do use `esup` and I like it. But once again, the interest of such a thing as `with-timer` is not so much to profile something thorougly. The real interest is that you *always* have profiling information. Whenever I start emacs, I have a buch of lines in my `*Messages*` buffer that tell me which part took how long. If I detect anything abnormal, I can then use any of the more adequate tools to profile and optimize things. – François Févotte Oct 01 '14 at 14:20
  • @JoeS Yes, `emacs-init-time` does produce interesting information. However, it only gives an inclusive elapsed time, without the possibility to break down individual parts of the initialization. – François Févotte Oct 01 '14 at 14:22
  • Error message: "Symbol’s function definition is void: form", Emacs v25.2.1 – CodyChan Jul 26 '17 at 04:17
  • There's also the [benchmark-init.el](https://github.com/dholm/benchmark-init-el) package. – izkon Nov 07 '17 at 22:50
  • You can also use the `elp` profiler. Try `M-x elp-instrument- TAB`. and later `M-x elp-results`. – Stefan Nov 07 '17 at 23:09
5

In addition to @Malabarba's answer, note that you can measure the compiled execution time of your code with benchmark-run-compiled. That metric is often much more relevant than the interpreted execution time that M-x benchmark gives you:

ELISP> (benchmark-run (cl-loop for i below (* 1000 1000) sum i))
(0.79330082 6 0.2081620540000002)

ELISP> (benchmark-run-compiled (cl-loop for i below (* 1000 1000) sum i))
(0.047896284 0 0.0)

The three numbers are the total elapsed time, the number of GC runs, and the time spent in GC.

Clément
  • 3,924
  • 1
  • 22
  • 37
1

Benchmarking is not only about getting the numbers, it is also about making decisions based on result analysis.

There is benchstat.el package on MELPA which you can use to get features that benchstat program provides.

It implements comparison-based benchmarking where you examine X performance properties against Y.

Benchstat functions can be viewed as a benchmark-run-compiled wrapper that not only collects the information, but gives it back in easy to read an interpret format. It includes:

  • Elapsed time delta between X and Y
  • Mean average time
  • Allocations amount

Very simple usage example:

(require 'benchstat)

;; Decide how much repetitions is needed.
;; This is the same as `benchmark-run-compiled` REPETITIONS argument.
(defconst repetitions 1000000)

;; Collect old code profile.
(benchstat-run :old repetitions (list 1 2))
;; Collect new code profile.
(benchstat-run :new repetitions (cons 1 2))

;; Display the results.
;; Can be run interactively by `M-x benchstat-compare'.
(benchstat-compare)

The benchstat-compare will render results in a temporary buffer:

name   old time/op    new time/op    delta
Emacs    44.2ms ± 6%    25.0ms ±15%  -43.38%  (p=0.000 n=10+10)

name   old allocs/op  new allocs/op  delta
Emacs      23.0 ± 0%      11.4 ± 5%  -50.43%  (p=0.000 n=10+10)

You will need benchstat program binary though. If you used Go programming language, most likely you have one in your system already. Otherwise there is an option of compiling it from the sources.

Precompiled binary for linux/amd64 can be found at github release page.

Iskander Sharipov
  • 73
  • 5