How do I patch an Emacs package?

Question

I wish to alter a package, test it and hopefully submit a pull request afterwards. How do I do it in a safe and efficient way? The question might feel too broad, I will accept the answer that covers the following issues:

1. I'd expect to install a separate branch of a package and be able to switch between it and stable branch on a whim, with recompilation performed automatically when it's necessary, but package.el doesn't seem to offer a straightforward way to do that. This answer on emacs-SE informs us that “If multiple copies of a package are installed, then the first one will be loaded” so I guess one could manually mess with load-path but this doesn't feel robust. What is the standard way to select a specific version of package among those installed?

2. Even if I manage to expose several branches to Emacs, for significant tweaks I need to make sure the unpatched branch is “unloaded” and its side-effects isolated. Does unload-feature handle this properly or maybe it has idiosyncrasies that every tester of multi-versioned packages should know about?

3. How do I install and test the local version? The answer appears to be dependent on whether the package is simple (= one file) or multifile. EmacsWiki says about multifile packages: “MELPA builds packages for you”. I doubt that I have to (or should) talk to MELPA every time I change a defun form in a multifile package but the question remains. At least I need to tell package manager about local version, and if so, how do I do it?

4. What names should I assign to local versions of packages? Suppose I want to work on multiple features or bugs simultaneously, which means having several branches. Emacs won't allow to name versions in a descriptive manner (along the lines of 20170117.666-somebugorfeature). I guess I could rename the package itself, a suffix per branch, but again, like manually messing with load-path in Q1, this is an ugly hack, so I won't try it with something I intend to send upstream unless it's a widely accepted practice.

The questions probably are naive, since I never wrote a patch neither applied one with git or a similar vcs. However, for plenty of Emacs users, patching an Emacs package might be their first ever (or maybe the only one) social programming endeavour, which is why, I believe, answers to this question would still be valuable.

Answer 1

To chime in with a slightly different work flow for loading different versions of packages, here's a couple of variations of what I do, both of which use the load-path to control which version I'm using (changing the name of the package is a bad idea if there are dependencies). I have the current version of "nice-package" installed in ~/.emacs.d/elpa using M-x package-install, and the package repo clone in ~/src/nice-package with git clone ....

With use-package

In init.el, I have

(use-package nice-package
  :load-path "~/src/nice-package"
  ...)

With the :load-path line uncommented, this will use the git version of the package. Commenting this line out, and reloading emacs uses the elpa version.

Similar without use-package

In init.el,

(add-to-list 'load-path "~/src/nice-package")
(require 'nice-package)
...

Now do the same commenting trick with the first line.

Using emacs -L

This is the same idea, but manipulating the load-path from the command line. You can load an instance of emacs with the git version of the package with

emacs -L ~/src/nice-package

which just prepends this path to the front of the load-path. That way, you can launch emacs from a different terminal and get the old and the new versions of the package running side-by-side.

Misc comments

1. Use M-x eval-buffer after editing a package file to load the new definitions you've created.
2. Checking what the compiler says with M-x emacs-lisp-byte-compile is handy too

Answer 2

Good question! The answer is that until now, there was no good answer, since none of the existing package managers were designed for this use case (except for Borg, but Borg does not attempt to handle other common package management operations like dependency handling).

But now, there is straight.el, a next-generation package manager for Emacs that addresses this problem as comprehensively as possible. Disclaimer: I wrote straight.el!

After inserting the bootstrap snippet, installing a package is as simple as

(straight-use-package 'magit)

This will clone the Git repository for Magit, build the package by symlinking its files into a separate directory, byte-compile, generate and evaluate autoloads, and configure the load-path correctly. Of course, if the package is already cloned and built, nothing happens, and your init time does not suffer.

How do you make changes to Magit? It's trivial! Just use M-x find-function or M-x find-library to jump to the source code, and hack away! You can evaluate your changes to test them live, as is general practice for Emacs Lisp development, and when you restart Emacs, the package will automatically be rebuilt, re-compiled, and so on. It's completely automatic and foolproof.

When you are satisfied with your changes, just commit, push, and make a pull request. You have total control over your local packages. But your configuration can still be 100% reproducible because you can ask straight.el to make a lockfile that saves the Git revisions of all of your packages, including straight.el itself, MELPA, and so on.

straight.el can install any package from MELPA, GNU ELPA, or EmacsMirror. But it also has a highly flexible recipe DSL that allows you to install from anywhere, as well as to customize how the package is built. Here's an example that shows some of the options:

(straight-use-package
 '(magit :type git 
         :files ("lisp/magit*.el" "lisp/git-rebase.el"
                 "Documentation/magit.texi" "Documentation/AUTHORS.md"
                 "COPYING" (:exclude "lisp/magit-popup.el"))
         :host github :repo "raxod502/magit"
         :upstream (:host github :repo "magit/magit")))

straight.el has ridiculously comprehensive documentation. Read all about it on GitHub.

Answer 3

These are all good questions!

Emacs works on a memory-image model, where loading new code alters the memory image of the running instance. Defining new functions and variables is easily undone, if you keep a list of them, but there are a lot of side effects a module might have that you would want to undo. It looks like unload-feature does make a pretty good go of it though.

I think what you're going to want to do is a combination of live coding and occasionally relaunching Emacs, loading the module you're working on from your branch rather than from where it's installed. If you do end up with a lot of these branches you might want a shell script that launches emacs with the correct load-path for the one you're working on at the moment. In any case I wouldn't rename the package; I think that would be even more confusing since emacs could then load them both.

As you develop your patches you can start by simply redefining the functions that you are changing right in your live Emacs session. This lets you test the new definitions immediately, without leaving Emacs. Specifically, as you edit an elisp file you can use C-M-x (eval-defun) to evaluate the current function in your current Emacs session. You can then call it to make sure it works. If you're changing something that happens at Emacs startup then you'll likely have to start and stop Emacs to test it; you might do that by starting and stopping a separate Emacs process so that your editing session isn't interrupted.

Answer 4

I don't think there's a good answer to that yet (I expect you can get a partial solution with Cask, tho I'm not sufficient familiar with it to give you a good answer using it; hopefully someone else will), but here's what I do (I rarely use a Elisp package without making local changes to it, so it's really my "normal" way):

• cd ~/src; git clone ..../elpa.git
• for each package cd ~/src/elisp; git clone ....thepackage.git
• cd ~/src/elpa/packages; ln -s ~/src/elisp/* .
• cd ~/src/elpa; make

• in your ~/.emacs add

  (eval-after-load 'package
   '(add-to-list 'package-directory-list
                 "~/src/elpa/packages"))
  

This way, all packages are installed "straight from the Git", a simple cd ~/src/elpa; make will recompile those that need it, and C-h o thepackage-function will jump to a source file that's under Git.

To "switch between it and stable branch on a whim", you'll need to git checkout <branch>; cd ~/src/elpa; make; and if you want it to affect running Emacs sessions it'll take more work. I generally recommend not to use unload-feature except in exceptional situations (it's a good feature, but it's not currently reliable enough).

It also doesn't satisfy many of your requirements. And it has some extra downsides, mostly the fact that many packages's Git clone doesn't quite match the layout and contents expected by elpa.git's makefile, so you'll need to begin by tweaking those packages (typically things that have to do with <pkg>-pkg.el, since elpa.git's makefile expects to build this file from <pkg>.el rather than have it be provided, but more problematically, the compilation is performed differently, so sometimes you need to play with the requires).

Oh and of course, this basically means you're installing those packages by hand, so you have to pay attention to dependencies. This setup does properly interact with other packages installed by package-install, tho, so it's not that terrible.

Answer 5

The other answers to this question, including my other answer, talk about patching an Emacs package by making changes to its code. But people finding this question via Google might be thinking of something else when they say "patch an Emacs package" -- namely, overriding its behavior without having to modify its source code.

Mechanisms for doing this include, in increasing order of aggressiveness:

• adding functions to hooks, or binding dynamic variables using let
• the powerful advice system
• just plain overriding the function you want to modify

Despite the power of the first two options, I found myself taking the third route pretty often, since there is sometimes no other way. But then the question is, what if the original function definition changes? You would have no way of knowing that you needed to update the version of that definition that you had copied and pasted into your init-file!

Because I'm obsessed with patching things, I wrote the package el-patch, which solves this problem as comprehensively as possible. The idea is that you define s-expression based diffs in your init-file, which describe both the original function definition and your changes to it. This makes your patches much more readable, and also allows el-patch to later validate whether the original function definition has been updated since you made your patch. (If so, it will show you the changes via Ediff!) Quoting from the documentation:

Consider the following function defined in the company-statistics package:

(defun company-statistics--load ()
  "Restore statistics."
  (load company-statistics-file 'noerror nil 'nosuffix))

Suppose we want to change the third argument from nil to 'nomessage, to suppress the message that is logged when company-statistics loads its statistics file. We can do that by placing the following code in our init.el:

(el-patch-defun company-statistics--load ()
  "Restore statistics."
  (load company-statistics-file 'noerror
        (el-patch-swap nil 'nomessage)
        'nosuffix))

Simply calling el-patch-defun instead of defun defines a no-op patch: that is, it has no effect (well, not quite—see later). However, by including patch directives, you can make the modified version of the function different from the original.

In this case, we use the el-patch-swap directive. The el-patch-swap form is replaced with nil in the original definition (that is, the version that is compared against the "official" definition in company-statistics.el), and with 'nomessage in the modified definition (that is, the version that is actually evaluated in your init-file).

Answer 6

When you make a lot of changes, I think you should use straight.el, see the answer by Radon Rosborough.

If you just want to make a one off change, let's assume to a project called fork-mode, do the following steps:

• Make a directory to store the git mkdir ~/.emacs.d/lisp-gits
• Make a fork of the project you wish to change, say at https://github.com/user/fork-mode
• Clone your fork cd ~/.emacs.d/lisp-gits && git clone git@github.com:user/fork-mode.git

Write the following code in your .emacs

(if (file-exists-p "~/.emacs.d/lisp-gits/fork-mode")
    (use-package fork-mode :load-path "~/.emacs.d/lisp-gits/fork-mode")
  (use-package fork-mode :ensure t))

(use-package fork-mode
  :config
  (setq fork-mode-setting t)
  :hook
  ((fork-mode . (lambda () (message "Inside hook"))))

Now you can use the emacs mode, using C-h f to find the functions you want to change. You will notice that when the package is installed in lisp-gits, you will jump to there. Use magit or other git commands to commit/push changes and then use github to send your pull requests.

Once your pull requests are accepted you can just remove the project in ~/.emacs.d/lisp-gits and let the package manager do its work.

Answer 7

With bitterness, I have to accept my own answer that amounts to “you can't do it in a civilised way.”

There are package manager issues with package.el that could be fixed with enough manpower, and there is code reload issue which, on the contrary, looks like a research problem. Because of the latter, we can conclude that the original question cannot be satisfactorily answered. You will have to restart your Emacs while developing Emacs packages, at least each time you want to make sure it compiles and loads correctly.

To work around package.el issues, I ended up writing and patching Gentoo packages for Emacs instead of patching (or writing) Emacs packages. This comes with its own set of problems but those are at least tractable on individual user level. In particular, Gentoo's package manager “portage” automatically restarts Emacs for compilation tasks. It does some other things that package.el does not: user patch, configure (particularly relevant for Emacs packages that ship dynamic modules), install from git branch, install from org-based sources. Most of this functionality is built-in as of 2020-03.