log keystrokes with timestamp and other metadata?

Question

If I evaluate this form:

(open-dribble-file "/home/joe/keys.log")

then type something, Emacs saves what I type to the keys.log file:

<return>hellow<backspace> keyloggi<down-mouse-5><mouse-5>ng world<down><down>
^E^X<C-down-mouse-4><C-mouse-4>^E^X^F^N^N<return>yes<return>

Let's stop that nonsense immediately.

(open-dribble-file nil)

What I would like is a similar function that would save each keypress together with a timestamp and some other metadata.

State of the art

There is command-log-mode, which can be adjusted to log every interactive command by setting

(setq clm/log-command-exceptions* '(nil))

It stores the time of the command on a :time text property, and that can be given a milisecond resolution by appending .%3N to the relevant format-time-string command. The only thing it doesn't do yet is actually save the text that I type. Instead, it records the first letter, and the number of letters typed:

t      self-insert-command [47 times] 

I'm guessing it will not be too hard to collect and save the entered text. I guess I'll report back when I figure out the text storage, unless someone beats me to it. :-)

(I won't need the exact text for basic analytics, for which what I have here is sufficient -- but it could be useful to save it for broader life-logging purposes.)

related

In the mean time, I noticed that there is a logkeys package in Ubuntu. It does something similar to what I want, but I actually only want to track Emacs keypresses, because anything else is not likely to be productive work. Here is some logkeys output as a sample:

2015-06-05 23:02:15+0100 > 3;<E-e1><E-e1>3<#+10>b3;ȁ
2015-06-05 23:02:22+0100 > 
2015-06-05 23:02:22+0100 > 
2015-06-05 23:02:22+0100 > 7'''u܂ccȁ3<#+7><CpsLk>3;q

(It is typing garbage because it doesn't know my keyboard layout, and only creates a timestamp when I press RET.)

note

When I first posted this question, I thought it would make sense to approach it via Emacs C, with a patch to record_char from keyboard.c, which is where the dribble file is written. But given that there's a nearly complete solution in Lisp, that was almost certainly the wrong intuition. I've mostly revised that out but I thought I should record the initial idea, even though it wasn't that apt.

Answer 1

I've added [in my fork] the following setup steps to command-log-mode to stash recently typed text in a variable.

(defvar clm/log-text t
  "A non-nil setting means text will be saved to the command log.")

(defvar clm/recent-history-string ""
  "This string will hold recently typed text.")

(defun clm/recent-history ()
  (setq clm/recent-history-string
    (concat clm/recent-history-string
        (buffer-substring-no-properties (- (point) 1) (point)))))

(add-hook 'post-self-insert-hook 'clm/recent-history)

(defun clm/zap-recent-history ()
  (unless (or (member this-original-command
              clm/log-command-exceptions*)
          (eq this-original-command #'self-insert-command))
    (setq clm/recent-history-string "")))

(add-hook 'post-command-hook 'clm/zap-recent-history)

Then I run the following snippet at a suitable point:

     (when clm/log-text
       (if (eq clm/last-keyboard-command 'self-insert-command)
           (insert "[text: " clm/recent-history-string "]\n")))

I've found that (setq clm/log-command-exceptions* '(nil ignore)) is a good setting for logging all commands, except those which have been explicitly ignored (like inadvertent mouse actions). And don't forget (setq command-log-mode-is-global t).

Here is a sample log:

H      self-insert-command [35 times] 
[text: Here is an example of how it works.]
RET    newline
P      self-insert-command [12 times] 
[text: Pretty nice.]
RET    newline
I      self-insert-command [30 times] 
[text: If you make mistakes when typo]
DEL    backward-delete-char-untabify
i      self-insert-command [41 times] 
[text: ing you have to read the lines separetly.]
<C-backspace>
       (lambda nil (interactive) (kill-sexp -1))
s      self-insert-command [11 times] 
[text: separately.]
RET    newline
C-x o      other-window

As Andrew Swann asked in a comment on this post: Where are the requested timestamps? They are there but really this requires an...

update: Command log mode stores time stamps in text properties, which is why you don't see them above. I added [commit dd497d] an export function that exposes them in plain text in the log file when that is written to disk . Here what that export looks like (for another bit of interaction):

[2015-06-15T00:46:24] ;    self-insert-command [19 times] 
[text: ;; One question is ]
[2015-06-15T00:46:28] <C-backspace>
[2015-06-15T00:46:28]      (lambda nil (interactive) (kill-sexp -1))
[2015-06-15T00:46:28] t    self-insert-command [47 times] 
[text: that seems worth asking is whether this noticeb]
[2015-06-15T00:46:35] DEL      backward-delete-char-untabify
[2015-06-15T00:46:36] a    self-insert-command [4 times] 
[text: ably]
[2015-06-15T00:46:36] H-s      ispell-word
[2015-06-15T00:46:37] SPC      self-insert-command [59 times] 
[text:  slows down the experience of typing.  If it does, I wonder]
[2015-06-15T00:46:48] RET      newline
[2015-06-15T00:46:48] ;    self-insert-command [25 times] 
[text: ;; if there are things ta]
[2015-06-15T00:46:52] <C-backspace>
[2015-06-15T00:46:52]      (lambda nil (interactive) (kill-sexp -1))
[2015-06-15T00:46:52] t    self-insert-command [32 times] 
[text: that could be done that would se]
[2015-06-15T00:46:56] DEL      backward-delete-char-untabify
[2015-06-15T00:46:56] p    self-insert-command [18 times] 
[text: peed it up.  It's ]

(Some other natural metadata to include would be the name of the buffer and the current mode, which would help with subsequent analytics... coming up soon.)