15

I use url-retrieve-synchronously to download a JSON file from a remote server. This function returns a buffer which contains the entire content of the response, including HTTP headers. However, I'm just interested in the body.

Now, short of parsing a HTTP response manually, how can I jump to the beginning of the body? I have found the variable url-http-end-of-headers, which works, but it's not documented at all, and using it feels “dirty”. Is there a proper API for this purpose?

3 Answers3

12

A higher-level alternative to using url-http-end-of-headers, but equally undocumented, is url-insert-file-contents:

(with-temp-buffer
  (url-insert-file-contents
   "https://api.stackexchange.com/2.2/questions/12464?site=emacs")
  (json-parse-buffer :object-type 'alist))

or equivalently, prior to Emacs 27 compiled with native JSON support:

(require 'json)
(with-temp-buffer
  (url-insert-file-contents
   "https://api.stackexchange.com/2.2/questions/12464?site=emacs")
  (let ((json-false :false))
    (json-read)))

These result in:

((items .
  [((tags .
     ["url" "http"])
    (owner
     (reputation . 10741)
     (user_id . 227)
     (user_type . "registered")
     (accept_rate . 89)
     (profile_image . "https://i.stack.imgur.com/ebO5J.jpg?s=128&g=1")
     (display_name . "lunaryorn")
     (link . "https://emacs.stackexchange.com/users/227/lunaryorn"))
    (is_answered . t)
    (view_count . 867)
    (accepted_answer_id . 29798)
    (answer_count . 3)
    (score . 4)
    (last_activity_date . 1517363132)
    (creation_date . 1431861037)
    (question_id . 12464)
    (link . "https://emacs.stackexchange.com/questions/12464/go-to-body-after-url-retrieve-synchronously")
    (title . "Go to body after url-retrieve-synchronously"))])
 (has_more . :false)
 (quota_max . 300)
 (quota_remaining . 276))

The function url-insert-file-contents wraps the better-documented functions url-insert-buffer-contents and url-insert and thus comes with the following goodies:

  1. Autoloaded by default.
  2. Exists since at least as far back as Emacs 21.
  3. Simple HTTP response error handling via url-http-response-status.
  4. Decodes data.

FWIW, it also comes with the guarantee of being used by lisp/emacs-lisp/package.el.

Basil
  • 12,019
  • 43
  • 69
  • I had a lot of trouble with using ```url-http-end-of-headers``` and I must say ```url-insert-file-contents``` is a much easier way, if all you care about is the payload. Thanks for the tip! – Marco Craveiro Feb 17 '21 at 15:23
  • Is `url-insert-file-contents` synchronous like the `url-retrieve-synchronously`? – scribe May 30 '21 at 00:38
  • 1
    @scribe Yes, `M-x find-function RET url-insert-file-contents RET` reveals that `url-retrieve-synchronously` is the first thing that `url-insert-file-contents` does. – Basil May 30 '21 at 00:48
11

Since there is a newline after the HTTP headers, the only alternative I'm aware of is using a re-search-forward:

(with-current-buffer
    (url-retrieve-synchronously my-url)
  (goto-char (point-min))
  (re-search-forward "^$")
  (delete-region (point) (point-min))
  (buffer-string))
mutbuerger
  • 3,434
  • 14
  • 22
  • 1
    Well, that's “parsing HTTP” for me, and I think `url-http-end-of-headers` is preferable to this approach. I had really hoped for something like `url-http-goto-body` or so :( –  May 17 '15 at 18:34
4

With respect to a "proper API", I think the short answer, at present, is "no" -- unless you are willing to rely on an external library. url-http-end-of-headers seems to be as good as it gets, although it isn't mentioned in the URL Programmer's Manual nor does it appear to be documented in emacs source.

dat
  • 271
  • 3
  • 10