`url-inhibit-uncompression` is ineffective; emulating curl with url.el

Question

Consider the following:

(let ((url-inhibit-uncompression t))
  (switch-to-buffer
   (url-retrieve-synchronously
    "http://api.stackexchange.com/2.2/filter/create")))

I would expect to get something that looks mostly like garbage as the content (gzipped response), but instead I get a 'reasonable' JSON object (formatted for sanity):

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/json; charset=utf-8
Content-Encoding: gzip
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST
Access-Control-Allow-Credentials: false
X-Content-Type-Options: nosniff
Date: Sat, 01 Nov 2014 03:19:40 GMT
Content-Length: 1903

{
  "quota_remaining": 178,
  "quota_max": 300,
  "has_more": false,
  "items": [
    {
      "filter": "default",
      "filter_type": "safe",
      "included_fields": [
        ".backoff",
        ".error_id",
        ".error_message",
        .................
        "write_permission.min_seconds_between_actions",
        "write_permission.object_type",
        "write_permission.user_id"
      ]
    }
  ]
}

Why is url-inhibit-uncompression not doing anything here? Spoiler alert: It's not used anywhere in the source (that I can find). How can I force output similar to curl without using curl (i.e. compressed and without headers)?

Malabarba · Answer 1 · 2014-11-02T10:36:21.243

I agree with @Constantine, this looks like a bug. Consider submitting a bug report.

As for how to fix it, I personally prefer temporary overriding instead of advices (too many bad experiences with them), but none of the options is ideal.

You can temporarily override functions by wrapping your code in a cl-letf.

(cl-letf (((symbol-function 'url-handle-content-transfer-encoding)
           #'ignore))
  (switch-to-buffer
   (url-retrieve-synchronously
    "http://api.stackexchange.com/2.2/filter/create")))

This assumes you want this function to do nothing. If you want to change its behaviour, just use a lambda or a function name instead of ignore.

Constantine · Answer 2 · 2014-11-01T22:14:35.140

This looks like a bug in url-handle-content-transfer-encoding: I believe it should check url-inhibit-uncompression. Consider submitting a bug report.

In the meantime you can add advice to this function, making it possible to (temporarily) disable it:

(defvar url-handle-content-transfer-encoding-disable nil)
(advice-add 'url-handle-content-transfer-encoding :around
  (lambda (orig &rest args)
    (if url-handle-content-transfer-encoding-disable nil
      (apply orig args))))

Now this

(let ((url-handle-content-transfer-encoding-disable t))
  (switch-to-buffer
   (url-retrieve-synchronously
    "http://api.stackexchange.com/2.2/filter/create")))

returns the HTTP header followed by compressed data.

`url-inhibit-uncompression` is ineffective; emulating curl with url.el

2 Answers2