87

I'm in a network using a proxy. I've got machines using lots of scripts here and there accessing each other over HTTP.

  • The network is 10.0.0.0/8.

  • My proxy is 10.1.1.1:81, so I set it up accordingly:

    export http_proxy=http://10.1.1.1:81/

  • I want to exclude my own range to be accessed with the proxy. I tried any combination available.

    export no_proxy='10.*'
export no_proxy='10.*.*.*'
export no_proxy='10.0.0.0/8'

None of the above work!

I'm testing with wget and it always tries to query the proxy, whatever IP address I want to connect to.

  • Since lots of scripts lie everywhere in all systems the --no-proxy option is actually not an option. I want to set it system wide.
Jeff Schaller
  • 67,283
  • 35
  • 116
  • 255
SamK
  • 1,610

5 Answers5

62

You're looking at it the wrong way. The no_proxy environment variable lists the domain suffixes, not the prefixes. From the documentation:

no_proxy: This variable should contain a comma-separated list of domain extensions proxy should not be used for.

So for IPs, you have two options:

1) Add each IP in full:

printf -v no_proxy '%s,' 10.1.{1..255}.{1..255};
export no_proxy="${no_proxy%,}";

2) Rename wget to wget-original and write a wrapper script (called wget) that looks up the IP for the given URL's host, and determines if it should use the proxy or not:

#!/bin/bash
ip='';
for arg; do
   # parse arg; if it's a URL, determine the IP address
done;
if [[ "$ip" =~ ^10\.1\. ]]; then
   wget-original --no-proxy "$@";
else
   wget-original "$@";
fi;
janmoesen
  • 2,710
  • Thank's a lot, I'm going to wrap my wget (and use hostnames instead of IP addresses). – SamK Oct 28 '11 at 14:27
  • 13
    Will having 255*255 addresses in no_proxy risk performance issues? – jtpereyda Dec 16 '15 at 22:50
  • @dafrazzman: that could well be. After all, it's almost 800 kilobytes large, which is somewhat extreme for an environment variable. It also depends on how large your environment can be. For this extreme case, I would recommend the wrapper approach. – janmoesen Jan 21 '16 at 10:13
  • Also, it should be 10.1.{0..255}.{0..255}, or at least 10.1.{0..255}.{1..255}. (I don't think the last bit can be 0 for "normal" devices, but I am not sure.) – janmoesen Jan 21 '16 at 10:15
  • 9
    Do not add each IP in full - it will be too long and you won't be able to execute any commands in bash. – Ross Jun 01 '16 at 18:47
  • 9
    NEVER DO THAT, this will generate something too long for any command, and, if you put this in /etc/environment file, this could break your server. – Thomas Decaux Dec 04 '17 at 15:56
  • This should be changed in wget. Other tools are able to parse the CIDR notation in no_proxy (curl, python, ruby, etc.). – Étienne Mar 05 '20 at 12:33
  • Also see: link – minus one Feb 22 '21 at 17:00
25

info wget says:

 `no_proxy'
     This variable should contain a comma-separated list of domain
     extensions proxy should _not_ be used for.  For instance, if the
     value of `no_proxy' is `.mit.edu', proxy will not be used to
     retrieve documents from MIT.

So the variable should contain a list of domains, not IP ranges. You'll need to set up proper local aliases for your local machines in /etc/hosts file(s).

Apart from this, bear in mind that setting an environment variable does not guarantee that a proxy will or will not be used. It is just an information that may be used by programs that support it.

rozcietrzewiacz
  • 39,269
  • 1
    If wget implements this the same way as curl does – and I suspect it does – then there is no difference if it is IP address or domain name as both are treated as strings – see https://github.com/curl/curl/issues/1208 – Piotr Dobrogost Dec 10 '18 at 15:52
  • @PiotrDobrogost they do not --> https://about.gitlab.com/blog/2021/01/27/we-need-to-talk-no-proxy/ – FreeSoftwareServers Sep 13 '22 at 21:10
22

Not exactly a correct solution but it might solve the problem for you. If you assume that accessing anything from the outside of the proxy will be using DNS-names and not directly using IP-addresses, you can set it like:

export no_proxy=0,1,2,3,4,5,6,7,8,9

As far as I know, no top domain ends with a number so if it does, it must be an IP-address.

muru
  • 72,889
Lars Berntzon
  • 319
  • 7
    That's a very smart solution :) Unfortunately, this will bypass the proxy for any IP, not only for local IPs... :( Is there no way (in 2018) to specify IP ranges or CIDR in the "noproxy" variable? – Sorin Postelnicu Mar 26 '18 at 16:19
  • For data center server this is a very clever trick. Has side effect, but acceptable. – notes-jj Jul 04 '18 at 17:20
  • 1
    this didn't work for myself .. had to use @iconoclast solution – Erik Nov 05 '18 at 07:54
  • It's a real hassle. Not all programs follow these variables – Time Killer Apr 17 '20 at 08:31
  • 3
    I think you need to use all numbers between 0 and 255 for this to work, otherwise, I think this solution only works for ip's ending in 0 and 9 not 0 and 255. At least your existing solution didn't work for me with curl with an IP ending in 67 until I did this: printf -v no_proxy '%s,' {0..255}; – Joel Pearson May 08 '20 at 03:29
  • @JoelPearson Followed by export no_proxy – kubanczyk Jun 03 '20 at 09:45
16

Here's a slightly simpler (one-line) approach based on janmoesen's solution. 

export no_proxy=`echo 10.1.1.{1..255} | sed 's/ /,/g'`

which has been further improved based on Cory Ringdahl's input:

export no_proxy="`echo 10.1.1.{1..254},` 10.1.1.255"

sed chokes on all the arguments passed from the expansion of 10.1.{1..255}.{1..255}, either in his code or mine. So if you really need to expand to 256 * 256 IP addresses, you might be able to further adapt Cory Ringdahl's approach, but if you merely need 256 or so, either of these should work perfectly for you.

I'm guessing the version where 2 octets are expanded would look something like this:

export no_proxy="`echo 10.1.{1..255}.{1..254},` 10.1.255.255"

but iTerm is giving me errors about argument lists being too long, so I can't say for sure whether there is a solution to make this work...

iconoclast
  • 9,198
  • 13
  • 57
  • 97
1

no_proxy seems to match the tailing part of a host with at least two chars, so set no_proxy containing 100 tailing part of a ipv4 will skip any ipv4 url

printf -v no_proxy '%s,' {10..99}
export no_proxy=".0,.1,.2,.3,.4,.5,.6,.7,.8,.9,$no_proxy"
curl -v http://any-ip/
James Z.M. Gao
  • 721