Reply on same interface as incoming?

Question

I have a system with two interfaces. Both interfaces are connected to the internet. One of them is set as the default route; a side effect of this is that if a packet comes in on the non-default-route interface, the reply is sent back through the default route interface. Is there a way to use iptables (or something else) to track the connection and send the reply back through the interface it came from?

The keyword to search for this is source-based routing, or policy-based routing (naming used in routers). Source-based because routing is selected based on the source IP of the packet, — sivann, Jul 16 '20 at 08:24
Worth noting if using modern systems with Netplan, see this complete answer: https://askubuntu.com/questions/1169002/how-to-use-netplan-to-create-two-separate-routing-tables — MadHatter, Sep 12 '23 at 16:34

score 93 · Accepted Answer · edited Nov 23 '20 at 19:13

93

echo 200 isp2 >> /etc/iproute2/rt_tables
ip rule add from <interface_IP> table isp2 prio 1
ip route add default via <gateway_IP> dev <interface> table isp2

The above doesn't require any packet marking with ipfilter. It works because the outgoing (reply) packets will have the IP address that was originally used to connect to the 2nd interface as the source (from) address on the outgoing packet.

edited Nov 23 '20 at 19:13

iBug

3,508

answered Oct 26 '11 at 19:10

Peter

946

7

Wow, this is exactly what I was looking for. In case anyone is looking, for RH-based distros, you can put the relevant rule and route commands in files named 'rule-eth0' or 'route-eth0' (for example) which will be added or removed on ifup/ifdown. Place these files alongside the ifcfg-eth0 file. For IPv6, there is 'route6-eth0' functionality built in, but no 'rule6-eth0' built in (yet). – Kyle Brantley Dec 30 '11 at 03:58
26

For me it worked only when I left out the dev param in the ip rule command, so running ip rule add from <interface_IP> table isp2 – cdauth Dec 13 '15 at 17:09
2

You can make these be created when the interface goes up by adding up ip rule add from <interface_IP> table isp2 and up ip route add default via <gateway_IP> dev ppp0 table isp2 to your /etc/network/interfaces under the relevant interface. – Gavin S. Yancey Apr 05 '16 at 04:59
6

I had to remove dev <interface> from ip rule to get it to work on my box. If I’m understanding right, dev <interface> was filtering out packets which were somehow set on the wrong interface which needed to be wrested over to the correct interface by the overridden route but the rule filtering by interface was preventing that from happening. – binki Oct 02 '16 at 03:52
I've tried this on my home machine and it works perfectly (with dev removed), but I can't for the life of me get it working on a dedicated server I rent. – goji Nov 08 '16 at 07:34
You may want to add "strict arp replies" to your configuration as well with this. that is, set the arp_filter flag to "1" - https://unix.stackexchange.com/questions/31096/sysctl-parameter-for-correct-arp-response - arp_filter - BOOLEAN / 1 - Allows you to have multiple network interfaces on the same subnet, IOW it allows control of which cards (usually 1) will respond to an arp request. / 0 - (default) The kernel can respond to arp requests with addresses from other interfaces. This may seem wrong but it usually makes sense, because it increases the chance of successful communication. – Daniel S. Sterling Jun 05 '17 at 20:10
Please correct answer with @cdauth suggestion! – lucaferrario Jan 09 '18 at 10:41
2

Like most other people, I had to remove dev <interface> from the ip rule command for this to work. Please update the answer ! Except for that detail, it worked like a charm. Thanks a lot, @Peter ! – MoonSweep May 14 '18 at 09:09
1

I was looking for a solution like this for Openvpn, that ended up being this: https://unix.stackexchange.com/questions/522136/how-to-route-all-vpn-traffic-via-specific-hardware-interface – Kevin Parker Mar 25 '20 at 04:12
2

For anyone reading this recently, the dev <interface> everybody is talking about has already been removed -- it was the dev <interface> on the rule line that was problematic, not the one on the route line (though I'm not convinced either of them is necessary). – markasoftware Dec 11 '20 at 02:20
Thanks for this answer, that's exactly what I needed to get a curl --interface <one_of_vpn_interface> <target_url> bash call to work properly, where <one_of_vpn_interface> was not the primary one. – Attila Jan 17 '21 at 13:51
@markasoftware The dev specifier on the route is only necessary if you have multiple interfaces with overlapping subnet ranges, which is usually an indication of a badly-designed network environment. – iBug Jan 18 '22 at 18:35

score 10 · Answer 2 · answered Nov 29 '10 at 23:16

The following commands create an alternate routing table via eth1 for packets that have the mark 1 (except packets to localhost). The ip command is from the iproute2 suite (Ubuntu: iproute Install iproute http://bit.ly/software-small, iproute-doc Install iproute-doc http://bit.ly/software-small).

ip rule add fwmark 1 table 1
ip route add 127.0.0.0/0 table 1 dev lo
ip route add 0.0.0.0/0 table 1 dev eth1

The other half of the job is recognizing packets that must get the mark 1; then use iptables -t mangle -A OUTPUT … -j MARK --set-mark 1 on these packets to have them routed through routing table 1. I think the following should do it (replace 1.2.3.4 by the address of the non-default-route interface):

iptables -t mangle -A OUTPUT -m conntrack --ctorigdst 1.2.3.4 -j MARK --set-mark 1

I'm not sure if that's enough, maybe another rule is needed on the incoming packets to tell the conntrack module to track them.

Nice. I forgot all about marking. That should get me there. – Shawn J. Goff Nov 30 '10 at 02:33 — Shawn J. Goff, Nov 30 '10 at 02:33
I'm sure you meant 127.0.0.0/8 not /0. – iBug Jan 07 '20 at 16:57 — iBug, Jan 07 '20 at 16:57

score 7 · Answer 3 · edited Oct 15 '18 at 20:33

I had issues with the locally generated packets with the solution suggested by Peter, I've found that the following corrects that:

echo 200 isp2 >> /etc/iproute2/rt_tables
ip rule add from <interface_IP> table isp2 priority 900
ip rule add from dev <interface> table isp2 priority 1000
ip route add default via <gateway_IP> dev <interface> table isp2
ip route add <interface_prefix> dev <interface> proto static scope link src <interface_IP> table isp2

NOTE: You may run into syntax issues with the 4th line above. In such cases the syntax for the 4th command may be this now:

ip rule add iif <interface> table isp2 priority 1000

Tried so much but nothing worked at my scenario, except this.. many thanks. — Dr. DS, May 31 '17 at 20:48

zaTricky · Answer 4 · 2021-08-21T13:34:49.920

I'm assuming you are running Linux and, further, that you are utilising a RedHat/CentOS-based distribution. Other Unix's and distributions will require similar steps - but the details will be different.

Start by testing (note that this is very similar to @Peter's answer. I am assuming the following:

eno0 is isp0 and has the overall default gateway
eno1 is isp1 and has the IP/range 192.168.1.2/24 with gateway 192.168.1.1

The commands are as follows:

$ echo 200 isp1 >> /etc/iproute2/rt_tables
$ ip rule add from eno1 table isp1
$ ip route add default via 192.168.1.1 dev eno1 table isp1

The firewall is not involved in any way. Reply packets would always have been sent from the correct IP - but previously were being sent out via the wrong interface. Now these packets from the correct IP will be sent via the correct interface.

Assuming the above worked, you can now make the rule and route changes permanent. This depends on what version of Unix you are using. As before, I'm assuming a RH/CentOS-based Linux distribution.

$ echo "from eno1 table isp1" > /etc/sysconfig/network-scripts/rule-eno1
$ echo "default via 192.168.1.1 dev eno1 table isp1" > /etc/sysconfig/network-scripts/route-eno1

Test that the network change is permanent:

$ ifdown eno1 ; ifup eno1

If that didn't work, on the later versions of RH/CentOS you also need to go with one of two options:

Don't use the default NetworkManager.service; Use network.service instead. I haven't explored the exact steps needed for this. I would imagine it involves the standard chkconfig or systemctl commands to enable/disable services.

or

Install the NetworkManager-dispatcher-routing-rules package

Personally I prefer installing the rules package as it is the simpler more supported approach:

$ yum install NetworkManager-dispatcher-routing-rules

Another strong recommendation is to enable arp filtering as this prevents other related issues with dual network configurations. With RH/CentOS, add the following content to the /etc/sysctl.conf file:

net.ipv4.conf.default.arp_filter=1
net.ipv4.conf.all.arp_filter=1

Reply on same interface as incoming?

4 Answers4

Linked

Related