0

I have a system with two network interfaces, both of which are connected to the internet with different IP addresses. I would like responses to incoming requests to be sent over the same interface that they were received at.

The public IP addresses of my network interfaces change every 24 hours due to the nature of German internet providers. This is the main difference to the similar question Reply on same interface as incoming?, whose responses all rely on defining IP rules based on the IP address of the interface.

Can I configure an IP rule that will choose the routing table based on the interface rather than its IP address? Ideas that I have had so far but that I have not managed to apply successfully yet are to use the iif or oif options or to set an fwmark using iptables mangle.

cdauth
  • 1,447
  • You probably want to use ct mark setting and matching based on the original iif or dst ether address to set meta mark / fwmark. My answer for this question might give you insight: https://serverfault.com/a/1095524/569990 – Tom Yan Mar 08 '22 at 05:42
  • (The case there is about gateway checking while yours is about interface checking; but the drill required is pretty similar. Just remember in your case the ether address that you want to match should be the dst one instead the src one of the traffic; or just match interface) – Tom Yan Mar 08 '22 at 05:53
  • Another option would be two have two network namespaces, give each one of the interfaces, and run the service(s) that respond to requests in both namespaces. Easier than fiddling with iptables and marking. – dirkt Mar 08 '22 at 08:16
  • @dirkt Network namespaces look interesting, but I guess that would mean running each service (such as sshd) twice, once for each namespace? That sounds complicated... – cdauth Mar 08 '22 at 10:49
  • Anyway, running UDP services (by opposition to TCP services) will get additional challenge in multi-homed environment: the service has to bind either to each address or else to use the IP_PKTINFO socket option. Else expect trouble, with the service's reply using the wrong IP address or interface and requiring additional bandaid. UDP isn't as transparent as TCP here. This issue is avoided with network namespaces. – A.B Mar 08 '22 at 10:53
  • Yes, you need to run each service (you didn't tell us which or how many you need) twice. However, that's easy if you are, say, using systemd already: docs – dirkt Mar 08 '22 at 15:01

1 Answers1

1

Thank you for all your hints in the comments. Based on them I have come up with two possible solutions.

Solution 1: ip rules

Assuming that I have two interfaces eth1 and eth2, and eth1 should be the default device to use for outgoing connections, I will define a second routing table that will use eth2 and set up an ip rule that uses this routing table in response to incoming requests on eth2:

iptables -t mangle -A INPUT -j CONNMARK -i eth2 --set-mark 2
iptables -t mangle -A OUTPUT -j CONNMARK -m connmark --mark 2 --restore-mark
ip route add 192.168.1.0/24 dev eth2 table 1234
ip route add default via 192.168.1.1 table 1234
ip rule add fwmark 2 lookup 1234

(For IPv6, repeat the same using ip6tables and ip -6.)

CONNMARK adds a mark to the connection (including the response) as opposed to MARK, which adds a mark to the (incoming) packet only. The INPUT rule sets the connmark on requests coming in on eth2 and their responses. The OUTPUT rule with --restore-mark will copy the connmark into a mark, which is necessary because the fwmark matcher of the ip rule matches only marks but not connmarks.

In the OUTPUT rule I added -m connmark --mark 2 to make sure that the mark is only set for packages that have a connmark of 2. I added this because otherwise it was overriding the mark of all outgoing connections, which was for example breaking Wireguard which relies on its own marks.

Routed connections

The above rules only match incoming connections on eth2 that are dealt with by the machine itself, but it does not match connections that are routed to another machine. To support that use case, use these additional rules:

iptables -t mangle -A PREROUTING -j CONNMARK -m connmark --mark 2 --restore-mark
iptables -t mangle -A PREROUTING -j CONNMARK -i eth2 --set-mark 2

The second rule will mark every packet coming in through eth2, while the first rule will match every packet whose connection already has a connmark of 2 and will copy that mark from the connection to the packet (making sure that the response to a packet that originally came in through eth2 gets marked as well).

For this to work, the route destinations need to be added to table 1234 as well!

Solution 2: ip namespaces

With this solution, the two network interfaces will be completely separated from each other. Each process will be assigned to one of the namespaces and will thus see only one of the two network interfaces. This means that services that should be reachable through both network interfaces (for example ssh) need to be started twice, once for each namespace. Since the processes and network interfaces are strictly separated from each other, the response is automatically sent over the right interface. As far as I understand, this even means that when I SSH onto the machine over one specific network interface, any outgoing network requests that I initiate from that SSH session will always use that same network interface.

I have not tried out this solution, since it is not quite perfect for my use case, but from the information that I could find online it should be configured like this:

ip netns add net2
ip link set eth2 netns net2

You can then run any command in the scope of namespace net2 by using ip netns exec net2 <command>. Any DHCP clients or other software configuring or using the network interface would have to be started this way. It seems to me that there is no wide support for namespaces among the network configuration solutions of the different Linux distributions yet, so some manual scripting will probably be necessary to get this to work.

cdauth
  • 1,447
  • 1
    But just remember that in solution 1 some UDP services (not all) will choose the wrong source reply: they will require additional fiddling (incl NAT). Probable reproduction: socat udp4-listen:5555,fork system:'printenv | grep ^SOCAT' will reply correctly only when queried on the "main" IP address, while (the more complex) socat udp4-recvfrom:5555,ip-pktinfo,reuseport,fork system:'socat udp4\:$SOCAT_PEERADDR\:$SOCAT_PEERPORT\,bind=$SOCAT_IP_DSTADDR\:5555\,reuseport system\:'\\\''printenv | grep ^SOCAT'\\\' will reply correctly in all cases. – A.B Mar 09 '22 at 09:42
  • I wonder if eliminating default route from the main table (in other words, use the trick for both interfaces) helps in the UDP case. (Assuming it doesn't pick a source address as per the intetface enumeration / randomly.) – Tom Yan Mar 09 '22 at 11:02
  • @TomYan there must be a rule + route with "from 0.0.0.0" somewhere, usually that's the main table (since "from all"). If it's not found, only applications with a socket bound to a local address will be able to emit correctly (so usually tcp services will since they answer using a socket with a local address already in place, then most client apps will fail and some udp services will fail). Anyway this assumes more ip rule from... in addition to those with the marks. – A.B Mar 09 '22 at 12:38
  • I have updated the rules from solution 1 after running into some problems. Your socat example seems to work for me with the new solution. I didn't try it with the old one. – cdauth Feb 14 '24 at 16:56