2

I've been looking around for a while and can't seem to find a good answer for this so I thought I'd ask before spending another couple of days banging my head on my desk.

I have a Ubuntu box with two physical interfaces and one virtual interface.

  eno1 - 172.16.0.100
  eno2 - 172.16.0.101
eno1:0 - x.x.x.x

What I'd like to get

  1. For responses to incoming packets I'd like the packets to go out on the interface its request came in on.
  2. For outgoing packets I'd like them to go out by default on...

    a. eno1 - for packets destined for private networks (multiple non-contiguous 172.16.x.0 ranges) b. eno1:0 - for packets destined for all other networks

Current Setup

ip rule list

0:  from all lookup local 
32760:  from all to x.x.x.x lookup eno1:0 
32761:  from x.x.x.x lookup eno1:0 
32762:  from all to 172.16.0.101 lookup eno2 
32763:  from 172.16.0.101 lookup eno2 
32764:  from all to 172.16.0.100 lookup eno1 
32765:  from 172.16.0.100 lookup eno1 
32766:  from all lookup main 
32767:  from all lookup default

ip route list table eno1:0

default via x.x.x.1 dev eno1 

ip route list table eno1

default via 172.16.0.1 dev eno1 
172.16.0.0/24 dev eno1  scope link  src 172.16.0.100 

ip route list table eno2

default via 172.16.0.1 dev eno2 
172.16.0.0/24 dev eno2  scope link  src 172.16.0.101

ip route list

default via 172.16.0.1 dev eno1 onlink 
x.x.x.0/23 dev eno1  proto kernel  scope link  src x.x.x.x 
172.16.0.0/24 dev eno2  proto kernel  scope link  src 172.16.0.101 
172.16.0.0/24 dev eno1  proto kernel  scope link  src 172.16.0.100

sysctl values for both eno1 and eno2

arp_filter=1
arp_ignore=1
arp_announce=2

Problems

  1. I can sporadically reach eno1 and eno2 from ranges outside their subnets but I can't reach eno1:0 at all.
  2. From the box I can't reach the internet (public IPs) at all.
UAdmin
  • 21

1 Answers1

0

Discovered - or rather re-discovered - the answer...

The problem is Unix's belief that the host is the network entity rather than the port which causes all kinds of arp problems on multi-homed servers. So basically I needed to limit arp responses on an interface to the IP addresses associated with that interface, otherwise Unix will respond on ANY interface regardless of the IP - which really confuses the heck out of all the other devices on the subnet.

So... Installed arptables

sudo apt-get -y install arptables

Limited the responses on each interface to the IPs on the interface

sudo arptables -n -v --line-numbers -L
Chain INPUT (policy DROP 6011K packets, 168M bytes)
1 -j ACCEPT -i eno1 -o * -d x.x.x.x , pcnt=2496 -- bcnt=69888 
2 -j ACCEPT -i eno1 -o * -d 172.16.0.100 , pcnt=294 -- bcnt=8232 
3 -j ACCEPT -i eno2 -o * -d 172.16.0.101 , pcnt=294 -- bcnt=8232 

Chain OUTPUT (policy DROP 0 packets, 0 bytes)
1 -j ACCEPT -i * -o eno1 -s x.x.x.x , pcnt=2503 -- bcnt=70084 
2 -j ACCEPT -i * -o eno1 -s 172.16.0.100 , pcnt=295 -- bcnt=8260 
3 -j ACCEPT -i * -o eno2 -s 172.16.0.101 , pcnt=294 -- bcnt=8232 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)

And voila, all IPs are responding on their correct ports.

Man! I hope I don't forget this again!

UAdmin
  • 21