1

I have a flat network (no routing yet) of 3 servers, each with a service (http, mysqld, doesn't matter) listening on 0.0.0.0 (ip_nonlocal_bind and ip_forward are on) and running keepalived.

virtual_server 10.0.0.80 3306 {
  delay_loop 2
  lb_algo rr
  lb_kind DR
  protocol TCP

  real_server 10.0.0.81 3306 {
    weight 10
    TCP_CHECK {
     connect_timeout 1
    }
  }

  real_server 10.0.0.82 3306 {
    weight 10
    TCP_CHECK {
      connect_port    3306
      connect_timeout 1
    }
  }

  real_server 10.0.0.83 3306 {
    weight 10
    TCP_CHECK {
      connect_port    3306
      connect_timeout 1
    }
  }
}

keepalived (as the service) is working and failing to each box as appropriate. However, the virtual_server is only serving pages (or database queries, or whatever) from the box keepalived currently has the IP on, they fail the other 2/3 of the time (weighted evenly).

Example: When BOX1 has the keepalived address, requests will WORK, FAIL, FAIL, repeat, it only answers with "box1". When BOX2 as the keepalived address, requests will FAIL, WORK, FAIL, only answering as "box2".

I am convinced the non-keepalived IP boxes are refusing to answer the query because they do not own or do not know they should be answering as the keepalived IP. How do I get the non-keepalived boxes to answer always?

This is not my first keepalived setup, but this is my first virtual_server setup. I just need a load balancer, I do not need high availability provided by HAProxy.

jnovack
  • 121

1 Answers1

1

Thank to the help in a serverfault.com question I was able to solve my issue.

Short Answer:

I add the virtual IPs to my dummy interface and ensure net.ipv4.conf.default.accept_source_route is 0.

Long Answer:

The purpose of a dummy interface is to easily disable/enable/failover keepalived without stopping and starting the service. Just up or down the dummy interface to fail the VIP to another server.

To do this easily, I create a systemd service called /usr/local/src/dummy.service.

[Unit]
Description=Create dummy network interface
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/sbin/ip link set dummy0 up
ExecStop=/usr/sbin/ip link set dummy0 down
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Then enable it.

# systemctl enable /usr/local/src/dummy.service

Additionally, I load the dummy module drivers.

File /etc/modules-load.d/dummy.conf:

dummy

File /etc/modprobe.d/dummy.conf:

alias dummy0 dummy
options dummy numdummies=1

I find it easier to shutdown -r now at this time to prove it works, but you can reload modprobe if you desire.

You should then see that you have a new interface:

# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: dummy0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 12:23:34:45:56:7a brd ff:ff:ff:ff:ff:ff
       valid_lft forever preferred_lft forever

Within keepalived.conf I track my vrrp_instance with:

vrrp_instance mysql {
  ...
  track_interface {
    dummy0
  }
  ...
}

Here's what I needed to change to work.

I needed to add the VIP IP address to the dummy0 interface. Modify /usr/local/src/dummy.service and # systemctl daemon-reload

ExecStart=/usr/sbin/ip link set dummy0 up && ip addr add 10.0.0.100 dev dummy0

I needed to ensure that source routing was not enabled so that any network device could answer the query, then reboot.

 # cat "net.ipv4.conf.default.access_source_route = 0" > /etc/sysctl.d/10-keepalived.conf

For completeness, my entire /etc/sysctl.d/10-keepalived.conf:

net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.conf.eno16777736.arp_ignore = 1
net.ipv4.conf.eno16777736.arp_announce = 2
net.ipv4.conf.eno16777736.rp_filter = 2
net.ipv4.conf.default.accept_source_route = 0
Community
  • 1
jnovack
  • 121