Iptables NAT / Kernel IP forwarding limited to ~10Mbit

Question

I am currently setting up my iot router and ran into some issues regarding connection speed. The router itself is a cascading router. My Internet speed is 100Mbit which I verified by directly connecting to the main router via speed test. However, if I connect it via my cascading router, I only get a connection speed of between 10-18Mbit. I think either Kernel IP forwarding or Iptables NAT is likely misconfigured.

The Operating system is Debian 8 Kernel Version 3.4 (Bananian Linux)
The Router itself is a Banana PI BPI R1
Iptables is running version v1.4.21

The relevant commands I ran to set up my network are

iptables -P FORWARD ACCEPT
iptables -t nat -A POSTROUTING -o eth0.101 -j MASQUERADE

(eth0.101 is the output interface that is connected to the main router.)

ip forwarding is enabled via systemctl
ipv6 is completely disabled
since the router’s network card uses an internal switch, I have to use vlans to separate the "lan" from the "wan" I achieve this via the tool swconfig

swconfig dev eth0 set reset 1
swconfig dev eth0 set enable_vlan 1
swconfig dev eth0 vlan 101 set ports '3 8t'
swconfig dev eth0 vlan 102 set ports '4 0 1 2 8t'
swconfig dev eth0 set apply 1

Why do I think this is the NAT/Forwarding? My first thought was, well my network card is not capable of higher speeds even though it says it is. However to confirm this I ran a socks5 proxy on my router and disabled IP forwarding for the test, when running a speedtest via this socks5 proxy I was able to achieve the 100Mbit which makes me conclude that it is not my network card that is bottlenecking this.

I have tried a few things, including increasing the sizes of packet queues for my VLAN interfaces as they were zero, this changed nothing.

I also do not think that the CPU of my router is too weak to run this, because why would it be strong enough to work with a generic socks5 proxy and too weak to work with iptables?

Here is an output of ifconfig:

eth0      Link encap:Ethernet  HWaddr 02:07:0b:02:15:ac
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:370503 errors:0 dropped:0 overruns:0 frame:0
          TX packets:365330 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:310436570 (296.0 MiB)  TX bytes:308685327 (294.3 MiB)
          Interrupt:117 Base address:0xc000

eth0.101  Link encap:Ethernet  HWaddr 02:07:0b:02:15:ac
          inet addr:192.168.178.2  Bcast:192.168.178.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:209032 errors:0 dropped:0 overruns:0 frame:0
          TX packets:171418 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:203959632 (194.5 MiB)  TX bytes:102579119 (97.8 MiB)

eth0.102  Link encap:Ethernet  HWaddr 02:07:0b:02:15:ac
          inet addr:10.8.0.1  Bcast:10.8.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:161471 errors:0 dropped:0 overruns:0 frame:0
          TX packets:193912 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:99807884 (95.1 MiB)  TX bytes:204644888 (195.1 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

as well as /etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0.101
iface eth0.101 inet static
        address 192.168.178.2
        network 192.168.178.0
        netmask 255.255.255.0
        gateway 192.168.178.1
        nameserver 8.8.8.8

auto eth0.102
iface eth0.102 inet static
        address 10.8.0.1
        network 10.8.0.0
        netmask 255.255.255.0

Any ideas would be very much appreciated.

Answer 1

The Banana R1/Lamobo R1 while an interesting piece of hardware has too many shortcomings.

Firstly, the "switch" internal interface bandwidth is shared. A theoretical 1GBps tops for all shared 5 ports; the official speed people been able to get from it per interface is around 300Mbit.

Second, it has to be setup for that, in the device tree (overlays?) by the OS being used - cannot remember the specific details. Or otherwise it will be slow.

Bananian Linux is an ugly hack, does not work well, it will give you problems, and may not setup well your gigabit switch. Furthermore, Bananian is officially a deprecated project since the end of the 1st quarter of 2017, and security updates for it will stop appearing in a couple of months.

I used the R1 with Armbian for a while; it worked well. I also cut physically the realtek wifi from it, it only created instability even when not being used.

You might also got power issues with mechanical hard disks; I used an SSD.

As a recommendation, stop using Bananian, and try ArmBian. Beware the switch interface is different in Armbian as it uses a more recent kernel 4.x.

Lastly, do not even waste time trying the OpenWRT version for R1. It is a botched job and is full of hacks for working around the big firewall of China.

Leaving now R1 specific considerations, and going to the routing side, one optimisation that can be done in most consumer-grade ISP routers is setting up a port with bridging and connecting your R1 there. Thus, your outside interface will get a public IP address, and your NAT won't have again a double NAT from the ISP. (I am doing the same here)

PS For readers coming here. The R1/R1S is not worth your time and money, get instead an AP router that can be hacked with OpenWRT.