Communication fails on DSA ethernet ports of Banana Pi R1 (Lamobo R1)

Question

I don't get TCP/IP communication running on my Banana PI R1 on the switch ports (i.e., not the separate LAN port which shows up as eth0, which is working). It runs "Debian GNU/Linux 9.8", upgraded from an Armbian image with Debian 8 [1].

This is my /etc/network/interfaces file:

allow-hotplug eth0
iface eth0 inet dhcp

# Local loopback
auto lo
iface lo inet loopback

# Router ports
auto lan1
iface lan1 inet static
    address 192.168.12.253
        netmask 255.255.255.0
        network 192.168.12.0
        broadcast 192.168.12.255
        gateway 192.168.12.253
        dns-search greatsoft.local

auto lan2
iface lan2 inet static
        address 192.168.13.253
        netmask 255.255.255.0
        network 192.168.13.0
        broadcast 192.168.13.255
        gateway 192.168.13.253
        dns-search greatsoft.local

auto lan3
iface lan3 inet static
        address 192.168.14.253
        netmask 255.255.255.0
        network 192.168.14.0
        broadcast 192.168.14.255
        gateway 192.168.14.253
        dns-search greatsoft.local

auto lan4
iface lan4 inet static
        address 192.168.15.253
        netmask 255.255.255.0
        network 192.168.15.0
        broadcast 192.168.15.255
        gateway 192.168.15.253
        dns-search greatsoft.local

After boot, ifconfig reports these lines (lo0/wlan0 omitted):

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.113  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::47:aff:fec1:f535  prefixlen 64  scopeid 0x20<link>
        ether 02:47:0a:c1:f5:35  txqueuelen 1000  (Ethernet)
        RX packets 4581  bytes 816911 (797.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2392  bytes 294521 (287.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 49  

lan1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.12.253  netmask 255.255.255.0  broadcast 192.168.12.255
        ether 02:47:0a:c1:f5:35  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lan2: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.13.253  netmask 255.255.255.0  broadcast 192.168.13.255
        ether 02:47:0a:c1:f5:35  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lan3: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.14.253  netmask 255.255.255.0  broadcast 192.168.14.255
        ether 02:47:0a:c1:f5:35  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lan4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.15.253  netmask 255.255.255.0  broadcast 192.168.15.255
        inet6 fe80::47:aff:fec1:f535  prefixlen 64  scopeid 0x20<link>
        ether 02:47:0a:c1:f5:35  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 13  bytes 1070 (1.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The output of "ip -d link show" is (lo0/wlan0 omitted):

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 02:47:0a:c1:f5:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
3: lan2@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 02:47:0a:c1:f5:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 switchid 00000000 
4: lan3@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 02:47:0a:c1:f5:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 switchid 00000000 
5: lan4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 02:47:0a:c1:f5:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 switchid 00000000 
6: wan@eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:47:0a:c1:f5:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 switchid 00000000 
7: lan1@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether 02:47:0a:c1:f5:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 switchid 00000000

For example, if I connect a cable on the lan 4 port and give the connected system an IP address of 192.168.15.250 with netmask 255.255.255.0, ping from Banana PI says:

From 192.168.15.253 icmp_seq=1 Destination Host Unreachable

arp -n shows

Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.15.250                   (incomplete)                              lan4

It is unclear to me why communication fails. I guess there must be anything missing. Does anybody have a hint for me?

[1] https://www.armbian.com/lamobo-r1/

The problem starts in that they are not really 5 gigabit ports, but one port with 5 different entities under a switch chipset that is not entirely a switch.... — Rui F Ribeiro, Mar 03 '19 at 20:44

score 4 · Answer 1 · edited Mar 24 '19 at 13:50

Per default in ArmBian and also in the former Bananian, the 4 ports besides the WAN port are bridged; they are only presented as a single IP interface, being the second interface the WAN port. That is why your IP OS level configuration was not working. Beware in the R1, the ports/chipset at boot time boots as a 5 port hub because the board lacks one resistor connected to the BCM53125 for that not to happen. Worse yet, if Linux kernel does not boot, it stays as a 5 port hub.

The BCM53125 switch chipset must be programmed to turn each port into an individual VLAN, not bridged with any other port.

Beware of the numbering of ports. At least in the old kernel config switching, have not tested the new DSA one, they were:

|2|1|0|4| |3|

The following configuration should be useful for setting up the switch VLAN, with each of the 5 ports as an independent port/VLAN.

When the boot is finished:

eth.10 (VLAN 10) will be your former eth0, 192.168.1.x interface
eth.12 (VLAN 12) will be your former lan1, 192.168.12.x interface
eth.13 (VLAN 13) will be your former lan2, 192.168.13.x interface
eth.14 (VLAN 14) will be your former lan3, 192.168.14.x interface
eth.15 (VLAN 15) will be your former lan4, 192.168.15.x interface

Beware those are internal VLANs to the chipset. The Broadcom BCM53125 does not support 802.1Q. So thus, the use of the untagged directive, in the following script.

Create the file at: /etc/network/if-pre-up.d/dsa

#!/bin/bash

# to avoid execute more than once
[ "$IFACE" == "eth0" ] || exit 0

ip link set eth0 up

# ** create the native VLAN **    
ip link add link eth0 name eth0.10 type vlan id 10
ip link add link eth0 name eth0.12 type vlan id 12
ip link add link eth0 name eth0.13 type vlan id 13
ip link add link eth0 name eth0.14 type vlan id 14
ip link add link eth0 name eth0.15 type vlan id 15

# ** ALLOCATION VLAN
bridge vlan add vid 10 dev wan pvid untagged
bridge vlan add vid 12 dev lan1 pvid untagged
bridge vlan add vid 13 dev lan2 pvid untagged
bridge vlan add vid 14 dev lan3 pvid untagged
bridge vlan add vid 15 dev lan4 pvid untagged




# ** Delete VLAN default 1 **
bridge vlan del vid 1 dev wan

bridge vlan del vid 1 dev lan1
bridge vlan del vid 1 dev lan2
bridge vlan del vid 1 dev lan3
bridge vlan del vid 1 dev lan4
bridge vlan del vid 1 dev eth0.10
bridge vlan del vid 1 dev eth0.12
bridge vlan del vid 1 dev eth0.13
bridge vlan del vid 1 dev eth0.14
bridge vlan del vid 1 dev eth0.15

# ** ACTIVATE /Start **
ip link set up wan

ip link set eth0.10 up
ip link set eth0.12 up
ip link set eth0.13 up
ip link set eth0.14 up
ip link set eth0.15 up

ip link set lan1 up
ip link set lan2 up
ip link set lan3 up
ip link set lan4 up

Make the file executable: chmod 0777 /etc/network/if-pre-up.d/dsa

Checking it all after boot ip -d link show eth0.10 ip -d link show eth0.11 ....

bridge vlan show bridge link show

Remove all rights from other VLAN files, so it is read-only: chmod 0444 /etc/network/if-post-down.d/vlan chmod 0444 /etc/network/if-pre-up.d/vlan

Script untested and adapted from the guide: BPi-R1 - Distributed Switch Architecture 2017

In the linked guide, there is also a script for turning off the switch.

Disclaimer: I won't claim the script works at first try, but should be a good start. I will be testing it in mine R1 in 6-7 weeks time too, have not the time before that.

PS: has your R1 random crashes every other day? Mine had mysterious read-only filesystem situations bi-weekly.

I tracked down the cause, and solved it cutting out the realtek wifi chipset.

Someone later on, doing electrical measurements, managed to track down that occasionally the realtek chipset overloads completely the R1, causing instability, even when it is not being used, confirming my empirical findings.

PPS: When I bought my R1, I tried to make it an AP. A lost cause. See related: Wi-Fi problems using ASUS USB-N13 adapter (realtek)

See also, related https://electronics.stackexchange.com/questions/236530/dht21-am2301-sensor-not-measuring-humidity/236751

Thank you very much for your answer. I am really sorry to tell you that your solution did not work. However, it helped me to better understand the problem. By searching for "linux BCM53125 set router mode", I came to https://github.com/armbian/build/issues/511 which contains some configuration snippets that I could compile for my needs. — Michael Kremser, Mar 05 '19 at 20:38
I was suspecting something could be missing. But you have a link for the full tutorial, which will be not as specific as you need. I am needing to do this to my R1 too, but will be abroad the whole month. I also have an interesting in putting it to work. Thanks for the feedback. — Rui F Ribeiro, Mar 05 '19 at 20:42

Michael Kremser · Answer 2 · 2019-03-05T20:49:03.673

The snippets in https://github.com/armbian/build/issues/511 gave me the right direction and I ended up with this /etc/network/interfaces file:

auto lo
iface lo inet loopback

auto eth0.101
iface eth0.101 inet dhcp
    pre-up ip link add br53125 type bridge
    pre-up ip link set wan master br53125
    pre-up bridge vlan add vid 101 dev wan pvid untagged
    pre-up bridge vlan del vid 1 dev wan
    pre-up ip link set wan up
    post-down ip link set wan down
    post-down ip link del dev eth0.101

auto eth0.102
iface eth0.102 inet manual
    pre-up ip link set lan1 master br53125
    pre-up ip link set lan2 master br53125
    pre-up ip link set lan3 master br53125
    pre-up ip link set lan4 master br53125
    pre-up bridge vlan add vid 102 dev lan1 pvid untagged
    pre-up bridge vlan del vid 1 dev lan1
    pre-up ip link set lan1 up
    pre-up bridge vlan add vid 102 dev lan2 pvid untagged
    pre-up bridge vlan del vid 1 dev lan2
    pre-up ip link set lan2 up
    pre-up bridge vlan add vid 102 dev lan3 pvid untagged
    pre-up bridge vlan del vid 1 dev lan3
    pre-up ip link set lan3 up
    pre-up bridge vlan add vid 102 dev lan4 pvid untagged
    pre-up bridge vlan del vid 1 dev lan4
    pre-up ip link set lan4 up
    post-down ip link set lan4 down
    post-down ip link set lan3 down
    post-down ip link set lan2 down
    post-down ip link set lan1 down
    post-down ip link del dev br53125
    post-down ip link del dev eth0.102

auth eth0.102
iface eth0.102 inet static
    address 192.168.12.254
    netmask 255.255.255.0
    gateway 192.168.12.254
iface eth0.102 inet6 static
    address 0db8:cafe:beef:c::
    netmask 64
    gateway 0db8:cafe:beef:c::

allow-hotplug wlan0
iface wlan0 inet static
    address 192.168.13.254
    netmask 255.255.255.0
    gateway 192.168.13.254
iface wlan0 inet6 static
    address 0db8:cafe:beef:d::
    netmask 64
    gateway 0db8:cafe:beef:d::

Unfortunately, IPv6 does not work yet on the LAN, but that should be handled in another question.

The result is that the switched LAN ports behave like it would be actually one port, so all of them share one IP address and the DHCP server gives out addresses for the same subnet 192.168.12.0. Having four different subnets would have been great, but for my purpose the current behaviour is acceptable.

Cool stuff. I will have to test it out too, in 1 months time! — Rui F Ribeiro, Mar 05 '19 at 20:48
hmmmm if they behave like 1 port for DHCP, the VLANs are not yet properly established then, you are just mixing several addresses/LANs in one segment. I have 4 IPs on my R1, 1 for the WAN, and 3 for the LAN side because I programmed it to bridge two of the ports, but unfortunately the configurations are in the old kernel switching technology. — Rui F Ribeiro, Mar 05 '19 at 21:02

Communication fails on DSA ethernet ports of Banana Pi R1 (Lamobo R1)

2 Answers2