BELK-AN-006: Enabling dual Gigabit Ethernet support on BoraEVB/BoraXEVB

From DAVE Developer's Wiki
Jump to: navigation, search
Info Box
Bora5-small.jpg Applies to Bora
BORA Xpress.png Applies to BORA Xpress
BORALite-TOP.png Applies to BORA Lite
Warning-icon.png This application note was validated against specific versions of the kit only. It may not work with other versions. Supported versions are listed in the History section. Warning-icon.png


200px-Emblem-important.svg.png

It is assumed to use ZYNQ SOC with speed grade -1 (even using command line script or GUI). In any case, there are no issues using speed grade -3 SOC provided with BELK kit


History[edit | edit source]

Version Date BELK/BXELK version Notes
1.0.0 September 2015 2.2.0 First release
1.1.0 January 2016 2.2.0, 3.0.0 Added support for BoraX/BoraXEVB platform

1.1.1

September 2016 2.2.0, 3.0.0 Added more information about MII buses organization
2.0.0 January 2020 4.1.0 / 2.1.0 AN migration to BELK 4.1.0 / BXELK 2.1.0

Introduction[edit | edit source]

Thanks to the migration to linux kernel 3.10.17, BELK 2.2.0 and BELK 3.0.0 allows to cleanly support dual Gigabit Ethernet configuration on BoraEVB and BoraXEVB. This application note describes how to implement such configuration, providing a reference design for Vivado 2014.4 and linux kernel configuration instructions.

Block diagram[edit | edit source]

Simplified block diagram of dual Ethernet configuration is shown in the following picture.


An-belk-006-bd.png
An-belk-006-boralite-01.png


First Ethernet port refers to J8 connector of BoraEVB and BoraXEVB carrier board and is based on Zynq's Gigabit Ethernet Controller 0 (Gem0). This controller is mapped at physical address 0xE000B000.

Second Ethernet port refers to J9 connector of BoraEVB and BoraXEVB carrier board and is based on Zynq's Gigabit Ethernet Controller 1 (Gem1). This controller is mapped at physical address 0xE000C000.

The fundamental difference between the two interfaces is the PHY interfacing. In case of Gem0, PHY is mounted on Bora and BoraX SoM and it is interfaced directly to Processor Subsystem (PS) via MIO pads. In case of Gem1 instead, PHY is populated on BoraEVB and BoraXEVB (U9) and it is interfaced to Programmable Logic (PL). Thus it is necessary to enable EMIO routing and to instantiate a GMII to RGMII bridge in PL as per PHY's interface requirement. About MII bus (MDIO, MDC), two different busses are used:

  • Bora PHY is connected to the signals ETH_MDC and ETH_MDIO (available on BoraEVB at JP18.7 and JP18.9 respectively)
    • this MII bus is associated to gem0
  • BoraEVB PHY is connected to PL's IO_L9N_T1_DQS_34 (ETH1_MDC) and IO_L9P_T1_DQS_34 (ETH1_MDIO); these signals are available on BoraEVB at JP18.8 and JP18.10 respectively
    • this MII bus is associated to gem1
    • it is worth to remember that a virtual PHY (whose address is 8) is connected to this bus as well; this PHY is implemented in the GMII/RGMII bridge and it is used to configure the bridge at runtime, depending on operating parameters such as the Ethernet physical link speed.

Vivado design[edit | edit source]

The following picture shows the block diagram of the Vivado project:

An-belk-006-01.png

Bora + BoraEVB[edit | edit source]

On BoraEVB PHY1 is interfaced to Programmable Logic (PL) pads that belong to bank #34.

WARNING: due to a restraint introduced, from Vivado version 2017.1 onwards, the signal ETH1_TXCK can be routed only to a pad that is MRCC or SRCC input. As result from BELK 4.0.0 an hardware rework is needed on the BoraEVB board:

  • Remove R183
  • Remove R232
  • Connect R183.2 with R232.1

This rework prevents the use of the PL SDRAM onboard of the BoraEVB (by default this ram is not mounted).

Here is the pinout assignment for the PHY1 on BoraEVB:

PHY1 Signal BORA SOM Signal
ETH1_TXD0 IO_L24P_T3_34
ETH1_TXD1 IO_L24N_T3_34
ETH1_TXD2 IO_L23P_T3_34
ETH1_TXD3 IO_L23N_T3_34
ETH1_TXCK IO_L20P_T3_34
ETH1_TXCTL IO_L20N_T3_34
ETH1_RXD0 IO_L5N_T0_34
ETH1_RXD1 IO_L5P_T0_34
ETH1_RXD2 IO_L7N_T1_34
ETH1_RXD3 IO_L7P_T1_34
ETH1_RXCK IO_L4N_T0_34 IO_L13P_T1_MRCC_34
ETH1_RXCTL IO_L4P_T0_34
ETH1_MDC IO_L9N_T1_DQS_34
ETH1_MDIO IO_L9P_T1_DQS_34

Since bank #34 is powered at 3.3V (High Range I/O mode), RGMII duty cycle distortion specification is not matched. In case of carrier board designed for production environments, it is recommended to use a lower voltage levels and thus a different PL bank. For more details please see section I/O Standard and Placement of PG160 GMII to RGMII LogiCORE IP Product Guide and this page.

The Vivado project can also be build with the procedure explained here.

BoraLite + Adapter + BoraXEVB[edit | edit source]

On BoraXEVB PHY1 is interfaced to Programmable Logic (PL) pads that belong to bank #13.

The BoraLite Adapter take care of rerouting the ETH1_RXCK to meet the Vivado requirements.

Here is the pinout assignment for the PHY1 on BoraXEVB:

PHY1 Signal BORAX SOM Signal
ETH1_TXD0 IO_L19P_T3_13
ETH1_TXD1 IO_L19N_T3_VREF_13
ETH1_TXD2 IO_L20P_T3_13
ETH1_TXD3 IO_L20N_T3_13
ETH1_TXCK IO_L16P_T2_13
ETH1_TXCTL IO_L16N_T2_13
ETH1_RXD0 IO_L17P_T2_13
ETH1_RXD1 IO_L17N_T2_13
ETH1_RXD2 IO_L18P_T2_13
ETH1_RXD3 IO_L18N_T2_13
ETH1_RXCK IO_L14P_T2_SRCC_13
ETH1_RXCTL IO_L14N_T2_SRCC_13
ETH1_MDC IO_L21N_T3_DQS_13
ETH1_MDIO IO_L21P_T3_DQS_13

I/O voltage of bank 13 must be set to 2.5V by configuring JP25 as shown in the following table.

Pins Setting
1-2 closed
3-4 open
5-6 closed
7-8 open
9-10 open
11-12 open

The Vivado project can also be build with the procedure explained here.

Borax + BoraXEVB[edit | edit source]

On BoraXEVB PHY1 is interfaced to Programmable Logic (PL) pads that belong to bank #13.

WARNING: due to a restraint introduced, from Vivado version 2017.1 onwards, the signal ETH1_TXCK can be routed only to a pad that is MRCC or SRCC input. As result from BXELK 2.0.0 an hardware rework is needed on the BoraXEVB board:

  • Remove RP84
  • Remove R232
  • Connect RP84.2 with R232.1

This rework prevents the use of the LVDS connector on BoraXEVB (J26).

Here is the pinout assignment for the PHY1 on BoraXEVB:

PHY1 Signal BORAX SOM Signal
ETH1_TXD0 IO_L19P_T3_13
ETH1_TXD1 IO_L19N_T3_VREF_13
ETH1_TXD2 IO_L20P_T3_13
ETH1_TXD3 IO_L20N_T3_13
ETH1_TXCK IO_L16P_T2_13
ETH1_TXCTL IO_L16N_T2_13
ETH1_RXD0 IO_L17P_T2_13
ETH1_RXD1 IO_L17N_T2_13
ETH1_RXD2 IO_L18P_T2_13
ETH1_RXD3 IO_L18N_T2_13
ETH1_RXCK IO_L15P_T2_DQS_13 IO_L12P_T1_MRCC_13
ETH1_RXCTL IO_L15N_T2_DQS_13
ETH1_MDC IO_L21N_T3_DQS_13
ETH1_MDIO IO_L21P_T3_DQS_13

I/O voltage of bank 13 must be set to 2.5V by configuring JP25 as shown in the following table.

Pins Setting
1-2 closed
3-4 open
5-6 closed
7-8 open
9-10 open
11-12 open

The Vivado project can also be build with the procedure explained here.

Enabling dual Ethernet configuration in linux kernel[edit | edit source]

To enable dual Ethernet user needs to get the pre-built binaries here

Alternatively kernel and device tree can be built from sources with the following procedure:

  • update Bora kernel repository (as described here)
  • build the bora-an006.dtb devicetree
  • build the updated kernel source as usual.

Put the binaries on the first (FAT32) partition of your BELK SD card, overwriting the original one if needed. Please note that you need the following files:

  • boot.bin
  • bora.dtb
  • uImage
  • fpga.bin
  • u-boot.img
  • uEnv.txt

Insert the SD card into BoraEVB or BoraXEVB and turn on the board.

During kernel boot, user can check if the second ethernet interface has been loaded succesfully:

root@bora:~# ifconfig -a
can0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          NOARP  MTU:16  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:10
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:60

eth0      Link encap:Ethernet  HWaddr 00:50:C2:B9:CF:82
          inet addr:192.168.0.209  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8424 errors:5 dropped:418 overruns:0 frame:0
          TX packets:5964 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7562500 (7.2 MiB)  TX bytes:922668 (901.0 KiB)
          Interrupt:54 Base address:0xb000

eth1      Link encap:Ethernet  HWaddr 66:94:55:CB:B1:3E
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:77 Base address:0xc000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:11 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:788 (788.0 B)  TX bytes:788 (788.0 B)

root@bora:~# ifconfig eth1 192.168.12.209
root@bora:~# [  228.492886] xemacps e000c000.ethernet: Set clk to 0 Hz
[  228.498079] xemacps e000c000.ethernet: link up (1000/FULL)
root@bora:~#

Pre-built binaries[edit | edit source]

Performance tests[edit | edit source]

To test the performances of the second Ethernet interface iperf based tests have been executed.

Test #1[edit | edit source]

In this test bed the two interfaces are connected on the same host machine (a PC running linux) via a Gigabit switch but are associated to two different subnets:

  • ETH0: 192.168.0.xxx
  • ETH1: 192.168.12.xxx

On the linux host machine iperf application is run in server mode:

bash# iperf -s

Here are the results of the test on the two Ethernet interfaces launched sequentially:

root@bora:~# iperf -c 192.168.0.210 && iperf -c 192.168.12.210
------------------------------------------------------------
Client connecting to 192.168.0.210, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.209 port 59288 connected with 192.168.0.210 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   899 MBytes   754 Mbits/sec
------------------------------------------------------------
Client connecting to 192.168.12.210, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.12.209 port 42238 connected with 192.168.12.210 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   889 MBytes   745 Mbits/sec
root@bora:~#


In case two iperf clients are run simultaneously on target, each instance's bandwitdh is roughly halved:

root@bora:~# iperf -c 192.168.0.210
------------------------------------------------------------
Client connecting to 192.168.0.210, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.209 port 59290 connected with 192.168.0.210 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   364 MBytes   305 Mbits/sec
root@bora:~#
root@bora:~# iperf -c 192.168.12.210
------------------------------------------------------------
Client connecting to 192.168.12.210, TCP port 5001
TCP window size: 48.1 KByte (default)
------------------------------------------------------------
[  3] local 192.168.12.209 port 42240 connected with 192.168.12.210 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   385 MBytes   323 Mbits/sec
root@bora:~#

Test #2[edit | edit source]

ETH0 set up is the same used in previous test. ETH1 is point-to-point connected to a second linux host machine instead.

This configuration allows to achieve better performance on the single ETH1 iperf test (780-820 Mb/s):

root@bora:~# iperf -c 192.168.12.208 -i 1 -t 6000
------------------------------------------------------------
Client connecting to 192.168.12.208, TCP port 5001
TCP window size: 43.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.12.209 port 50715 connected with 192.168.12.208 port 5001
[ ID] Interval       Transfer     Bandwidth
....
[  3] 368.0-369.0 sec  94.6 MBytes   794 Mbits/sec
[  3] 369.0-370.0 sec  92.4 MBytes   775 Mbits/sec
[  3] 370.0-371.0 sec  97.1 MBytes   815 Mbits/sec
[  3] 371.0-372.0 sec  97.1 MBytes   815 Mbits/sec
[  3] 372.0-373.0 sec  96.1 MBytes   806 Mbits/sec
[  3] 373.0-374.0 sec  94.1 MBytes   790 Mbits/sec
[  3] 374.0-375.0 sec  96.6 MBytes   811 Mbits/sec
[  3] 375.0-376.0 sec  96.8 MBytes   812 Mbits/sec
[  3] 376.0-377.0 sec  97.8 MBytes   820 Mbits/sec
[  3] 377.0-378.0 sec  93.9 MBytes   787 Mbits/sec
[  3] 378.0-379.0 sec  95.9 MBytes   804 Mbits/sec
[  3] 379.0-380.0 sec  98.2 MBytes   824 Mbits/sec
[  3] 380.0-381.0 sec  93.2 MBytes   782 Mbits/sec
[  3] 381.0-382.0 sec  96.9 MBytes   813 Mbits/sec
[  3] 382.0-383.0 sec  92.0 MBytes   772 Mbits/sec

In case of simultaneous iperf instances on ETH0 and ETH1 ports, ETH1 bandwidth drops down to 330-350 Mb/s:

[  3] 383.0-384.0 sec  42.6 MBytes   358 Mbits/sec
[  3] 384.0-385.0 sec  41.4 MBytes   347 Mbits/sec
[  3] 385.0-386.0 sec  40.9 MBytes   343 Mbits/sec
[  3] 386.0-387.0 sec  40.9 MBytes   343 Mbits/sec
[  3] 387.0-388.0 sec  40.9 MBytes   343 Mbits/sec
[  3] 388.0-389.0 sec  41.2 MBytes   346 Mbits/sec
[  3] 389.0-390.0 sec  41.4 MBytes   347 Mbits/sec
[  3] 390.0-391.0 sec  39.8 MBytes   333 Mbits/sec
[  3] 391.0-392.0 sec  40.8 MBytes   342 Mbits/sec
[  3] 392.0-393.0 sec  39.9 MBytes   334 Mbits/sec
[  3] 393.0-394.0 sec  42.2 MBytes   354 Mbits/sec