What happen when the VCE is having multiple tunnels to different SD-WAN Gateways (VCG)?
In the housekeeping section, it is mentioned the SD-WAN Edge (VCE) is by purpose to get assigned with a single VCG. This is because it will make manipulate the packet loss and latency value much easier. However, in production environment, the SD-WAN Edge is assigned with at least two VCG. Then in this situation, SD-WAN Edge Internet WAN link will have at least two overlay tunnels, one to each of the VCG. Since two overlay tunnels will have different latency/jitter/packet loss, this section will check which tunnel value will be used for the SD-WAN Edge to select WAN link for Internet Local Breakout.
Edge Configuration for Test 5.1
Firstly, the business policy of Edge-1 is configured with link steering Auto for Internet local breakout traffic. The following is the screen capture of the business policy configuration:
In order to check when there is multiple overlay tunnels on a single Internet WAN link to different VCGs, Edge-1 will be assigned with two VCG with vcg-40-sfpg02 (24.11.0.55) as primary and vcg-40-sfpg01 (24.11.0.54) as secondary, the following screen captures from remote diagnostics and “debug.py –path” shows now the Edge-1 is have two VCG assigned, which each Internet WAN link forms overlay tunnel to two VCGs:
The output of “debug.py –path”:
The “debug.py –gateway” can help to confirm vcg-40-sfpg02 (24.11.0.55) is the primary gateway:
The following is the “Monitor–>Overview” of Edge-1, GE3 (98.1.2.19) is configured with 5Mpbs/5Mbps while GE4 (184.1.2.30) is with 10Mbps/10Mbps.
Test 5.1 – Introduce 4% packet loss to overlay tunnel between GE4 (184.1.2.30) and VCG vcg-40-sfpg02 (24.11.0.5)
In the situation where all the overlay tunnels having latency/jitter/packet loss in GREEN, according from previous section “When use “Auto” link steering, and every links are GREEN, which link will be picked?”, we understand the Internet WAN link with larger remaining downstream bandwidth will be selected. In this case (where there is no traffic), the Internet WAN link selected will be GE4 (184.1.2.30) as the downstream bandwidth is 10Mbps which is larger than GE3 (98.1.2.19) downstream bandwidth with 5Mbps.
In Test 5.1, packet loss with a target of 4% will be introduced to the overlay tunnel between GE4 (184.1.2.30) and vcg-40-sfpg02 (24.11.0.55). And there will be no packet loss for overlay tunnel between GE4 (184.1.2.30) and vcg-40-sfpg01 (24.11.0.54). This is a specifically created scenario such that GE4 (184.1.2.30) is RED to primary gateway vcg-40-sfpg02 (24.11.0.55) while GREEN to secondary gateway vcg-40-sfpg01 (24.11.0.54).
Let’s check the “debug.py –path” output with this scenario:
The above screen capture shows, the introduction of packet loss is applied to the overlay tunnel we want. In the output, there are 4 overlay tunnels, three of them are without any packet loss. The only overlay tunnel with packet loss is the tunnel between (184.1.2.30) and vcg-40-sfpg02 (24.11.0.55), which having 3.08% loss at the Rx direction and 4.41% loss at the Tx direction.
Let’s check the output of “debug.py –dec”:
The output of “debug.py –dec” shows overlay between (184.1.2.30) and vcg-40-sfpg02 (24.11.0.55) is RED on every type of traffic as the packet loss is larger than 3%. The output actually gives us a hint for predicting the result. Take a look on the two lines for GE4 with “Dest” N/A, these two lines show GE4 as the WAN link only (not with specific destination peer) is GREEN.
Let’s generate some traffic from Client100 (192.168.200.100), firstly, access web service of wordpress05 (43.254.254.14). The following is the web access log from wordpress05 (43.254.254.14):
From the wordpress05 (43.254.254.14) access log, the source IP is 184.1.2.30, this means Edge-1 selected GE4 (184.1.2.30) for local Internet breakout when link steering is Auto.
Let’s give a try to generate traffic by iperf3 from Client100 (192.168.200.100) to wordpress05 (43.254.254.14). The following is the screen capture from the iperf3 client:
The following screen capture is the iperf3 server at wordpress05 (43.254.254.14):
With the screen capture from the iperf3 server, it is showing the connection is from 184.1.2.30:20006. This means Edge-1 selected GE4 (184.1.2.30) for local Internet breakout when link steering is Auto.
The following is the screen capture of the “List Active Flows” for the completeness of the test.
With the above test result, the conclusion is, when SD-WAN Edge is having multiple overlay tunnels to different VCGs in a WAN link. Only the best quality overlay tunnel is used to determine that particular WAN link is GREEN/YELLOW/RED. Take the above test as an example, GE4 (184.1.2.30) is having two overlay tunnels, one to vcg-40-sfpg02 (24.11.0.55) which is RED. And there is a another overlay tunnel to vcg-40-sfpg01 (24.11.0.54) which is GREEN. In this scenario, the GE4 (184.1.2.30) is consider as GREEN as the best quality overlay tunnel is GREEN.
Note: I believe the idea behind this is, if the best quality overlay tunnel is poor, such as YELLOW/RED, then it is very likely this WAN link is with poor quality at that point of time. That’s why only the best quality overlay tunnel is used to determine the WAN link’s quality.