Why the slow Site-to-site VPN Speeds with Open VPN and IPSec?

VPN
1

Anyone else experiencing this?
It appears Im unable to get anything higher than 2 or 11Mbps through a site to site VPN Tunnel that traverses across the Internet. :woozy_face:

However, CLIENT vpn tunnels see speeds utilizing the full internet pipe being offered. On top of that, any other services being offerd also see the full utilization of both ends of the pipe.

I have set this up in my lab, and I am getting the same results. That was surprising to me as I really had thought maybe the intenet had something to do with this. But im getting the same results :man_shrugging:.
Any Ideas on what I maybe missing or mis-understanding?

Below are my environmental variables from my lab as well as real world.

  • I have a mixture of PFSense, Unifi as well as meraki routers with various Site-to-Site links all going back to PFSense running on Dell R630 hardware.
  • Some use default ports other use custom ports.
  • hardware specs range from two dell R630 servers with an S2S link between each other, down to the little small unifi 3p routers and PFSense. - same VPN performance across all hardware configurations
  • lab network is a flat 1gbps basic 16port netgear switch that mimics the internet with each routers wan port plugged into it.
  • outside the lab, our actual real world connection at the main VPN concentrator is a set of R630’s hosted in a data center behind 2 dedicated 1gbps Fiber links in a Lagg configuration.
  • all other connectivity and speeds are performing up to specs. Including Client side VPN, which takes the same routes.
  • real world load of our pipe is roughly 30-40% utilization with infrequent bursts up to 80%
  • packet switching and routing is also nominal.
  • all measurements and speeds I have given are perfomed using IPerf.
  • experiencing this on IPSec as well.
  • only seeing these speeds with Site-to-site configurations

Any help would be appreciated :+1:

2

Few questions:

  • do all end points support AES-NI and do you have it enabled?
  • what’s the processor load look like when you run a test? Monitor on both ends with top -HaS
  • how are you testing? What’s the command?
  • what encryption are you using on the tunnels?
3

I have a site-to-site IPSec/IKEv2 tunnel between my main office and my home, and I’m iperfing ~140/140.

Tunnel setup:AES256-GCM (256 bits) `SHA256 DH14 (2048 bit)

Main office box is an MBT-2220 running 2.4.4:

CPU Type Intel(R) Atomâ„¢ CPU E3826 @ 1.46GHz2 CPUs: 1 package(s) x 2 core(s)AES-NI CPU Crypto: Yes (active)Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM

Home office box is an SG-1100 running 22.05 (equivalent to 2.6):

CPU Type ARM Cortex-A53 r0p42 CPUs:CPU 0: ARM Cortex-A53 r0p4 affinity: 0CPU 1: ARM Cortex-A53 r0p4 affinity: 1SafeXcel Crypto: Yes (active)Hardware crypto AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256,SHA384,SHA512

Line speeds are 1000/1000 on both sides, but I’m router limited even for basic internet access, as speedtest reports 400/400 at the office and 250/400 at home.

Physical distance between the two sites is less than 10 miles, and ping reports ~10 ms.

I had been getting only 30/30 until yesterday, because I was using encryption settings that were neither secure nor accelerated. Check your crypto accelerators and use something that both sides support.

Oddly enough, async crypto actually slows it down by ~10 - 15 Mbps.

Even my ancient APU (AMD G-T40E Processor, 2 CPUs: 1 package(s) x 2 core(s), AES-NI CPU Crypto: No) will do AES256-GCM (256 bits) SHA256 DH14 (2048 bit) at 56/56 Mbps.

4

Thanks for everyone’s replys! Apologies for the delay in getting back to yall. I had some person things come up that I needed to tend to.

I have taken your questions and have answered them following each question. I did go around and ensure each site/endpoint is receiving the subscribed speed. The slowest site speed is 250Mbps Down and 50mbps up. That is on a unifi box. The fastest centeral box that these connect back to is a PFSense box and it is on a dedicated symmetrical fiber link of 5Gbps Lagged across 5 1Gbps Fiber pairs.

  • Do all endpoints support AES-NI and do you have it enabled?
    • Most endpoints support AES-NI. However, I have a few Unifi Boxes that appear to not support it. Those PFSenser boxes supporting it are seeing speeds no more than 75mbps and the unifi units are seeing 26mbps.
  • What’s the processor load look like when you run a test? Monitor on both ends with top -HaS
    • I am not seeing any significant (Less than 30%) CPU Load on either side.
  • how are you testing? What’s the command?
    • I’m using IPerf in PFSense on both ends, with one listening as the server.
    • IPerf3 -s
  • what encryption are you using on the tunnels?
    • OpenVPN - AES-256-CBC - SHA256 - (PFSense to PFSense) - 76Mbps on a dedicated 1Gbps Sonet fiber circuit
    • OpenVPN - AES-256-GCM - SHA256 - (PFSense to PFSense) 76Mbps on a Coax 500Mbps down by 100Mbps up
    • OpenVPN - PF-CBC - SHA1 (PFSense to Unifi) - 20Mbps on ATT uverse shared 1Gbps fiber
    • IPSec - AES-256 - SHA1 (PFSense to Meraki) - 21Mbps - COX Coax service at 250Mbps down and 50Mbps up.
  • Has MTU Been compensated for?
    • At this time, no.
5

Addendum…

  • Has MTU been compensated for