We have about 40 employees that connect to the VPN on a daily basis. Multiple times a week I see people put in tickets for VPN being down.
I also work from home quite a bit, and I hate to be that guy, but my VPN is connected from when I sign in in the morning to when I sign off for the day. It never disconnects. Sometimes if I forget to disconnect it will maintain a connection through the night.
This, along with a general lack of alerts or notifications on the firewall lead me to believe that the issue is related to their internet specifically. People don’t seem to like to hear that. I even had someone go as far as call Comcast and complain, to which Comcast did a “remote check” of some sort and concluded that it HAD to be our VPN or network infrastructure at the office.
Now that they’ve heard “officially” from a “network expert” that the issue is on our end, they’re not accepting any of my explanations or suggestions. This is now got the visibility of the CEO, and he’s asked me “why can’t IT just get this right already?” He is clearly frustrated.
Have you had to deal with this? Is there anything else I can check to cover my own ass?
Back when my company had a small WFH footprint (Myabe 10-15 people) I’d spend a lot of time going over the logs to see if I could find anything indicating an issue with our ISP, Firewall, etc.
I can’t recall a time I ever found an issue on our end.
Now that we have a larger WFH footprint I look to see what the longest connection times are.
If they’re all relatively long, I tell the person with the issue that it’s either their home network or their ISP and suggest they contact their ISP because I have 200+ other people who haven’t reported any issues.
We narrowed it down to our local ISP/s combo modem/router boxes but they still refuse to admit it.
Users in my area with the combo router/modem from the local ISP get dropped and reconnected fairly often. Users with their own modem and router do not.
I tell them to keep calling the ISP as it works fine without their hardware.
99.5% of the time when I investigate these type of complaints, the issue is due to packet loss on the users ISP. (almost never on their WiFi, the packets drop after their WiFi gateway on the way to the internet)
And since they have “Home” level ISP service, when they call support, they get asked if they can get to web pages and when they say yes, the ISP says “Some packet loss is normal, if you can get to web pages its working as designed”
I have started having the users call up and complain they cant stream Netflix, and that usually gets the ISP to at least look into it…
Watch the results for a while. There is a good chance that you’ll see something like:
20ms
25ms
22ms
130ms
25ms
30ms
500ms
28ms
212ms
…
Everyone has multiple phones, tablets, PCs, doorbells, etc., and they all connect via WiFi. Routers with QoS (which is most of them nowadays) will prioritize every single device on the user’s network ahead of the VPN traffic. This creates a surprising amount of latency. If the latency spikes high enough, the session often drops. Everyone has a hard-on for bandwidth, but it’s latency that kills sessions.
Connecting via ethernet often resolves the issue because users have a dozen wireless devices and none that are hardwired. Most routers will allow you to manually configure QoS to prioritize a specific device.
First I would establish if their home internet sucks. Should be able to run a constant ping to the outside world and see if there is packet loss. If not it could be the tunnel, could be your firewall or any hop in-between. I would bet it’s 99.9% of the time thiyer isp.
If you have random users through out the week, I would suggest…
Have them use an Ethernet cable - no WiFi
User need to have high Bandwidth (100+ Mbps), because consumers bandwidth fluctuates during peak hours
User ISP modem might need to be replace
If you have multiple users in a day then it’s probably…
Firewall firmware
VPN client - Cisco Anyconnect clients does well on low bandwidth
Company bandwidth is randomly fluctuating
Maybe there’s a router on your LAN or outside of your LAN that’s receiving a lot of packets that can cause your VPN to disconnect. This happened to me.
I always remind my boss “vpn is meant to be secure, not reliable.” If they want more reliability, then I offer a hardware solution that the user has at their home or on their person to travel.
Most of the time, once additional costs come into play, people start quieting down.
Big questions would be what your users are getting onto the VPN to use? If there is an issue with VPN reliability, then maybe it would be time to talk with management about updating infrastructure to be less reliant on the VPN and using always-on proxies like Zscaler, app proxies, or cloud services to host apps and data.
Had a similar issue with our Sophos UTM and Radius/Azure MFA. there was a setting that had the VPN reconnect every 4 hours. Before MFA, it just reconnected. After we enable MFA, it required the approval push. Lengthened the timeout to 12 hours for our workaholic wfh devs and haven’t had an issue since.
Check power management related stuff. Wireless and wired NICs default to allowing the os to turn the device off to save power. If they’re using a laptop on battery power the power management will be even more aggressive. Or they’re just not active enough on the machine to keep it from sleeping.
Not saying that this is necessarily your issue, but something that we tend to overlook.
Had a user complaining the other day so I sent a 10 minute ping plot showing them that they had about 20 incidents of >50% packet loss. They called Comcast and eventually got a tech to come out and found a bad splitter. “But my internet works fine, I’m watching Netflix.” Yeah, why do you think all streaming service have a buffer…
This gives me anxiety reading about. Was in your boat, in a small company with a few dozens of wfh and various issues and no possibility to see the actual cause. Because we cannot go to user’s home or trace connection through all the hops. Error log in VPN is very generic, connection lost, why? who knows? And it is very hard to explain this to users. They don’t know tech and only see your end of tech and blame what they can see and understand. We were using Fortinet VPN. Usually i was able to point to something about user’s ISP, ask them to try another, go to a room closer to their Wifi. Sometimes issues were too vague or strange. I left year before pandemic. Not sure what they did, maybe changed VPN, maybe something else. Now i don’t have to deal with VPN that much.
One user had constant disconnects every 5-10 minutes and when talking to her finally she said that her ISP came and replaced old router. On a whim i asked her to tell them to bring old router back to test. They did and it never disconnected. ISP swore configs are identical and good, but something was not playing nice in that mix of OS, drivers, VPN, wifi.
On my current job one user complained they can’t connect to Citrix while at home. Another teammate could work fine. We suggested that teammate come to her home and it didn’t work for her either. So, user talked to ISP more firmly and they said that they changed her plan to cheaper one, but that also blocked some ports, that apparently Citrix needed. After they moved her to previous plan, it started working. And we spend weeks with emails back and forth with her and her manager blaming IT and demanding to fix VPN, desktop support replacing her laptop, reinstalling apps and VPN and drivers.
We actually have used this type of “troubleshooting” a few times now. You have troubles at home and your teammate not? Invite them for a cup of tea and test
This is what I do. Make them use Ethernet connection to router. Update all drivers. Update VPN client if you have one. If that doesn’t work tell them it’s the ISP and not your fault or problem. The office internet is always working so maybe WFH isn’t for them. Or just deal with it.
Everyone bitches about the VPN. The best solution we implement was to have two. We have one included with our firewall and pulse (now ivanti). Having two VPNs solves almost all the “I can’t connect” requests, just throw users to the other VPN” it almost always works.
Sometimes you need users to add routes in there computer’s route table, if their network overlaps with yours.
If your VPN isn’t OpenVPN, I’d recommend that as a secondary. Open source and easy, and most importantly open source, so free , as in beer
