Here’s what’s making life miserable for us right now:
If our users are connected to VPN, and there is a sudden loss of power (power flickers, user trips over power cord, etc.), upon restart of their PCs, they will get to the Windows 10 login screen, be able to enter their password, but well NEVER advance past the spinning “Welcome” screen.
Our Setup:
Affected OS: Windows 10 Pro (1903 - 1909)
Affected PCs: Dell Optiplex 7060 & 7070s (micro)
VPN Server: OpenVPN Access Server (version 2.6.1)
VPN Client: OpenVPN Connect 2.6.0.100
Clients access private subnets using: Routing
Split-Tunneling is disabled - all traffic must traverse VPN
After the PCs are in this semi-bricked state, users cannot login as themselves, and I cannot login with either a domain admin or local admin account. The “Welcome” screen will simply sit there and spin forever.
If we reboot in safemode (with or without networking), we are able to login using the cached credentials without a problem. However, I’ve not discovered a way to remedy the issue and have the PC function when booted up normally.
Right now, our work around is to have users simply boot into “Advanced Repair” and Reset their PCs (electing to keep the files). This is pure insanity! Have any of you run across this?
We saw a similar phenomenon with the same OpenVPN connection and Windows 10 Pro (version 180x) last year. The fix was to apply this Local GPO:
Computer Configuration > Administrative Templates > Network > DNS Client > “Turn off smart multi-homed name resolution” = ENABLED.
This does not appear to be a fix with the very same power-cord-kicking issue in Win10 190x.
Any suggestions from the hive-mind? We’re losing it!
Plus run the following:
Powershell as Administrator DISM /Online /Cleanup-Image /RestoreHealth
You said this:
The “Welcome” screen will simply sit there and spin forever.
What is “Forever” - You should actually time how long it is trying to login. I would say let it actually sit at this screen for 30 minutes and see if it ever does anything.
Is this VPN connected pre-login, or after the user logs in under normal circumstances?
Being stuck at the “Welcome” screen - To me - implies it is attempting to contact the Domain Controller during logon, but you mentioned this also occurs with Local Admin accounts? What do the Event Logs show when you boot into safe mode with networking?
After you use Advanced Repair and reset their computers - Can the issue be recreated?
Have you got verbose login/logoff and start/shutdown display messages set up via GPO? There might be something that it’s hanging on that you can’t see because it’s showing the “welcome” message instead of something more useful.
I’m willing to bet the farm that this is a DNS issue. Tweaking that multi-homed DNS setting fixed this issue in previous versions of Win10.
Under normal VPN circumstances, the VPN connection will direct the PC to use the DNS servers in our data centers for all name resolution magic. Since our connection is interrupted instantaneously by the power being pulled, would it be a fair to assume that Windows is chocked-up by DNS servers specified by the VPN connection, and thus is trying to use these same DNS servers upon reboot?
When we see the ‘globe’ icon (and we know our internet connection/source is good), we can almost always safely assume that there is a DNS issue somewhere.
I’m seeing that same globe icon at the login page.
Does anyone know some commands that could be run to get the system to release/ignore those DNS servers specified in the VPN connection?
I’ve noticed similar behavior with OpenVPN in a similar config (full-tunnel). Was able to resolve it by upgrading both the server and the client. The 3.1 beta client is both more resilient to power/network loss situations and easier for users. The catch - they’ll have to rebuild their connection profiles.
It is actually a dns issue. I have been experiencing similiar problem with our OpenVPN.
The trick to fix it is to boot in safe mode and find Dnspolicyconfig in regedit. It hosts openvpndnspolicy which SHOULD clear itself upon exiting the vpn software but when crashes happen it stays.
I have no idea why it bricks the PC fully but upon removing the faulty items from regedit everything starts working normally and you only need to reinstall the vpn client.
On mobile so I dont have the regedit path to give you but search in hklm computer system currentcontrolset and something, I can post it tomorrow from work.
Is the VPN always on or do they have to start it after they’re logged in?
Is this isolated to certain users?
Do you have folder redirection setup for desktop and documents?
This sounds like a GPO setting that is set to apply before presenting the user with a login screen. I forget the name of the actual settings, but a start would be to move the computer to a test OU and block all inheritance, reset GPO to default settings manually then see if the problem goes away. I’d really be suspect of GPO settings because this is not normal behavior, even if users lose internet connection altogether they should be able to authenticate with cached credentials.
Look into this. HERE Perhaps it is not even an issue with OpenVPN. I mean once you are on the VPN I assume it can not hit its authentication source. However if you set the devices to remember x number of logins they would be cached.
Are you able to ping the device from the lan when it’s in this state (make sure fw rules allow pings from the local subnet before trying)?
What about logging in via rdp?
Also try powershell remoteing as that might let you better enumerate what the network state is and what’s actually going on.
I don’t known if there’s anything like it for windows but some sort of terminal based packet capture (like tcpdump for linux) also might be handy to see what actually happening network wise when you try to log in.
Much appreciated! OpenVPN.net replied to my ticket stating that this is a known issue with client 2.6.1.
They also recommended we bump up to v2.7.1, so that is what we’ll do.
I bench tested v2.7.1 and can report that this does address the issue.
The longest I ever let v2.6.1 sit at the “welcome” screen was 2 hours. Even if it resolved itself at 2 hours and 1 minute, I’d never convince users to wait that long, plus the business would find that unacceptable.
I did some lab testing on v2.7.1 this morning and find that upon restoring power, the “Welcome” screen spins for almost exactly 1 minute and 52 seconds every time. That’s much better!
DevinSysAdmin,
I’ll try your recommendations in the lab environment. Thanks!
To answer your other questions:
Forever = at least 2 hours. Users report that 4 minutes = forever…
VPN does not connect pre-login, users must connect post-login.
The issue can be recreated 100% reliably after the Advanced Repair/Reset
Thank you everyone for your suggestions and help. I suspect I’ll never get to the bottom of what was happening here, but if it can be fixed by just pushing out an updated version of the VPN client (and scheduling updates to the production VPN Server), then that is good enough for me!
Sadly, it appears that this isn’t providing useful information.
I applied the GPO and did a few clean reboots to confirm the verbose messages were showing. When it was time to yank the cord and test “user style”, I’m only getting the “Welcome” spinny message. Nothing useful displayed…
A ticket was raised last year when we had a spout of power-cord-kicking going on in one of our remote offices. The normally helpful folks at OpenVPN.net did not have a solution. After days of research and trail and error, I found that applying the GPO I mentioned seemed to prevent this from being an issue.
Tried it. No luck.
An interesting observation: At the user sign-in screen, windows displays that ‘globe’ icon for the network indicator. If device is WiFi connected, Windows will report “Connected, secured, no internet”.