Four Years of Packet Loss

VPN
21

Yeah, I’m still working on this. First two months were spent diligently setting up meetings and troubleshooting with support, and then… life happened. Had to stop working on it for a handful of weeks. But, I’m getting back into it now.

I moved to a new apartment, around 10 minutes from my old place. I have xfinity at the new place. After setting up my PC, I noticed I was still experiencing packet loss. Ran the tests here… exact same results that I was getting at my old place. Same problem IPs on pingplotter.

I talked about this with a friend who lives in a neighboring state. They mentioned that they both have xfinity & has been experiencing some pretty similar lag for a while now. They ran the tests… same results. Same problem IPs.

Talked with a few more friends scattered across the state. Asked them to run it. Those that replied reported the exact same results.

Pretty much confirms that this isn’t an issue with either of us. It’s a problem with a small handful of servers- probably just a couple bad NICs or something.

I’m certain that everyone connected to these problem servers- which could potentially be all xfinity customers within a 2 state radius- is experiencing packet loss as well. And likely has been for at least… five years now?

That seems like a pretty big deal.

My modmail support ticket timer ran out, so I’ll be making a second post here in a couple days that has more details.

22

Not much progress, unfortunately. PC melted & I won’t be able to spare the cash for a new CPU for a good while. I’ve yet to formally report the faulty servers bc all of my logs are on that SSD. I only have a list of identified IPs on-hand.

This said, I’d heavily encourage pressing onwards. I’m genuinely beginning to suspect that a majority of Comcast customers in the midwest have been suffering with this packet loss for… god knows how long.

If you’d like, I can dig up the IPs I’ve found & DM them to you. I’ve got logs from at least 4 separate people somewhere in my Google sheets I can pass over.

I figure that the louder people are about this, the more likely it is they’ll do something about it.

Edit: I’ve heard a couple people say that using a VPN to connect to specifically servers on the west coast helps. Haven’t been able to verify myself, but figured it’d be worth mentioning

23

Sounds super helpful. Gonna look into this later, cheers

24

I’m fairly certain it isn’t tied to individual events. Behavior hasn’t changed for the full four years.

I can’t rule out fixed environmental factors. We’ve relocated the modem & shifted the cables around a handful of times, and it’s a pretty straight shot from here to the pole, but it’s definitely still possible that they’re the source.

25

This issue has not yet been solved.

26

Still not solved. Just taking me a long time to get my next appointment scheduled.

27

I’m on a Comcast 800Mbps connection. Just to give you some comparison numbers:

Your latency is better than mine. I show 96ms.

Your jitter is worse than mine. I show 2ms.

Your packet loss does look pretty severe. It’s notable that all of your loss is upload loss. I would think that would be a clue for a tech, but I’m not really a signals guy.

My initial thought would have been try a different modem, but I see you’ve tried that. I am not familiar with the user interface to Comcast modems. Since your packet loss is upstream, I would be looking at the “upstream bonded channels” section. A little googling tells me the power level you see depends on how many bonded channels you have.

While you’re checking things, you can also check the downstream channels just to be safe. SNR should be 30db or greater.

Quote:
For some networks, the upstream power limits for 3 to 4 channels are 35 to 51 dBmV. Ideal levels are approximately 40 to 50 dBmV for single channels, 37 to 48 dBmV each for 2 to 4 channels.
End quote.

Again for comparison, I have 4 upstream bonded channels, all showing approximately 49dBmV.

Because you have the same results with Windscribe active, I think that conclusively rules out theories 1 and 2. I also think this points to a problem with either your modem or your (local) cabling.

My guess is that you need a really good tech to look at this. I hope Comcast sends you one.

Notes from Comcast on how to access the modem (to see signal levels):

​With a computer, phone, or tablet that’s connected to your network, open a web browser and go to the Admin Tool at ​​http://10.0.0.1​​. If you’ve already changed your Admin Tool ID and password, log in with your information.​

​Pro tip: ​​This username and password aren’t the same as your WiFi name and password.​

​If you haven’t changed the username and password, log in with: ​

​ ​
​Username: ​​admin​
​ ​
​Password: ​​password​

​If you changed your ID or password but don't remember it, you can do a​​ factory reset (See Factory Reset Note below)​​.​
​ ​
28

Have you validated these results by using a different application like the command prompt?

The traceroute didn’t shown anything wrong. Every hop doesn’t respond to ICMP requests.

29

There’s an old email online that I’ve read is no longer active. I can understand why. Snail mail may be your best bet. Their corporate office is in Philly.

30

We have been keeping things open for you, u/Coyotezzz. Please feel free to send us a Modmail message whenever you have an opportunity, and we’d be more than happy to continue looking into this.

31

That all makes sense, I am not sure what Xfinity’s approach is to addressing packet loss. I would imagine that they have visibility of it, but maybe due to costs, they won’t address it until customers speak up. Speed tests may produce great results, and the average Joe is typically satisfied with that. Gamers, for example, will notice packet loss as it impacts their gameplay experience and some games (e.g. Fornite) have the option to enable network diagnostics where packet loss is clearly indicated when present.

The packet loss that my son is experiencing in gaming is intermittent and yesterday, it was really bad so after testing with NordVPN, we determined that no VPN or a NordVPN connection to NY produced similar packet loss results. However, a NordVPN connection to VA or GA produced 0% packet loss consistently after at least a dozen tests with this site: https://speed.cloudflare.com/

32

Go ahead, I am curious to see if they are the same ones. I am in the midwest too.

33

This pretty much sums up my experience with xfinity. Switching to AT&T fiber.

34

Just some dumb questions: When you installed Windscribe, did you initiate a VPN connection through the Windscribe GUI? Try multiple different VPN endpoints? Run packetlosstest.com with the VPN active?

BTW, the packetlosstest.com numbers I shared above for my connection are all with a Windscribe VPN active because I run a full-time VPN on my router so every device in the house goes through the VPN.

35

https://imgur.com/a/vU94fME

Does this help?

36

Better than nothing. Thank you!

37

Funny enough, I just reconducted all tests this morning, and all connections, non-VPN and VPN, report 0% packet loss. So, I suspect that if something is faulty within Xfinity’s backbone to certain regions (e.g NY), it only may surface during times of heavy network utilization. This would make it tough to find but I suspect that somewhere, in a data center, there’s a faulty switch, load balancer, network cable, something…that is causing packet loss during high network utilization in the NY metro area.

38

Fumbling around with channel checking right now. Here’s what I found for upstream:

https://imgur.com/a/Dz6xEyF

I’m gonna test all of this out with a 3rd party modem (rather than one provided by Comcast) as soon as I can.

Windscribe stuff: Through the GUI. Multiple different endpoints returned the same results on packetlosstest & pingplotter.

Looking like this is probably gonna end up being a cable issue.

Side note, I just stumbled across something strange. I ran bitmeter for a little while and found that every upload packet loss spike seems to correspond with a spike in amount downloaded.

https://imgur.com/a/P1ISbJa

39

Like I said, I’m not a signals guy, but it feels like you’re homing in on something.

Packet upload loss corresponds with download data? Crosstalk issue? Don’t know what I’m talking about here!

In your picture of your modem’s signal levels, the values for index 5 look all kinds of wrong to me. Unless you have one of the newer Comcast connections with high speed uploads, I’m not sure you should even have an index 5 active. I only have 4 upload channels active, and all 4 of mine are much the same values as your first four.

I would think both of these points are potentially important to bring to the attention of a tech.

40

I’m fairly certain we recently upgraded to the “latest and greatest” package xfinity has to offer. That said, the power level on 5 does look incredibly strange. I’ll check our plan here in a minute.