Four Years of Packet Loss

VPN
1

#Four Years of Packet Loss.

#1/30/24: JUMP TO BOTTOM OF POST

I have been experiencing non-stop packet loss for over four years. It’s caused hours and hours of headache. I’ve tried every fix that I can think of. It’s become unbearable, and I’ve had enough of it. I’d like to finally get to the bottom of this issue and solve it.

I’ve found that the majority of our packet loss occurs at jumps to specific Comcast IP addresses. All of our connections jump to at least one of these problem IPs at some point. It has not been seen to subside once, in all 4 years that I’ve been tracking it.

#Describing the Issue

  • We are experiencing constant packet loss.

  • Packet loss comes in spikes.

  • Spikes happen approximately once every 8-10 seconds, never skipping a beat.

  • Spikes last approximately 0.3-1 seconds.

  • During these spikes, packet loss at the identified problem jumps rises as high as 90%.

  • Occasionally, there are periods where we experience an extreme increase in these spikes.

  • During this time, spikes last 1-3 seconds long and often show total packet loss.

  • Periods of intensity last anywhere from 10 minutes to 24 hours. Their average duration seems to be somewhere around an hour. Shorter periods often come in chains- 30 minutes of intense loss, 5 of normal, repeat.

  • Periods of intensity have been gradually growing in frequency and length over the course of the past few years. There has been a rapid increase over the course of the past month.

  • Things that require moderate to complete packet transfers are unreliable and unusable during this time. Most of the important things our household does involve video streaming/screensharing, voice calls, and online gaming. One of the people here works from home, and these issues have begun to interfere with their work.

  • It’s been practically unusable for anything requiring real-time uploads for the past two weeks.

#Attempted Solutions - Tech Support Checklist

#####Are you using wifi?

  • This happens on both ethernet and wifi.

  • …so, plugging into the source & relocating our router are right out.

#####Does this happen on multiple devices?

  • This happens on multiple different devices. Windows PCs, Android phones & tablets, Smart TVs, Playstation 4s, and even a Chromebook.

  • This answer invalidates the following “potential fixes” (SEE BOTTOM OF POST):

  • Restart device.

  • Disable firewalls.

  • Update OS.

  • Update drivers.

  • Replace faulty hardware.

  • Look for potential security issues- viruses.

  • Factory reset device.

#####Does this happen to everything, or just streaming, chatting, and gaming?

  • This happens while performing any activity. It’s just more noticeable when doing things that are reliant on all of your packets going through.

  • This happens on every site that I have tested.

#####Reduce the amount that you are currently downloading/uploading.

  • This happens even while we are not downloading/uploading anything.

  • …on that single connected device. After a cycle/reset.

  • Excessive downloading and uploading does not intensify the packet loss.

#####Does this happen at certain times of the day?

  • This happens consistently, regardless of time of day, week, and year.

  • Periods of intensity also do not follow any set pattern.

  • So, it’s safe to assume that it’s not due to ISP network overload, bottlenecks, overloading our modem/router, ect.

#####Disconnect all of your devices and check again.

  • This happens even while only one device is connected… after both power cycling our router and resetting our gateway.

  • We’ve done this hundreds of times.

#####Reset your modem/router.

  • This continues to happen after resetting (power cycling) the router.

  • We’ve done this hundreds- perhaps thousands- of times.

#####Reset your gateway.

  • This continues to happen after gateway resets.

  • We’ve done this dozens of times.

  • It’s the “solution” that the tech line usually settles for. It does not work.

#####Swap your DNS servers.

  • We can’t change what DNS servers we connect to. Your provided hardware does not permit it. We haven’t been able to check to see if this happens on a VPN because of this. However…

  • This has persisted between (at minimum) 4 DNS changes that you have initiated to attempt to solve the problem.

#####Disable your modem/router’s built in firewalls.

  • Tried this. Sadly didn’t do anything.

#####Replace your modem/router.

  • This has persisted between three separate Xfinity routers.

  • Router swaps have not changed issues.

#####Replace all relevant cables.

  • This has happened before and after replacing the following cables:

  • Gateway cable, modem to the outside of the house.

  • Gateway cable, outside of the house to the line.

  • Modem/Router power cable.

  • The Modem/Router has been tested on multiple different circuits.

#Recent Customer Support Interactions

I’ve recently connected with your base T1 customer support- once through your online service, and once on the phone. These connections took about an hour and a half each, and achieved nothing. They completely ignored all information I attempted to provide, bashing their head against the problem assuming I was referring to intermittent periods of total service loss. To be quite honest, it felt belittling. After 90 minutes of running around, the person on the line claimed they were going to “escalate my issue to their dedicated support team.”

The first attempt at “escalation” was through the online chat service. After troubleshooting, the individual I was talking with said that they said their team would contact me via phone within 30-40 minutes. I waited for two hours before I gave up and went to sleep. I did not receive a call that I could identify as being Comcast while asleep, or during the course of the next day.

The second attempt was through your phone line. The woman I spoke to said that, after around an hour, I would receive a text from their line asking if the issue was resolved. If the problem wasn’t fixed, I was to respond with “NO”, and a T2 tech would call within a few minutes. I never received this text.

So, I’ve decided to take to Reddit.

#Source Theories & Proposed Solutions

Looking at this list of failed solution attempts, I cannot see how this issue could be caused by factors within my control. I have a few theories on what could cause this, and proposed solutions that could either solve the problem or narrow down its source.

First theory: I believe there may be an ISP-side firewall incorrectly flagging & blocking filtering chunks of the packets we send. In the few similar situations I’ve found online, this was often found to be the issue. I believe that disabling these firewalls will solve this issue, as the packets that are being incorrectly blocked will be allowed through. I’m not certain why this theoretical firewall error would cause the packet loss to come primarily at 10 second intervals, but I couldn’t care less if disabling it solves our issue.

Second theory: I believe there may be some sort of issue with the DNS servers that we are connecting to. I’m not really qualified to speculate on this one. The solution I could try out is to connect us to a non-comcast DNS. Seeing as the provided hardware does not support this relatively basic feature, this will require us to buy a new 3rd party modem/router. If this solves the issue… it’ll sting a bit. To know that 4 years worth of package upgrades and attempted repairs were for nothing, and that we’ve been receiving sub-par service due to faults in your service/hardware would be highly upsetting.

Third theory: I believe the cable running from the outside of our home to the pole is damaged, and was not properly repaired the last time a tech came to have a look at it. If this cable is found to be undamaged, perhaps it is the cable connecting our street to the network?

Fourth theory: I believe there may be an issue with your servers themselves? It’s… the only place I think that the issue could be, at this point.

Fifth theory: There’s something that I have missed, likely due to a lack of technical knowledge.

I’m tired. My patience is waning. I hope that this post reaches the eyes of someone who knows how to fix this.

EDIT: Learned more about DNS servers. Tried a VPN, no change.

Theory 1 is out. Theory 2 is out.

Found something interesting in bitmeter. The packet loss spikes seem to directly line up with tiny download spikes.

CMD prompt tracert: https://imgur.com/a/m7eTBTB

Pingplotter & packetlosstest results: https://imgur.com/a/tCIu9rX

Gif comparing CMD prompt “ping -n 100 1.1.1.1” to pingplotter results: https://imgur.com/a/vU94fME

Bitmeter graph: https://imgur.com/a/P1ISbJa

Upstream channels: https://imgur.com/a/Dz6xEyF

EDIT 2: I’ve got a tech coming soon to look at the cables. I’ll edit this post after the tech appointment. Might have to make a new one.

EDIT 3: Problem is not resolved after tech visit.

Alright, we’ve done a bunch of troubleshooting.

Had a tech come. He was stellar. Polite, patient, efficient, inquisitive, knowledgeable. Worked with me every step of the way to troubleshoot this. He went the extra mile running additional post- tests to double check for flaws/ensure the evidence was airtight. That man deserves a raise.

I gave him a quick rundown as he started setting up to replace the coax.

It’s worth noting that the tech observed the issue, in person, multiple times.

We removed the current coax & all old cables around the area. Then, the tech ran a fresh coax all the way from the pole to the gateway downstairs. The line runs far away from every bit of potential electrical interference I can think of.

No luck.

So, he replaced the port connecting our coax to the main line.

No luck.

He ran a series of tests connecting to multiple different sites, at both the box & the fresh port. All of them showed absolutely no packet loss.

But, we still witnessed packet loss from inside the house. Our next assumption was that there was something damaged our outdated in our gateway’s firmware… and that the issue had persisted between our gate replacements 'cause we kept swapping our old XB7s for new XB7s. So, we set up to replace the gateway and the tech submitted his report. Gateway was replaced today.

Still no luck.

While waiting for the opportunity to swap the gate, I discovered another interesting potential puzzle piece. When connecting to certain sites, our target IP will occasionally change. These target swaps line up with changes in packet loss intensity frequently enough for me to want to mark it down here. Might be nothing, but more information is better than less.

Example: https://imgur.com/a/PuPGhzc

#####The most important thing I’ve found thus far:

This thread: ‎Mysterious & Frustrating Intermittent Packet Loss & Disconnects | Xfinity Community Forum

It’s one of the only things I can find anywhere online that resembles my issue. These things seem to match up with one of the previous theories. I currently think I’m dealing with cable tilt.

I’ve got another tech coming out in a couple of days. I don’t really know what they’re going to do, but I’ll welcome anything that resembles progress with open arms.

Edit 4: three months later, I’ve moved places. Getting the same issue at my new apartment. A friend a state away is also experiencing identical drops at IPs that match mine. It’s a server issue.

Edit 5: Another few months. I’ve found that a bunch of friends across the US are also having this issue. Still collecting as much data as I can, think I’m close to a fix.

Edit 6: Here’s to five years! I have created a spreadsheet that contains links to 80+ posts on the xfi subreddit/xfi support forum, each showing test results that match the ones described in this post. Several large clusters of these posts reference the same set of IPs. A significant percentage of Xfi servers across the US seem to have sustained some kind of damage- leaning towards faulty on-site PONs- that is causing them to drop packets in a way similar/identical to what I’ve described. Xfi tests show up “green” either because packet loss isn’t hitting the 5% this-is-a-problem threshold or because the tests are being routed through private/optimized/specific servers (and thus never see the damaged ones).

If you’ve found this post while searching for a fix, know that you are not alone.

This is not happening because of something you’ve done, some piece of damaged or outdated equipment you have, some hidden firewall you forgot to turn off. It’s happening because several of the servers you are being routed through are damaged.

My advice? Full 3rd party hardware and a strong VPN. Try to dodge as many problem IPs as you can. Either that, or find a way to get Xfi to repair every single damaged server you’re being routed through.

Questions, comments, concerns, evidence, arguments? Comment. Post. Tag me somewhere. If I can help, I will.

2

Thank you for joining us here on r/Comcast_Xfinity, your official source on Reddit for help with Xfinity services. As members of the Digital Care Team here at Xfinity, we can help with a wide array of concerns including troubleshooting, billing, plan changes, and more.

If you have not already, please review both the Posting Guidelines and Rules here on the sub. While you’re waiting for assistance, check out the Xfinity App for your smartphone where you can pay your bill, view your plan details, change or upgrade your services, and experience 24/7 real time support you can count on, anytime you need it.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

3

I don’t know how to fix this, but I have some feedback.

First off all, how do you know your problems are due to dropped packets? Have you actually tested for this, or are you just assuming it based on observed behavior of applications?

Try this:

Theories 1 and 2 don’t make sense to me. If it were an ISP firewall, it would be affecting a lot of people. And DNS issues just don’t behave the way you are describing (a DNS record is looked up once and then retained for a while before it’s looked up again, and a connection that’s established - like a video call - shouldn’t be doing any DNS lookups).

Theory 3 does make sense to me. It would have to be a cable that’s only for you, not your block’s connection, or your neighbors would be having problems - have you checked with them?

I would think a good test WOULD be to connect through a VPN. For some reason, you think you can’t do this. All you need to do is install some free VPN software on an affected PC. I recommend trying Windscribe’s free tier. I’ve used Windscribe for years. Comcast cannot stop you from trying a VPN. You do not need to change your DNS servers to use a VPN - the VPN tunnel will take care of that for you.

4

Good evening u/Coyotezzz, and thank you for reaching out to our Reddit Team tonight. We are sorry to hear about your longstanding connection issues, this is never the experience we want for any of our valuable customers and understand your frustrations. We appreciate you sharing the detailed troubleshooting steps you’ve performed in the past. Our team would like to look further into your concerns and help find a way to get this resolved. Can you please send us a Modmail Message with your first and last name along with the service address on the account?

5

Do you have alternate options? Any fiber providers in your area? I finally had to switch due to high latency and ping spikes, and now my connection is smooth, faster, and it was cheaper. Fiber for $30-40 less per month…

6

I had this problem years ago and it was infuriating and so I feel your pain! My issue was a bad cable that was buried underground. In the end, Comcast came and replaced the line and all problems went away. I struggle with 1, 2 and 4.

One easy way to troubleshoot this is schedule a ping to run at some frequency and capture the output. That will tell you if your Internet connection drops. The Ping test wbesite works, but it is short term, you want something that runs for a longer time and captures the output. PRTG is a good idea, but something simpler like this would work.

7

Do you have any traceroutes?

8

Awesome post! I feel your frustration and we are not alone. I believe the only way to get this resolved is to send this post to Brian Roberts, Comcast chairman and CEO. You tried everything else. I’ve had success in the past on Reddit with Comcast Technical Support. Recently, no success. Same with Comcast forums. AI responses have not helped. Best of luck! Please keep us updated.

9

FWIW, I am currently experiencing 10-12% packet loss through Xfinity’s network but noticed that I’d experience 0% packet loss when connecting to a VPN outside of the NY area. I suspect that some sort of ISP backbone routing issue or malfunctioning switch or network gear is causing the issue, but it seems to only affect connections to the NY area from NJ (where I’m based out of). I tested VPN connections through VA and GA and had 0% packet loss when conducting a test to NY here: .https://speed.cloudflare.com/

I’ve set my VPN to autoconnect and plan to wait it out to see if it resolves itself. Otherwise, I’ll have to contact support.

10

I too am having the same exact issues and have done everything you have listed off besides having a tech come out, which is scheduled. Did they ever solve the issue with the server? I am using pingplotter and anytime I have packet loss its always on a Comcast hop.

11

I have this same issue. It’s kind of frustrating.

12

Honestly I didn’t read the entire thing but when I ultimately had to go to war with Comcast in a similar situation I spun up PRTG in a datacenter to monitor my gateway latency, packet loss, and uptime via my public IP. The aforementioned gives you handy graphs and eliminates the variable of everything within the network.

I was able to finally convince Comcast and they fixed equipment somewhere down the line which corrected it.

13

What area are you in? Like depending on the area, there may be random fiber cuts due to really annoying road work.

Or, are there any electronics near your modem or line? Like the story of a company network going down every day at the beginning and end of the day only to eventually find out that the line was run through the elevator shaft area and not properly shielded so, a giant metal container going up and down constantly caused major issues.

This can happen with almost any kind of line depending on how the run is made.

I get it though… I’m switching to fiber in my area since I can’t get xfi complete in my area and get more than 10mbps up and I really need better for my work. And I also get some stupid packet loss. Sometimes, it’s just down and reports nothing down in my area on the app but, I go online and find an outage not even a mile away.

When it gets really bad, I clear out my ipconfig and flush dns and then renew. Doesn’t work if there’s an outage that’s not reported. I’ve been busy at work and just had to go to my hot spot on my phone because the internet just goes out for a couple of hours…

14

This post was marked as solved. Should you experience further issues, please create a new post

15

This post was marked as closed. Should you experience further issues, please create a new post.

16

First off, I really, truly do appreciate the feedback. It’s refreshing to hear from a person who knows more about this than I do. Honestly, I’m just coasting off of googling, reading forums, and watching youtube. I’m shocked that it’s gotten me as far as it has.

Thanks for the brief explanation on how DNS stuff works. Makes a lot more sense now. There was something to it that just wasn’t sticking in my head before.

Data has been acquired via pingplotter and packetlosstest.com. In hindsight, I probably should’ve included screenshots in the main post. I’ll attach a miscellaneous handful to the bottom of this comment, and a few more specific ones tomorrow.

Just downloaded Windscribe and did a couple tests. Same results as before.

Theory 1 hinges off the idea that there’s something that’s causing only our packets to get flagged. I have no idea what that something would be, I’ve been holding onto a sliver of hope that it exists. Maybe there’s a magic switch Comcast can flip that’ll make it all better.

Theory 2 is indeed nonsense. Lack of understanding on my part.

Theory 3 is- frustratingly- the most likely of the bunch. Still, I find it baffling that the issue would persist through multiple cable swaps. Perhaps a tech mucked up a past repair/replace job that would’ve fixed the issue?

Neighbors haven’t reported any issues that I know of. I’ll try to get in contact with one tomorrow, ask if I can connect to their router to check.

https://imgur.com/a/tCIu9rX

17

Haven’t done much research on it as of right now. I’ll probably be looking into it if I can’t get this fixed

18

Just tried a VPN. Same results.

Comcast servers. Theory 4 is mostly longshot-spitballing. I’m waaay out of my depth here.

This happens on both ethernet and wifi. This happens on multiple different devices. Specifically computers: 3x Windows PCs, 2x Windows laptops.

Once I get some free time & cash, I’ll give the UPS stuff a go.

Power is something I’ve looked at, but haven’t gone too deep into. I’ll try running the router on battery later, see if that solves it.

I’ll do another hunt for something that could be interfering with it in the morning. Shouldn’t take long. Since the problem has existed for four years, there’s a lot I can rule out. I’d rather not rip up my floorboards and tear out my appliances, but if that’s what it takes…

19

https://imgur.com/a/m7eTBTB

https://imgur.com/a/tCIu9rX

20

Yeesh. All the way to the top, huh? I’ll give it a shot. Would you happen to know a good way of getting this to him? A number I can call, user I can DM, or email I can send to, or address I can write to?