#Four Years of Packet Loss.
#1/30/24: JUMP TO BOTTOM OF POST
I have been experiencing non-stop packet loss for over four years. It’s caused hours and hours of headache. I’ve tried every fix that I can think of. It’s become unbearable, and I’ve had enough of it. I’d like to finally get to the bottom of this issue and solve it.
I’ve found that the majority of our packet loss occurs at jumps to specific Comcast IP addresses. All of our connections jump to at least one of these problem IPs at some point. It has not been seen to subside once, in all 4 years that I’ve been tracking it.
#Describing the Issue
-
We are experiencing constant packet loss.
-
Packet loss comes in spikes.
-
Spikes happen approximately once every 8-10 seconds, never skipping a beat.
-
Spikes last approximately 0.3-1 seconds.
-
During these spikes, packet loss at the identified problem jumps rises as high as 90%.
-
Occasionally, there are periods where we experience an extreme increase in these spikes.
-
During this time, spikes last 1-3 seconds long and often show total packet loss.
-
Periods of intensity last anywhere from 10 minutes to 24 hours. Their average duration seems to be somewhere around an hour. Shorter periods often come in chains- 30 minutes of intense loss, 5 of normal, repeat.
-
Periods of intensity have been gradually growing in frequency and length over the course of the past few years. There has been a rapid increase over the course of the past month.
-
Things that require moderate to complete packet transfers are unreliable and unusable during this time. Most of the important things our household does involve video streaming/screensharing, voice calls, and online gaming. One of the people here works from home, and these issues have begun to interfere with their work.
-
It’s been practically unusable for anything requiring real-time uploads for the past two weeks.
#Attempted Solutions - Tech Support Checklist
#####Are you using wifi?
-
This happens on both ethernet and wifi.
-
…so, plugging into the source & relocating our router are right out.
#####Does this happen on multiple devices?
-
This happens on multiple different devices. Windows PCs, Android phones & tablets, Smart TVs, Playstation 4s, and even a Chromebook.
-
This answer invalidates the following “potential fixes” (SEE BOTTOM OF POST):
-
Restart device.
-
Disable firewalls.
-
Update OS.
-
Update drivers.
-
Replace faulty hardware.
-
Look for potential security issues- viruses.
-
Factory reset device.
#####Does this happen to everything, or just streaming, chatting, and gaming?
-
This happens while performing any activity. It’s just more noticeable when doing things that are reliant on all of your packets going through.
-
This happens on every site that I have tested.
#####Reduce the amount that you are currently downloading/uploading.
-
This happens even while we are not downloading/uploading anything.
-
…on that single connected device. After a cycle/reset.
-
Excessive downloading and uploading does not intensify the packet loss.
#####Does this happen at certain times of the day?
-
This happens consistently, regardless of time of day, week, and year.
-
Periods of intensity also do not follow any set pattern.
-
So, it’s safe to assume that it’s not due to ISP network overload, bottlenecks, overloading our modem/router, ect.
#####Disconnect all of your devices and check again.
-
This happens even while only one device is connected… after both power cycling our router and resetting our gateway.
-
We’ve done this hundreds of times.
#####Reset your modem/router.
-
This continues to happen after resetting (power cycling) the router.
-
We’ve done this hundreds- perhaps thousands- of times.
#####Reset your gateway.
-
This continues to happen after gateway resets.
-
We’ve done this dozens of times.
-
It’s the “solution” that the tech line usually settles for. It does not work.
#####Swap your DNS servers.
-
We can’t change what DNS servers we connect to. Your provided hardware does not permit it. We haven’t been able to check to see if this happens on a VPN because of this. However…
-
This has persisted between (at minimum) 4 DNS changes that you have initiated to attempt to solve the problem.
#####Disable your modem/router’s built in firewalls.
- Tried this. Sadly didn’t do anything.
#####Replace your modem/router.
-
This has persisted between three separate Xfinity routers.
-
Router swaps have not changed issues.
#####Replace all relevant cables.
-
This has happened before and after replacing the following cables:
-
Gateway cable, modem to the outside of the house.
-
Gateway cable, outside of the house to the line.
-
Modem/Router power cable.
-
The Modem/Router has been tested on multiple different circuits.
#Recent Customer Support Interactions
I’ve recently connected with your base T1 customer support- once through your online service, and once on the phone. These connections took about an hour and a half each, and achieved nothing. They completely ignored all information I attempted to provide, bashing their head against the problem assuming I was referring to intermittent periods of total service loss. To be quite honest, it felt belittling. After 90 minutes of running around, the person on the line claimed they were going to “escalate my issue to their dedicated support team.”
The first attempt at “escalation” was through the online chat service. After troubleshooting, the individual I was talking with said that they said their team would contact me via phone within 30-40 minutes. I waited for two hours before I gave up and went to sleep. I did not receive a call that I could identify as being Comcast while asleep, or during the course of the next day.
The second attempt was through your phone line. The woman I spoke to said that, after around an hour, I would receive a text from their line asking if the issue was resolved. If the problem wasn’t fixed, I was to respond with “NO”, and a T2 tech would call within a few minutes. I never received this text.
So, I’ve decided to take to Reddit.
#Source Theories & Proposed Solutions
Looking at this list of failed solution attempts, I cannot see how this issue could be caused by factors within my control. I have a few theories on what could cause this, and proposed solutions that could either solve the problem or narrow down its source.
First theory: I believe there may be an ISP-side firewall incorrectly flagging & blocking filtering chunks of the packets we send. In the few similar situations I’ve found online, this was often found to be the issue. I believe that disabling these firewalls will solve this issue, as the packets that are being incorrectly blocked will be allowed through. I’m not certain why this theoretical firewall error would cause the packet loss to come primarily at 10 second intervals, but I couldn’t care less if disabling it solves our issue.
Second theory: I believe there may be some sort of issue with the DNS servers that we are connecting to. I’m not really qualified to speculate on this one. The solution I could try out is to connect us to a non-comcast DNS. Seeing as the provided hardware does not support this relatively basic feature, this will require us to buy a new 3rd party modem/router. If this solves the issue… it’ll sting a bit. To know that 4 years worth of package upgrades and attempted repairs were for nothing, and that we’ve been receiving sub-par service due to faults in your service/hardware would be highly upsetting.
Third theory: I believe the cable running from the outside of our home to the pole is damaged, and was not properly repaired the last time a tech came to have a look at it. If this cable is found to be undamaged, perhaps it is the cable connecting our street to the network?
Fourth theory: I believe there may be an issue with your servers themselves? It’s… the only place I think that the issue could be, at this point.
Fifth theory: There’s something that I have missed, likely due to a lack of technical knowledge.
I’m tired. My patience is waning. I hope that this post reaches the eyes of someone who knows how to fix this.
EDIT: Learned more about DNS servers. Tried a VPN, no change.
Theory 1 is out. Theory 2 is out.
Found something interesting in bitmeter. The packet loss spikes seem to directly line up with tiny download spikes.
CMD prompt tracert: https://imgur.com/a/m7eTBTB
Pingplotter & packetlosstest results: https://imgur.com/a/tCIu9rX
Gif comparing CMD prompt “ping -n 100 1.1.1.1” to pingplotter results: https://imgur.com/a/vU94fME
Bitmeter graph: https://imgur.com/a/P1ISbJa
Upstream channels: https://imgur.com/a/Dz6xEyF
EDIT 2: I’ve got a tech coming soon to look at the cables. I’ll edit this post after the tech appointment. Might have to make a new one.
EDIT 3: Problem is not resolved after tech visit.
Alright, we’ve done a bunch of troubleshooting.
Had a tech come. He was stellar. Polite, patient, efficient, inquisitive, knowledgeable. Worked with me every step of the way to troubleshoot this. He went the extra mile running additional post- tests to double check for flaws/ensure the evidence was airtight. That man deserves a raise.
I gave him a quick rundown as he started setting up to replace the coax.
It’s worth noting that the tech observed the issue, in person, multiple times.
We removed the current coax & all old cables around the area. Then, the tech ran a fresh coax all the way from the pole to the gateway downstairs. The line runs far away from every bit of potential electrical interference I can think of.
No luck.
So, he replaced the port connecting our coax to the main line.
No luck.
He ran a series of tests connecting to multiple different sites, at both the box & the fresh port. All of them showed absolutely no packet loss.
But, we still witnessed packet loss from inside the house. Our next assumption was that there was something damaged our outdated in our gateway’s firmware… and that the issue had persisted between our gate replacements 'cause we kept swapping our old XB7s for new XB7s. So, we set up to replace the gateway and the tech submitted his report. Gateway was replaced today.
Still no luck.
While waiting for the opportunity to swap the gate, I discovered another interesting potential puzzle piece. When connecting to certain sites, our target IP will occasionally change. These target swaps line up with changes in packet loss intensity frequently enough for me to want to mark it down here. Might be nothing, but more information is better than less.
Example: https://imgur.com/a/PuPGhzc
#####The most important thing I’ve found thus far:
This thread: Mysterious & Frustrating Intermittent Packet Loss & Disconnects | Xfinity Community Forum
It’s one of the only things I can find anywhere online that resembles my issue. These things seem to match up with one of the previous theories. I currently think I’m dealing with cable tilt.
I’ve got another tech coming out in a couple of days. I don’t really know what they’re going to do, but I’ll welcome anything that resembles progress with open arms.
Edit 4: three months later, I’ve moved places. Getting the same issue at my new apartment. A friend a state away is also experiencing identical drops at IPs that match mine. It’s a server issue.
Edit 5: Another few months. I’ve found that a bunch of friends across the US are also having this issue. Still collecting as much data as I can, think I’m close to a fix.
Edit 6: Here’s to five years! I have created a spreadsheet that contains links to 80+ posts on the xfi subreddit/xfi support forum, each showing test results that match the ones described in this post. Several large clusters of these posts reference the same set of IPs. A significant percentage of Xfi servers across the US seem to have sustained some kind of damage- leaning towards faulty on-site PONs- that is causing them to drop packets in a way similar/identical to what I’ve described. Xfi tests show up “green” either because packet loss isn’t hitting the 5% this-is-a-problem threshold or because the tests are being routed through private/optimized/specific servers (and thus never see the damaged ones).
If you’ve found this post while searching for a fix, know that you are not alone.
This is not happening because of something you’ve done, some piece of damaged or outdated equipment you have, some hidden firewall you forgot to turn off. It’s happening because several of the servers you are being routed through are damaged.
My advice? Full 3rd party hardware and a strong VPN. Try to dodge as many problem IPs as you can. Either that, or find a way to get Xfi to repair every single damaged server you’re being routed through.
Questions, comments, concerns, evidence, arguments? Comment. Post. Tag me somewhere. If I can help, I will.