ATT Fiber Routing Issues (SE FL) - Back Office Tech Support Denies Issues

VPN
1

-----UPDATE-----

Based on what I was seeing on some other threads, where some folks have had luck purchasing a static IP block, and getting routing on the new provided subnet, I went ahead and put the order in. I waited a few days, as they wanted to send a tech out for some reason to do the setup. After reading a few posts here, I decided to call their tier 2, and get the static IPs setup sooner.

I was able to get ahold of Michael in the Czech Republic, and spend literally an hour and 50 minutes as he spent a lot of time working the issue from the beginning, in addition to getting the static IP settings applied in the router.

Once the IPs were assigned to my account, I went ahead and configured my internal home router to one of the static IPs, with the new gateway assigned. We’ve seen a dramatic drop in latency, in addition to things like Plex, and the other sites listed below working again. I don’t think it brought us back to pre-issue performance, but so far, the extra $30/month is worth it.

I’ll keep y’all updated on whether we see any other changes, or re-introduction of issues.

-----ORIGINAL POST-----

For the past week, I’ve been experience very specific routing issues from my ATT Fiber router, here in the Southeast US (Northeast Florida). I’ve spoken to 3 diff techs, and escalated to back office via those techs twice. No resolution yet. Here are three examples:

  1. https://plex.tv
  2. http://archive.ubuntu.com
  3. http://security.ubuntu.com

I can confirm an ATT routing issue by connecting to a VPN and routing with no problem.

Without VPN Example:

user@DESKTOP:~$ traceroute archive.ubuntu.com

traceroute to archive.ubuntu.com (185.125.190.82), 64 hops max

1 192.168.48.1 0.715ms 0.154ms 0.126ms

2 192.168.1.254 3.946ms 0.960ms 0.556ms

3 XXX.XXX.XXX.XXX 10.955ms 1.046ms 0.856ms

4 99.166.205.128 31.350ms 31.125ms 30.945ms

5 * * *

6 * * *

7 * * *

8 4.68.111.49 33.518ms 33.149ms 34.183ms

9 * * *

10 * * *

11 * * *

12 * * *

13 * * *

14 * * *

15 * * *

16 * * *

17 * * *

With VPN Example:

user@DESKTOP:~$ traceroute archive.ubuntu.com

traceroute to archive.ubuntu.com (185.125.190.83), 64 hops max

1 192.168.48.1 0.442ms 0.111ms 0.108ms

2 10.5.0.1 32.895ms 32.369ms 32.497ms

3 XXX.XXX.XXX.XXX 32.684ms 32.519ms 32.399ms

4 185.156.45.242 32.584ms 32.633ms 32.409ms

5 * * *

6 154.54.10.58 32.904ms 33.047ms 32.688ms

7 171.75.8.115 129.092ms 129.146ms 129.183ms

8 212.113.9.30 129.349ms 128.968ms 129.002ms

9 185.125.190.83 138.324ms 138.217ms 138.194ms

I spoke to Zharko in Technical Support (Supervisor), who has been the 3rd person that I’ve talked to. He confirmed to me that he’s talked to multiple customers with the exact same issues, with multiple different websites, over the last week. In each case when I’ve called, and for the customers that called him, there is a back office team that they talk to, that tells them they don’t see a problem, and then nothing happens. Zharko is trying to create an escalatory ticket, but needs a few more individual customers that get routed to him that he can attach to the ticket.

At this point, SOMETHING is going on with ATT routing, but in my mind, the back office team is either A) Tech team isn’t really checking correctly, or B) Deliberately ignoring the issue, or C) Purposely blocking certain sites.

In addition to various sites not working, my kids have noticed significant jump in overall latency when gaming, which would make sense with bad routing, Also they’ve had issues with certain discord voice chats not working, or poor quality. Something is definitely going on.

Anyone else seeing this??

2

You might very well just be experiencing the latency issues that many on here are complaining about. If you are, all you can do is wait for the resolution on their end.

3

Its a routing issue.

I did a whoip using the traceroute IP,s and the traffic moved from cogento, to Canonical , london, and centurylink. Looks like its outside AT&T network.

Kindly confirm , when was the last time it worked fine?

Also, the B2C customer support that you are talking to and this includes the supervisor, they dont have access to the ticketing tool for AT&T to setup a ‘HIGH’ priority ticket regarding the same. The reason that supervisor is asking for multiple customers, is because then he will collate that data and send it to his manager and they can then bring it up to AT&T on a conference call. Its with AT&T to raise a priority ticket for the issue.

4

I had a similar issue in Chicago a couple weeks ago.
I was able to get AT&T to open 2 tickets to fix the issue, but it still has not been resolved as of today.

Oddly enough, I was able to get around this issue by buying a static IP block. This is also how I proved to AT&T techs that the issue was indeed their network (6 hours on the phone with them to get them to admit this).
The last tech was able to tracert the sites I was having issues with from my gateway with both my dynamic IP and my static IP and see the latency spike as well as the fact that the pages were inaccessible on the dynamic IP address.

Fast forward to today, I still have my static IP block and I will keep them until they fix this issue.

5

You are wasting your time in tier-1 hell. The best thing you can do is call and tell the IVR that you want to cancel your service. You will go to the retention department, and they will work with you and get a tech dispatched. You should request that you want the tech to have a “helper,” which is an advanced tech. I know this because I have been dealing with this crap for six weeks - always have latency and dropped packets between 8:30 & 10:30 PM ET.
I really got a winner with the tech they dispatched. Truly wanted to fix my service and has become a friend. Very responsive, communication is great. I am sure your mileage will vary.
And then today, I get a call from him stating he would not be able to come to my house tomorrow, Saturday 8/17, because they are striking at midnight.
Please fight this fight. AT&T needs to fix their issues. They make too much money to pull this crap.
Oh yeah, if they do strike, they will probably replace workers with contractors, so expect the worst.

6

I’m also in FL and have had latency issues for some time now. I called about a month ago, was told “everything looks fine” and that’s as far as the tech would go.

I updated my BGW-500 to 6.28.7, issues remained.

I haven’t found the time to call in and push/escalate harder, but clearly I need to. I’ll give it a shot tonight. I’ve run similar trace routes and see high response times.

I considered even trying to bypass my BGW (currently in pass through to my pfsense box) but I’m on Alt-Optics GPON, and the little bit of searching I did says it’s not possible in my case. Though this issue still seems to be on the network and not my hardware.

7

If any of you have ticket / case numbers. Please SHARE THEM. I plan to call a bit later and getting all of mine I’ve called in the last 2 weeks. There are multiple I should have.

Just noticed a huge spike in High Latency for me. I have a UDM-SE and it shows me when this is typically happening. Running PingPlotter shows me MASSIVE Latency in these hops to Google DNS (8.8.8.8):

https://pasteboard.co/SRuFbLxfOvIv.png

  1    <1 ms    <1 ms    <1 ms  192.168.86.1
  2    38 ms    <1 ms    <1 ms  192.168.1.254
  3   112 ms     1 ms     1 ms  107-199-112-1.lightspeed.cicril.sbcglobal.net [107.199.112.1]
  4     2 ms     2 ms     2 ms  71.151.17.172
  5    46 ms     3 ms     3 ms  12.242.114.43
  6    65 ms   100 ms     3 ms  12.255.10.40
  7   225 ms   305 ms   353 ms  74.125.251.149
  8   268 ms   154 ms     4 ms  142.251.60.7
  9   484 ms   606 ms   206 ms  dns.google [8.8.8.8]

I can also confirm it’s a routing issue because as soon as I fire up a VPN and run another traceroute to the same IP (8.8.8.8) I get much better results:

https://pasteboard.co/I6CS3IMOaFNl.png

  1     *        *        *     Request timed out.
  2     2 ms     2 ms     2 ms  172.20.21.254
  3     3 ms     3 ms     7 ms  172.20.22.3
  4     *        *        *     Request timed out.
  5     2 ms     3 ms     2 ms  unn-149-34-240-124.datapacket.com [149.34.240.124]
  6     3 ms     2 ms     3 ms  vl202.chi-cs1-core-2.cdn77.com [138.199.0.234]
  7     3 ms     3 ms     3 ms  chi-b24-link.ip.twelve99.net [62.115.187.241]
  8     5 ms    14 ms     4 ms  google-ic-371217.ip.twelve99-cust.net [62.115.154.243]
  9     4 ms     4 ms     4 ms  209.85.142.117
 10     3 ms     3 ms     3 ms  142.251.60.1
 11     3 ms     3 ms     4 ms  dns.google [8.8.8.8]

However… you can see right from the jump in my first test (not on VPN) there is Latency right at the start at the Gateway itself… I have a feeling that this is an issue with the Broadcom / Qualcomm chipsets that are in the BGW-320’s as noted on other posts in this subreddit.

8

Allz I know when you start tinkering with the BG320 ATT techs don’t have anything for you, all they’re willing to do and tell you is that everything is good from their end to the fiber modem/router, after that whatever settings you done are equipment you added is on you

9

Wanted to ask…how’s everything going post-static IP? I called again tonight, but got told the back office/Tier 2 works weekdays and to call back tomorrow…I don’t really want to pay more but at this point my internet service is so inconsistent I just might.

10

have you tried using custom DNS if you are using at&t’s DNS? The VPN would definitely be using a different DNS.

11

Glad for confirmation on the routing issues for sure. I was out of town, so I don’t have an exact date, but based on text messages from my kids, it was around 8/1, plus or minus a day or two. EDIT: I should have added that 8/1 is when the plex.tv routing killed plex, but we’ve been seeing an increase in latency across the board since June/July.

That makes sense on the ticketing process based on what the supervisor was telling me. Assuming it is outside of their network and they don’t have direct control, they should be able to change their routes to bypass whatever is impacting correct?

EDIT: Also, I assume then that the IP-Passthrough should be non-effective?

12

Almost all of my issues happen immediately after hopping to this IP: https://who.is/whois-ip/ip-address/99.166.205.128

And a few IPs that get stuck after this IP: https://who.is/whois-ip/ip-address/32.130.25.32

Given the broad issues ATT is seeing with people across the country, I doubt it’s localized to North Palm Beach, but interesting nonetheless.

13

Might be worth the cost to at least try purchasing a static IP and see if it helps the problem. Thanks for the tip!

14

Love the idea on the cancellation threat, may give it a shot. I find it crazy that something like this is so prevalent, yet the higher ups appear to be doing nothing to address.

15

Glad to hear that I’m not going insane based upon what I’m seeing in this subreddit, but mad that so many are being impacted without ATT taking responsibility.

I may try the IP passthrough route tonight to see if it makes a difference. Several threads on here that that has helped others, so at this point will to try anything.

16

So far, so good. It’s not to pre-issue days, but the kids stopped complaining, and the services that weren’t working, are working again, so for now will keep running with it.

17

Yeah, I don’t think it’s a DNS issue. The IPs are resolving quickly either way, just routing issues once resolved and attempting to connect. I’m no expert for sure, but it doesn’t appear to be a DNS issue to me.

18

No matter what you do: IP Passthrough, DNS change (Which is actually not possible on AT&T gateways), etc…) It wont change.

However, you can try disabling IPv6 and save and then reboot the AT&T gateway. I have seen improvement at latency level.

You will find IPv6, in at&t gateway settings via 192.168.1.254

‘Home network’ tab > ‘IPv6’ tab > Select to OFF and save.

Reboot AT&T gateway to check for latency.

19

Both IP’s belong to devices which exist in AT&T network. As stated by me, its a routing issue and can only be fixed by AT&T. AT&T network team could make changes to the routing table.

20

I have yet to see the issue, however BGP works in interesting ways, so I may just never see the issue. Also, I use my own custom DNS, not ATTs, so if it’s a DNS related issue I won’t see it.