Forum Discussion

wildbailey's avatar
wildbailey
New Contributor III
4 years ago

Packet loss: Cox split the node, but it didn't help!?

I'm extremely saddened, disappointed, and heartbroken right now. We've been having extreme packet loss and latency spikes for the last 7 months. I posted about the experience four months ago here:

https://forums.cox.com/forum_home/internet_forum/f/internet-forum/28432/upstream-packet-loss-and-jitter

We weren't able to make any progress with technicians/support teams so we were about to begin the process of switching to a new provider when several neighbors jumped in, said we're not alone, and even provided us with an official date for a node split – June 30, 2021, which was a few months away. We decided to wait it out.

Yesterday morning we received notice that they were starting on the upgrade, and a few hours later they were finished. I was monitoring latency and packet loss throughout the day with PingPlotter and saw no improvement. Below is an example with 3 hours of packet loss at times peaking 25% over 5 minute intervals. 

We checked in with the neighbors, and all of them said that their internet was flawless (edit: some neighbors are now reporting that there has been no improvement).

Before the node split, we had two technicians come to the residence and confirm that everything was good from our end. They confirmed the equipment and signals were all working nominally. 

Edit: I had another technician come on June 2 and he found no issues within the residence but did see unusual latency at the tap. He didn't see packet loss (he arrived when it was relatively calm), but packet loss started shortly after he finished testing the tap and he kindly refused my request to go test again for a couple more minutes. 😅 Regardless, it's pretty likely that the latency he saw is related to the packet loss.

I'm making one last massive push to get this fixed. I will continue posting updates to this thread until we resolve the issue. I really feel like we can figure this out. Just need to find a way to get someone out here to diagnose the issue beyond the tap! 

  • This is resolved.

    Cox has informed me that it's normal for them to continue monitoring and making adjustments to the area for up to 14 days after the node split.

    The latency, packet loss, and jitter issues we experienced were remedied by increasing the number of upstream channels in the area. I'm not sure if a node split is a prerequisite to that, but from what I understand, they were planning out and working on this entire upgrade for several months at least.

    Now I have some advice for others with similar issues:

    • Don't focus on packet loss. It seems that Cox's tools for testing their network are far less eager to declare a packet "lost" than most software. If you notice that your packet loss occurs alongside spikes in latency, then emphasize the latency spikes instead.

    • Don't give up if the first technician can't find anything. Have a couple more come out until someone spots it. If possible, try to schedule techs around times that you know the problem occurs most frequently.

    • It seems that Cox does not entirely respect PingPlotter or treat its results with much weight. In fact, 2 of the 3 technicians who visited our residence gave their own unique excuse for why it's not reliable. That said, it's a great tool to identify times of day where the issue occurs most frequently for the purpose of scheduling a technician.

    • Once a technician does finally manage to see the issue, they will open an internal maintenance ticket. At this point, the issue is beyond technicians and you will need to wait (potentially several months or longer) for a resolution. 

    The most frustrating part of this entire situation has been the lack of information. Throughout all of the emails, calls, chats, and technicians, I was unable to get Cox to definitively confirm:

    • If they were aware of our issue.
    • If they were aware of a wider issue with our area.
    • That work was planned or ongoing in the area to address our issue.

    It seems that an informal FCC complaint is the "way to go" for answers like those. I didn't file one while the issue was ongoing due to a neighbor providing us with information.

    Thanks all for the help along the way!

  • wildbailey's avatar
    wildbailey
    New Contributor III

    Uploading a couple more images for documentation...

    Tech support (over email) said they're seeing timeouts on the modem. I unfortunately have no way to access such logs.

  • wildbailey's avatar
    wildbailey
    New Contributor III

    Update: Good news!

    We had an outage last night. Our connection has been completely stable since. I checked the modem, and it looks like they added 3 new upstream channels (we went from 5 to 8). Perhaps there simply wasn't enough upstream bandwidth for the area, and a node split wouldn't help with that?

    I'll continue monitoring and hopefully I can mark this as resolved in a couple of days! 🙂

    • shipracer's avatar
      shipracer
      New Contributor III

      We all hope for the magical outages, you are lucky!.

      Can you post a ping plotter of this magic?  thanks for the update

      • wildbailey's avatar
        wildbailey
        New Contributor III

        There were a few instances of packet loss throughout the day, but they were instantaneous (<1s) and without latency, plus at least one was related to a sudden spike in usage. No idea what happened at 7am, but as long as it's isolated I won't complain!

  • wildbailey's avatar
    wildbailey
    New Contributor III

    This is resolved.

    Cox has informed me that it's normal for them to continue monitoring and making adjustments to the area for up to 14 days after the node split.

    The latency, packet loss, and jitter issues we experienced were remedied by increasing the number of upstream channels in the area. I'm not sure if a node split is a prerequisite to that, but from what I understand, they were planning out and working on this entire upgrade for several months at least.

    Now I have some advice for others with similar issues:

    • Don't focus on packet loss. It seems that Cox's tools for testing their network are far less eager to declare a packet "lost" than most software. If you notice that your packet loss occurs alongside spikes in latency, then emphasize the latency spikes instead.

    • Don't give up if the first technician can't find anything. Have a couple more come out until someone spots it. If possible, try to schedule techs around times that you know the problem occurs most frequently.

    • It seems that Cox does not entirely respect PingPlotter or treat its results with much weight. In fact, 2 of the 3 technicians who visited our residence gave their own unique excuse for why it's not reliable. That said, it's a great tool to identify times of day where the issue occurs most frequently for the purpose of scheduling a technician.

    • Once a technician does finally manage to see the issue, they will open an internal maintenance ticket. At this point, the issue is beyond technicians and you will need to wait (potentially several months or longer) for a resolution. 

    The most frustrating part of this entire situation has been the lack of information. Throughout all of the emails, calls, chats, and technicians, I was unable to get Cox to definitively confirm:

    • If they were aware of our issue.
    • If they were aware of a wider issue with our area.
    • That work was planned or ongoing in the area to address our issue.

    It seems that an informal FCC complaint is the "way to go" for answers like those. I didn't file one while the issue was ongoing due to a neighbor providing us with information.

    Thanks all for the help along the way!

  • BenS1's avatar
    BenS1
    Former Moderator
    Hi @Wildbailey

    I know how frustrating it is to constant packet loss and unreliable internet. If repairs were just done less than 24 hours ago, first check all of your connections, bypass splitters, and amplifiers. And if still is not resolved, please reach out to us and schedule a technician visit.


    Ben S.
    Cox Support Forums Moderator
  • wildbailey's avatar
    wildbailey
    New Contributor III

    Update: A technician came and tested upstream from the tap for 8 minutes. By chance, he came at a time where network performance was relatively calm, except for some jitter. He saw the jitter and will report it, but refused to acknowledge any packet loss even when presented PingPlotter graphs.

    We now have some neighbors reporting that their internet service is still unstable. Some are still reporting that their service has improved drastically after the node split.

    • BenS1's avatar
      BenS1
      Former Moderator
      @Wildbailey

      Because the technician did not find any packet loss, are you able to do a Continous Ping and a traceroute and then post your results?


      Ben S.
      Cox Support Forums Moderator
      • wildbailey's avatar
        wildbailey
        New Contributor III

        Hi bens1 and thanks for all the suggestions.

        That's exactly what PingPlotter does. I have it running constant traceroutes against Google DNS, Cloudflare DNS, our modem gateway, as well as what I believe is our CMTS (the second hop for all outgoing traceroutes). Packet loss always starts with that second hop.

        Here are PingPlotter traceroute results on 6/30 between 6:34 PM and 6:35 PM. The values shown at the top are averages among all 120 traceroutes performed within that minute. (Note: 100% PL for 192.168.0.1 is normal, the Cox Panoramic Gateway always does that even when the network is performing flawlessly - proof here).

        To demonstrate that the issue is still ongoing, here is another average of 120 traceroutes from 7/2 between 6:00 PM and 6:01 PM.

        To reiterate some important previous points:

        • These results have been corroborated across multiple devices (both LAN and WiFi) repeatedly.
        • Whenever there are large spikes in packet loss, I have no issue downloading anything, it's only upload that is affected.
        • I've run tests with packetlosstest.com during some episodes and it also confirms that all packet loss is only upstream. There's a screenshot of this in my previous thread.

        Next time there's a nasty packet loss episode, I will take screenshots of ping/tracert commands for you, as well as the modem's own ping test tool.

        Thank you!

  • wildbailey's avatar
    wildbailey
    New Contributor III

    Posting some 24 hour graphs to help find correlations and to identify the best times to schedule technicians. Each graph starts at at midnight. Anything outlined in blue should be ignored as expected.

    7/1/2021

    7/2/2021

    7/3/2021

    7/4/2021

    7/5/2021

    7/6/2021 (After possible fix)