Time Warner Cable Responds To World Of Warcraft Disconnections

Jeff Simmermon, the Digital Communications Director for Time Warner Cable, has responded to the charges that TWC is responsible for the lags and disconnections plaguing East Coast World of Warcraft players. He took a look at the traceroutes posted on Blizzard’s user forums and sent the response.

I’m the director of digital communications at Time Warner Cable, and I’d like to bring some clarity to this discussion. We’re happy to to take our lumps when we’ve earned them, but it doesn’t seem to be the case in this instance.

Take a look at some of the traceroutes posted to the thread in question … starting here, at comment #446: http://tinyurl.com/5gqe27

If you follow the commenter’s posted trace results, you’ll notice that it’s only on TWC’s Roadrunner (rr) network for the first 6 hops — with maximum response times of 10 ms. The response time jumps drastically at hop # 11 — when the trace is no longer on the Roadrunner network.

Scroll down further on the same page to comment #456, and you’ll see something similar — a giant leap in lag times. However, this trace never touches our network. It starts at Verizon, goes to Alter.net at hop #5, and then jumps to ATT.net’s network at hop #8. Hop #9 shows a response time of 114 ms — quite a jump from the 49ms at hop #8.

On the first page of the thread, you’ll see something similar: http://tinyurl.com/3hfs9k

At comment #10, the lag time leaps from 18ms on our network at hop #6 to 150ms at hop #7 — on Level3, an Internet backbone.

At comment #18 (same page), the trace again never touches our network. The lag jumps from 15 ms at hop #3 to 261 ms at hop #4, while on the Verizon network. The hops vacillate between high and low response times throughout the trace.

Blizzard’s comment at the top of the thread that “Unfortunately this means that the only commonality between all the players experiencing these disconnects and extreme latency is Time Warner/Road Runner” is a pretty interesting choice of words, in light of the fact that several of the troublesome traceroutes posted in the forum itself never touch our network.

Jeff Simmermon
Director, Digital Communications
Time Warner Cable

So, is Blizzard just trying to pass the buck to the best available scapegoat? If both companies aren’t responsible for the lags and disconnects, whos is? Does anyone know how this crazy Internet works?

(Photo: ashley_dryden)

Comments

Edit Your Comment

  1. backbroken says:

    This would never have happened if we had elected Al Gore.

  2. chiieddy says:

    Sounds like there might be a bad router in the routing tables. RR COULD adjust their tables to remove the bad router.

  3. Sorshha says:

    Time Warner Cable doesnt care about WoW people

  4. HPCommando says:

    Anyone else having ATT connection issues outside of iPhone problems?

    My tethering stopped being able to connect for my laptop, but the PDA still sees the Internet. Likewise, my backup dial-up account was also unable to connect.

    Reason is, since Monday, any of my customers with ATT/SBC/PacBell/Yahoo/Prodigy accounts or connections have also been reporting connectivity issues, not counting any iPhone issues.

    Calls to tech support net a “not our equipment” from ATT, which is the stock answer whether true or not.

  5. Snowblind says:

    Well, that is the problem with TCP/IP, it guarantees delivery, not performance.

    Eventually, the packet will get there…

    It may very well be that neither TW or Blizzard’s provider is in control of the offending router. In fact, there may be no direct connection between the two networks, leaving them at the mercy of the 3rd party. In this case it is Level 3, the backbone provider.

  6. They should hit their routers with a Sword of Swift Transfer.

  7. EYELAZERZ says:

    MOAR TUBES!

  8. I heard it was just a series of tubes. Better get some draino.

  9. Lithium542 says:

    Well, I’d say the problem is with Time Warner’s backbone, level 3. Level 3 provides fiberoptic & wireless based bandwidth pipes to ISP’s, and some of their routing looks screwy. Methinks Time Warner needs to step up, and give their service provider a boot to the groin.

  10. Dobernala says:

    Time Warner certainly has a choice in who their upstream providers are, so they aren’t completely void of fault.

  11. Canino says:

    I bet Geek Squad can fix it.

  12. pmathews says:

    @backbroken:

    Just give him a pair of tights and a mask and he’ll fix this problem just like he fixed the environment…Are there Nobel Prizes for internet saving?

  13. mannyv says:

    Well, this probably isn’t the best forum to discuss the traceroute results, but TWC may be right. It also may be that there’s some kind of config problem on all the TWC edge routers. It’s strange that there’s that much latency right after the TWC edge, especially if it happens across multiple peers…so a common config may be the culprit.

    Maybe they’re doing QoS at the edge and WoW stuff is on the losing end of that?

  14. eliblack says:

    Blizzard is known for passing the buck as fast as possible. When they were suffering massive lag and usability problems a couple years ago, they tried to pass it off as a backbone problem, and finally admitted that it was a problem with their servers. It was fixed, eventually, but they were very quick to defer blame.

  15. MyPetFly says:

    “This would never have happened if we had elected Al Gore.”

    We did.

  16. Mfalconieri says:

    Does anyone know how this crazy Internet works?

    Of course…..ask Ted Stevens. I am sure you can get a straight answer from him about how the internet works.

  17. magic8ball says:

    See, the actual problem here is that the internet is not a truck, it’s a series of tubes. And right now the tubes are clogged with enormous amounts of material.

  18. kaptainkk says:

    @Canino: “I bet Geek Squad can fix it.”

    I don’t know why but I busted out laughing when I read your comment…LOL!

    @MyPetFly:

    “This would never have happened if we had elected Al Gore.”

    “We did.”

    This is so true.

  19. hubris says:

    @magic8ball: Ted Steven’s e-mail, mostly.

  20. OnceWasCool says:

    Blizzard has turned into a money grubbing horrible company since Sierra bought them.

    money money money

  21. @Lithium542: Umm…what? L3 is a different company than TWC/RR. I’m not sure what you expect TWC to do about it other than to notify L3 about the latency experienced through some of their routers. L3 would most likly respond two TWC along the lines of, “Your customer’s satisfaction is important to us…”

  22. Bladefist says:

    TW response sounds reasonable. If one of the routers between the users and blizzard is blocking it, it’s likely a whole region would be blocked as well.

    Time warner takes you to the internet. Once you get there, you bounce around on tons of different routers until you get to the ISP of the destination. It’s hard to explain in non-technical terms.

    All I can say is, a long time ago, I was blocked from Yahoo Mail, and it was neither my ISP nor Yahoo. Some idiot in the middle had a badly configured router.

  23. Norcross says:

    @Dobernala: unfortunately, it’s not as easy for them to change backbones as it is for us to change cell phone companies or ISPs (I can’t imagine what their ETF cost would be).The fact that TCW laid out the details of the latency issues shows that (a) they researched it, and(b) identified the problem.

    HOWEVER, since this proves that nothing is completely network-specific (that they all share backbone), then maybe they can all stop pretending that we pay out the nose for service because they own the lines?

  24. Crymson_77 says:

    Sounds like ATT’s hub is having issues………again…….

  25. Bladefist says:

    Have someone in your family who lives somewhere else host a vpn server for you, connect to that, and pipe your WoW through the VPN. That’ll work :)

  26. kaptainkk says:

    Sorry one more: I hear the all the time when lag times between hops are high. “The latency is astronomical”

    Who is to blame here? Well with networks you can always pass the buck to the ISP, the routers and switches along the path, the backbone provider, the end servers, etc. Who knows what the real problem is? Maybe just too many WoW heads trying to battle it out at the same time?

  27. OmniZero says:

    I think that Time Warner might be right for once. However I bet it’s something they can talk to those providers about instead of sitting on their butts. They might have more influence on Level 3 than us regular consumers.

  28. Skellbasher says:

    A few years back I experienced a similar problem on the a section of the Adelphia network that ended up with Time Warner.

    There’s multiple problems with all this, and none will help find the root cause.

    First, the traceroute utility in Windows runs the trace using ICMP. ICMP traffic is rate limited across just about every network now, so a Windows trace may not give you a clear picture. Your ‘live’ data may get somewhere in 40ms, but your Windows traceroute could show 100ms if some rate limits were being hit.

    Secondly, the Blizzard tech support folks in their forums are not technically proficient enough to understand the underlying network issues that could cause something like this. They don’t have those people on staff because they don’t really need them. All the WoW realms are co-located in AT&T cages, they don’t need a lot of network experience to set that up.

    Finally, you have a classic situation where and end user blames their ISP, the ISP blames the content provider, and the root problem is on a carrier network somewhere in the middle. Until one of them mans up and has someone competant look seriously at the problem, nothing will change.

    In my case, the problem was a sketchy interface on an AT&T router that most of my area ended up passing through. I was only able to find that out because of my professional contacts. Blizzard support was no help, nor was Adelphia. I was actually threatened by Adelphia’s Director of network security at the time, because he thought I was contacting providers and pretending to be employed by them.

    It’s a crappy situation, unfortunately there’s no easy resolutions to it.

  29. Sorshha says:

    @Canino:
    you forgot the “‘t” after can

  30. Teh1337Pirat3 says:

    that is pretty lol, I’m glad they atleast investigated the issue and owned those WoW fags in the face

  31. wcnghj says:

    Sounds like TWC is putting in a delay or something between 10 and 11. I have no problems on my local DSL provider.

    [upitall.googlepages.com]

  32. ibored says:

    looking at that thread…it does seem like all of them occur at the edge of TWC. My guess is that to try and stretch bandwidth to the nth degree, TWC is slashing QoS. I bet money this isn’t going on with just WOW they just have a big enough community to make noise.

    I think Ted Stevens should have to clean the inter-tubes as parrt of whatever punishment he gets for being a corrupt bastard.

  33. headhot says:

    Trace Route is fucking meaningless. It uses ICMP packets not UDP. With TW’s deep packet inspection, it could be passing ICMP unmollested (as screwing with them would be a dead give away) but hosing UDP.

    Games use UDP, so if thier DPI is throttling UDP and not ICMP, traceroute would show nothing.

  34. categorically says:

    I blame the Russians and their cyber war.

  35. tedyc03 says:

    My best guess is that TWC is lying. They say they can’t pick up the phone and call other networks and say “hey your lag time is unacceptable”? Bullshit.

  36. admiral_stabbin says:

    Ha! My diabolical plan to destroy the “World of Warcraft” is coming to fruition…thank you, Time Warner for your help.

    OK, seriously though…while I somewhat agree with the spirit of the TWC Directors message (that it may not be entirely their fault), I don’t agree with the message…as it’s still their problem. Further, the packets do touch their network…the packets don’t magically jump from hop #1 to hop #6.

    I can only begin to guess what the root cause of the problem is, but, coming across like you’re passing the buck will only put fuel to the fire…

  37. BlairLizard says:

    “If you follow the commenter’s posted trace results, you’ll notice
    that it’s only on TWC’s Roadrunner (rr) network for the first 6 hops —
    with maximum response times of 10 ms. The response time jumps
    drastically at hop # 11 — when the trace is no longer on the Roadrunner
    network.”

    And if he’d followed that line of thought to hope #12, he’d see the
    ping times drop back down, which means traffic going THROUGH hop #11 is
    fine, but the ping time goes up going to 4.68.16.133 (hop#11)

    “At comment #18 (same page), the trace again never touches our network.
    The lag jumps from 15 ms at hop #3 to 261 ms at hop #4, while on the
    Verizon network. The hops vacillate between high and low response times
    throughout the trace.”

    It jumps up again on one hop and then drops back down; which again means
    traffic going through hop #4 (and #7 and #10) is not latent, just
    traffic destined to those hops

    I’m not sure this guy understands traceroute and I wouldn’t put much
    weight in his troubleshooting of network issues

  38. CrazyMann says:

    Can someone explain to me why it took the Digital Communications Director of TWC to come up with an answer to the problem? Each person that called into TWC should have reached a tech that should have been able to see and provide the same information. Causing a lot less frustration to TWC customers.

  39. headhot says:

    Also,
    TW edge routers decide what back bones to use to get various places. If they are choosing poor routes, even though the slow down is outside of TW’s network, its still their fault. TW needs to call their peer up and find out what the problem is instead of shrugging and passing the buck.

  40. Everyone’s speaking a language I do not understand.

    /quietly leaves thread

  41. Consumerist-Moderator-Roz says:

    @OnceWasCool: What does that have to do with this article? Please keep your comments on topic.

  42. Canino says:

    After looking at some notes from my tech support days, I can definately say the problem is kinked cables.

    You see, your cables have to be straight or smoothly curved, not kinked. If you have a kinked cable the zeros can get through ok, but the ones get stuck and cause a log jam.

    All these WoW users should make double sure they have no kinked cables. If they find a kink, they should straighten it and shake it a little to get the ones moving again. It might take a while for it to all clear out, but the speed will return as soon as the backed up bits have gone through.

    You’re welcome.

  43. Lithium542 says:

    @valarmorghulis

    I’m guessing you’ve never worked for an ISP.

    Just like you have a choice in ISP’s, they have a choice in bandwidth providers. Global Crossing, Covad, Cogent, etc. are all viable alternatives. If Level 3 is giving them poor service, then they need to move. If they have a flawed routing table, or poorly functioning network hardware in specific locations, other companies that use Level 3 for bandwidth would theoretically complain too.

    When I used to work with Cogent, they’d occasionally forget to pay their routing/DNS bills, and you’d only get half the internet….

  44. xkaluv says:

    Wow, MR. Director. You really know tact… way to respond with excellent PR skills! Wahoo, YOU SUCK!

  45. quagmire0 says:

    In other news, due to the rediculous lag times, the WoW geek walked outside, saw his shadow, and promptly hid. 6 more weeks of summer!

  46. JeffSimmermonTWC says:

    @CrazyMann — I can field that one, having recently sat in for a shift at one of our call centers in Queens. Our CSRs process a tremendous number of calls every day, and don’t have the time to do the research that I did.

    If you — or anyone else — has any questions, feel free to e-mail me at jeff.simmermon@twcable.com

  47. ShadowFalls says:

    @eliblack:

    Every company is quick to defer blame, or do you not see what TWC is doing here? When you call up support, it always everyone else’s fault but theirs first.

    Those issues with latency are really not horribly bad, not enough to cause these issues.

    I want to point out one thing though. You can have lag and disconnects even with a good ping. This can have anything to do with bad equipment to traffic filtering.

  48. swedub says:

    It’s probably the NSA’s fault. They have to inspect every packet, WoW or not, for “terrorism” of course. Skellbasher’s comment above said “WoW realms are co-located in AT&T cages”. AT&T has allegedly provided the NSA with a room specifically for them them to inspect every packet through their network.

    [en.wikipedia.org] <– Hepting v. AT&T (lawsuit filed by the EFF)

    Maybe terrorists are masking their communications through WoW?

  49. warf0x0r says:

    Shouldn’t TWC be able to change its routing tables to skip those hops?

    I work in software and I would do exactly what blizzard said. They don’t have anything to do with how you connect to their game, they just make their servers availible. It’s the responsibility of the ISP to make sure they’re customers can get to locations on the if they are selling internet access (unless they specifically tell you, “No, we don’t let you go there”).

    Ultimately this means that someone has to contact the company with bad hops, and TWC doesn’t want to do it, Blizz doesn’t want to do it, so its left in the hands of the consumer, which quite frankly is F***ing BS.

  50. Dustbunny says:

    @InfiniTrent:

    Me too. But look — there’s a cute kitty in the pic! I can play wif kitteh instead of playing WoW?

  51. BeastMD says:

    Filtering specific ports could be the cause, but the reponse times on the TW network look fine. The problem probably lies in the equipment at the end of the TW “network” but so there is a 50/50 shot its their problem.

  52. Also I would love to know what addons these guys are using cause im sure they are tx and rx’ing a lot of data in the background using up upload bandwidth.

  53. JeffMc says:

    I think we’re missing the point here, guys.

    It doesn’t matter to the user who’s fault it is, the fact of the matter is they’re paying TWC a monthly fee for internet access and it’s reasonable to assume that with this access people can play World of Warcraft.

    So Time Warner needs to stop passing the buck and talk to whoever is having the problem and get it resolved. Their customers don’t care who’s fault it is, they care about who they’re paying $50 a month to.

  54. TorrentFreak says:

    In other words, Time Warner’s response is eat shit.

  55. BrandonW says:

    Have they tried turning if off and then on again?

    God, I hope someone gets this, although I may be at the wrong Gawker site for that…

  56. chrisdag says:

    @Bladefist: uhhh. so you want to push an *already* latency sensitive application through a random VPN enpoint? Seriously?

  57. admiral_stabbin says:

    @JeffSimmermonTWC: I assume you have a TWC provided cable modem at home. Have you personally installed WoW (yep, it’s even Mac compatible if you need it to be!) and given it a go? If not, I’d encourage you to do so.

    It’s a large download, and, it can be addictive, but, I’ll help you get started. I’m P1mPMaster5000 on the Nathrezeim Realm.

  58. chrisdag says:

    @JeffMc: If you read the fine print for residential internet access agreements you will find that your main assumption: “… it’s reasonable to assume that with this access people can play World of Warcraft” has no basis in fact.

    TimeWarner has no such obligation to ensure that you can play WoW, especially for a case like this where latency seems to be the main gripe.

    Seriously read the fine print. These companies have lawyers write up these things so that they are on the hook for basically nothing — no stated performance goals, latency goals, uptime or even time-to-fix.

    The alternative is to upgrade to a business connection. You pay more money but you get a service level agreement with explicit promises regarding speed, uptime and time-to-fix. There are also penalties defined for when the SLA goals are not met. The final benefit is that you’ll be able to open a support ticket for problems like this and a single person will be responsible for making things right.

    For me the big payoff was back in the DSL days – with a business DSL circuit it did not matter if the problem was with the copper wire (owned by Verizon), with the DSLAM in the CO (owned by Covad Communications) or with the ATM uplink to the internet (owned by my ISP) — a residential customer would end up in multi-vendor fingerpointing hell while a business customer would have a single ticket open with the ISP that would resolve the issue regardless of what company actually was causing the problem.

    You get what you pay for when it comes to broadband and you don’t get much in the way of promises with the subsidized and oversubscribed residential offerings.

  59. Karl says:

    At comment #10, the lag time leaps from 18ms on our network at hop #6 to 150ms at hop #7 – on Level3, an Internet backbone.

    Sounds like TWC’c connection to Level3 is congested, and when that happens, packets are dropped. What packets are dropped is usually a policy decision, and they COULD be dropping packets from long-standing (e.g. WoW client-to-server) flows.

    Also, traceroutes are helpful, but don’t tell you the whole story. For example, the burst of lag at hop 11 in the first traceroute? That could have happened ANYWHERE up to the router listed. Or, that router could have simply been slow on sending back replies. Since the latency after that router drops significantly, those measurements don’t show the true latency to that router, and should be taken with a grain of salt.

    Traceroutes also generally don’t take into account any traffic shaping policies. It’s very possible for traceroute packets to get a much higher priority than TCP packets using WoW’s ports. It’s possible that TWC is confusing these TCP flows for P2P traffic.

  60. TomCruisesTesticles says:

    Haha a lot of the WoW geeks I know have been talking about this. Maybe we’ll actually be able to do something this weekend besides hang out in _ _ _ _ _’s room watching him

  61. Quilt says:

    …I didn’t understand any of that.

  62. TomCruisesTesticles says:

    @TomCruisesTesticles: All I can say is pWned!1

  63. stopNgoBeau says:

    @Consumerist-Moderator-Roz: Much like the comments about Al Gore being elected? If you’re going to call out one person, please do it to all.

  64. jstonemo says:

    @backbroken: True, since he DID invent the internet (and global warming)!

  65. Routing is scary stuff.. I’m on the Roadrunner system here in Tampa, but I never had any WoW issues (before I cancelled) even though we fall under the Bright House name these days. I would suspect that the geographical routing might be an issue but maybe I lucked out with my choice of server? I started out on Skywall (PVE) and ended on The Forgotten Coast (PVP) and it was pretty smooth sailing.

  66. xthexlanternx says:

    @Bladefist:

    I’m one of the people who has slow Internet from Time Warner. It continues for months on end and I actually stopped playing online games for long periods of time because of it. I called them pretty much nightly for months at a time complaining over and over again, getting the same bullshit responses every time (lets not forget about the waiting 30+ minutes every night on hold as well). I am very good with computers and networks and I have fixed many a problems in my time. I can say with 100% certainly that all MY problems were with Time Warner. I hit bad Time Warner hops every single night for months on end. It would go away for a week or two and come right back. They were not someone else’s fault, they were Time Warner hops that were slowing everything down. They refuse to fix anything and just repeatedly blame it on your problem or someone else’s. They have phone techs who know about as much about computers as my 85 year old grandmother. Time Warner is pretty much the sole reason I do not play online games anymore. Oh, and its the only (reliable) broadband in my town too.

  67. jstonemo says:

    Two words:

    Summoner Geeks!

  68. linoth says:

    When you have a pairing between an ISP and another carrier, the destination carrier isn’t the only one responsible for maintenance on the pairing.

  69. JustThatGuy3 says:

    @Norcross:
    They do own the lines that matter. Backbone is cheap to build, cheap to run, it’s the local network that’s costly. How many miles of Interstate highway in the US? How many miles of local streets?

  70. Since when does Level3 have poor ping times? If I get the option, they’re one of the top Tier 1′s I choose for routing. Much better than CogentCo atleast…

    I’m curious to know what happens at this point as it seems to happen commonly:

    20 * * * Request timed out.

    21 * * * Request timed out.

    22 * * * Request timed out.

    23 * * * Request timed out.

    24 * * * Request timed out.

    25 * * * Request timed out.

    26 * * * Request timed out.

    27 * * * Request timed out.

    28 * * * Request timed out.

    29 * * * Request timed out.

    30 * * * Request timed out.

  71. AndyAndy719 says:

    It seems due to Comcast’s traffic shaping of BitTorrent that it’s now the automatic assumption whenever theres a network issue.

    I would agree with many commenters that ICMP doesnt give the whole picture. Filtering ICMP on a core router level isnt common, as it ads to CPU. (if the router has to scan every packet to make sure to route, then that taxes the router.) Its certainly not deep-packet inspection either, as ICMP is on the internet layer (same as with IP – which TCP/UDP traffic rides on top of).

    Time Warner cant route around the blockage because routes are announced based on destination usually from the destination (or destination’s ISP). This is done via BGP (Border Gateway Protocol), and annoucing parties have a ASN (Autonomous System Number). The annoucements for IP routes are done in blocks of IP addresses. For example, blizzard’s IP block (according to ARIN – american registry of internet numbers):

    Blizzard Entertainment, Inc. ATTENS-010058-005011 (NET-206-17-110-0-1)
    206.17.110.0 – 206.17.111.255

    Which is a smaller block of:

    CERFnet CERFNET-BLK-206 (NET-206-16-0-0-1)
    206.16.0.0 – 206.19.255.255

    CERFnet is AT&T. (prolly acquired)

    Blizzard is only using AT&T (AT&T is the only announced route). Fixed orbit (which shows which ASN is announcing IPs):

    [www.fixedorbit.com]

    Whereas the Consumerist (hosted at Datagram) has their IP announced by Datagram:

    [www.fixedorbit.com]

    But if you click the ASN # for datagram, you get all the peering ASNs.

    AT&T has a lot of peering – but Time Warner has no direct route to AT&T, and the border router @ time warner prolly determined that level 3 is the best route to get to Blizzard (as evidenced by the traceroute), which then peers to AT&T, and then to Blizzard.

    Blaming time warner for this is probably way premature. It could be a Level 3 — AT&T peering issue, and theres too many middlemen. The easy fix would be for Blizzard to get another carrier (redundancy/diversity anyone?) or for time warner to get an AT&T link/peer. Diagnosing a network issue between carriers is not easy/instant, but I’m sure with all the attention they’ve got some significant resources working on it.

  72. Of course nobody knows how the Internet works. Your request goes into the magic cloud and arrives at its destination.

    Duh.

  73. AndyAndy719 says:

    @Papa Midnight: The point where the traceroute commonly ends:

    13 12-122-254-14.attens.net (12.122.254.14) 11.029 ms 9.874 ms 9.751 ms
    14 mdf001c7613r0003-gig-10-1.bos1.attens.net (12.130.0.170) 9.797 ms 9.630 ms 9.897 ms
    15 * * *

    Is probably where AT&T hands off to Blizzard, and Blizzard probably filters ICMP inside their network.

  74. rick.h says:

    @Karl: Ping times can also jump because for a lot of routers, priority is given to forwarding packets. When you do a traceroute, packets are sent directly to the router, and it has to respond to them. Responding to packets is the router’s last priority. So if it happens to be under heavy load, or doing a maintenance task or such, you’ll see a single jump in response times like this.

    However, a consistent high ping time on that router could also indicate that that router is not properly spec’d for the job, and it could be dropping packets.

    Another thing that most people don’t understand is the use of switching gear in backbone networks. In a 10 hop traceroute, it shows the 10 IP addressable routers that your traffic is passing through. What it doesn’t show is more than likely the 20+ carrier switches that your traffic is passing through, again any one of which could be causing issues.

    @Papa Midnight: You’re picking peering bandwidth for someone and you don’t know what “Request timed out” on a traceroute means? I hope I don’t ever send any packets across any of YOUR networks.

  75. ModernDemagogue says:

    I’ve had similar problems with streaming video. TWC gave me the following explanations after showing them traceroute tables with the same error as the Level3 one above.

    1) I was told that TWC does not prioritize traceroute’s and ping’s, so my results were not reliable indicators of what was actually going on. While this seems believable, its also an explicit admission of packet shaping or some form of non-neutral behavior.

    2) They also said since the problem was happening once the data got off their network, its unfortunately not their problem; when I called “bs” and responded by pointing out that I am not paying for “access to their network” but in fact paying for internet access, the dialogue began to make some progress.

    3) When I then informed them that I work in post, and frequently download HD files of 2gb or greater and asked if this could this be causing a problem? TWC said they don’t packet shape, but wait one sec, and that I should try my connection again now. It was instantly back to full speed, as though some setting was changed. Use of certain applications and sites continues to appear to trigger a slowdown.

    My theory that I have yet to test completely, is that rather than “shaping” traffic by type per se, TWC has different routing methods for different customer classifications, ie PowerUsers such as WoW players, Movie Downloaders, etc… are placed into a different class and share a different routing path which ultimately limits throughput. My most direct evidence for this is that I’ve piggybacked on my neighbors TWC internet, and found absolutely no problems and completely different routing paths. Also, WoW play and streaming media from YouTube or Hulu would appear similar to a traffic-type-blind throttling system.

    TWC has recently launched its “PowerBoost” service to allow greater speeds when downloading files. I feel like this might be related.

    This response from the director of digital communications is complete crap. If there is that much packet loss going off their Tier-2 Backbone to Level-3 a true Tier 1 Network (defined by not having settlement charges), it is very much TWC’s fault.

    A quick search of Wiki turns up that TWC in fact does not even purchase/settle with Level-3, they purchase upstream from Sprint Nextel Corporation/as1239. If this is true, why is traffic even being routed to Level-3?

    All of this seems suspicious; and I can’t wait for FiOS to hit the rest of Manhattan.

  76. ModernDemagogue says:

    Site’s I’ve experienced the same type of timeout on traceroutes and pings;

    Facebook
    Yahoo
    CNN
    Google
    Hulu

    I’m not convinced this is localized to Blizzard’s connectivity, I’m much more inclined its either a connectivity, configuration, or intentional routing issue on TWC’s end…

  77. ModernDemagogue says:

    And yeah, YouTube, as well as my company’s FTP server, physically located within 10 blocks on a full T3.

  78. @stopNgoBeau: Much like the comments about Al Gore being elected?

    Well, he invented the Internet, so clearly an Al Gore presidency would have fixed all of the Internet’s problems. He’d’ve even made sure that “reply all” gets changed to “reply to sender” when you’re about to embarrass yourself.

  79. Breach says:

    The traffic is probably hitting a crappy router out there somewhere in the web, that may be impossible to trace.

  80. beavis88 says:

    @headhot:

    This.

  81. bwcbwc says:

    Maybe Sprint and other backbone carriers are giving preferential treatment to their own customers over Time-Warner’s? That would explain the discrepancy in the statements.

  82. AndyAndy719 says:

    @headhot: Traceroute actually uses UDP, and waits for the ICMP replies.

    [www.freesoft.org]

  83. headhot says:

    @Breach:

    The web is an protocol that runs over the internet. The web is not the internet.

    And a crappy router is traceable. Its that whole “route” part of the router that tells you where it is.

  84. stopNgoBeau says:

    @Michael Belisle: Yeah, and the Blizzard comment that got the commenter’s hand slapped was pertaining to the WoW being part of the problem as mentioned by many other folks.

  85. purplesun says:

    @Teh1337Pirat3: Can we get a mod on that offensive comment? There really needs to be a report-a-comment feature.

    On topic – this situation has been going on for a very long time. While I can appreciate the person from Time Warner responding, I personally do not find that very satisfactory from a customer stand point. At the very least, you’d think Time Warner would at least attempt to contact the 3rd parties involved and resolve the situation, since it is impacting the opinion of their customers – a negative opinion that’s spread like wildfire.

  86. Bryan Price says:

    Sounds to me like their peering sucks. Since this happens just outside RR’s network, they know who the problem is, and they know how to talk to them to fix the problem.

    On a side note, I’ve noticed that Comcast (for my area) is no longer using AT&T for their backbone, they actually have their own backbone setup. One of my sites fell over while I was uploading something, and it recovered (it was supposedly working all the time according to my buddy, the owner, but once things were working, it was obvious the server fell over. Comcast got pretty far before they went off their network, which is encouraging for them.

    And yes, I’m a geek, a networking geed, and damn proud of it.

  87. @AndyGoodwin: Well, I know since it’s inception, Blizzard has used AT&T for their network solutions. Since Battle.net went live with Diablo, they’ve always used AT&T, though something seems to be amiss here. It may not be Time Warner to blame… Entirely. Though, judging from the post made by their representative, they seem to be looking for an excuse to defer blame from themselves.

    I reside in Maryland right now and use a Comcast ( :( ) line. I don’t have much of a problem when I hit Level3 as Time Warner customers seem too, but once we hit the handoff at hop 19, you see where the problems become similar. Notice, this is a tracert to a battle.net server.

    Tracing route to useast.battle.net [63.240.202.127]
    over a maximum of 30 hops:

    1 <1 ms <1 ms <1 ms WRT54G [Redacted]
    2 1 ms 2 ms 1 ms Redacted
    3 14 ms 7 ms 31 ms Redacted
    4 9 ms 8 ms 8 ms ge-3-13-ur01.owingsmills.md.bad.comcast.net [68.
    85.170.241]
    5 10 ms 9 ms 8 ms te-8-4-ar01.howardcounty.md.bad.comcast.net [68.
    87.136.1]
    6 * 9 ms 13 ms po-10-ar02.whitemarsh.md.bad.comcast.net [68.87.
    129.34]
    7 16 ms 13 ms 18 ms te-0-3-0-0-cr01.philadelphia.pa.ibone.comcast.ne
    t [68.86.90.73]
    8 16 ms 15 ms 16 ms pos-0-8-0-0-cr01.newyork.ny.ibone.comcast.net [6
    8.86.85.10]
    9 69 ms 14 ms 16 ms xe-11-3-0.edge1.NewYork2.Level3.net [4.71.186.5]

    10 16 ms 14 ms 16 ms ae-43-99.car3.NewYork1.Level3.net [4.68.16.197]

    11 16 ms 33 ms 15 ms att-level3-oc192.NewYork1.Level3.net [4.68.127.1
    50]
    12 74 ms 51 ms 34 ms tbr1.n54ny.ip.att.net [12.123.3.57]
    13 40 ms 32 ms * cr2.n54ny.ip.att.net [12.122.16.133]
    14 33 ms 23 ms 23 ms cr2.wswdc.ip.att.net [12.122.3.38]
    15 26 ms 32 ms * tbr1.wswdc.ip.att.net [12.122.16.54]
    16 23 ms 24 ms 19 ms 12.123.8.33
    17 27 ms 28 ms 34 ms 12-122-254-98.attens.net [12.122.254.98]
    18 30 ms 30 ms 26 ms mdf001c7613r0004-gig-12-1.wdc1.attens.net [63.24
    0.193.14]
    19 * * * Request timed out.
    20 * * * Request timed out.
    21 * * * Request timed out.
    22 * * * Request timed out.
    23 * * * Request timed out.
    24 * * * Request timed out.
    25

    Time outs for the rest of the tracert.

    @OnceWasCool: Like YouTube operates independently from Google, Blizzard activates independently from Activision.

    On top of that, your statement is mis-information. Sierra Entertainment is a subsidiary of the newly crowned, Activision-Blizzard, which is in turn, partially owned by Vivendi VA (Formerly known as Vivendi Universal) – Vivendi owns 52% of Activision-Blizzard stock.

  88. Quatre707 says:

    TCP/IP was not designed for video streaming, VOIP, massive online games!!! It’s been said numerous times by many of the very people who created the protocols.
    The Quality of Service requirements for such applications are HUGE!

  89. AndyAndy719 says:

    @Papa Midnight: So judging by best practices w/ DNS naming – i’m willing to bet the following:

    tbr1.n54ny.ip.att.net (and the cr2 hostnames) are core/backbone routers.

    mdf001c7613r0004-gig-12-1.wdc1.attens.net – that’s probably the interface on the router right before the handoff to blizzard.

    the mdf == main distribution frame, 001 is probably rack/location. c7613 == cisco 7613. r0004 is unknown, gig-12 probably denotes the interface on the router, and the -1 is one side of the connection.

    If you do a DNS lookup on 1 up lower, 63.240.193.13, you get mdf001c7613r0001-gig-12-2.wdc1.attens.net. gig-12-2 = same interface as above, but the -2, so this is prolly the /30 subnet used for the interface IPs.

    That would explain why traceroutes stop here – blizzard wont give insight into their network. Firewall/network prevents insight into the network. It doesnt mean somethings busted, they’re just not going to make it easy for people to figure out which machines/routers to hack.

    Which also says they might have a gigabit connection here. Again, no one really knows, but maybe their connection is saturated? A gigabit connection probably isnt enough for all their traffic, but who knows.

    @Quatre707: I dont think this is an issue with protocol design, but more of routing issues. That, and getting companies to work together to make all their customers happy and not blaming each other.

  90. viqas says:

    I think the NSA is unable to keep up with all the WOW activity

  91. 3drage says:

    Sounds like Time Warner is probably packet shaping the important stuff (They recently roll out VoIP service?) Sounds like their QOS gets pushed down on the priority list by the time it’s reached Verizon or AT&T. Instead of playing the corporation blame game, Blizzard/Verizon/AT&T and Time Warner need to get their intelligent guys (the ones who actually do the work and not this schmo who has a degree in management at best) and get their routers to start talking to themselves the most efficient way possible.

    Passing the buck helps no one, if this guy was really concerned for his customers he’d say. “I just came across this information and am working getting contacts between the other companies to work on router and peer configs to provide the best service possible.”

    I really hate this about upper management, he probably didn’t even talk to his technicians before posting that information.

  92. 3drage says:

    Quatre707, how would you describe then why other ISPs have no issues with WoW, or any other MMRPG? You gotta learn your history, TCP/IP may not have been designed for gaming, but games designed themselves around TCP/IP. I’m on Comcast and haven’t had constant repeating issues. I’ve streamed video and downloaded large files while playing with no noticeable difference in latency. To say that everyone should expect poor service on a broadband connection is just plain wrong.

  93. Rctdaemon says:

    @hubris: No, it’s his internets!
    “I just the other day got… an Internet was sent by my staff at 10 o’clock in the morning on Friday, I got it yesterday [Tuesday].”

    NEEDS MOAR JON STEWART!

  94. OzBob says:

    Guh.

    Okay. So most likely TWC has oversubscribed their total bandwidth, or if they haven’t, they’ve breached their peering agreement with other providers. The final and least likely solution is the QoS that has been suggested by many.

    However, TWC should be ashamed of itself. Working for a company that purchases significant bandwidth, I know that you can definitely bitch and get some resolution.

    Clearly the TWC rep has no concept of technical skill other than tracert.

  95. the_wiggle says:

    @xthexlanternx: cox (oh so aptly named) is no better. been there done that. no more pc gaming for me :(

  96. Krrose27 says:

    @the_wiggle:

    TWWC/RR need to complain to level3 and other internet backbones for u.

  97. GhostMul says:

    I love how people are blaming the ISP, but could it also be the fact that they are connecting to a server that has a lot of activity? The more populated “realms” the worse the lag.

    Also if this is happening to east coast users that would make sense because distance does matter when dealing with networks. Most of the servers are located in the alternate universe known as California. If they had servers in the Midwest USA or on the East coast then they wouldn’t see as much of a problem.

    WoW is a fun game but Blizzard has a track record of shoving their head up their butt when there is a technical issue. They will always blame someone else.

  98. PinkBox says:

    @: I’m located on the east coast and play WoW. My latency is fine, and I play on both low and high pop realms.

    Then again, I don’t use TWC.

    Even if TWC isn’t “directly” causing it, the fact that (apparently) a majority of their users cannot access WoW because of their backbone, it makes sense to me to avoid them.

  99. Anonymous says:

    @EYELAZERZ:
    LMAO That was brilliant.

    @Consumerist
    What do you mean Blizzard is “passing the buck”? We just read that these slow hops are on Level3 backbones and other various companies; that can’t possibly have anything to do with Blizzard. They know it isn’t them, so they’re legitimately assuming the next likely culprit: TWC.

  100. psychos says:

    This guy obviously does not understand ICMP deprioritization on intermediary routers, return path variables, or routing in general. But again, this is not the right forum to discuss such things.

  101. Wormfather is Wormfather says:

    @Sorshha: You win, hands down man. That was ZOMG funny.

  102. Bladefist says:

    @chrisdag: Yes. You have 2 options. Dont play WoW, or play WoW over a vpn. So uhh, if you want to play, the choice is clear.

  103. campredeye says:

    People suck at reading… blaming TimeWarner, when clearly if their read their own tracert they can see it has absolutely nothing to do with TWC. It is the routing that is the problem, sucks TWC is taking so much flak!

  104. campredeye says:

    @Sorshha: Haha

  105. absentmindedjwc says:

    From what I am seeing, the blame isnt on either company. The last mile provider (Time Warner) should step up and contact Level3 and report a problem with the network in between the backbone and their line. But that is not saying that it is their fault there is a problem here.

    Granted, in most cases, the ISP does not contact the backbone provider, it waits for the provider to contact them. What I would recommend is for the users that are having the problem to contact Level3 and inform them of the problem with the connection, stating that the problem is taking place off of TW’s network.

    Then again… a little break from WoW has never killed anyone, and they may be better off without it ;)

  106. William Mize says:

    @: Only if there’s porn involved and then you may never get your internet back.

  107. TancredoLaevinus says:

    I work for one of the tier 1 backbones. Two days ago we began investigating problems with another backbone announcing routes with the AS path stripped off on some of their peering points and creating inconsistant and flapping routes. This was causing 50% packetloss on the RETURN path. You guys did realize that IP packets don’t have to return on the same exact path as you see in your FORWARD trace, right? The problem with all of these lay-people wanting to diagnose Internet connectivity problems is the same as having your senator perform brain surgery on you. If it’s not your field, listen to experts. If Blizzard wants to condem one of the backbones, they need to publish their evidence showing the path from Blizzard to RR or TWC or whatever they’re calling themselves these days. Stop jumping on the hate-bandwagon folks.

  108. 3drage says:

    @: You aren’t using correct logic when diagnosing the problem. If it were a server issue, it wouldn’t be just TWC customers having the problem. Think man!

  109. 3drage says:

    @ GhostMul, also the servers are spread out geographically. Servers have Pacific, Central, East coast, and Australian time. China have their own server farm that are also spread out geographically.

  110. Anonymous says:

    Hard to say where the truth is in this instance, however I’ve been fighting a similar battle with Cox.

    I’ll be sending 14 pages of e-mail correspondence to you guys later this week.

    In 2004/5 I was able to “crash” my network by booting up WoW. The trick was in how they were responding to P2P like activity (eg: online gaming). They would kill the signal in quick 2-3 second intervals causing bad lag, and eventually forcing a modem reboot (before it would communicate with the server again).

    Here’s audio of two Cox techs giving me 100% different answers as well as a link to my previous post recorded earlier this month.

    [virtualwayfarer.com]

  111. FijianTribe says:

    Rather than trying to rely on tracerout, test your network speeds using http:www.dslreports.com and test different site locations. Make sure you are even getting the speeds you are promised.

  112. JeffSimmermonTWC says:

    Hi there — this is Jeff from TWC again. Thought you might like to know that I just got off the phone with the VP of Operations. His team has been working since yesterday with Blizzard, AT&T, Level3, AboveNet and WoW users to identify the source of the problems. As I mentioned at the top, there are a number of WoW users who aren’t using TWC at all that are also experiencing latency/lag issues.

    Rest assured that people from all the aforementioned companies are testing and searching together to identify and fix the problem, resulting in a better experience for the end user. I’ll share more when I know more myself …

  113. FrugalFreak says:

    I too am with Roadrunner, but I go through Brighthouse. I am on the net about 20 hours a day and I noticed a HUGE slowdown around the time the RoadRunner e404 browser redirect page scandal went live and around that same time the new 15 mbps service was about to be offered. I personally think it is the punishment of consumers opting OUT of the browser redirect after it was made public and a intentional slowdown of service to make the new 15Mbps service for $10 more look like a good upgrade. I was sold 7mbps in Alabama and I never go above 4.5mbps. They do control and pick their net provider and think they went with some backroom deal partnership with backbone providor to have less expenditure all the while cramming HD/broadcast/phone/Net services into what they already had. Getting more from less is Brighthouse’s motto.

  114. digitalklepto says:

    As a tech that works for Road Runner, I have had my hand in on dealing with this sort of situation from time to time. As many of you know and show, narrowing the problem down to a specific piece of equipment is not usually a difficult task. We have on many cases had single customers, or a small handful of customers experience issues like this, actually with WoW. If it is actually a piece of equipment that is maintained by Road Runner, the fix is usually relatively quick. When the equipment is operated by another service provider, it is often difficult to arrive to a quick fix. Point in case here was a situation where there was a router on another service provider’s network that was causing the lag. After a considerable amount of time consumed just trying to contact the proper level of support for that equipment, it was days before they resolved the issue. Personally, I would attribute that to a competing service provider not caring whether or not our customers can use their connection to their own liking or not.

    Now, with no desire to pour through literally thousands of NOC (National Operations Center) tickets for confirmation, I can assure you that if enough of you guys with this problem have called in to complain, there will be a ticket open, and steps have been taken to work with Level 3 (or whoever) to correct the issue. That being said, short of an entire region having routing issues, whoever said it above was correct, they aren’t likely to change the routing tables for a small handful of customers experiencing this problem.

  115. noodleslayer says:

    I find this response to be incredibly disgusting from a “director of digital communications” I would hope that they’d know how tracert actually works.

    It doesn’t give you latency BETWEEN routers, it gives you the latency to that router. So in short it isn’t necessarily particularly good at telling you where there is latency in the network.

    For those who haven’t taken a college level comp sci networks course, tracert works by sending a packet with a low TTL (time-to-live) out to a specified destination, starting with a value of zero and incrementing upwards, it then waits for a TTL exceeded error packet to come back to it and lookups up the address on that packet. This means it sends out a new packet each time it goes to the next router, so the latency times are for that particular trip through the network, not one single trip. It means that temporary network effects (increase in traffic, collisions, etc.) might increase the latency on some trips, but not others, but it’s not necessarily between the two specific routers that it appears, but rather ANY routers on that path up to that point.

    Now on a healthy network with stable equipment apart from momentary congestion and other network hickups the general trend of a traceroute should be that of cumulatively higher pings as it goes through more hops, for example, here’s a tracert from where I am to google:

    Tracing route to http://www.l.google.com [64.233.169.104]
    over a maximum of 30 hops:

    1 22 ms 25 ms 20 ms ip72-205-192-1.sb.sd.cox.net [72.205.192.1]
    2 9 ms 8 ms 21 ms 68.6.13.125
    3 17 ms 21 ms 23 ms 68.6.13.133
    4 27 ms 25 ms 25 ms paltbbrj02-ae0.0.r2.pt.cox.net [68.1.0.235]
    5 68 ms 113 ms 123 ms 216.239.49.250
    6 26 ms 27 ms 29 ms 209.85.243.123
    7 84 ms 108 ms 98 ms 209.85.242.210
    8 86 ms 87 ms 71 ms 209.85.243.116
    9 73 ms 75 ms 75 ms 209.85.241.20
    10 81 ms 77 ms 71 ms 209.85.241.23
    11 99 ms 90 ms 92 ms 209.85.242.209
    12 97 ms 92 ms 91 ms 72.14.236.200
    13 99 ms 97 ms 87 ms 216.239.49.149
    14 102 ms 102 ms 87 ms yo-in-f104.google.com [64.233.169.104]

    Again, the cumulatively increasing trend of latency is very clearly visible on this traceroute. There’s a couple places where there’s been minor increases in latency on a given run, but nothing too drastic. The traceroutes shown in that thread show a very different story.

    It more is indicative of there being periodic lag spikes on the network, the fact that for different people it’s happening in different places means that it’s more likely to be on the roadrunner network imo. Unless you happened to run the traceroute at the exact right time, it’s going to blow through the nearest hop routers much faster, so statistically speaking it’s much more likely to show greater lag when it’s trying to ping a further router.

  116. zithero says:

    it’s not TWC’s backbone. you can’t blame TWC because the server that these WoW players are trying to get to is too far away from them, and that other companies back-bones are slowing the packets down.

    I work for Cablevision, where recently I had to field a call from a sub who was complaining of ping spikes on a counter strike game. He has the optimum Boost service, which offered 30mbps down, and 5mbps up stream. After checking his speed, I found no issues, and no ping issues when pinging him from our server – he complained again that his pings to the server were spiking, to which I asked, “How many people are logged into that server?” he said upwards of twenty people, and they all weren’t complaining of ping spikes.

    I then gave him a brief education on the way a network and server work – as I am pinging him from our call center (Which is NOT our headend btw… the headend is several miles away from the call center, and actually to be honest, closer to the customer calling), I am not only getting no slow pings, and all of them were under 10ms. his buddy seemed to chime in for him after this and go “Well of course HE can ping you fine, he’s at the source of your network.” – I will say this many times, if you’re getting poor ping times or ping spikes or getting booted from servers… it’s the SERVER – not the ISP (usually).

    I was never disrespectful or unsympathetic to the fellow, but there’s only so much I can do from my end – I finally had advised him to hook up directly to the modem rather than going through his router – thus eliminating a hop. after a few moments he did say his pings were slightly better. which I told him: “Well, any time you can save the packet a trip through the internet, one of the hops being your router, you’ll find better pings.”

    I’m not saying that slow speeds are never the responsibilities of the ISP, but things like Ping spikes are the nature of the beast known as the internet.