Mystery Solved? Using OpenDNS Results In Glacial YouTube Downloads For Qwest Customers
Earlier this week, we posted an email from a frustrated Qwest customer who said he couldn't download YouTube and other online videos at a speed equivalent to the Qwest service he was paying for. Qwest wrote to us, and spoke to the customer, and swore they were not interfering with any download rates. Instead, it looks like the problem is with OpenDNS, a free service that usually speeds up downloading, but that seems to have an issue when it comes to certain video streams.
First, here are two entries from the OpenDNS forums by a different customer, posted back in March, that describe the problem:
When I use OpenDNS for my DNS servers, YouTube loads videos very slowly (it is actually impossible to watch anything). The second I switch back to my ISP DNS, YouTube videos load fast.
I KNOW OpenDNS is ONLY a DNS server and should not affect my download speeds — BUT IT DOES, at least on YouTube. It does not matter if I use a Mac (10.5.6) or a PC (XP SP3) I get the same results.
Here's what happens with OpenDNS:
[removed screen movie from server but basically the "loading" spinner in youtube would sit there spinning, then the video would play for 1-2 seconds, then go back to the loading spinner for 20 seconds, then play a couple seconds, etc]
Here's what happens with my ISP DNS:
[removed screen movie from server but with my ISP DNS the movie loads fine and plays without a problem]
EDIT: Adding youtube.com to my whitelist does nothing.
I did some more digging and figured out that the .swf file is appears to be calling a .flv file that is hosted at:
v19.cache.googlevideo.com
When using my ISP DNS, it wants to pull the video from 74.125.165.88 — and the video downloads in about 90 seconds. In fact, the first 1/4th of the video comes in at a whopping 800 KB(ytes)/sec, then it seems it is trottled at the source end down to about 100 KB/sec for the rest (I'm assuming this is how they do it for youtube to keep their bandwidth in check).
When using OpenDNS, it wants to pull the video from 74.125.100.100 — and the video download crawls along and the estimated completion time is 20+ minutes.
(I am downloading just the FLV file separate from the youtube page using my browser's download manager)
If I manually put this IP in the URL for the .flv (no matter what I have my DNS set to), I get the same results. That is, I can put the "good/fast" IP in the URL and get a fast download even when using OpenDNS servers.
Apparently something is happening where OpenDNS is returning this "bad/slow" IP when looking up v19.cache.l.googlevideo.com
Our own OP performed a similar experiment and says his YouTube streaming is back to normal:
I ran a traceroute to v23.iscache5.googlevideo.com, the apparent server within Google that was serving up the video. I was a bit surprised to see the the traceroute cut off at an OpenDSN router (line 10 in the included traceroute) from my IP to the cited Google server. I have been using OpenDNS's servers for a a few years with for security reasons. I've been more than pleased with their service (free to individuals).
I reconfigured my network's DNS to point to QWests own nameservers. I again attempted viewing the same video. This time the download completed without stalling or interruption in the display. FlashGot's instrumentation showed a rate of 60Kbs. A traceroute to the same google video server yielded an entirely different route to the destination. See the included traceroute.
I ran these experiments several times and the results were consistent.
I would imagine that the reason OpenDNS routed traffic via their own IP-space is to identify potential malware and to protect its users from same, a laudable goal, and part of their service. I'll stay with QWest's nameservers until QpenDNS investigates my trouble ticket and fixes it or refutes my suspicion.
Did I get it right this time? Time will tell.
Traceroutes follow ...
Using opendns's name servers (208.67.220.220)
traceroute to v23.iscache5.googlevideo.com (208.67.216.132), 30 hops max, 40 byte packets
1 gateway (192.168.1.1) 1.166 ms 1.647 ms 2.089 ms
2 * * *
3 tukw-dsl-gw22-214.tukw.qwest.net (63.231.10.214) 59.107 ms 68.397 ms 68.847 ms
4 tukw-agw1.inet.qwest.net (71.217.184.169) 70.277 ms 70.600 ms 71.929 ms
5 sea-core-01.inet.qwest.net (67.14.1.194) 73.591 ms 73.757 ms 73.912 ms
6 sea-brdr-01.inet.qwest.net (205.171.26.82) 74.155 ms 73.224 ms 75.141 ms
7 POS1-1.BR1.SEA1.ALTER.NET (204.255.174.177) 74.957 ms 54.444 ms 55.264 ms
8 0.so-4-2-0.XT1.SEA1.ALTER.NET (152.63.105.82) 56.941 ms 56.282 ms 57.602 ms
9 195.ATM4-0.GW10.SEA1.ALTER.NET (152.63.104.1) 58.294 ms 56.959 ms 57.516 ms
10 opendns-gw.customer.alter.net (63.65.72.74) 58.111 ms 59.424 ms 59.580 ms
11 * * *
12 * * *
13 * * *
...
30 * * *Using QWest's nameservers (205.171.3.65)
1 gateway (192.168.1.1) 0.924 ms 1.327 ms 1.745 ms
2 * * *
3 tukw-dsl-gw22-214.tukw.qwest.net (63.231.10.214) 56.385 ms 57.373 ms 58.440 ms
4 tukw-agw1.inet.qwest.net (71.217.184.169) 60.214 ms 61.207 ms 62.190 ms
5 sea-core-01.inet.qwest.net (67.14.1.194) 63.198 ms 64.190 ms 65.183 ms
6 sea-brdr-01.inet.qwest.net (205.171.26.82) 66.152 ms 66.203 ms 66.916 ms
7 63.146.26.198 (63.146.26.198) 267.621 ms 213.340 ms 213.519 ms
8 sl-gw20-sea-0-0-0.sprintlink.net (144.232.6.8) 54.976 ms 54.198 ms 54.257 ms
9 sl-googl13-199181-0.sprintlink.net (144.224.13.138) 55.146 ms 54.540 ms 54.197 ms
10 209.85.249.32 (209.85.249.32) 107.489 ms 138.085 ms 128.856 ms
11 72.14.233.117 (72.14.233.117) 109.982 ms 108.183 ms 108.998 ms
12 209.85.242.209 (209.85.242.209) 131.685 ms 164.573 ms 137.819 ms
13 72.14.239.95 (72.14.239.95) 218.838 ms 209.426 ms 228.598 ms
14 209.85.251.41 (209.85.251.41) 248.187 ms 210.056 ms 223.112 ms
15 74.125.6.100 (74.125.6.100) 211.362 ms 219.634 ms 211.790 ms
It turns out another Qwest customer, Andrew, wrote to us yesterday to suggest the same solution to the problem:
I had the exact same problem with Qwest and YouTube; I am, I guess you could call "a heavy user". However, I realized that this began to happen after I used OpenDNS on my router. After I removed OpenDNS, YouTube worked great and every video downloaded very fast.
"Why does OpenDNS make YouTube load slowly?" [OpenDNS Forums]
Post a comment
Comments:
@Tallanvor: You are correct. Best path is part of the DNS game.
I think it's cool that OpenDNS provides the services they do, however, this is one of the biggest cons to using such a service. I debate the efficacy of additional safety in using their service. Maybe I like going to the sites they have blacklisted.
The moral of the story is that it's best to stick with the name servers of your ISP.
@Tallanvor: Or more likley all openDNS users are hitting one server instead of having it roundrobin'd like it would be normally
This really isn't a mystery. Big content providers (and Akamai especially) have boxes at many ISPs and use DNS tricks to point your computer towards the closest cache -- usually one at your ISP. By using OpenDNS, you're using Google's server that's closest to OpenDNS's server. There are probably additional bottlenecks between your connection and the cache closest to OpenDNS.
Bottom line: use your own ISP's DNS server for best performance on large files from big sites.
YouTube like most large sites uses a CDN (Content Distribution Network) to offset the load of all the video downloads and provide servers closer to users, which results in faster downloads. This is done by using a CNAME entry. If you are using OpenDNS your getting a cached response from OpenDNS rather than your own ISP as a result it won't be the optimal server.
This is normal behavior.
The solution is obviously to use the perfectly good DNS your ISP provides. OpenDNS provides a few toys, but none you couldn't reproduce yourself with a decent router or even just a program or two on your computer. Malware blocking is common on all major browsers now. It's not OpenDNS specific.
I switched to OpenDNS years ago when I was using Earthlink DSL for my ISP, but it had nothing to do with security. Earthlink got the bright idea that instead of serving a 404 error if you made a typo in your URL, they would instead serve a page full of ads and, supposedly, helpful search results.
Not only were these new ad pages slow to load, I didn't appreciate being forced to see ads despite already paying for service. I also didn't like the precedent of an ISP monkeying with web standards (they're called standards for a reason after all) so I stopped using their DNS servers.
Quite frankly, in that situation I'm happy to put up with slow Youtube downloads in the name of not having to put up with a bunch of BS from my ISP.
My OpenDNS experience was even unhappier. Regardless of what site I went to, it told me it couldn't resolve the name (there's some DNS for ya) and gave me a page of keyword advertising. Reliable, repeatable, regular behavior. If I waited a minute or so, a second attempt to reach the site would succeed.
But I have no interest in OpenDNS forcing me to see a page of ads before every single site I want to read, so I stopped using it pretty quickly. I don't even care whether this was some weird technical issue or a genuine scam, because it was so predictable and consistent.
@Kryndis: My ISP does that too, but there's an opt-out link at the bottom of the page. Just click that and you don't have to deal with the ad pages.
@eelmonger:
Wow, thanks, I hated the domain not found page for the same reasons in the parent post, and I was thinking of switching to OpenDNS to get rid of that. However, I looked at the not found page, and found the opt out link right at the bottom. I never thought to actually look at the page for an opt out, I just assumed that it was some 'feature' that could not be removed.
I use OpenDNS and have found it to be many times more reliable than Charter's name servers, at least in my area. I've never really looked at domain resolving speed, but I regularly visit YouTube and have never noticed any slowdowns or oddities in loading. I've even noticed that it seems to resolve typos automatically (like when I type ".ocm"). I, for one, will be sticking with it.
I used OpenDNS for awhile but recently it started taking a long time for DNS queries. At the moment I am using 4.2.2.1 and 4.2.2.2 which is closer to me and seems to be a lot faster.
In my opinion however, the best you can do is to set up your own DNS server. Seriously - it's not that hard, and a lot of people will have an old computer lying around. (I unfortunately do not.) You don't need an expensive computer. I bet you a Pentium 1 would run BIND on, say, Slackware just fine.
I gotta say, ShortBus (for all his/her possible shortcomings) has got this one right. There is so much nonsense going on in this thread it's scary.
For example:
1) OpenDNS serves up ads on negative hits negating one poster's stated reason for using it.
2) Your ISP's DNS server doesn't know where you are (at least in the vast majority of cases). Client to destination hop count isn't part of the protocol.
Hey guys -- David from OpenDNS here. It's late tonight, so excuse my brevity. Bottom line is that I haven't heard about this issue before but you can be sure we'll be figuring out what's going on. I definitely don't want people to have a slower experience using OpenDNS no matter the site.
I have colleagues in the networking department at Qwest and folks in the netops dept. at Google/YouTube so hopefully we'll get the issue figured out and resolved quickly; whatever it may be. :-)
Thanks!
-David
I wonder if the OP feels like an ass now..? It sounds like a case of assuming you know more than you really do. And as everyone knows, you can't spell assume without ass. As I said before I haven't seen Qwest throttle or do anything sketchy. I thought maybe he was abusing his connection and getting slowed because of that. It turns out Qwest wasn't doing a thing. And oh yeah, and just to reiterate, there is NO 3GB CAP!!!
I have previously read about video servers being placed "closer" to ISPs to minimize the number of hops and congestion. I don't use opendns myself but when I read those posts, it was sort of a "duh, of course!" moment. I hadn't even considered the possibility of him using different DNS servers. For any bandwidth heavy sites, you should be using your ISP's DNS. Yes, it could affect things in this way, as others have tried to explain.
Now, OP, would you please admit that you don't entirely understand what's going on, leave your DNS to qwest, and just pipe down? I'm sick of the righteous indignation and the paranoia about how Qwest is screwing with your connection. They aren't.
@davidulevitch: David, Please keep us up to date. I recently installed OpenDNS for its filtering capabilities, and about the same time I noticed a 50% or more reduction in speed across the board. I have not yet investigated further, so I can not confirm if it is OpenDNS, or some other issue (dying router, dying FiOS NID etc.), but this article and timing of my speed hit have me thinking I want to investigate further.
@ShortBus: And if you can't tell us exactly which comments are incorrect or correct, comments like yours are totally worthless...
@chris_d: Well, if I called my ISP (like OP did) and they told me they were throttling my connection, then said "oh nevermind we're not", I might be a little confused, too.
@Luke Karmazin: My ISP actually does have them at the node, but they are for shit. Half the time, my DNS queries to my ISP time out. Put them through to OpenDNS, though, and they work.
I hadn't noticed a problem with YouTube, BTW.
@Cocoa Vanilla: Yep, I run BIND on an old Pentium III (running Ubuntu) as a caching server to speed up web queries. It uses OpenDNS as it's upstream DNS and I've never had a problem with it.
Hi,
Long time reader first time commenter. If you want to check the performance of various DNS server's Steve Gibson at GRC has a program in Beta but its still pretty solid. You can find it here: [www.grc.com]
Of course that only shows the how quickly they resolve domain names and doesn't take into account other features OpenDNS may offer over your ISP's DNS system.
I used OpenDNS for a bit when Roadrunner's DNS servers were being DOS bombed awhile back. I had problems with it though, and the moment Roadrunner got the attacks under control I switched back again.
BTW, here's the Wiki on DNS for those who don't understand it. Simply put, DNS is what resolves the numerical IP address of a website when you type it in your browser... [en.wikipedia.org]
@Kryndis: OpenDNS sadly returns the IP to a 404 style page. You know HTTP is not the only thing that needs to resolve names. That bugs me about OpenDNS, but at least they do not use broken caching DNS servers that fail on anything other than the 4 most common records.
@admiral_stabbin: I don't use any of their blacklists at home. When I managed the network at a former workplace, I only used the phishing and porn lists.
I also don't agree with your moral. I'd say you should test both and choose the one that works best for you. I don't have any problems viewing YouTube videos, and I'm a user in Norway connecting to their London servers.
the server opendns-gw.customer.alter.net is apparently owned by verizon, and is dropping packets, but still reporting to other routers that it has a route to that server.
Hence the problem is with Verizon. Either they have a misconfigured router, or they are doing deep-packet inspection and deciding to slow down packets that were requested with OpenDNS. I lean towards the later explanation because the name of the server has "opendns" in there.
If that's true, why would they want to depreciate traffic that isn't using their nameservers?
@Rachacha: Yes please let us know what you find out, either here or on the OpenDNS blog. I've had to switch away from the amazing OpenDNS for this purpose and can't wait to come back.
@chris_d: The OP knows how to use tracert and do many other diagnostics that key into his intelligence regarding network routing. I had this same problem and assumed that it was Qwest, as DNS is a system used to resolve a site to an IP address, not to serve that site itself. It's definitely the last thing to think of that the DNS could affect how a site was being served, but in this case it is the culprit. Especially since he talked to Qwest first and they told him that maybe he was being throttled but they couldn't really find a right answer for him. That would lead me to believe that Qwest was acting sketchy since they were talking out of both sides of their mouth.
The Word Of The Day is "Geolocation".
Just about every large internet service has multiple data centers nation/worldwide. What's happening here is that Youtube's name servers will see the source IP address of the DNS request from your configured resolver, look up that IP's location in a database, and then return the IP address of a server in the datacenter that is most optimal (usually the closest, both other factors can be put into the equation) for the source IP presented.
That said, the geo-lookup is done on the IP address of the *resolver*, not your own IP address. That means that if your resolver is on the other side of the country or world, you'll get handed a rather suboptimal IP for the site.
And looking at the network that OpenDNS instructs you to use, it appears that their data center is in the Washington, DC metro area.
So, when you use OpenDNS, youtube thinks you're in Asburn, VA, not Tukwila, WA, and sends you to one of their east coast datacenters, thus giving you a much slower download than what you'd get when you use Qwest's resolvers, which correctly put you in the pacific northwest.
Even without OpenDNS "filtering" the traffic, this mis-location will happen. It's even worse if you're an international traveler, and keep your DNS settings to your stateside resolvers...
@Cocoa Vanilla: 4.2.2.1-4.2.2.4 aren't actually "closer" to you, they're anycast addresses which likely resolve to either your ISP's dns servers, or your ISP's backbone provider's dns server.
I've been using OpenDNS here in Southern California since the beginning of the year and haven't had any real problems compared to the normal service which was buggy and slow as hell. In fact, the only time where I've experienced slower service was just a few days ago and that stopped as abruptly as it began. Other than that, no complaints at all.
@ShortBus: Or, perhaps share what you believe to the the correct information. The info I posted appears to have been regurgitated multiple times...and, if I'm wrong...I'd love to know how. Only fools don't like to learn from their mistakes...
@davidulevitch: You rock for swinging by and sharing such an encouraging comment. While I don't personally have a current need for OpenDNS service, I do generally like what you folks do.
I'm thrilled to see your organization taking an interest in making things better for your users.
@scooby2: No, that's impossible. All OpenDNS does it resolve the hostname to an IP address. The video itself doesn't pass through them or involve them in any way.
What's happening is one of two things:
1) OpenDNS, by sheer bad luck, happens to have cached the address of a YouTube server that's slow or broken in some way.
2) YouTube makes broken assumptions about a person's DNS server being in close network proximity to their own network pipe and by sheer good luck when the poster disables OpenDNS, the DNS server happens to be closer.
The problem is almost certainly on YouTube's end because all OpenDNS does it point the person to YouTube.
There's more that needs to be said however. You are correct that this is an issue and that using your providers DNS "could" be better.
That depends on your area and your provider however. Cable companies are notorious for having terrible DNS server. (especially Comcast and Time Warner) Conversely though most Telco companies run pretty good DNS.
Best DNS option is to run your own at home and have it forward unknown requests to an external DNS server.
Here is wikipedia discussing the most popular ones: [en.wikipedia.org]
there is nothing faster than running your own DNS off an old unused PC. Or if you have Tomato on a Linksys WRT54G/GL/GS you can do this on router. Other routers have DNS SW built in to them but I like the separate PC behind the firewall option better.
@axiomatic: And for what it's worth, most Unix systems (not sure about OS X) do this automatically with a small program called "nscd". No idea how Windows handles it, though.
@ajlei: Make sure your version of Flash is 10 or higher. FireFox had a problem with Flash 9 that caused the sound problem with YouTube. For awhile, you had to use the beta version of Flash 10 as a workaround, but now you should just be able to upgrade to Flash the normal way on Adobe's site.
I'm on Qwest and i have been having this YouTube problem recently as well... but i've NOT been using OpenDNS, i've been on the 4.2.2.x ones.
Having removed those from my router's DNS configuration and reverted back to Qwest's own DNS servers the YouTube issue seems to be resolved... but the fact that i wasn't using OpenDNS makes me wonder.
Is it possible that the anycast addresses could have routed me to an OpenDNS server? If so, how could i find that out?
I also use open DNS and it dosent slow down my youtube videos so that I can notice, however I only have one question to ask and that is; Why would you download youtube videos anyway unless you are looking at instructional videos I guess any how I really really miss [stage6.com] it was the best web site ever created for media content all in hi def glorry and super fast streams of data. Hope someone out there revices such a site :(
I've had the same problem (slow YouTube downloads) and was using OpenDNS on my router, but had an old Comcast DNS IP address on my laptop (from back the the AT&T days).
Deleting the old AT&T/Comcast DNS address fixed it. I'm still using OpenDNS on my router and everything is fine (YT is now loading very fast again).
















If using OpenDNS usually speeds up downloading, then the DNS software (or nameserver) you use must affect how quickly you receive those precious packets.