Enraged Verizon FiOS Customer Seemingly Demonstrates Netflix Throttling 398
MojoKid (1002251) writes The ongoing battle between Netflix and ISPs that can't seem to handle the streaming video service's traffic, boiled over to an infuriating level for Colin Nederkoon, a startup CEO who resides in New York City. Rather than accept excuses and finger pointing from either side, Nederkoon did a little investigating into why he was receiving such slow Netflix streams on his Verizon FiOS connection. What he discovered is that there appears to be a clear culprit. Nederkoon pays for Internet service that promises 75Mbps downstream and 35Mbps upstream through his FiOS connection. However, his Netflix video streams were limping along at just 375kbps (0.375mbps), equivalent to 0.5 percent of the speed he's paying for. On a hunch, he decided to connect to a VPN service, which in theory should actually make things slower since it's adding extra hops. Speeds didn't get slower, they got much faster. After connecting to VyprVPN, his Netflix connection suddenly jumped to 3000kbps, the fastest the streaming service allows and around 10 times faster than when connecting directly with Verizon. Verizon may have a different explanation as to why Nederkoon's Netflix streams suddenly sped up, but in the meantime, it would appear that throttling shenanigans are taking place. It seems that by using a VPN, Verizon simply doesn't know which packets to throttle, hence the gross disparity in speed.
Re:Alternative explanation (Score:5, Informative)
Thats how the internet is paid for. The sending provider pays the receiving provider for the bandwidth, and this is the only rational way it can be.
No. That is not how it works. The truth is that the smaller provider pays the larger provider, no matter which direction the traffic flows. Some companies, like Netflix, are nice enough to not use their size as an excuse to charge people -- they offer free peering at internet exchanges. Other companies are maximally greedy.
Re:Could be a different route involved for the VPN (Score:5, Informative)
It is also possible the the VPN packets are transiting a different upstream peer from Verizon and bypassing the peering bottleneck at issue. Assuming that Verizon is performing inspection of packets and throttling only Netflix packets is quite a leap.
This is exactly what's happening. I do the same thing for a specific server I use. Standard routing via FiOS results in consistent 1mb download speeds. I set up a GRE tunnel to my VPS host and I get consistent 10mb download speeds. The culprit appears to be a shitty peering connection between so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net (130.81.151.246) and 0.ae2.XL3.CHI13.ALTER.NET (140.222.225.187).
Re:Could be a different route involved for the VPN (Score:3, Informative)
You do not understand how BGP works.
The problem isn't the data that verizon is sending to netflix, it's the data netflix is sending to verizon. Verizon messing with routing policy on netflix's announced prefix's wouldn't have an effect on verizon's streaming speeds. The traffic flows from Netflix to Verizon.
Therefore, in order to influence streaming speeds, Verizon would have to change their routing policy on how they announce their own routes in order to influence which links netflix traffic can come in on. The problem is, there's no way within BGP for Verizon to say I want Netflix traffic to only come in over these specific links. It would influence *all* traffic from that peer. Routing policy is destination based, not source based, and not source-destination based. By simply announcing the routes are more preferable over Level 3 saturated links, that forces traffic Level3 delivers for those prefixes to come in over those links.
Sure, Level3 could do some traffic engineering of their own and ignore or mutate some parts of Verizon's route announcements, and force that traffic in over unsaturated links Verizon may have with Level 3 (if there are any), but Level3 is a middleman. Doing so would take them out of their middleman status and put them firmly on Netflix's side. Verizon's likely response would be to immediately de-peer Level3.
The only folks who can effectively change how the traffic reaches Verizon's network is Netflix. They determine their outbound routing policy, but only up until their own border. Once it transits to another AS, it will be forwarded according to the upstream AS's routing policy. If Netflix wants to avoid saturated Verizon-L3 interconnects, the only thing they can do is not send traffic to Level3 for Verizon prefixes. They could easily modify their inbound route policy to send traffic for Verizon's prefixes via another peer. This is something that Level3 does not want, because it effects their revenue, hence their seeming to take sides with Netflix on the matter. It's one thing for Level3 to have an opinion on what Verizon is doing, that doesn't really effect operations. The second you change operations to try and force that opinion, well, you're likely to invoke the Law of Unintended Consequences.
Now, for whatever reason, Netflix has decided to go ahead and keep sending Verizon traffic to Level3. The reality is that if Verizon has decided to be douchebags about this, then they can do the same thing for whatever peer the traffic is ingressing through. Maybe all of Netflix's other peers ultimately transit to Verizon via Level3 anyway, which would make any change of forwarding policy moot.
About the only way for Netflix to solve this is to go ahead and cut out the middleman and just pay Verizon directly for interconnects into their network. This is what Verizon wants: another revenue source for traffic their going to deliver anyway. This is what Level 3 does not want: When you cut out the middleman, the middleman makes no money.
Netflix has already done it with Comcast and AT&T, so it's not surprising that Verizon wants in on this action as well, and will continue to be douchebags about it.
In the meantime, savvy customers can come up with their own solutions in order to avoid having netflix traffic destined for them coming in over saturated links. VPN and tunneling are two perfectly valid solutions.
Re: Alternative explanation (Score:5, Informative)
You obviously missed the article where Netflix supplies a tower-pc sized box with all of netflix on it to ISPs for free:
Netflix Boxes [gizmodo.com]
Re:Could be a different route involved for the VPN (Score:3, Informative)
there are a dozen video providers i can name who stream video with no problems. Hulu, HBO Go, Amazon, Vudu, Cinemanow, Apple itunes, ABC, History Channel, Disney, Lifetime, PBS, Fox, ESPN, NBC and others. most of this is video on demand, but they have some live video they stream as well.
only netflix is having these issues. difference is that everyone uses a CDN to host their content inside the ISP's network or close to. there are at least a half dozen CDN's that do most of the hosting
the way it has worked for 15 years since Akamai made the first CDN because people figured out a long time ago you can't send large files over the internet. the way it works is the CDN pays the ISP for hosting and bandwidth and the customer pays the CDN
netflix used to use one of these CDN's, forgot exactly who but they let the contract expire last year
instead netflix came up with a cool CDN box but they demand ISP's host it for free and give them free bandwidth as well. i guess netflix got so big they think they can extract payment from everyone
and if there is a lawsuit it will be brought up that this is only a netflix issue and there would be lots of discovery on netflix's business for the last 5 years
Re:Could be a different route involved for the VPN (Score:5, Informative)
Re: Alternative explanation (Score:4, Informative)
Bandwidth is perhaps cheaper than you suspect.
I worked for a regional ISP that serves about 50.000 subscribers. We had multiple 10 Gigabit Ethernet connections to various peering points, one of which happens to be where Netflix peered with us. Total cost for that peerage: the cost of the extra fiber capacity, plus engineering the peer.
As opposed to housing Netflix servers at our data center. First off, to service that many potential streams might require a few boxes and a not insignificant storage array. We actually did have a similar arrangement with another very large content provider: their stuff took about a half-rack. It then needs to be added to network monitoring, and you need to train your NOC staff what to do when that little red light comes on. And the equipment will fail: the "other content providers" equipment had a MTBF of a couple of months. The hard drives will take a pounding.
And we were small enough that when we asked Netflix to co-locate in our data center for free they actually said "Not interested."