Sandvine CEO Says Internet Monitoring a Necessity

Khalid Baheyeldin writes in with a CBC interview with the CEO of Sandvine, Dave Caputo (bio here). Sandvine is the Waterloo, Ontario-based company that provides the technology that Comcast and other ISPs use to overrule Net neutrality by, for example, injecting RST packets to disrupt Bittorrent traffic. Caputo says, among other things, that Internet monitoring is a necessity. Some of the comments to the interview are more tech-savvy than the interviewee comes across.

Of course it's needed

And we can sell you just the product you need for that.

Gotta love those statements.

CBCNews.ca: Has the internet always been managed, because this idea of network management almost seems recent?

Caputo: I had dinner with Vint Cerf [a U.S.-based computer scientist often called the father of the Internet] probably a year back and I think he'll be the first to admit that he's surprised and shocked at what his authoring of TCP/IP has meant. If you look at that underlying transmission protocol, when you send a flow of packets -- if they're getting through -- they get bigger until you get congestion, then the packets get smaller. The idea of flow control in the internet has been a tenet of it since day one.

It really depends on where you draw the line on what management is. The service provider has to figure out the business model of how much service they're going to give a subscriber and how much bandwidth they're going to provide to the internet. That oversubscription ratio is their business model.

For every five megabits they sell you for $40, they buy a quarter of a megabit because they're planning on you not using your computer 24/7. They count on you being away at work or being asleep. They simply cannot provision that five megabits because that costs way more than what they're selling it to you for. They need people not using the internet for it to work at $40 a month. Now CBC may buy its one-megabit connection for $800 a month because it's a dedicated one-megabit connection.

...and...

CBCNews.ca: So theoretically an internet service provider could sell customers a dedicated peer-to-peer router?

Caputo: Conceivably. The beauty is to let the market figure it out, and it will.

So he wants to sell technology that allows the ISP's to OVERSELL their bandwidth while LYING to their customers and he refers to that as "the market".

How about just telling the customers EXACTLY what they're paying for?

For $40 you get a guaranteed MINIMUM bandwidth of X with a potential to burst to Y.

If you want more, you pay for more.

Re:Gotta love those statements.

All ISPs oversell, with our without Sandvine's products. Your ISP tells you you're getting a certain amount of bandwidth, but you aren't, at least not 24/7. This has always been the case from day one.

This company isn't doing anything particularly brilliant. ISPs have been doing ad hoc versions of it for years and years.

Re:Gotta love those statements.

Its more like selling access to the keg, telling everyone they can drink the whole thing, and expecting everyone to blackout before they notice its gone

Re:Gotta love those statements.

Everybody in my neighborhood picked up the phone at the same time and half of them couldn't get through!

Overselling is not a bad thing. It can just mean that you sell based on statistical maximums rather than theoretical maximums which never happen. When done this way, there's absolutely nothing wrong with it.

When 90% of your customers are offline at any given time, there's no point in provisioning more than one tenth of the bandwidth you would need to support all of them downloading at the maximum rate simultaneously.

The problem is not overselling. The problem is that some ISPs oversell too much. They aren't willing increase capacity to match actual use, but instead try to reduce usage to match actual capacity. This is wrong. But the simple fact of overselling is the only sane way to do business.

Re:Gotta love those statements.

Exactly right. So complain about insufficient capacity, and not about overselling which is necessary, common, and entirely reasonable.

It just gets me how it seems like everybody in these discussions does not actually understand reality. "Ooh, the evil cable company promised 100 people in my neighborhood 5MBit connections but they don't actually have 500Mbit of bandwidth serving us! What a bunch of liars!" Sorry guys, but that's not actually how it works!

Now if people will complain about a lack of capacity then I'll be right there with them. But everybody just jumps straight to complaining about "overselling" and it makes them look like a bunch of fools.

To take your analogy, if you know from past behavior that you can sell beer "subscriptions" and only purchase half the beer that your subscriptions would require because most of your customers won't drink their full subscription, this is just good business practice and it's a good thing to do.

Re:Gotta love those statements.

That's all true, however, the point being made is that companies are not selling/telling their customers this. They are advertising it as unlimited.

To build on the analogy, you can have unlimited beer but if we see you drinking more than a six beers a day we'll cap how much beer you can have. See how ridiculous that is?

Overselling is not unreasonable. Advertising as unlimited is.

Re:Gotta love those statements.

I disagree. Advertising as unlimited is perfectly reasonable, if you can provide it. There's nothing that says you can't. This should be obvious simply by observing that a huge number of ISPs over a very long period of time have advertised and provided unlimited access with no problems.

The problem comes when you no longer want to provide it but still want to advertise it, which is what these large US ISPs are beginning to do, and this is indeed unreasonable.

Back to the beer analogy, let's say you sell a beer subscription that's limited to 1 beer an hour but is otherwise unlimited. However you only provision your restaurant for 10 beers an hour despite the fact that you've sold 100 subscriptions. Nothing wrong with this so far. If you worked out your numbers to see what your peak demand is and that peak demand is 10, then you're in good shape!

The problem comes when people start drinking more, and so your peak demand increases past 10 beers per hour. At this point you have two honorable choices. One is to say, sorry, we can no longer offer the unlimited subscription, would you like a subscription which comes with 30 beers per month, and a charge per each beer after that? Another is to increase your supply of beer. If your subscribers are now peaking at 15 beers per hour then arrange for that amount to be delivered. Unfortunately these US ISPs are taking a cowardly way out. They are, essentially, continuing to offer the unlimited beers but are finding all the guys who constantly come in for one beer every hour nonstop, intercepting them on the way out the restaurant, dragging them into the alley, and beating them up.

But if you just increase your supply to match the actual demand, there's nothing wrong with overselling while advertising unlimited service, since that is in fact exactly what you are providing.

Re:Gotta love those statements.

So obvious that everybody knows about it and solves it.

It's not a problem. You size your infrastructure for peak demand. Yes, that peak demand tends to happen between 6PM and 11PM. Yes, a lot more people use their connection a lot more than they do at other times of the day. But no, that peak demand is still well below the theoretical maximum. If you oversell by sizing your network to average demand then, yes, you will fail hard. But if you oversell by sizing your network to actual peak demand then you will succeed in providing what you promise while still provisioning only a small fraction of the theoretical maximum usage.

The concept of overselling isn't very hard to grasp. I don't know why so many people here just don't get it. Aren't you people supposed to be smart?

Re:Gotta love those statements.

Could you explain what would be lost if such as page was clearly marked as highly technical, and was optional to read?

I'm thinking something along the lines of the link text being "high technical information" and the page having a header that goes "The information on this page is meant for people who want to know the technical details of how internet service is provided by $ISP. It's written with the assumption that the reader knows what TCP window sizes, anycast routing and best-efforts networks are and which practical implications they have. If these terms are new to you, you probably want $USER_FRIENDLY_DOC."

I'm with you on the point that you shouldn't try to force your users to understand the technology (just as the car stereo salesman doesn't wax on/wax off about how frequency modulation works and the benefits of optical versus magnetic storage). But not having to explain something is different from having to not explain it. Why not make both groups of users happy?

Beating Sandvine

http://redhatcat.blogspot.com/2007/09/beating-sandvine-with-linux-iptables.html [blogspot.com] [blogspot.com]
If you are running linux or a linux based router with iptables give this a try. My speeds returned to pre-sandvine levels.

"If you are using a Red Hat Linux derivative, such as Fedora Core or CentOS, then you will want to edit /etc/sysconfig/iptables. First, make a backup of this file. Next, open this file in your favorite text editor. Replace the current contents with this, substituting 6883 with your BitTorrent port number:

*filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0]
-A INPUT -i lo -j ACCEPT
#Comcast BitTorrent seeding block workaround
-A INPUT -p tcp --dport 6883 --tcp-flags RST RST -j DROP
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
#BitTorrent
-A INPUT -m state --state NEW -m tcp -p tcp --dport 6883 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 6883 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

Reload your iptables firewall with service iptables restart. You should now see a great improvement in your seeding.

If you are using Ubuntu or another non-Red Hat Linux derivative, then place the following in a file and execute that file as root.

#!/bin/sh
#Replace 6883 with you BT port
BT_PORT=6883

#Flush the filters
iptables -F

#Apply new filters
iptables -A INPUT -i lo -j ACCEPT
#Comcast BitTorrent seeding block workaround
iptables -A INPUT -p tcp --dport $BT_PORT --tcp-flags RST RST -j DROP
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
#BitTorrent
iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport $BT_PORT -j ACCEPT
iptables -A INPUT -m state --state NEW -m udp -p udp --dport $BT_PORT -j ACCEPT
iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited

Your firewall is now configured and you should have great upload speed now. You will have to run this script every boot, by the way. One easy way is to call the script at the end of /etc/rc.local."

Re:Beating Sandvine

On the other hand, if a sufficiently large population of Torrent users made the change, that particular attack vector would pretty much go away.

How fscking stupid can a suit get?

From TFA:

For every five megabits they sell you for $40, they buy a quarter of a megabit because they're planning on you not using your computer 24/7. They count on you being away at work or being asleep. They simply cannot provision that five megabits because that costs way more than what they're selling it to you for. They need people not using the internet for it to work at $40 a month. (Emphasis added)

So let me get this straight--poor planning on their part somehow does constitute some form of emergency on my part?

Re:How about selling what you have?

So you can't provide those fantastillion megabits per sec for 40 bucks. Ok, I can see that. How about ... I dunno... selling what you can sell?

The problem is, that a megabit still costs $300/mo or $700/mo. There's no way around that.

You can get un-fucked-with bandwidth for that price, or you can live with the fact that your concentrated. You can't have it both ways.

The more you buy, the cheaper it gets, so you could order a T3 or something for like $5000/mo and then sell it to your neighbors for like $200/mo... (not including the cost of the routers).

... but one thing you couldn't do is sell unfiltered unconcentrated bandwidth to your neighbors for $40/mo.

I don't know about you, but I'm happy to have 3megs part of the day for $30/mo instead of my old ISDN line for $145/mo. Or maybe dialup? No thanks. I'll take the concentrated 3megs for $40.

It's just not realistic to expect to get more for your $40 than they get for their $300.

Re:How about selling what you have?

http://www.dslreports.com/shownews/Cogent-McBandwidth-Gets-Cheaper-95203/ [dslreports.com]

$7/mbit (of course talking about decent volumes here with the cheapest provider and I guess with fiber already in the ground)

However that should give you a clue how much everyone is overcharging everywhere. The expensive part is the digging, but it is good (money earning) business to charge big money for small traffic volumes on lines that in reality could support far higher volumes. Not to mention how inefficent a big part of the industry is.

Atleast that is the only way I can explain how some countries are managing to supply such nice bandwidth to their citizens without getting economically ruined.

What has overselling to do with monitoring?

As stated in the article is that the ISP's are selling you 1 megabyte while really buying you 1/4th of a Megabyte... Network monitoring is in other words necessary to ensure you in other words only use 1/4th of a Megabyte for every Megabyte you buy. It's right there in his argument!

This is the "perfect storm"

That I have talked about several times in the past. When you combine corporate greed,the ease of buying off congress,and the desire by the telecos not to bother sinking any of their massive profits into infrastructure upgrades even though the taxpayers shelled out for them,you have the "perfect storm" that in the end will leave the Internet nothing but a crippled former shell of itself.

After all,it isn't but a single step to go from "We are doing RSTs to save our network!" to "We can use this technology to "guide" our customers to our services and to our affiliates and to "discourage" them from using our competitors and make even greater profits!".

Mark my words,the Internet will end up a bunch of "walled gardens" like in the days of AOL and Compuserve. The amount of bandwidth they give you for "non-affiliated" services will be so pathetic as to not matter. They will offer the few big boys like Google a free pass to keep them from fighting it while the rest can just starve. The days of a wild and free Internet are coming to a close IMHO. And the world will be a much worse place for it. After all I'm sure that each "garden" will have their own "free" news feed where only approved views will be heard and the corporate spin will always be considered gospel. But that is my 02c,YMMV

ISPs should never send an RST

ISPs should never muck with a TCP stream. They're entitled to send ICMP messages. ICMP Destination Unreachable has codes for things like "(13) Communications Administratively Prohibited" and "(10) Destination host administratively prohibited". Then at least the user knows 1) that somebody along the route didn't like the packet, and 2) who to blame. There's a right way to do this, and sending an RST isn't it.

Client software may not pass all the ICMP info up to the user, but that could be fixed easily enough.

Re:"Honour"

Back in my day we had a honor system that basically said "don't sell 100 gallons of milk when you only have 20".

Re:What the fuck is 'churning'?

Churn is an industry term for the percentage of your users will leave for somebody else and the percentage of their users that leave for you. Frequently these users are the same damn people swapping back and forth.

So despite gaining and losing lots of users, everyone's base stays roughly the same, like a churning ocean, but each one of those churners costs you $X every time they switch sides(freebies, paperwork, number portability, etc).

Apparently this is now the superlative of "discontinuing service", i.e. "you guys suck, I'm leaving for your competitor."

Re:Churn is nothing new

That's the dumbest term I've heard since people calling single enemies "mobs" in online games

That's MOBs for you, not "mobs". It is an ancient MUD game engine acronym which stands for "Mobile OBject". One of those archaic game lingo terms which still survives but the origins of which most of the young whipper-snappers do not have clue about.

Now about that lawn of mine ...

Re:Maybe I'm being selfish

Where is it written that it is all-you-can-eat?

All over ISPs' advertisements. Unless they've redefined the word "unlimited".

An Internet which is not neutral is less useful than an Internet that is. If web browsing is sped up at the expense of streaming video, that's going to hurt some people more than others. If streaming video is sped up at the expense of games, a whole other group is affected. Since people come up with new ways of using the Internet all the time, and we can't predict new uses, the best strategy is to give all packets equal measure.

Rather than throwing out Net Neutrality, it'd be more productive for ISPs to find business models that don't involve overcommitment, or at least make it less painful. Like some of the recent attempts to make P2P software favor nodes within the same ISP.

Re:Maybe I'm being selfish

Read your contract - the ISP may say unlimited; but the DON'T guarantee a bandwidth. All unlimited means is that they don't cut you off or charge you more if you exceed a certain data volume.

Let's get real here. If an ISP was really selling you a guaranteed dedicated bandwidth you would be paying a much higher price than you do now. Why do you think T1 is hundreds of dollars per month at 1.5 Mb/s? Because of the service guarantee, that is why.

Packet switching works economically because it is shared bandwidth relying on a statistical distribution of traffic on the network. During peak loads traffic will be slower than at off peak times unless the network is extremely over-provisioned.

There is another technology out there that gives a guaranteed bandwidth for every customer - which is rapidly being displaced because of its inefficiency - it is called circuit switched, and it is what the phone companies use to carry analog voice. Every call gets it's own dedicated bandwidth. All I can say is that you would not want an internet based on this network model. It is slow, inefficient and inflexible.

Now ISPs have a problem with users that run applications that present a high constant load because they don't fit the statistical model. High volume P2P is the primary offender right now. If people are using these sorts of applications when the network is heavily loaded it seems to me quite reasonable that traffic based on interactive applications (VOIP, video, HTTP) should receive priority. ANY good computing system should favor interactive applications over non-interactive applications. It is a basic system design principle.

Sorry to inform you, but to do this you need to monitor.

A lot of people whine that this breaks the idea of network neutrality. I disagree; network neutrality must not allow one type of communications stream or application to seriously degrade the performance or usability of all of the other applications. If that occurs you do not have a neutral network. You have a network that is dedicated to that one application. That is NOT what I as an end user want.

Re:speeding up 'your' Internet

That's not what I'm advocating at all.

Civility means getting what you pay for. Civility means behaving when there's a traffic jam. Civility means not having what you bought and paid for surreptitiously examined, weighted, and thwarted.

I'm not interested in jamming my neighbor's pipe. I AM interested in not being lied to, and for getting what I paid for, and not having my information sniffed by a cockamamie CIVIL liberty-avoiding bone head that calls him/herself a service provider.

Where, praytell, is the civility in THAT?

Re:Honestly, I'm SHOCKED!

I am shocked because Sandvine is a frequent supporter of Open Source Operating Systems and has contributed to BSD Conferences. I would have thought that they would support the openness of the internet too. Apparently, their monetary sponsorship of open source conferences are just a PR Stunt.

Sandvine is one of many telecomm gear companies that strongly support OSS. I used to work at a similar company with at least one ex-Sandvine co-worker. Basically, they build "devices" which they sell to ISPs and other big network operators. They build those devices with custom or off the shelf hardware combined with on OSS operating system, toolchain, and applications, plus a few closed source applications that contain their core competency and money proposition. This is often referred to as the "secret sauce" code.

These companies do support OSS and build their entire business model around it (in combination with some closed source). They aren't OSS zealots, but most of the employees are strong supporters of OSS and the companies are very good about contributing code back. A lot of the code in Linux and the BSDs is contributed by these companies. They support OSS conferences and the like, because they want to promote OSS, because it is a good way to recruit new talent, and because the improvements that come out of those conferences are often beneficial to their bottom line. A lot of people think OSS is created by hobbyists, but really Sandvine is a good example of who really makes up the OSS community and contributes code. It is mostly businesses who use it to make money in conjunction with hardware, services, or additional closed source software.