Enhancement To P2P Cuts Network Costs 190
psycho12345 sends in an article in News.com on a study, sponsored by Verizon and Yale, finding that if P2P software is written more 'intelligently' (by localizing requests), the effect of bandwidth hogging is vastly reduced. According to the study, redoing the P2P into what they call P4P can reduce the number of 'hops' by an average of 400%. With localized P4P, less of the sharing occurs over large distances, instead making requests of nearby clients (geographically). The NYTimes covers the development from the practical standpoint of Verizon's agreement with P2P company Pando Networks, which will be involved in distributing NBC television shows next month. So the network efficiencies will accrue to legal P2P content, not to downloads from The Pirate Bay.
400%? (Score:5, Insightful)
Re:400%? (Score:4, Informative)
Sloppy, but we can understand what they were trying to say.
Re: (Score:2, Interesting)
Interestingly this mistake doesn't happen with small changes like 10 or 20 percent. But as soon as something doubles i
Re: (Score:2)
Re:400%? (Score:4, Informative)
Re:400%? (Score:5, Funny)
Re: (Score:2)
Re: (Score:2)
Re:400%? (Score:5, Insightful)
The problem that leaps to my mind is that either you're going to have to collect a huge chunk of routing information so your client can figure out which peers are "close" to you, or a third party is going to have to manage the peering...Neither one of those thrills me, especially since an ISP is pushing the technology, which would make them the obvious third party.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re:400%? (Score:5, Insightful)
"Using the P4P protocol, those same files took an average of 0.89 hops"
How do you possibly get an average of LESS than one hop, unless you're getting the file from yourself?
Re:400%? (Score:5, Funny)
Easy! They ran it in simulation, using VMware. Have you ever used VMware? It's an amazing tool that makes an excellent platform for simulations and prototypes, especially when you need to know exactly how applications will perform in the real world.
Game developers, for example, routinely use VMware sessions. Especially the hard-core, 3D FPS developers.
No, really!
Re: (Score:2)
0.89 hops (Score:3, Insightful)
Usually when people are talking about hops, they are referring to routers. The only way you would not go through a router is if you and the source were in the same LAN. If you get an IP address from your ISP and it is one of the private ones (ie. 192.168.0.0/16) then you will likely have to go through a NAT machine before you will be able to see anyone. If your IP address is a publicly route-able address
Re: (Score:2)
Re: (Score:2)
Re:400%? (Score:5, Informative)
So in the field testt we saw data transmission distance drop from an average of 5.5 'hops' to 0.89 'hops'. This happens because P4P provides network mapping information, allowing the p2p network to encourage localized data transfers. Generic p2p moved only 6.27% of data within a metro area, while p4p intelligence resulted in 57.98% same-metro area data transfer. Thus deliveries are both faster and cheaper.
Re: (Score:3, Informative)
That's an improvement, but if there's information about the structure of the ISP's network you can connect people within their network much more efficiently. For example, Verizon Internet has customers all over the US, Japan, Europe, etc., and it's better to connect people with (for example) the New York metro area to each other first, and avoid moving data through tra
Re: (Score:2, Funny)
Re: (Score:2)
What information are we talking about? (Score:4, Interesting)
Network topology isn't & can't be a secret...
Re: (Score:2, Informative)
How much capacity a device has, how many links it has, how much it might cost a carrier to use those links. How much capacity the switching devices in that network have, what firewall/filtering might be in place. Where the devices are phyiscally located.
There's a lot more to a network that just IP Addresses.
Re: (Score:2)
Re: (Score:3, Informative)
But wouldn't a protocol that learns and adjusts to the number of hops be nearly as efficient? If preferential treatment were given to connections with fewer hops and the same subnet I bet they'd see similar improvements.
Re: (Score:2)
www.maxmind.com
If your project is open source their database is free.
Re:What information are we talking about? (Score:5, Informative)
Your traceroute program doesn't tell you when your traffic is being routed four hops through a tunnel to cut down on visible hops and to save space in the ISP's main routing table. Without the routing tables at hand you don't know the chances of being routed through your usual preferred route and through a backup route kept in case of congestion. Nothing from the customer end shows where companies like Level 3 and Internap have three or four layers of physical switches with VLANs piled on top between any two routers. Nothing tells you when you're in a star build-out of ten mid-sized cities that all go to the same NOC vs. when you're being mesh routed over lowest latency-weight round robin, although you might guess by statistical analysis and mesh routing of commercial ISP traffic outside the main NAPs is getting more and more rare.
There's a lot you can easily deduce, especially if your ISP uses honest and informative PTR records. There's still much that an ISP can do that you'll never, ever know about.
I worked for one ISP where we had 5 Internet connections in four cities to three carriers, but we served 25 cities with them. We had point-to-point lines from our dial-in equipment back to our public-facing NOCs. We had a further 18 or so cities served by having the lines back-hauled from those towns to our dial-in equipment. We had about 12k dialup customers and a few hundred DS1, fractional DS1, frame relay, and DSL customers. Everyone's traffic went through one of two main NOCs on a good day, and their mail, DNS, AAA, and the company's web site traffic never touched the public Internet unless we were routing around trouble. In a couple of places we even put RADIUS slaves and DNS caching servers right in the POP.
I worked for another that served over 40k dial-up and wireless customers by the time they sold. We had what we called "island POPs". Each local calling area we served had dial-in equipment and a public-facing 'Net connection. Authentication, Authorization, and Accounting, DNS, Mail, and the ISP's website traffic all flowed over the public Internet except in the two towns we had actual NOCs. There were tunnels set up between routers that made traffic from the remote sites to the NOCs look like local traffic on traceroute, but that was mainly for our ease of routing and to be able to redirect people to the internal notification site when they needed to pay their late bills. We (I, actually) also set up L2TP so that we could use dial-up pools from companies like CISP who would encapsulate a dial-in session over IP, authenticate it against our RADIUS, and then allow the user to surf from their network. We paid per average used port per month to let someone else handle the customer's net connection while we handled marketing, billing, and support.
The first ISP I worked for had lines to four different carriers in four different NAPs in four different states, lots of point-to-point lines for POPs, and a high-speed wireless (4-7 MBps, depending on weather, flocks of birds, and such) link across a major river to tie together two NOCs in two states. Either NOC could route all of the traffic for all the dozens of small towns in both states as long as one of our four main connections and that wireless stayed up (and all the point-to-point ones did, too). If the wireless went down, the two halves of the network could still talk, but over the public Internet. That one got to about 10k customers before it was sold.
At any of those ISPs, I couldn't tell you exactly who was going to be able to get online or where they were going to be able to get to without my status monitoring systems. On one, all the customers could get online even without the ISP having access to the Internet, but they could only see resources hosted at the ISP. Yet that one might drop five towns from a single cable break. Another one might keep 10k people offline due to a routing issue at a tier-1 NAP, but everyone else was okay. However, if that one's NOC went offline, anyone surfing in other
Re: (Score:2)
Re: (Score:3, Interesting)
They simply do not tell, and there is no established protocol to get that information reliable. This p4p would give this information in a way usable to p2p applications.
One disadvantage of p4p is that not everyone will be equal according to p4p. It might reason that all Americans can b
Re: (Score:2)
Thus all peer-to-peer software, regardless of type or legality, could be done more efficiently.
"legal" content? (Score:2)
Is it just that I'm naive ... (Score:4, Insightful)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
So.... (Score:2)
Re: (Score:2)
a) view it WITH commercial breaks every 5 minutes (or worse, since it's now on the interwebs, it might also contain a lot of Cialis and Viagra ads)
b) use it only on the computer you downloaded it to
c) be unable to fast forward (or backward) without restarting a commercial
This will off course add to the revenue and on the other hand turn people off the format so they'll go back to get it from TPB.
Re: (Score:2)
Re: (Score:2, Insightful)
New math (Score:5, Interesting)
I honestly can't figure out where "reduce by 400%" came from. They say the average hops were reduced from 5.5 hops to 0.89 hops, which is either 84% if you're not an idiot or 616% if you are. So I'm really quite confused here. Go figure.
Re: (Score:3, Funny)
Is it possible that pixies and angel farts are carrying packets between peers in your model?
Re:New math (Score:5, Informative)
5.5 * 0.89 - 0.89 = 4.0050 or 400%
As opposed to:
( 5.5 - 0.89 ) / 5.5 = 84%
Re: (Score:2)
Though as an end-user, I don't care about "hops", I care about download speed. I'd prefer my client connect to the fastest sources, not the closest.
Re: (Score:2)
Though as an end-user, I don't care about "hops", I care about download speed. I'd prefer my client connect to the fastest sources, not the closest.
Closer sources are more likely to be faster. I mean, if you are a cable modem user, you have full bandwidth and low latency to everybody else on your segment, very low latency to everybody else in the neighborhood, slightly higher latency to everybody else in the city on that cable network, and very high latency outside the city. Local peers are faster peers.
Many P2P systems try to take advantage of this, and torrent systems are starting to try to take advantage of this, but the problem is that it's somewh
Re: (Score:2, Funny)
Localizing means less anonymity (Score:4, Insightful)
Re: (Score:2)
Your computer is broadcasting an IP address!
Seriously, if your tinfoil hat is on that tight, I have some "security" software to sell you. P2P isn't anonymous, not the way it's normally implemented. If you actually want anonymous P2P, you need to go to something like Freenet [freenetproject.org].
Re: (Score:2)
So it would be a double edged sword...
innumeracy (Score:4, Informative)
That works out to an average 84% reduction.
Re:innumeracy (Score:5, Funny)
Less than one hop on average? Wow, they must use patented "You downloaded that three months ago, you wanker! Look on your damn file server!" technology.
Re: (Score:2)
Fixed (Score:2)
There, fixed that for you
Alternative NYT article (Score:5, Informative)
Not a bad idea actually (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Interesting)
That would help peers pick between known peers to exchange data with. The problem is that if you're in a large swarm, you'll only know about a small subset of the swarm, and thus almost certainly miss the best peers to connect to. For example, if you're in a swarm with 10,000 peers, and you know about a random 50 peers, you are 99.5% likely not to find out about the closest peer on the fi
Re: (Score:2)
Re: (Score:2)
Verizon actually doesn't suck (Score:5, Interesting)
Re: (Score:2)
Even a residential subscriber can get business FIOS, for about double the monthly fee. It has a static IP, and multiple IPs are available. However, for some obscure reason business FIOS doesn't play well with FIOS TV (which uses the 'Net connection to download video-on-demand and program guide info).
what p2p protocol? (Score:2)
Generally, what matters is acceptance (Score:4, Insightful)
What the list includes? Easy:
1. Encryption
2. Onion routing
For very obvious reasons. And neither of them decreases bandwidth used. Quite the opposite.
Re: (Score:2)
Encryption: useful. Onion routing: not useful, probably even negatively useful (it's got to slow things down).
Increasing efficiency is useful too, because one way or another, the user will end up paying for the bandwidth they use. The sad thing is that right now, you are currently paying for it very indirectly. When the ISPs make it more direct (i.e. tiered pricing) you'll actually feel the market force that makes you care about efficiency. Intelligent people feel it right now (they are able to perceiv
Re: (Score:2)
Re: (Score:2)
Where "buy" and "cheap" isn't limited to monetary concerns but also spans to things like ads spliced into the stream (amount of interruptions and length).
Why shouldn't I be allowed to watch content from all over the world, provi
Geographically isn't what's needed (Score:5, Informative)
How about a BitTorrent client that gives preference to peers on the *same ISP*?
Yeah, less hops and all is great, but if an ISP can keep from having to hand off packets to a backbone, they'll save money and perhaps all the hue and cry over P2P will die down some. I'm sure Comcast would rather contract with UUnet to handle half of the current traffic destined for other ISPs than they do now.
Sort of a 'be nice to the ISPs and they'll be nicer to the users' scenario.
Re: (Score:2)
Re: (Score:3, Interesting)
Going back to
Re: (Score:2)
Newsgroup servers routinely distribute and cache content locally to minimize overall network traffic. They can distribute only the headers of the news feed, and then cache the content after it's been requested and downloaded.
This is a *very* efficient content distribution system, and ironically, it's a system more resistant to takedown notices and the like than BT. (It's virtually impossible to entirel
Re: (Score:2)
Re: (Score:2)
Hey, I've got a study too... (Score:5, Informative)
It would have made Internet broadcasting much more efficient, but it never took off. Why? Because providers never wanted to turn it on, fearing their tubes would get filled with video. So what happened? People broadcast videos anyhow, they just don't use the more efficient Mbone multicasting method.
Furthermore, when I download a video via Bittorrent, there are usually only a few people, whether they have a complete seed or not, who are sending out data. So how local they are doesn't matter. If there are more people connected, usually most people are sending data out at less than 10K, while there is one (or maybe 2) people sending data out at anywhere from 10K to 200K. So usually I wanted to be hooked to them, no matter where they are - I am getting data from them at many multiples of the average person.
I care about speed, not locality. The whole point of the Internet and World Wide Web is locality doesn't matter. Speed is what matters to me. For Verizon however, they would prefer most traffic goes over their own network - that way they don't have to worry about exchanging traffic with other providers and so forth. Another thing is - there is tons of fiber crisscrossing the country and world, we have plenty of inter-LATA bandwidth, the whole problem is with bandwidth from the home to the local Central Office. In a lot of countries, natural monopolies are controlled by the government - I always hear about how inefficient that would be and how backwards it would be, but here we have the "last mile" controlled by monopolies and they have been giving us decades-old technology for decades. In fact, the little attacks by the government have been rolled back, in a reversal of the Bell breakup, AT&T now owns a lot of last mile in this country. Hey, it's a safe monopoly that the capitalists, I mean, shareholders, I mean, investors can get nice fat dividends from in stead of re-investing in bleeding edge capital equipment, so why give people a fast connection to their homes? Better to spend money on lawyers fighting public wifi and the like, or commissars and think tanks to brag about how efficient capitalism is in the US of A in 2008.
Re: (Score:2)
The investors in AT&T have lost about 10% of their value in the last year.
They never recovered from 2001 are still at about 60% of their value then.
This is true for many large corporations today.
The executive class is looting and pillaging corporations at the expense of
a) the workers (1 executive pay == 6000 $40k workers)
b) the investors (see stock performance above-- think about adding $155 mill in profits that went to one man who took Home depot into the toile
Re: (Score:2)
Yeah, sure, right (Score:2)
This is why any "massive improvement" on this aspect makes me skeptical. We all know the reason they want to tie it to local is to save bandwidth costs using only their own uploaders basically which would slow speeds down astronomically. Overseas hosts that can do 300KB/s or more on an upload vs a local that can do a cap of 40KB/s. You decide.
ASN matching (Score:2)
Only work if they open the topology data... (Score:4, Informative)
Most true p2p systems use something called a Distributed Hash Table (DHT) [wikipedia.org] to store and search for metadata such as file location and file metadata. Examples are Pastry [wikipedia.org], Chord [wikipedia.org], and (my favorite) Kademlia [wikipedia.org]. These systems index data by ids which are generally a hash (MD5 or SHA1) of the data.
Without going into the details of the algorithms, the search process exploits the topology of the DHT, which becomes something called an "overlay network" [wikipedia.org]. This lets you efficiently search millions of nodes for the IDs you're interested in in seconds, but it doesn't guarantee the nodes you find will be anywhere near you in physical or network topology space.
The trick some of us are playing with is including topology data in our DHT structure and/or search, to weigh the search to nodes which happen to be close in network topology space.
What they are likely doing is something along these lines, since they have the real topology instead of what we can map using tools like tracert.
If they really want to help p2p, then they would expose this topology information to us p2p developers, and let us use it to make all our applications better. What they're likely planning is pushing their own p2p, which will be faster and less stressful on their internal network (by avoiding peering point traversal at all costs, which is when bandwidth actually costs THEM). The problem is their p2p will likely include other less desired features, like RIAA/MPAA friendly logging and DRM, and then they'll have a plausible reason to start degrading other p2p systems which aren't as friendly by their metrics, such as distributing content they don't control or can't monetize... Then again, maybe I'm just a cynic...
Re: (Score:3, Informative)
P4P isn't a p2p network. P4P is an open standard that can be implemented by any ISP and any p2p network, and which has been tested so far on BitTorrent (protocol, not company) and Pando software, and the Verizon and Telefonica networks. Participants include all of the major P2P companies and ma
The awful routing performance of p2p (Score:4, Insightful)
This has been my main criticism of "p2p" user-level networking for years. The selection of "peers" has no clue about network structure. The routing performance is just awful. Finally, someone is doing something about it.
One problem is that, from an endpoint perspective, it's tough to extract network topology and bandwidth. Hop count is only moderately useful. But there are a few tricks one can use.
There are several basic numbers of interest - bandwidth, delay ("lag"), hops,"bottleneck points" and commercial boundary crossings. Each of these can be measured.
Delay, or lag, is the easiest to measure. A few pings and you've got it.
With bittorrent, you're not committed to staying with a peer for an entire download. So you can observe the bandwidth of the peers you're talking to and preferentially use the higher bandwidth ones. You really have to transmit for a while to get a solid bandwidth number, especially since Comcast introduced "Boost" quality of service, which increases bandwidth allocation for a few seconds on demand, then reduces it.
If you do a traceroute, you'll usually observe that many hops show low lag (those are usually hops within a single data center) while others show higher lag. The number of high-lag hops is the number of "bottleneck points" in the path.
Commercial boundary crossings occur then packets cross from one ISP to another at a peering point. Users don't notice this much, but carriers are very interested in minimizing that traffic. Converting IP addresses to autonomous system numbers, as someone mentioned, can tell you when you're crossing a boundary.
So it's possible to collect enough data to do intelligent routing without much help from the network provider. What to do with that data is a separate question, but a solveable one.
obvious (Score:3, Informative)
Re:P2P - P4P? (Score:5, Insightful)
Re:P2P - P4P? (Score:5, Funny)
Personally I'm waiting for the next binary progression, Peer Ate Peer, or P8P. I'm not sure what it will do, but I'll bring popcorn to watch...
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Interesting)
tragedy of the commons, Oh Hamlet (Score:3, Insightful)
THis may turn out to be a classic economics case of the tragedy of the commons.
let's say that if I use P4P that the total path length (i.e. measured in router hops, or cable value, not distance) traversed by my data is less thereby not utilizing as much network resources. However suppose that by voluntarily restricting myself to nearby peers that my download time increases by a factor of 50% (just to make something up). Personally it costs me no more if I use P2P while e
Re: (Score:2)
Re: (Score:2)
Re: (Score:2, Funny)
Re: (Score:3, Informative)
In the case there are not enough peers to fill your tube it will work the same as P2P
This is true, but if 80% of your P2P bandwidth is going within your ISP's local network, then only 20% is going out their interconnect links, which is better than 95% that would happen normally. And by better, I mean better for the ISP. If you have less traffic going out their interconnect links, that is more available bandwidth for other non-P2P traffic. I think their goal is to allow P2P to happen with having the minimal impact on the non-P2P customers.
Re: (Score:3, Insightful)
Re: (Score:2)
Re:P2P - P4P? (Score:4, Funny)
mmmm... Beer.
Re: (Score:2, Funny)
Re: (Score:2)
The RIAA perhaps?
Re: (Score:2)
Re: (Score:2)
Arr, matey, it ain't be making ME nervous! Only thing that be makin' ME nervous is when me blunderbuss is empty and me sword breaks and I drop me knife 'caus I'm full o' rum and they make me walk the plank and keel haul me! Nothin' else makes me nervous.
What's all this bloody "P2P" nonsense anyway, ye damned landlubbers? AAAARRR!!!!