TYPICAL. You fight tooth and nail to bury a technology, arrest almost 3000 people using it for nefarious purposes, and then along comes some smart alec who says you got it all wrong. The same technology could make your fortune and have huge benefits for consumers, too. And on this occasion that smart alec is no minnow.
Music and film executives’ minds are probably still recovering from the announcement in the UK earlier this month that the BBC is planning a pilot project to test a file-sharing system to distribute its TV programmes to viewers over the internet. Until now the entertainment industry has viewed this sort of system as the spawn of the devil.
But a growing number of projects demonstrate that these systems could benefit everyone from TV broadcasters, computer-game creators and software developers who sell their wares online, to academics who want to share their digital archives.
Advertisement
Over the next three months, 1000 hand-picked individuals from across the UK will test the BBC’s new Interactive Media Player (iMP) software, which allows people with a broadband internet connection to download TV programmes and view them for up to a week after they are broadcast. The iMP is based on the reviled peer-to-peer (P2P) technology, which has caused ructions in the entertainment business because it has been used to distribute video and music files that infringe copyright.
P2P is an umbrella term for technologies that allow users to share files over computer networks. For example, users of the first incarnation of Napster checked a central server that logged where each file was, and then connected to an individual users’ machine to download that file. Other systems, such as Gnutella and Kazaa, have no central server, so instead sophisticated search algorithms explore the network to locate files.
Sentiments against P2P have run high in the entertainment industry ever since Napster gained notoriety as the number one place to download pirated music. After a lengthy legal battle, it was forced to close down in July 2001 though it has now re-emerged as a legal music-download site.
When the plug was pulled on Napster, file sharers switched to less controllable networks, such as Kazaa and Gnutella. Closing them down is practically impossible. And the piracy problem kept growing because, unlike with Napster, no single company could be held responsible for controlling access to the networks’ content.
In July 2002, 19 US congressmen wrote to the Department of Justice voicing their concerns. “Over the past few years, we have witnessed a staggering increase in the amount of intellectual property pirated through peer-to-peer systems,” they said. “We urge you to prosecute operators of peer-to-peer systems [and] individuals who intentionally allow mass copying from their computer over peer-to-peer networks.”
Comments like these, together with pressure from the Motion Picture Association of America, an organisation representing the film industry, and the Recording Industry Association of America, which represents the record industry, culminated in March this year in draft US legislation, known as the Pirate Act, which suggests cracking down on file sharers by setting the penalty for copyright infringement at several years in jail.
But this attitude to P2P networks may soon have to be revised. A few ingenious projects are proving that, far from being uncontrollable scourges of the online world, P2P networks are one of the most efficient, cost-effective ways of distributing large amounts of data. And the BBC’s iMP is at the forefront. “During the trial we’re going to try out several different methods of data distribution to find out which works best,” says Ben Lavender, chief architect of the project.
A 30-minute TV programme compresses into anything from a 66 to a 234-megabyte file, depending on the compression system used. If more than a handful of people tried to download the same programme simultaneously, the usual method of making the clips available from a central server would require an enormous amount of bandwidth, at huge expense. So Lavender decided to use software that distributes the downloading process using a technique known as swarming.
Files are chopped up into pieces. When a user requests a file, a central directory tells his computer where to find each piece, whether on the central server, or from peers – other computers that have already downloaded some or all of the pieces – or a combination of the these. Where each piece is taken from will depend on the load each machine is dealing with at the time of the request. The software will download the least prevalent piece on the network first, thus making itself more useful to its peers (see Graphic).
Lavender and his team bolted together an existing P2P file-sharing system with off-the-shelf digital rights-management software – that gives the BBC control over how long a programme can be viewed – to create the first incarnation of iMP. Because of copyright issues, access to content will be restricted to computers with UK-based internet addresses.
The P2P technology he used is being kept under wraps for commercial reasons, but he says it is based on the principles used by one of the latest stars in the P2P galaxy, BitTorrent. It is the brainchild of 28-year-old computer programmer Bram Cohen, who had become frustrated by existing P2P networks because many people would download files but then not make them available to others. Besides being selfish, it disrupted the balance of the network and meant that certain servers would become overloaded because they contained popular files. So he set about designing a network that rewarded those that put most into it by giving them the most back.
The result was a system that dramatically speeded up downloading times. The download method built into the BitTorrent client asks for the least-copied parts of the file, so a new downloader starts by picking up a chunk that is wanted elsewhere in the network and in this way is pulled into the network. As a result, downloading files using BitTorrent has the odd effect of starting off slowly and getting faster as you pick up more chunks.
Lavender says the BBC’s key motivation for adopting this sort of system is that it uses other people’s bandwidth to spread the costs of distributing the files. This means a large number of people can get hold of the same file much more quickly, and the pressure is not all piled onto one server. Also, punishing those who don’t work with the network by making their download slower keeps the network running as efficiently as possible.
Another trend has recently emerged. More software is being sold and downloaded online so companies selling their software in this way – who often pay their internet service provider on a per-megabyte-downloaded basis – have experienced soaring bandwidth-usage bills. As a result, several of these firms have recently begun to look at alternative systems to distribute their products. Computer-game publisher Valve – famous for its shoot-’em-up game, Half Life – was particularly keen on BitTorrent. It hired Cohen to help develop a system based on BitTorrent to distribute updates to existing games. The new software is being made available to other games developers so they can sell their games without incurring packaging and retail costs.
Paying for an internet connection fast enough to distribute their content is too expensive for small games developers, says Gabe Newell, Valve’s chief executive. “Using the peer-to-peer model, they can push those replication costs onto their user base,” he says, which users are generally happy to pick up.
BitTorrent also found a fan in Linspire – a company that sells a Linux-based alternative to Microsoft Windows. In March this year, it realised it could save so much money by getting people to download its software, that it halved the cost – to $25 – for anyone who used BitTorrent to download the program. “Most content companies view P2P as evil, but there are phenomenal commercial applications,” says Lindows’ CEO Michael Robertson. “P2P allows us to serve more customers faster than previously possible and at a huge saving, which we are able to pass on in lower prices.”
However, as successful and popular as BitTorrent is, it does not hold all the answers. “BitTorrent is about providing rapid, low-cost collaborative downloads of popular items,” says Ross Anderson of the Computer Laboratory at the University of Cambridge. “That’s fine for popular stuff but doesn’t help with archives of ancient manuscripts in which only a few hundred scholars are interested.”
Anderson is pursuing an alternative called Chord. This uses a tried and tested technique called distributed hash tables. Hash tables apply a simple formula to pieces of data to split them into identifiable groups. “It is simply filing, like having 26 folders, one for each letter of the alphabet. But you put those 26 folders on 26 different machines,” says Frans Kaashoek, of Massachusetts Institute of Technology, co-creator of Chord.
Anderson is working with the University of Cambridge Library to provide distributed storage for its Genizah collection of ancient manuscripts from a Cairo synagogue. They are being digitised, creating very large files. If they were stored in one place, it would be a huge drag on resources, but with a distributed system, the load can be spread across an almost unlimited number of machines.
Kaashoek is using Chord to build an alternative distribution system for Usenet, the online newsgroup system that pre-dates the internet. “The number of sites that can offer its full feed is small because of the bandwidth and storage needed,” Kaashoek says.
It is still early days, but interest is rife. New variations of both BitTorrent and Chord such as PDTP, Circle and DSecure are blurring the distinction between the two groups. Anderson thinks it is just a matter of time before P2P will be used everywhere. “An average bank will currently hold only three to six months of your account history online and the rest it will keep on microfilm. But the bank has enough PCs, if built into a distributed network, to contain all that information.”
Security and access control may seem to be an obvious worry: would you want your banking history spread out on your neighbours’ machines? But Anderson says such concerns are unfounded. “Security is irrelevant,” he says. “If you want to keep data confidential, you encrypt it.”
It is still early days, but the BBC’s tacit endorsement of P2P systems is a watershed for a technology that has been shunned by the establishment for too long. Together with the growth in broadband connections and digital video recorders, the demand for cheap and easy ways to share large amounts of data looks set to rocket. And where the BBC goes others are sure to follow.
