The MP3: A History Of Innovation And Betrayal
"I don't like the title 'The Father of MP3,'" says Karlheinz Brandenburg. But he kinda is. "Certainly I was involved all the time from basic research [to] getting it into the market."
Brandenburg was part of the group that gave the MP3 its name. The Moving Picture Experts Group (MPEG) lent its name to the process of digital encoding by which audio and video is compressed into a file small enough to be transferred easily. That process — MPEG Audio Layer III — and the resulting file — the MP3 — is ubiquitous today. But the development wasn't simple, and its outcome wasn't inevitable.
The story of the MP3 is the story of how intellectual property became the commodity over which the Internet's greatest wars would be fought, and also how the work that goes into innovating can be forgotten in the face of a technology's rapid spread. Google "history of the MP3" today, and you'll find two options, neither satisfying: brief timelines that privilege the user experience over the process of invention and relentlessly technical, acronym-studded descriptions of the differences between various algorithms.
Today, Brandenburg is a professor at Ilmenau Technical University at the Fraunhofer Institute for Digital Media Technology – the same organization that funded his efforts as part of the MPEG. Joel Rose spoke with him about the story of the MP3, and we jumped at the opportunity to get the professor to walk us through the history of the format. As he told Joel, that story begins almost three decades ago and ends in 1999, at the dawn of the file-sharing network.
1982: Brandenburg's involvement in the process begins with a challenge: his PhD thesis advisor wants to patent a process of transferring data that, at the time, is considered impossible. So he asks Brandenburg for help.
Karlheinz Brandenburg. Courtesy of the Ilmenau Technical University hide caption
"My thesis advisor for my PhD work thought of what to do with digital phone lines and composed and submitted a patent application that ISDN digital phone lines should be used to transmit music and the patent examiner told him, 'This is impossible, we can't patent impossible things.'"
1986: Real progress only arrived as technology advanced.
"I always say the real progress started in 1986. We had better computers, better probabilities to experiment with music on the computers — in fact it was in early '86 that at some I point I had the idea to do things in a different way from others in the field had done and, in fact, that idea went into the first patent in 1986, and still can be found in MP3 and ISDN."
The process still took quite a bit of time to perfect. "It's not that easy [to explain]," says Brandenburg. Here comes the technical bit: Early encoding processes were designed to filter a signal into layers of sound which could each be saved or discarded depending on relative significance. But the system was "very structured and inflexible." So a new system was proposed that would exploit the limitations of human hearing.
"Others had the idea that we really should use psycho-acoustic use and knowledge about what we hear, what we don't hear — so-called "masking," that sometimes we hear something, sometimes it's masked by other sounds. With this system I gained, on the one hand, efficiency — being able to reduce bitrates for the music, the compressed music — on the other hand, I got the flexibility to adapt better to the properties of the human auditory system."
1988: An international team is convened after the International Organization for Standardization (ISO) calls for standards in audio encoding. That team was called the Moving Picture Experts Group, abbreviated MPEG. Brandenburg credits a colleague, Leonardo Chiariglione, with "the vision that such standards could be useful."
"In a standards group, people have to get consensus on a good idea for a system. ... That was the original starting point for work of MPEGs. And at that time, everybody working on the subject around the world thought, 'Okay, that's our chance, let's contribute to the standard.'
"The idea of Leonardo Chiariglione and other people was originally to bring video to CD-Rom. That was the first application — but the list of applications at that time already listed everything else we now use compressed video and audio for ... So we had the audio subgroup within the motion pictures group ... In the end we all together had a compromise worked out which had different modes, so-called Layer I, Layer II, Layer III ... And most of our ideas went into the modes of compression in MPEG audio ... which was the most complex one and the one giving best quality at low bitrates — that was called Layer III."
MPEG Audio Layer III hits a snag.
"In 1988 when I thought, 'OK, that's near perfect, what we've done,' I heard about this song and then heard it myself — it's the acapella version [of Suzanne Vega's 'Tom's Diner']. The CD is Solitude Standing. The way it's recorded — with Suzanne Vega in the middle and little bit of ambience and no other instruments — is really a worst case for the system as we had it in 1988. Everything else sounded quite OK, and Suzanne Vega's voice was destroyed."
More collaboration followed. Brandenburg worked with Jim Johnston of AT&T on different psycho-acoustic models and data coding, to try and fix the encoding system and save Vega's voice.
"Finally [we] perfected the system and then Suzanne Vega's voice was easy, but it gave us quite some work to have her voice in full fidelity ... I think over time I have listened to the song 500 or 1,000 times. In fact, I still like it. That was the good part about it ... Interesting thing, later on I met Suzanne Vega and I heard her singing this song in a live performance. It was really astonishing — [it] was exactly like on the CD. It was a like a curtain opening because I knew all the little details in how she sings it, and she still does it the same way."
1992: MP3 is a reality. The ISO includes it as one of a number of possibilities for encoding audio. But for some time, Brandenburg says, it didn't catch on.
"In the early days most people, especially people at the big consumer electronics companies, thought that Layer II is a good compromise. Layer III is too complicated to be of real use. So the first run of applications went to the Layer II camp."
1993 - 1994: The system is in place. Where are the users?
"We had to look for different ways to market our technology. We had first companies like Telos Systems in Cleveland, Ohio ... they were the first to use ... Layer III to send audio over ISDN from some off-site recording site to the studio. So in some sense it was the original idea of sending music over phone lines.
"In 1994, we had some internal strategy meeting in Erlangen, and we discussed what to do and somebody said, 'We have a window of opportunity to let MPEG Layer III become the internet audio standard,' but I don't think we had an idea of what that really meant."
1995: The MP3's birthday.
"We had a time in 1994 – 1995 where we really identified the Internet as a big application area for Layer III. We needed a file extension, so we have some birthday — on 14th of July in '95 — we decided in Erlangen to use the file extension "dot m-p-3" for all our software encoding or decoding: .MP3. It really has a birthday in July."
To answer your question: Yes. In 2005, there was a birthday party for the MP3.
"[After] 10 years, we did a party, yes. Not every year, but from time to time. We still have the internal e-mail around where we announced the file extension internally at Fraunhofer."
The Internet emerges as the MP3's true home, and the source of what Fraunhofer hoped would be a workable business model. Here's how that business model would work: encoding tools – to be used by large companies – would be expensive. But decoding tools – which would take the encoded digital files made using the MPEG Audio Layer III process and turn them back into audio that people could hear – would be available cheaply.
"Winamp was one of the early software which got widely distributed. The knowledge how to do MP3 decoding was out there in the open. We helped people to get that knowledge. There were still patent rights; we early on decided for a business model where we wouldn't go after freeware authors. The people doing Winamp at some point paid a patent fee."
1997: As MPEG's encoding system got out into the world, it began to spark the imagination of users who definitely weren't thinking of Fraunhofer's bottom line. Before its second birthday, the MP3 was setting fires that its creators couldn't put out. The story of how the technology was hijacked and adapted for widespread consumers contains not only the roots of the war that the music industry would later wage over the tiny, compressed, user-friendly files, but also echoes of some of the very ideas that war was fought over: intellectual property, copyright, technology, theft, control and the free distribution of ideas and products that had taken years to realize.
"There were more and more people using this technology to store music on their hard discs. The idea was [originally] that encoders would be much more expensive. ... In, I think it was '97, some Australian student bought professional grade — from our point of view — encoding software for MP3 from a small company in Germany. He paid with a stolen credit card number from Taiwan. He looked at the software, found that we had used some Microsoft internal application programming interface ... racked everything up into an archive and wired some Swedish side, [and] put that to a U.S. university FTP site together with a read-me file saying, 'This is freeware thanks to Fraunhofer.'
"He gave away our business model. We were completely not amused. We tried to hunt him down. We told everybody, 'This is stolen software so don't distribute it,' but still the business model to have expensive encoders and cheap decoders [was] done. From that time on, we reduced the cost for encoders. There was a company, Music Match, which allowed people early on to take a CD's music, read it into the computer and then have their own music jukebox on that. And they were legal, they paid for the patent fees so that was fine.
"When we found out that people used our technology to do unauthorized distribution of music over the Net — that was not our intention, very clearly. I have to say, I don't say that everything the music industry does is correct or good, but still I think we should have respect for the work of the artists and everybody involved and it's only fair that they get paid for it.
"It was in '97 when I got the impression that the avalanche was rolling and no one could stop it anymore. But even then I still sometimes have the feeling like is this all a dream or is it real, so it's clearly beyond the dreams of earlier times."
For the music industry, it would be a nightmare, but one of those nightmares where – when it starts — you have no idea how bad things are going to get.
"We tried to tell the people from music industry early on, and we tried to discuss possibilities how to react to this ... The idea was that the music industry wouldn't just be able to go on, they would have to adapt to the situation as well, and if we now look back these 15 years we have to say they finally did but it was too slow and some strategic errors in there."
1999 – 2000: Napster is invented in 1999, and even in its slow early days, allows the transfer of digital files from the hard drive of one user to another, with no regard to geography. Another group was convened in 1999 and 2000 to define methods for "secure, legal distribution of music over the Internet," Brandenburg says.
"My advice was that they should shoot for a technical standard to get interoperability for all these upcoming services and MP3 players, music players and so on.
"And ... that hasn't changed, very clearly if we don't reach interoperability for a secured format, then the only surviving format will be format without copy protection and that is what happened in the end."
In the end, convenience won out.
"That was very clear. We told them usability, convenience is the most important factor.
"At some point in the late '90s, MP3 was technically the best system out there and at the same time it was accessible to everybody. That made it early technical standard and nowadays the one format which can be played [on] all the devices who play music and that of course means that everyone, media-wise has to support that format as well.
"I think at some point we might get to completely different formats, without compression but that's still some years out."