The Pluralist
The Pluralist appeared to be having too much of a good time, smiling, frisking through tracks, and tapping her fingers on the table in time with the music. She was fearless about volume level and didn’t hesitate to ask us to crank it up while scrutinizing certain portions of the tracks. But she found the exercise to be anything but a breeze. “I’m pretty sure about this,” she told us, but quickly added, “sort of.” And then: “Wait, can I hear them again?”
Despite this last-minute waffling, she was very confident about identifying the quality levels of her own track, as well she should have been: She correctly identified all three. With songs she was not familiar with, or with tracks that made her burst out laughing (see the last sentence of the Hipster’s report), she identified only two of the 160Kb/s tracks and a single 320Kb/s track. The final result—six out of 12 correct—doesn’t seem particularly impressive until you consider that this was the highest score of the entire group!
| Score Card | ||||
| Pluralist's Track | Curmudgeon's Track | Hipster's Track | Audiophile's Track | |
| Uncompressed | pass | fail | fail | fail |
| 320kbit/sec VBR | pass | pass | fail | fail |
| 160kbit/sec VBR | pass | fail | pass | pass |
Although ostensibly objective and cool-headed, the curmudgeon was none too happy once the listening tests began. He was initially grumpy, and as the minutes flew by, he became increasingly restless and even hostile at times. He wanted to know, for example, how the test administrator knew that the files were done correctly if only the staff member who labeled the tracks knew which was which.
Nonetheless, he marched forward, and was particularly demanding of A/B testing, comparing portions of his own track with numerous repeats. Apparently, the diligence didn’t pay off: The curmudgeon was unable to correctly identify the quality level of any of the three versions of his own track! In an eerie coincidence, however, he dissed the Hipster’s track much as she dismissed his (“I thought Ani DiFranco got all her anger out already....”), yet correctly identified the quality levels of the three versions of that track. He was also able to identify a 320Kb/s version and an uncompressed version of the other track sets, but that’s all, for a modest total of five out of 12 correct.
When the test results were revealed, the curmudgeon threw a hissy fit, questioning our methodology before sinking into an oppressive quiet behind his keyboard.
| Score Card | ||||
| Pluralist's Track | Curmudgeon's Track | Hipster's Track | Audiophile's Track | |
| Uncompressed | pass | fail | pass | fail |
| 320kbit/sec VBR | fail | fail | pass | pass |
| 160kbit/sec VBR | fail | fail | pass | fail |
Comments are closed on this article
anon
February 09, 2010 at 3:22pm
"I didn't read about this. I heard it."
So rather than providing evidence such as an ABX test, just claim that it sounds better. Funny. And I am waiting for your rebuttal links, rokikiki.
fnordfnord
October 09, 2009 at 9:12am
"but allowing the bit rate to wander upwards during more complex
passages—as variable bit rate encoding does—and throttle down during
quieter"That's not how it works. A loud sine wave takes a lot less data to encode than quiet white noise.
smarter
December 15, 2008 at 7:14am
comparing a record to a cd is like comparing a canvas to photoshop...... simple as that. Continuing beyond the scope of this article, recording an mp3 or wav to vinyl wouldn't exceed the range of human hearing, thus limiting the "feel" of the vinyl.
fnordfnord
October 09, 2009 at 9:05am
Comparing a record to a CD is like comparing a film photograph to a digital photograph, you mean.
nantucketbob
October 22, 2008 at 6:06am
This is a good article. The comment by Chris is revealing in that he is a trained listener, and he has access to professional equipment. I would like to see a test done with quality consumer grade equipment (PC, MP3 players, stereo systems) with professional trained listeners, such as "Chris." I would also like to see a test on professional grade equipment by both trained and untrained listeners, even though the conclusion should be obvious. Trained listeners would include audio engineers as well as musicians (probably conductors). Enjoyable article!
anon
September 07, 2008 at 3:45am
Greg Calbi and you can believe whatever you want. I am not asking you to believe me. I am asking disinterested readers to go to those three links, read, use their brains, and decide for themselves. The quality of the wire does matter. That is not to say that expensive wire will always sound better. That is to say that expensive wire will never sound better. It is to say that many audiophiles are suffering from confirmation bias and post-purchase rationalization.
orlanthrex
September 03, 2008 at 6:28am
I have not met an audiophile who hasn't been an @sshole.
they like all other tech cultists, (Apple devotees, and Volkswagen cultisti just to name a few) are insufferable to be around.
if you actually can hear the difference between them....good for you....you must be a miserable human to be constantly disappointed in life.
Ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn
In his house at R'lyeh dead Cthulhu waits dreaming
ctm
October 10, 2008 at 8:38am
True, my friend. I was thinking of going on a journey to find the perfect stereo that would show me exactly how bad my music sounds. Then I was thinking of simply turning the stereo I currently own WAY UP and bobbing my head to some groovy tunes while I carry on in ignorant bliss. shoot, the bass sounds awesome!
fanfarefan
June 04, 2008 at 5:44am
Actually the scores are not necessarily humiliating, and they are very good when you look at favorite songs only. I would say the test indicates that people can indeed hear the difference, although not every time.
Expected avg random guess score: 4
The actual scores: 3, 6, 5, 5
Avg score: (3+6+5+5) / 4 = 4.75
Favorite songs score: 3+3+0+1 = 74.75, the average score for the four testers, represents 19% more right answers than predicted by chance.
Not bad, and it gets much better when we consider everyone's favorite song only. The single-song scores for the favorites are (3, 3, 0, 1). So the total score for the four favorites is 7, almost double the number of right answers the testers would have been likely to get by guessing.
To figure the expected average random guess, consider that for each song, the testers are being asked to put the three versions in order by quality. For three things, there are six possible orders {ABC, BCA, CAB, ACB, CBA, BAC}. If "ABC" represents all three right, then the six orders give scores of {3, 0, 0, 1, 1, 1} respectively. Random guesses would randomly get one of those scores, for each song. So for each song, the expected avg random guess score is (3+0+0+1+1+1) / 6 = 1. For four songs, the random score would be four times that: 4 x 1 = 4.
anon
May 28, 2008 at 5:53am
There is a lot of nonsense regarding what can and what cannot be heard; the only proper way to do this is with double-blind listening tests. First, use what may be the best MP3 encoder—LAME 3.97. Second, use a program to perform double-blind listening tests--http://wiki.hydrogenaudio.org/index.php?title=ABX.
EarthBoundMisfit
November 17, 2007 at 11:02pm
Registered Linux User #404122
Microsoft has encountered a critical system error and must now shut down. Better get Bill Gate$ on the phone for this one.......
[img]http//i35.photobucket.com/albums/d183/NunofyaBidness/404122.png[/img]A very nice demonstration of what I already knew.
I have a friend who claims he 'can tell' when he hears an mp3...at ANY bitrate.
As a former disc jockey for 2 1/2 years...what many folks do not know...is that music you hear overnights or a lot of programs...is now in mp3 format.
Nicely done.
Glenn Condrey
Gulfport, Mississippi
HWood
June 12, 2007 at 7:50am
This can be a bit misleading and confusing. Anyone trying to make a comparison using headphones should think about a few things. First headphones are not a good speaker to perform a comparison. Lets go back in time a little when Compact Disc came about most people said CD was better than LP's. Most people have never owned or listened to a good Turnable Phono Cartridge setup ever. So how would they even know. Now with that said, most people now are listening to Earbuds, car stereo's, computer speakers, and House audio systems. Which are all very forgiving of the source. Most of those people have never owned or heard a good stereo system (Not Home Theater System). I am not surprised by the results. A very simple but conclusive test could have been done using a good consumer grade CD player, Stereo Receiver and a nice pair of floor standing speakers or a Subwoofer and 2 smaller speakers. I believe the results would be different.
anon
May 28, 2008 at 6:44am
Another myth is that records sound better than CDs:
http://www.theaudiocritic.com/downloads/article_1.pdf
http://www.soundstageav.com/mastersonaudio/20050415.htm
http://www.mastersonaudio.com/audio/20030101.htm
soggybomb
June 01, 2007 at 5:24pm
I have had great success with Creative's Mediasource Audio Converter. Because Creative values audio quality (and because the software is packaged with my X-Fi soundcard) it sounds great after encoding.
transmothra
May 26, 2007 at 11:27pm
All you really need to hear it instantly is a song with cymbals and a piano. Boom! If it's wobbly-sounding, or kind of garbled like it's almost underwater, crank up the bitrate. Nothing to it.
If you don't notice a difference, then it doesn't matter - just don't share your crappy mp3s with me.
~jer
http://jeremyjarratt.com/
SysAnalyst
May 07, 2007 at 12:06pm
For the longest time MPC recommended using Exact Audio Copy and LAME for encoding to MP3. Recently, Will Smith, I believe, mentioned using straight iTunes for ripping. This was a surprise to me as I had been following the EAC/LAME method for my whole CD collection. Why this change in method?
lacalaca
April 30, 2007 at 7:32am
Your test can be misleading. Using a cheap CD-drive to grab music data and make them play via "the finest consumer-level soundcard" as reference is not the real quality-alternative to MP3 listening.
CD sound is compressed anyway (in term of dynamics) and represents a 25-year-old digital technology (which at the moment of its birth had its compromisses: low sample rate, low resolution). A cheap PC-CD-drive will read only a portion of that low-rate music data, and the software tries to interpolate the missing data (which are missing because of the low sample rate AND the poor disc-reading). It sounds like a lossy compression, isn't?
I suggest you a test:
Config A: Your PC with that "finest consumer-level soundcard"
Config B: A really good consumer-level (not an "audiophile") CD-Player (try a Marantz SA 7001 with its built-in headphone output, or use a decent external headphone-amp with reasonable cables).First, drop in two identical original CDs, and compare the uncompressed PC-sound to the Marantz. You don't need to lean forward to hear the difference... And if you switch to MP3 on the PC, that difference will be more clear (watch the trebles, the stereo room-rendering, and the body of the instruments). Use acoustic music and human voice to reveal the unnatural sound of MP3.
And then drop an SACD into the Marantz (the Miles Davis "Kind of Blue" does the job), and compare to the CD-ripped, MP3-compressed PC-sound of the same music... Now, THAT is the real alternative.
rogergraham
May 30, 2007 at 7:12am
"Your test can be misleading. Using a cheap CD-drive to grab music data and make them play via "the finest consumer-level soundcard" as reference is not the real quality-alternative to MP3 listening.
CD sound is compressed anyway (in term of dynamics) and represents a 25-year-old digital technology (which at the moment of its birth had its compromisses: low sample rate, low resolution). A cheap PC-CD-drive will read only a portion of that low-rate music data..."
*cough*bullshit*cough*
Any cheap PC CD drive will retrieve every last bit of every CD, barring bad scratching etc.
How many programs fail to isntall due the CD drive read errors? How much data do people loe due to their drive not reading properly? You guessed it: absolutely fark all.
cris
September 02, 2008 at 11:38pm
Haha. Oh yeah. You're an expert... NOT.
It's not bullshit. You're an ignoramus.
"Any cheap PC CD drive will retrieve every last bit of every CD, barring bad scratching etc."
You see the devil is in the details. Data discs are VERY different to audio discs. When you read a data disc in a pc cd rom, there is a thing called a checksum, and if it doesn't match, the sector is read and re-read until they do (or it fails). Thus a data disc will read perfectly, or not at all.
However, ripping an audio CD is a totally different task. It suffers from something called "jitter", and it DEFINATELY affects the quality of the sound. A trained ear and top gear is needed to hear it (the gear used in this test is ANYTHING but top shelf). Look jitter up. Have a look at the Plextor mastering drives, and their different error rates.
And you should stop pretending to knoweverything and realise you have a lot to learn. Believe me. I've been an Audio Engineer for over a decade now. The equipment used here will not reveal the difference. It is simply not good enough. They are dynamic headphones FFS, so you are looking at a good 20dB roll of from about 15kHz at least. So you are losing all the fine detailed high frequency information where you REALLY CAN TELL the difference between compressed tracks and uncompressed tracks.
All this test proves is the equipment they used is unable to resolve the subtle differences between the tracks.
snowcarver75
April 16, 2009 at 12:38pm
First of all, all CDs are digital. Whether you call a CD audio or data is just a matter of labeling with a few flags set here and there to allow devices to recgnize the CD type. What you don't seem to understand is that in terms of audio extraction, the Red Book standard does not require block-accurate addressing when the CD drive is in audio mode. This provides a certain level of tolerance so that the user experience is not terribly interrupted when the device has a problem reading the CD. So if you press the play button the CD drive, you may or may not get the correct bits, resulting in those little clicks.
Data extraction is subject to Yellow Book standard and requires software/driver to automatically perform data correction. As an engineer, you should know that data is data, and audio is nothing than data that is packaged in a certain format. When an audio CD is extracted, especially to the uncompressed WAV format, either in software or via drivers performing overlapping reads, incorrect bits (seek jitters) are eliminated. In other words, the same audio CD when opened/browsed/accessed by the computer in data mode will behave as if it were a data CD (thus you can always extract audio CDs as ISO's).
Like the original author stated, unless there is a bug, or the CD is unreadable, audio extraction is guaranteed to be 100% accurate.
Stick to audio hardware next time okay?
rokikiki
December 13, 2009 at 7:04pm
Snowcarver, it would seem your post contradicts its purpose (proving that the quality of the CD reader is moot during extraction). Yeah, the computer does have all the time in the world to perform correction but it has no choice but to read whatever data comes from the lightweight, vibrating drive it has installed in its bay. Checksums take care of your *computer* data (such as text documents), as the Audio Engineer wrote. If there were no checksums, your text files might come with e440r5; since there _are_ checksums, the block will be re-read until it comes out right or until the machine gives up. With an audio CD, the system just plays the CD while glossing over the errors.
The levels of tolerance needed for audio CDs you write about are real, they are a "necessary evil", but can be reduced, for instance, by putting a big brick on a cheap CD player. Which makes it heavier and less prone to vibration. So there is less jitter to be taken care of by the error correction mechanisms. Of course better CD drives resort to other solutions than bricks.
I believe there might be many other reasons why the setup used for testing is unsuitable for showing the quality of non-compressed music, some of them were mentioned before. Among them there is the DAC stage, which is pathetic in a soundcard (sorry, gamers, I know you _really_ heard that explosion on the right side). There is the cheap preamp, the hum induced by other components and so on.
I suggest visiting someone with a decent stereo (not home theatre) system and listening to various sources (LP, SACD, CD, mp3). The differences in quality and listener experience are staggering. Accept this from a mere mortal (I'm not an audiophile yet).
Speaking of cables and vinyl records: I have neither expensive cables nore a record player but am sorry to report, that - no matter how many rebuffals get published, good cables and vinyl records _do_ sound better than bad cables and CD's. I didn't read about this. I heard it.
Peace, R, from Hungary
anubisjr
April 23, 2007 at 3:29am
...if you want a fair test, use either the LAME or Fraunhauer codecs. iTunes uses an inferior mp3 encoder so their AAC will sound better by comparison. Don't believe me? Try it for yourself using both iTunes and Music Match Jukebox. The results are startling, even at 320kbps.
sackynut
April 22, 2007 at 9:30pm
Personally I rip all my music in apple lossless. It lets me have a lossless copy of my music in the case I lose the physical CD, plus i have plenty of HD space.
For me, I need SUPER high quality headphones, and then songs that I can tell. Some songs (even with good headphones) I cant tell the difference. But others I can. I guess it depends on the song, quality of recording and quality of headphones (or speakers).
This is an awesome article and I was actually looking for my old MaxPC issue with an article similar to this, I was bummed I didnt find it. but now i found it yay!
Enlightenment
April 20, 2007 at 8:01pm
I started converting my 500+ CD collection back in 1998 with L.A.M.E., way before most of the iPod punks ever heard of MP3 or any compressed audio. I had a necessity to do it because it was a major hassel to haul hundreds of CD's to DJ parties.
Back in those days when you couldn't buy 100's of GB hard drives for next to nothing so I had to experiment with what was reasonable. Even 9 years ago, I knew that 128Kbps CBR (constant bit rate) MP3 didn't sound good enough. I really wanted to use 320Kbps, but hard drives weren't large enough to hold my entire collection at that rate, so I used the next best thing of VBR (variable bit rate), over time I re-ripped my collection multiple times, and years ago finally said enough is enough, and just went to 320Kbps CBR MP3.
I have tried some comparison tests of my own over the years, and come to the conclusion that if I picked a lower bit rate, it might sound ok for a lot of my songs, but eventually I would find a song that didn't sound very good. The solution was to just encode everything at the highest bit rate of 320Kbps and just not worry about the storage that it used. Now that I can buy 1TB drives, even 320Kbps isn't a big deal, heck I could rip everything as WAV files and still not fill a 1TB drive.
Audio tests are hard to do, but I can tell you for a fact when I hear a really badly encoded song off the internet, oh my it hurts to hear it. Good examples of songs to find for the tests are ones that have: cymbols, snare drums, anything that rings, anything that has very high frequencies, anything that has stereo movement of sound, wide dynamics of volume in a short amount of time. When I listen to the music and not trying to do a test, sometimes I hear weird encoding problems and they are usually associated with those types of sound.
maxhodges
April 20, 2007 at 6:31pm
thanks. could have been done better, but wasn't done terribly either.
in the end some people just CAN'T tell a difference between compression for some types of music, while others CAN tell a difference for some types of music, but not all types of music.
pretty much confirms things I've already known.
for most people, moderately compressed MP3s are 'good enough', and that's why they exist: because there isn't a lot of market pressure driving higher bitrates. The unfortunately thing is that very avid listeners who buy things like $1,600 amplifiers and $350 speaker wire (like me), can't enjoy the conveniences of digital downloads because their bitrates are determined by the tastes of the lowest-common-denominator mass-market consumer.
garbage in, garbage out
another thing is that a lot of the music they were listening to was probably already damaged pretty badly by the time it got on CD. sound engineers these days really mutilate the music by compressing dynamic range and trying to get the signal as hot as possible--so that it stands out the radio--but clip transients in the process. etc.my own test CD is Miles Davis "Kind of Blue"--but that wouldn't work for other people unless they've listened to it enough to know the duration of every breath. reminds me: I went into a Home Entertainment once (think they got bought up by Tweeter) to audition some speakers, and when I put my disc on I instantly noticed that the speakers were wired backwards. I prodded the salesperson a bit, but he didn't seem very willing to straighten things out, so I went and dropped my $3000 budget somewhere else.
anon
May 28, 2008 at 6:22am
Two myths are that expensive amplifiers and wires sound better than inexpensive amplifiers and wires: http://bruce.coppola.name/audio/Amp_Sound.pdf
http://bruce.coppola.name/audio/Greenhill.pdf
http://www.theaudiocritic.com/
rokikiki
December 13, 2009 at 7:03pm
• expensive houses are larger than inexpensive ones
• there's more leg room in business class
• porsche is more reliable than yugo
• an euro is worth more than a yen
We're waiting for your rebuffal links for these ones.
cris
September 02, 2008 at 11:42pm
You should tell that to Greg Calbi. I'd believe him over you anyday. The quality of the wire does matter. That's not to say that EXPENSIVE wire will always sound better. Its to say QUALITY wire will sound better.
livewyre
April 20, 2007 at 5:04pm
"Many people agree that a Vorbis file remains indistinguishable from the original source material at q5 (~160kbps), whereas for MP3 a bit-rate of 192kbps is required to reach the same standard."
Quote from my website discussing optimum bit-rates for compressed audio (Vorbis being the variable-rate). For more please see:
http://www.cassette2cd.co.uk/mp3.htm
Good study, though after invesigating this myself, I am not at all surpised to 'hear' your results. I disagree in principle with the view that monitors (even huge theatre speakers) would reveal the imperfections, but that in itself would make another interesting experiment. Just because a piece of equipment delivers massive volume doesn't naturally make it more accurate when it comes to faithful reproduction does it?
anon
May 28, 2008 at 6:04am
Perhaps about 175 kbps: http://wiki.hydrogenaudio.org/index.php?title=Lame_Compiles#Recommended_encoder_compiles_and_source_code.
azogue
April 20, 2007 at 10:47am
As a professional theatrical sound designer and engineer, I am regularly asked to accept mp3s from directors and dancers.
Maybe not on headphones, but on LOUD concert and theatrical sound systems the compression can certainly be identified, even with variable bit rates.
Additonally, editing is not accurate with mp3s.
Casual listening, in the car or on headphones...fine.
But for any other use give me 24-bit 96 khz!
Cybermynd
April 20, 2007 at 8:11am
Although many would consider headphones to be the ultimate transducer for this type of testing there are people who consider themselves audiophiles that will argue there are more revealing systems that should be used for testing.
Not to say that most people will ever acheive or exceed the excellence of your test setup. I guess my point is that perhaps using a high end, speaker based setup might convince more people.
Then again, anyone who buys $8000 speaker cables can safely be considered a questionable listener...
Thanks for an illuminating article!
Print
