Every Headphone IEM, Amp, DAP and DAC that has came through my door for the past 2 years has been subjected to testing under as close a set of conditions that I could manage. The reason I do this is that I go through a lot of equipment and I like to catalogue them so as to be able to hold better comparative ability in the future. I got sick of reading the subjective reviews on Head-Fi from people listening to X equipment with X variables then making comparisons to other equipment and undergoing a completely different level of evaluation the next time around. There was very simply little to no standardisation of process. It got tired fast reading about how “this headphone has way more bass and is way more open than this etc”, in most cases people were recommending pieces off of a loose memory from the time that they owned said equipment with different songs and source equipment. The other peeve of mine and I'm sure one that you have noticed is audio reviewers flat out making up terminology and writing about a pair of headphones as if it was the twilight saga. I am not a 12 year old girl and I'm sure most of the folk interested in high end audio are not either. The Following is the way I do it, its not necessarily the right method, I don't think we will ever get to that stage, but have found it works for me. I hope you enjoy reading this and please feel free to comment.
Myself I had always kept loose listening notes from as far back as the release of Shure's e2c IEM. I wondered if I took just a few hours every time I got something new to use a standardised method of evaluation to documenting its performance what sort of data I would accumulate. The answer is a lot, far more than a hobby it’s now an OCD riddled obsession. I now have a massive catalogue of reviews and listening notes all performed under similar condition and a compulsion to go through that process with every piece of gear I get.
I use Rhodia or Moleskine pads and either a Cross fountain or ballpoint. The one with the red Quiver pen holder is headphones; the orange earphones and my yellow one is for Amps Dacs DAPS etc.
I like to conduct my reviews in stages as you can probably tell from my written reviews that every single one follows the exact same format. This is in the hope that when someone is comparing one headphone to another that they can find a better ability to draw comparisons and whilst this method is in no way perfect it is an attempt at least to cut out some of the BS that I found so frustrating when I was searching for the perfect set of cans many years ago.
Stage 1: Packaging
Firstly, I examine the packaging and presentation as I think this is important for 2 reasons, how does it make you feel? Does it add to the excitement? and second is how well packaged is the product to withstand delivery.
Stage 2: Accessories
Next will be an evaluation of the accessories. In this case there is a flat base of expectation and anything above gets marked higher. For example, if I were buying a set of earphones I would expect at least a 3 pack of varied size replacement tips. The manufacturer may then add in bonuses such as additional tips, cable management solutions and storage cases and can get extra points in this category for the quality of them (ie a food soft case does not equal a good hard case).
Stage 3: Build Quality / Aesthetics
After accessories are taken care of I move on to examining the build quality and style of the piece. Does it feel sturdy? Are there any manufacturing defects? Does it feel premium or cheap? How does it look? Are they eye catching? What environment would they fit into? How durable are jack points and strain reliefs? etc.
Stage 4: Fit
Now we are getting closer to the good part as the earphones are inserted or the IEM’s place in the ear. Im about to take a look at the fit. Are they comfortable? How is the cable management? Length of the cable and how prone to micro phonics it is. In the case of headphones I will look at the weight, the clamping force and the comfort of the pads among other things.
Stage 5: Acclimatisation
The product is then used for minimum 2 days in general use to remove any shock to the system and minimise perceived differences due to familiarity with a previous piece of equipment I was using. Think of this as a palette cleansing of sorts before the main meal.
Stage 6: Critical listening and sound quality testing
Finally is the best part and the one to which I will go into greatest length on. Up until this point my notebook is filled with around a page worth of scribble in relation to the values and impressions gathered from the above sections. The next few pages will usually be filled with detailed testing notes as I work my way through the playlist for the critical listening test.
When listening I always use the same tracks, without fail. This has a few benefits but most notable is that after listening to the same songs over and over and over you begin to get an idea of how it should be reproduced. If something is off then it stands out like a sore thumb, I urge all enthusiasts to create a playlist specifically for this purpose. The downside? Well some of the songs you do get a bit sick of.
OK lets dive in. I fire it up starting with classical, specifically Mozarts – Requim (Colin Davis and LSO) and 1812 Overture by Tchaikovsky (Telarc Digital Canons). Moving a little more modern now with Vanessa Mae – Storm and Ludovico Einaudi – Divenire. The reason the testing is so heavy in classical is due to the large frequency range in the genre. There is an instrument in there that will cover all frequencies well and thus is an excellent starting point for any evaluation. There are wind instruments for highs, stringed for decay and there is a lot going on in orchestral tracks so as to better determine separation learning to love classical is something I urge everyone to do both for the enjoyment and to improve your technical listening ability.
Vocals are up next and for female its Diana Krall - and Nico’s – These Days (Krall managing the Upper and Nico the lower ranges of a female vocal). For male Vocals Bocelli and …… wait for it…… Everlast – White Trash Beautiful (It has a wonderful grit and weight in the voice).
Electronic comes in the form of Choas Royale - Anna Supernova which is an excellent track to determine bass quantity and impact, sub- bass is inherently present throughout. Infected Mushroom - Pletzura is an extremely complicated track which goes through many movements and has lots of switching from right and left channels.
Rock? Led Zepplin – Whole Lotta Love for both the vocals and reproduction of Jimmy Pages iconic guitar riff. Live rock I chose - In this River by Black Label Society because of emotional vocals and environmental reasons, artefacts picked up from the crowd and so on (plus the guitar solo is ridiculous, he's playing a large part with his teeth).
Finally I finish off with Amber Rubarth – Full moon in paris (Chesky) for some binaural goodness.
I included the links to the youtube videos if you want to click but this is only as a guide I urge you to grab the songs in at least 320kbps mp3 or higher to get the full effect.
In all of these tracks I listen for a range of factors from the following: specific frequency response, separation, decay, overall balance/tuning, separation and many more. I evaluate the highs mids and lows with specific sections I am familiar with in the above playlist and through the entire test I take scribbled notes that I come back to when writing my review to elaborate on.
So that’s it? Not quite. After the testing is done I will use them for a while with the intention of getting more familiar with the product. I take them out in public and on commutes to evaluate their fitness for purpose and in the case of IEM’s each will come to the gym with me at least once to see how they perform. Why do I not take headphones to test at the gym? Well....You shouldn’t use headphones at the gym (a pet peeve). Gym evaluation for the earphones doesn’t involve sitting around on a stationary bike etc. I am a big fan of calisthenics, parallel bars Muay Thai and gymnastics so the earphones are put through the absolute ringer. Usually cable over the ear earphones perform best in this field due to having less chance of popping out and having lowered microphonics.
In the Gym with the ProGuard P2+1 Expect every earphone to go through a few days of this.
During these next few days I then spend time listening to whatever I am into at that particular moment and will continue to make further observations on the performance and adding to the notebook.
The Ratings System
I rate each category out of 10. Again this is to find some level of quantification and comparability for the above categories between reviews. Here is what the ratings are intended to mean.
1-3 Terrible, the lowest of the low
9-Very good almost top Tier
10-Top tier one of the best in class
Lets take for example the average category you could say that a 4 for example is just a step above Terrible but still passable whereas a 6 is still average but on the upper end and almost good. This is meant purely as a guidance and I urge folks to read the words in each section rather than relying too heavily on a number. Rating headphones and earphones to me in minute decimals seems absurd very occasionally you will see a 9.5 when a headphone is just missing that special something to take it to the next level but there wont be 4.3 6.1 etc.
As always the best way to test any headphone is to do it yourself and get some hands on time with them. Don’t get to hung up on incremental differences as the one thing I have learned is that the difference between a good set of headphones is very very small and usually down to tuning at the top end of the market. We are talking at minute technical levels in this game, a game where the law of diminishing returns is ever present the higher up a price scale you go. Its fine to spend thousands extra to get to that next level but understand that expectations should be managed. Beginners don’t get too hung up on the gear and don’t feel you need to go all in at the beginning. Progression is part of the fun of the hobby, experimenting and building up to your dream gear is a journey. If you take your time building up you will have a greater appreciation for the nuances that make 1 piece of equipment technically better so take your time and develop your listening skills.
Always remember it should be about enjoying the music first. Even with owning $1000’s worth of gear I can still enjoy myself with a set of ibuds if needed and I would rather have good music I love on a crappy set of buds rather the entire One Direction back catalogue on a set of Sennheiser Orpheos.