Inside The Consumer Reports Testing Facility

Ever wonder how Consumer Reports figures out which products to recommend? For one, it takes mad science, like this echo-free room that sits on a different foundation from the rest of the building. I was up at the Consumer Reports HQ yesterday for a planning meeting related to a blogger’s conference they’re planning for June, and they were nice enough to give me a quick tour of their testing facilities. I snapped some 33 pictures with my cellphone camera. Check them out in the interactive photo essay gallery, inside…

Comments

Edit Your Comment

  1. clevershark says:

    When I buy a house I’m sooooo putting that sort of sound insulation on the walls of my bedroom… imagine having one room in your place that’s *perfectly* quiet!

  2. deviationer says:

    Consumer Reports of BS

  3. ekthesy says:

    Very cool, Ben! Now I see why the magazine is so expensive.

    May I ask what cellphone you use to take such great photos? Compared to my cellphone photos yours look positively hi-res.

  4. Beerad says:

    @deviationer: Thank you for advancing a meaningful discussion with your words. Your statement is so pithy and insightful, and promotes a better understanding of the issues. God bless you, deviationer!

  5. Ben Popken says:

    @ekthesy: Verizon LG enV 9900. Camera is 2 megapixels.

  6. weakdome says:

    @clevershark: Have a neighbor with a “loud girlfriend”, do you?

  7. weakdome says:

    @clevershark: was supposed to also have a link to: [xkcd.com]

  8. Darkwish says:

    Sweet! I wondered what their testing facilities were like. I think it would be cool to work there.

  9. HeyBeav says:

    Amazing! The Listening Room looks exactly like my media room, except the metal grid flooring at my place is to allow Cheeto crumbs to fall harmlessly away and not sully the soles of my socks.

    Tomato, tomahto.

  10. am84 says:

    Awesome. I want to work there.

  11. Pro-Pain says:

    Time to max out the Marshall stacks WOO HOO!

  12. IphtashuFitz says:

    That speaker testing room reminds me of the new symphony hall in Seattle that I got a tour of a couple years ago. It’s built on top of a subway line so they did the whole “isolate a building-inside-a-building” thing so that you don’t hear or feel any rumbling of trains underfoot. When you walk from the outer lobby into the symphony hall you can also detect the subtle sudden lack of background noise.

  13. trujunglist says:

    Anechoic chambers are actually not as cool as they look. My school had one, and as an acoustics major I had to do some work in there. It’s not actually a very pleasant place to be, because you don’t hear ANYTHING other than the sounds your own body makes. The tinnitus in my ears was annoyingly loud because there was nothing else to mask the noise. I spent as little time as possible in there during the TEF/speaker analysis. I much preferred the reverb chamber, which was made from a bank vault and had a RT60 of something like 4 seconds at lower Hz.

  14. trujunglist says:

    @clevershark:

    You’d soooo not like it and end up losing your mind. Anechoic chambers are really fun… until you’re in one for more than a couple of minutes.

  15. SpdRacer says:

    30 minutes inside an anechoic chamber, by yourself, and you start to go a little nutty. The Orfield Labs Anechoic Chamber has been awarded the Guinness World’s Record for the quietest chamber, they have a standing bet for any employee to last 45 min alone in the chamber for a case o’ beer. Ain’t happened yet.

  16. Alex Chasick says:

    “Need to spit” and “want to spit” are how I’m going to describe all sub-par foods from now on.

  17. SigmundTheSeaMonster says:

    @clevershark: You would still hear outside noise unless the walls the acoustic-absorbing material is attached to are solid concrete/plaster.

    @SpdRacer: Yep. You hear your heart, tinnitus (if you have it), and if you speak, there is no echo. Creepy.

  18. Xavoc says:

    @SigmundTheSeaMonster: I work in an office designed for 100 people, I’m one of 6 people who work here now… I’m so far from everyone, and it’s so quiet, that without music all I hear is my ears ringing and the environmental system.

    Ugh.

  19. Maulleigh says:

    I’m so envious. I’m not kidding.

    Now it’s a little too “dear” for my budget–a subscription. When I go to the library to read them, I have to ask special cuz they’re always getting stolen!

  20. Soldmysoul says:

    Wow, they test things at consumer reports? I just assumed that the companies that paid the most money got the best ratings…

  21. Xavoc says:

    @Soldmysoul: Actually, Consumer Reports doesn’t even accept free product from a company to test. If they cannot go to a store and purchase the item off of the shelf, they don’t test it.

    This includes automobiles, and other expensive big ticket items.

  22. theblackdog says:

    @weakdome: Dude, that comic rocks!

  23. JiminyChristmas says:

    @SpdRacer: Orfield Labs is in my hometown and I took a field trip there several years ago as part of an architectural acoustics course. The anechoic chamber is pretty weird. Someone can be talking to you from three feet away and it sounds like a whisper.

    Fun fact: The Orfield Labs building used to be a music studio. ‘Funky Town’ was recorded there.

  24. celyn says:

    That speaker testing room reminds me of the new symphony hall in Seattle that I got a tour of a couple years ago. It’s built on top of a subway line

    Benaroya Hall is built near a bus tunnel. Would that it were a subway line. This is actually a sore point for Seattleites. We have no rapid transit system, just buses that run on surface streets (and some that run through the aforementioned downtown bus tunnel).

  25. SAugsburger says:

    @Xavoc: Just because someone doesn’t take advertising doesn’t mean their reviews are any good. I can pull tons of random joes who never worked for any company in the industry or even got any free samples from a company and their product reviews are terrible.

    I got a free subscription to Consumer Reports just to get a free book about how to clean stuff. Not only was the free book lame(no big surprise), but even the magazine wasn’t very good. Most of their reviews on computers and even printers often are printed long after the products are discontinued, which means that easily half of the products on the market aren’t even reviewed by Consumer Reports.

    Probably the worst part is that the information they give is that it is vague. I remember telling one fan of Consumer Reports that they didn’t give any methodology to their reviews. She said well it says “this computer has above average for ease of use.”

    WTF? What does that mean? Why? If they all run the same software(Windows Vista plus the same AV software) than how much easier can computer A be than computer B? Virtually every other magazine that reviews electronics gives me not only some information about why, but more importantly they give some type of methodology of how they tested stuff. They will say we used xyz benchmark software to compare our top 10 desktops and laptops this month. If I wanted to I could go see whether their data was right, but does Consumer Reports provide this? Nope. They just give a circle that is full, half full, or empty. What does that mean? Even PC World, which usually only writes half page reviews at least talks about the product. Most magazines have at least half a dozen different ratings they can give a product. Using a four point scale with half points there are 8 different scores a product could get from an abysmal 1/2 point to 4 points for the perfect product that would be difficult to improve.

    Somehow I think that Consumer Reports “labs” are just for show. Consumer Reports on several occasions has had to retract articles as inaccurate and they have even lost a libel suit. Not taking ads doesn’t make the publication accurate. Many political opinion journals take little or no advertising, but no one calls them objective. The less information a magazine gives me on their methods of testing a product the less I trust it. It tells me they have something to hide, which is most likely their incompetence in testing products that they claim to have expertise on. I don’t care if they have zero advertising. If they refuse to share even the slightest amount of information on their methods of testing chances are their methods are BS, and by conclusion their article is BS.

    The reverse is also true. If a magazine tells me exactly how they performed their tests, I don’t care that they accept ads. Their methodology might be dubious, but at least I can judge for myself. Furthermore, others can attempt to verify to information and or question the validity.

    Consumer Reports relies upon a aura of credibility by not not taking ads while ignoring any good publications standards of writing reviews that are informative(CR fails badly at this) and attempt to be as transparent as possible(CR is one of the few publications where I virtually never see any descriptions on test methods). Good reviews aren’t black boxes with a bad, good or great rating system.

    One reason why there are a lot of good publications and websites that accept advertising is because most reputable publications have had long standing rules that separate their ad divisions and their product reviewers. The publishers realize that if their reviews are considered phony then no one will read the publication and by extension there will be no eyeballs for advertisers. I have seen more than a few products that were advertised in PC Gamer for example that were give some of the lowest scores the publication has ever given. It is pretty clear that advertisers don’t buy themselves a minimum rating from the reviews.

    A another technology example would be Tom’s Hardware. They have plenty of advertising, but they are pretty transparent about their testing methods. Some of their testing methods sometimes are useful for your particular situation, but like any tech benchmarks they are only good within a context, but I have seen very few people who have attempted to reproduce their tests find their data inaccurate. My past use of them as a source in buying products has been pretty positive so I continue to visit and use their reviews amongst others in buying decisions. It is important to note that they aren’t the only source. Like anything in life you try to finding corroborating or conflicting evidence because one source is rarely enough especially on a high end purchase.

    Trust in reviews for me is based upon transparency of methodology and past accuracy of their articles. Consumer Reports fails on both accounts.

    I ended up canceling my subscription before they started charging me for it. I frankly can’t comprehend what people see in the publication. If they get stuff wrong that I do know about why should I trust them for stuff I don’t know about.

  26. Groovymarlin says:

    Lucky! They sure have a cool facility.

    You guys know you can subscribe online for substantially less than the cost of the printed magazine, right? As far as the value of their testing and reviews, of course everyone is going to have their own opinion and experiences. In my experience, the information they provide has been very valuable and I’ve never regretted following any of their recommendations. Naturally, your mileage may vary. :)

  27. Groovymarlin says:

    Ugh, my bad on the last comment. Actually subscribing online costs the same as the print subscription.

    Print subscription: $26/year, can then add online subscription for another $19/year.
    Online subscription: $26/year.

    I find the online subscription very useful since it gives me a large database of product reviews and articles to search when I’m thinking about buying something.

  28. SAugsburger says:

    @Groovymarlin:

    “Lucky! They sure have a cool facility.”

    Some of the pictures are either unimpressive or difficult to even tell what they are for. The sound proof room for audio testing hardly impresses me. Even college radio stations have sound proof rooms! A college radio station’s soundproof studio may be small, but even with their tiny budget they can afford sound proofing all the walls. You need to get out more if you seriously were impressed.

    My real issue with Consumer Reports is why pay money for something when for a lot of product categories I can get better reviews for free online? Unlike CR, most publications provide some if not all of their reviews for gratis, ad supported of course. I have no qualms about spending money if I feel I get a superior product, but anybody who seriously thinks Consumer Reports provides quality reviews on any tech items either hasn’t compared publications dedicated towards consumer technology or they are putting the lack of ads over the quality of the reviews.

  29. larry_y says:

    I find CR useful. I go there as a starting point for washers, dryers, lawn mowers, cordless drills, elliptical machines, etc. However, it isn’t for computers or electronics. They simply can’t keep up.

    And if you’re reading Tom’s Hardware or Anandtech, you’re not their audience anyway. Average consumer wants something that doesn’t suck and is good enough, but won’t spend their free time figuring out acronyms and technical jargon.