How objective are bike reviews in magazines?

neeb

Sometimes you see bike reviews in magazines and on their commercial websites (such as Bike Radar) that are suspiciously "glowing".. I'm sure that if you asked any of these magazines they would claim that they give their honest opinions in reviews based on extensive experience of testing many different bikes. I'm not necessarily saying that this isn't the case, or that it isn't the case for some magazines, but what guarantees and assurances do we have as consumers that this is the case?

Clearly there must be huge commercial pressure from the marketing departments of the big bike brands to obtain good reviews. How does the law stand in the U.K. on the independence of reviews in magazines that claim to be objective? Would it be legal for a bike manufacturer to explicitly pay a publisher to write a good review and pass it off as objective and unconnected to advertising? I must admit I have no idea myself.

Even if it was illegal for a manufacturer to blatantly pay for a good review, what safeguards are there in place to ensure that reviews aren't influenced (either slightly or not so slightly) by the relationships between manufacturers and journalists and/or publishers? If a manufacturer pays for a bunch of journalists to come on a trip to the Alps to test their latest bike, wines & dines them and gives them a bunch of freebies, how can the subsequent review be truly objective?

Perhaps I am being naive here and everyone else just accepts implicitly that bike reviews are just another form of advertising and there is no chance of any genuine objectivity. Or perhaps I am being too cynical, and bike journalists are all well-intentioned enthusiasts who always try to give their honest opinions, even in the face of pressure from the marketing departments of the big brands.

diamonddog

Any review be it of bikes,cars, watches, wine etc etc can only ever be subjective IMO. I always prefer to personally research whatever it is thoroughly before I commit to buying it. Any review is just an opinion or collection of opinions.

hibster

i know of times when albums have been slated for personal reasons nothing to do with the music, so not even commercial pressure, but a bias against people in the band by the reviewer
not the same as bikes I know, but a similar premise to the OP
it's not just bikes thata could be affected here, but bike kit too

thegreatdivide

Marcel Wüst does a good bike review for Pro Cycling - if you're in the market for a very top end velo.

neeb

diamonddog wrote:

Any review be it of bikes,cars, watches, wine etc etc can only ever be subjective IMO. I always prefer to personally research whatever it is thoroughly before I commit to buying it. Any review is just an opinion or collection of opinions.

Maybe "objective" was the wrong word for me to use. A review is always going to be subjective as you say, but I would like to think that the reviewer is giving his/her honest opinion uninfluenced by marketing pressure, or at least is trying not to be influenced...

mercia_man

As an evening newspaper journalist (now retired), I regularly road tested motorcycles in the 1980s. There was never any question of manufacturers or importers paying money for good reviews and I always reported truthfully on my experiences. That applied to all the news stories I wrote as well. However, we would occasionally be told to lay off doing stories or following up a particular line following pressure from major advertisers - such as the boss of an estate agent having a cosy chat with our managing director. I guess this sort of pressure applies to all publications. It's morally wrong and journalists get angry about this kind of interference. But then the adverts do pay our wages. Being a member of the NUJ with its code of conduct and professional representation does help journalists stand up to advertiser pressure.

I was occasionally wined and dined by motorcycle manufacturers launching new ranges of bikes. And it was great fun. I fondly remember trips to Donington Park race circuit where all the manufacturers had their ranges lined up and you could choose whatever superbike you wanted for a thrash around the track. Such a brief assessment of a motorcycle does not give a complete picture of what it's like and I'm sure the average reader is intelligent enough to realise that the journalist's views are tinged by the enjoyable time had at the press launch. The subsequent full road test is much more to be relied on, as long as the journalist knows what he or she is talking about.

I don't believe that road tests by journalists are just another form of advertising. Although I do sometimes feel I cannot always rely on some of the modern reviews in Cycling Plus as the authors sometimes reveal a degree of ignorance which was not displayed by previous expert testers like Paul Vincent.

markhewitt1978

I find bike reviews mostly useless anyway as the reviewer is rarely commenting from the same position as you are, and may be commenting e.g. that a bike doesn't handle very well, but they want fast handling, whereas for someone else it's super stable etc.

neeb

Mercia Man wrote:

As an evening newspaper journalist (now retired), I regularly road tested motorcycles in the 1980s. There was never any question of manufacturers or importers paying money for good reviews and I always reported truthfully on my experiences. That applied to all the news stories I wrote as well. However, we would occasionally be told to lay off doing stories or following up a particular line following pressure from major advertisers - such as the boss of an estate agent having a cosy chat with our managing director. I guess this sort of pressure applies to all publications. It's morally wrong and journalists get angry about this kind of interference. But then the adverts do pay our wages. Being a member of the NUJ with its code of conduct and professional representation does help journalists stand up to advertiser pressure.

I was occasionally wined and dined by motorcycle manufacturers launching new ranges of bikes. And it was great fun. I fondly remember trips to Donington Park race circuit where all the manufacturers had their ranges lined up and you could choose whatever superbike you wanted for a thrash around the track. Such a brief assessment of a motorcycle does not give a complete picture of what it's like and I'm sure the average reader is intelligent enough to realise that the journalist's views are tinged by the enjoyable time had at the press launch. The subsequent full road test is much more to be relied on, as long as the journalist knows what he or she is talking about.

I don't believe that road tests by journalists are just another form of advertising. Although I do sometimes feel I cannot always rely on some of the modern reviews in Cycling Plus as the authors sometimes reveal a degree of ignorance which was not displayed by previous expert testers like Paul Vincent.

Thanks for that information, very interesting! It sort of confirms my gut feeling that reviewers are trying to be fair most of the time, but are inevitably influenced a little, sometimes, by commercial pressure. But intelligent readers should be able to put any review in context, and usually the context is explicit.

I hadn't thought of the NUJ aspect and how that helps to enforce a professional code of conduct

neeb

markhewitt1978 wrote:

I find bike reviews mostly useless anyway as the reviewer is rarely commenting from the same position as you are, and may be commenting e.g. that a bike doesn't handle very well, but they want fast handling, whereas for someone else it's super stable etc.

Again however, you can put that in context - it's usually obvious if a reviewer doesn't know what they are talking about or if they are coming from a particular perspective, so you can make allowances / take with a pinch of salt as appropriate.

markhewitt1978

Lets face it, if a reviewer is invited to press events, wined and dined, especially if it's at the manufacturers expense, they aren't going to give a daming product review, for fear of not being invited again - and moreover upsetting the publications management who don't want to be on a black list.

It's well publicised that Apple (for example) have a black list of publications who don't get invited to events because they've been negative about them in the past, IMO that just leads to even more negative press!

markhewitt1978

neeb wrote:

markhewitt1978 wrote:

I find bike reviews mostly useless anyway as the reviewer is rarely commenting from the same position as you are, and may be commenting e.g. that a bike doesn't handle very well, but they want fast handling, whereas for someone else it's super stable etc.

Again however, you can put that in context - it's usually obvious if a reviewer doesn't know what they are talking about or if they are coming from a particular perspective, so you can make allowances / take with a pinch of salt as appropriate.

Which is why user product reviews are important IMO. If you read enough of them you'll get a good idea - after all some reviews are going to say, best product ever, some saying worst product ever, but you can get the general feeling.

ugo.santalucia

When they use the classic jargon: light/stiff/responsive/holds speed and all that nonsense that is supposed to differenciate between virtually identical products, then you know it's just garbage.
I don't even know how they can judge a frame when they are given a bike with Shimano RS 10 wheels or Aksium... clearly that takes any liveliness or alleged performance away and it becomes guesswork

neeb

ugo.santalucia wrote:

When they use the classic jargon: light/stiff/responsive/holds speed and all that nonsense that is supposed to differenciate between virtually identical products, then you know it's just garbage.
I don't even know how they can judge a frame when they are given a bike with Shimano RS 10 wheels or Aksium... clearly that takes any liveliness or alleged performance away and it becomes guesswork

You do sometimes read reviews where they say that they swapped out the wheels for better ones to allow a better assessment of the frame.

I'm in two minds about the jargon. I have had carbon frames that have differed really noticeably in stiffness under pedalling (with the same wheels), and haven't liked the more flexy ones. You only notice it when climbing / accelerating out of the saddle, but I do think it makes a difference. And a light & stiff bike is certainly more responsive than a heavy & flexy one. I also notice front end stiffness a bit (although it's less of a factor for me being quite light), and some bikes do seem to inspire more confidence when descending than others (although I suspect that's as much to do with things such as angles / fork offset / trail, which differ between different frame sizes and are hardly ever talked about in detail in reviews).

ugo.santalucia

neeb wrote:

I'm in two minds about the jargon. I have had carbon frames that have differed really noticeably in stiffness under pedalling (with the same wheels), and haven't liked the more flexy ones. You only notice it when climbing / accelerating out of the saddle, but I do think it makes a difference. And a light & stiff bike is certainly more responsive than a heavy & flexy one. I also notice front end stiffness a bit (although it's less of a factor for me being quite light), and some bikes do seem to inspire more confidence when descending than others (although I suspect that's as much to do with things such as angles / fork offset / trail, which differ between different frame sizes and are hardly ever talked about in detail in reviews).

Thing is Neeb, I am lucky enough to own a 1980 frame and I have ridden a few carbon frames of the latest generation. The difference is not that massive, it's there, but it's not lifechanging... and that's over 30 years of filling our heads of "significantly improved stiffness"... You and I both know it's mostly bollxxks.
So, when a spotty reviewers says that the new Cervelo is massively stiffer than the previous one, you know it's just trying to fill your head with more sxxt

Calpol

I think you have to be very selective in what you believe. i have some recent first hand experience in this area albeit not with bikes. Lets say that for these purposes the product concerned generally cost much more than a racing bike.

A well known magazine published a review of one of our companies products which our PR department were very unhappy about and indeed was at odds with several other press opinions. The review was incredibly subjective so as a result we pulled all our advertising with the magazine.

Next thing you know they have published a 5 star review of one of our products that actually in no way would merit 5/5 thus completely undermining any credibility. Anyone that tells you advertising revenues don't influence product reviews in the glossies is naive in my opinion. There might be a load more integrity with more local press or some web media but I am now even more cynical about these sorts of things.

Jim C

^^^^ ugo. This
take reviews with a large pinch of salt.....for lots of reasons

neeb

ugo.santalucia wrote:

Thing is Neeb, I am lucky enough to own a 1980 frame and I have ridden a few carbon frames of the latest generation. The difference is not that massive, it's there, but it's not lifechanging... and that's over 30 years of filling our heads of "significantly improved stiffness"... You and I both know it's mostly bollxxks.
So, when a spotty reviewers says that the new Cervelo is massively stiffer than the previous one, you know it's just trying to fill your head with more sxxt

Maybe I've just been lucky/unlucky to have owned frames that are at one extreme or the other. I had a steel Mercian made out of Reynolds 725 (bought 2nd hand, I think it was the now discontinued "superlight" model and late 1990s/early 2000s vintage). That was seriously flexy, so much so that I basically avoided climbing out of the saddle on it and remained seated as much as possible. But my Enigma Extensor (Columbus XCR) is a world apart, much, much stiffer (admittedly the geo is a lot more aggressive too). And while my Scott Foil and Trigon rqc-29 are both nice and stiff, the scuro rs I had a few years back was disappointing (maybe it was a Chinese knock-off / factory reject, it came from a small retailer in France).

neeb

Calpol wrote:

I think you have to be very selective in what you believe. i have some recent first hand experience in this area albeit not with bikes. Lets say that for these purposes the product concerned generally cost much more than a racing bike.

A well known magazine published a review of one of our companies products which our PR department were very unhappy about and indeed was at odds with several other press opinions. The review was incredibly subjective so as a result we pulled all our advertising with the magazine.

Next thing you know they have published a 5 star review of one of our products that actually in no way would merit 5/5 thus completely undermining any credibility. Anyone that tells you advertising revenues don't influence product reviews in the glossies is naive in my opinion. There might be a load more integrity with more local press or some web media but I am now even more cynical about these sorts of things.

Also very interesting, thanks.

ugo.santalucia

neeb wrote:

Maybe I've just been lucky/unlucky to have owned frames that are at one extreme or the other. I had a steel Mercian made out of Reynolds 725 (bought 2nd hand, I think it was the now discontinued "superlight" model and late 1990s/early 2000s vintage). That was seriously flexy, so much so that I basically avoided climbing out of the saddle on it and remained seated as much as possible. But my Enigma Extensor (Columbus XCR) is a world apart, much, much stiffer (admittedly the geo is a lot more aggressive too). And while my Scott Foil and Trigon rqc-29 are both nice and stiff, the scuro rs I had a few years back was disappointing (maybe it was a Chinese knock-off / factory reject, it came from a small retailer in France).

Yeah, well, despite the understandable love for all things british, there is a fundamental reason why Eddie Merckx (to name one) was riding frames made in Italy and not those made in England. A Hetchins or a Mercian are a pretty things, but I doubt they ever won a race of any relevance (and the milk race is not!).
However, take a decent Italian frame from the days of SLX and it's more than adequate even for today's demending stiffness standards

mercia_man

I admit I do feel sorry for cycle magazine journalists finding something different to say about modern carbon bikes. They are mostly made in the Far East in the same factories and are equipped with factory wheels and Campag, Shimano or Sram components. OK, one bike might have quicker or more stable handling than another, one might have a longer head tube and another might have fancy carbon handlebars. Doesn't give writers much to say. No wonder they resort to cutting and pasting manufacturers' PR handouts to pad out their reviews.

But I don't believe that magazine and website writers generally write glowing reviews because of freebies or advertiser pressure. They may be inexperienced cyclists or technically ignorant, they may be experts but with a racing rather than a touring or recreational background and so prefer a different type of bike than some readers, or they may have to write a rushed review to meet a deadline. Their reviews are subjective and should be considered as such - sometimes useful, sometimes not. The reader just needs to use intelligence to decide how much weight to put on a reviewer's comments.

It's the same with comments on this forum. I always take heed of what people like ugo, Monty dog, cycleclinic and hoopdriver say because I think they know what they are talking about - even if I don't always agree with them.

I find forums a very useful way to assess whether components are any good. For example, if you rely on Bike Radar and Cycling Plus reviews, Zipp Firecrest wheels are the best you can buy. But a bit of investigation on this forum and you realise the rear hub design was suspect, judging by people's reports of failures, and that Zipp has now changed the spoke pattern and beefed up the hub for 2014.

Of course, another way to write a cycle review is the exhaustive comparisons done by Tour magazine from Germany. Every frame is tested for stiffness in a laboratory, every wheel for aerodynamics, every component for efficiency and then a best bike in the world accolade is awarded. German bikes normally do very well in these tests. But does such an objective approach really tell us how good a bike is to ride?

markhewitt1978

The same is true of cars isn't it? Unless you're reviewing some supercar (a la Top Gear) then they are pretty much all the same at similar price points. Which only leaves branding and unquantifiable stuff like image, mostly how the purchaser wants to be viewed by others.

Greenbeetle

We all know subjective reviews are pretty silly really, but still we like to see them, and even allow ourselves to be influenced by them when it comes to buying choices. I'd be surprised if there were anything underhand going on, but then I tend to expect people not to be deliberately dishonest...

The kind of tests done by Tour Quarterly, on the other hand, are worthwhile, and faking those results would be downright criminal.

alihisgreat

You've got to be careful about the causality here: are reviews generally quite positive (and similar) because bikes and quite good (and similar), or are they generally quite positive because publishers are afraid of printing a bad review?

I think the best way is to read the text of the review and only take note of the actual score if it is unusually high or low. The text will usually give you a good idea of the strengths and weaknesses of the bike - and you can combine this with your own research to determine the extent to which a strength or weakness would bother you.

Steve236

neeb wrote:

Sometimes you see bike reviews in magazines and on their commercial websites (such as Bike Radar) that are suspiciously "glowing"..

Are you thinking about the Synapse Hi Mod review that's appeared on BR this week? It certainly reads more like an ad than a review!

Jim C

From a couple of posts back, re: British frames not winning major races

I believe Laurent Fignon won Le Tour on a Raleigh. Albeit Special Products, Ilkeston, but most certainly British built. From memory, it wore Peugeot transfers

Edit , Fignon didn't win when he rode his Raleigh, but Zoetemelk did in 1980.
60 odd TdF stages. Plus a whole host of classics - Flanders, Paris Roubaix, Ghent Wevelgem

the first great things to come out of Ilkeston, prior to Mr Brailsford

neeb

Steve236 wrote:

neeb wrote:

Sometimes you see bike reviews in magazines and on their commercial websites (such as Bike Radar) that are suspiciously "glowing"..

Are you thinking about the Synapse Hi Mod review that's appeared on BR this week? It certainly reads more like an ad than a review!

Yup, that, and also the Felt AR2 review:
http://www.bikeradar.com/road/gear/cate ... -14-47802/

Supposedly the only negative comment they could come up with was:

LOWS: The AR2’s all-round ability might make a couple of your other bikes redundant

I mean, it could be the best thing since sliced homemade wholemeal sourdough with almond butter for all I know, but you can't help wondering...

mercia_man

I agree that the Felt AR2 review is too gushing. Next year no doubt we'll be told by a reviewer that this AR2 model was too stiff or something and that the latest 2015 replacement is so much better. Just one of those reviews filled with PR guff that discerning readers will take with a pinch of salt. I wish they would put bylines on all their reviews so we know who has written it. Some writers clearly have more cycling experience and technical knowledge than others.

neeb

Mercia Man wrote:

I agree that the Felt AR2 review is too gushing. Next year no doubt we'll be told by a reviewer that this AR2 model was too stiff or something and that the latest 2015 replacement is so much better. Just one of those reviews filled with PR guff that discerning readers will take with a pinch of salt. I wish they would put bylines on all their reviews so we know who has written it. Some writers clearly have more cycling experience and technical knowledge than others.

Give it a few years until paper publication has disappeared entirely and we will all get reviews customised to our personal tastes and levels of knowledge..

alihisgreat

Mercia Man wrote:

I agree that the Felt AR2 review is too gushing. Next year no doubt we'll be told by a reviewer that this AR2 model was too stiff or something and that the latest 2015 replacement is so much better. Just one of those reviews filled with PR guff that discerning readers will take with a pinch of salt. I wish they would put bylines on all their reviews so we know who has written it. Some writers clearly have more cycling experience and technical knowledge than others.

Is there any reason why a bike can't be that good though? Its not as if they are giving everything a glowing review and 5*s then we might need to be more concerned.

bobones

Yes, however the 2014 Felt AR series is the real deal http://aerogeeks.com/2013/09/24/2014-fe ... -analysis/

mercia_man

ALIHISGREAT wrote:

Mercia Man wrote:

I agree that the Felt AR2 review is too gushing. Next year no doubt we'll be told by a reviewer that this AR2 model was too stiff or something and that the latest 2015 replacement is so much better. Just one of those reviews filled with PR guff that discerning readers will take with a pinch of salt. I wish they would put bylines on all their reviews so we know who has written it. Some writers clearly have more cycling experience and technical knowledge than others.

Is there any reason why a bike can't be that good though? Its not as if they are giving everything a glowing review and 5*s then we might need to be more concerned.

I'm sure that bike is really good but I just feel that the AR2 write up and the Synapse one mentioned above seem so gushing that they might as well have been written by a PR person. I feel it's over enthusiasm by the reviewers rather than pressure from advertisers or marketing people. I know how difficult it is to write an objective review to a tight deadline while avoiding PR speak and cliches like those dreaded "bombproof wheels" we are always reading about.

Each new year's model is better than the last - or so we are generally told in the magazines. I don't accept that. Much modern stuff is great. But other developments are not necessarily an improvement.

What magazine and website reviews don't normally tell us is how these fantastic frames and components perform in the long term. There are plenty of comments on this forum from people suffering broken spokes on their supposedly bombproof wheels or squeaks from their supposedly ultra stiff press fit bottom brackets.