Forrester Reports: Unreliable and Untrustworthy

I’ve just finished reading a Forrester Report by Shar VanBoskirk’s titled: “The Forrester Wave™: Bid Management Software Providers, Q4 2012” and was extremely surprised by its biased and low quality content. It’s not the first time I’ve seen what I would describe as “a low quality report” from Forrester. However, this is the worst perceived “report of authority” I think I’ve ever read, from a publisher of this status. Online publishers with less authority such as Search Engine Land publish articles that supersede the quality of this ‘report’ by a significant margin. Even paid advertorial would exceed the quality of the content within this report.

My first observation is that this “report” appears to be extremely biased towards particular vendors and I’m wondering if Shar VanBoskirk was paid by Kenshoo to ensure they scored well, while making Google look bad. I’m not a Google fan boy by any means, but this “report” seemed to focus on Keshoos’ strengths while not scoring areas where other vendors were the industry leaders although these comparisons were highlighted in the preface of the report?

For example, some of the content, which I interpret as being overly, biased included a spurious “Foresters’ weighting” whereby they compare various key features from the software vendors. One comparison is the ‘cost and pricing structure’ of the various products; however, no score is attributed to the ‘cost and pricing structure of the vendors’. This comparison looks as though it was ignored because the Googles’ product would score full marks in this category as it’s completely free and therefore this “key comparison” has been left unsecured in its entirety.

Other issues include: scoring Google just one point out of a possible five for ‘revenue’ when compared to the other vendors. This is just wrong. Firstly because Googles’ revenue completely dwarfs the other companies in this report; however, if Google scored just a single point because its product is free then it’s evident that the scoring is completely biased towards certain vendors while at the same time, appearing hostile to others.

Other areas within this “report” that were highly questionable included how they have collected this data which was from a absurdly small and in addition very biased pool of respondents, i.e.: the vendors themselves and JUST twelve of their recommended customers! Therefore a survey of which had only seventeen respondents.

In addition to all of this, “report” also scores each vendor on “vision” and the “strength of the management team”. Did Forrester score each company based on what their own surveys or customers reported, or perhaps they scored the companies based on what their direct competitors has said about one other?

The “report” also makes very strong and seemingly biased statements such as “Kenshoo is the only Leader” and “Google is a risky bet” and in addition to all of this, the “report” is littered with grammatical errors and clumsy valueless information. For example: “software solutions exist to help scale paid search programs”. This is just repetition and was used in the opening of the report as a ‘key takeaway’ of the entire report. Seriously.

This “report” states that using Googles’ tools is a risky bet which maybe true; however, I believe the risk is far greater to base any business decision on this Forrester report and from what I’ve read recently, many other Forester reports too.

One thought on “Forrester Reports: Unreliable and Untrustworthy

  1. katy says:

    you’re such a geek.

Leave a Reply

Your email address will not be published. Required fields are marked *