How trustworthy are assessment vendors claims?

A new article questioning the trustworthiness of business psychology research is about to be published.

One of the authors is a pioneer in this field – Michael McDaniel from Virginia Commonwealth University. McDaniel is one of the foremost researchers into what is known as ‘reporting bias’ - the tendency for people and companies to only publish positive results or ones that further their arguments or products. His work has previously covered the pharmaceutical industry, and the tendency for drug manufacturers to sometimes overstate the efficacy of their drugs. More recently he has turned his attention to the field of talent assessment and the trustworthiness of the research from psychometric test providers into the effectiveness of their tools. What makes his work particularly interesting is that he has named the vendors involved, and one of them – Hogan Assessment systems – is a very well-known test provider. In a 2005 paper, McDaniel concluded that the data provided by Hogan Assessment Systems to show the effectiveness of their HPI personality tests has “been selectively chosen for publication”.

The irony of this is that Hogan Assessment Systems are probably one of the relative good guys out there, in that at least they do actually provide some data into the efficacy of their tools. As Bob Hogan himself has noted, there are estimated to be over 2,000 test publishers in the US alone and only a minority of them engages in any proper validity studies. So only a small percentage of vendors can say with any objective authority that they know that their measurement methods genuinely work.

Measurement is, of course, a business and I understand that in this commercial environment vendors need to present themselves well. But recent research shows that reporting bias is far more prevalent than you might expect in an industry that professes to be grounded in science. At a broad level, for example, there is evidence that academic research findings are less favorable about the success of measurement than research produced by vendors. More specifically, studies have identified reporting bias by some very well-known psychometric test publishers. The publishers of one of the most globally used personality tests, for instance, state that the tool has great validity, yet a review by a respected independent body has concluded that, “The test suffers from questionable reliability and unknown validity. Its use is not recommended.”

Probably the most public example of the issue is the tale of “emotional intelligence”. In the mid-1990s the psychologist and author Daniel Goleman brought to the fore the idea that emotional skills were important for leadership success. On the back of the book came a number of tools claiming to measure emotional intelligence, and with them came claims that they could account for 80 percent of the factors that determine success.

Almost 20 years on, however, there is now overwhelming independent research showing that emotional intelligence measures are actually some of the less effective predictors of success. This does not mean that emotional intelligence is not important for leadership. It simply means that measures of it are nowhere near as good as initially claimed at predicting success. Yet if you Google these measures, you will find the same original weighty claims still being made by some big-name vendors selling them, without mention of the decades’ worth of independent research findings to the contrary.

This prevalence of biased validity figures makes the recent actions of one of the biggest test providers in the world all the more concerning. They appear to have changed their contractual terms to prevent independent research into the validity of their tests without their approval and permission. This effectively throws any kind of pretence about objective science straight out of the window.

Talent measurement is a complex, technical and all-too often impenetrable field, and knowing who and what to trust is hard enough without this sort of rubbish. So please, cut it out vendors. After all, it’s self-defeating. In your eagerness to sell us your wares, you are undermining our ability to trust you.