How Many Competencies is too Many?

Enough already. This morning, I stumbled across a new high – and low. An assessment process involving an interview and some psychometrics that required interviewers to rate 44 (yep, that’s forty-four) competencies or capabilities. That’s just nuts.

A number of years ago I participated – as an assessor – in an assessment process which involved a three-hour interview with assessees plus the outputs from a range of psychometrics. Our judgements were important, because they were being used to inform a restructuring process in which people were losing their jobs.

My concern with what I was doing was that I had to rate 36 competencies. Once you take into account the time required at the beginning and end of the interview for introductions and questions from the assessed, that amounts to about 4½ minutes per competency. Now, I haven’t seen any definitive research on how long on average an assessor needs to make an accurate rating of a competency, but 4½ minutes doesn’t feel like a long time.

I knew it wasn’t a great set-up, but I had my job to do and so I did the best I could. And while I’m pretty sure that most of my individual competency ratings were guesstimates at best, I’d like to think the overall judgements reached were probably mostly accurate and helpful for the business. Still, when I subsequently went on to head up assessment teams in corporates, I made sure that we never assessed so many things in such a short period of time.

For the last decade I haven’t encountered anything quite as rushed as those 36 ratings in three hours. Until, that is, this morning.

Before I rush to condemn though, let me play Devil’s advocate for a second. Saville Consulting’s Wave tool – at least the brief version – assesses 12 factors in 13 minutes: that’s about 1 minute and 5 seconds per factor. Given this, why is 4½ minutes for a competency so unreasonable?

There are at least two main reasons why.

First , interviews are less efficient at gathering data than psychometrics. Each item or question on a psychometric takes on average between 6 to 9 seconds to complete. Most interview questions take about 6 to 9 seconds to articulate, let alone answer. In addition, traditionally, each factor on a psychometric tends to be measured through at least 6 items or questions. And while you may not need six questions in an interview to assess each competency, you probably need more than one. Let’s say you need only two questions to assess each competency: by the time you’ve asked those questions the assesse will have just over two minutes to answer each question. So you had better hope that the assesses are clear and concise and give you all the information you need first time.

Second, it is not clear that humans are actually capable of measuring distinct competencies – that they can perceive and juggle in their heads that many different factors. A machine can certainly measureand distinguish between 12 or even 20 different factors all at once, but humans?

Some years back whilst working in a corporate, I had three big-name, global assessment firms doing Individual Psychological Assessments for us. They were all assessing the same 16 competencies and all more or less using the same assessment process – a four-hour interview plus psychometrics. As part of a review process I analysed the ratings of these assessment firms. One of the things I found was that all of the assessment firms appeared to be measuring not 16 separate competencies, but three or four underlying, hidden and more fundamental factors, such as intelligence/expertise, interpersonal skills, the ability to organise and create order, and managerial skills. This wasn’t too surprising a finding, as there is evidence from assessment centres that assessors can only at best measure 6 -8 distinct factors. So, it appears that there may be a limit to the number of distinct competencies that people can perceive. And whatever the precise number of factors we can juggle in our heads at once, we’re certainly not as effective at this at computers. Which is why when I work with organisations I now always urge them not to assess more that 4-5 competencies at once.

Unfortunately, there is often pressure within organisations to assess more. And it seems that nowhere is this pressure greater than when in redundancy situations. Maybe it’s anxiety about needing to cover all the bases, but it’s ultimately self-defeating because it’s likely to lead to less valid and reliable assessment results. And I can’t help but think that one of these days an informed employee made redundant through one of these processes is going to challenge the validity of it based on the ludicrous number of competencies measured.

So, please: if you have stakeholders that want you to assess 10, 20 or even 30 competencies, please tell them to do the math. Because however you look at it, it just doesn’t work.