Beyond Corporate Astrology: The Science and Strategy of Personality Assessments in Hiring

Read The Transcript:

We’re trying out transcribing our podcasts using a software program. Please forgive any typos as the bot isn’t correct 100% of the time.

Jason Hreha: The issue is a lot of these tests are just very ill-constructed from a psychometric point of view, and they’re also just based on bad science to begin with. That’s the issue with a lot of these assessments is a lot of these assessments out there have reliability problems and they also have predictive validity problems. Some of these assessments can be actually pretty reliable, but if you’re reliably measuring something that doesn’t matter or that doesn’t predict anything useful, who cares?

David Rice: Welcome to the People Managing People podcast. We’re on a mission to build a better world at work and help you create happy, healthy, and productive workplaces. I’m your host, David Rice. My guest today is Jason Hreha. He is the Founder and CEO of Persona. We’re gonna be talking about personality assessments and behavioral science in hiring.

Jason, welcome!

Jason Hreha: Thanks. Yeah, thanks for having me.

David Rice: Let’s start broad here. I kinda wanna get your perspective around why these tools get used and where they tend to sort of fail or succeed. So what’s the problem that we’re really trying to solve with personality assessments in the workplace and in the hiring process?

Jason Hreha: Yeah, so I would say in the workplace and in the hiring process, those are two different domains or two different problems in the hiring process. We’re just trying to get an honest or an accurate assessment of how somebody will behave and how they’ll perform over time. The big issue is that a lot of hiring methods are just not that reliable.

So for example, if you just sit down and interview somebody. I mean, if you just have a general conversation, it’s very hard to really get a holistic sense of that person and how they’re going to react or behave across a variety of different circumstances since you only get to talk to them for 30 minutes or an hour and a half.

And in general, people aren’t structured with their interviews. They tend to just have a general conversation. And if that’s the case, it also makes it hard to cross compare because if you’re asking one set of questions to one person and another set of questions to another person, it’s just hard mentally to say, okay, who’s better?

Who answered these questions better? So what personality assessments really do is they give you a holistic understanding of a person in the hiring process, and they allow you to cross compare very effectively in the workplace. I would say that the purpose is. Similar in the sense that you want to get an accurate understanding or a holistic understanding of a person, but the use case is different, right?

So that may be for helping a person gain better self-awareness so that they can work on their weaknesses or kind of enhance their strengths, et cetera. Or if you’re building a new team within your organization and you want a mixture of different personalities so that you have a very well-rounded team, you could use personality assessments for that purpose as well.

And so, different use cases, different needs, but the core. Kind of purpose overall of personality assessments is, let me get an accurate understanding of this person’s holistic set of behavioral and emotional predispositions.

David Rice: When we chatted before this, you know, I mentioned a phrase that I sometimes like, I don’t know, people use it for some reason, it makes me smile.

But when people describe these assessments, they’ll often call it like corporate astrology, right? And in discussion you said many of these assessments out there really are no better than astrology. That’s true. You know, I’m curious, what is it that drives a test scientific validity, and what should HR leaders be really looking out for when they look at these tools?

Jason Hreha: Yeah, so the problem with a lot of these assessments is a lot of them were developed in a very long time ago, and they’re just outdated, right? So that’s one problem. Another problem is that a lot of these assessments were just. Some person at some point had an idea or had a theory about human behavior or performance or whatever, and they just created a test in order to try to measure their pet theory, their armchair theory.

The problem is that a lot of these theories that in the first place, that theory that these people came up with was just incorrect. And so garbage and garbage out, right? If you just have an incorrect theory of human nature, human behavior, or outdated, right? Like a theory of human nature, human behavior.

It doesn’t matter. If you can measure that accurately and reliably, if the theory that it’s all based on is not valid then doesn’t really matter. I would say that overall, the issue is a lot of these tests are just very ill constructed from a psychometric point of view, and they’re also just based on bad science to begin with, bad theories, as I mentioned.

And so really what you wanna look for when you’re dealing with a personality assessment is there are two key variables you want to keep in mind. And they’re called reliability and predictive validity. What reliability means is. You have a person take the test today, and then you have them take it in a week or a month or six months or a year.

Are the results roughly the same? Do they not deviate that much? And then the second thing that you really want to pay attention to is predictive validity, which is, do the results actually predict things that you care about in the real world? And that’s the issue with a lot of these assessments is a lot of these assessments out there have reliability problems and they also have predictive validity problems.

So, some of these assessments can be actually pretty reliable, but once again, if you’re reliably measuring something that doesn’t matter or that doesn’t predict anything useful, who cares? And so I’d say that those are the two variables that you should always be analyzing in your head when you’re looking at an assessment.

You should try and find the research around the assessment. You should try to find the white paper and you should look to see is it reliable? What are the reliability, statistics, and then predictive validity, like what did these people do in order to show that the results are actually going to predict the behavior or the outcome of a person accurately in the future.

And I would say that on most fronts, these tests are not very good across those two different variables. So for example, like Myers Briggs. Not that reliable, doesn’t really predict things all that reliably. You can just look at, there’s been a lot of research on this over the years. I think that the big issue there is that people tend to flip types fairly frequently.

Where a person will, for example, on one version of the te, or if they take the test once, there may be an ENTP, let’s say they take it again, they can flip from E to I or from, you know, like on one of the other types as well. And so that’s an issue there. And so a test like that, for example. You can have people take it, and I think it’s okay to have people take it just to try and gain maybe some self-awareness or just because it’s fun or it’s a good team building exercise or it can spark interesting conversation.

But I wouldn’t make any serious decisions based upon that information.

David Rice: Yeah. It’s almost like they saw that people like to take those Buzzfeed tests, you know, like what butterfly or a species of you know, animal are you? And then they thought, well, can we apply this to the workplace? I agree. Like I think there’s some issue around, like you said what are you measuring?

Does it matter? And based on that, I guess my question would be like, or going further is how much weight should a hiring manager actually place on a personality test result?

Jason Hreha: I think personality by itself is pretty important. It’s not the most important thing, but it is quite important. If I just had to pull a number out of a hat, I would say something like, it should be like 25% of the weight of the hiring decisions.

Something along those lines. I think that the other most important variable is just gonna be the person’s cognitive ability. Are they generally intelligent? Can they learn new things really quickly? Are they good at solving problems they haven’t seen before? I’d say that’s the single most important thing, because if you think about what is work, no matter your role work is.

You go into an organization or you work from home and you’re presented with problems and challenges, a lot of these problems are challenges you haven’t seen before. And you have to figure out what’s going on and you have to solve it, and you have to solve it accurately, right? And that’s just like raw, intellectual horsepower or learning ability.

And so I’d say that’s the single most important variable behind that. I would probably put personality. Then after that I would put other things such as just do you like the person? Do you get along? You’re gonna have to work with them day in and day out. And you want somebody that really fits interpersonally with you and the rest of the team so that everybody enjoys working together.

So things like that I think fall in after that. But I would say that it’s probably the second most important variable. But if I had to put a percentage on it, I’d say, let’s say a quarter of the weight.

David Rice: Okay. One of the things I think is interesting too, ’cause like I talk to a lot of folks who work in recruiting.

Particularly in the tech space right now, right? You’ve got a lot of interview processes that are bloated. Yeah. It’s sort of a collaborative hiring process. Right. And so I think one of the big challenges is sort of comparing candidates across interviews because they may interact with this person really well, but not have any sort of, maybe they were just having a bad day that day, or maybe they just didn’t get the same reaction to the out of another person.

I’m curious, how do psychometric tools help solve for some of that? Because obviously like across these different interviews, it can vary how you might respond to them.

Jason Hreha: Yeah, I mean, it’s a good question. I think that it really depends on the organization and it depends on what your goals are, and it depends also on your applicant flow, et cetera, right?

So for example, at our company we get over a hundred thousand applicants a month. And so because of that, it’s like we can’t have a recruiter talk to every single person because I mean, we’d have to have the world’s biggest recruiting team, or maybe not the world’s biggest one, but a very large one. And so for us, you know, we try to understand using these sorts of tests, like in the beginning okay, is the person do we think that this would be a good fit for them?

Not only are they gonna perform on the role, but will they enjoy and find the role or like this sort of position meaningful, and you can actually ballpark that from these sorts of assessments. And so I’d say if you get a ton of application volume, if you get a ton of applicant volume, you can put this closer, these tests closer to the beginning and incorporate it that way.

If you’re not getting as much application volume, and let’s say you’re dealing with more of a leadership position, or let’s say you’re only dealing with 10 or 15 or 20 serious applicants, then I think you can incorporate it later on in the process. Let’s say towards the end, so let’s say you talk to everybody, you look at their resumes, you get references, et cetera.

You’re pretty intent on two or three. There’s two or three people that interest you a lot, but you’re just like, you know, you’re not really sure and you need a tiebreaker. Then that’s when I would bring these assessments in and say, okay, these people, all three of them seem like they’d be great, or these two people seem great.

If I flipped a coin, you know, we could choose that way. Or we could bring these assessments in as sort of a tiebreaker to understand. More accurately and more richly and more deeply how they think, what they like doing and whether or not this would just be a good fit for them. So that’s how I think about it there.

David Rice: Yeah, sort of, it kinda seems like it probably adds a layer of structure and consistency.

Jason Hreha: Yeah, exactly. So I mean that’s, I think the key problem with recruiting processes in general is that they tend to be all over the place and it makes it hard to, as I mentioned before, cross compare. And so for example, like structured interviewing.

Significantly better than unstructured interviewing, right? If you just have a bunch of conversations with the applicants that’s okay, but the best thing to do is, okay, for this role, what questions should we ask? And let’s just ask the same questions to every single applicant and have that like a structured conversation.

And then that way we, you can actually evaluate, you can compare the different answers to one another. So you can say, okay, this applicant said this, that applicant said that. That one said that. And then it makes it very easy to say, okay, actually this person had by far the best, most in depth answer that expressed mastery.

And so therefore we should pick that person versus the other two people. So I’d say that these tests bring that level of structured evaluation. They bring it to its extreme, or it makes it actually very easy to cross compare because with an interview. The data you get back is still fuzzy.

It’s a person just talking for 30 seconds or a minute or two minutes or five minutes, and then you have to read that and think about it and say, okay, well which answer is better? Whereas with these tests, if you’re dealing with a problem solving or cognitive ability test, you have a one number.

So you’re just comparing a number to a number, and with these personality assessments, you’re given a bunch of different numbers and it’s the combination and the specific relationship of all the numbers that matters. But it still is easier to compare that than just a person, like 10 people’s different two minute answers, so.

David Rice: Yeah. You mentioned leadership positions when you were speaking just a minute ago, and I know you’re building a new personality assessment that’s really focused on leadership, and I’m curious, you know, like sort of real world patterns. Have you seen what do current tests for leadership myths when it comes to executive capabilities and what gaps are you trying to address?

Jason Hreha: Yeah, it’s interesting. Yeah, so I actually haven’t seen any throughout my career. I haven’t really seen any good tests that really focus on leadership as a specific thing. And this has been a personal kind of project of mine over the last year or so, let’s say.

I’ve just been very interested in what are the different types of leaders, what makes a great leader? And then how could you therefore measure that and predict it? And I think I’m pretty close to cracking this problem. I’m not gonna claim. That I’ve solved it, that I’ve solved the problem yet, but I think I, you know, there’s a good chance that in the coming year or two I could really crack the code here.

So I’ve just been doing a lot of reading and a lot of research, and this was actually the reason I embarked on this project is I’m a leader, so I wanna understand, okay, like how could I measure my own leadership ability and my understand my strengths and weaknesses so that I could strengthen my strengths and strengthen my weaknesses so that I could just become better all around.

And over the years, you know, I’ve worked at large corporations, I’ve worked at startups, I’ve seen different types of great leaders. So I’ve seen a lot of great startup founders who are just absolutely incredible. And then I’ve seen, you know, incredible leaders at Walmart where I worked for a period of time, and I think Walmart has some of the best corporate leaders in the world.

But what I noticed when I was at startups then, when I was also at Walmart, is that. There are very different types of leaders out there. So the type of person who would just crush it at Walmart wouldn’t necessarily be good in a four person startup in a garage in San Francisco, building a totally new technology or product from scratch.

And so they’re obviously, in my mind, a very different types of leadership, and I think that you can really put them leaders into two major buckets. I think it’s a spectrum. I don’t think it’s just like you’re in this bucket or you’re in that one. I really think it’s more of a spectrum, but what I call transformational leadership and operational leadership.

This is something that in the academic world actually has been studied quite a bit, and an operational leader would be more of the, okay, you’re going into an existing organization that has existing processes and an existing product market fit. And the whole goal is just to run things really well.

Scale things, make things better. So you’re taking a preexisting machine and just making sure that it doesn’t break down and that it just keeps functioning really well and getting better and better. The other type of leader I call transformational leader. This is a term that’s also used in the academic literature and the basic idea there.

That’s like more of a Steve Jobs, somebody who comes in to a preexisting organization and says, okay, we’re building computers now. We’re gonna build this smartphone thing. We’re gonna build a phone that can connect to the internet, et cetera. They are, they revolutionize organizations. They build new product lines, they create completely new things.

De novo. A startup founder would be an example of that too. This is somebody who says, Hey, nothing exists here today. I want to build an organization and a product and everything from scratch. And if you think about it, these are two very different skill sets and they’re two very different types of people.

In order to be a transformational leader, you have to be more comfortable with, I think, chaos, disorganization, uncertainty. You have to be a bit more creative, a bit more visionary. You know, those sorts of things really matter a lot. And I mean, these are just like some kind of surface level examples of the differences.

You know, we can get very detailed here. And then with an operational leader, it’s really, a lot of it is actually, yeah, are you really organized? Are you really good at like just understanding systems, understanding how they fit together and just. Sticking to them, staying with them. Are you very good socially at understanding the preexisting culture, fitting into it, exemplifying it, and making sure that everybody feels good, et cetera?

And so I do think that these are very different types of people. And so what I’ve been trying to do is understand using the academic literature and then I’m gonna be doing my own studies as well, really understand on a very deep level, okay, how could I identify an individual’s kind of propensities or scores in each of these different areas?

Then accurately measure it so that number one, I can understand myself better. Two, I can understand just the nature of leadership and what makes people great better, just ’cause I think it’s an intellectually interesting thing. And then the eventual goal would be to help companies give this to potential executives, potential hires, like key hires and say, okay, great.

We’re building a completely new division here. We’re building a completely new business line, a completely new product. We probably need somebody who is actually, has quite a bit of operational leadership capability, but is more of a transformational leader. And so I want to help companies make these sorts of key decisions so that they, that 25% of the hiring pie, right?

I told you before that I think 25% roughly of the decision for hiring should be the personality component. I want to fill that and make it so that companies can say. Okay, we have once again three finalists. This person’s personality is way better suited for this type of leadership problem. Therefore we should probably hire this person. So I really want to de-risk that for companies.

David Rice: You know, I have to ask ’cause it’s in everything now. It’s the AI component and all this. Where does it fit in? Are we close to sort of a future where we can, I don’t know, I guess automate hiring through psychometrics and AI’s ability to maybe understand that?

Jason Hreha: So I think that psychometrics and AI are fundamentally different. ’cause AI is more probabilistic, right? Like you ask the same question to an AI three times, you’re gonna get three slightly different answers. They may be very similar, but there is still some differences in each answer. And as I was mentioning before, cross comparison is really the key advantage.

Of something like a psychometric test because you can perfectly cross compare individuals ’cause they’re more or less taking a standardized or semi standardized test. And then based upon that you’re saying, okay, this person answered this, that person answered that. It makes it very easy to cross compare and just determine who’s a better fit here.

But Theis, you’re just degrading that cross comparison ability because once again, each time the answer is slightly different. And so I actually think that psychometrics and this sort of testing will continue to live on. In all of its glory in the coming years, because once again, I think the AI can augment that or it can really help you.

It can be an extra component maybe that you add on top, like an AI analysis or using different AI evaluation tools. But I think that fundamentally these tests are just gonna continue to be just as valuable in the future. And I don’t think AI is gonna replace that. Yeah.

David Rice: Okay. What do you think companies most often get wrong when they sort of try to applied behavioral science principles to these talent discussions that they’re having. If you had to give sort of a practical critique, what would it be?

Jason Hreha: So I’ve been doing applied behavioral science work my whole career. Right. And what I’ve noticed throughout my entire career, whether or not we’re dealing with talent discussions or when not whether or not we’re dealing with product design and behavior change problems, is that the number one thing I’ve noticed throughout my whole career is that people do not do a good job.

Of thinking about behavior and getting clear about behavior. And what I mean by that in this context is, okay, let’s say you’re hiring a new leader for your, I don’t know, I’ll just pull something out. Let’s say you’re Amazon and you’re creating a new, let’s say, robotic helper division where you wanna build robots that help people, and you can sell them through Amazon or whatever.

At a high level, you’re like, okay, I, we just need a leader. We need a leader maybe with some. Engineering or some manufacturing background to come in and lead this thing. But that’s super generic and super broad. Okay, somebody with technical background and a manufacturing background. I think that’s just so broad.

It’s not very helpful in thinking about what should we measure or how should we evaluate this position. I think you always want to get as specific about the behavior and as crystal clear about the behavior as possible, and this is, once again, this is true whether or not you’re trying to change behavior or whether or not you’re just trying to measure it so that you can pick the right person.

So. What I would say is, okay, great. We want somebody who has a background in manufacturing or a background in technology creation. Well, for this position, what does this person need to do? What specifically do we want them to come in and do? Okay, well, we need them to recruit a team. So we need them to actually recruit 10 to 20 or 30 or 40 or 50 individuals.

That’s one thing that this leader needs to do. We need this leader to actually sit down and let’s say draw out the plans for the new manufacturing plant or for the new product or to, and to lead that whole process and could, so that’s the next set of tasks slash behaviors we need them to do. And then you can just build out an in-depth profile of okay, this person, if they came in tomorrow and they just executed perfectly.

What are all the activities? What are all the behaviors that they would do? And then once you have a crystal clear understanding of those activities and behaviors, then you can think very specifically about, okay, well then what should we ask? Like now we know what to ask them about. We know what to ask references about.

We know maybe the types of aptitudes or skills that they would need to have in order to execute on or perform all these different behaviors. And if we give them a personality test. We would then know what things to look for because, okay, we want the person to be able to recruit, let’s say 10 to 30 people.

If you then give them a personality assessment, you find that one of the applicants is not very social at all. They answer that they don’t really like social contact. They don’t like going to parties, they don’t you know, these sorts of things. Well, okay, maybe that person’s not quite the right fit ’cause you need this person to recruit an entire division from scratch, whereas somebody else.

If it already existed and they didn’t need to do as much recruiting, maybe it’d be okay to like give up on that or to really not select for that. And so I think that the key thing that people really need to do in general is get extremely clear on what you’re looking for. And I find that the most helpful way to do that in this sort of context is think specifically about this person’s daily activities and the responsibilities that they’re gonna have and the things that they’re gonna need to accomplish or do think at the behavioral level.

And then once you think at the behavioral level, then based upon that, you can really construct out the right evaluation process and figure out who the right person is, if that makes sense.

David Rice: Yeah, absolutely. Before we go, there’s a couple things I always like to do. First is I want to give you a chance to plug anything that you wanna plug.

It’s something you’re working on or, yeah, just where to find you and make contact with you. Learn more about what you do.

Jason Hreha: I have a company called Persona, so we’re a recruiting and staffing company and I think we do the best job of any company that I know of. Just really rigorously evaluating talent and finding just the best non-technical roles from all around the world.

We really focus on non-technical remote roles. So we work with a lot of technology companies, helping them find things like project managers, administrative assistants, executive assistants, et cetera. And so I think we do a great job there. So if you’re looking for any of those roles, I think we’d be a great fit. On top of that, you know, I do a decent amount of writing at my website.

It’s thebehavioralscientist.com. So I just post a decent amount there. If you’re interested in this sort of stuff, you can follow me on LinkedIn. My name is Jason Hreha, the last name is spelled strangely, so it’s H-R-E-H-A. It’s a weird Czechoslovakian name, so you can follow me at any of those places.

And you know, I’m gonna continue to work on cool applied behavioral science stuff for the rest of my life. I’m always thinking about personality and these sorts of things and critiquing personality assessments and talking about what’s good in this area. So if you wanna learn more about that, follow me at either my LinkedIn or my website and you can learn as much as you want there.

David Rice: Love it. Love it. Last thing that we do on each episode, I always give each guest an opportunity to ask me a question. So feel free. Shoot.

Jason Hreha: Yeah. So I mean, related to the conversation that we’ve just been having, I just wonder, you know, for you, because I know you, you just built this personality test recommendation engine or this quiz, so I’m just curious like what are your favorite tests and how have you used them?

David Rice: You know, I haven’t used them extensively. Yeah. The reason I created the quiz was to. Take what I was getting from experts and then sort of like feed that back to the audience, but also to learn about them myself because there was a bunch that I hadn’t, I didn’t know much about, right? Like I knew the Enneagram and the Myers-Briggs because I’ve been forced to do those at different times or because.

It’s like one of those things everybody became really enamored with for a while and they started popping up on the, you know, you see ’em in people’s like LinkedIn descriptions of you know, their job or who they are. You know, it’s like right next to their pronouns. I started to get interested because I, you know, there’s a lot of controversy.

You see it in different threads, and I think that the user view of it and the I should say the HR slash hiring person’s view of it versus. The candidate view of it is very different. So I found that kind of fascinating. And I’ve learned ones about different ones. You know, you had mentioned to me the ocean, I think it was called.

Yeah. Ocean. Yeah. Yeah. And Kolby was another one somebody brought up to me. And so I’m learning more about these and sort of what drives their effectiveness and why they’re effective for different things sometimes. It’s been just been fascinating to get every, everybody’s different perspective on ’em.

And like how they should be used and, ’cause there’s no standard answer. It’s yeah, no, definitely. You get one wildly skeptical answer and then you’ll get, you’ll talk to somebody else and they’re a big believer. And so it’s been just been fascinating from that perspective. And I’m interested to hear what people think when they take this quiz.

You know, did this introduce them to something that they didn’t know about? Or that they thought was you know, more effective than what they’d been using in the past, so.

Jason Hreha: Yeah. Makes sense. Okay, cool. No, that’s a good answer.

David Rice: Well, thanks for coming on today. I really appreciate it. I gotta talk about this stuff all day, so.

Jason Hreha: Cool. Yeah, no, I’m happy to talk about it anytime. And glad I could share.

David Rice: Well, listeners, if you haven’t done so already, head on over to peoplemanagingpeople.com/subscribe. Get signed up for the newsletter.

If you also haven’t done so, check out the Personality Assessment Quiz. Give it a quick go. Tell us what you think about your current use of personality assessments. I don’t collect any, you know, private information. I’m not gonna share yours, your story with the whole audience. I just wanna know generally your thoughts on how effective they’ve been for you in your hiring process and whether or not you use them. And then, yeah, take the quiz and find out which one might be a good fit for you and let me know if we were right or we were wrong.

Until next time, enjoy the summer.