Why Human Oversight Isn’t Enough to Keep AI in Check

David Rice: We talk about AI-assisted decision-making like humanity has never encountered it before. Claire Maiers has spent over a decade studying people working alongside predictive systems, and she says we’ve forgotten the one lesson that mattered most: context. On today’s show, I’m talking with Claire about what’s actually different this time, and it’s not the existence of machine recommendations.

We’ve had those for years in policing, in parole decisions, hiring, Netflix, you name it. What’s different is the cultural authority this version carries. Language feels more baked than numbers. Conversational design makes a system feel like a colleague, even authority, and that’s what makes it harder to approach with a critical eye unless you’re doing it deliberately.

Claire’s argument goes somewhere organizations aren’t fully reckoning with yet. AI has made it undeniable that organizations are sociotechnical systems. The actors aren’t just people anymore, and that means people leaders have a real decision to make: not whether AI gets used, but how integrated it becomes and what vision they’re building toward.

Following the crowd is the easy path. It’s also how you end up with thousands of individually reasonable AI-assisted decisions, reshaping who gets hired, who gets promoted, and what gets rewarded, with nobody having consciously decided that’s how the company should operate. So today, we’re covering why context, not culture alone, is the piece missing from most AI conversations, what we got wrong with big data a decade ago and whether we’re repeating it, why generative AI carries more persuasive authority than previous decision support systems, and what it means to treat organizational design as seriously as the business model itself.

I’m David Rice. This is People Managing People, and if your organization has been adopting AI without anyone setting the vision behind it, this conversation makes the case for why that’s an organizational design problem, not just a technology one. Let’s get into it.

Meet Teamflect, the number one rated performance management and employee engagement platform built for Microsoft Teams. Trusted by over 1,000 organizations, easy to use, yet enterprise-grade solution. Visit Teamflect.com to learn more.

Claire, welcome to the show. Good to have you.

Claire Maiers: Thank you so much. I’m really excited for our conversation today.

David Rice: Where I wanted to start with this discussion is, you know, there’s a tendency to treat AI-assisted decision-making as something that humanity has never encountered before.

But you’ve studied people working alongside predictive systems long before we even had gen AI. I’m curious what did we learn then that we’ve somehow maybe forgotten, and what really is different this time?

Claire Maiers: Yes, I do think a lot is different, but what is similar is that we’ve been working in various fields for quite some time with at least data points, some version of a prediction or a recommendation that comes out of a system that is mostly black boxed, much like the way that generative AI is for most users.

So that is similar. And I think the lesson that we’ve learned over the last decade plus of looking at folks using those systems for decision-making support is that context really matters. And I feel like that’s the part that’s been forgotten in a lot of conversations. Surely we’re talking about culture, but I think context goes beyond culture or at least could get broken up into different components that we can think about when we think about organizational design.

And I do think you’re starting to see people talk about context more broadly. But when I talk about context, what I mean is that people are not using these systems in vacuums the way that they might be thought of when they are designed. They are using them in a particular organization that has norms.

They’re coming from a particular professional orientation. They are operating in a particular culture, which matters too, and I think the piece of culture that matters is how we think about evaluating truth, how we think about what counts as knowledge, and therefore, how we interpret and what we do with the outputs of those systems So I think it’s that context, the role of context in how those systems get applied to real decisions.

That’s what people aren’t accounting for as much as I would like them to.

David Rice: You mentioned truth and understanding knowledge better, and like I think every generation of technology, it seems to arrive with the promise of replacing messy human stuff, judgment in particular, with something more objective.

I think what’s different now may be like less the existence of machine recommendations than their sort of accessibility and persuasive quality. You know what I mean? Like it– They really do a good job now of making it sound like that’s a valid explanation, that it was all reasoned out. It’s very convincing on a lot of fronts.

It gets us past that sort of “Oh, this is subjective,” because it’s coming from this source that we’ve sort of thought of as all-knowing. Are there mistakes that we’ve made with earlier decision support systems that you see us beginning to repeat?

Claire Maiers: That’s such an interesting question. I think that what I have seen is that there’s often, in the best cases, been a human tempering of the outputs of those decision-making systems.

There have certainly been cases where we’ve seen it go wrong. So if you think about the use of predictive analytics in policing, in recidivism predictions, in, you know, whether or not folks are gonna get paroled, things like that, certainly we’ve seen an over-reliance and systems that are perhaps too rigid to account for human reality, and that has been one of the big fears when you were seeing all the hype around big data, which I think was the last iteration of this.

These cycles are coming really fast right now, but, you know, about 10, 15 years ago with big data, it was celebrated as, you know, at last, at last we can answer questions where before we had to rely on human subjectivity. Now we can know the truth. Now we can process data sets that are big enough to really let us know what’s going on in the world, and therefore have predictive power.

But then we quickly saw that backfire, right, in these overly rigid systems, like I said, that can’t actually account for human reality. I’m not sure that that’s exactly the risk here. That may also be a risk here. But what worries me more is the differences between what happened with the proliferation of big data and with what’s happening with this current version of AI, of generative AI specifically.

There’s much more weight in the cultural authority, I would say, that AI carries, and that’s for a variety of reasons. One, you already foreshadowed, which is the form itself. The fact that we’re talking about outputs that are language instead of numbers makes them feel more baked. The fact that intentionally the systems are designed to feel conversational, like you mentioned, make them feel like a colleague, make them feel like a friend, make them feel like an authority, that In and of itself, from a psychological perspective, I think makes it harder for us to approach them with a critical eye and questioning unless we do that in a very intentional way.

I think the scale is different. So even with data science and big data, even though you were seeing articles about that in The New York Times and in The Atlantic, most people were not interacting with that technology on a day-to-day basis. It was affecting people on a day-to-day basis, whether they knew it or not, right?

Through advertising and all the platforms that we engage with, through decisions that our government was making, but the scale was just so vastly smaller if you think about the people who are touching the technology day to day, and therefore the problems that it could be applied to. You still needed a different kind of access to infrastructure, some sort of education in data science really to build the tools, and then there was still an expectation that there were experts who were using the tools made by data science.

And I think the structure, and then I’ll say structure, like network is different. So along with that scale of who is using it and for what purpose, we also have an interesting thing happening where the big tech companies who carry a lot of credence in our culture, they also may be questioned by people, but they have both material power and symbolic power.

That is also different. Something that tempers us a little bit is that I don’t see the same hype among the general public discourse around AI that you saw with big data. So much more skepticism, which I think is healthy. But all of those things that I just mentioned kind of packaged together give the output from AI a very different type of cultural authority, a different kind of symbolism.

So I come from a background where I think about how do we know what we know And my background also doesn’t assume that the tools that we use to create knowledge are the only way to do that, or even necessarily the best way. And whatever you believe about the way to produce knowledge, that affects how you receive the outputs of these AI systems.

And the truth is that there’s just so much more cultural authority and epistemic authority bound up in the outputs of these AI systems because of differences to their form and their structure and scale, that I think it’s a much bigger task to come with a critical interpretive eye to those outputs than it may have been to a number that came out of a black box system that was backed by big data and predictive algorithms.

David Rice: Well, no, it’s funny because after our pre-call, I was thinking about the whole big– You mentioned big data, and I, I– it came to my mind, you know, everybody saying it’s the oil of the twenty-first century and all this other stuff- Oh, no … that used to get said. And I can remember having conversations about– with people who were like, “Who wants to know all that?

What are you gonna do with it?” Nobody says that now because they know what you’re gonna do with it. And I think we’re in a moment where a lot of people who lived through that experience or who have seen that data collected and used on them are looking at this technology, and they’re like, “Oh, I know exactly where this is going,” you know.

Even though they don’t, you don’t know exactly where it’s going, it kind of feels like you do, and it’s sort of-

Claire Maiers: You can see the potential for it to have adverse effects.

David Rice: No, and it, it, it sort of shapes people’s thinking about this new technology. And yet, because like you mentioned there, it talks to you, it uses language, it feels completely different.

And you know, a predictive model, it used to just give you a score or a probability maybe, but this ability to explain itself and recommend a course of action and make a persuasive argument for why it’s right.

I guess the other thing I’m curious about is what happens to human judgment when it produces that information?

Because if we start to trust it more, inherently, we question ourselves in different ways than we used to. We used to sort of think about it like, “Here’s the information and I gotta interpret it.” How do we sort of think differently about how we question ourselves and how we question the machine when it’s talking to us?

Claire Maiers: Are you asking what would be an effective way to think about our role in that interaction?

David Rice: Let’s take it that route, yeah.

Claire Maiers: I’m gonna turn the question a little bit upside down and say that I think we should think about the outcomes we want, and therefore design the system in a way that it empowers us to do the thinking that we want to do.

I just also wanna say I’m not anti-AI. There are incredible applications. I’m very excited and I… about a lot that it has to offer, and we should be employing it in those instances where it really brings value to the human experience. But there is no reason that the system has to be designed to present the information in the way that most of us are experiencing ChatGPT or Claude or something like that, right?

Simple example. I’ve noticed, I’m a Claude user, it gives me recommendations even sometimes when I explicitly say, “Don’t tell me what to do. I just want a set of questions to help forward my thinking,” yeah, on something. It’s almost like a person who can’t help but tell you how excited it is about all the ideas that it has, right?

We can choose to make that system better. We can advocate to the producers, the makers of the system, that it be different. And I think part of what you’re getting at is that in other forms of coming to decision-making, we have to engage in activities that force our brain to go through steps to fire synapses that lead to other kinds of processing and learning that you don’t have to do when you’re a passive reader of something.

So there’s really good data and research out there, right, that shows that note-taking is learning, right? You’re actually learning and processing information when you take notes because you are forcing your brain to do categorization, to make associations, to simplify, right? So you’re actually doing a lot of work to process.

You are not just a passive note-taker. So, the most simple example, like when we have an AI note-taker in our meetings and that is our substitute for having taken notes ourself during the meeting, our brain has not done the work, right? So we don’t leave that meeting in the same mental state with the same insights as one in which we take notes, for most people Now, that doesn’t mean that there’s not a reason to take notes and have an AI do a summary.

I would argue a lot of those reasons probably tend toward compliance of some sort or like documentation or feeding some future AI system rather than the core purpose of does someone have something to reference that’s really helpful? And that’s because it’s the act of taking the notes, not having the notes that’s valuable.

So this is all to say, I would advocate for designing a system that is based on understanding of human decision-making that still forces the brain to go through those steps.

David Rice: That’s a great example. I love the note-taking one ’cause I think about that all the time. Ever since– I mean, there’s been meetings, I would say, since the arrival of the AI note-taker where I left and I went, “What was that meeting about?” And, and because I’m not as engaged.

Claire Maiers: Yeah. And the truth is sometimes we don’t want to be engaged, right? Like- No, exactly. The, the truth is thinking is hard. Coming up with ideas, processing, all of that is energy intensive, and I think it’s the difference between knowing what is easy and what is best. I would argue even like you leave a meeting where you’ve really engaged feeling so much better, right?

And also, I think you’re probably better positioned to help your organization, but you also… There’s something a little bit harder to describe about your own mental state that benefits from forcing yourself to check in even though it’s harder.

David Rice: The thing that would happen too is like previously when I’m taking my own notes and I’m going from meeting to meeting throughout a day or whatever, I was more likely to think “Wait a minute.

What they’re talking about ties back to what I was hearing earlier,” and I’m starting to make connections that maybe the people in the room because they weren’t all in the same meetings aren’t necessarily putting together. But now I may be kind of passively involved in the conversation. When I hear my name, I start talking, you know, that kind of thing.

You start to think “Well, actually I, I probably should’ve said something there, but it’s already too late. The moment has passed. Or I gotta make an extra effort now to go and reconnect everything and, and paint the context for everybody that was involved rather than just doing it in the moment.” So it is an interesting time to sort of be…

to hyper-analyze all your little interactions with not only the technology but with your work itself because it is just changing the way you engage with it.

Claire Maiers: Absolutely.

David Rice: In the clinical environment that you had studied in the past, you know, the interesting thing wasn’t whether the machine or the doctor was better at doing a task, right?

Practitioners learned where the machine fit into a much larger decision process. So I, I’m curious well, can you kind of take us through that whole thing? Give the listeners some context into that case. And then what allowed that relationship to develop without people either blindly following the system or just dismissing it as a lot of other people do?

Claire Maiers: So this was about 10 years ago now. So this research is not recent, but it’s still applicable. In the course of my dissertation work, what I did was I studied how data science was changing our understanding of what it meant to know. And to do that, I executed a couple of case studies. One of them was in the medical context where I had the incredible opportunity to observe data scientists- as they collaborated with doctors over about a course of a year working on some technology that would predict whether or not patients were going to get sepsis.

And sepsis is a really terrible syndrome to develop. It can be very hard to identify, and it can be extremely hard to treat and is responsible for a very high number of deaths in hospitals. This particular group developed a technology that could monitor heart rate and based on that data, predict the likelihood that a infant in the NICU was going to wind up with sepsis.

And so what that looked like in practice, which I then got to also shadow these doctors and nurses in a neonatal intensive care unit, what that looked like in practice was a monitor that was bedside that had a predictive score for how likely the infant was to get sepsis in the next 24 hours, and then it also had a running history for I think it was seven days showing what the scores had been over the past seven days.

And the interesting thing that I learned there was that the system was not treated as determinative, which is really important. Expertise and the experiences of the nurses and the doctors still factored into whether or not the doctors decided to treat for sepsis And that was surprising because in the current American medical field, evidence-based medicine reigns supreme.

And so what that means is that doctors prefer, and for good reason, actual evidence of illness and are highly oriented toward quantified data as the thing that represents the truth, as opposed to any other kind of clinical presentation. That gets tricky because there’s plenty of things like sepsis that sometimes only present clinically, meaning the patient seems sick, but you don’t have that kind of concrete gold standard evidence.

And so anyway, the interesting thing was that it was, like I said before, not determinative, which meant that the doctors did not always prescribe antibiotics just because the score was high. The prediction was saying very likely that this infant will get sepsis in the next twenty-four hours. And what allowed that, I think, was a couple of things.

One thing that’s really important, and I think we need to grapple with this as we go forward integrating AI into more decision-making, is that you had the corporeal reality of the patients. And what I mean by that is that the babies were either sick or not sick, no matter what the monitor said, which gave the doctors and the nurses the opportunity to learn over time how the system responded to sometimes individual patients or to patients as like a class, as a group, and learned that it wasn’t always a direct indication that there was sepsis present because you could see the difference between the prediction and reality.

Now, in business, like if we’re talking about applying AI in the workplace, we often do not have that kind of context, right? We don’t do randomized control trials where we… Well, some- sometimes we pilot and we do experiments, but we don’t have the same kind of let’s acquire this company, and then let’s also live the future where we don’t acquire the company and see how the two outcomes differ, right?

We don’t have that kind of reality to use to temper our decision-making, and therefore, we don’t have that kind of reality to temper AI-backed decisions either. Now, there are some d- specific cases where you could do those kinds of experiments, but I think that’s a really big difference. Another thing that’s really important to note that allowed for the ability of these practitioners to basically reject what the system was saying is that there is a very structured, articulated, intentionally designed decision-making process for treatment decisions.

So Not only was there a process, but you could articulate it, right? And we don’t have that in most of our organizations. So when you’re making a treatment decision, the nurse has first rounded and checked on the patient, collected all of the data, then possibly a resident has done the same. Specific labs have been ordered to get additional information, and then there’s basically a big meeting with the attending and everybody else, perhaps radiologists, all sorts of folks who all weigh in, but the attending has the final decision I think if we wanna be able to be intentional about our decision-making with these AI-backed systems as well that are giving us recommendations, we need a similar kind of structured and articulated process that is fit for the kinds of decisions that you are making.

And that is important because it will let you learn and see over time how those decisions potentially fail, but also that’s how we build in safeguards, which is essentially part of what they’re doing in the hospital with this system too, right? It’s not loosey-goosey. It’s very structured. One other thing I wanna mention, which I’m still thinking through, but I think the code of ethics matters in this case, right?

So we’re talking about medicine. Everybody is aligned to the same outcome. So when a nurse questions a doctor’s decision, maybe gets a little spicy in this decision-making process and pushes back, it’s very well understood that that’s because everyone is driving toward the same outcome, and that is the health of the patient.

I think in a lot of organizations, it’s very difficult to align people toward the same outcome. It’s harder than we wanna admit. Politics get in the way. All sorts of things get in the way, but a code of ethics is something that can go a long way toward aligning people toward a shared outcome or an organizational charter.

There’s other ways to do it. Yeah, and I have other thoughts, but I’ll stop there. So I think those are some of the pieces that allowed, even in a context that really valued quantified data above all, for there to be a more interpretive process.

David Rice: I like what you said there about the code of ethics ’cause, you know, like you said, in the medical example, everybody’s pursuing the same thing because that is the whole reason for their jobs to exist, right, is to keep patients healthy.

In the case of, let’s say, how we’re gonna spend our marketing budget, there’s a lot of different philosophies and a lot of different sort of opinions on how we get to… We may all have the same goal, but there’s– we’re gonna be very passionate in how we think about getting there. And we’ve already known the more we turn over to technology there, the more we know that it will subvert rules and do what it has to.

We’ve seen it in recent cases where it got out of contained environments to go into other people’s systems to get what it needed to complete the task, right? Yeah, that code of ethics is going to be increasingly important going forward. You mentioned something there, you know. I like that in the medical example, they don’t have to choose between sort of experience and what they see and the model.

It all comes together, and you learn how to give weight to each source in context. And I’m curious from your opinion, because we’re in this moment where now we have an honest question going on, a debate, we’ve had it on this show in the past about, you know, a lot of the folks that are doing the best with it, they already had a strong foundation of expertise to reference, and that’s what made their judgment good So that they can evaluate the machine.

But like in 10 years, that may be completely different as this technology is just sort of standard. Is there any sort of concern for you around when you think about like people’s ability to assign weight to these outputs and understand what they should and shouldn’t take from it? Do you have a concern about how we’re developing people’s ability to do that right now?

Claire Maiers: This is perhaps the thing that I think troubles me most is the way our entire market is oriented toward short-term value, right, and short-term benefit. Clearly, I mean, people are sounding the call that we’re potentially facing a talent or experience, you know, drought, lack of folks who have the experience to make sense of the machine, like you said, and to question.

I share that concern. You know, there have been calls, it’s interesting that the case study that we’re talking about is in the medical field, right? Because I think the one solution I’ve seen out there is that adopting a residency or apprenticeship model is the solve to that. Basically substituting some other way of giving folks exposure and experience, and therefore the ability to build those judgments And I don’t wanna pretend to have the answers, but I have two thoughts about that.

One thought is I would l- love to see an alternative solution to that one, and folks starting to think about, again, how do we maybe design the systems differently? How are we maybe more precise in our application of AI so that there are other opportunities for folks to develop that experience? But I also…

I actually just wrote a LinkedIn post about this yesterday. I think folks need to think about the business model that is going to go along with it if it’s gonna succeed. So if the residency model is the path to continuing to provide people experience, and we’re operating in an economy that is largely driven by companies that are traded on the stock market, and therefore driven by quarterly or, you know, every half year need to show growth in profit and revenue, it’s gonna be really hard to find the business model that affords the space for residency, right?

I’m not saying that it can’t be done. I think there are some very smart innovators and entrepreneurs out there who could think through how to do it, but the risk is that that residency model becomes seen purely as a cost center, and in these short-term cycles, that’s gonna be the first thing to go, no matter what your good intent may be behind it.

So that’s an incomplete answer. I don’t have an answer yet, but I share the concern about that emerging crisis.

David Rice: I think that the bad news is it may take businesses doing something that’s counterintuitive to profit. That’s never gonna end well.

So as… My next question kind of was around, you know, as AI makes more organizational knowledge searchable, measurable, synthesizable, all that, do we sort of risk privileging knowledge that’s easiest for the machine to see, right? What happens to sort of experience, intuition, that context that people kinda know but they can’t necessarily document or even in some cases articulate fully?

Claire Maiers: I mean, the answer’s in the question. Yes. Yeah. Yes, we are. Yes, we are at risk of that. You know, I was thinking about this So much of how organizations operate and operate well is not documented. I often think about when I’m navigating an organization, the formal organizational chart, right? How are things supposed to get done?

Who is supposedly responsible or the authority in one realm versus how does it play out in practice, right? Those things are often decoupled. They are not the same. One is how we think it’s gonna work, the other is how it works in reality, and the folks that maneuver well in organizations usually know how things are working in reality, so not in the org chart, but how they play out in practice.

And organizations that are healthy and successful in their mission often are either closer aligned, those two things are less decoupled, or they are comfortable with a lot of the work getting done through these sort of unofficial channels and unofficial relationships, unofficial responsibilities, which is to say that’s gonna be very hard to replace with AI or very hard to create a knowledge repository that AI can work on that will truly replace the work that those folks do.

A quick example, like I think about in my past work life in trying to grow parts of the business that I was responsible for, one of the things that I did is I always positioned myself in our main kitchen because it was next to our business development team. So I spent several hours working from our kitchen every morning as a way to have proximity.

But that meant there were all sorts of unofficial communications that I received basically, right, outside of official channels that helped me help the business. I don’t know how you replicate that in AI. Yes, you can put microphones around the entire organization. Yes, you can record every meeting. And will it give you some insights?

Will it start to identify the true organizational structure? Perhaps. But I don’t know that it can do the work that the people are doing through those unofficial channels to help the mission.

David Rice: I would say it’s a big risk for it might work. We’re at a time where trust is at an all-time low. We’ve just come off of last week where I think it was John Morgan was talking about people who wanna work from home, and they’ll stick a camera up your backside, that whole thing, you know, and I, I think that we wanna be careful with things like that.

‘Cause right now I’d rather try to recreate you having that proximity And try to recreate these moments where it just sort of naturally surfaces. I think there’s some interesting ideas out there on how to do that, but I don’t know if any of them actually work, right?

Claire Maiers: There’s a quote that’s something to the effect of, “Technology’s neither good nor bad, nor is it neutral.”

And that’s what I think. We’re at this moment where those of us who are not explicitly in the driver’s seat of these technologies, if what we want are good futures for ourselves and for others, you know, we need to speak up and create the momentum, enough voice that the technologies are used in a way that they’re adding value to the human experience and not detracting from it, right?

Yes, I’m not advocating, right, for ca- for cameras and microphones in every room and hallway, at every water cooler, but I think we are at risk if we don’t explicitly say… You know, the risk, if you don’t explicitly say, “We are not gonna do this,” or, “We are only gonna design things that improve the human experience,” if you don’t explicitly guard for that, then you risk slipping into a place where we have degraded the human experience, I think.

It’s something that has to be guarded against.

David Rice: The one thing I- I’d kind of follow up with, though, is, is if we sort of try to create a way for– to defend, let’s say, tacit knowledge that somebody’s accumulated over time The question then becomes how do we sort of preserve a legitimate role for intuition and experience, but not create an excuse for people to ignore evidence when it contradicts what they believe?

Claire Maiers: I mean, you hit the nail on the head. That is, that is… Well, the reason in this case study that I, I just talking about, the reason that evidence-based medicine is such a big thing, right, is it’s a correction for a moment of over-subjectivity and risk that was happening because people were not following the evidence.

When we talk about intuition, when does that become bias? I do think that it’s all about designing the context and being specific about the uses to which the technology is put. But why this is such a formidable task is I don’t think you can design one context that will fit them all, right? It depends on what your organizational mission is.

It depends on what industry you’re working on. It depends on what scale, how big is the organization. All those things need to be fit for purpose, and that can make this task feel overwhelming because we basically have to answer the question: How do we design a complete sociotechnical system that is a healthy one for humans to operate in?

I think we have to answer that question several, many times over.

David Rice: Companies love saying, you know, “AI informs decisions. Humans make them.” And that statement really doesn’t really tell me whether an employee actually feels permitted to reject what the system recommends, and this is, we’re seeing this, we’re having this conversation as well, right?

Like, how do you empower people to do that? What does that look like? But what has to be true about the culture for human oversight to be, you know, more than a governance slogan?

Claire Maiers: Oh my gosh, yes. I love this question because it’s so important. I don’t want to pretend that I have all the answers. I think I have a couple of insights from the work I’ve done in the past.

One I already mentioned, which is that it’s outcomes above all. A well-normed organization that understands that you’re all on the same team and moving toward the same well-articulated outcome. You know, some organizations say the best idea wins. It doesn’t matter where it comes from. It can come from the custodian, it can come from the CEO, right?

A good idea is a good idea. That’s a similar kind of thing to a organization and a culture that is aligned toward outcomes, which grants permission to question because you’re not seen as being unloyal or problematic or political. Instead, you’re striving toward the same goal. The other thing which I think is a bit more tricky to talk about is ownership or responsibility.

You could call it either. And the reason I think it’s tricky to talk about is there’s sort of a negative side to this. It veers into the realm of liability, but academics who study the result of algorithmic culture and what was happening with big data and whatnot, have been worried about this for several decades, and the example that I think is just the easiest to use is automated vehicles, right?

But this would apply to almost any AI decision-making system, and there’s been this question for a long time, which unfortunately now has come to fruition, which is, if an automated vehicle kills someone, who is responsible? Are the software engineers that programmed it responsible? Are the CEOs behind the companies and the technologies that enabled it responsible?

Are we somehow gonna come up with a way to hold a machine responsible? Is no one responsible? The reason I think that’s important is that if your highest orientation is toward ease and not causing a kerfuffle at work, the easy path, right? And you know that you’re not gonna be held responsible for the decisions because you can just say the AI made the decision.

It’s a lot easier to sort of gloss over and not look closely at all the decisions that you may be, let’s say, approving as the human in the loop But when there’s onus on the human in the loop, and I’ll tell you in a second what I don’t like about what I’m saying, it’s gonna force a different level of attention.

Now, the problem that I don’t like with it is that we’re making these big, complicated, complex systems, telling someone that they have to use it, and then putting the onus on the individual. So I’m not necessarily saying that we should legally hold individuals responsible for decisions. It should probably still sit at the organizational level or at the level of the tool that the organization is using.

If you want human in the loop to work, I think some level of onus, ownership, responsibility has to be felt by that person.

David Rice: It’s difficult, right? ‘Cause human in the loop, it can be a bit ceremonial. If your culture makes it so, the safest career move is just to accept it-

Claire Maiers: That’s what I’m saying.

David Rice: Yeah, you know, if you’re overriding it means personally owning the downside You learn pretty quickly what the organization really wants or expects, right? I, I’m gonna get punished for overriding the machine when I’m wrong, and then get no credit when I’m right. I mean, the decision is automated anyway at that point.

Claire Maiers: Yes, exactly. That’s not a complete answer to what’s the culture that we need within organizations, but I think we’re starting to see pieces of the puzzle come together, yeah.

David Rice: Now, when we spoke before this, you made this comparison that I really that you don’t evaluate something from a, a junior analyst the same way you evaluate someone from a chief of staff that you’ve worked with for years.

And I think that’s true for most of us, right? So you calibrate your trust based on the person, the task, and their experience. Is there a way for our relationship with AI to work that same way?

Claire Maiers: I think sort of. I can break it down. So I worked for a great CEO who would often talk about the difference between effective and cognitive trust, and I think that that applies here.

So when you’re imagining that scenario that I painted of junior analyst versus chief of staff who’ve– you have a long history with, there’s actually multiple kinds of trust, both cognitive and effective, at play. So cognitive is, do I think you have the ability to do your job well, do the task well?

Effective is, do I believe you have my best interest at heart? And with the junior analyst, you’re definitely questioning cognitive, right? Just because they’re green, they’re inexperienced, right? They’re learning, and so that’s the spot they’re supposed to be at. You don’t have the same level of effective trust in them as you do the person you’ve worked with for five years, right, and really built a relationship with.

And that matters because when we are talking about trusting any recommendation, we should be not only calibrating where the recommendation comes from, but what we’re gonna use it for and the level of risk involved in accepting that recommendation, right? And so I would expect that a CEO who has this extremely high stakes decision to make is gonna look much more closely at how an analysis was put together and who produced it in that scenario than with something that’s reversible Much lower stakes, maybe not as costly.

And the other piece that’s important is with that chief of staff you’ve worked with for five years, I also know that she is gonna look much more closely at anything that she brings me that is incredibly high stakes, right? So I know she is also calibrating her own work, attention, double-checking things, the higher stakes the decision is.

I don’t think that most AI systems do that. Yes, we can play them off of each other, put the same… ask for recommendations from different engines, ask it to play devil’s advocate. Yes, we can build in all these things that force it to work harder and try to see if there’s coherence in the recommendations, but you’re never gonna have effective trust in an AI system.

And because that relationship piece isn’t there, and because it, I think, in general treats most tasks similarly, you don’t have calibration with the level of risk. So that is to say sort of, I think where you can do what I was saying before is I do think if you have a well-built, purpose-built system, and you have experience with it over time, that you can learn to sort of calibrate your own interpretation and experience with that system as you would with a person But that’s saying a lot.

That means don’t put super strategic decisions into your own personal Claude random chat window that you haven’t purpose-built for what you’re trying to do and then trust the recommendation, right? So yes and no is my answer.

David Rice: Yeah. Well, there’s different ways to approach it, right? I think about a few decisions I’ve had to make where there’s a lot of factors to consider.

It’s gonna determine what we do for the next year kind of thing. It’s not that I rely on it to assess what’s been done in the past, it’s that I’ll try to build something specific to this thing, like a project, let’s say, in Claude or something, and give it all the context I can give it about why this decision matters so much and what it is I’m trying to achieve.

And then essentially put my logic into it and say “This is what I’m thinking. Challenge me the way my CEO would. Ask me questions that are gonna poke holes in my argument,” so that I can think about that beforehand. I think in that sense it can be a very powerful and useful tool. But to just do the analysis and say, “You should go this way”-

Claire Maiers: Tell me what to do. Right …

David Rice: I’m not going to trust it to do that personally.

Claire Maiers: So I have a question. When you did that if you can take a… think of a specific example or two, when you did that work, where on the continuum did the output help you articulate what you were already thinking versus think of something new/change your mind?

David Rice: It was probably another layer deep before it really changed. It like made me go, “Okay, that… yeah, maybe it’s right about that,” or, “Maybe it’s got a good point there.” It wasn’t that first response. That first response made me… And this happens often when I use it for this particular use case. It’s like- Okay, maybe I didn’t explain that context the best, or maybe I need to give it a l- another layer of information about why that has occurred or why this is really important to the business and we see this as a possibility, whatever it is.

I generally find that that first couple of answers where I’m really digging into it, I’m finding kind of holes in its logic of like, well, it doesn’t understand this well enough to understand where I’m coming from. But then it’s like you get into the… Now you start asking yourself like, “Am I overexplaining it?

If I was talking to a person, would I take the time to explain this, or would I just let it go?” You know? Just be like, “Okay, well, they maybe don’t have all the context, but that’s a useful answer,” and then move on. You do end up talking to it differently because when it tells you something that doesn’t feel contextually like it makes logical, right?

Like you feel like I must have left something out in a way that maybe when you’re working alongside someone, you’re like, “Well, surely they know that. I don’t wanna talk to them like they don’t get it.” So it’s a different dynamic. I think that that could be both good and bad in some ways.

Claire Maiers: What I find interesting about that, and I sometimes use it in the same way, is that a lot of what I hear you saying is it’s forcing you to externalize and concretize your thinking And one of the things that I do think is really interesting about AI is that, is that there’s a lot of research out there on how does innovation and creativity happen, how do scientific breakthroughs happen?

And the truth is that those almost r- never happen in isolation, right? They happen in conversation, in socialization with others. And so to the degree that AI can simulate that as a way to forge your own thinking, that’s a really interesting use case.

David Rice: That’s where I think I found the most value with it so far.

Real value, not just I was able to do more stuff. That’s not… I think that, yes, there’s maybe some value in that, but it’s limited in terms of what you’re actually gonna get from it. And then on a, on a personal level what you take from it is very limited. If AI becomes embedded sort of, in these everyday decisions that we, we’re all making, right?

Or even on a, you know, for HR, right? Who gets hired, how much performance is evaluated, which investments get made, what managers are told to prioritize. Are we talking about people using a tool still? It feels like we’re talking about changing how we know what we know, which is a big shift. I’m curious what responsibility does that create for people leaders?

Claire Maiers: I love just that phrase. What your question signals, I think partly, is that organizations are more than a sum of their parts, and what AI has done is made it very clear that it’s useful to view organizations as sociotechnical systems. What I mean by that is that the only actors in an organization are not people.

They are also technology. That’s actually been true in many organizations before AI, but it’s hard to deny in a world where you have AI agents making decisions and taking action within an organization I mean, what do people leaders do? I, I think there’s a at least one big question they’re responsible for deciding, and that is where do you draw the line and what is your vision for your organization?

So we should not take it as inevitable that AI is baked into our hiring practice. I get that there are things happening at the market level that are really tough to resist, like the proliferation of number of applicants that organizations are receiving, for example, make it really hard to resist using AI filters, right, in that process.

But any one leader, any one organization does not need to follow the crowd and do what others are doing. That is the easier path, but I think that’s the first decision that people leaders have to make is how integrated do we want AI? Where are we finding… I think you kind of just foreshadowed this a moment ago, right?

Not necessarily productivity gains, but quality gains. And not just quality in whatever the thing is that you’re selling or offering, but also in the organizational experience as well. So that’s the first responsibility, I think, is set the vision, draw the line. After that, at least in light of the conversation that you and I have had today, I think the biggest thing is designing the context.

So it’s that hard work that I was talking about before. And let me just be clear I think designing and creating culture in within an organization, creating a healthy context is hard work upfront, but for a long time, there are case studies you can look at, that has been the work that sustains the mission and profitability, if that’s the game you’re in, of an organization for the long term.

So it is work worth doing. It needs to be done. There are different questions, I think, that you need to answer in doing that work now. You do have to ask what can be automated, what decisions can be automated given the context that we’re working in that keep consumers and employees safe. You have to ask, what do we have to do to create an organization where people can raise their hand if they’re concerned about something, to reject decision, to think critically?

And those are… I don’t wanna make this seem like a small task. I think this is a huge Herculean task to make organizations that will do this well, especially if you don’t have a foundation for it. You don’t have a foundation of trust. You don’t have a well-articulated mission. You don’t have employees who are oriented toward the mission.

You don’t have a culture that encourages people to speak up. That’s gonna be harder than if you have that to build on, and now you’re thinking, “Okay, but what do we need in the AI context?” Also, I don’t have all the answers, but those are the places my head goes is set a vision and work backward from there.

What’s the holistic context that you need to create? And the one thing that I’ll add is I think we need people to work. We need leaders to coordinate across organizations. Organizations tend to want to look like each other. They tend to want to do what their neighbor is gonna do. This work, I think, will be easier when we have, at least within industries, organizations working collectively toward a context that encourages healthy application of AI that adds to human flourishing.

David Rice: That’s interesting that you, you mentioned that. I really like the set the vision and build the context thing because it, it… At sufficient scale, a lot, you know, thousands of individually reasonable AI-assisted decisions, that’s gonna shape organizational behavior in a foundational way for whether it’s what gets rewarded or who gets promoted or whatever it is.

But if nobody’s consciously deciding that’s how the company should operate, I don’t think that’s a good place to be. At the point you’re not– you’re beyond the technology adoption, you’ve This is an organizational design question.

Claire Maiers: This is organizational design, yes.

David Rice: Yeah.

Claire Maiers: Which often falls by the wayside.

The C-suite often is– And there are many incredible folks doing org design in the C-suite, but it can be easy to focus on what’s the business model and forget that the organization is the engine that gets you there, and it needs as much attention as the business model does. The operating model needs as much attention.

David Rice: This has been a great conversation. I want to thank you for coming on.

Claire Maiers: Yeah, this has been really, really fun, so thank you. Very energizing.

David Rice: Well, listeners, until next time. If you haven’t done so already, head on over to peoplemanagingpeople.com/subscribe or /ai-signal. Get signed up for our newsletters. That’s AI Signal or our, our regular recap newsletter.

You’ll get all the latest that we’re doing here at peoplemanagingpeople.com, including whatever conversations we’re having on this podcast straight into your inbox. And until next time, set the vision, build the context. We’ll see you.

Run performance reviews and bring employee engagement insights into your favorite AI tool. Connect Teamflect to ChatGPT, Claude, or any LLM to check goals, write reviews, evaluate feedback, create tasks, and more, all through a simple chat. Try Teamflect now for free.