Your AI Agents Will Make the Same Mistakes as Your New Hire

I recently tried using AI to make advertising assets for a company. The experiment was a success, so I set about training the company’s new marketing specialist to do it.

I demonstrated and explained, and he tried to replicate it, but kept making the same mistakes. Days went by with no results, and I couldn’t sit next to him all day long.

In fact, I did exactly what I usually advise against. I decided since I’m good at this job, it’s enough to show the new person the process, and they’ll pick up my skills simply by watching. 

This belief is a common mistake, because being able to do something yourself and being able to convey the logic behind your decisions to someone else are two different things. I’d already tried combining them and it didn't work

When scaling up my company, STEP IT Academy, a global IT education network, I trained the branch directors myself. The training took about three months, and during that time I was unable to carry out my own work. Once we had extracted and structured the expert knowledge so that training no longer depended on me or other people, the training of directors was reduced to two weeks, and instead of opening 2–3 new branches a year, we were able to open dozens.

The problem of knowledge transfer now also complicates the integration of AI into work processes. An AI agent can access instructions and corporate data, but not the logic by which an experienced specialist makes effective decisions. So an AI agent will make the same mistakes as an inexperienced newcomer.

Below, I’ll explain why this is the case and how to turn your best employees’ knowledge into a company asset. 

Your Knowledge Base is Missing the Most Important Part

When you’ve been doing something for years, your actions become automatic, and it’s surprisingly difficult to answer the question "how do you do that?"

This is called tacit knowledge: a person knows more than they can explain. And this is the case for many professionals in their field.

That’s why the advice to "pair a newcomer with your best employee" doesn’t always work. The expert is pulled away from their own work, trying to explain the procedure, often without covering everything. The newcomer doesn’t gain the necessary knowledge: some things were explained to them on the fly, some they figured out for themselves, some they had to ask about, while other aspects they didn’t think of and ended up making a mistake.

As a result, both waste time, and the desired outcome isn’t achieved.

Corporate regulations and instructions set out a standard process: what to open, where to click, and to whom to send a document. But they rarely explain why an experienced specialist follows this process in one situation, yet in another immediately realizes that they need to deviate from it.

There’s no need to make the expert explain this to others over and over again. Instead, structure the logic behind their decisions so that it no longer depends on their presence. 

Let the Expert Speak

Writing things down is a skill in its own right, and it takes far more time than it seems. It’s better to conduct an interview and let the expert speak.

Send the questions in advance so that the person has time to recall practical examples. If the business operates in different niches, do not ask them to comment on each one – there will be no end to it. You need to identify 2–3 key triggers that influence an employee’s actions. For example, these might be the type of market (B2B or B2C) or the length of the deal cycle (short or long).

Record the conversation. AI can transcribe it and put together a preliminary structure, but the expert must check it and add to or correct it if necessary.

AI does a good job of organizing what has been said, but it cannot guarantee that the expert has covered all the key points. The outcome of the interview depends first and foremost on the questions. If asked "Tell me how you work," a person will reply in general terms. So what should you ask? 

Ask About Specific Actions

I have been involved in developing training programs for staff for many years and have noticed that specialists almost always describe their work in abstract terms: "analyze," "assess," "figure out." But such words do not reveal exactly what needs to be done. 

That is why it is worth pausing on each one and asking: "What exactly do you do? What is the first step? Where do you look for information?"

Let’s say a specialist says: "First, you need to assess how well the sales department is performing."

Ask: "What exactly do you do to achieve this?" And it will turn out that they open the managers’ tasks in the CRM and check the proportion of overdue ones. If more than half are overdue, this is a sign to them that the department’s work is disorganized. This sort of logic can then be passed on to someone else.

When a job involves a wide range of tasks, one of the most common mistakes is to try to describe every possible scenario straight away. In a complex business, there may be hundreds of them.

Start by looking at it the other way around: ask what remains constant regardless of the client, project or market. This helps you identify the core of the process. For a new employee, this is the foundation from which they begin their work and for an AI system, it is a set of rules that do not change with the context.

Examine Edge Cases, Not Just Ideal Process

Ask an expert what rules they follow in their work, and they’ll name two or three obvious ones. The rest they follow without realizing it and it is precisely in these that their experience lies.

These rules will emerge if you analyze differing case studies. Ask the expert to recall a project where everything went particularly well, and a situation where things nearly fell apart.

In the first instance, ask which decision or action at the outset had the greatest impact on the success. In the second, ask what went wrong and what exactly the expert did to rectify the situation.

Once, a member of my team told me how, at a general meeting, an intern sharply criticized the work of someone on the client’s side. That person shut down, and the project ground to a halt. A senior specialist had to step in, spoke to the partner in private and explained that nobody was planning to change the way they worked. On the contrary, they wanted to help them achieve the result themselves.

This leads straight to two rules: do not criticize someone’s work in public, and ensure that your work primarily benefits those whose area of responsibility it affects.

It is precisely these edge cases that are valuable to AI as well. Standard documentation most often describes the happy path, but true expertise comes to the fore when data is lacking and the standard process fails to deliver the expected result. Such scenarios can be turned into edge cases and examples against which the AI system is then tested.

Define What the Result Should Look Like

Let’s imagine you’ve conducted an interview with the company’s top expert. AI tools have helped organize it into a clear structure, and you now have a precise step-by-step guide for every new employee. How can you tell if they’ve really mastered it?

The first thing that springs to mind is a test. And that’s the worst option: it shows what the person has memorized, not what they can actually do. It’s better for a newcomer to complete a real-world task. You need to define straight away what the result should look like: a completed spreadsheet, a finished document, a configured screen in the CRM, or a recording of a conversation with a customer.

And to define the quality criteria, ask an expert. Imagine that a newcomer has handed in their work to you, but you don’t have time to read it all. What single detail would you look at to tell whether they’ve really understood it or just copied a template?

You can go even further. Ask an expert what typical mistake a newcomer makes, and incorporate it into a training scenario. See if the employee spots the problem and chooses the correct course of action.

The same principle should be applied to an AI agent. Give it real-world cases, especially those where the standard scenario doesn’t work, and compare its solution with the one an experienced specialist would have chosen.

If the agent took a different approach, this doesn’t necessarily mean the model isn’t good enough. Perhaps you failed to identify an important exception or rule during the interview. In that case, the knowledge base needs to be updated and the test repeated. 

What This Achieves

Expert knowledge needs to be formalized so that expertise is no longer tied to a single person. Once you’ve done this, onboarding will no longer depend on whether the best specialist is available: every new recruit receives the same operational logic, rather than whatever the expert manages to recall between tasks. 

The same body of expert knowledge can work in two directions – for people and for AI systems. You can provide them with decision-making rules, examples, exceptions and scenarios, which can then be used to test their performance.