What Your Agents Can’t Finish Without You
Core Shift: Upwork now sells work delivery, where systems combine people and machines to achieve outcomes rather than fill roles.
Benchmark Limits: Upwork’s automation benchmark tested selected, low-complexity jobs, so its results cannot represent most platform work.
Human Value: Human feedback improved agent completion, but required substantially more time and never produced reliable success across all jobs.
Verification Gap: Automation works best when outputs have objective checks; judgment-heavy, cross-functional work still needs accountable human ownership.
Business Test: Measure automation by accepted outputs, price expert review as production, and protect mid-level roles that develop future judgment.
Upwork has stopped calling itself a talent marketplace. Now, the term is “work delivery platform,” and the shift is more than positioning.
In a marketplace, a client hires a person. On a work delivery platform, a client states an outcome and the system decides what combination of people and machines produces it.
Andrew Rabinovich, who leads AI at Upwork, describes the unit being bought as something smaller than a job and smaller than a role. Two decades of platform data, he said at AI4, suggest that the enormous variety of work posted on Upwork decomposes into roughly 500 to 1,000 recurring tasks. He calls them "atomic units of work", basis functions that recombine into any assignment a client might post.
Create a Free Account to Keep Reading—and Keep Leading Smarter
Unlock this piece and join a community of forward-thinking leaders discovering tools, playbooks, and insights for thriving in the age of AI.
Have an account? Log In
A company that intends to sell decomposed work inherits an obligation. It has to know which decomposed pieces a machine can finish on its own, and it has to know that with enough precision to put money behind the answer.
So Upwork ran real client jobs through a benchmark, published the results as the Human+Agent Productivity Index, and documented the methodology in a paper released through arXiv in December.
The headline finding is what the agents finished. The more useful finding is what it costs to make them finish it, and who that cost turns out to be.
The Most Favorable Work to Automation
The methodology deserves more attention than the results, because it sets the ceiling on what the results can mean.
Upwork pulled 322 real jobs already posted by paying clients and already completed to client acceptance by human freelancers. Expert freelancers wrote a rubric for each one, between 5 and 20 acceptance criteria, sorted into critical, important, optional and pitfall. A job counted as complete only if every critical and important criterion passed.
Three models ran the work: Claude, Gemini and GPT-5.
The selection criteria are where an executive should slow down. Jobs were restricted to fixed-price, single-milestone contracts with clearly defined scope. Durations ran from nine hours to more than 100 days, so these weren't uniformly micro-tasks. Open-ended and complex work was excluded on the stated grounds that agents had no reasonable chance at it.
Upwork describes that excluded category as the vast majority of what runs on the platform.
One more constraint matters for anyone about to dismiss the results. The agents were kept deliberately basic. Each one read the job post and any attachments, made a plan, produced a deliverable and stopped. No web searching, no access to outside data, no additional tools and no training on the task type.
As a result, the authors note that the findings are not representative of tool-rich systems and describe their own numbers as conservative estimates.
So every figure below was measured on the cleanest work available, with the hard cases removed in advance, using agents stripped of the tooling most companies actually deploy.
Admin Support is Harder Than It Looks
Working alone, agents cleared every criterion on somewhere between 4 and 68% of jobs depending on the category and model. Web and software development sat at the top. Data science followed close behind.
Admin support finished last on average across the three models, scoring 6% on two of them.
That ordering runs directly against how most automation roadmaps get sequenced. Admin work gets targeted first, on the reasonable-sounding assumption that it is the simplest thing in the building.
The benchmark suggests that simple to describe and simple to verify are separate properties, and only the second one predicts whether an agent can finish the job.
Rabinovich's own framing shows how easy the assumption is to make. Asked about the effect on freelancers, he described low-level work, building logos, writing summaries of things, as already completely automated away. His company's benchmark scored writing between 4% and 51% depending on the model, with Gemini at the bottom of that range. The intuition and the measurement point in different directions.
Adding a round of expert feedback moved overall completion for every model. Claude went from 39.8% to 51.2% on low complexity tasks after just one human intervention. Gemini went from 19.9% to 32.3%. GPT-5 went from 19.6% to 33.5%. In absolute terms that's 11 to 14 percentage points improvement from a single human involvement.
Upwork markets the top of the relative range as a lift of up to 70%, which is accurate arithmetic on GPT-5's low baseline and worth reading alongside the absolute figures.
Category-level movement was uneven. Claude's data science completion went from 64% to 93%. Gemini's writing went from 4% to 16%. Gemini's data science didn't move at all.
Nothing reached 100%. Not even on the easiest available work, with an expert guiding it.
Verification is the Part We Haven’t Solved
Rabinovich puts the qualitative share of work on the platform above 90%, and he is direct about what that means for evaluation. Deterministic work carries its own check. A translation can be verified by another model. Code either compiles and passes tests or it doesn't. For everything else, the only available verification is a person with judgment looking at the output and deciding whether it's any good.
He described this as largely unsolved, and went further.
Solve reliable validation of qualitative output and you have more or less solved AI.
The paper's authors are careful about what their own numbers prove on this point. They did not run matched agent-only second attempts on the jobs that failed, which means the improvement can't be separated from the effect of simply letting the model try again. That's their disclosure, listed in their limitations.
What the benchmark establishes is narrower than the Upwork framing suggests. A second pass following expert feedback outperforms a first pass. How much of that belongs to the feedback and how much to the retry is unmeasured.
The ceiling is the durable finding. Under conditions selected for agent success, roughly one in five initially failed jobs, got rescued by a human-guided second attempt, and completion topped out at 51% for the strongest model. The human-guided runs also took about four times longer, 14.5 minutes against 3.6.
Upwork's own product decision points the same direction. UMA, the company's flagship AI system, was built as an orchestration agent rather than a worker. It interprets intent, decomposes it, routes pieces to humans or machines, and checks the result. It doesn't do the task.
A company with every commercial incentive to sell autonomous delivery designed its central system around the assumption that a human stays in the work.
Two Questions Before You Fund a Workflow
There’s two key questions that are important to ask about what you want agents to do.
Can the output be checked against something objective? A test suite, a reconciliation, a compliance rule, a number that either matches or doesn't.
Does the work stay inside one function and one format? Or does it cross from backend to frontend to copy to brand approval before anyone can call it done?
Objective check plus self-contained work is where autonomous agents belong. Fund it, measure it, expand it.
Objective check plus cross-functional work usually means the pieces are automatable but the seams are not. Automate the components and keep a person owning the handoffs.
Judgment-based plus self-contained is the quadrant that produced the benchmark's biggest lifts, and it's the one that gets budgeted wrong. More on that below.
Judgment-based plus cross-functional is where agents broke in every category Upwork tested. Treat it as human work with AI assistance and stop promising a headcount number against it.
Upwork's own economic modeling lands in roughly the same place. Agent-only wins on low-value tasks where the cost of failure is small, collaboration wins in the middle, and full human execution wins on high-value work where a bad output costs more than the automation saves.
The Job Upwork Chose to Show
The HAPI page includes five sample jobs with deliverables you can download. One is a lead generation task. Filter a spreadsheet of mobile apps down to the US-headquartered ones, find two to four marketing contacts at each, populate nine named columns.
Objectively checkable. Single function, single format. The rubric criteria are visible on the page and none of them require taste.
Both deliverables are marked failed evaluation. The agent working alone failed. The agent working with an expert freelancer guiding it also failed.
Upwork put that on its own marketing page, next to a claim about collaboration lifting completion by up to 70%. Give the company credit for the disclosure and then take the lesson, because that job is a close match for work companies are already handing to an agent.
The Right Person in the Loop?
Every vendor deck sells human in the loop as a safety feature. Read the evaluator criteria and a different question surfaces.
Upwork's reviewers all held a 100% Job Success Score with Top Rated or Top Rated Plus status. Collectively they had earned over $1 million on the platform across more than 96,000 hours.
That is the top of a global talent marketplace. It is also the only reviewer population tested. The paper notes that the team did not profile evaluator expertise or study how a reviewer's background changes the value of their feedback. There was no junior condition.
So every number in the study describes a best case for human oversight, and the effect of staffing that role with whoever had capacity is unmeasured by anyone.
That's my argument rather than the benchmark's, and it carries two consequences most workforce plans haven't priced.
The first is that review is production capacity, not overhead. In the judgment-based quadrant, the expert isn't a checkpoint bolted onto an automated process. The expert is the input that turns a draft into something shippable, and the four-fold time difference between the two conditions is what that input costs.
Classify it as oversight and you understate the workflow while overstating the savings. The math that justified the investment stops describing the process you actually built.
The second consequence is slower and worse. Every plan that thins mid-level roles to fund AI spending is also cutting the population that becomes senior enough to overrule a model in three years.
Expert judgment is not a credential, it's residue. It accumulates from doing the work, badly at first, with someone more experienced correcting you. Automate the entry rungs and you keep today's reviewers while removing the mechanism that produces tomorrow's.
That's a leadership problem before it's a staffing problem, and it belongs to whoever signs the automation business case.
If Agents Become Buyers, Judgment is the Product
Rabinovich's read on where this market goes next is that agents become a source of demand rather than only a source of supply. A model working through a task it can't verify, a medical question or a restaurant recommendation or anything resting on taste, hires a human in real time to check it. The human doesn't deliver the project, they simply validate the output.
That's a platform executive describing his own company's upside, and it should be weighed as such.
The workforce implication survives regardless of whether the mechanism arrives in that form. If verification is the durable human contribution, then the roles that hold value are defined by judgment about output quality rather than production volume.
Those roles currently sit scattered across seniority levels, job families and cost centers that often go unmapped, which means most organizations can't say who they are or what they'd cost to replace.
Four Moves
So what can you do to assess if you’re ready to start handing things over to agents to complete at least some of the work?
- Pull ten outputs from your most-automated workflow and put them in front of whoever used to own that work. Ask how many they would have sent to a client unchanged. That number is your real automation rate.
- For each automated workflow, name the person with authority to reject the output. Then check whether they can exercise it without escalating.
- Reclassify review time in workflows that fall in the judgment quadrant. Move it from oversight to production and rerun the business case.
- Read your vendor contracts for what completion means. Rubric-scored acceptance against every criterion and "task attempted" are different claims, and only one of them is what the benchmark measured.