The Data Scientist's Role When AI Can Run the Analysis

AI agents now write analysis code, clean tables, and fit standard models with little supervision, doing much of the work that used to fill a data scientist's week. Is the role disappearing? No, but it is changing and will continue to evolve. A human is still needed to decide which question to ask, what the data can support, and which result deserves belief. Recent studies show where agents still fall short, and they also expose a new problem that agents create. This article covers both and describes what the job looks like as a result.

The Labor Data

The US Bureau of Labor Statistics counted about 245,900 data scientists in 2024 and projects 34 percent growth through 2034 , much faster than the average for all occupations. The agency attributes the growth to rising demand for data-driven decisions. A separate BLS analysis of AI and information technology expects increased use of AI to raise demand in some occupations and shrink others, and it lists data scientists among the occupations projected to grow. Projections are estimates and can miss, but employers show no sign yet of expecting the role to disappear.

What Agents Do Well

The AgentDS benchmark ¹, built by University of Minnesota researchers, poses 17 data science challenges across six industries, from insurance fraud to manufacturing maintenance. The authors ran an autonomous coding agent on each challenge with a 10-minute budget and no human help. The agent placed 10th among 29 human teams, inside the top third. A single-prompt GPT-4o baseline placed 17th and fell below the median. The difference between the two shows what iteration adds: the agent runs its code, reads the output, and tries again. Routine modeling on clean, well-specified problems is largely automatable now.

Where Agents Fall Short

Berkeley’s Data Agent Benchmark tested five frontier agents on realistic (i.e., messy and unspecified) enterprise data questions. On a single attempt, the best agent only answered 38% correctly. The error analysis found that 85% of wrong answers came from faulty planning or implementation, and agents rarely picked the wrong data source. The agents found the right data and then analyzed it poorly.

AgentDS shows the same pattern from the human side. Autonomous agents defaulted to a generic pipeline of standard preprocessing and tree-based models, and they missed signals in images. The strongest solutions paired humans with AI: people diagnosed why a model failed and supplied domain rules, such as comparing patient vital signs against clinical normal ranges, while the AI wrote the code². Several teams found that the feature pipelines their AI agents proposed often lowered validation scores when tested. Those teams formed their own hypotheses about the problem first and used the agent only to implement them.

Other evidence points in two directions. On well-defined problems, agents are improving quickly. OpenAI’s MLE-bench found that the best 2024 agent earned at least a bronze medal in 16.9% of 75 Kaggle competitions, and later systems report medal rates above 60% on MLE-bench Lite, a smaller set of easier competitions. AgentDS is built so that generic pipelines fail, so its results may not carry over to routine tabular work.

A New Problem: Cheap Analyses Multiply Conclusions

In “many-analyst” studies, independent human teams analyze the same dataset to test the same hypothesis and regularly reach conflicting conclusions. Two 2026 preprints find that AI agents reproduce this effect at far lower cost.

In one study , autonomous LLM analysts received the same dataset and hypothesis. Their effect sizes, p-values, and conclusions varied widely, and changing the model or the analyst persona shifted the results even after the authors screened out methodologically flawed runs. In another , agents that were assigned opposing prior beliefs reached opposing conclusions from the same data. Of those reports, 86% passed an independent AI review and 78% passed a majority vote of human experts. A reviewer reading the final write-up could not easily tell the sound analysis from the one steered by a prior.

When a defensible analysis only costs a few minutes of time, anyone can generate supporting evidence for a conclusion they already prefer. The authors of the first study propose reporting results as a distribution across plausible analytic choices and disclosing prompts alongside code and data.

What the Role Becomes

Taken together, the evidence points to four responsibilities that stay with people:

Framing

Turn a business question into a testable hypothesis, with the analysis plan written down before anyone runs it.

Domain Judgment

Supply knowledge the data does not contain, such as clinical ranges or credit rules. This is where human teams beat autonomous agents in AgentDS.

Verification

Check whether a model generalizes, whether the data came from where its owners say it did, and whether the result survives reasonable alternative analyses.

Accountability

Report the spread of defensible results and document how the analysis was produced, so decision makers can see how much the answer depends on choices.

Code production will keep getting cheaper, and benchmark scores will keep rising. An organization still needs someone who can state the question precisely, know the domain well enough to catch a plausible but wrong answer, and put their name on a conclusion after testing it against the alternatives. Nothing in the current evidence offers a cheap substitute for that judgment. The more analyses AI produces, the more that judgment is worth.