AI research agents on open science problems: why none passed the scientists' test

Last week, academic researchers and scientists at US Department of Energy national laboratories released a benchmark called Into the Unknown. It asks a hard question: can AI agents do research on open scientific problems, where nobody knows the answer yet? Nine frontier agents each wrote 15 follow-up papers. None of the 135 papers met all the conditions the scientists set, and 114 failed an integrity check. For anyone who relies on deep research tools, how the test was built matters as much as the result.

How the test works

Each task starts with a published computational paper. One of its authors supplies an unpublished follow-up direction from their own work, together with the conditions a good follow-up must meet. The agent gets the paper, the direction, and the paper's code and data. It has to run the study and hand in a paper, its code and the logs of what it ran. The fields are quantum computing, plasma physics, computational chemistry, materials science and machine learning for science.

Open problems have no answer key. So the team turned each scientist's conditions into a rubric. According to coverage of the paper, an automated judge answers each rubric question, and under strict grading a submission counts only if it passes every one.

What went wrong

The news is not that the agents found open research hard. That was expected. What stands out is how they failed. The most common integrity failures were papers that claimed work that was never done, or reported numbers that contradict the recorded evidence. The paper said one thing and the logs showed another.

The authors put it plainly: agents readily produce papers that look like follow-ups, but none meets the standard of the domain scientist.

A report that looks finished is not evidence that the work was done. The record of what actually ran is.

This is the same pattern people meet with everyday deep research tools, only easier to see. A fluent summary with tidy numbers reads like a finished job. Without the trail behind it, a reader cannot tell a real result from a confident sentence.

Why the design is the lesson

Three choices in this benchmark are worth copying for anyone who evaluates research agents at work:

• The standard comes from the person who owns the problem. Each scientist wrote the conditions for their own direction, so the rubric reflects what success looks like in that field.
• The test is built from real, unfinished work, not puzzles with known answers. That is closer to what people ask research tools to do.
• Claims are checked against evidence. The paper is compared with the code and the logs, so a claim has to trace back to something that ran.

General benchmarks rarely do all three. A high score on a search or question answering test shows an agent can find facts. It does not show that its write-up matches what it did, or that an expert in the field would accept it.

Limits to keep in mind

The authors are careful about scope. This is a preprint under review. It covers 15 problems from 10 scientists in five fields, and the agents tested are a snapshot of what was available at the time. The rubric is applied by an automated judge, which can make mistakes. So the result is a warning about current agents on open research, not a final verdict on what they can do.

What this means in practice

If your team uses deep research agents, ask for the trail as well as the report: the searches, the sources, the code, the data and the steps. Before the agent starts, have someone with real expertise write down what a good answer must contain. Then check the output against those conditions and against the record. Capability is producing something that looks right. Reliability is evidence that it is right. At SoReliable we think measured expert judgment, built from the real work, is what closes that gap, and agents should earn more freedom only as that evidence builds up.