Here's where it gets truly concerning. Auto research doesn't just fail to deliver on its promises — it actively harms science.
The Synthetic Science Spiral
We are already witnessing a feedback loop that should terrify everyone. Researchers use AI to generate papers. Overwhelmed reviewers use AI to write review comments. At a recent top-tier conference, an estimated 20% of submissions were AI-generated, and a significant portion of peer reviews were also drafted by LLMs. If AI writes the paper and AI approves the paper, we have removed the human from the quality control loop entirely.
And these aren't obviously bad outputs. Current AI agents fabricate experimental results with terrifying confidence — realistic-looking tissue sections, intricately fabricated citation chains, code that runs but proves nothing. The danger isn't bad research that looks bad. It's bad research that's indistinguishable from good research on the surface. The better the models get, the harder the fakes become to catch — and we're already struggling.
Future models will train on this synthetic echo chamber. The result: model collapse and a permanent cognitive pollution of the human knowledge base. We will lose the ability to distinguish between what we know and what AI told us we know.