8B-Parameter Model Beats Claude Opus 4.8 at Generating Scientific Ideas

The problem: diversity in idea generation

One of the earliest stages in any research pipeline is generating ideas. Large language models (LLMs) are increasingly used to automate this step, but most work in the field focuses on the quality of individual ideas while ignoring how diverse they are. That creates a bottleneck: a model can produce high-quality suggestions that all look alike.

Methodology: set-level reinforcement learning

Researchers at Salesforce AI Research (authors: Manya Wadhwa, Pranav Narayanan Venkit, Hiroaki Hayashi, Yu Li, Tuhin Chakrabarty, Chien-Sheng Wu) took a different approach. They trained an 8B-parameter model using a set-level reinforcement learning objective — a reward function that evaluates an entire set of ideas rather than scoring each idea in isolation.

In a single generation pass, the model writes K structured idea cards, each grounded in source scientific papers (document-grounded ideation). The key design choice: when the set is scored jointly, each idea is weighted by its distance from ideas generated earlier in the same batch. That trains the model to deliberately steer away from its own prior suggestions and produce maximally diverse sets.

Results: outperforming Claude Opus and GPT-6

On a contamination-free benchmark (one that excludes data the models could have been trained on), the 8B model achieved higher creative utility — a metric combining quality and diversity — than both Claude Opus 4.8 and GPT-6 Astra.

In a pilot study with four human experts, idea sets from the 8B model were rated as just as promising as outputs from Claude Opus 4.8. The experts noted that the Salesforce model's ideas contained slightly more unique proposals.

A crucial finding: when a coding agent ran proof-of-concept experiments based on the generated ideas, the Salesforce model's success@k (the share of ideas that led to a successful implementation) was comparable to Claude Opus 4.8. In other words, the increased diversity doesn't come at the cost of practical executability — the ideas remain implementable.

Why it matters

The results show that a smaller model trained on the right objective (set-level reinforcement learning) can outperform far larger closed-source models on a specialized task — generating scientific ideas that are both diverse and high-quality. That matters for research organizations looking to automate ideation at the earliest stages of their pipeline.

The project was released by Salesforce AI Research with open source code and a contamination-free benchmark, allowing the community to reproduce and build on the results.