Ai2 released a study, dataset and code evaluating six LLM judges of research-idea novelty. In one experiment, changing a small part of the prompt shifted accuracy by more than 50 percentage points on the same idea pairs, while retrieval and larger reasoning budgets did not consistently improve results. Human papers were grouped using novelty assessments in ICLR reviews, but the lower-novelty labels for generated ideas are explicitly weak labels. The work demonstrates instability in particular evaluation designs, not a measure of creativity across all fields. A single LLM judge’s score is therefore an insufficient basis for assessing an automated research agent’s performance.