Cognitive science met computational methods sooner than many scientific domains, but hasn’t yet fully embraced *benchmarks*: Shared evaluation challenges that focus on open data and reproducible methods (doi.org/10.1162/99608f92.b91339ef). How could we get benchmarking right for cognitive science? 🤔

Cognitive science aims for more than mere prediction: We aim to build theories. Yet, evaluations in cognitive science tend to be narrow tests of a specific theory. How can we create benchmarks to make empirical validation more systematic, while preserving our goal of theory-driven cognitive science?

This GAC focuses on three debates/questions around benchmarks in cognitive science (the what, why, and how): (1) Should data or theory come first? (2) Should we focus on replication or exploration? (3) What incentives should we build up, if we choose to invest effort as a community?

Want to contribute to this debate at #CCN2025? Please come to our session today, fill out the anonymous survey (forms.gle/yDBBcBZybGjogksC8), and comment on the GAC page (sites.google.com/ccneuro.org/gac2020/gacs-by-year/2025-gacs/2025-1)! Your perspectives will shape our eventual GAC paper. 👥