SO is an incredibly valuable and unique source of information about programming. It contains millions of questions asked, answered, & edited by people with different perspectives, curated with votes and tags. Its posts are contained in datasets used to train LLMs like the Pile.

LLMs are a pretty decent substitute for many questions we might look up or ask on Stack Overflow. But other people don't see the questions you ask ChatGPT, or the answers you get. Only OpenAI can.

In the paper we also find - heterogeneities across programming languages - no change in post quality and we present supporting evidence from the SO User Survey. We also observe further decline in posting past the point where our counterfactuals are valid.