Eva Vivalt
Assistant prof in economics at the University of Toronto, research on cash transfers and evidence-based decision-making, J-PAL affiliate. evavivalt.com
- Other site: all Claude Code all the time. This site: crickets. Hmm.
- I remember visiting New Orleans and hanging out with some people doing research on the coast. There are areas where people know the water's getting close, but they're like "great, in 10 years it will be on my doorstep!" They don't seem to think about year 11. Analogy for AI?
- Blatantly lying is a poor strategy when people are not that stupid.
- I was into crypto arbitrage in its heyday. I remember the crunch for time, when every minute felt (and was) insanely valuable. The current moment, with Claude Code etc., feels like that.
- This is the clearest video analysis, lining up the different videos. The guy's feet are clear, he leans over.
- 🚨 New working paper! How well do people predict the results of studies? @sdellavi.bsky.social and I leverage data from the first 100 studies to have been posted on the SSPP, containing 1,482 key questions, on which over 50,000 forecasts were placed. Some surprising results below.... 🧵👇
- [Not loaded yet]
- That's a great question. We hope that soon there will be a few "sets" of related projects to forecast, via collaborations, which could let us look at this, but to date we have only had studies on similar topics be posted on the platform by chance, so I'd say there's not enough info quite yet.
- Together, these results show that forecast data contain real, exploitable information that can help decision-makers prioritize projects and treatment arms, benchmark expected effect sizes, and design better studies. For more results, check out the paper! www.nber.org/papers/w34493
- Several excellent RAs worked on this project, most recently Kevin Didi, Malek Hassouneh, Rohan Jha, and Francis Priestland. Kevin is a pre-doc of mine currently applying to PhD programs! He also assisted with the recent guaranteed income papers. He's great - watch out for him!
- Our results also highlight that who is placing the forecasts really matters. We see a clear gap between academics and non-academics, but field and subfield expertise don't improve accuracy in a meaningful way.
- Again, one of the advantages of this paper is that we can track individuals. There is a lot of work on, e.g., the wisdom of crowds, perhaps in part because you can estimate effects of crowds without a panel. But getting the right people to forecast matters a lot, too. (Both are important!)
- In other words, controlling for forecaster fixed effects, we regain the result that confidence has a (weak) positive correlation with accuracy. But some people are much better forecasters than others, and those who place more accurate forecasts also tend to be more uncertain.
- Our panelists are great! We have a paid forecaster panel that takes the majority of the surveys posted on the platform. And they do very well in comparison to other users.
- If you are a funding body, policymaker, or researcher, you could benefit from collecting forecasts, but shrink their effect sizes by about 1/2.
- Interestingly, high self-reported confidence is associated with lower accuracy. This is dissimilar to most of the literature. In our setting, we can track individual forecasters over time. And thus we can observe: this result is driven by overconfident forecasters.
- Many of the forecasts are thus of *causal* phenomena, like "what effect will X have on Y?" Super relevant for decision-makers trying to understand the potential impact of their actions.
- First result: forecasters tend to overestimate treatment effects - but there is a lot of signal in the forecasts made. This means that forecasts can be informative in power calculations or determining which interventions to trial.
- For context, this paper is based on the most comprehensive set of forecasts of research results, to our knowledge. It uses data from the Social Science Prediction Platform, a platform researchers use to collect forecasts of what their studies will find.
- It's really fun to be working in a cafe and overhear someone talking about basic income and your work!
- An inspired choice! I was thinking there may be a prize for innovation this year, and it is so richly deserved.
- 🎖️ The Prize in Economic Sciences is awarded to Joel Mokyr, Philippe Aghion and Peter Howitt! Over the last two centuries the world has seen sustained economic growth. This year’s laureates explain how innovation provides the impetus for further progress. #PrizeinEconomicSciences #NobelPrize
- We've updated a paper on the 3-year, $1000/month U.S. guaranteed income study. New results, in 3 figures: 🧵 1) Subjective well-being significantly improved in the treatment group in year 1, but there were no significant differences between the treatment & control group after that. 1/
-
View full threadBut you can check out results from different years in the paper. 16/
- Those are my three favorite new plots in the revised paper. Follow for more updates as we continue to put out results about this exciting program. And here is the full paper: evavivalt.com/wp-content/u... 17/17
- Also, some people have wondered about whether results were driven by the COVID-19 pandemic. We can't make conclusive statements here, but it's important to note the majority of the negative labor supply effects only materialized late. 14/
- Most people pointing to the pandemic seem to think effects would be better afterwards. If anything, during the pandemic the effects on labor supply were more muted. 15/
- As you can see, the results show a clear time trend, with impacts on employment growing over time until near the end of the program, when the gap starts to close. 12/
- Importantly, if the transfer were of a shorter duration or if we had followed participants for a shorter period of time, we might have come to very different conclusions! 13/
- Yes, there are some negative impacts on labor supply and income excluding the transfers. People also do stuff with that money. This second new figure helps illustrate the overall effects - and what doesn't move. 10/
- 3) We previously included quarterly regression results, but we obtained some updated administrative data and made some nicer plots. For example, here is an event study plot looking at employment status. 11/
- In any case, improving people's subjective well-being over the long term is hard. This isn't necessarily a fault of the intervention - it could be something about human nature or subjective well-being measures. 8/
- 2) Zooming out a bit, what can we say about the broader effects of cash transfers? Some people have focused on the negative effects of cash transfers on labor supply. Others have focused on the consumption the transfers have enabled. 9/
- But there are other things that could explain it. For example, it's possible that those receiving the transfers are not doing as well in years 2-3 as they expected. 6/
- Many experts expected the transfers - which represent 40% of baseline household income - to make a bigger difference. If participants thought similarly, they might be disappointed. Treated participants may face unexpected challenges as they make life changes. 7/
- This might be surprising, but there is a large literature on "hedonic adaptation" and that could be part of what is happening here. People who win the lottery, for example, often revert to baseline levels of happiness. Many positive shocks only temporarily improve well-being. 4/
- When I saw these results, my first reaction was that we have good data to be able to pick up this pattern. 5/
- Previously, we found similar effects on stress and psychological distress (Miller et al. 2024): declines in stress in year 1 that do not persist. 2/
- Here, we measure subjective well-being in several different ways: a measure of life satisfaction, an index of satisfaction across several domains, and a measure of "affect balance" (SPANE). All show the same trend. 3/
- Nice quote! One of the reasons for the Social Science Prediction Platform. A few of the cash transfer evaluations (including our own) collected ex ante forecasts there. @sdellavi.bsky.social
- The domestic UBI studies: — Are disappointing given the stronger results from studies of cash programs in Africa — Still bolster the case that converting SNAP, LIHEAP, Section 8, etc to cash would be good — raises Qs about priorities www.slowboring.com/p/what-cash-...
- [Not loaded yet]
-
View full threadJust a note it's not just them. We see pretty similar mixed results on children for three-year $1000/month transfers in the US, albeit not targeted at new moms: www.nber.org/papers/w34040
- Evidence for cash transfers is more positive for children is in LMIC countries. But the "it's just one study" framing here is misleading.
- Okay, now that I've used it for some work on forecasting, I have to say this is a fantastic tool! We will be changing some things in our paper as a result. Highly recommended!
- Giving a seminar today at the World Bank's Strategic Impact Evaluation Fund, pulling together insights from three papers on the evidence-to-policy pipeline! Excited to engage with this group.
- Niche, but did you know there is a TTC shop? This is such a perfect fit for my interests (maps, public transit, puzzles) that I have to share. Probably other transit systems make them, too?
- [Not loaded yet]
-
View full threadThanks! We were pretty lucky in having a good experience overall, but there were also places you aren't seeing that were very interested until they heard of the results. (And some places perhaps only interested because of a result....) The media is funny.
- The people I know on Baby's First Years are quite open about their findings, fwiw.
- A salute to this man, who brought so much entertainment and camaraderie to so many. 💔 Several generations appreciating his music. www.youtube.com/watch?v=frAE...
- [Not loaded yet]
-
View full threadNonetheless, if you were hoping that effects on children would provide a slam-dunk case for guaranteed income, you will probably be disappointed. Our results highlight that effects on children are complex. Check out the paper for more details: www.nber.org/papers/w34040 19/
- Small aside: this is the first paper I've worked on with "Certified Random" author ordering. 😀 10/10, would randomize again. Wonderful to work with this great group of people: Patrick Krause, @elizabethrds.bsky.social @smiller.bsky.social @alexbartik.bsky.social @dbroockman.bsky.social 20/20
- There are also reasons to think that effects could still turn out to be more positive over time. First, the literature suggests that effects on children might be largest when children are young. But the youngest children in our sample haven't aged into the admin data yet. 17/
- Further, sometimes treatments can have delayed benefits. If the positive effects of the transfers on parenting lead to better parent-child relationships, for example, this could lead to improvements in other outcomes in the long term. 18/
