"This paper uses Bayesian methods — an alternative to classical statistics — to reanalyze results from three studies in the HtE Demonstration and Evaluation Project, which is testing interventions to increase employment and reduce welfare dependency for low-income adults with serious barriers to employment. In interpreting new data from a social policy evaluation, a Bayesian analysis formally incorporates prior beliefs, or expectations (known as “priors”), about the social policy into the statistical analysis and characterizes results in terms of the distribution of possible effects, instead of whether the effects are consistent with a true effect of zero."
The main question addressed in the paper is whether a Bayesian approach tends to confirm or contradict published results. Results of the Bayesian analysis generally confirm the published findings that impacts from the three HtE programs examined here tend to be small. This is in part because results for the three sites are broadly consistent with findings from similar studies, but in part because each of the sites included a relatively large sample. The Bayesian framework may be more informative when applied to smaller studies that might not be expected to provide statistically significant impact estimates on their own." (p. v) (Abstractor: Author)
Full Publication Title: A Bayesian Reanalysis of Results from the Enhanced Services for the Hard-to-Employ Demonstration and Evaluation Project
Major Findings & Recommendations
“For Welfare to Work (WtW), the reanalysis suggested that the programs effects were likely to be small but positive …the reanalysis suggests that the Working toward Wellness program probably reduced depression slightly, while the original analysis indicated that the impact on depression severity was not statistically significant. The Bayesian reanalysis provided greater confidence in small impacts for two reasons. First, results were generally consistent with results from other telephonic care management studies….Second, the Bayesian reanalysis resulted in a reinterpretation of results that focused less on whether impacts were statistically significant but instead provided more information on the likely distribution of impacts. For the Center for Employment Opportunities (CEO), the reanalysis suggested that the programs effects were likely to be very small and the reductions in recidivism may have been much smaller than suggested by the published CEO results….The Bayesian reanalysis…provides a reason to be skeptical about CEOs effects on recidivism, and the ongoing replication of CEO will provide important information on the programs likely true effects….This is in part because prior evaluations of subsidized employment for welfare recipients occurred decades ago, but also because earlier studies differed in important ways”… by including more intensive training opportunities than were provided by the Philadelphia program….The transitional jobs approach may or may not have produced modest employment gains and reductions in welfare receipt in the medium term. This paper suggests three ways in which social policy evaluations might be altered. First, evaluations may want to put more effort into placing their results into the context of earlier findings….Second, evaluations may want to place less emphasis on statistical significance tests and more emphasis on the implied distribution of estimated effects.... A policymaker may benefit more from knowing there is an 85 percent chance that a programs effect is positive than from knowing the estimate is not significantly different from zero at conventional significance levels….Finally, a Bayesian framework may allow for smaller evaluations in some cases….This may be the case in deciding whether to expand a policy that has been evaluated in one set of sites or for one target population to a different location or group." (p. 41-42) (Abstractor: Author)