“If you are a program director or administrator at a human services agency that serves low-income families, you must constantly decide how to use and invest your agency’s resources most effectively. You want to use those programs and service delivery strategies that have been shown effective at moving clients and their families toward greater self-sufficiency, stability, and well-being. Unfortunately, many programs and strategies have little or no proof of effectiveness, and the available evidence may come from studies without a strong research design or from a context different from your own. Faced with these challenges, it may seem sensible to forge ahead with the approaches and strategies you know best, even though they may lack rigorous evidence of effectiveness. The risk in this approach is that you could spend years and resources using a strategy that isn’t effective at improving outcomes for your clients and their families….
This guide is intended to help [human service agencies] introduce an accessible, reliable, and efficient approach for conducting low-burden [Randomized Controlled Trials] RCTs —sometimes known as opportunistic experiments—by identifying common situations that are well-suited for an experiment and then walking you through the research process. The guide explains the benefits of using random assignment to conduct an experiment, answers common questions and concerns, and provides practical, step-by-step guidance on how to conduct an RCT in a way that both minimizes cost and disruption and provides an opportunity to inform ongoing program improvements in a timely way” (p.4).(Abstractor: Author and Website Staff)
Major Findings & Recommendations
The guide recommends that human service agencies partner with research centers or researchers at universities to design and conduct RCTs, but notes that human service agencies play “… a critical role in identifying opportunities in time to conduct RCTs” (p.8). The guide also provides human service agencies a step-by-step guide on how to conduct an RCT: “Once you have identified a question of interest, there are several steps involved in conducting a successful RCT:…. “Step 1: Find a research partner. You may have staff in your agency who can conduct RCTs. If you do not, it can be helpful to collaborate with a research partner” (p.9). “Step 2: Identify participants. You need to identify participants—the sites, clients, or even staff (if, for example, the intervention is related to staff training)—who will serve as members of the treatment and control groups in the study” (p.9). “Step 3: Conduct and monitor random assignment. As previously emphasized, the assignment of participants to treatment and control groups needs to be random. This is not necessarily difficult: Microsoft Excel® spreadsheets have random-number generators that can be used to randomly assign groups” (p.9). “Step 4: Collect data. For some RCTs, your agency may already collect the necessary data (such as administrative records of benefits received, employment in a given month or quarter, and earnings). You should first confirm within your agency that staff would not violate any privacy protections by using these data for a study” (p.10). “Step 5: Analyze data. An important benefit of using random assignment is the simplicity of the analysis: To assess an intervention’s impact, you calculate the difference in the average outcome you are measuring between the treatment and control groups” (p.10). “Step 6: Share results with others. Research is most useful when its lessons are shared with others. Agencies can report their results to local sites through internal channels or in group forums or meetings” (p.11). (Abstractor: Author and Website Staff)