At the Allegheny County Department of Human Services (DHS), we work with more than 400 community organizations to keep children and older adults safe from abuse and neglect, connect them to effective mental health and substance use treatment, provide housing for those without shelter and more. As stewards of more than $1 billion in public funding, we constantly scrutinize the programs and services we fund to ensure that they are most effectively helping clients make measurable progress toward their goals. We pay close attention to the data, including client feedback and surveys, and use these data to determine when it’s in clients’ best interest to modify existing programs or launch new ones. Recently, we have been increasing our commitment to the use of randomized pilots to measure the effectiveness of various strategies and determine how to make the best use of available resources. In keeping with our values of transparency and the appropriate use of data for decision-making, we want to share our thinking behind the use of pilots as an effective tool to further the Department’s goals and values while improving client outcomes.
What are randomized pilots?
When we ask, Was this program effective?, We want to know if client outcomes are better for program participants when compared to the outcomes of others not involved in the program. There are various ways to make this comparison. For example, we can compare client status before and after the program or we can compare their outcomes to clients who chose to not participate in the program. The weakness of both of these approaches is their reliance on the assumption that the groups are similar, i.e., on average, they would have performed the same if neither had received the intervention. Thus, we are left wondering, “How can we distinguish the program impact from the effects of time or characteristics of the participants?” Randomized pilots solve this problem by breaking the link between characteristics that could drive differences between groups and the program itself. In a randomized pilot, participants are assigned to different programs or policies randomly, based on the equivalent of a coin flip. These methods are standard practice in medical and drug trials to measure the effectiveness of health interventions with confidence; in recent years, they have become more popular with governments entities, nonprofit organizations and businesses.
Why start with a pilot?
Many organizations have highlighted the value of randomization (see here and here for examples). Rather than rehash their points, we expand upon three DHS core values that are supported by the use of pilots.
- Accountability and Transparency: Too often, the performance of government-led initiatives or strategies are unclear, making it impossible to measure progress or demand accountability. Change, if it does occur, is not based on credible data nor connected to measurable outcomes. Without clear information, program continuation or termination may be based on subjective decisions that might be inaccurate. In contrast, DHS wants to be held accountable and to ensure that we are funding programs that measurably improve the outcomes of the clients who rely on us for critical support, services and resources. We seek to provide clear feedback to ourselves and our stakeholders, demonstrating either that our investments are producing their intended outcome(s) and that money is being spent effectively or that the program needs to be modified or even ended. Randomized pilots promote this accountability and transparency because they are simple to understand, analyze and report on.
- Betting boldly: Pilots allow us to bet on new, unproven or controversial solutions without committing to expensive or unproven investments that don’t produce results. By getting clear data on performance of new interventions, we can make informed decisions that are in the best interest of those we serve. At DHS, pilots will never be used to generate a known answer to a question or to withhold resources for an intervention clients would otherwise be entitled to receive (e.g., publicly funded childcare subsidies).
- Continuous improvement: Given the complexity and range of the problems we face—from the opioid epidemic to community violence—we need to use every tool at our disposal to make progress. In government, as in any endeavor, it is hard to improve without feedback. Pilots are a key part of a continuous quality improvement cycle that starts with asking whether programs are effective and how they could be better. They guard against poor decision-making by providing data on an initial investments before making longer term decisions about funding and large-scale program launches.
Pilot ethics
Earning and keeping clients’ trust is essential for our work. In line with our strategic initiatives, work on algorithms and funding decisions, we will continue to strive for transparency and community engagement as we develop pilots and report on their results and ultimate funding decisions. Our commitment to transparency and
protections for clients includes:
- Seeking participants’ informed consent for sharing data and voluntary participation in all pilots.
- Targeting studies to individuals and neighborhoods that can benefit from the solution.
- Compensating clients for completing surveys and other forms of participation.
- Protecting participants’ data—study data will always be reported only in aggregate form, protecting the individual identity of all participants.
When partnering with outside academics who propose to analyze and report on the implementation and outcomes of a pilot, they will be asked to submit their analysis plan to their institutions’ Institutional Review Board (IRB), thus minimizing the risk of unintended negative outcomes and ensuring informed consent. In any case, when we implement and analyze the results of a pilot to inform decisions about launching the program, we commit to publishing the results and decisions on our website. We welcome suggestions about potential interventions where a randomized pilot can increase our understanding of the specific intervention and of the kind of interventions community members are interested in testing (contact us at DHS-Research@alleghenycounty.us). Randomized pilots have the potential to increase our data-informed program decisions and improve the quality and relevance of program strategies; they will also challenge us to make the best use of our resources. We look forward to the challenge and hope you’ll join us as we expand this continuous quality improvement strategy. We’ll update this website as we begin new pilots and report on interim and final results.