Applying the Research and Evaluation Provisions of the Family First Prevention Services Act

Source: Wilson et al. (2019)

---

In recent decades, several clearinghouses have been established as repositories of evidence-based programs, including clearinghouses that focus primarily on interventions of interest to child welfare systems.[5] Although the Prevention Services Clearinghouse’s method of rating programs is informed by the methods that these other clearinghouses use, criteria and evidence ratings may differ across clearinghouses. For this reason, programs that are typically referred to as “evidence-based” may or may not satisfy all requirements of the Prevention Services Clearinghouse.

Programs with multiple variations or formal adaptations will be reviewed as separate programs by the Prevention Services Clearinghouse. For example, Multisystemic Therapy (MST) and Multisystemic Therapy – Child Abuse and Neglect (MST-CAN) were assigned separate evidence ratings as their manuals and model components differ. Program adaptations such as adding, subtracting, or significantly changing content; changing the format of service delivery; significantly changing content for cultural relevancy; and implementing the program with substantially different types of staff would all be considered new programs that require separate review by the Prevention Services Clearinghouse.

As the Prevention Services Clearinghouse continues to review interventions, the federal Children’s Bureau has released guidance that allows states to submit documentation of an independent systematic review concluding that a program not yet rated by the Prevention Services Clearinghouse qualifies as promising, supported, or well-supported. With approved documentation of an independent systematic review and all other requirements met, states can receive reimbursement (called “transitional payments”) for these services until the Prevention Services Clearinghouse completes its review of that program. However, the final evidence rating of these services will ultimately be determined by the Prevention Services Clearinghouse (U.S. DHHS, 2019a).

Tribal Title IV-E agencies are not required to adhere to the criteria for promising, supported, and well-supported prevention programs and must instead determine their own practice criteria for identifying prevention services that are relevant to their culture and context. These may include criteria such as longevity of the practice and community leader/elder approval (U.S. DHHS, 2018c).

Continuous quality improvement and evaluation requirements

For each promising, supported, or well-supported prevention program for which states are seeking federal reimbursement, the Family First Act requires states to outline (1) how they will implement the programs with fidelity using a continuous quality improvement (CQI) framework, and (2) how they will evaluate each program using a well-designed and rigorous process.

By implementing programs with fidelity, states will ensure that all key program elements associated with positive outcomes for children and families are maintained. Many program developers have created tools such as fidelity checklists to support supervisors and program administrators to regularly monitor these core elements. Used in combination with other program data, these measures feed into a cycle of continuous quality improvement through which states can strengthen prevention service delivery.

Beyond the fidelity requirements, states must develop and submit an evaluation plan for each prevention service. This requirement may be waived for well-supported interventions with documentation that the program has compelling evidence and the state will meet the CQI requirements. While the law does not define requirements for well-designed and rigorous evaluation strategies, states should consider the study quality standards established by the Prevention Services Clearinghouse to ensure that their evaluation activities will contribute to the evidence base. The Children’s Bureau has released a “tip sheet” for preparing evaluation plans. Meant to serve as a resource for states and their partners, the tip sheet outlines key components of evaluation plans and related reference materials (U.S. DHHS, 2019b). States can evaluate their prevention services or participate in a cross-site evaluation, if approved by the Administration for Children and Families (ACF).

Tribal Title IV-E agencies are afforded flexibility in their evaluation designs, with the recognition that some rigorous evaluation designs may not be feasible in the tribal context. Alternative evaluation strategies may include, for example, exploratory studies, qualitative studies, and community-based participatory research (U.S. DHHS, 2018c).

States must collect and report child-level data to the federal government. States are required to report the services that were provided to each child/family and the cost and duration of services, the placement status of each child 12 months after the prevention plan start date, whether the child entered foster care within two years of the prevention plan start date, whether the child is pregnant or parenting, and certain demographic characteristics. Some data elements are not required for pregnant or parenting youth (U.S. DHHS, 2019c).

The U.S. Department of Health and Human Services (HHS) may conduct evaluations and gather data to assess whether programs reduce foster care placements, increase kinship care, and/or improve child well-being. HHS may directly carry out this provision or enter into an agreement with external partner(s).

Kinship navigator services

When children cannot remain safely in their homes, placement with kin provides an opportunity to keep children connected to their families and culture. Kinship navigator programs support kin caregivers to learn about, find, and use programs and services for the children in their care and themselves. Kinship navigator programs use a variety of approaches to connect kin caregivers to public benefits and community resources (e.g., designated staff, peer-to-peer support, help lines). Kinship navigators also serve an important role in educating child welfare agency and community partners about the needs of kin caregivers (Casey Family Programs, 2018).

Levels of evidence

Under the Family First Act, states can receive federal reimbursement for up to 50 percent of their expenditures on kinship navigator programs that meet the criteria for promising, supported, or well-supported practice. The evidence requirements for promising, supported, and well-supported are the same as those for prevention services, with the exception of the target outcomes. Kinship navigator programs are reviewed for their effects on child safety, child permanency, child well-being, adult well-being, access to services, referral to services, and/or satisfaction with services. Beyond the evidence requirements, several other practice requirements must be met for reimbursement. For example, kinship navigator programs must coordinate with other state/local agencies; consult with kin, youth, and community stakeholders in designing and implementing programs; and establish information and referral systems, among other requirements (U.S. DHHS, 2018d). The Prevention Services Clearinghouse reviewed two kinship navigator programs in its first round of review, and neither program met the criteria.

Qualified residential treatment placements

Complementary to its focus on supporting families through prevention and kinship navigator services, the Family First Act directs federal funding to encourage agencies to place children who must be removed from their homes in families, not group settings, when appropriate. Exceptions are made for children assessed to have needs that necessitate treatment in a qualified residential treatment program, or children and youth who qualify for specified settings. This policy shift is consistent with the research evidence that children in family settings fare better across a variety of outcomes, compared to children in group settings (Annie E. Casey Foundation, 2015; Casey Family Programs, 2017). The Family First Act defines qualified residential treatment programs (QRTPs) as programs that are designed to address the needs of children with serious emotional or behavioral disorders or disturbances, using a trauma-informed treatment model. Specific requirements related to assessments, program staff, family engagement, discharge planning, aftercare, and accreditation are outlined in the law.

Evidence-based assessment tools

The Family First Act specifies several requirements for assessing the appropriateness of every placement of a child in a QRTP; these include using a validated, evidence-based assessment tool to determine the appropriate placement setting. The assessment tool must be administered by a trained professional or licensed clinician who is not employed by the state agency or affiliated with a placement setting. While this provision does not require further research on the assessment tools being used, it is an important element of the Family First Act and reinforces the focus on effectiveness.

---

Challenges and Next Steps

The Family First Act’s research and evaluation requirements ensure that children and families receive the most effective programs and services to meet their individual needs. The emphasis on evaluation, CQI, and evidence-based tools spurs child welfare stakeholders to work together to continue building the evidence base and monitoring program implementation and outcomes to ensure effectiveness.

Anticipating challenges

As with any significant change in policy and practice, states are likely to face challenges in implementing provisions of the Family First Act. Challenges for research and evaluation may include the following:

• Identifying evidence-based programs. The evidence-base for child welfare programs lags behind that of other disciplines. While programs studied on other populations may be eligible for federal reimbursement, it remains to be seen how many of them will eventually qualify as promising, supported, and well-supported. The systematic review of interventions by the Prevention Services Clearinghouse is time-intensive, with only a small number of programs meeting these criteria to date. States can seek transitional funding for programs based on their own assessment of the evidence; however, this will require independent application of complex research standards and does not guarantee continued funding if the Prevention Services Clearinghouse reaches a different rating in their subsequent review.
• Ensuring that programs produce desired outcomes. The Prevention Services Clearinghouse considers evidence from populations that do not have child welfare involvement (e.g., adults receiving substance abuse treatment or children receiving mental health services through the behavioral health system) in its reviews. It is possible that programs that are effective for other populations will not produce the same desired outcomes for children at risk of foster care entry and their families. The evaluation and continuous quality improvement requirements of the Family First Act will be essential for ensuring that such programs are indeed effective for families involved with the child welfare system.
• Identifying culturally relevant, evidence-based programs. The existing evidence for programs may come from study samples that do not match the characteristics, needs, or experiences of children and families served in a local jurisdiction. When used with different populations, services may not work the same way or achieve the same results. Additionally, programs may not be designed to serve specific subgroups of children and families, and refinements to better align these services may result in formal adaptations that require a separate review by the Prevention Services Clearinghouse. Formal adaptations will require new research that will take time to conduct and be reviewed.
• Building the evidence base for emerging programs. By requiring that programs be rated as promising, supported, or well-supported for states to receive federal reimbursement, the Family First Act incentivizes states to implement programs that already have some evidence of effectiveness. It will therefore be important to identify other funding sources to implement and continue building the evidence base for new and emerging programs and services that have not yet met the evidence requirements.
• Aligning programs with families’ needs. At the family level, states may select promising, supported, or well-supported programs that ultimately do not meet the needs of individual families. By understanding the needs and characteristics of the families they serve, states can work to build a continuum of services that provides responsive, tailored supports.
• Mobilizing support and financial resources for rigorous evaluations. Child welfare stakeholders may resist RCTs due to concerns about withholding services from some children and families; moreover, rigorous evaluations can be cost-prohibitive. While the Family First Act does specify that HHS may directly engage in or partner with other entities through grants, contracts, or interagency agreements to conduct evaluations, funding for this provision comes from the same allocation as the funding for the Prevention Services Clearinghouse, which limits availability of resources.

Next steps

For state agencies and legislators implementing the prevention, kinship navigator, and QRTP provisions of the Family First Act, key next steps related to research and evaluation include:

• Assess the current landscape of services and alignment with child and family needs. Mapping existing programs and their corresponding evidence base will help states identify existing prevention and kinship navigator programs eligible for reimbursement. Concurrently, states can work to identify gaps in the service array by analyzing the needs of children identified as “candidates for foster care” and their families and reviewing programs’ effectiveness in addressing these needs. State legislators can support these efforts by allocating resources and requesting information on the availability and evidence base of existing prevention and kinship navigator services, as well as data on child and family needs.
• Partner with researchers to assess opportunities for scaling up effective programs and building the evidence base for emerging programs. Building off the landscape analysis, states may engage research and program partners to plan next steps for each program, including identifying opportunities and resources needed to scale existing evidence-based programs and strengthening the evidence base where needed.
• Invest in new and emerging programs and corresponding rigorous evaluations. Given that federal funding can be directed to programs rated promising or higher, states should seek other funding sources to implement and evaluate new and emerging programs. Opportunities for state legislators include appropriating funds, convening stakeholders in the philanthropic and research communities to develop partnerships, requesting regular updates on program implementation and fidelity, and requesting to see the data states submit to HHS.
• Explore inter-state partnership opportunities for program training, implementation, and evaluation. Multiple states seeking to implement the same program may achieve economies of scale by jointly training providers, developing shared fidelity measures, and participating in cross-site evaluation, if approved by HHS. Partnering in these ways increases the potential for larger, more diverse study samples as well as more rigorous evaluation designs, and states can share the costs of the evaluation. To better understand these opportunities, state legislators may participate in site visits to relevant programs in surrounding states.
• Review the evidence base for assessment tools used to determine appropriateness of residential treatment. By identifying currently used assessment tools and reviewing their evidence base, states can determine whether existing tools meet the evidence requirements or need to be replaced or further studied.

For researchers partnering with states to build the evidence base, key next steps include:

• Design evaluation plans that fit local contexts. The Prevention Service Clearinghouse weighs evidence from both RCT and QED studies, which gives states and their evaluation partners more flexibility in study design if an RCT is not feasible. There may also be natural opportunities for researchers to use RCT designs as states expand services. A random assignment process ensures that all children and families have the same chance of receiving the services when program demand exceeds capacity, which often occurs with incremental expansion efforts.
• Align evaluation plans with the Prevention Services Clearinghouse standards. States are required to develop well-designed and rigorous evaluation plans for prevention services. By aligning these evaluation plans with the Prevention Services Clearinghouse standards, researchers can ensure that their findings will contribute to the evidence base.
• Identify what works and why. Through the combination of fidelity monitoring and rigorous evaluation, the child welfare community and state legislators will ultimately gain a deeper understanding of whether programs achieve their intended goals, and the mechanisms through which this occurs. This understanding can help to strengthen programs and identify new opportunities for innovation to better meet the needs of all children and families.

---

Conclusion

The Family First Act presents a significant opportunity to embrace the power of research to better serve children and families. It directs resources to what we know works best for children and families by supporting parents and kin caregivers to safely care for children in their homes and prioritizing family-based placements when children must enter the foster care system. The new requirements around evidence-based programs and tools hold great potential for improving the lives of children and families; however, significant work remains to ensure that states have the support they need to identify and implement evidence-based programs and evaluate their efforts in ways that continue to expand the evidence base. By deepening our collective understanding of what works and why, and channeling resources to proven interventions, the Family First Act contributes to ensuring that children and families receive high-quality, effective supports.

---

Glossary

Comparison study: A study that compares a group receiving an intervention with a group that receives no intervention, minimal intervention, or services as usual. Wilson et al. (2019) define two types of comparison studies as follows:

• Randomized controlled trial (RCT): “A study in which units are assigned to intervention and control conditions via a random process (e.g., a lottery)” (p. 19).
• Quasi-experimental design (QED): “A study in which units are assigned to the intervention and control conditions via a non-random process” (p. 19).

Formal adaptations: “Alternative manualized versions of the original program designed to address particular issues or populations” (Wilson et al., 2019, p. 4). Changes to a program’s content or approach (e.g., changing from group to individual treatment) are examples of formal adaptations.

Target outcomes:

• For prevention services: Child safety, child permanency, child well-being, and/or adult well-being.
• For kinship navigator services: Child safety, child permanency, child well-being, adult well-being, access to services, referral to services, and/or satisfaction with services (Wilson et al., 2019)

Usual care setting: A non-research setting where an existing provider delivers prevention or kinship navigator services. (Wilson et al., 2019)

Fidelity: Adherence to the specified components of a program.

Continuous quality improvement (CQI): An ongoing process of using data to identify and understand strengths and challenges and to test, implement, analyze, and strengthen solutions (National Child Welfare Resource Center for Organizational Improvement & Casey Family Programs, 2005).

Cross-site evaluation: An evaluation of the same program across multiple locations.

---

References

Annie E. Casey Foundation (2015). Every kid needs a family: Giving children in the child welfare system the best chance for success. Retrieved from https://www.aecf.org/m/resourcedoc/aecf-EveryKidNeedsAFamily-2015.pdf.

Bartlett, J.D., & Steber, K. (2019). How to implement trauma-informed care to build resilience to childhood trauma. Retrieved from https://hvuonowcn1ghw30ph4pi2gevw.js.wpenginepowered.com/publications/how-to-implement-trauma-informed-care-to-build-resilience-to-childhood-trauma.

Casey Family Programs (2017). What are the outcomes for youth placed in congregate care settings? Retrieved from: https://caseyfamilypro-wpengine.netdna-ssl.com/media/SF_CC-Outcomes-Resource.pdf.

Casey Family Programs (2018). What are kinship navigator programs? Retrieved from:  https://caseyfamilypro-wpengine.netdna-ssl.com/media/SF_Kinship-navigator-programs.pdf.

Children’s Defense Fund (2018). The Family First Prevention Services Act: Historic reforms to the child welfare system will improve outcomes for vulnerable children. Retrieved from https://www.childrensdefense.org/wp-content/uploads/2018/08/family-first-detailed-summary.pdf.

Child Welfare Information Gateway (2017). Child maltreatment prevention: Past, present, and future. Washington, DC: U.S. Department of Health and Human Services, Children’s Bureau. Retrieved from https://www.childwelfare.gov/pubPDFs/cm_prevention.pdf.

Child Welfare Information Gateway (2019). Long-term consequences of child abuse and neglect. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Children’s Bureau. Retrieved from https://www.childwelfare.gov/pubpdfs/long_term_consequences.pdf.

Kolko, D.J., Hurlburt, M.S., Zhang, J., Barth, R.P., Leslie, L.K., & Burns, B.J. (2010). Posttraumatic stress symptoms in children and adolescents referred for child welfare investigation. Child Maltreatment, 15(1), 48-63.

Moore, K.A. (2008). Quasi-experimental evaluations. Retrieved from https://childtrendsstg.wpengine.com/wp-content/uploads/2008/01/Child_Trends-2008_01_16_Evaluation6.pdf.

Moore, K.A. & Metz, A. (2008). Random assignment evaluation studies: A guide for out-of-school time program practitioners. Retrieved from: https://childtrendsstg.wpengine.com/wp-content/uploads/2008/01/Random-Assigment-Evaluations.pdf.

National Child Welfare Resource Center for Organizational Improvement & Casey Family Programs (2005). Using Continuous Quality Improvement to Improve Child Welfare Practice. Retrieved from: http://muskie.usm.maine.edu/helpkids/rcpdfs/CQIFramework.pdf.

Supplee, L. H. & Metz, A. (2015). Opportunities and challenges in evidence-based social policy. Social Policy Report, 28(4), 1-16.

U.S. Department of Health and Human Services. (2018a). ACYF-CB-PI-18-09: State requirements for electing Title IV-E prevention and family services and programs. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/pi1809.pdf

U.S. Department of Health and Human Services. (2018b). Attachment C to ACYF-CB-PI-18-09: HHS initial practice criteria and first list of services and programs selected for review as part of the Title IV-E Prevention Services Clearinghouse. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/pi1809.pdf

U.S. Department of Health and Human Services. (2018c). ACYF-CB-PI-18-10: Tribal Title-IV E agency requirements for electing Title IV-E prevention and family services and programs. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/pi1810.pdf.

U.S. Department of Health and Human Services. (2018d). ACYF-CB-PI-18-05: Fiscal Year 2018 funding available for developing, enhancing or evaluating kinship navigator programs. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/pi1805.pdf.

U.S. Department of Health and Human Services. (2019a). ACYF-CB-PI-16-06: Transitional payments for the Title IV-E prevention and family services programs. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/pi1906.pdf.

U.S. Department of Health and Human Services. (2019b). ACYF-CB-IM-19-04: Evaluation plan development tip sheet. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/im1904.pdf.

U.S. Department of Health and Human Services (2019c). Technical bulletin #1: Title IV-E prevention program data elements. Retrieved from https://www.acf.hhs.gov/sites/default/files/cb/tb_1_iv_e_data_elements.pdf.

Wilson, S. J., Price, C. S., Kerns, S. E. U., Dastrup, S. D., & Brown, S. R. (2019). Title IV-E Prevention Services Clearinghouse Handbook of Standards and Procedures, Version 1.0, OPRE Report # 2019-56, Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. Retrieved from https://www.acf.hhs.gov/sites/default/files/opre/psc_handbook_
v1_final_508_compliant.pdf.

---

Footnotes

[1] The services provided under the Family First Act can be categorized as tertiary prevention services, designed to support families already affected by child maltreatment. These interventions support child and family well-being by strengthening protective factors and reducing risk factors for maltreatment recurrence (Child Welfare Information Gateway, 2017). While states should consider a range of primary prevention efforts (i.e., services aimed at the entire population to prevent maltreatment before it happens) and secondary prevention efforts (i.e., services that are targeted at populations at higher risk for maltreatment), only tertiary prevention is covered under the Family First Act.

[2] The Prevention Services Clearinghouse considers a range of factors when determining study design quality, further outlined throughout the Title IV-E Prevention Services Clearinghouse Handbook of Standards and Procedures (Wilson et al., 2019). Examples of these factors include, but are not limited to, the similarity between the treatment and control groups at the start of treatment, and whether participants dropping out of the study could bias results.

[3] For a program to qualify as promising, supported, or well-supported, the Family First Act requires evidence of significant positive effects in well-designed and well-executed stud(ies). The Family First Act explicitly requires a QED or RCT study design for programs to be rated as supported or well-supported. According to the Act, promising programs may be evaluated using some form of control, with examples including an untreated group, placebo group, or waitlist study (section 471(e)(4)(C) of the Act). The Prevention Services Clearinghouse has interpreted the language of the Act to require an RCT or QED for all relevant research studies (Wilson et al., 2019).

[4] The Family First Act specifies that well-supported programs be evaluated in at least two studies (section 471(e)(4)(C)(v)). Because researchers often publish more than one analysis of the same sample, the Prevention Services Clearinghouse specifies that findings must come from two or more non-overlapping samples.

[5] Examples of related clearinghouses include the Department of Education’s What Works Clearinghouse, US DHHS’s Home Visiting Evidence of Effectiveness (HomVEE), Blueprints for Healthy Youth Development, and the California Evidence-Based Clearinghouse for Child Welfare.

[i] For a summary, see: Child Welfare Information Gateway (2019).

[ii] For more information on the design and implementation of randomized controlled trials, see Moore and Katz (2008).

[iii] For more information on the design and implementation of quasi-experimental evaluations, see Moore (2008).