Ethics and Empathy in Using Imputation to Disaggregate Data for Racial Equity: A Case Study Imputing Credit Bureau Data
Table of Contents
Table of Contents
Report Publish Date: July 2021
This report describes an experiment with adding racial and ethnic identifiers created through statistical calculations onto a dataset that omitted them. Based on their findings, the authors lay out ethical steps researchers must take to create data they can disaggregate.
Primary Takeaways
The authors found it important to identify and mitigate potential bias at three points in the process. They recommend:
- Auditing input data for bias before beginning the imputation process.
- Looking for the possibility of introducing bias at each step during imputation.
- When imputation is complete, asking whether the imputed race and ethnicity data are accurate enough for ethical use, depending on the researchers’ purpose.
Overview and Objectives
Disaggregating data by race and ethnicity helps researchers understand and describe the racialized systems at work in our society. But race and ethnicity are not always included in data sets, such as credit bureau data which could help researchers study racial home ownership gaps and the biased effects of using credit screens in hiring. Imputation, or adding racial and ethnic identifiers to datasets using statistical calculations, can aid researchers who want to disaggregate data. The authors of this report wanted to show how to use imputation ethically. They used the example of adding race and ethnicity markers to credit bureau data.
Hypothesis or Approach
“We used multiple imputation to add a combined race and ethnicity variable onto a 2013 dataset from a major credit bureau that represents a 2 percent random sample of adults with credit records in the United States,” the authors write.
How This Influences Change
“Despite the risks involved, data imputation can fill critical gaps in disaggregated data to give researchers, policymakers, and community leaders a chance to make more ethical and more equitable decisions,” the authors write. “To realize the wide-ranging benefits that better, more granular data offer, producers and users of imputed data must mitigate the risks by proactively centering equity in their methodological decisions.”
Research: Go Deeper
Disaggregating data by race and ethnicity is a critical method for shining light on racialized systems of privilege and oppression. As City of Austin chief equity officer Brion Oaks told the Urban Institute, “Only when the city can segment data can we see what is truly happening. Aggregates can conceal reality.” But many high-value datasets do not collect or report information on race and ethnicity. For example, such information is missing in credit bureau data, which has inhibited efforts to examine how credit scores affect racial homeownership gaps and to challenge the use of credit screens in hiring.
Imputation is a powerful tool for disaggregating data by appending racial and ethnic identifiers onto datasets lacking that information. Although failing to disaggregate data by race and ethnicity can pose considerable harm to Black people, Indigenous people, and other people of color, efforts to fill data gaps using imputation can risk the same or even greater harm, particularly if done without a proactive focus on equity.
This report describes lessons we learned from a case study in which we proactively incorporated equity in imputing race and ethnicity onto a nationally representative sample of credit bureau data. We organize these lessons around three “ethics checkpoints” where we examine our source datasets, our imputation methodology, and the resulting race and ethnicity imputations for potential racial bias and inaccuracy. At each checkpoint, we share how we approached mitigating bias where possible and transparently communicating any bias that could not be mitigated; we also discuss how to determine when the unmitigated risk is unacceptably high and therefore warrants terminating the production or use of the imputed data.
Although this report focuses on how to implement these ethics checkpoints, just as important are the researchers involved in the process and the institutional structures that hold teams accountable to the checkpoint outcomes. It is vital for researchers to engage impacted communities from the very beginning of the research process and collaborate with them at each checkpoint to identify potential risks and weigh those risks against the potential benefits of disaggregated data for their communities. Moreover, at the outset, researchers should create institutional structures, such as community advisory boards, that give community members power to affect the imputation process and hold researchers accountable for following the outcomes of the ethics checkpoints. Before beginning any imputation process, we recommend that researchers consult our ethics and empathy standards guide for guidance on creating diverse teams and accountability structures to ensure the ethics checkpoints outlined in this report yield equitable results.
Urban Institute, July 2021
Research Team
This study and report was conducted and created by the following people.
- Alena Stern
- Ajjit Narayanan
- Steven Brown
- Graham MacDonald
- LesLeigh Ford
- Shena Ashley
Similar Research to Explore
Health Data
Gathering and sharing accurate data can help communities, policymakers, journalists, and others understand the barriers to health some face and take action to overcome them. Explore public health data for analysis of the many factors that shape health.
