Editors and Researchers Beware

Various methods of calculating eligibility rates in random digital dial (RDD) surveys lead to substantially different response rates.

Journal editors consider survey response rates very important when deciding whether or not to publish an article. Response rates are particularly important—and complicated—in surveys using random digital dial (RDD). For example, how do the researchers consider the eligibility (or not) of a phone call unanswered or a respondent who hangs up?

The two most commonly used response rate calculation methods for RDD surveys—from the American Association for Public Opinion Research (AAPOR) and the Council of American Survey Research Organizations (CASRO)—treat eligibility requirements differently.

The Robert Wood Johnson Foundation’s Aligning Forces for Quality Consumer Survey of adults with chronic conditions provided these investigators data to analyze.

They calculated the AAPOR and CASRO response rates which treat cases of unknown eligibility differently. For example, AAPOR considers calls broken off in introduction to be eligible non-interviews and thus non-responses. AAPOR considers a portion of cases in which the phone rings and with no answer as eligible. CASRO assumes a proportion of both “quick hang-ups” and no answers eligible.

Average response rates for the 14 communities in the survey varied from 32 percent when calculated by AAPOR methods to 50 percent calculated by CASRO, a 55 percent difference.

"Health services researchers must consider strategies to standardize response rate reporting, enter into a dialog related to why response rate reporting is important, and begin to utilize alternate methods for demonstrating that survey data are valid and reliable," the authors conclude.