Methods in Public Health Services and Systems Research

According to the largest systematic review to date of methods used in public health services and systems research (PHSSR), there is an over-reliance on cross-sectional and descriptive methodologies, which limits the field’s sophistication and progress toward its goals.

There are six PHSSR goals, all related to the linkages between characteristics of public health agencies and systems, performance, outcomes, and expenditures. But meeting these goals requires using different methodological strategies. The purpose of this review is to assess to what extent PHSSR is incorporating these necessary strategies. The reviewers evaluated 327 U.S. empirical studies included in the library of the University of Kentucky Center for PHSSR.

Key Findings:

  • More than 68 percent (224) of the studies were quantitative and another 18 percent (58) were mixed methodologically. But over 80 percent of these (228) were cross-sectional.
  • Of the 87 qualitative or methodologically “mixed” studies, 46 used a case-study approach and 53 collected data at a single point in time.
  • Over 60 percent (198) of the studies collected data on individuals, as opposed to at the program, health department, or government agency level.
  • Just 16 studies (5.7 percent) mentioned statistical “power,” (a measure of sensitivity that helps determine sufficient sample size); and, of those, only six studies reported adequate power.
  • Many studies relied on descriptive, as opposed to complex and inferential statistics.

The reviewers note the use of limited methodologies prevents PHSSR from providing externally valid, generalizable information, and that the focus on individual-level data is in conflict with PHSSR goals which are focused at the public health system level. The authors make recommendations to improve research, but note these recommendations are not new, and neither is their message that PHSSR researchers must “improve methodological strength and sophistication…in order to best answer the big questions facing the field.”