July 2009

Grant Results

SUMMARY

The Robert Wood Johnson Foundation (RWJF) subcontracted with two independent researchers to produce a report titled Issues in Evaluating Health Department Web-based Data Query Systems: Working Papers. A health department Web-based data query system offers users access to population health data at the state and local levels, typically including data on mortality, disease surveillance and maternal and child health.

Key Findings
According to the report:

  • A survey of all 27 state health departments with operating Web-based data query systems found that 19 had formally or informally assessed user or agency needs prior to developing their systems while 24 had done so during or after their development. Only three states (California, New Hampshire and Rhode Island) had a budget specifically for evaluation.
  • Despite a need for evaluation to keep pace with the development and implementation of Web-based data query systems, most states do not devote adequate resources to this task. States that did conduct evaluations found the process useful for improving their efforts, according to the researchers.
  • In order to invest sensibly in the development and refinement of Web-based data query systems, health departments must clarify their purpose, identify their users and place the system within the context of an overall health department information strategy.

Funding
RWJF provided $153,895 to fund this project.

 See Grant Detail & Contact Information
 Back to the Table of Contents


THE PROBLEM

Web-based data query systems sponsored by health departments offer users access to population health data at the state and local levels. Available since the late 1990s, these systems are the primary means by which public health and health care professionals, advocacy groups, policy-makers and the general public gain access to data on mortality, disease surveillance, maternal and child health and other health-related topics.

Despite their importance as tools for assessing community health and setting local public health priorities, no evaluation of a state Web-based data query system had been published in a peer-reviewed journal, according to the researchers. Without evaluation data health departments cannot make evidence-based decisions about funding, developing or revising these systems.

 Back to the Table of Contents


THE PROJECT

RWJF's Public Health team conducted a collaborative project with two independent researchers to assess evaluation activities among states with Web-based data query systems and develop evaluation guidelines.

They issued subcontracts to the researchers, Daniel J. Friedman, Ph.D., and R. Gibson Parrish, M.D., to conduct the assessment, select contributing authors and produce a report. Friedman, of Population and Public Health Information Services, is an expert on population and public health infrastructure development and an adjunct associate professor at the University of Massachusetts Amherst School of Public Health and Health Sciences. Parrish is also an expert on population health information systems and an adjunct associate professor at the Dartmouth Medical School.

An advisory workgroup convened by the RWJF Public Health team—consisting of representatives of national agencies and organizations that work with data, state and local health department staff and academics—convened two meetings to assist the researchers.

The team produced Issues in Evaluating Health Department Web-based Data Query Systems: Working Papers. The report opens with an introductory paper explaining the importance of evaluating these systems. The remaining papers are divided into four content areas:

  • Conceptual issues, including:
    • Definitions and classifications to help health departments plan evaluations of their Web-based data query systems.
    • Logic models, which visually describe the sequence of events needed to accomplish a goal.
    • Perspectives from 12 key interviews with experts in population health information systems, former state health officers and frequent users of data query systems.
  • Methods of evaluation, including:
    • An examination of the range of possible indicators.
    • A scientific approach to designing and improving systems.
    • Statistical issues in using Web-based data query systems.
  • Survey findings from the 27 state health departments operating Web-based data query systems.
  • Examples of evaluation efforts in three states: California, Missouri and Rhode Island.

 Back to the Table of Contents


FINDINGS

Findings and qualitative observations of the contributing authors reported in Issues in Evaluating Health Department Web-based Data Query Systems: Working Papers included:

  • Surveys of the 27 state health departments with Web-based data query systems revealed that:
    • Although most managers said the purpose of the system was being achieved, its intended users were being reached and their questions were being answered, the data from which these impressions were drawn were generally limited and informally gathered.
    • Nineteen states had either formally or informally assessed user or agency needs prior to developing their systems while 24 did so during or after development. Only three states (California, New Hampshire and Rhode Island) had a budget for evaluation.
    • One-third of system managers said they planned to replace their systems, but those decisions appeared to be based on their own informal assessments, rather than a formal evaluation or an informal evaluation conducted by others.
  • Despite a need for evaluation to keep pace with the development and implementation of Web-based data query systems, most states do not devote adequate resources to this task. The states that did conduct evaluations found the process useful in improving their Web sites, according to the researchers.
  • In order to invest sensibly in the development and refinement of Web-based data query systems, health departments must clarify their purpose, identify their users and place the system within the context of an overall health department information strategy.
  • Evaluation should occur at all stages of the development of a Web-based data query system:
    • Prior to development, evaluation should include needs analysis and development of a formal business case.
    • During development, evaluation requires testing to ensure that queries return correct answers.
    • Following development and public release, an impact evaluation should occur.
    • As functions and modules are added, additional needs analysis, testing or impact evaluation may be necessary.
  • Empirical evaluation methods can include surveys, focus groups, laboratory-based user testing, review of use logs, e-mails and telephone calls from users and expert review.

 Back to the Table of Contents


AFTER THE GRANT

The project activities ended with the grant.

The researchers served as guest editors and authors of the January/February 2009 issue of the Journal of Public Health Management and Practice (see the Bibliography), which examined the evaluation of community health assessments and touched on some of the issues raised in the RWJF study.

 Back to the Table of Contents


GRANT DETAILS & CONTACT INFORMATION

Project

Evaluation of health department Web-based data query systems

Grantee

Robert Wood Johnson Foundation (Princeton,  NJ)

  • Amount: $ 153,895
    Dates: March 2007 to August 2008
    ID#:  060228

Contact

Daniel J. Friedman, Ph.D.
(617) 232-6710
danieljfriedman@verzon.net
R. Gibson Parrish, M.D.
(802) 592-3358
gib.parrish@gmail.com

 Back to the Table of Contents


BIBLIOGRAPHY

(Current as of date of this report; as provided by grantee organization; not verified by RWJF; items not available from RWJF.)

Articles

Friedman DJ and Parrish RG (issue eds). "Evaluating Community Health Assessments" (issue focus). Journal of Public Health Management and Practice, 15(1): 1–81, January/February 2009. Journal is available for purchase. Table of contents is available online.

Reports

Friedman DJ and Parrish RG (eds). Issues in Evaluating Health Department Web-based Data Query Systems: Working Papers. Robert Wood Johnson Foundation, 2008. Available online.

 Back to the Table of Contents


Report prepared by: Nina Berlin
Reviewed by: Karyn Feiden
Reviewed by: Molly McKaughan
Program Officer: Russell A. Brewer

Most Requested