Evaluation of joint services installation pilot project and Bionet syndromic surveillance systems—United States, 2004

Introduction: Key elements that can be measured objectively to evaluate the effectiveness of a surveillance system include sensitivity, specificity, and timeliness. Statistical algorithms and data sources have been evaluated frequently; however, factors that affect whether public health professionals will use a system (e.g., acceptability, generalizability, flexibility, representativeness, and reliability) are more difficult to assess and are reported less frequently. Through the Joint Services Installation Pilot Project (JSIPP), the Department of Defense provided nine military installations with enhanced capabilities to detect and respond to weapons of mass destruction. The Department of Homeland Security is funding BioNet to improve outbreak management in San Diego, California, by integrating military and civilian information. Both programs implemented versions of the Electronic Surveillance System for the Early Notification of Community-based Epidemics (ESSENCE) IV for medical surveillance and required evaluation within these environments.

Objectives: This study measured JSIPP and BioNet users' perspectives of syndromic surveillance and the use of ESSENCE IV.

Methods: For JSIPP, registered ESSENCE users were surveyed regarding system usage, utility of features, user friendliness, and suggested improvements. For BioNet, potential ESSENCE users were identified by program administrators and surveyed before implementation to identify qualities and features they viewed as important to syndromic surveillance.

Results: The JSIPP survey response rate was only 34% (17 of 50), probably because registered but infrequent users were included in the survey population. The majority of respondents found the system easy to use and valued having access graphics and summary statistics of disease trends. Seven (41%) of 17 reported concern about the inability to obtain patient identifiers in a timely manner, which diminished their ability to investigate suspicious alerts. Barriers to system use included dislike of the layout and difficulties in interpreting nonclinical data sources. The BioNet survey response rate was 59% (13 of 22). Respondents ranked the usefulness of four elements of syndromic surveillance: 1) usefulness during an outbreak investigation; 2) ability to detect outbreaks rapidly; 3) reassurance that no ongoing outbreaks are occurring; and 4) capability to generate summary reports. Major weaknesses included 1) difficulty in interpreting and responding to alerts; 2) uncertainty in outbreak detection capability; 3) use of abstract data sources; and 4) difficulty in sustaining a system for long-term use. Additional concerns included high false-positive rates, timeliness, reliability, and cost.

Conclusion: Users often continue to have reservations about the utility of these systems. Feedback on which data sources and system features users value can help system developers direct resources for development.

Nicola Marsden-Haug, V. Foster, S. Hakre, A. Anderson, J. Pavlin

Walter Reed Army Institute of Research, Silver Spring, Maryland

Corresponding author: Nicola Marsden-Haug, Walter Reed Army Institute of Research, Division of Preventive Medicine, 503 Robert Grant Avenue, Silver Spring, MD 20910. Telephone: 301-319-7355; Fax: 301-319-9104; E-mail: nicola.marsden-haug@na.amedd.army.mil.

Disclosure of relationship: The contributors of this report have disclosed that they have no financial interest, relationship, affiliation, or other association with any organization that might represent a conflict of interest. In addition, this report does not contain any discussion of unlabeled use of commercial products or products for investigational use.