Do false positive alerts in naïve clinical decision support system lead to false adoption by physicians? A randomized controlled trial

Chung You Tsai, Shi Heng Wang, Min Huei Hsu, Yu Chuan Jack Li

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

Objectives: False positive alerts in patient-safety-related clinical decision support systems (CDSS) are defined as alerts which incorrectly prompt when no-risk patients are encountered. It is an unfavorable condition which may potentially mislead physicians. The aim is to investigate physician responses toward false positive (FP) and true positive (TP) alerts in CDSS for the prevention of contrast-induced nephropathy (CIN). Methods: A two-arm cluster randomized controlled trial was conducted in university hospitals. Eligible physicians were randomized to receive alert intervention or no intervention (groups 1 and 2, respectively). The alert system was embedded with a deliberately non-specific risk detection tool in order to generate TP and FP alerts. The naïve alert system would alert the physician to cancel the order regardless of the patient being at-risk or not at-risk. CIN risk was stratified as at-risk and no-risk according to a patient's pre-existing renal function. Contrast imaging order-cancellation rate was measured as primary outcome. Results: 3802 contrast-enhanced examination orders from 66 physicians were analyzed. Demographic data and risk distributions of patients were similar and well-balanced between two groups. In the intervention group, a total of 1892 alerts were generated (332 TP alerts and 1560 FP alerts). Order-cancellation rates were 5.1% versus 1.4% in groups 1 and 2 for at-risk patients (relative risk [RR] = 3.69) from TP alerts, and 1.0% versus 1.4% for no-risk patients (RR = 0.71) from FP alerts. Using generalized linear model with generalized estimating equation, the FP alerts had no order-cancellation effect when compared to the control arm (adjusted RR = 0.69; 95%CI, 0.36-1.32). The TP alerts had a larger order-cancellation effect than that of the control arm (adjusted RR = 2.95; 95%CI, 0.94-9.27), which revealed a marginal trend toward significance. However, the effect was not statistically significant (adjusted RR = 1.24; 95%CI, 0.71-2.18) if TP and FP alerts were mixed. Conclusions: Physicians are not likely to adopt recommendations provided by false positive alerts in patient-safety-related CDSS. If reporting only the adoption rate of CDSS as a whole without differentiating between TP and FP alerts, the effects of TP and FP alerts will be mixed, and thus, will lead to an underestimation of system effectiveness.

Original languageEnglish
Pages (from-to)83-91
Number of pages9
JournalComputer Methods and Programs in Biomedicine
Volume132
DOIs
Publication statusPublished - Aug 1 2016

Fingerprint

Clinical Decision Support Systems
Decision support systems
Randomized Controlled Trials
Physicians
Patient Safety

Keywords

  • Alert system
  • Clinical decision support system
  • Contrast-induced nephropathy
  • False positive alert
  • Patient safety
  • Randomized controlled trial

ASJC Scopus subject areas

  • Computer Science Applications
  • Software
  • Health Informatics

Cite this

@article{dcca7547543f49ee8f3beeb6c632eac3,
title = "Do false positive alerts in na{\"i}ve clinical decision support system lead to false adoption by physicians? A randomized controlled trial",
abstract = "Objectives: False positive alerts in patient-safety-related clinical decision support systems (CDSS) are defined as alerts which incorrectly prompt when no-risk patients are encountered. It is an unfavorable condition which may potentially mislead physicians. The aim is to investigate physician responses toward false positive (FP) and true positive (TP) alerts in CDSS for the prevention of contrast-induced nephropathy (CIN). Methods: A two-arm cluster randomized controlled trial was conducted in university hospitals. Eligible physicians were randomized to receive alert intervention or no intervention (groups 1 and 2, respectively). The alert system was embedded with a deliberately non-specific risk detection tool in order to generate TP and FP alerts. The na{\"i}ve alert system would alert the physician to cancel the order regardless of the patient being at-risk or not at-risk. CIN risk was stratified as at-risk and no-risk according to a patient's pre-existing renal function. Contrast imaging order-cancellation rate was measured as primary outcome. Results: 3802 contrast-enhanced examination orders from 66 physicians were analyzed. Demographic data and risk distributions of patients were similar and well-balanced between two groups. In the intervention group, a total of 1892 alerts were generated (332 TP alerts and 1560 FP alerts). Order-cancellation rates were 5.1{\%} versus 1.4{\%} in groups 1 and 2 for at-risk patients (relative risk [RR] = 3.69) from TP alerts, and 1.0{\%} versus 1.4{\%} for no-risk patients (RR = 0.71) from FP alerts. Using generalized linear model with generalized estimating equation, the FP alerts had no order-cancellation effect when compared to the control arm (adjusted RR = 0.69; 95{\%}CI, 0.36-1.32). The TP alerts had a larger order-cancellation effect than that of the control arm (adjusted RR = 2.95; 95{\%}CI, 0.94-9.27), which revealed a marginal trend toward significance. However, the effect was not statistically significant (adjusted RR = 1.24; 95{\%}CI, 0.71-2.18) if TP and FP alerts were mixed. Conclusions: Physicians are not likely to adopt recommendations provided by false positive alerts in patient-safety-related CDSS. If reporting only the adoption rate of CDSS as a whole without differentiating between TP and FP alerts, the effects of TP and FP alerts will be mixed, and thus, will lead to an underestimation of system effectiveness.",
keywords = "Alert system, Clinical decision support system, Contrast-induced nephropathy, False positive alert, Patient safety, Randomized controlled trial",
author = "Tsai, {Chung You} and Wang, {Shi Heng} and Hsu, {Min Huei} and Li, {Yu Chuan Jack}",
year = "2016",
month = "8",
day = "1",
doi = "10.1016/j.cmpb.2016.04.011",
language = "English",
volume = "132",
pages = "83--91",
journal = "Computer Methods and Programs in Biomedicine",
issn = "0169-2607",
publisher = "Elsevier Ireland Ltd",

}

TY - JOUR

T1 - Do false positive alerts in naïve clinical decision support system lead to false adoption by physicians? A randomized controlled trial

AU - Tsai, Chung You

AU - Wang, Shi Heng

AU - Hsu, Min Huei

AU - Li, Yu Chuan Jack

PY - 2016/8/1

Y1 - 2016/8/1

N2 - Objectives: False positive alerts in patient-safety-related clinical decision support systems (CDSS) are defined as alerts which incorrectly prompt when no-risk patients are encountered. It is an unfavorable condition which may potentially mislead physicians. The aim is to investigate physician responses toward false positive (FP) and true positive (TP) alerts in CDSS for the prevention of contrast-induced nephropathy (CIN). Methods: A two-arm cluster randomized controlled trial was conducted in university hospitals. Eligible physicians were randomized to receive alert intervention or no intervention (groups 1 and 2, respectively). The alert system was embedded with a deliberately non-specific risk detection tool in order to generate TP and FP alerts. The naïve alert system would alert the physician to cancel the order regardless of the patient being at-risk or not at-risk. CIN risk was stratified as at-risk and no-risk according to a patient's pre-existing renal function. Contrast imaging order-cancellation rate was measured as primary outcome. Results: 3802 contrast-enhanced examination orders from 66 physicians were analyzed. Demographic data and risk distributions of patients were similar and well-balanced between two groups. In the intervention group, a total of 1892 alerts were generated (332 TP alerts and 1560 FP alerts). Order-cancellation rates were 5.1% versus 1.4% in groups 1 and 2 for at-risk patients (relative risk [RR] = 3.69) from TP alerts, and 1.0% versus 1.4% for no-risk patients (RR = 0.71) from FP alerts. Using generalized linear model with generalized estimating equation, the FP alerts had no order-cancellation effect when compared to the control arm (adjusted RR = 0.69; 95%CI, 0.36-1.32). The TP alerts had a larger order-cancellation effect than that of the control arm (adjusted RR = 2.95; 95%CI, 0.94-9.27), which revealed a marginal trend toward significance. However, the effect was not statistically significant (adjusted RR = 1.24; 95%CI, 0.71-2.18) if TP and FP alerts were mixed. Conclusions: Physicians are not likely to adopt recommendations provided by false positive alerts in patient-safety-related CDSS. If reporting only the adoption rate of CDSS as a whole without differentiating between TP and FP alerts, the effects of TP and FP alerts will be mixed, and thus, will lead to an underestimation of system effectiveness.

AB - Objectives: False positive alerts in patient-safety-related clinical decision support systems (CDSS) are defined as alerts which incorrectly prompt when no-risk patients are encountered. It is an unfavorable condition which may potentially mislead physicians. The aim is to investigate physician responses toward false positive (FP) and true positive (TP) alerts in CDSS for the prevention of contrast-induced nephropathy (CIN). Methods: A two-arm cluster randomized controlled trial was conducted in university hospitals. Eligible physicians were randomized to receive alert intervention or no intervention (groups 1 and 2, respectively). The alert system was embedded with a deliberately non-specific risk detection tool in order to generate TP and FP alerts. The naïve alert system would alert the physician to cancel the order regardless of the patient being at-risk or not at-risk. CIN risk was stratified as at-risk and no-risk according to a patient's pre-existing renal function. Contrast imaging order-cancellation rate was measured as primary outcome. Results: 3802 contrast-enhanced examination orders from 66 physicians were analyzed. Demographic data and risk distributions of patients were similar and well-balanced between two groups. In the intervention group, a total of 1892 alerts were generated (332 TP alerts and 1560 FP alerts). Order-cancellation rates were 5.1% versus 1.4% in groups 1 and 2 for at-risk patients (relative risk [RR] = 3.69) from TP alerts, and 1.0% versus 1.4% for no-risk patients (RR = 0.71) from FP alerts. Using generalized linear model with generalized estimating equation, the FP alerts had no order-cancellation effect when compared to the control arm (adjusted RR = 0.69; 95%CI, 0.36-1.32). The TP alerts had a larger order-cancellation effect than that of the control arm (adjusted RR = 2.95; 95%CI, 0.94-9.27), which revealed a marginal trend toward significance. However, the effect was not statistically significant (adjusted RR = 1.24; 95%CI, 0.71-2.18) if TP and FP alerts were mixed. Conclusions: Physicians are not likely to adopt recommendations provided by false positive alerts in patient-safety-related CDSS. If reporting only the adoption rate of CDSS as a whole without differentiating between TP and FP alerts, the effects of TP and FP alerts will be mixed, and thus, will lead to an underestimation of system effectiveness.

KW - Alert system

KW - Clinical decision support system

KW - Contrast-induced nephropathy

KW - False positive alert

KW - Patient safety

KW - Randomized controlled trial

UR - http://www.scopus.com/inward/record.url?scp=84964905309&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84964905309&partnerID=8YFLogxK

U2 - 10.1016/j.cmpb.2016.04.011

DO - 10.1016/j.cmpb.2016.04.011

M3 - Article

C2 - 27282230

AN - SCOPUS:84964905309

VL - 132

SP - 83

EP - 91

JO - Computer Methods and Programs in Biomedicine

JF - Computer Methods and Programs in Biomedicine

SN - 0169-2607

ER -