Fears that AI discrimination could lead to patient harm

Artificial intelligence in healthcare has left experts urging caution that a focus on predictive accuracy over treatment efficacy could lead to patient harm.

- STORM NEWTON

Researchers in the Netherlands warn that while AI-driven outcome prediction models (OPMs) are promising, they risk creating “self-fulfilling prophecies” due to biases in historical data.

OPMs utilise patient-specific information, including health history and lifestyle factors, to assist doctors in evaluating treatment options. AI’s ability to process this data in real time offers significant advantages for clinical decision-making.

However, the researchers’ mathematical models demonstrate a potential downside, namely, if trained on data reflecting historical disparities in treatment or demographics, AI could perpetuate these inequalities, leading to suboptimal patient outcomes.

The study highlights the crucial role of human oversight in AIdriven healthcare. Researchers emphasise the “inherent importance” of applying “human reasoning” to AI’s decisions, ensuring that algorithmic predictions are critically evaluated and do not inadvertently reinforce existing biases.

The team then created mathematical scenarios to test how AI may harm patient health and suggest that these models “can lead to harm”.

“Many expect that by predicting patient-specific outcomes, these models have the potential to inform treatment decisions and they are frequently lauded as instruments for personalised, data-driven healthcare,” researchers said.

“We show, however, that using prediction models for decision-making can lead to harm, even when the predictions exhibit good discrimination after deployment.