Evaluating compliance of computed tomography referrals with iGuide imaging guidelines using artificial intelligence approaches

Most hospitals and clinics offer diagnostic imaging services. While X-ray remains the most frequently performed diagnostic imaging examination worldwide, and ultrasound is also widely used, computed tomography is valued for its speed and diagnostic comprehensiveness. The increasing demand for this imaging modality has led to a significant rise in worldwide scanning frequency. The associated risk of stochastic radiation effects is not negligible, especially when unnecessary imaging is performed. Therefore, justification should be a core focus of imaging departments. However, this is not the case, as evidenced by high rates of unjustified scans in Europe and abroad. The existing clinical decision support system for choosing appropriate imaging comes in the form of clinical referral guidelines. Given the high rates of unjustified imaging, it is unsurprising that their uptake has been poor. As of today, a manual retrospective justification audit is the only reliable method to quantify local and national justification practices. Although considered a gold standard method, manual auditing is time-consuming, subject to inconsistencies between human experts, and costly. Nonetheless, it is essential for monitoring compliance with the guidelines, quality of care, financial expenditure, and more. The primary aim of this thesis was to investigate the feasibility of automated justification analysis of the most frequent computed tomography scan nationally—the brain scan—in line with iGuide referral guidelines, using various AI approaches, including natural language processing, supervised machine learning, deep learning, and large language models, namely ChatGPT and Gemini. Retrospective justification auditing was conducted to generate target features, which also yielded important findings regarding the compliance rates of four Irish clinical sites. The secondary aim was to present this new use case of AI in the context of the increasing commercialisation and potential of AI in medical imaging. The aim of Study 1 was to retrospectively audit the justification of an initial sample of adult brain CT referrals from a tertiary referral hospital in Dublin and subsequently train prediction models to categorise referrals according to their justification. The study hypothesised that the rate of justified referrals would be lower than 80% and that machine learning algorithms could identify underlying patterns associated with justified and unjustified referrals, achieving high classification accuracy. Study 2 focused on developing multi-class prediction models to demonstrate the feasibility of automated iGuide categorization of referrals. This experimental, multicentre study involved four human experts analysing the justification of a larger sample of referrals. It was again hypothesized that the justification rate would be lower than 80%, The second hypothesis assumed high multi-class classification accuracy with machine and deep learning, and better analysis consistency than that of humans. In the final study, ChatGPT’s and Gemini’s ability to justify referrals from the test set of Study 2 was investigated and compared against human experts and the best-performing multi-class prediction model. It was hypothesised that the two chatbots would be inferior and that their recommendations would not be entirely consistent with iGuide. In summary, the justification rates in four Irish clinical sites are suboptimal. Automating the justification analysis of unstructured brain CT referrals using machine and deep learning is feasible. This could facilitate and standardise regular retrospective auditing and provide real-time clinical decision support for better implementation of referral guidelines. Commercially available large language models, ChatGPT and Gemini, need to improve in processing real-world clinical text of varying quality.