A new framework to compare AI models

Artificial intelligence (AI) is impacting on many aspects of biomedical sciences and healthcare.

We have been implementing AI methods that use whole genome sequencing data to identify pathogen mutations to predict drug resistance, geography, and outbreaks. The idea is to develop and implement AI-enabled profiling tools to assist clinical management and surveillance. One difficulty with AI approaches is that they tend to be “black box” and it is difficult to directly compare published models. In new work by Jody and team, we establish a framework for the community sharing of models, particularly in M. tuberculosis genomics (called TB-ML), thereby facilitating the direct comparison of AI approaches.

Infectious diseases (ID), including TB, have a high public health burden. Whole genome sequencing (WGS) data can be used to identify pathogen mutations linked to drug resistance, geography, and outbreaks. Informative mutations are inferred from the analysis of large genomic databases, then linked to informatics tools to “profile” pathogens from WGS data (e.g. TB-Profiler). This approach uses a static database of informative mutations, and we have recently implemented (and shared) AI models to identify new mutations (e.g. Treesist-TB; ML-TB). Here, we propose to fully integrate AI models and informatics profiling systems, across several pathogens, including through automated public and researcher “crowd” data capture, linked to re-training and evaluation of models and their predictions. Building on existing collaborations that are generating WGS data to inform infection control (e.g. UKHSA), we propose to develop AI-enabled profiling tools to assist clinical management and surveillance.