AI World Executive Summit recap: AI in Clinical Diagnostics


Lux Research recently helped co-chair the Artificial Intelligence (AI) World Executive Summit in San Francisco, CA – One of the many wonderful presentations given at the conference was by David Ledbetter, a data scientist at the Children’s Hospital in Los Angeles. David and his team are currently working to leverage the advances in machine learning to develop a host of tools for use within the children’s hospital. Some of the projects that he and his team are currently working on include a pediatric early warning system, which would help clinicians identify any subtle underlying patterns in any given child’s
condition that may be indicative of a child’s potential need for heightened treatment within the next six hours; a physiology trajectory predictor system, which attempts to predict how vital signs will behave in order to help clinicians make better informed decisions; and a probability of survival tool, which calculates how dire a patient’s condition is to give doctor’s a benchmark to assess how treatment is influencing the chances of a patient’s

To develop accurate models for the health care environment, data scientists attend rounds and shadow nurses to understand what is going on in the unit, what information is being exchanged, and how the decision-making process takes shape. The team also has access to about a decade’s worth of data gathered by the Children Hospital of Los Angeles, which includes about 20,000 pediatric intensive care unit (ICU) encounters and 60,000 regular floor encounters. Some of this data includes qualitative data such as the patient’s demographics, family history, symptoms, and diagnoses; however, David states that the “meat and potatoes” of the model developed by his team comes from time series data such as vital signs, laboratory measurements, and time of drug administration. Given that much of the model is contingent upon the interpretation of time series data, the deep learning model that the group uses is the Recurrent Neural Network (RNN). Moreover, David explained that the team uses the Keras deep learning framework due to its ability to use either TensorFlow or Theano on the backend (see the October 25, 2016 LRIBDAJ).

In order to validate their results the team compared their research to some of the expert systems currently in use for clinical diagnostics, such as the Pediatric Index of Mortality (PIM). David demonstrated to the audience that one of the systems developed by the team did, indeed, outperform traditional systems by showing the audience the results of each model on a Receiver Operator Characteristic (ROC) curve. A ROC curve is a plot of the false positive rate of a binary classifier model against the true positive rate of the model as the threshold of the model is varied; a model is considered good if one of the thresholds can
simultaneously achieve a high true positive rate while minimizing the false positive rate. However, David was quick to note that this level of performance would not have been possible had the team not had access to the copious amount of training data that was at their disposal.

He then demonstrated this point by illustrating how the performance of the system varied with when trained using different fractions of the training data available data. Since the team was using a supervised learning technique, quite naturally the obvious conclusion was that more data produced better system performance. To this end, one of the major focuses
of his team’s efforts is working with other leading pediatric institutions to aggregate as much pediatric clinical care data as possible.

Though David demonstrated the value of machine learning within the context of the health care industry, the message of his presentation applies to any industry looking to benefit from the advantages of AI. As mentioned time and time again (see the November 22, 2016 LRIBDAJ), machine learning models that use supervised learning techniques need a great deal of quality data to perform well. If companies truly wish to effectively apply this technology to their enterprises, then they must get into the habit of not keeping their data stored in silos across their business units, and maintain consistent, quality records. More
importantly, just as the hospital started collecting copious amounts of data years before it implemented an AI approach, businesses need to understand that they do not need to have an analytics platform in place in order to start preparing for the day when they do need abundant data.

Moreover, David’s anecdotal reference of how he and his team shadowed clinicians to tailor the models to reflect the know-how of the caretakers interacting with the patients is a great example of how businesses need to more effectively harmonize the collaboration between technology developers and business workers to capture their industry experience in the technology. Otherwise, the resultant technology will not reflect the core knowledge of the
business and so, consequently, it will do little to add value to the overall business workflow. And though this level of collaboration may be straightforward for smaller businesses, larger corporations will have to face a paradigm shift from the traditional compartmentalization of IT departments and operating business units, or risk being outperformed by competitors.

by Ahmed Khalil, Lux Research, Inc.