Medical Device manufacturers in the US need to comply with reporting requirements outlined by the US Food and Drug Administration (FDA) and submit adverse events electronically to the FDA for review. Failure to do comply could lead to financial penalties or even criminal sentences.
An adverse event can refer to any unfavorable occurrence involving a medical device. However, FDA requires that companies report an event if the device:
Caused or contributed to a death or serious injury
Malfunctioned and could potentially cause or contribute to a death or serious injury if the malfunction were to recur
Today such reports need to be filed within 30 calendar days of an incident. However given the vast underreporting, this timeline is under review by the FDA and it is expected to be relaxed somewhat.
Today most device makers face several challenges in identifying adverse events. Although they use a software system (typically a case management system or CRM) to track all uses of their devices, they rely on humans (doctors, clinical specialists, etc) to enter the right amount of detail in cases. Even after such information is entered there’s either an inefficient manual process where a team of specialists comb through these cases every day to determine if a case is deemed to have an adverse event. Some companies use a rules based system to try and automate this. Such a system looks at various cases and makes a list of keywords and phrases that identify adverse events. However given the vast number of ways a case is described, not to mention typographical and grammatical errors that can happen quickly renders such a system very ineffective.
At SpringML we have developed a system that applies AI to automatically detect adverse event cases. We use several Natural Language Processing (NLP) techniques such as Word2Vec, TFIDF and others to correctly predict adverse events. The field of study that focuses on the interactions between human language and computers is called Natural Language Processing, or NLP for short. It sits at the intersection of computer science, artificial intelligence, and computational linguistics (Wikipedia).
As you can imagine adverse events occur in a tiny percentage of cases for any device maker. Typically this is in the 5% range. As such a model that simply classifies all cases will be 95% accurate, and a 100% useless. Our models achieve a high overall accuracy but more importantly also perform very well in predicting the adverse event cases. In modeling terms we measure a metric called F1 to account for this. We will be providing more detail on modeling and associated measures in a detailed technical blog to be written shortly.
Here’s how the model results can be surfaced for end users – this is a simple dashboard that shows which cases have a high probability of having an adverse event, along with additional information on why the model predicted that way.
One of the reasons why our model performs well is because we look at several data points both internal and external to a company’s CRM system. For instance apart from the case notes, we look at the hospital and physician responsible for the case, their history – number of cases, number of previous adverse events, etc. We also look at external data such as demographics of the location where the case occurred. Combining all such information in a meaningful way helps improve the algorithm’s accuracy.
Finally a great model is of not much use if it’s not used. Our application integrates the results of the model into Salesforce and surfaces predictions in easy to use dashboards as well as daily/weekly reports. This process helps device manufacturers comply with FDA reporting requirements in a fast and efficient manner.
