Did IBM overhype Watson Health’s AI promise?

In recent weeks, IBM has changed leadership at its Watson Health division and  announced a new business strategy for deployment that relies on a hybrid cloud, not a  public- or private-cloud only model.

Over the past year, Watson Health – particularly Watson for Oncology – has come under criticism for not meeting expectations or even offering physician users inaccurate advice. (Watson for Oncology is IBM’s commercial cognitive computing cloud platform that analyzes large volumes of patient healthcare data and published medical studies to offer physicians cancer treatment options.)

Laura Craft, a vice president of research for Gartner’s Healthcare Strategy business, said IBM’s Cognitive Computing Division did not do well in recent third-quarter results, “and that was largely driven by the healthcare component.”

Craft also pointed to the recent leadership changes as indicative of internal problems.

Reports of trouble, and IBM’s defense

In July, the healthcare news publication Stat published a report claiming “internal IBM documents” showed the Watson supercomputer often spit out erroneous cancer treatment advice and that company medical specialists and customers identified “multiple examples of unsafe and incorrect treatment recommendations,” even as IBM was promoting its AI technology.

Stat cited several slide decks it had obtained from a presentation made by IBM Watson Health’s deputy chief health officer in 2016. The slides mostly blamed problems on the training of Watson by IBM engineers and staff at the Memorial Sloan Kettering Cancer Center (MSKCC).

Separately, an article by the Wall Street Journal claimed Watson Health had not made progress in bringing AI to healthcare.

In August, John Kelly (then IBM’s senior vice president of Cognitive Solutions and Research), fired back in a blog about the reports: “It is true, as the article reports, that we at IBM have placed a big bet on healthcare. We have done this for two reasons: 1) Most importantly, we know that AI can make a big difference in solving medical challenges and supporting the work of the healthcare industry, and 2) We see an enormous business opportunity in this area as the adoption of AI increases,” he wrote.

“To suggest there has been no patient benefit,” Kelly continued, “is to ignore both what we know The Wall Street Journal was told by a number of physicians around the world and these institutions’ own public comments – which we believe speak for themselves.”

Kelly pointed to five healthcare facilities and the Department of Veterans Affairs that offered physician testimonials on how Watson for Oncology, Watson for Clinical Trial Matching and Watson for Genomics had reduced physician time and effort, increased clinical trial participants or increased the volume of research data available to the AI engine.

IBM

Asked whether IBM’s Watson for Oncology had relayed erroneous cancer treatment advice, Ed Barbini, IBM’s vice president of external relations, flatly denied the charge.

While IBM faces declining revenue overall, and its recently released third-quarter earnings showed revenue from cognitive offerings was down 6% from last year, Watson Health saw growth, according to Barbini. He noted that IBM does not release numbers specific to Watson Health for “competitive reasons.”

Barbini admitted that developing Watson Health and, specifically, Watson for Oncology is not an easy task but it remains an important one.

“That’s why IBM dove into it three years ago. Did you really think oncology would be mastered in three years?” Barbini said. “However, let’s look at the facts. More than 230 hospitals are using one of our oncology tools. We’ve had 11 [software] updates over last year and half and we’ve doubled the number of patients we’ve reached to over 100,000 as of the end of the third quarter of this year.”

Earlier this month, the head of Watson Health for the past three years, Deborah DiSanzo, stepped down and Kelly took over. DiSanzo is continuing to work with IBM Cognitive Solutions’ strategy team, according to a company spokesperson.

IBM has been the most aggressive among technology companies using AI in support of evidence-based medicine for health care. But recent problems have reportedly led to the loss of some of its larger hospital clients.

Craft said the Watson for Oncology unit is garnering most of the bad press; she believes its because IBM’s marketing department promised something they could not deliver. Watson Health should have remained in an incubator stage far longer so more data could be ingested, enabling better evidence-based medicine treatment options.

IBM’s “party line” has been to deny the reports and publish select development partner and client testimonials, Craft said.

IBM

“I think if I understand IBM’s vision…, it was to really move toward personalized medicine. The ambition was to get to the state where they can target and correlate therapies and drugs to what the patient will respond best to,” Craft said. “I think from a medical research perspective, they are going to get there in 10 or 15 years when we’ve got better, more consistent, real-world data – all things I don’t think Watson has the benefit of today.”

As a result, IBM has failed deliver on the capabilities it marketed to healthcare providers. “They have disappointed clients and they’ve certainly created skepticism around the integrity of their more advanced products,” Craft said.

It’s not the tech, it’s the time

Watson’s AI technology is not the problem, Craft said; it simply hasn’t had enough time or quality data input to become the personalized medicine engine IBM has pitched.

Cynthia Burghard, IDC’s research director for Value-based Healthcare IT Transformation Strategies, said IBM “shot themselves in the foot” by choosing to aim Watson at something as complex as healthcare right out of the gate.

“Part of Watson for Health’s challenges is they were very aggressive with marketing, which is kind of an IBM trait. And, then it came [to] delivering it and they chose oncology, they chose genomics – really tough nuts to crack,” Burghard said.

IBM had hoped to offer Watson as a software product, Burghard said, where oncologists could simply plug in patient data and they would receive recommendations for treatment. “That’s the commercial version of Watson Health they were hoping, but that’s not been realized,” she said.

Instead, IBM has had to work with the hospital clients to ensure Watson functions properly, Burghard explained.

“In order for it to be commercial, IBM shouldn’t be putting their hands on the data. They shouldn’t have analysts in there building models for the clients; that should all be available to the client,” Burghard said.

Another problem, Burghard pointed out, is Watson was trained using data from IBM development partner MSKCC. Because the system was trained using the hospital system’s data, its query results tend to be biased toward that institution’s cancer treatments and is not as inclusive with data from other hospitals, such as the Mayo Clinic or other smaller facilities.

While that may work for simple cancers whose treatments are relatively generic, Burghard said complex cancers are treated differently in various facilities. And, smaller hospitals may not even have access to the same treatment options as their larger, regional counterparts.

Because of its failure to produce the results promised, Burghard said IBM has lost momentum and “there’s such skepticism that unless they have some silver bullet sitting behind the curtain, I think it’s just going to plug along until the stock holders say they’ve had enough.”

Early problems

In 2012, one of the first pilots of Watson for Oncology took place with the M.D. Anderson Cancer Center at the University of Texas. The hospital used IBM’s Watson supercomputer to expedite clinical decision making by matching cancer patients to clinical trials to improve outcomes “worldwide.” At a final cost of $62 million, the cancer center’s Watson-backed Oncology Expert Advisor (OEA) never got off the ground and was halted after an external audit was sought by the university.

The initial scope of OEA system development was for MDS leukemia, but it was expanded in February 2013 to include five additional types of leukemia, then in December 2014 to include lung cancer.

The audit revealed the Watson Oncology system could not integrate with M.D. Anderson Cancer Centers’ EPIC electronic medical record (EMR) system, so internal pilots of the OEA for Leukemia and Lung cancer were conducted using the prior medical records system (ClinicStation).

The Cancer Center and IBM Watson ceased active development in 2015. And IBM ended support for the OEA Pilot System and for the OEA Demo System effective Sept.  1, 2016. The system is not in clinical use and has not been piloted outside of M.D. Anderson, according to the the audit.

IDG

The IBM agreement at the time the project was halted said the system “is not ready for human investigational or clinical use, and its use in the treatment of patients is prohibited” except as needed to test and evaluate the system, according to the University of Texas audit.

When asked by Computerworld why the project failed, M.D. Anderson Cancer Center said via email: “While a variety of approaches have been examined, a final approach using [cognitive computing] to benefit patients has not been determined at this time. MD Anderson is committed to continuing to explore how digital solutions can accelerate the translation of research into advanced cancer care for patients.”

Computerworld also reached out to Memorial Sloan Kettering Cancer Center and the Mayo Clinic, two of IBM’s top development partners on Watson Health who’ve been cited as success stories for training Watson and using it for clinical trial matching.

Begun in 2014, Watson’s job at the Mayo Clinic was to sift through thousands of medical studies and ensure that more patients are accurately and consistently matched with promising clinical trials. (IBM has announced that enrollment rates for breast cancer clinical trials at the Mayo Clinic had increased dramatically.)

A request for comment from the Mayo Clinic on Watson’s effectiveness was not returned. A Mayo Clinic spokesperson said multiple attempts had been made at reaching the physician in charge of the Watson project but were unsuccessful.

A Memorial Sloan Kettering Cancer Center spokesperson referred questions to IBM, stating that IBM receives feedback on Watson for Oncology directly from its customers, and while the hospital trains Watson’s AI with its data, “we do not use it here.”

Another clinic touted early on by IBM is the Highlands Oncology Group (HOG), which participated in a feasibility study of IBM Watson to increase the efficiency and accuracy of the clinical trial matching. Located in Northeast Arkansas, HOG has 15 physicians and 310 staff members working across three sites; the facility’s pilot lasted 16 weeks and used data from 2,620 visits by lung and breast cancer patients.

In an initial pre-screening test, the HOG clinical trial coordinator took 1 hour and 50 minutes to process 90 patients against three breast cancer screenings. Conversely, when the Watson’s clinical trial matching platform was used, that job took 24 minutes. “This represents a significant reduction in time of 86 minutes or 78%,” HOG said in a statement.

Computerworld reached out to HOG about the Watson trial, and asked specifically if there were any problems during the pilot; HOG’s medical director said the clinic had signed a confidentiality agreement with IBM and was not allowed to give out any information.

“So, IBM Watson would be the ones that provide you the concerns and road blocks they’ve run into,” a HOG spokesperson wrote via email.

An IBM buying spree, and what comes next

In 2015, IBM purchased healthcare data analytics-firm Explorys and patient communications company Phytel for an undisclosed amount. In 2016, IBM acquired Truvan Health Analytics for $2.6 billion. Upon completing all three acquisitions, IBM boasted its Watson Health Cloud housed “one of the world’s largest and most diverse collections of health-related data, representing an aggregate of approximately 300 million patient lives acquired from three companies.”

“They all in their own right, before they were acquired, were very successful companies and had good, strong, loyal client bases and were plugging along. I think IBM thought, ‘We should buy these guys and throw in some AI and really take the market by storm,’” Burghard said. “As far as I can tell, that hasn’t happened.”

At least one of those acquisitions, Truvan, was recently cited by IBM’s Kelly as key to moving insurance provider data onto the IBM Watson Health platform now that it’s going to be offered through a hybrid cloud.

In late October, IBM announced plans to seed its new hybrid cloud model for Watson by first moving data from insurance payer systems. For that, Truvan will be key.

“They [Truvan] are very big in the payer space,” Kelly said. “We process payer claims and we have payer records. So, what does it cost for a certain procedure in a state or in a hospital – that is a very rich data set we can apply AI to to dramatically reduce cost.”

Once payer data is moved to the hybrid cloud, the electronic medical records (EMRs) acquired through the Explorys acquisition will follow, Kelly said.

Article source: https://www.computerworld.com/article/3321138/healthcare-it/did-ibm-put-too-much-stock-in-watson-health-too-soon.html