My Reaction to Roger Schank’s Odd Critique of Watson


note: Yesterday AI Trends curated a controversial blog by Roger The Fraudulent Claims Made By IBM About Watson and AI.  As expected, it raised some eyebrows in the market. Here’s an rebuttal by Adrian Bowles of StormInsights

Several people have pointed out Roger Schank’s recent blog The Fraudulent Claims Made By IBM About Watson and AI  and wondered if his scathing critique of IBM and Watson, and his concern about the next AI Winter, were valid. I think not.

In this self-congratulatory piece Schank asserts:

“What I am concerned about are the exaggerated claims being made by IBM about their Watson program. Recently they ran an ad featuring Bob Dylan which made laugh, or would have, if had made not me so angry. I will say it clearly: Watson is a fraud. I am not saying that it can’t crunch words, and there may well be value in that to some people. But the ads are fraudulent.”  (sic, emphasis added)


“Really? I am a child of the 60s’ and I remember Dylan’s songs well enough. Ask anyone from that era about Bob Dylan and no one will tell you his main theme was “love fades”. He was a protest singer, and a singer about the hard knocks of life. He was part of the anti-war movement. Love fades? That would be a dumb computer counting words. How would Watson see that many of Dylan’s songs were part of the anti-war movement? Does he say anti-war a lot? He probably never said it in a song.” (emphasis added)

and, in fairness, he almost gets it right with:

“ I asked a 20-something about Bob Dylan a few days ago and he had never heard of him. He didn’t know much about the 60’s. Neither does Watson. You can’t understand words if you don’t know their context.”

By that logic, perhaps the 20-something was also suffering from a cognitive deficit. Ending his screed with this:

“IBM is simply lying now and they need to stop.”

was the last straw. Or, it was until I saw another of his posts on LinkedIn in which he said:

“But, today, with IBM promoting something that is not that much more than ELIZA people are believing every word of it. “ (emphasis added)

ELIZA – a program developed in the 1960s at MIT by Joseph Weizenbaum – used pattern matching algorithms to construct natural language responses to user input based on a script. Today, we would call it a chatbot. I doubt that ELIZA would fool a teenager today, never mind beating human Jeopardy! champions in a game that requires some level of understanding – or representation – of data, knowledge, and relationships (context?) in a variety of domains. Should Watson be relegated to a comparison with an aging chatbot because it shares cognitive characteristics with a college-aged human? Let’s take a closer look.

First, I should acknowledge that I do respect Roger Schank’s contributions to the field. 32 years ago he wrote a book called The Cognitive Computer and he has a legitimate claim to being called a pioneer. It is understandable that he may be upset that the world hasn’t universally adopted his approach or nomenclature. Science, technology and language do all evolve, but that doesn’t make his blanket assertions of malfeasance valid.

Now, the disclaimer. IBM has been a client of mine in recent years, and the Watson team was a client in 2013-2014. One might therefore dismiss my comments as biased, but I believe that I have looked into Watson more closely than those who are quick to dismiss it as an insignificant advance based on conventional statistical methods and very large data sets. Nobody from IBM was aware that I was writing this note, and it was not subject to or submitted for outside review. Any errors or omissions are mine alone.

For context, my interest in Watson began when I spoke with David Ferrucci (who headed the original development team) about architecture and algorithms on the night of Watson’s Jeopardy! tournament victory. I’ve met with many team members and interviewed the early adopters since then, and last year I co-authored a book on cognitive computing which included a chapter on Watson, but my background goes back much further. I got my undergraduate degree in psychology and first worked with NLP while studying computer science in the 1970s. I co-authored Early Experiences With CAPS: A Conversational Adaptive Programming System, which was based on a statistically-driven NLP application we developed in 1978 and published in Cybersoft 80 Volume 4 of Documents International Association for Cybernetics. I studied AI and symbolic logic and mechanical theorem proving in the PhD program at Northwestern before Prof Schank arrived. In the 1980s I taught AI and game theory and worked on RCA’s design submission for a DoD project that was one of the largest expert systems projects ever attempted (the Air Land Battle Management System). I take this stuff seriously.

It is all very well to smugly dismiss Watson’s analysis of Dylan’s lyrics with the benefit of personal historical context – Schank and many of us were around when Dylan wrote those songs. Let’s not forget, though, that the entire catalog of Dylan’s work was not about protest; that’s as gross a simplification as “love fades.” Schank notes that he spoke to one 20-year old who didn’t know anything about Dylan. If a class of college freshmen today was asked to read the complete set of Dylan lyrics and identify the primary themes, would they do any better than Watson? I doubt it – there is no reason to expect otherwise. So let’s look a little deeper. Is Watson the embodiment of artificial general intelligence? Of course not. Can it deal with multiple levels of inferences? Yes. Does it improve its performance based on experience with data rather than by re-programming (the textbook definition of machine learning)? Yes. Does that mean that it learns in a manner similar to humans? No, but I haven’t seen anyone claim that, either.

To understand how Watson works – which would be nice before one dismisses it – one really should read some of the technical articles on the system as it was developed for Jeopardy! Start with Building Watson: An Overview of the DeepQA Project which explains the approach they developed for parallel hypothesis generation and evaluation. The deep QA techniques the Watson team developed use semantic analysis at a very sophisticated level. If you’ve ever worked with natural language processing – or even compiler design for fully specified programming languages – you will understand the distinction between syntactic analysis and semantic analysis.  Schank’s critique falls flat because you simply cannot answer the types of questions Watson answered for Jeopardy! without some level of semantic analysis. Dismissing that as “crunching words” or as a rough equivalent to Eliza is ridiculous. Frankly, it strikes me as insincere, ill-informed or delusional, and seems to reflect another agenda or a gross misrepresentation or misunderstanding of how Watson works.

Read the source blog at StormInsights