|
|
|
|
Cuneiformist
The Imperfectionist
USA
4955 Posts |
 Posted - 02/19/2011 : 12:50:57
|
As most regulars to SFN know, the popular game show Jeopardy! recently hosted a special game between two of its biggest former winners and an IBM supercomputer named Watson. They played two games over three days, with the scored being cumulative. The result was decisive: Watson won $77,147 while Ken Jennings placed second at $24,000 and Brad Rutter third at $21,600.
Much of the commentary in the aftermath of this exhibition (the bulk of the 1.5 million dollar jackpot went to charity-- IBM donated its million to charity, and half of the winnings the humans won ($300,000 and $200,000 respectively) similarly went to charity) is how we're on our way to an AI-dominated world (e.g. here), with some (perhaps jokingly) talking about a Terminator or HAL9000-like scenario. (Seriously-- just Google Watson + IBM + Victory + Terminator and see the hits.)
But I have to say, I'm not quite as impressed as everyone else in this. I am very impressed-- very that the programmers have created a machine that does incredibly well in recognizing what they call "natural language"-- the language we speak in our everyday lives. I recall playing the old game Space Quest, where you had to type in what you wanted your character on the screen to do. So you'd move the little guy around with the arrow keys, and when he came up to some object, you'd frantically start typing "look at rock" or "pick up rock" or whatever, and if the computer didn't understand you, you'd get back a message saying as much. And the problem was that because of how the program worked, you had to essentially type the phrase perfectly, or the computer wouldn't understand. That people have designed a machine to recognize the various phrasings of the Jeopardy questions (technically, they're answers to which contestants have to provide the questions, but you get the point) is impressive.
Moreover, I see the incredible value of such technology. On a promo for the project, they talked about how this technology would improve medicine, as a Watson-type machine could quickly process a medical case and with its algorithms accessing hundreds and hundreds of gigabytes of data, provide a superior regimen for a cure than what a human doctor could provide. One could easily come up with various other situations that such a system would be a huge improvement.
However, in watching the two Jeopardy games, I wasn't so impressed. Essentially, Waston's advantage was simply one of speed. And not speed of accessing its massive multi-terabyte database. I'm talking about the speed (and timing) that the computer was able to buzz in to answer.
Some background: you're not allowed to buzz in until Alex is finished reading the question. In the actual studio, there's a light that flashes to indicate that it's safe to answer (I actually saw a taping last year). If you answer too soon, you're locked out for an extra fraction of a second. Now, the computer in this game couldn't actually "hear" the question. Instead, it was fed a text file as Alex was reading. Though I don't know for sure, it must have been linked into the special indicator so that it could properly buzz in. But because it's a machine, it would never buzz in early, and moreover, as it works at the speed of light, it would be able to buzz in with a high level of precision. And that's exactly what it did.
I watch the show regularly, and I am confident that I can answer 40-50% of the questions correctly without much thought. I'm sure the experts who make it on the show can do better (as they've committed to memory lists of presidents, Oscar winners, etc.). And it was clear to me that the humans on the show knew the bulk of the answers. Their primary disadvantage was that they buzzed in as humans, and thus they did so with some degree of error.
To be sure, if they had played in a real trivia show and asked serious trivia ("this man was the Secretary of the Interior under Benjamin Harrison"), the computer would have killed. Humans just can't possibly have ready access to, basically, every Wikipedia page like Watson can. But this wasn't real trivia. It was Jeopardy! and the bulk of the questions aren't that hard.
So while I think that Watson is an incredible development and I'm glad it's happening, I don't feel like humans got beat as bad as it sounds.
|
|
|
Dave W.
Info Junkie
USA
26020 Posts |
Posted - 02/19/2011 : 13:12:19 [Permalink]
|
Here was my comment after Wednesday night's episode:
So, IBM's "Watson" computer system just spanked two extremely competent past winners of Jeopardy! One of them noted that he, for one, welcomes our new computer overlords - after visibly being annoyed by how often Watson was beating him to the button during the games.
IBM got a huge amount of advertising out of the three days on the show, and IBM's charity picks are now a million bucks richer, but hopefully this technology will become something more useful and long-lasting than a big Jeopardy! win. |
- Dave W. (Private Msg, EMail)
Evidently, I rock!
Why not question something for a change?
Visit Dave's Psoriasis Info, too. |
 |
|
|
Cuneiformist
The Imperfectionist
USA
4955 Posts |
Posted - 02/19/2011 : 17:58:15 [Permalink]
|
Originally posted by Dave W.
Here was my comment after Wednesday night's episode:
So, IBM's "Watson" computer system just spanked two extremely competent past winners of Jeopardy! One of them noted that he, for one, welcomes our new computer overlords - after visibly being annoyed by how often Watson was beating him to the button during the games.
IBM got a huge amount of advertising out of the three days on the show, and IBM's charity picks are now a million bucks richer, but hopefully this technology will become something more useful and long-lasting than a big Jeopardy! win. |
I'm sure it will be more useful in the end. I just don't think it was that they were "spanked" quite as badly as everyone says. I have watched almost every episode of Jeopardy for the last 7 months or so and seen when someone really spanks the others. But in those cases, you really see someone dominating because they actually know more of the answers. In this case, it seemed more an example of one contestant-- Watson-- just being faster at the buzzer.
The show took pains to show how Watson had to actually buzz in with a mechanical buzzer. However, unlike humans, Watson could never buzz-in early (thus triggering a short time-delay penalty). When people are allowed to buzz in, it is indicated by a light in the studio. Watson couldn't see the light, of course, and so it was told (electronically) when it was OK to buzz in. At that moment-- assuming it had a high enough confidence level-- it could then send an electric impulse to the button-pushing device to do so. The amount of time that the buzzing process takes is virtually nil, and certainly faster than a human. So, all other things being equal (i.e. knowing the answer), the computer is going to win the bulk of the time.
Indeed, in Jeopardy you'll notice a common strategy of simply buzzing in-- and timing your buzz with the indicator light--regardless of whether or not you know the answer. You just buzz in and then use the few seconds you have to actually think of it.
The only time the humans did well was when the computer was still trying to process the answer and hadn't reached the correct confidence level. If I were the programmers and doing a rematch, I'd simply tell the computer to buzz in when it was able to, and then use the extra few seconds to keep processing and answer with what it had before the '"time's up" signal. It would win virtually every buzz (a few times, the humans would get it simply because they try to anticipate the signal), and as anyone who watched the game saw-- even when its confidence level was at 50 or 60%, the answer it was most confident with was still correct.
Again, I feel like what they've done is an amazing leap forward in computer technology, and indeed, it's what we've always felt like computers should do (like the Stark Trek ship computers, where you ask it what to do in "natural language" and it taps its huge database (I'm sure on Stark Trek, its memory was up in the hundreds of petabytes range). I just don't think that this victory on Jeopardy was so impressive because of the mechanical advantage involved with buzzing in. |
|
|
|
Dave W.
Info Junkie
USA
26020 Posts |
Posted - 02/19/2011 : 19:55:57 [Permalink]
|
The look on Jennings' face near the end of the third night said that he knew he was getting a spanking.
You're right in that there are two parts to the game - buzzer-pushing and knowledge - but you won't last long in the game if your only skill is buzzing in first. Someone who buzzes in first every time but only gets questions correct 50% of the time will wind up with a big fat $0 on average (ignoring Daily Doubles). Someone like my mother-in-law, who watches the show religiously but couldn't get close to 50% right would go well into the negatives if she buzzed in first every time. Watson obviously has both skills (not perfect in either one, of course), or it wouldn't have worked as well as it did.
I saw Watson's top-three list change once (and only once) when one of the other players got a chance to answer. At first, its responses were all deep in the red, and then one changed to yellow and popped to the top, but then a fraction of a second later, the human spoke the right answer. Given that I'd seen Watson miss the buzzer at least a couple dozen times, but only saw its answers change once, how much would the extra time really help it? I wonder if it'd be worth the risk of never finding a suitable response for it to buzz in first every time regardless of what it thought at the moment it was signalled. |
- Dave W. (Private Msg, EMail)
Evidently, I rock!
Why not question something for a change?
Visit Dave's Psoriasis Info, too. |
|
|
|
Elmo the Clown
New Member
31 Posts |
Posted - 02/20/2011 : 06:27:17 [Permalink]
|
Feed it history and economics, and set it on the floor of Congress. I think we should run it for a Senate seat. 'course it isn't old enough.
|
Support a clown, buy a luury cruise from www.ChicLuxuryCruises.com (or any cruise...) |
| Edited by - Elmo the Clown on 02/20/2011 06:28:46 |
|
|
|
Dude
SFN Die Hard
USA
6891 Posts |
Posted - 02/20/2011 : 07:43:18 [Permalink]
|
It (Watson) is an interesting search tool. What I'd do if I were the IBM exec in charge of this project is send off an email to Google... "Hey, guys, I have this wicked fast/accurate speech recognition tool that can search massive databases for answers to questions in fractions of a second, and it runs entirely on voice input. Want to get in on the action? Hit me back!"
|
Ignorance is preferable to error; and he is less remote from the truth who believes nothing, than he who believes what is wrong.
-- Thomas Jefferson
"god :: the last refuge of a man with no answers and no argument." - G. Carlin
Hope, n.
The handmaiden of desperation; the opiate of despair; the illegible signpost on the road to perdition. ~~ da filth |
|
|
|
|
Cuneiformist
The Imperfectionist
USA
4955 Posts |
Posted - 02/20/2011 : 16:41:51 [Permalink]
|
Originally posted by Dave W.
The look on Jennings' face near the end of the third night said that he knew he was getting a spanking.
You're right in that there are two parts to the game - buzzer-pushing and knowledge - but you won't last long in the game if your only skill is buzzing in first. Someone who buzzes in first every time but only gets questions correct 50% of the time will wind up with a big fat $0 on average (ignoring Daily Doubles). Someone like my mother-in-law, who watches the show religiously but couldn't get close to 50% right would go well into the negatives if she buzzed in first every time. Watson obviously has both skills (not perfect in either one, of course), or it wouldn't have worked as well as it did. |
No, of course. I wouldn't advocate just anyone buzzing in first. You have to know something, too! But if you're Watson, you have access to terabytes of data that you can scan in a tiny fraction of a second.
| I saw Watson's top-three list change once (and only once) when one of the other players got a chance to answer. At first, its responses were all deep in the red, and then one changed to yellow and popped to the top, but then a fraction of a second later, the human spoke the right answer. Given that I'd seen Watson miss the buzzer at least a couple dozen times, but only saw its answers change once, how much would the extra time really help it? |
It's hard to say. You actually have a few seconds when you answer, but in this game, no one really took that time because they're all pretty good.
We also never got to see what Watson could do if given a few seconds to think because he never employed the "buzz-first" strategy. I won't pretend to know how the machine's algorithms work, but if it's like a chess computer, then time is important. A good chess machine with access to every GM game ever played can certainly find a good move for a given position in a short amount of time. For every second more you give it, though, you're going to get an even better one.
|
|
|
|
 |
|
|
|
Watson, Jeopardy and Humanity
