A fairer comparison would be Eliza vs ChatGPT.
deleted by creator
Its because of all the people saying that LLMs can reason and think and the human brain works just like an LLM and… some other ridiculous claim.
This shows some limitations on LLMs.
Human brains lose to computerized chess all the time, though. So I guess this is a win for AI tech bros?
Why the special qualifier of “computerized” chess? Do humans regularly lose to Atari’s at chess? LLMs are computerized too.
I meant a specialized application, like the Atari one that beat the LLM.
But humans not trained (made) for chess would make stupid mistakes too
Why are so many people mad when it’s pointed out that the shitty chatbots are just shitty chatbots.
I knew there would be these kinds of comments making this obvious point. This is just a demo of how these language models are not going to achieve the “General” part of AGI. It’s going to take a new paradigm
Now apply this to like, everything else ever.
Machine designed to convincingly fake human internet conversation sucks at ____________!
ChatGPT can’t make a rug as well as a 300 year old loom.
Too many people forget that specialized, purpose-driven software is often if more effective and efficient. LLMs and other AI are nice when you don’t have a properly defined spec or a flexible algorithm but you pay, literally, for the convenience.
40 year old machine designed to play chess*
I think people in the replies acting fake surprised are missing the point.
it is important news, because many people see LLMs as black boxes of superintelligence (almost as if that’s what they’re being marketed as!)
you and i know that’s bullshit, but the students asking chatgpt to solve their math homework instead of using wolfram alpha doesn’t.
so yes, it is important to demonstrate that this “artificial intelligence” is so much not an intelligence that it’s getting beaten by 1979 software on 1977 hardware
A chess-specific algorithm beat a language model at chess. Shocking!
Try training a chess model. Actually I think it’s already been done, machines have been consistently better at chess than humans for a while now.
deleted by creator
I’m shocked! — shocked to find that LLMs aren’t superhuman intelligences that will soon enslave us all. Other things they’re not good at:
- Summarizing news articles. Instead of an actual summary they’ll shorten the text by just leaving things out, without any understanding of which parts are important.
- Answering questions about anything controversial. Based on subtle hints in the wording of your question they’ll reflect your own biases back at you.
- Answering questions about well-known facts. Seemingly at random when your question isn’t phrased exactly the right way they’ll start hallucinating and make up plausible bullshit in place of actual answers.
- Writing a letter. They’ll use the wrong tone, use language that is bland and generic to a degree that makes it almost offensive, and if you care about quality the whole thing will need so much re-writing that it’s quicker to do it yourself from the start.
- Telling jokes. They don’t really get humour. Their jokes tend to have things that superficially look as if they should be punchlines but aren’t funny at all.
- Writing computer code. Correcting their mistakes is even more laborious in computer languages. Most of the time they’re almost as bad at it as they are at playing chess.
Still they are amazingly clever in some ways and pretty good for coming up with random ideas when you’ve got writer’s block or something.
This is useful for dispelling the hype around ChatGPT and for demonstrating the limits of general purpose LLMs.
But that’s about it. This is not a “win” for old school game engines vs new ones. Stockfish uses deep reinforcement learning and is one of the strongest chess engines in the world.
EDIT: what would be actually interesting would be to see if GPT could be fine-tuned to play chess. Which is something many people have been doing: https://scholar.google.com/scholar?hl=en&q=finetune+gpt+chess
In other news, my toaster absolutely wrecked my T.V. at making toast.
How did alpha go do?
