I’m happy with the Oxford definition: “the ability to acquire and apply knowledge and skills”.
LLMs don’t have knowledge as they don’t actually understand anything. They are algorithmic response generators that apply scores to tokens, and spit out the highest scoring token considering all previous tokens.
If asked to answer 10*5, they can’t reason through the math. They can only recognize 10, * and 5 as tokens in the training data that is usually followed by the 50 token. Thus, 50 is the highest scoring token, and is the answer it will choose. Things get more interesting when you ask questions that aren’t in the training data. If it has nothing more direct to copy from, it will regurgitate a sequence of tokens that sounds as close as possible to something in the training data: thus a hallucination.
This can be intuitively understood if you’ve gone through difficult college classes. There’s two ways to prepare for exams. You either try to understand the material, or you try to memorize it.
The latter isn’t good for actually applying the information in the future, and it’s most akin to what an LLM does. It regurgitates, but it doesn’t learn. You show it a bunch of difficult engineering problems, and it won’t be able to solve different ones that use the same principle.
I’m happy with the Oxford definition: “the ability to acquire and apply knowledge and skills”.
LLMs don’t have knowledge as they don’t actually understand anything. They are algorithmic response generators that apply scores to tokens, and spit out the highest scoring token considering all previous tokens.
If asked to answer 10*5, they can’t reason through the math. They can only recognize 10, * and 5 as tokens in the training data that is usually followed by the 50 token. Thus, 50 is the highest scoring token, and is the answer it will choose. Things get more interesting when you ask questions that aren’t in the training data. If it has nothing more direct to copy from, it will regurgitate a sequence of tokens that sounds as close as possible to something in the training data: thus a hallucination.
This can be intuitively understood if you’ve gone through difficult college classes. There’s two ways to prepare for exams. You either try to understand the material, or you try to memorize it.
The latter isn’t good for actually applying the information in the future, and it’s most akin to what an LLM does. It regurgitates, but it doesn’t learn. You show it a bunch of difficult engineering problems, and it won’t be able to solve different ones that use the same principle.