Popularity is not correctness. You’re using a sloppily defined term. I’m using the fundamental definition. Your (Verners) concept muddles matters pointlessly.

The fact is, self-refining LLMs can very possibly exhibit the intelligence explosion fundamental to Von Neumann or I.J. Good’s definition. They are already beginning to alter the way human society operates (coding, school, replacing jobs). They easily pass the Turing test with the right prompts. Your whole point is that it’s not “real” intelligence because they don’t really “understand”, but I can say the same for you. For all I know, you’re an LLM and there’s literally no way that you can prove you aren’t.

Lines in the sand about “real” intelligence are purely philosophical, and that kind of hyperopic philosophizing is exactly the sort of behavior that dooms humanity via underestimation. I’d rather we didn’t find ourselves under machine overlords because “technically they aren’t even really intelligent”.