Technology

A neurotypical human mind, acting rationally, is able to remember the chain of thought that lead to a decision, understand why they reached that decision, find the mistake in their reasoning, and start over from that point to reach the "correct" decision.

Even if they don't remember everything they were thinking about, they can reason based on their knowledge of themselves and try to reconstruct their mental state at the time.

This is the behavior people are expecting from LLMs but not understanding that it's something they're fundamentally incapable of.

One major difference (among many others, obviously) is that AI models as currently implemented don't have any kind of persistent working memory. All they have for context is the last N tokens they've generated, the last N tokens of user input, and any external queries they've made. All the intermediate calculations (the "reasoning") that led to them generating that output is lost.

Any instance of an AI appearing to "correct" their mistake is just the model emitting what it thinks a correction would be, given the current context window.

Humans also learn from their mistakes and generally make efforts to avoid them in the future, which doesn't happen for LLMs until that data gets incorporated into the training for the next version of the model, which can take months to years. That's why AI companies are trying to capture and store everything from user interactions, which is a privacy nightmare.

It's not a compelling argument to compare AI behavior to that of a dysfunctional human brain and go "see, humans do this too, teehee!" Not when the whole selling point of these things is that they supposed to be smarter and less fallible than most humans.

I'm deliberately trying not to be ableist in my wording here, but it's like saying, "hey, you know what would do wonders for productivity and shareholder value? If we fired half our workforce, then found someone with no experience, short-term memory loss, ADHD and severe untreated schizophrenia, then put them in charge of writing mission-critical code, drafting laws, and making life-changing medical and business decisions."

I'm not saying LLMs aren't technically fascinating and a breakthrough in AI development, but the way they have largely been marketed and applied is scammy, misleading, and just plain irresponsible.