The critical thing to remember about LLMs is that they are probabilistic in nature. They don’t know facts, they don’t reason, they don’t evaluate. All they do is take your input string, split that string into tokens that are about 3-4 characters long, and then go back into their vast, vast, pretrained database and say “I have this series of tokens. In the past when similar sets of tokens were given, what were the tokens that were most likely to be associated with them?” It will then construct the output string one token at-a-time (more sophisticated models can do multiple tokens at once so that words, phrases and sentences might hang together better) until the output is complete (the probability of the next token being relevant drops below some threshold value) or your output limit is reached.

Pretty much.

But it only takes words as input. Numbers are just another type of word. Sentences are just a series of words with a “.” word on the end.

This is why when you get a LLM to do basic math it fails so often - it has no concept of a number or operations on those numbers.

There’s a part of our brain called the salience network, that continually models and predicts our environment and directs our conscious attention to things it can’t predict. When we talk to each other, most of the formal content is predictable, and the salience network filters it out; the unpredictable part that’s left is the actual meaningful part.

LLMs basically recreate the salience network. They continually model and predict the content of the text stream the same way we do—except instead of modeling someone else’s words so they can extract the unpredictable/meaningful part, they model their own words so they can keep predicting the next ones.

This raises an obvious issue: when our salience networks process the stream of words coming out of such an LLM, it’s all predictable, so our brains tell us there’s no actual message. When AI developers ran into this, they added a feature called “temperature” that basically injects randomness into the generated text—enough to make it unpredictable, but not obvious nonsense—so our salience networks will get fooled into thinking there’s meaningful content.

mathematically “correct” sounding output

It’s hard to say because that’s a rather ambiguous way of describing it (“correct” could mean anything), but it is a valid way of describing its mechanisms.

“Correct” in the context of LLMs would be a token that is likely to follow the preceding sequence of tokens. In fact, it computes a probability for every possible token, then takes a random sample according to that distribution* to choose the next token, and it repeats that until some termination condition. This is what we call maximum likelihood estimation (MLE) in machine learning (ML). We’re learning a distribution that makes the training data as likely as possible. MLE is indeed the basis of a lot of ML, but not all.

*_{Oversimplification.}

And our brains probably do something very similar. If llms are not intelligent our brains aren’t either.