How Does AI Actually Work On The Inside?
You type a question into ChatGPT, Claude, Gemini or another AI:
"Why is the sky blue?"
A moment later, several paragraphs appear explaining the answer.
It can feel as though the AI has understood your question, thought about it and then written a response.
What's actually happening is very different — and rather remarkable.
Behind that simple conversation are tokens, embeddings, neural networks, parameters and weights, working together on an enormous scale.
You don't need to understand the mathematics to get a useful picture of how it works.
It starts with training
Before an AI model can answer your questions, it has to be trained.
During training, a large language model is exposed to enormous amounts of text and other data. Depending on the model, this can include material from websites, books, articles, code and other sources.
The model doesn't simply store all this information as a giant library that it can search later.
Instead, training changes billions — and in some models hundreds of billions — of numerical values inside the model.
These are its parameters.
To understand how that happens, we first need to talk about tokens.
AI doesn't see words quite like we do
When you read:
"The cat sat on the mat."
you see six words and understand what they mean.
A language model doesn't receive text in quite the same way.
Text is first broken into smaller pieces called tokens.
A token might be a whole word, part of a word, punctuation or another small piece of text.
For example, a sentence might effectively become something like:
"The" | "cat" | "sat" | "on" | "the" | "mat" | "."
Each token is represented by a number.
But simply numbering pieces of text isn't enough. The AI needs some way of learning relationships between them.
That's where embeddings come in.
Embeddings turn tokens into something the AI can work with
An embedding represents a token using a collection of numbers.
Those numbers place it within a huge mathematical space where useful relationships can be represented.
Words and concepts that occur in similar contexts can develop related representations.
So concepts such as:
cat
dog
rabbit
may have relationships that are different from:
car
train
aeroplane
This doesn't mean the AI has a dictionary inside it saying that a cat and dog are animals.
Those relationships have emerged mathematically through training.
Embeddings give the model a much richer representation of language than simply assigning each word a number.
Then comes the neural network
At the heart of a modern language model is a large neural network.
Despite the name, this isn't a miniature electronic brain.
It's a vast mathematical system made up of interconnected calculations.
Information passes through many layers of this network.
During training, the model is repeatedly given text and asked, in simplified terms:
What is likely to come next?
Imagine it encounters:
"She poured the tea into the…"
It might initially give too much probability to:
car
and not enough to:
cup
The training process measures how far its predictions are from the expected result and makes tiny adjustments throughout the neural network.
Those adjustments change the model's weights.
What are weights?
Weights are numerical values controlling how strongly different pieces of information influence calculations inside the neural network.
There are an extraordinary number of them.
During training, these values are adjusted over and over again as the model processes examples.
One individual weight tells you almost nothing useful.
But billions of weights working together can represent extraordinarily complicated patterns.
Those patterns can include relationships involving language, facts, grammar, writing styles, programming, reasoning patterns and connections between concepts.
This leads us to another word you'll often encounter: parameters.
Weights and parameters
A parameter is a value within the model that is learned during training.
Most of the parameters in a large language model are weights, although there can be other learned values too.
So when you hear that an AI has "billions of parameters", you're hearing something about the enormous number of adjustable numerical values that make up its learned behaviour.
Training determines what those values become.
Once training is complete, those learned parameters collectively form much of what we call the model.
How does predicting text become intelligence?
This is one of the most surprising parts.
Predicting the next piece of text sounds rather basic.
Your phone has been predicting your next word for years, and nobody would mistake that for ChatGPT.
The difference is largely one of scale and sophistication.
To become extremely good at predicting what comes next across an enormous variety of text, a model has to learn a remarkable number of underlying patterns.
Consider:
"Paris is the capital of…"
Predicting France requires a factual relationship.
But consider:
"If Sarah has three apples and gives one to Tom, she has…"
Now the model needs a pattern involving arithmetic and language.
Or:
"Dear Sir or Madam, I am writing to complain…"
Predicting what follows involves recognising the structure and style of a formal letter.
Programming code, stories, recipes, explanations and conversations all contain their own patterns and relationships.
At sufficient scale, learning to predict what comes next requires the model to capture a surprisingly rich representation of language and the patterns contained within its training data.
Transformers and attention
Modern large language models generally use an architecture called a Transformer.
One of its important ideas is called attention.
Attention helps the model determine which parts of the text it's processing are particularly relevant to other parts.
Consider:
"John put the chocolate in the cupboard because the fridge was broken. Later, he went back to get it."
What does "it" refer to?
You immediately understand that it probably means the chocolate.
The model needs mechanisms for recognising relationships like this too.
Attention allows different tokens to influence one another according to their relevance.
And it does this across many layers and many different attention calculations.
Training is finished. Now you ask a question.
Suppose you type:
"Why do leaves change colour in autumn?"
The AI doesn't normally search through its original training material looking for a paragraph containing the answer.
Instead, your prompt is processed by the trained model.
First, your text is converted into tokens.
Those tokens are transformed into embeddings.
They then pass through the layers of the neural network.
The model's learned parameters and weights influence the calculations throughout those layers.
Attention helps it take account of relationships between the different parts of your prompt and the text already generated.
Eventually the model produces probabilities for what token should come next.
Perhaps possibilities include:
Leaves — 18%
During — 12%
In — 9%
Trees — 6%
These numbers are only illustrative, but the principle is important.
The model produces a range of possible next tokens with different probabilities.
A token is selected.
Now perhaps the answer begins:
"During"
The model then runs again, taking into account:
Your original question + "During"
and predicts the next token.
Perhaps:
"autumn"
Then it runs again.
Your question + "During autumn"
And again.
And again.
Your answer is being created as you watch
This is an important point.
The AI usually isn't composing the entire answer somewhere behind the scenes and then revealing it one word at a time.
It is generating the response token by token.
Very roughly:
Your prompt
↓
Tokens
↓
Embeddings
↓
Transformer neural network
↓
Attention + billions of learned parameters
↓
Probabilities for the next token
↓
Select a token
↓
Add it to the conversation
↓
Predict again
↓
Repeat
It happens so quickly that you see sentences appearing almost immediately.
So is AI just guessing the next word?
Yes — but "just guessing" gives the wrong impression.
It's a little like describing a Formula 1 car as "just turning its wheels".
Technically true, but it misses nearly everything interesting about how it achieves the result.
A modern language model is calculating the next token using the patterns represented by billions of learned parameters, your prompt, the conversation so far and the relationships it identifies between them.
That can produce remarkably sophisticated behaviour.
Why doesn't it give exactly the same answer every time?
The model doesn't necessarily choose the single most probable token every time.
There can be an element of controlled variation in how tokens are selected.
That means asking exactly the same question twice can produce different wording — or sometimes a somewhat different answer.
Settings can make the model more predictable or more varied.
This variation is useful for things such as writing and brainstorming, but it is also one reason AI shouldn't be thought of as a traditional database that simply retrieves a fixed answer.
And this explains why AI can get things wrong
The same mechanism that makes generative AI so flexible also creates one of its biggest weaknesses.
A language model is designed to generate a plausible continuation.
It isn't inherently performing a database lookup to establish whether every sentence it generates is true.
If the patterns within the model lead towards a convincing but incorrect answer, it can produce one.
This is sometimes called a hallucination.
The answer may sound confident, be beautifully written and still be wrong.
Modern AI systems use additional techniques and tools to reduce this problem. Some can search the web, retrieve information from documents, use calculators or call other software.
But the underlying language model is still generating tokens.
That's why important AI answers should be checked rather than trusted simply because they sound convincing.
Putting all the pieces together
The terminology can initially sound intimidating, but the main concepts fit together quite neatly.
Tokens are the pieces into which text is broken.
Embeddings turn those pieces into numerical representations the model can work with.
A neural network performs the enormous collection of calculations that processes those representations.
A Transformer is the type of neural-network architecture behind most modern large language models.
Attention helps the Transformer work out which parts of the text are relevant to one another.
Parameters are the values the model learns during training.
Weights make up most of those parameters and influence how information flows through the network.
Training repeatedly adjusts those parameters so that the model becomes better at predicting.
And when you provide a prompt, all of this machinery is used to predict a response, one token at a time.
The remarkable part
There isn't a tiny encyclopaedia hidden inside ChatGPT.
Nor is there a little person inside it thinking about your question.
What exists is an enormous mathematical model whose parameters have been shaped by training.
Your words enter as tokens. They become numerical representations. Those representations travel through layer after layer of mathematical operations influenced by billions of learned values.
At the other end come probabilities.
One token is generated.
Then another.
Then another.
And from that process can emerge a recipe, a computer program, a poem, an explanation of quantum physics — or an answer to a simple question about why leaves change colour.
That's the strange and fascinating idea at the heart of today's generative AI.
