How language models work

What does a language model do?

A language model does, in essence, the same thing as your cell phone keyboard when it suggests the next word: predict which word is most likely to come next, given the previous text.

The difference is one of scale. Mobile autocomplete works with simple patterns (what words usually go together). A language model like GPT-4, Claude, or Gemini has been trained with enormous amounts of text and is capable of generating long, coherent sequences, maintaining the thread of a conversation, or producing an entire essay. But the underlying mechanism is the same: calculate the probability of the next word and choose one.

This is important: a language model does not "understand" text in the human sense of the term. It has no beliefs, no intentions, no representation of the world. Calculates probabilities on sequences of text.

Where do these probabilities come from?

The probabilities that a language model handles come from its training: a process in which the model has processed massive amounts of text (books, articles, web pages, forums, technical documentation...) and has extracted statistical patterns from them.

What kind of patterns? Basically, what words, phrases and structures tend to appear together and in what contexts. For example:

• After "good", it is very likely that "days" or "results" will appear.
• After "the patient has a fever and...", medical terms are likely to follow.
• A text that begins with "Once upon a time..." will probably continue with a narrative tone.

An analogy: if you had read thousands of cooking recipes, you could "predict" that after "add salt and..." probably comes "pepper." Not because you understand cooking or because pepper is the correct answer in all cases, but because it is the most frequent pattern in the texts you have seen. That's exactly what a language model does, but with all types of texts and on a vastly larger scale.

Key concept

The model does not memorize concrete data or store a database of facts. What it stores are probability distributions over sequences of text: statistical patterns that allow it to generate new text that looks like the text it was trained on.

In more depth

Introduction to Machine Learning

Implications of being a statistical tool

Understanding that a language model is, fundamentally, a statistical prediction tool allows us to understand both its capabilities and its limitations.

There is no understanding, there is correlation

The model produces text that is statistically plausible, not necessarily true. This has a direct consequence known as "hallucination": the model can easily generate false information, invented quotes or non-existent data.

This is not a "bug" in the usual sense. The model does not distinguish between true and false because it does not have access to "the truth": its only criterion is whether a text sequence is consistent with the statistical patterns it has learned. A made-up fact may be perfectly plausible from a statistical point of view.

Practical implication

Any information generated by a language model must be verified by the user. The model is not a reliable source of information: it is a text generation tool.

The prompt modifies the probabilities

The text we provide as input (prompt) directly conditions the probabilities of the output. By changing the formulation of a question, we change the statistical context and, therefore, the distribution of probable answers.

This explains why:

• Rephrasing a question can produce very different answers.
• Giving detailed instructions improves results (we provide more statistical context).
• Prompt engineering works: it is not "magic", it is the manipulation of the context that the model uses to calculate its probabilities.

Inherent limitations

• Frozen Knowledge: The model can only generate text based on the patterns learned during its training. You do not know what has happened after your court date unless you are given that information.
• Approximation, not reasoning: The model can produce text sequences that look like logical reasoning, because it has seen many examples of reasoning in its training data. But it does not perform formal logical deduction: it reproduces text patterns that are associated with reasoning.
• Uncalibrated Confidence: The model generates text with the same fluidity regardless of whether the information is correct or not. It does not have an internal mechanism of doubt or uncertainty.

Biases as a statistical consequence

The term "bias" in the context of language models refers to the systematic tendency of a model to produce results that reflect certain associations, stereotypes, or imbalances. However, it is essential to understand where this bias comes from so as not to fall into simplistic analyses.

Why does a language model reproduce biases?

The answer is straightforward: because your training data contains them.

A language model learns statistical patterns from the texts with which it was trained. If in those texts certain professions are more frequently associated with a gender, if certain groups are described with different adjectives, or if certain stereotypes appear recurrently, the model will learn those associations because that is exactly what it does: extract and reproduce the statistical regularities of your data.

This is not a "defect" of the model, nor a "decision" that the model makes. It is the direct result of its design: being a statistical reflection of the texts it has processed. And these texts are a reflection of the world in which they were written, with its imbalances and its conventions.

Perspective

The question is not "why is AI biased?", but: what does the data on which it was trained contain, and what are the implications of using a tool that faithfully reproduces the statistical regularities of that data?

Types of biases in data

• Sampling bias: training data does not adequately represent all groups. For example, a facial recognition system trained predominantly with images of light-skinned people may be less accurate with other skin tones, simply because it has fewer examples to learn from.

• Historical bias: the data reflects social prejudices of the time and place in which they were generated. A documented example: Amazon's resume selection system (2018) penalized CVs that contained the word "woman" because it had been trained with historical data from an industry where the majority of hires were men. The model learned that statistical correlation.

• Measurement bias: data is collected or measured in a way that does not accurately represent what we want to evaluate. The measured variable does not faithfully capture the actual characteristic.

• Algorithmic bias: introduced by the design decisions of the algorithm itself, such as which function is optimized or what simplifications are assumed. It may be that the metric being optimized does not capture aspects such as equity.

• Labeling bias: the people who label the training data introduce their own prejudices and preconceptions into the labels, which the model learns as if they were objective truths.

Examples

The grading algorithm in the United Kingdom (2020)

• Problem: Due to the cancellation of exams due to the pandemic, the British government used an algorithm to predict students' grades.
• The bias: The model gave significant weight to the school's historical performance. This systematically penalized bright students from historically poorer schools (generally in lower-income areas), while benefiting students from elite schools.
• Consequence: Thousands of students saw their grades unfairly lowered, which prevented them from accessing their desired university. The case sparked massive protests and the government had to back down.
• Analysis: This is a case where the bias is not a technical failure of the algorithm, but rather the direct consequence of using historical data that reflects pre-existing socioeconomic inequalities. The algorithm did exactly what it was designed to do: reproduce the statistical distributions of the data. The problem was in the decision to use these distributions as individual evaluation criteria.

Biases corrected with biases

This example of Google Gemini generating historically inaccurate images is especially illustrative. Trying to correct for a statistical bias (the overrepresentation of white people in the training data) with an artificial correction produced absurd results. It demonstrates the fundamental difficulty of "fixing" biases that reflect real data:

"You can't have a system that is unbiased, or perceived as unbiased by everyone" (Yann LeCun)

From biases to ethics

Understanding that biases are an inherent consequence of the statistical performance of these models — and not a “defect” that can simply be eliminated — is the foundation for critical and responsible use of AI. The ethical implications of this reality and practical considerations for teachers are addressed in the [Ethics in AI] section (61-etica.md).

Reasoning models

Reasoning models represent an evolution that improves results in tasks that require structured thinking, at the cost of greater processing time. They integrate the chain of thought (CoT) strategy into their operation: they have been trained with explicit examples of step-by-step reasoning, which allows them to generate intermediate sequences that decompose a problem into simpler steps.

This has allowed significant improvements in mathematical and logical benchmarks. However, it's important to keep perspective: these models are still statistical tools. Their "reasoning" consists of generating sequences of text that look like logical reasoning, because they have been trained with many examples of reasoning. This allows them to solve many more problems than conventional models, but it does not imply that they perform formal logical deduction in the strict sense of the term.