How Language Models Actually Work 🧠

The one-sentence version

A large language model is a function that, given some text, predicts the next chunk of text. That is the whole thing. Everything else, the conversations, the code, the apparent reasoning, is that one operation run over and over, very fast, at enormous scale.

Tokens: the pieces text is chopped into

Models do not see letters or words. They see tokens: chunks that are often a word, sometimes part of a word, sometimes punctuation. Here is a toy tokeniser that splits on the same rough boundaries, so you can feel the idea:

import re

def toy_tokenise(text):
    """Split into word-ish and punctuation tokens. A real tokeniser is cleverer."""
    return re.findall(r"\w+|[^\w\s]", text)


sentence = "Jarvis, what's the weather? It's 25 degrees."
tokens = toy_tokenise(sentence)

print(tokens)
print(f"{len(tokens)} tokens for {len(sentence)} characters")

['Jarvis', ',', 'what', "'", 's', 'the', 'weather', '?', 'It', "'", 's', '25', 'degrees', '.']
14 tokens for 44 characters

Real tokenisers (the model's, not this toy) use subword pieces learned from data, so a common word is one token and a rare one is several. A useful rule of thumb for English: one token is about four characters, or roughly ¾ of a word. This matters because you pay per token, and the context window is measured in tokens.

Prediction, made concrete

"Predict the next token" sounds abstract, so here is a tiny model that genuinely does it. It reads some text, learns which word tends to follow which, and then generates new text by sampling. This is a Markov chain: a language model with the intelligence turned almost all the way down, but the exact same shape.

import random
from collections import defaultdict

random.seed(7)

corpus = """the cat sat on the mat the cat ate the fish
the dog sat on the log the dog ate the bone the cat ran"""

# Learn: for each word, which words have followed it?
following = defaultdict(list)
words = corpus.split()
for current, nxt in zip(words, words[1:]):
    following[current].append(nxt)

# Generate: start somewhere, then keep predicting the next word
word = "the"
output = [word]
for _ in range(11):
    choices = following[word]
    word = random.choice(choices)
    output.append(word)

print(" ".join(output))

the log the dog sat on the log the cat sat on

That is next-token prediction, in eight lines. A real model does the same thing, but instead of a lookup table of one-word history it has hundreds of billions of learned parameters weighing the entire preceding text, so its "which comes next" is breathtakingly more informed. The mechanism is identical; the sophistication is not.

Temperature: the creativity dial

The model does not output one next token, it outputs a probability for every possible token. Temperature controls how you pick from those probabilities. Low temperature always takes the likeliest; high temperature spreads the love. Here is the actual maths, on a toy set of scores:

import math

def softmax_with_temperature(scores, temperature):
    """Turn raw scores into probabilities, sharpened or flattened by temperature."""
    scaled = [s / temperature for s in scores]
    biggest = max(scaled)
    exps = [math.exp(s - biggest) for s in scaled]
    total = sum(exps)
    return [e / total for e in exps]


tokens = ["cat", "dog", "banana"]
scores = [3.0, 2.0, 0.5]        # the model's raw confidence in each

for temp in [0.2, 1.0, 2.0]:
    probs = softmax_with_temperature(scores, temp)
    shown = ", ".join(f"{t} {p:.0%}" for t, p in zip(tokens, probs))
    print(f"temp {temp}: {shown}")

temp 0.2: cat 99%, dog 1%, banana 0%
temp 1.0: cat 69%, dog 25%, banana 6%
temp 2.0: cat 53%, dog 32%, banana 15%

At temperature 0.2 the model almost always says "cat": predictable, focused, a little boring. At 2.0 it will surprise you, sometimes wonderfully, sometimes with "banana". For an assistant that answers factual questions you want low temperature; for a brainstorming partner, higher. Note: some of the newest models manage this internally and do not expose a temperature knob, which is a design choice, not a limitation.

The context window: its entire short-term memory

A model has no memory between calls. Everything it "knows" about your conversation is the text you send it each time, and that text has a size limit called the context window, measured in tokens. Modern windows are large (the models you will use hold around a million tokens, hundreds of pages), but they are finite.

• Everything counts: your system prompt, the whole conversation history, any documents you paste in, and the reply being generated.
• When you build a chatbot, you resend the history every turn. The model is not remembering; you are reminding. Lesson 55 builds exactly this.
• Run past the window and the oldest text must be dropped or summarised, which is why long chats sometimes "forget" the start.

What this means for building Jarvis

import re

def toy_tokenise(text):
    return re.findall(r"\w+|[^\w\s]", text)


for text in ["Hello!", "antidisestablishmentarianism", "don't", "user@example.com"]:
    print(f"{text!r:35} -> {toy_tokenise(text)}")

print()
print("A 500-word email is roughly", round(500 / 0.75), "tokens")

'Hello!'                            -> ['Hello', '!']
'antidisestablishmentarianism'      -> ['antidisestablishmentarianism']
"don't"                             -> ['don', "'", 't']
'user@example.com'                  -> ['user', '@', 'example', '.', 'com']

A 500-word email is roughly 667 tokens

import random
from collections import Counter

random.seed(1)

corpus = "the cat sat the cat ran the cat ate the dog sat"
words = corpus.split()

following = {}
for current, nxt in zip(words, words[1:]):
    following.setdefault(current, Counter())[nxt] += 1


def next_word(word, greedy):
    counts = following.get(word)
    if not counts:
        return "the"
    if greedy:
        return counts.most_common(1)[0][0]      # always the likeliest: temp near 0
    population = list(counts.elements())
    return random.choice(population)            # weighted by frequency: temp near 1


for greedy in (True, False):
    word, out = "the", ["the"]
    for _ in range(7):
        word = next_word(word, greedy)
        out.append(word)
    label = "greedy (cold)" if greedy else "sampled (warm)"
    print(f"{label:15} {' '.join(out)}")

greedy (cold)   the cat sat the cat sat the cat
sampled (warm)  the cat ate the cat sat the dog