Running Models Locally 🏠

Why run a model yourself?

A cloud model is someone else's computer. That is often exactly right: you get the most capable models with zero setup. But it means your data leaves your machine, you pay per token, and you need a connection. Running an open-weight model locally flips all three.

Ollama: the easy way in

Ollama is the simplest way to run open models. Install it, pull a model, and it runs a local server on your machine that speaks a familiar API. Downloading a model is one command:

# install Ollama from ollama.com, then:
ollama pull llama3.2          # download an open model (a few GB)
ollama run llama3.2           # chat with it right in the terminal

# it now serves an API at http://localhost:11434

That is genuinely it. You are now running a capable language model with no account, no key, no bill, and no data leaving your laptop. The models are open-weight releases from Meta (Llama), Mistral, Google (Gemma), Alibaba (Qwen) and others, and there are small ones (a few gigabytes, runs on a laptop) up to large ones (needs a serious GPU).

Talking to it from Python

# Ollama exposes a simple local HTTP API. requests (Lesson 42) is all you need.
import requests


def ask_local(prompt, model="llama3.2"):
    """Send a prompt to a model running locally via Ollama."""
    response = requests.post(
        "http://localhost:11434/api/chat",
        json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "stream": False,
        },
        timeout=120,      # local models can be slow on modest hardware
    )
    response.raise_for_status()
    return response.json()["message"]["content"]


# print(ask_local("Explain recursion in one sentence."))
print("Same chat shape as the cloud: messages in, a reply out. Just a different URL.")

Look at the shape: a list of messages, a reply out. It is the same mental model as the cloud API, which is the whole point. Ollama even offers an OpenAI-compatible endpoint, so much cloud code runs against it by changing only the base URL and dropping the key.

# Ollama also has its own tidy Python library (pip install ollama)
import ollama


def chat_local(messages, model="llama3.2"):
    """A local chat that streams, mirroring Lesson 56's cloud version."""
    stream = ollama.chat(model=model, messages=messages, stream=True)
    reply = ""
    for chunk in stream:
        piece = chunk["message"]["content"]
        reply += piece
        print(piece, end="", flush=True)
    print()
    return reply


# Same streaming feel as the cloud, running entirely on your machine.
print("Local streaming chat: identical ergonomics, zero data leaving the room.")

The honest limitations

The hybrid pattern: use both

You do not have to choose once and forever. A smart assistant routes: cheap, private, local for the easy majority of tasks, and a frontier cloud model for the hard minority. This is exactly the "pick the right tool" thinking from Base Camp 4 and the performance lesson (Lesson 51), applied to models.

def route(task_difficulty, is_private):
    """Choose local or cloud based on the task. A real router might ask a
    cheap model to judge difficulty, or use rules like these."""
    if is_private:
        return "local"                      # sensitive data never leaves
    if task_difficulty == "hard":
        return "cloud-frontier"             # worth the cost and the round trip
    return "local"                          # the cheap, private default


for difficulty, private in [("easy", False), ("hard", False), ("easy", True), ("hard", True)]:
    choice = route(difficulty, private)
    print(f"{difficulty:5} / private={private!s:5} -> {choice}")

easy  / private=False -> local
hard  / private=False -> cloud-frontier
easy  / private=True  -> local
hard  / private=True  -> local

Note the bottom row: a hard and private task stays local even though local is weaker, because the privacy requirement outranks the capability preference. Encoding that priority in your router, rather than always reaching for the best model, is what makes an assistant trustworthy.

def ask(prompt, backend="local"):
    """One interface, two backends. The caller never has to know which."""
    if backend == "local":
        import requests
        r = requests.post(
            "http://localhost:11434/api/chat",
            json={"model": "llama3.2",
                  "messages": [{"role": "user", "content": prompt}],
                  "stream": False},
            timeout=120,
        )
        r.raise_for_status()
        return r.json()["message"]["content"]

    if backend == "cloud":
        import anthropic
        client = anthropic.Anthropic()
        resp = client.messages.create(
            model="claude-sonnet-5", max_tokens=1024,
            messages=[{"role": "user", "content": prompt}],
        )
        return resp.content[0].text

    raise ValueError(f"unknown backend: {backend}")


# The rest of Jarvis calls ask(prompt, backend) and stays blissfully unaware.
print("One function, two worlds. Swap backend without touching anything else.")