12  Large Language Models

~8 hours How LLMs Work Conceptual + Practical

Learning Objectives

  • Understand how LLMs are trained and how they generate text
  • Grasp the Transformer architecture conceptually
  • Learn about RLHF and alignment
  • Use LLMs effectively through APIs and prompting
  • Understand limitations and risks

Table of Contents

11.1 What Are LLMs?

Large Language Models (LLMs) are neural networks trained to predict the next word in a sequence. Despite this simple objective, scaling up has led to emergent abilities: reasoning, coding, translation, and more.

The Key Insight

LLMs are "just" predicting the next word. But to predict well, they must learn grammar, facts, reasoning patterns, and even social conventions--all compressed into billions of parameters.

The LLM Landscape

Model Family Organization Access
GPT-4, GPT-4o OpenAI API, ChatGPT
Claude 3, 3.5 Anthropic API, claude.ai
Gemini Google API, Gemini app
LLaMA 2, 3 Meta Open weights
Mistral, Mixtral Mistral AI Open weights, API

11.2 How LLMs Work

Tokenization

Text is split into tokens--subword units. "Tokenization" might become ["Token", "ization"]. This handles rare words by breaking them into common pieces. 

# Example: How text becomes tokens
text = "Hello, how are you doing today?"

# Tokenized (roughly):
tokens = ["Hello", ",", " how", " are", " you", " doing", " today", "?"]

# Each token maps to an integer ID
token_ids = [15496, 11, 703, 527, 499, 3815, 3432, 30]

# Models work with token IDs, not text
# Typical: 1 token is approximately 0.75 words

The Transformer Architecture

LLMs use the Transformer architecture (Vaswani et al., 2017). The key innovation is self-attention, which lets each token "look at" all other tokens when computing its representation.

Self-Attention Intuition

Consider: "The cat sat on the mat because it was tired."

To predict what comes after "it," the model needs to know that "it" refers to "cat." Self-attention computes a weighted average of all previous tokens, with weights learned to capture such relationships.

Next-Token Prediction

Given tokens [t1, t2, ..., tn], the model outputs a probability distribution over the vocabulary for t(n+1). During generation, it samples from this distribution and repeats. 

# Simplified generation loop
prompt = "The capital of France is"
tokens = tokenize(prompt)  # [464, 3361, 315, 9822, 374]

for _ in range(max_tokens):
    # Model outputs probability for each vocabulary word
    probs = model(tokens)  # Shape: [vocab_size]

    # Sample next token (or take argmax for greedy)
    next_token = sample(probs, temperature=0.7)

    # Append and continue
    tokens.append(next_token)

    if next_token == END_TOKEN:
        break

# Detokenize back to text
output = detokenize(tokens)
# "The capital of France is Paris."

Temperature and Sampling

  • Temperature = 0: Always pick the most likely token (deterministic)
  • Temperature = 1: Sample according to model probabilities
  • Temperature > 1: More random/creative
  • Temperature < 1: More focused/deterministic

11.3 Training LLMs

Pre-training

LLMs are trained on massive text corpora--often trillions of tokens from the web, books, code, and more. The objective is simple: predict the next token. This is called self-supervised learning because no human labels are needed.

Model Parameters Training Tokens
GPT-2 1.5B ~40B
GPT-3 175B ~300B
LLaMA 2 70B 2T
GPT-4 ~1.8T (rumored) ~13T (rumored)

Scaling Laws

Kaplan et al. (2020) and Hoffmann et al. (2022) showed that performance scales predictably with compute, data, and parameters. More of each leads to better models. This motivated the race to train ever-larger models.

11.4 RLHF and Alignment

Pre-trained models are good at predicting text but not at being helpful or safe. RLHF (Reinforcement Learning from Human Feedback) fine-tunes models to behave as intended.

The RLHF Process

  1. Supervised Fine-Tuning (SFT): Train on high-quality examples of helpful responses
  2. Reward Model: Train a model to predict which responses humans prefer
  3. RL Fine-Tuning: Optimize the LLM to maximize the reward model's score
Why RLHF Matters

Without RLHF, models might:

  • Be unhelpfully verbose or terse
  • Follow harmful instructions
  • Hallucinate confidently
  • Be inconsistent in style

RLHF teaches models to be helpful, harmless, and honest--the "HHH" criteria.

Constitutional AI

Anthropic's approach uses a set of principles (a "constitution") to guide AI behavior. The model critiques and revises its own outputs according to these principles, reducing reliance on human labeling.

11.5 Prompt Engineering

Prompt engineering is the art of crafting inputs that elicit the best outputs from LLMs. Small changes in wording can dramatically affect results.

Key Techniques

Technique Description
Few-shot Provide examples of input-output pairs
Chain-of-thought Ask model to show reasoning steps
Role prompting "You are an expert economist..."
Structured output Request JSON, markdown, or specific format 
# Zero-shot
prompt = "Translate to French: Hello, how are you?"

# Few-shot
prompt = """Translate to French:
English: Hello
French: Bonjour

English: Thank you
French: Merci

English: How are you?
French:"""

# Chain-of-thought
prompt = """Solve step by step:
If a train travels 120 miles in 2 hours, then continues
for 3 more hours at the same speed, what is the total distance?

Let's think step by step:"""

# Role + structured output
prompt = """You are a data analyst. Given this data,
identify the top 3 trends. Format as JSON:
{"trends": ["trend1", "trend2", "trend3"]}"""

11.6 Using LLM APIs

# Using OpenAI API
from openai import OpenAI

client = OpenAI()  # Uses OPENAI_API_KEY env var

response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain regression discontinuity."}
    ],
    temperature=0.7,
    max_tokens=500
)

print(response.choices[0].message.content)

# Using Anthropic API
from anthropic import Anthropic

client = Anthropic()

message = client.messages.create(
    model="claude-3-sonnet-20240229",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Explain difference-in-differences."}
    ]
)

print(message.content[0].text)

11.7 Limitations and Risks

Known Limitations

  • Hallucinations: LLMs confidently generate false information
  • Knowledge cutoff: Training data has a date limit
  • Context limits: Can only process limited text at once
  • Math/reasoning: Struggles with complex calculations
  • No real-time info: Can't access the internet (unless given tools)
Critical for Research

Never trust LLM outputs without verification. They can generate plausible-sounding but false citations, statistics, and claims. Always verify facts from primary sources.

Ethical Considerations

  • Bias: Models reflect biases in training data
  • Privacy: May memorize and reveal private information
  • Misuse: Can be used for spam, disinformation, cheating
  • Environmental: Training requires enormous compute/energy

11.8 LLMs in Research

LLMs are increasingly used as research tools--for coding, literature review, data analysis, and writing. Use them wisely.

Appropriate Uses

  • Debugging code
  • Explaining concepts
  • Brainstorming research questions
  • Drafting and editing prose
  • Translating between programming languages

Inappropriate Uses

  • Generating citations (they hallucinate!)
  • Performing calculations you can't verify
  • Replacing critical thinking
  • Submitting AI-generated work as your own (check your institution's policy)
Further Reading