10 History of NLP » Text Analysis Today

Module 10 showed you how we got here. This module shows you what to do now. We walk through every major technique used in modern text analysis — from basic preprocessing to transformer-based models and LLM APIs — with production-ready Python code you can run today. A single running example (analyzing Federal Reserve communications) ties every technique together so you can see how each tool fits into a real research or business pipeline.

10A.0  The Text Analysis Pipeline

Every text analysis project — whether a hedge fund parsing earnings calls or a political scientist studying parliamentary speeches — follows the same core pipeline. The techniques differ, but the structure is universal:

The single most consequential decision you make is at the Represent stage. The table below summarizes the three tiers of text representation available today, from simplest to most powerful:

10A.1  Environment Setup

Before writing any analysis code, set up a reproducible environment. Pin your library versions so that anyone can replicate your results months or years later.

Installation

# requirements.txt — pin EXACT versions for reproducibility
# Save this file and run: pip install -r requirements.txt

# Core NLP
nltk==3.9.1
spacy==3.8.4

# Classical ML & text representation
scikit-learn==1.6.1
gensim==4.3.3

# Transformers & embeddings
transformers==4.47.1
sentence-transformers==3.4.1
torch==2.5.1

# Topic modeling
bertopic==0.16.4

# Financial sentiment
pysentiment2==0.1.1

# LLM APIs
anthropic==0.42.0
openai==1.58.1

# Data & visualization
pandas==2.2.3
matplotlib==3.9.3
seaborn==0.13.2
wordcloud==1.9.4

# After installing, download required models & data
import nltk
nltk.download('punkt_tab')
nltk.download('stopwords')
nltk.download('wordnet')
nltk.download('vader_lexicon')

# Download spaCy English model
# Run in terminal: python -m spacy download en_core_web_sm

Reproducibility Seed

Set random seeds once at the top of every script. This ensures that any method involving randomness (topic models, train/test splits, embeddings initialization) gives identical results every time.

import random
import numpy as np

def set_seed(seed=42):
    """Set all random seeds for full reproducibility."""
    random.seed(seed)
    np.random.seed(seed)
    try:
        import torch
        torch.manual_seed(seed)
        torch.cuda.manual_seed_all(seed)
        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False
    except ImportError:
        pass

set_seed(42)

Loading Our Running Example

We will work with a small corpus of FOMC press statements. In practice, you would scrape these from the Federal Reserve website. For this tutorial, we define a sample directly:

import pandas as pd

# Sample FOMC statements (abbreviated for pedagogy)
fomc_data = [
    {
        "date": "2008-12-16",
        "text": """The Federal Open Market Committee decided today to establish a target range
for the federal funds rate of 0 to 1/4 percent. Since the Committee's last meeting,
labor market conditions have deteriorated, and the available data indicate that
consumer spending, business investment, and industrial production have declined.
Financial markets remain quite strained and credit conditions tight. Overall, the
outlook for economic activity has weakened further."""
    },
    {
        "date": "2015-12-16",
        "text": """The Committee judges that there has been considerable improvement in labor
market conditions this year, and it is reasonably confident that inflation will
rise, over the medium term, to its 2 percent objective. Given the economic outlook,
the Committee decided to raise the target range for the federal funds rate to
1/4 to 1/2 percent. The stance of monetary policy remains accommodative after
this increase, thereby supporting further improvement in labor market conditions."""
    },
    {
        "date": "2020-03-15",
        "text": """The coronavirus outbreak has harmed communities and disrupted economic activity
in many countries, including the United States. The effects of the coronavirus
will weigh on economic activity in the near term and pose risks to the economic
outlook. In light of these developments, the Committee decided to lower the target
range for the federal funds rate to 0 to 1/4 percent. The Committee expects to
maintain this target range until it is confident that the economy has weathered
recent events."""
    },
    {
        "date": "2022-06-15",
        "text": """Overall economic activity appears to have picked up after edging down in the
first quarter. Job gains have been robust in recent months, and the unemployment
rate has remained low. Inflation remains elevated, reflecting supply and demand
imbalances related to the pandemic, higher energy prices, and broader price
pressures. The Committee decided to raise the target range for the federal funds
rate to 1-1/2 to 1-3/4 percent and anticipates that ongoing increases in the
target range will be appropriate."""
    },
    {
        "date": "2024-09-18",
        "text": """Recent indicators suggest that economic activity has continued to expand at a
solid pace. Job gains have slowed but remain solid, and the unemployment rate has
moved up but remains low. Inflation has made further progress toward the Committee's
2 percent objective but remains somewhat elevated. The Committee decided to lower
the target range for the federal funds rate by 1/2 percentage point to 4-3/4 to
5 percent. The Committee has gained greater confidence that inflation is moving
sustainably toward 2 percent."""
    }
]

df = pd.DataFrame(fomc_data)
df["date"] = pd.to_datetime(df["date"])
print(f"Corpus: {len(df)} FOMC statements, {df['date'].dt.year.min()}–{df['date'].dt.year.max()}")

10A.2  Text Preprocessing

Raw text is messy. Before any analysis, you must normalize it into a consistent format. The goal is to reduce noise without destroying signal. Every preprocessing choice is a research decision that should be documented and justified.

The Intuition: Why Preprocess at All?

Consider the sentence “The Fed’s rates are RISING!”. To a human, the meaning is obvious. To a computer, it is a string of characters that could match or not match other strings in unpredictable ways. "Rising" ≠ "rising" ≠ "RISING" ≠ "rise" — yet they all convey the same concept. Preprocessing systematically resolves these surface-level differences so the algorithm can focus on meaning.

There are four core preprocessing steps, each motivated by a specific problem. Let’s walk through them on a real sentence from our FOMC corpus:

Each Step Explained

Tokenization splits continuous text into discrete units (tokens). This sounds trivial but is surprisingly subtle. Should “New York” be one token or two? What about “don’t” — “do” + “n’t”, or “don’t” as one unit? Tokenizers make these decisions using language-specific rules. Modern subword tokenizers (BPE, WordPiece) split rare words into meaningful pieces: “unemployment” → “un” + “employ” + “ment” (Manning, Raghavan & Schütze, 2008, Ch. 2).

Stopword removal discards high-frequency function words (“the,” “is,” “and”) that appear in virtually every document and therefore carry little discriminative information. Standard stopword lists contain 100–300 words. Caution: in some applications (e.g., authorship attribution), function word frequencies are themselves the signal (Mosteller & Wallace, 1964).

Option A: NLTK (transparent, step-by-step)

from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer

# Step 1: Tokenize — split text into individual words
text = df.loc[0, "text"]
tokens = word_tokenize(text.lower())
print(f"Raw tokens ({len(tokens)}): {tokens[:10]}")

# Step 2: Remove stopwords and punctuation
stop_words = set(stopwords.words("english"))
filtered = [t for t in tokens if t.isalpha() and t not in stop_words]
print(f"After filtering ({len(filtered)}): {filtered[:10]}")

# Step 3: Lemmatize — reduce words to base form
lemmatizer = WordNetLemmatizer()
lemmatized = [lemmatizer.lemmatize(t, pos="v") for t in filtered]
print(f"After lemmatization ({len(lemmatized)}): {lemmatized[:10]}")

Option B: spaCy (production-grade, faster)

import spacy

# Load the English pipeline (tokenizer + POS tagger + lemmatizer + NER)
nlp = spacy.load("en_core_web_sm")

def preprocess_spacy(text):
    """Preprocess a single document using spaCy.

    spaCy handles tokenization, POS tagging, and lemmatization
    in a single pass — faster and more linguistically accurate
    than doing each step separately.
    """
    doc = nlp(text)
    return [
        token.lemma_.lower()
        for token in doc
        if not token.is_stop       # remove stopwords
        and not token.is_punct     # remove punctuation
        and token.is_alpha         # keep only alphabetic tokens
        and len(token) > 1        # drop single characters
    ]

# Apply to entire corpus
df["tokens"] = df["text"].apply(preprocess_spacy)
df["clean_text"] = df["tokens"].apply(lambda t: " ".join(t))

# Inspect
for _, row in df.iterrows():
    print(f"{row['date'].strftime('%Y-%m')}: {row['tokens'][:8]}...")

10A.3  Text Representation: BoW & TF-IDF

Machine learning algorithms operate on numbers, not words. Text representation converts documents into numerical vectors. This is the foundational challenge of all computational text analysis: how do you turn language — something inherently symbolic and contextual — into a mathematical object you can compute with?

The Core Problem

Think of it this way: if you have a spreadsheet where each row is an observation and each column is a variable, you can immediately run a regression. But with text, your “data” is a column of paragraphs. You need to transform those paragraphs into rows of numbers — but which numbers? The answer to this question has driven 30 years of NLP research. We start with the simplest answer.

Bag of Words (BoW)

The simplest representation: count how many times each word appears in each document. Ignore word order entirely. Treat the document as a “bag” of words thrown together — as if you shook all the words out of their sentences and just counted them.

Here is a tiny worked example with three documents:

After removing stopwords (“the,” “is,” “and”), we count each remaining word per document:

Each document is now a vector of numbers: Doc 1 = [1, 1, 0, 0, 0], Doc 2 = [0, 0, 1, 1, 0]. You can now measure similarity between documents (using cosine similarity), cluster them, or feed them into a classifier. This is the document-term matrix — the foundational data structure of computational text analysis (Turney & Pantel, 2010).

from sklearn.feature_extraction.text import CountVectorizer

# Build a Bag of Words matrix from our preprocessed texts
bow_vectorizer = CountVectorizer(
    max_features=100,       # keep top 100 words
    min_df=1,               # word must appear in at least 1 doc
    max_df=0.95             # ignore words in >95% of docs
)
bow_matrix = bow_vectorizer.fit_transform(df["clean_text"])

print(f"Document-term matrix shape: {bow_matrix.shape}")
print(f"Vocabulary size: {len(bow_vectorizer.get_feature_names_out())}")
print(f"Sample words: {bow_vectorizer.get_feature_names_out()[:15].tolist()}")

TF-IDF: Weighting by Importance

Raw word counts treat every word equally. But a word like “committee” that appears in every FOMC statement is less informative than “coronavirus” which appears in only one. TF-IDF (Term Frequency – Inverse Document Frequency) solves this by asking a deceptively simple question: how distinctive is this word for this document?

The intuition has two parts:

• Term Frequency (TF): How often does this word appear in this document? More occurrences → higher score.
• Inverse Document Frequency (IDF): How rare is this word across all documents? Rarer words → higher score.

The product TF × IDF rewards words that are frequent locally but rare globally — exactly the words that characterize a document’s distinctive content.

Worked Example: Computing TF-IDF by Hand

Using our three mini-documents from above (N = 3 documents):

For Doc 3, the word “slowing” gets TF-IDF = 1 × 1.10 = 1.10, while “economy” gets only 1 × 0.41 = 0.41. TF-IDF automatically identifies “slowing” as the most informative word in Doc 3 — exactly what a human reader would conclude. This is why TF-IDF is often called a “statistical summary of what makes a document special” (Spärck Jones, 1972; Salton & Buckley, 1988).

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer = TfidfVectorizer(
    max_features=100,
    ngram_range=(1, 2),    # unigrams AND bigrams
    min_df=1,
    max_df=0.95
)
tfidf_matrix = tfidf_vectorizer.fit_transform(df["clean_text"])

# Show the most distinctive words for each statement
feature_names = tfidf_vectorizer.get_feature_names_out()

for i, row in df.iterrows():
    # Get TF-IDF scores for this document
    scores = tfidf_matrix[i].toarray().flatten()
    # Sort by score, take top 5
    top_idx = scores.argsort()[-5:][::-1]
    top_words = [(feature_names[j], round(scores[j], 3)) for j in top_idx]
    print(f"{row['date'].strftime('%Y-%m')}: {top_words}")

10A.4  Sentiment Analysis

Sentiment analysis assigns a polarity score (positive, negative, neutral) to text. This is arguably the most widely used text analysis technique in both business and research — from hedge funds scoring earnings calls in real time (Loughran & McDonald, 2016) to political scientists measuring the tone of legislative debate (Gentzkow, Shapiro & Taddy, 2019).

The Fundamental Idea

At its core, sentiment analysis rests on a simple premise: certain words carry emotional valence. “Excellent” is positive. “Terrible” is negative. “Quarterly” is neutral. If you count the positive and negative words in a document, you get a rough measure of its overall tone. This is the lexicon-based approach — and it is where the field began.

But language is not that simple. Consider these three sentences:

These examples illustrate the three generations of sentiment analysis: (1) generic lexicons that miss domain context, (2) domain-specific dictionaries that handle financial vocabulary, and (3) contextual models (transformers) that understand how words interact in sentences.

Approach 1: Lexicon-Based (VADER)

VADER (Valence Aware Dictionary and sEntiment Reasoner; Hutto & Gilbert, 2014) works by maintaining a dictionary of ~7,500 words, each hand-rated on a scale from −4 (most negative) to +4 (most positive) by human annotators. When you pass a sentence to VADER, it:

1. Looks up each word in the dictionary
2. Applies heuristic rules for intensifiers (“very good” > “good”), negations (“not good” flips sign), and capitalization (“GREAT” > “great”)
3. Normalizes the aggregate score to a −1 to +1 range (the “compound” score)

from nltk.sentiment import SentimentIntensityAnalyzer

sia = SentimentIntensityAnalyzer()

print("VADER Sentiment Scores (compound: -1 = most negative, +1 = most positive)\n")
for _, row in df.iterrows():
    scores = sia.polarity_scores(row["text"])
    print(f"{row['date'].strftime('%Y-%m')}  compound={scores['compound']:+.3f}  "
          f"pos={scores['pos']:.2f}  neg={scores['neg']:.2f}  neu={scores['neu']:.2f}")

Approach 2: Loughran-McDonald Dictionary (Finance-Specific)

The breakthrough paper by Loughran & McDonald (2011, Journal of Finance) showed that nearly three-quarters of “negative” words in the Harvard General Inquirer dictionary are not negative in financial contexts. Words like “tax,” “cost,” “capital,” “liability,” and “risk” are classified as negative by general-purpose dictionaries but are routine, neutral vocabulary in financial texts.

To fix this, they manually classified >4,000 words from 10-K filings into six categories: Negative, Positive, Uncertainty, Litigious, Constraining, and Superfluous. Their dictionary is now the de facto standard for sentiment analysis in finance and accounting research, cited in over 8,000 papers.

import pysentiment2 as ps

# Load the Loughran-McDonald financial dictionary
lm = ps.LM()

print("Loughran-McDonald Financial Sentiment\n")
for _, row in df.iterrows():
    tokens = lm.tokenize(row["text"])
    score = lm.get_score(tokens)
    print(f"{row['date'].strftime('%Y-%m')}  Polarity={score['Polarity']:.3f}  "
          f"Positive={score['Positive']}  Negative={score['Negative']}")

Approach 3: FinBERT (Transformer-Based, State of the Art)

FinBERT (Araci, 2019) is a BERT model further pre-trained on 46,000 sentences from financial news and then fine-tuned on 4,845 manually labeled sentences. Unlike dictionaries, it does not look up individual words. Instead, it reads the entire sentence at once through 12 layers of self-attention (see Module 10, Section 10.6), building a rich representation of how all the words relate to each other. Only then does it output a classification.

This contextual understanding is what makes transformer-based models fundamentally more capable than dictionary lookup, at the cost of being less interpretable and more computationally expensive (Huang, Wang & Yang, 2023).

from transformers import pipeline

# Load FinBERT — first run downloads the model (~420 MB)
finbert = pipeline(
    "sentiment-analysis",
    model="ProsusAI/finbert",
    tokenizer="ProsusAI/finbert"
)

print("FinBERT Financial Sentiment\n")
for _, row in df.iterrows():
    # FinBERT has a 512-token limit; truncate if needed
    result = finbert(row["text"][:512])
    print(f"{row['date'].strftime('%Y-%m')}  {result[0]['label']:<10s}  "
          f"confidence={result[0]['score']:.3f}")

10A.5  Topic Modeling

Topic modeling discovers the latent themes in a collection of documents without any prior labeling. It answers the question: “What is this corpus about?” This is unsupervised learning — the algorithm has no labels, no categories, no guidance. It must discover the themes on its own by finding patterns in which words tend to co-occur.

LDA (Latent Dirichlet Allocation)

LDA (Blei, Ng & Jordan, 2003) is one of the most influential models in machine learning. To understand it, think of a generative story — a fictional process by which documents could have been written. LDA doesn’t claim authors actually follow this process; it uses it as a mathematical model to reverse-engineer the topics.

The Key Hyperparameter: How Many Topics?

LDA requires you to specify K, the number of topics, before fitting the model. This is both its strength (you have control) and its weakness (you might choose poorly). Common approaches: (1) try several values of K and compare coherence scores, (2) use domain knowledge (“I expect ~5 major policy themes”), or (3) use hierarchical models that learn K automatically. In practice, researchers often run K = 5, 10, 15, 20 and inspect the topics qualitatively (Chang et al., 2009).

from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer

# LDA requires a raw count matrix (not TF-IDF)
count_vec = CountVectorizer(max_features=200, min_df=1, max_df=0.95)
dtm = count_vec.fit_transform(df["clean_text"])

# Fit LDA with 3 topics
lda = LatentDirichletAllocation(
    n_components=3,            # number of topics to discover
    random_state=42,           # reproducibility!
    max_iter=50,               # training iterations
    learning_method="online"   # faster for small corpora
)
lda.fit(dtm)

# Display the top words per topic
feature_names = count_vec.get_feature_names_out()
for topic_idx, topic in enumerate(lda.components_):
    top_words = [feature_names[i] for i in topic.argsort()[-8:][::-1]]
    print(f"Topic {topic_idx}: {', '.join(top_words)}")

# Show topic distribution for each document
print("\nDocument-topic distributions:")
doc_topics = lda.transform(dtm)
for i, row in df.iterrows():
    dist = [round(x, 2) for x in doc_topics[i]]
    print(f"  {row['date'].strftime('%Y-%m')}: {dist}")

BERTopic (Neural Topic Modeling)

BERTopic (Grootendorst, 2022) takes a fundamentally different approach from LDA. Instead of modeling the generative process of documents, it uses a four-stage pipeline that leverages modern neural embeddings:

The key innovation is c-TF-IDF: after clustering, BERTopic treats all documents in a cluster as one big “class document” and computes TF-IDF at the cluster level. This gives each cluster a ranked list of the words that are most distinctive to it — which become the topic labels. Because it uses neural embeddings for clustering, BERTopic captures semantic similarity (not just word co-occurrence), making it superior for short texts where word overlap is sparse.

from bertopic import BERTopic
from sentence_transformers import SentenceTransformer

# BERTopic pipeline: Embed → Reduce dimensions → Cluster → c-TF-IDF
# For a small corpus, we lower min_topic_size
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

topic_model = BERTopic(
    embedding_model=embedding_model,
    min_topic_size=2,          # allow small topics (for demo)
    nr_topics="auto",          # let the model decide how many
    verbose=False
)

topics, probs = topic_model.fit_transform(df["text"].tolist())

# Display discovered topics
topic_info = topic_model.get_topic_info()
print(topic_info[["Topic", "Count", "Name"]])

# For larger corpora, BERTopic also supports interactive visualizations:
# topic_model.visualize_topics()        # Inter-topic distance map
# topic_model.visualize_barchart()       # Top words per topic
# topic_model.visualize_over_time(...)   # Topic evolution

10A.6  Named Entity Recognition

Named Entity Recognition (NER) identifies and classifies proper nouns and specific references in text — people, organizations, dates, monetary values, locations. Think of NER as converting unstructured text into a structured database: from a paragraph of prose, you extract a table of who, what, where, and when (Nadeau & Sekine, 2007).

What NER Sees

Here is how a NER model annotates a sentence. Each colored span is a detected entity:

How NER Works: The BIO Tagging Scheme

Under the hood, NER models classify each individual token (word) using the BIO tagging scheme. Each token gets one of three labels: Beginning of an entity, Inside an entity (continuation), or Outside any entity (not part of one). This allows multi-word entities like “Federal Reserve” to be captured as a single unit:

Modern NER models (spaCy, HuggingFace) are typically fine-tuned BERT models that predict BIO tags for each token simultaneously. The model sees the full sentence context, so it can distinguish “Apple” (the company) from “apple” (the fruit) based on surrounding words (Li et al., 2020).

Entity Types: What Can NER Extract?

import spacy

# Use the small model for speed; for best accuracy use en_core_web_trf
nlp = spacy.load("en_core_web_sm")

# Analyze the 2024 FOMC statement
text = df.loc[4, "text"]
doc = nlp(text)

print("Named Entities Found:\n")
print(f"{'Entity':30s} {'Type':12s} Explanation")
print("-" * 65)
for ent in doc.ents:
    print(f"{ent.text:30s} {ent.label_:12s} {spacy.explain(ent.label_)}")

For more complex NER (extracting organization names, people, geopolitical entities from news or research texts), use the transformer-based spaCy model or a HuggingFace NER pipeline:

from transformers import pipeline

# HuggingFace NER pipeline (BERT-based)
ner_pipeline = pipeline(
    "ner",
    model="dslim/bert-base-NER",
    aggregation_strategy="simple"   # merge sub-word tokens
)

# A richer example text
rich_text = """Federal Reserve Chair Jerome Powell announced on Tuesday that
the central bank would maintain interest rates. Goldman Sachs analysts
expect GDP growth of 2.3% in the United States for 2025."""

results = ner_pipeline(rich_text)
for entity in results:
    print(f"{entity['word']:25s} {entity['entity_group']:8s} score={entity['score']:.3f}")

10A.7  Zero-Shot Classification

Traditional text classification requires labeled training data — hundreds or thousands of examples per category. Zero-shot classification lets you classify text into categories you define at inference time, with no training examples at all. This is a game-changer for research, where creating labeled datasets is expensive and time-consuming (Yin, Hay & Roth, 2019).

The Core Trick: Reframing Classification as Textual Entailment

The insight behind zero-shot classification is a clever reframing of the problem. Instead of training a model to classify text into categories, we use a model already trained on Natural Language Inference (NLI) — the task of deciding whether one sentence logically follows from another.

An NLI model takes two inputs: a premise (a statement of fact) and a hypothesis (a claim), and outputs one of three judgments: entailment (the hypothesis follows from the premise), contradiction (they conflict), or neutral (can’t tell).

The beauty of this approach is that the NLI model (typically BART or RoBERTa trained on the MultiNLI dataset of 433,000 sentence pairs; Williams, Nangia & Bowman, 2018) has never seen any of your specific labels during training. It generalizes because it has learned the deep semantic relationship between premises and hypotheses — essentially, it has learned what it means for one statement to “be about” something.

from transformers import pipeline

# Load zero-shot classifier
classifier = pipeline(
    "zero-shot-classification",
    model="facebook/bart-large-mnli"
)

# Define policy categories (no training data needed!)
policy_labels = [
    "monetary policy tightening",
    "monetary policy easing",
    "economic growth",
    "labor market",
    "inflation concerns",
    "financial stability"
]

print("Zero-Shot Policy Classification\n")
for _, row in df.iterrows():
    result = classifier(
        row["text"],
        candidate_labels=policy_labels,
        multi_label=True   # a statement can be about multiple topics
    )
    # Show top 3 labels
    top3 = [(l, f"{s:.2f}") for l, s in zip(result["labels"][:3], result["scores"][:3])]
    print(f"{row['date'].strftime('%Y-%m')}: {top3}")

10A.8  Semantic Similarity & Embeddings

Embeddings are arguably the most important conceptual breakthrough in modern NLP. The idea is deceptively simple: represent each word (or sentence) as a point in a high-dimensional space, such that words with similar meanings are near each other.

The Core Intuition: Meaning as Geometry

Consider a 2D space where the x-axis represents “economic sentiment” and the y-axis represents “policy action.” In this space:

In this space, “recession” and “crisis” are near each other (similar meaning), while “recession” and “growth” are far apart (opposite meanings). Crucially, the direction from “recession” to “growth” is similar to the direction from “decline” to “recovery” — they capture the same semantic relationship. This is the famous “vector arithmetic” property: king − man + woman ≈ queen (Mikolov et al., 2013).

Real embeddings live in 100–768 dimensions (not 2), but the geometric intuition holds. The distance between two points tells you how semantically similar the corresponding words are. This is measured using cosine similarity:

How Are Embeddings Learned? The Skip-Gram Intuition

Word2Vec’s Skip-gram model (Mikolov et al., 2013) learns embeddings by training a simple neural network on a single task: given a word, predict the words that tend to appear near it.

Static vs. Contextual Embeddings

For most tasks in 2025, sentence-transformers (Reimers & Gurevych, 2019) are the recommended approach. They encode entire sentences or paragraphs into a single dense vector, making them ideal for document similarity, semantic search, and clustering.

Static Embeddings (Word2Vec)

Although superseded by contextual models for most tasks, Word2Vec remains important: it is computationally cheap, its geometric properties are well-understood, and it is widely used in published economics research (e.g., measuring changes in the meaning of “inflation” across decades of Fed communications; Ash & Hansen, 2023).

from gensim.models import Word2Vec

# Train Word2Vec on our (small) corpus
# In practice, you'd use a much larger corpus or pre-trained vectors
sentences = df["tokens"].tolist()

model = Word2Vec(
    sentences,
    vector_size=50,      # embedding dimensions
    window=5,             # context window size
    min_count=1,          # include all words (small corpus)
    workers=4,            # parallel training threads
    sg=1,                 # 1 = Skip-gram (better for small data)
    seed=42,              # reproducibility
    epochs=100            # more passes for small corpus
)

# Find words similar to "inflation"
try:
    similar = model.wv.most_similar("inflation", topn=5)
    print("Words most similar to 'inflation':")
    for word, score in similar:
        print(f"  {word:20s} cosine_similarity={score:.3f}")
except KeyError:
    print("'inflation' not in vocabulary (corpus too small)")

Sentence Embeddings (Sentence-Transformers)

Sentence-transformers encode entire sentences or paragraphs into dense vectors. Unlike Word2Vec, the same word gets a different vector depending on its context. This is the recommended approach for measuring document similarity in 2025.

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Load a fast, high-quality embedding model
model = SentenceTransformer("all-MiniLM-L6-v2")

# Encode all FOMC statements into 384-dimensional vectors
embeddings = model.encode(df["text"].tolist())
print(f"Embedding shape: {embeddings.shape}")  # (5, 384)

# Compute pairwise similarity
sim_matrix = cosine_similarity(embeddings)

# Display as a labeled matrix
labels = [d.strftime("%Y-%m") for d in df["date"]]
print(f"\n{'':<10s}" + "  ".join(f"{l:>8s}" for l in labels))
for i, label in enumerate(labels):
    row_str = "  ".join(f"{sim_matrix[i][j]:8.3f}" for j in range(len(labels)))
    print(f"{label:<10s}{row_str}")

10A.9  LLMs for Text Analysis at Scale

Large Language Models (Claude, GPT-4) represent a paradigm shift in text analysis. Unlike all previous methods, LLMs can perform complex, multi-dimensional analysis in a single pass — extracting sentiment, topics, entities, causal claims, and nuanced reasoning simultaneously. They do this through natural language instructions (prompts) rather than specialized training.

When to Use LLMs vs. Traditional Methods

Prompt Engineering for Text Analysis

The quality of LLM-based text analysis depends almost entirely on the quality of the prompt. Three principles matter most (Ziems et al., 2024):

import anthropic
import json

# Initialize the Claude client
client = anthropic.Anthropic(api_key="your-api-key")  # use env variable in practice

def analyze_fomc_statement(text):
    """Analyze an FOMC statement using Claude with structured output."""

    prompt = f"""Analyze this FOMC statement. Return ONLY valid JSON with these fields:

{{
  "overall_sentiment": "hawkish" | "dovish" | "neutral",
  "confidence": 0.0 to 1.0,
  "rate_decision": "raise" | "lower" | "hold",
  "key_concerns": ["concern1", "concern2", ...],
  "forward_guidance": "brief summary of what the Fed signals about future policy",
  "economic_conditions": {{
    "growth": "expanding" | "contracting" | "mixed",
    "labor_market": "strong" | "weak" | "mixed",
    "inflation": "above_target" | "at_target" | "below_target"
  }}
}}

FOMC Statement:
{text}"""

    response = client.messages.create(
        model="claude-sonnet-4-20250514",   # pin exact version
        max_tokens=1024,
        temperature=0,                     # deterministic output
        messages=[{"role": "user", "content": prompt}]
    )
    return json.loads(response.content[0].text)

# Analyze each statement
for _, row in df.iterrows():
    result = analyze_fomc_statement(row["text"])
    print(f"\n{row['date'].strftime('%Y-%m')}:")
    print(f"  Sentiment: {result['overall_sentiment']} (conf={result['confidence']})")
    print(f"  Rate decision: {result['rate_decision']}")
    print(f"  Key concerns: {result['key_concerns']}")
    print(f"  Forward guidance: {result['forward_guidance']}")

Batch Processing with Rate Limiting

When analyzing thousands of documents, you need to handle rate limits gracefully:

import time
import json
from pathlib import Path

def batch_analyze(texts, labels, output_path, delay=0.5):
    """Process a corpus with checkpointing and rate limiting.

    Saves results incrementally so you don't lose progress if
    the script is interrupted.
    """
    results = []
    output_file = Path(output_path)

    # Resume from checkpoint if exists
    if output_file.exists():
        with open(output_file) as f:
            results = json.load(f)
        print(f"Resuming from checkpoint: {len(results)} already done")

    for i in range(len(results), len(texts)):
        try:
            result = analyze_fomc_statement(texts[i])
            result["label"] = labels[i]
            results.append(result)
        except Exception as e:
            results.append({"label": labels[i], "error": str(e)})

        # Save checkpoint every 10 documents
        if (i + 1) % 10 == 0:
            with open(output_file, "w") as f:
                json.dump(results, f, indent=2)
            print(f"  Checkpoint: {i+1}/{len(texts)} processed")

        time.sleep(delay)  # respect API rate limits

    # Final save
    with open(output_file, "w") as f:
        json.dump(results, f, indent=2)

    return results

10A.10  Putting It All Together

Let’s combine every technique into a single, end-to-end analysis pipeline. This is the kind of script you would use for a research paper or business report.

"""
Complete Text Analysis Pipeline: FOMC Statements
=================================================
This script demonstrates a full, reproducible text analysis workflow.
It can serve as a template for any corpus-level text analysis project.
"""

# === 0. SETUP ===
import random, numpy as np, pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from nltk.sentiment import SentimentIntensityAnalyzer
from sentence_transformers import SentenceTransformer
import spacy

random.seed(42); np.random.seed(42)

# === 1. LOAD DATA ===
# (In practice, load from CSV/database)
# df = pd.read_csv("fomc_statements.csv")

# === 2. PREPROCESS ===
nlp = spacy.load("en_core_web_sm")

def preprocess(text):
    doc = nlp(text)
    return " ".join([
        t.lemma_.lower() for t in doc
        if not t.is_stop and t.is_alpha and len(t) > 1
    ])

df["clean"] = df["text"].apply(preprocess)

# === 3. TF-IDF: What makes each statement distinctive? ===
tfidf = TfidfVectorizer(ngram_range=(1, 2), max_features=100)
tfidf_matrix = tfidf.fit_transform(df["clean"])
features = tfidf.get_feature_names_out()

# === 4. SENTIMENT: Multiple methods for robustness ===
sia = SentimentIntensityAnalyzer()
df["vader_compound"] = df["text"].apply(
    lambda t: sia.polarity_scores(t)["compound"]
)

# === 5. EMBEDDINGS: Semantic similarity over time ===
embed_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = embed_model.encode(df["text"].tolist())
sim_matrix = cosine_similarity(embeddings)

# === 6. VISUALIZE ===
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Panel A: Sentiment over time
axes[0].plot(df["date"], df["vader_compound"], "o-", color="#2563eb", linewidth=2)
axes[0].axhline(y=0, color="gray", linestyle="--", alpha=0.5)
axes[0].set_title("FOMC Sentiment Over Time (VADER)", fontsize=13)
axes[0].set_ylabel("Compound Score")
axes[0].set_ylim(-1, 1)

# Panel B: Semantic similarity heatmap
labels = [d.strftime("%Y-%m") for d in df["date"]]
sns.heatmap(
    sim_matrix, annot=True, fmt=".2f", cmap="YlOrRd",
    xticklabels=labels, yticklabels=labels, ax=axes[1]
)
axes[1].set_title("Semantic Similarity Between Statements", fontsize=13)

plt.tight_layout()
plt.savefig("fomc_analysis.png", dpi=150, bbox_inches="tight")
plt.show()

print("Analysis complete. Results saved to fomc_analysis.png")

10A.11  Reproducibility Checklist

Reproducibility is not optional — it is the difference between a finding and an anecdote. Follow this checklist for every text analysis project, whether for a course assignment, a business report, or a journal submission.

• Pin all library versions in a requirements.txt with exact version numbers
• Set random seeds at the top of every script (random, numpy, torch)
• Document every preprocessing decision: what you removed, what you kept, and why
• Store raw data separately from processed data — never overwrite raw data
• Version your prompts if using LLM APIs — save the exact prompt template used
• Pin model versions: use ProsusAI/finbert not just “finbert”; use claude-sonnet-4-20250514 not just “claude”
• Report results across multiple seeds: run with seeds 42, 123, 456, 789, 1011 and report mean ± std
• Validate against human labels: randomly sample 100–200 documents and have a human code them
• Report all metrics: precision, recall, F1 per class — not just accuracy
• Use version control: commit your analysis scripts to Git (see Module 9)
• Include a README with instructions to reproduce your results from scratch

References & Further Reading

Foundational Methods

• Harris, Z. S. (1954). “Distributional Structure.” Word, 10(2–3), 146–162.
• Spärck Jones, K. (1972). “A Statistical Interpretation of Term Specificity and Its Application in Retrieval.” Journal of Documentation, 28(1), 11–21.
• Salton, G. & Buckley, C. (1988). “Term-Weighting Approaches in Automatic Text Retrieval.” Information Processing & Management, 24(5), 513–523.
• Blei, D., Ng, A. & Jordan, M. (2003). “Latent Dirichlet Allocation.” Journal of Machine Learning Research, 3, 993–1022.
• Griffiths, T. L. & Steyvers, M. (2004). “Finding Scientific Topics.” Proceedings of the National Academy of Sciences, 101(S1), 5228–5235.
• Turney, P. D. & Pantel, P. (2010). “From Frequency to Meaning: Vector Space Models of Semantics.” Journal of Artificial Intelligence Research, 37, 141–188.
• Mikolov, T. et al. (2013). “Efficient Estimation of Word Representations in Vector Space.” arXiv:1301.3781.
• Hutto, C. & Gilbert, E. (2014). “VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text.” AAAI ICWSM.
• Devlin, J. et al. (2019). “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” NAACL-HLT.

Text as Data in Economics & Finance

• Loughran, T. & McDonald, B. (2011). “When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks.” Journal of Finance, 66(1), 35–65.
• Baker, S., Bloom, N. & Davis, S. (2016). “Measuring Economic Policy Uncertainty.” Quarterly Journal of Economics, 131(4), 1593–1636.
• Hansen, S. & McMahon, M. (2016). “Shocking Language: Understanding the Macroeconomic Effects of Central Bank Communication.” Journal of International Economics, 99, S114–S133.
• Gentzkow, M., Kelly, B. & Taddy, M. (2019). “Text as Data.” Journal of Economic Literature, 57(3), 535–574.
• Gentzkow, M. & Shapiro, J. M. (2010). “What Drives Media Slant? Evidence from U.S. Daily Newspapers.” Econometrica, 78(1), 35–71.
• Ash, E. & Hansen, S. (2023). “Text Algorithms in Economics.” Annual Review of Economics, 15, 659–688.

Modern NLP Tools & Models

• Araci, D. (2019). “FinBERT: Financial Sentiment Analysis with Pre-Trained Language Models.” arXiv:1908.10063.
• Reimers, N. & Gurevych, I. (2019). “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” EMNLP.
• Yin, W., Hay, J. & Roth, D. (2019). “Benchmarking Zero-Shot Text Classification.” EMNLP-IJCNLP.
• Williams, A., Nangia, N. & Bowman, S. (2018). “A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference.” NAACL-HLT.
• Grootendorst, M. (2022). “BERTopic: Neural Topic Modeling with a Class-Based TF-IDF Procedure.” arXiv:2203.05794.
• Li, J. et al. (2020). “A Survey on Deep Learning for Named Entity Recognition.” IEEE TKDE, 34(1), 50–70.
• Nadeau, D. & Sekine, S. (2007). “A Survey of Named Entity Recognition and Classification.” Lingvisticae Investigationes, 30(1), 3–26.

LLMs for Text Analysis & Research Methods

• Gilardi, F., Alizadeh, M. & Kubli, M. (2023). “ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks.” Proceedings of the National Academy of Sciences, 120(30).
• Ziems, C. et al. (2024). “Can Large Language Models Transform Computational Social Science?” Computational Linguistics, 50(1), 237–291.
• Huang, A. H., Wang, H. & Yang, Y. (2023). “FinBERT: A Large Language Model for Extracting Information from Financial Text.” Contemporary Accounting Research, 40(2), 806–841.

Reproducibility & Best Practices

• ACL Rolling Review. “Responsible NLP Research Checklist.”
• Chang, J. et al. (2009). “Reading Tea Leaves: How Humans Interpret Topic Models.” NeurIPS.
• Blei, D. M. (2012). “Probabilistic Topic Models.” Communications of the ACM, 55(4), 77–84.
• Manning, C. D., Raghavan, P. & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.
• Jurafsky, D. & Martin, J. H. (2024). Speech and Language Processing, 3rd edition (draft).