Study Guide · Natural Language Processing · 7 min read

How Modern NLP Works Under the Hood

Modern language AI follows one recipe: chop text into tokens, turn tokens into vectors, let attention mix meaning across the passage, then predict what comes next — trained on a library's worth of text.

Tokens and embeddings

Models don't see words; they see tokens — word fragments mapped to numbers ('understand' might split into 'under' + 'stand'). Tokenization is why models sometimes fumble unusual spellings or rare scripts.

Each token maps to an embedding vector, thousands of numbers encoding meaning. In this space, 'doctor' and 'physician' are neighbors; 'doctor' and 'sandwich' are distant strangers.

Attention: context is everything

Attention lets each token look at every other token and ask 'how does this affect my meaning?' In 'the bank of the river', attention pulls 'bank' toward water and away from money.

Stack dozens of these attention layers and the model builds progressively richer understanding — grammar early, facts and reasoning deeper in. This is the transformer, introduced by the 2017 paper 'Attention Is All You Need'.

Pretraining, prompting, fine-tuning

Pretraining teaches general language by predicting missing words across trillions of tokens. The result is a foundation model with broad but unfocused competence.

Teams adapt it two ways: prompting (instructions and examples at run time — instant, cheap) or fine-tuning (additional training on your data — durable style and format knowledge). Most projects should exhaust prompting before considering fine-tuning.

Key Points

  • Tokenization converts text to numbers; embeddings give those numbers meaning.
  • Attention dynamically re-weights context so meaning fits the sentence.
  • Pretraining builds broad skill; prompting adapts behavior instantly; fine-tuning bakes in knowledge.
  • Prompt engineering is real engineering — try it before any training project.

All study guides for this term: NLP, Explained Simply · How Modern NLP Works Under the Hood · NLP in the Real World