Portfolio · Written for this search · llm
LLM
Computation desk
Large language models transform text sequences into probability distributions over vocabulary. They estimate which token should follow given context, using neural networks trained on internet-scale corpora.
- Input tokens
- Embedding layer
- Transformer blocks
- Output distribution
- Next token
Architecture
The transformer stack processes tokenised text through repeated blocks of multi-head attention and feedforward layers. Each attention head learns to weight input positions differently, computing query-key products scaled by dimension then passed through softmax. Residual connections and layer normalisation stabilise training across dozens of blocks.
Position encodings inject sequence order because attention itself is permutation-invariant. Absolute encodings add sinusoidal patterns to embeddings; relative methods modify attention scores by learned offsets. Models now exceed a hundred billion parameters, with embedding and unembedding matrices accounting for substantial memory when vocabulary reaches fifty thousand tokens.
Training
Pre-training uses next-token prediction on web text, code repositories, and books. The cross-entropy loss drives gradient descent across parameter space, with batch sizes in the millions of tokens distributed over accelerator clusters. Learning rate schedules warm up then decay; gradient clipping prevents exploding updates in early phases.
Fine-tuning narrows behaviour through instruction datasets and human preference feedback. Supervised examples teach format adherence; reinforcement learning from comparisons steers tone and refusal patterns. The result is a base model conditioned toward assistant dialogue, though the underlying mechanism remains autoregressive prediction over the same vocabulary.
Inference
Generation samples from the output distribution repeatedly, appending each token to context until a stop sequence appears. Temperature rescales logits before softmax, flattening or sharpening probabilities. Top-k and nucleus sampling truncate the tail, preventing rare tokens from appearing through unlucky draws.
Key-value caching stores attention states for existing tokens, avoiding recomputation as the sequence grows. Memory bandwidth becomes the bottleneck when batch size is small; latency scales with model depth and context length. Quantisation to eight or four bits reduces transfer cost, accepting small accuracy loss for doubled throughput on consumer hardware.
Finding from this desk
The architecture relies on self-attention mechanisms that compute relationships between all positions in a sequence. This permits parallel training and captures long-range dependencies that earlier recurrent designs missed.