Transformer on XEDCZQ Blog

Transformer 20-Step Visual Study Notes

Fri, 05 Jun 2026 23:10:00 +0800

Transformer 20-Step Visual Study Notes

This note is based on the interactive explanation from Transformer Explainer. It follows the site’s 20 steps and organizes them into a study note for understanding how GPT-style Transformers perform next-token prediction. The site uses GPT-2 small as the example model and visualizes the full pipeline from input tokens to output probabilities.

First, remember one sentence: the core task of GPT-style Transformers is next-token prediction. Given the prompt:

Data visualization empowers users to

The model needs to answer:

What is the most likely next token?

To answer this, a Transformer goes through tokenization, embedding, positional encoding, repeated Transformer blocks, self-attention, MLP, logits, probability distribution, sampling strategy, and other steps.

All screenshots in this note are taken from Transformer Explainer, developed by the Georgia Tech Polo Club team. They are used here for personal study notes. It is best to read this together with the original interactive site.

What Is Transformer

Transformer is the most common foundation architecture for modern large language models. Text-generation models such as GPT, Llama, and Gemini can all be understood as expanded versions of the Transformer architecture.

Its most important ability is not “memorizing answers”, but learning language patterns from large amounts of text and then predicting the next token from context at inference time. This prediction is repeated: predict one token, append it to the text, then predict the next one.

Source: Transformer Explainer, https://poloclub.github.io/transformer-explainer/

How Transformers Work

When a Transformer generates text, it does not write an entire paragraph at once. It generates step by step. Each step performs the same task: predict a probability distribution for the next token based on the existing context.

For example, if the current input is:

Data visualization empowers users to

The model may assign the highest probability to visualize, while also assigning relatively high probabilities to tokens such as create, see, and make. The final selected token is also affected by sampling parameters such as temperature, top-k, and top-p.

Source: Transformer Explainer

Transformer Architecture

A text-generation Transformer can be divided into three major parts:

Embedding: converts human-readable text into vectors the model can process.
Transformer Blocks: repeatedly refine each token representation, mainly through Self-Attention and MLP.
Output Probabilities: converts the final vector into probabilities over all tokens in the vocabulary.

At a high level, the information flow is:

text input -> token -> embedding -> Transformer Blocks -> logits -> probabilities -> sample next token

Source: Transformer Explainer

Embedding

The model cannot directly understand strings. Embedding converts each token into a sequence of numbers, or a vector. This vector is not manually designed; it is learned during training.

If two tokens often appear in similar contexts, their embeddings tend to be closer in high-dimensional space. You can think of embedding as an internal “semantic coordinate” used by the model.

GPT-2 small has a hidden dimension of 768, so each token is represented as a 768-dimensional vector.

Source: Transformer Explainer

Token Embedding

Tokenization splits input text into tokens. A token can be a full word or a subword. For example, empowers in the example is split into em and powers.

Each token has a unique ID. GPT-2 has a vocabulary size of 50,257, so its token embedding matrix is roughly:

$$ 50257 \times 768 $$

After the model gets the token ID, it looks up the corresponding 768-dimensional vector from this large matrix.

Source: Transformer Explainer

Positional Encoding

Self-Attention does not naturally know sequence order. If the model only receives a set of token vectors, it does not know which token comes first and which comes later.

So positional encoding is needed. GPT-2 uses learnable positional embeddings and adds the semantic token vector and the position vector:

$$ x_i = \text{TokenEmbedding}_i + \text{PositionEmbedding}_i $$

This lets the model know both “what this token is” and “where it is.”

Source: Transformer Explainer

Repetitive Transformer Blocks

Embedding is only the input representation, not a fully contextualized semantic representation. The real context modeling happens inside Transformer Blocks.

GPT-2 small has 12 Transformer Blocks. Each block roughly contains:

Multi-Head Self-Attention: lets tokens exchange information.
MLP: nonlinearly processes each token representation.
Residual, LayerNorm, and Dropout: stabilize training and improve generalization.

The meaning of stacking multiple layers is that lower layers tend to capture local and lexical information, while higher layers more easily form complex semantic and task-related representations.

Source: Transformer Explainer

Multi-Head Self Attention

The goal of Self-Attention is to let each token update itself based on context. For example, the token to has different meanings in different sentences. It needs to “look at” the previous context Data visualization empowers users to form a more accurate representation.

Multi-Head means the model does not use just one attention perspective. It uses multiple heads in parallel. GPT-2 small has 12 attention heads. Different heads can learn different relationships, such as syntactic relations, short-distance collocations, and long-distance semantic dependencies.

Source: Transformer Explainer

Query, Key, Value

Self-Attention maps each token’s input vector into three vectors:

Query (Q): what information the current token wants to search for.
Key (K): features by which each token can be matched by others.
Value (V): the actual information content to be aggregated and passed along.

They come from linear transformations:

$$ Q = XW_Q,\quad K = XW_K,\quad V = XW_V $$

A simple analogy is a search engine:

Query is the search query.
Key is the page title or index.
Value is the page content.

The model first computes relevance between Query and Key, then reads Value with weighted aggregation.

Source: Transformer Explainer

Multi-Head

GPT-2 small has an embedding dimension of 768 and 12 attention heads, so each head usually processes:

$$ 768 / 12 = 64 $$

The benefit of multi-head attention is learning multiple relationships in parallel. One head may focus on adjacent words, another on subject-verb relationships, and another on more distant semantic hints.

Multiple heads are not duplicate work; they give the model multiple “viewing angles.”

Source: Transformer Explainer

Masked Self Attention

GPT-style models generate text from left to right. When predicting the current position, they must not peek at future tokens, so causal mask, also called masked self-attention, is used.

The core formula is:

$$ \text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right)V $$

Where:

$QK^T$: computes pairwise similarity between tokens.
$\sqrt{d_k}$: scaling factor that prevents dot products from becoming too large and making softmax too sharp.
$M$: mask matrix that sets future positions to $-\infty$.
softmax: turns scores into probabilities.
multiplying by $V$: aggregates information according to attention weights.

Source: Transformer Explainer

Attention Output and Concatenation

Each head outputs a context-enhanced token representation. Since GPT-2 small has 12 heads, it produces 12 sets of results.

The model then concatenates the outputs from these heads and applies a linear projection back to the original hidden dimension 768:

head_1, head_2, ..., head_12 -> concat -> linear projection

The purpose is to let different heads extract information separately, then fuse those perspectives into one unified representation.

Source: Transformer Explainer

MLP

Attention handles information flow between tokens. MLP processes each token’s own representation nonlinearly.

GPT-2’s MLP usually contains two linear transformations with GELU activation in between:

$$ \text{MLP}(x)=W_2\cdot \text{GELU}(W_1x+b_1)+b_2 $$

The first layer expands the dimension from 768 to 3072, and the second layer compresses it back to 768. Expanding the dimension lets the model represent more complex features in a higher-dimensional space.

Note that MLP does not communicate across tokens like Attention. It processes each token independently.

Source: Transformer Explainer

Output Logit

After all Transformer Blocks, the model takes the output vector at the last position and uses it to predict the next token.

This vector goes through the final linear layer and is mapped to the vocabulary size:

$$ \text{logits}=h_{\text{last}}W_{\text{vocab}}+b $$

GPT-2 has a vocabulary size of 50,257, so logits are a vector of length 50,257. Each number corresponds to the raw score of a candidate token.

A logit is not a probability. It can be any real number and must go through softmax to become a probability distribution.

Source: Transformer Explainer

Probabilities

Softmax converts logits into probabilities:

$$ p_i=\frac{e^{z_i}}{\sum_j e^{z_j}} $$

After conversion:

Every token probability is between 0 and 1.
The probabilities of all tokens sum to 1.

In the figure, after the example input, the model considers tokens such as visualize, create, see, and make to be likely next tokens.

Source: Transformer Explainer

Temperature

Temperature scales logits before softmax:

$$ p_i=\frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)} $$

Where $T$ is the temperature:

$T < 1$: the probability distribution becomes sharper. High-score tokens are more likely to be selected, and output is more stable.
$T = 1$: logits are not additionally adjusted.
$T > 1$: the probability distribution becomes flatter. Low-probability tokens have more chances to be selected, and output becomes more diverse.

In simple terms, lower temperature is more conservative, while higher temperature is more divergent.

Source: Transformer Explainer

Sampling Strategy

After obtaining a probability distribution, the model still needs to decide how to choose the next token. Common strategies include:

Greedy Search: always choose the highest-probability token. Stable, but can be rigid.
Top-k: keep only the k highest-probability tokens, then sample from them.
Top-p: keep the smallest token set whose cumulative probability reaches p. Also called nucleus sampling.

Top-k is more like a fixed candidate pool, while top-p is a dynamic candidate pool. In practice, temperature and top-k/top-p are often tuned together.

Source: Transformer Explainer

Residual Connection

A residual connection adds a layer’s input directly to its output:

$$ y = x + F(x) $$

Its purpose is to preserve original information and make gradients pass through deep networks more easily. Without residual connections, training becomes harder as the model gets deeper, and information from early layers is more likely to be lost.

In Transformers, residual connections usually surround both Attention and MLP.

Source: Transformer Explainer

Layer Normalization

Layer Normalization normalizes the values inside a token vector, making the mean and variance more stable:

$$ \text{LayerNorm}(x)=\gamma\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta $$

It reduces training instability and makes the input distribution of each layer more controllable. GPT-2 uses a pre-norm style: LayerNorm is applied before entering Attention and MLP.

Intuitively, LayerNorm is like adjusting the numerical scale to a more suitable range before each key computation.

Source: Transformer Explainer

Dropout

Dropout is a regularization method used during training. It randomly sets part of the connections or activations to zero, preventing the model from over-relying on certain local features.

The intuition is: during training, do not let the model follow the exact same path every time, forcing it to learn more robust representations.

Important notes:

Dropout is mainly used during training.
Dropout is disabled during inference.
Many newer large models use less Dropout than early models because their training data is extremely large.

Source: Transformer Explainer

One Flowchart Summary

The inference flow of a GPT-style Transformer can be compressed into this chain:

flowchart LR
 A["Input text"] --> B["Tokenization"]
 B --> C["Token Embedding"]
 C --> D["Positional Encoding"]
 D --> E["Transformer Block x N"]
 E --> F["Multi-Head Self-Attention"]
 F --> G["MLP"]
 G --> H["Final Linear"]
 H --> I["Logits"]
 I --> J["Softmax Probabilities"]
 J --> K["Temperature / Top-k / Top-p"]
 K --> L["Sample next token"]

Key Differences from RNN

Combining this with the previous RNN note, the difference can be understood like this:

Aspect	RNN	Transformer
Information transfer	Passed step by step through hidden state	Self-Attention lets tokens directly read one another
Parallelism	Strong time-step dependency, hard to parallelize	Tokens in the same layer can be computed in parallel
Long-distance dependency	Long path, easy to decay	Any positions can directly connect
Context representation	Compressed into hidden state	Keeps explicit token representations for the whole context
Large-model training	Less efficient to scale	Better suited to large-scale GPU/TPU matrix computation

This is why modern LLMs mainly use Transformers: they are not only strong in modeling, but also better engineered for large-scale pretraining.

Sources

RNN Recurrent Neural Network Study Notes

Thu, 04 Jun 2026 00:00:00 +0800

RNN Recurrent Neural Network Study Notes

This note summarizes what I learned from Andrej Karpathy’s classic blog post The Unreasonable Effectiveness of Recurrent Neural Networks. The original article was published in 2015 and uses character-level language models to show how RNNs/LSTMs can learn spelling, formatting, structure, local syntax, and even interpretable “state memory” from raw text.

The main thread of this note is:

Why RNNs are suitable for sequences.
The core formulas and computation process of RNNs.
How character-level language models are trained and sampled.
What the classic experiments in Karpathy’s article demonstrate.
Why RNNs were later replaced by Transformers in mainstream NLP.

What Problem RNNs Solve

Ordinary feedforward neural networks usually assume that inputs and outputs are fixed-length vectors, such as taking an image as input and outputting a class label. But many real tasks are naturally sequential:

Text: a sentence consists of multiple tokens or characters.
Speech: audio frames are arranged over time.
Video: frames are arranged over time.
Translation: one sentence is converted into another sentence in another language.
Generation: previously generated content affects what comes next.

The key idea of an RNN is: when processing the current input, the model does not only look at the current input; it also maintains a hidden state that compresses past context.

Source: Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks. Red represents input, blue represents output, and green represents recurrent state. The diagram shows common modes such as fixed input/output, sequence output, sequence input, sequence-to-sequence, and synchronized sequence input/output.

You can think of an RNN as a repeatedly called step function:

rnn = RNN()
y = rnn.step(x)

Each time step(x) is called, the RNN reads the current input x_t, combines it with the previous hidden state h_{t-1}, updates a new hidden state h_t, and produces the current output y_t.

Core Formula of a Vanilla RNN

The most basic RNN update is:

$$ h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t + b_h) $$$$ y_t = W_{hy}h_t + b_y $$

Where:

$x_t$: input at time step $t$
$h_{t-1}$: hidden state from the previous time step
$h_t$: hidden state at the current time step
$W_{xh}$: input-to-hidden weights
$W_{hh}$: hidden-to-hidden recurrent weights
$W_{hy}$: hidden-to-output weights
$\tanh$: nonlinear activation that squashes values into $[-1, 1]$

In code, it roughly looks like this:

class RNN:
 def step(self, x):
 self.h = np.tanh(np.dot(self.W_hh, self.h) + np.dot(self.W_xh, x))
 y = np.dot(self.W_hy, self.h)
 return y

The most important thing to understand is the meaning of h: it is not a hand-written rule, but a “context summary” learned by the model during training. If the input is text, h may carry information such as whether the model is inside quotes, inside a URL, whether a bracket is open, or what words have appeared before.

Why RNNs Can Model Context

Take the string hello as an example. Suppose the vocabulary only contains the four characters h, e, l, o. During training, the input and target can be:

Input: h e l l
Target: e l l o

Notice that the target after the first l is l, while the target after the second l is o. If the model only looks at the current character, both time steps have the same input, so it cannot tell what the next character should be. The RNN must use its hidden state to record “what has already been seen.”

Source: Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks. The model reads hell one character at a time, outputs scores for the next character at each step, and the green targets show the correct characters whose scores should be increased.

The training objective is usually cross-entropy loss at every time step:

$$ \mathcal{L} = -\sum_t \log p(x_{t+1}\mid x_{\le t}) $$

Where:

$x_{\le t}$ means the context up to and including the current position.
$p(x_{t+1}\mid x_{\le t})$ means the probability of the next character predicted from historical context.

After training, text generation works as follows:

Give the model a starting character or prompt.
Get the probability distribution for the next character.
Sample one character from the distribution.
Feed the sampled character back into the model.
Repeat the process.

This is the simplest generation loop for a character-level language model.

BPTT: How RNNs Are Trained

An RNN reuses the same set of parameters at every time step. During training, the recurrent structure is unrolled over time and then backpropagation is applied. This is called Backpropagation Through Time, BPTT.

If the sequence is very long, fully unrolling it is expensive. A common approach is Truncated BPTT, where gradients are only propagated back for a fixed number of steps. For example, Karpathy’s Paul Graham experiment used truncated BPTT with a length of 100 characters.

The main difficulty in RNN training comes from long chains of gradients:

$$ \frac{\partial \mathcal{L}}{\partial h_{t-k}} $$

This gradient must pass through many matrix multiplications and nonlinear functions. When the chain is long, gradients may:

Become smaller and smaller: vanishing gradients, making long-term dependencies hard to learn.
Become larger and larger: exploding gradients, making training unstable.

This is why LSTM and GRU became widely used.

LSTM: A Stronger Recurrent Unit

The experiments in Karpathy’s article actually use LSTM. LSTM is still part of the RNN family, but it uses a more complex hidden-state update with gates, making it easier for the model to keep or forget information.

Typical LSTM components include:

Forget gate: decides how much old information to keep.
Input gate: decides how much new information to write.
Output gate: decides how much current state to expose to the output.
Cell state: provides a more stable information channel.

Intuitively, a Vanilla RNN mixes the old state and the new input at every step and compresses them again, so long-term information is easily overwritten. LSTM gives the model mechanisms for “writing, keeping, and reading”, making it more suitable for long sequences.

Classic Experiments in Karpathy’s Article

Learning Structure from Characters

The original article shows RNN/LSTM results trained on different kinds of text, including:

Paul Graham essays
Shakespeare plays
Wikipedia Markdown/XML
Algebraic geometry LaTeX
Linux source code
Baby name lists

The common point is that the model is not given an explicit dictionary, grammar rules, Markdown rules, XML tree rules, or C language rules. It is only trained to “predict the next character.” Yet after training, it can generate text that looks like the distribution of the original data.

This shows that: next-character prediction looks simple, but it forces the model to learn multi-level structure.

Character level: spelling, spaces, punctuation
Word level: common words, names, variable names
Syntax level: quotes, brackets, indentation, tag closure
Style level: Shakespeare-like dialogue, Wikipedia-like entries, source-code comments

Capability Evolution During Training

Karpathy uses War and Peace as an example to show how sampled text changes over training iterations:

Early stage: almost random characters, but spaces begin to appear.
Middle stage: short words, periods, quotes, and other local structures appear.
Later stage: more English-like words, names, and sentence forms appear.

My understanding is that an RNN does not learn “language” all at once. It first learns the most local and frequent patterns, then gradually forms longer-range dependencies.

Hidden Units Learn Interpretable States

One of the most classic parts of the article is the visualization of LSTM hidden unit activations. Some neurons activate inside URLs, some activate inside Markdown link contexts such as [[...]], and others seem to track quoted regions.

Source: Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks. This figure shows a hidden unit that activates strongly in URL regions, suggesting the model may have learned an internal state for “currently inside a URL.”

Source: Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks. This figure shows a hidden unit responding to the [[...]] Markdown environment.

Source: Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks. These visualizations show that some hidden units learn state-detection functions that humans can interpret.

The point is not that every neuron has a clear semantic meaning. Rather, end-to-end training can lead the model to discover intermediate states that are useful for the task. For next-character prediction, knowing whether the model is inside a URL, bracket, or quote genuinely improves prediction accuracy.

RNNs Can Handle Non-Traditional Sequential Tasks

Karpathy also mentions that even when the data itself is not a sequence, the processing procedure can be designed as a sequence. For example, a model can move attention step by step to read an image, or generate an image step by step on a canvas.

Source: Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks. The experiment on the left is related to Recurrent Models of Visual Attention.

Source: Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks. The experiment on the right is related to DRAW: A Recurrent Neural Network For Image Generation.

This gives an important perspective: RNNs are not only for “processing sequence data”; they can also represent “a computation process executed in order.”

Sampling Temperature: Why Generation Changes

A character-level language model outputs a probability distribution for the next character. During sampling, a temperature value is often used to adjust the distribution:

$$ p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)} $$

Where:

$z_i$ is the logit of the $i$-th character.
$T$ is the temperature.

The effect of temperature:

$T < 1$: the distribution becomes sharper; the model is more conservative and more likely to repeat high-probability patterns.
$T = 1$: normal sampling.
$T > 1$: the distribution becomes flatter; output is more diverse, but errors also increase.

So the “creativity” and “stability” of a generative model are often a tradeoff.

Advantages of RNNs

RNN advantages can be summarized as:

Naturally suitable for streaming input: when data arrives one time step at a time, an RNN can continuously update its state.
Parameter sharing: the same step function is reused for sequences of arbitrary length.
State compression: the hidden state can summarize past context.
Simple generation intuition: predict the next token, then feed the output back into the model.
Still valuable for small models and certain temporal tasks, such as sensor sequences, real-time speech, and low-latency tasks on edge devices.

Limitations of RNNs

The main limitations are also clear.

Sequential Computation Is Hard to Parallelize

An RNN must compute $h_{t-1}$ before it can compute $h_t$. This means the time steps inside one sequence are hard to fully parallelize.

For short sequences this may not matter much, but in large-scale language model training, both data size and model size are huge. Not being able to fully use GPU/TPU parallelism becomes a core bottleneck.

Long-Distance Dependencies Are Difficult

In theory, the hidden state can carry all historical information. In practice, it is hard for a fixed-length vector to losslessly compress a long context. Earlier information passes through more state updates and is more likely to be overwritten or weakened.

LSTM/GRU mitigate this issue, but do not completely solve it.

The Information Path Is Too Long

If the first token needs to affect the 1000th token, the information in an RNN must pass through about 1000 recurrent updates. The longer the path, the harder optimization becomes and the easier information is lost.

Hidden State Is a Bottleneck

RNNs compress the past into one hidden vector. This vector must both store context and participate in the next computation. Karpathy also mentions in the outlook section that RNNs couple representation capacity with per-step computation cost: the larger the hidden state, the more expensive each matrix multiplication becomes.

Why RNNs Were Replaced by Transformers

Transformers did not replace RNNs because “RNNs are completely useless.” They replaced them because, for large-scale NLP tasks, Transformers have better engineering properties and modeling scalability.

Transformers Are Easier to Train in Parallel

RNNs recur over time:

$$ h_t = f(h_{t-1}, x_t) $$

Transformer self-attention can compute relationships among all positions in a sequence within the same layer. During training, token representations in a batch can be heavily matrixized and parallelized.

This is one of the core motivations of Attention Is All You Need: remove recurrence and convolution, and build sequence transduction models using attention only, improving parallelization and reducing training time.

Shorter Paths for Long-Distance Dependencies

In an RNN, information between distant tokens must pass through many time steps. Transformer self-attention lets any two positions directly connect within one layer.

A rough comparison:

Model	Information path between distant tokens
RNN	$O(n)$
CNN	Depends on convolution depth and receptive field
Transformer self-attention	$O(1)$

Shorter paths usually make long-distance dependencies easier to learn.

Attention Explicitly Reads Context

RNNs rely on hidden states to compress history, while Transformer attention dynamically reads information from other positions for the current position:

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

This allows the model to retrieve information from context according to relevance, instead of relying entirely on a recursively compressed state.

Transformers Fit Large-Model Scaling Better

Modern large language models rely on large data, large parameter counts, and large compute. Transformers are matrix-multiplication-heavy and parallel-friendly, which makes them better at using hardware efficiently. RNNs are limited by time-step dependencies and have lower throughput.

In the era of large-scale pretraining, the advantage of Transformers is not only algorithmic performance, but also hardware efficiency, training stability, ecosystem tooling, and scalability.

But RNNs Have Not Disappeared Completely

RNNs are still useful in some scenarios:

Streaming inference: input arrives continuously and recomputing the whole context is undesirable.
Low-latency edge tasks: small models, fixed state, and controllable inference cost.
Time-series tasks: some sensor or control tasks do not necessarily need full self-attention.
New architecture research: state-space models, linear attention, and RWKV-like models reuse ideas from recurrence.

So a more accurate statement is: Transformers replaced traditional RNN/LSTM models in mainstream NLP and large-model training, but the idea of recurrent state continues to exist in many newer architectures.

Study Summary

The core of RNNs is not a complicated formula, but a simple and powerful abstraction: use the same function repeatedly to process a sequence, and use a hidden state to carry past information.

Karpathy’s article is classic because it does not start by stacking theory. Instead, it uses character-level generation experiments to show that if the training objective is general enough, the model will spontaneously learn spelling, formatting, brackets, quotes, URLs, code structure, and other multi-level patterns in order to predict the next character.

But the recurrent structure of RNNs also creates natural bottlenecks at scale: difficult parallelization, hard optimization of long dependencies, and limited hidden-state compression capacity. Transformers use self-attention to let sequence positions interact directly and greatly improve parallel training, which is why they became the mainstream architecture for modern NLP and large language models.

Sources

Andrej Karpathy, The Unreasonable Effectiveness of Recurrent Neural Networks, 2015-05-21
Andrej Karpathy, char-rnn GitHub repository
Andrej Karpathy, Minimal character-level language model with a Vanilla Recurrent Neural Network, in Python/numpy
Ashish Vaswani et al., Attention Is All You Need, 2017