LLM on XEDCZQ Blog

AI Interview Project: llm-provider Module

Tue, 09 Jun 2026 00:00:00 +0800

Llm-provider Module Design and Implementation

This note records the design and API implementation of the llm-provider module in the interview-guide project. This module is responsible for unified management of large model Provider configuration, including model lists, default models, Embedding capability, connectivity testing, and ASR/TTS runtime configuration for voice interviews.

Module Capability Overview

Provider management: supports querying, creating, updating, and deleting LLM Providers.
Dual storage modes: supports both DB mode and Legacy configuration-file mode.
Secret protection: in DB mode, API Keys are encrypted with AES-GCM and masked before being returned by APIs.
Default model management: separates the default Chat Provider from the default Embedding Provider.
Cache reload: after Provider changes, clears ChatClient and EmbeddingModel caches and rebuilds them on next use.
Embedding validation: validates model type, dimensions, and capability switches when creating, updating, or setting the default Embedding Provider.
Connectivity testing: supports sending real HTTP test requests to LLM Providers.
Voice configuration management: supports reading and updating Qwen ASR/TTS configuration and reloading runtime services.

Flowchart

stateDiagram-v2
 [*] --> ProviderConfigPage: Open LLM Provider config page

 ProviderConfigPage --> ProviderListLoaded: GET /api/llm-provider/list
 ProviderListLoaded --> ProviderDetailLoaded: GET /api/llm-provider/{id}

 ProviderListLoaded --> CreatingProvider: POST /api/llm-provider
 CreatingProvider --> ProviderSaved: Validation passed and saved
 CreatingProvider --> Error: duplicated id / invalid params / write failure

 ProviderDetailLoaded --> UpdatingProvider: PUT /api/llm-provider/{id}
 UpdatingProvider --> ProviderSaved: Partial fields updated
 UpdatingProvider --> Error: Provider missing / invalid params / write failure

 ProviderSaved --> RegistryReloaded: registry.reload()
 RegistryReloaded --> ProviderListLoaded: Reload list

 ProviderDetailLoaded --> TestingProvider: POST /api/llm-provider/{id}/test
 TestingProvider --> TestSuccess: External LLM API connected
 TestingProvider --> TestFailed: connection failed / auth failed / model unavailable
 TestSuccess --> ProviderDetailLoaded
 TestFailed --> ProviderDetailLoaded

 ProviderDetailLoaded --> UpdatingDefaultChat: PUT /api/llm-provider/default-provider
 UpdatingDefaultChat --> DefaultChatUpdated: Provider exists
 UpdatingDefaultChat --> Error: Provider missing / defaultProvider empty

 ProviderDetailLoaded --> UpdatingDefaultEmbedding: PUT /api/llm-provider/default-embedding-provider
 UpdatingDefaultEmbedding --> DefaultEmbeddingUpdated: Embedding supported
 UpdatingDefaultEmbedding --> Error: Provider missing / Embedding unsupported

 DefaultChatUpdated --> RegistryReloaded
 DefaultEmbeddingUpdated --> RegistryReloaded

 ProviderDetailLoaded --> DeletingProvider: DELETE /api/llm-provider/{id}
 DeletingProvider --> ProviderDeleted: Not a default Provider
 DeletingProvider --> Error: Provider missing / default Provider cannot be deleted
 ProviderDeleted --> RegistryReloaded

 ProviderListLoaded --> ManualReloading: POST /api/llm-provider/reload
 ManualReloading --> RegistryReloaded

 ProviderConfigPage --> VoiceConfigLoaded: GET /voice/asr or GET /voice/tts
 VoiceConfigLoaded --> UpdatingVoiceConfig: PUT /voice/asr or PUT /voice/tts
 UpdatingVoiceConfig --> VoiceConfigSaved: Write YAML and reload ASR/TTS service
 UpdatingVoiceConfig --> Error: config write failure

 VoiceConfigLoaded --> TestingAsr: POST /voice/asr/test
 TestingAsr --> AsrTestSuccess: WebSocket port connected
 TestingAsr --> AsrTestFailed: connection failed
 AsrTestSuccess --> VoiceConfigLoaded
 AsrTestFailed --> VoiceConfigLoaded

 Error --> ProviderConfigPage: Show error and return to config page

Core Design

The core of the llm-provider module is managing model configuration reads, secret protection, default model selection, and runtime client caches in one service.

In DB mode, Provider configuration comes from the database. After the service reads LlmProviderEntity, it decrypts the API Key, masks it, and converts the result into ProviderDTO for the frontend. The plaintext API Key only appears briefly at runtime on the server side and is never returned by the API.

In Legacy mode, Provider configuration comes from ConfigurationProperties. Create, update, and delete operations also modify the YAML configuration file and .env file, then reload the Provider registry after the update.

The module uses rwLock to control concurrent reads and writes. Query APIs use a read lock, while create, update, delete, and default-value updates use a write lock to avoid inconsistent configuration during concurrent access.

Provider List Query

`GET /api/llm-provider/list` Get All Providers

Returns:

Result<List<ProviderDTO>>

Call chain:

providerController.listProviders();
providerService.listProviders();
globalSettingRepository.findById(1L);
providerRepository.findAll();
encryptionService.decrypt(nonce, ciphertext);

Flow:

Controller calls listProviders().
Service acquires rwLock.readLock().
In DB mode, it first queries global settings to identify the default Chat Provider and default Embedding Provider.
It queries all LlmProviderEntity records.
It iterates over each Provider:
- Decrypts the API Key.
- Calls maskApiKey(...) to mask it.
- Calls resolveEmbeddingDimensions(...) to resolve vector dimensions, using the global default when not configured.
- Maps it to ProviderDTO.
In Legacy mode, it reads in-memory configuration from properties.getProviders().
It returns the Provider list.

Key points:

DB read failures throw BusinessException(PROVIDER_CONFIG_READ_FAILED).
API Keys are never returned to the frontend in plaintext.
There is a current issue: if DB storage is enabled for LLM configuration, changes to configuration files and API Keys will not automatically sync to DB even after restarting the project, unless DB mode is disabled or the database configuration is cleaned.

`GET /api/llm-provider/{id}` Get a Single Provider

Returns:

Result<ProviderDTO>

Flow:

Controller receives the Provider id.
Service acquires the read lock.
In DB mode, it queries global settings and the target Provider.
If the Provider does not exist, it throws BusinessException(PROVIDER_NOT_FOUND).
It decrypts the API Key and masks it.
It resolves Embedding dimensions and builds ProviderDTO.
In Legacy mode, it gets the Provider by id from in-memory configuration.

Provider Creation and Update

`POST /api/llm-provider` Create a Provider

Returns:

Result<Void>

Call chain:

providerService.createProvider(request);
providerRepository.existsById(request.id());
validateEmbeddingConfig(...);
encryptionService.encrypt(apiKey);
providerRepository.save(entity);
registry.reload();

Flow:

Controller receives CreateProviderRequest.
@Valid ensures id, baseUrl, apiKey, and model are not blank.
Service starts a transaction and acquires the write lock.
In DB mode, it first checks whether the Provider ID already exists.
It performs secondary non-blank validation on baseUrl, model, and apiKey.
It calls validateEmbeddingConfig(...) to validate Embedding configuration.
It encrypts the API Key with encryptionService.encrypt(apiKey).
It saves LlmProviderEntity.
It calls registry.reload() to clear runtime caches.

Legacy mode handling:

Checks whether properties.getProviders() already contains the same ID.
Builds ProviderConfig and puts it into the in-memory Map.
Calls writeProviderToYaml(...) to write back to YAML.
Calls writeEnvValue(...) to write to .env.
Calls registry.reload() to reload caches.

Embedding validation logic:

supportsEmbedding = true
embeddingModel == null // throw error
looksLikeChatModel(...) // throw error and recommend a model
embeddingDimensions <= 0 // throw error

`PUT /api/llm-provider/{id}` Update a Provider

Returns:

Result<Void>

Call chain:

providerService.updateProvider(id, request);
providerRepository.findById(id);
validateEmbeddingConfig(...);
encryptionService.encrypt(newApiKey);
providerRepository.save(entity);
registry.reload();

Flow:

Controller receives Provider id and UpdateProviderRequest.
Service starts a transaction and acquires the write lock.
In DB mode, it queries the Provider by id.
If the Provider does not exist, it throws BusinessException(PROVIDER_NOT_FOUND).
It updates fields selectively:
- baseUrl: null means no update; empty string is illegal.
- model: null means no update; empty string is illegal.
- apiKey: null means no update; empty string is illegal; re-encrypted when updated.
- embeddingModel: can be null to clear it.
- embeddingDimensions: updated from the request value.
- supportsEmbedding: updated from the request value.
- temperature: updated from the request value.
It calls validateEmbeddingConfig(...) for full validation.
It saves the entity and reloads caches.

Notes:

UpdateProviderRequest does not use @Valid, and all fields are optional.
null means do not update.
Empty strings are treated as invalid input.

Provider Deletion and Reload

`DELETE /api/llm-provider/{id}` Delete a Provider

Returns:

Result<Void>

Flow:

Service starts a transaction and acquires the write lock.
In DB mode, it reads global settings.
It checks whether the current Provider is the default Chat Provider or default Embedding Provider.
If it is a default Provider, it throws BusinessException(PROVIDER_DEFAULT_CANNOT_DELETE).
It queries the target Provider, confirms it exists, and deletes it.
It calls registry.reload() to clear runtime caches.

Legacy mode handling:

Checks whether the Provider is the default Provider.
Removes the configuration from the in-memory Map.
Calls removeProviderFromYaml(...) to remove the YAML node.
Calls removeFromEnv(...) to remove the API Key line from .env.
Calls registry.reload() to reload caches.

Protection mechanism:

Default Chat Provider and default Embedding Provider cannot be deleted directly.
The default value must be switched before deleting the original Provider.

`POST /api/llm-provider/reload` Manually Reload Provider Cache

Returns:

Result<Void>

Logic:

registry.reload();
clientCache.clear();
embeddingModelCache.clear();

Notes:

This API does not acquire a lock.
It does not start a transaction.
It does not access the database.
It only clears in-memory ChatClient and EmbeddingModel caches.
The next call to getChatClient() or Embedding model retrieval rebuilds clients from the latest configuration.

Provider Connectivity Test

`POST /api/llm-provider/{id}/test` Test Provider Connection

Returns:

Result<ProviderTestResult>

Flow:

Service acquires the read lock.
It reads runtime configuration based on the current mode:
- In DB mode, calls getProviderRuntimeConfigOrThrow(id).
- In Legacy mode, calls toRuntimeConfig(...).
It builds a RestClient:
- connectTimeout = 5s
- readTimeout = 10s
- Header: Authorization: Bearer {apiKey}
It builds the test request body:

{
 "model": "xxx",
 "messages": [
 {
 "role": "user",
 "content": "Reply with OK only."
 }
 ],
 "max_tokens": 1
}

It builds candidate test URLs:
- baseUrl + "/chat/completions"
- If baseUrl does not contain a version number, also try baseUrl + "/v1/chat/completions"
It sends POST requests to candidate URLs in order.
If any URL succeeds, it returns success.
If all URLs fail, it returns the last failure reason.

Notes:

This is the only Provider management API that directly calls an external LLM API.
The test sends a real HTTP request.
HTTP errors record status code and response body, while other exceptions record exception type and message.

Default Provider Management

`GET /api/llm-provider/default-provider` Get Default Providers

Returns:

Result<DefaultProviderDTO>

Flow:

Service acquires the read lock.
In DB mode, it queries globalSettingRepository.findById(1L).
It returns the default Chat Provider ID and default Embedding Provider ID.
In Legacy mode, it builds the response from properties.defaultProvider and properties.defaultEmbeddingProvider.

Response structure:

{
 "defaultProvider": "dashscope",
 "defaultEmbeddingProvider": "dashscope"
}

`PUT /api/llm-provider/default-provider` Set Default Chat Provider

Returns:

Result<Void>

Flow:

Service starts a transaction and acquires the write lock.
It reads request.defaultProvider().
If the default Provider is empty, it throws BAD_REQUEST.
It queries the target Provider and confirms it exists.
In DB mode, it updates GlobalSettingEntity.defaultChatProviderId.
It saves global settings.
It calls registry.reload().

Legacy mode handling:

Validates that the Provider exists.
Updates properties.setDefaultProvider(providerId).
Calls writeDefaultProviderToYaml(providerId) to write configuration back.
Removes the old module-defaults configuration.
Calls registry.reload().

`PUT /api/llm-provider/default-embedding-provider` Set Default Embedding Provider

Returns:

Result<Void>

Flow:

Service starts a transaction and acquires the write lock.
It reads request.defaultEmbeddingProvider().
If the default Embedding Provider is empty, it throws BAD_REQUEST.
It queries the target Provider and confirms it exists.
It validates that the Provider supports Embedding:
- supportsEmbedding must be true.
- embeddingModel must exist.
- validateEmbeddingConfig(...) must pass.
In DB mode, it updates GlobalSettingEntity.defaultEmbeddingProviderId.
It saves global settings.
It calls registry.reload().

Difference from default Chat Provider:

Setting the default Embedding Provider includes additional Embedding capability validation.
A Provider that does not support Embedding cannot be set as the default vector service.

ASR Configuration Management

`GET /api/llm-provider/voice/asr` Get ASR Configuration

Returns:

Result<AsrConfigDTO>

Flow:

Service acquires the read lock.
It reads voiceProperties.getQwen().getAsr() from VoiceInterviewProperties.
It builds AsrConfigDTO:
- url
- model
- language
- format
- sampleRate
- maskedApiKey
- enableTurnDetection
- turnDetectionType
- turnDetectionThreshold
- turnDetectionSilenceDurationMs
- VAD-related parameters
It returns the masked ASR configuration.

Notes:

ASR configuration comes from VoiceInterviewProperties.
The configuration prefix is app.voice-interview.
This configuration does not use DB storage.

`PUT /api/llm-provider/voice/asr` Update ASR Configuration

Returns:

Result<Void>

Flow:

Service acquires the write lock.
It reads runtime ASR and TTS configuration references.
It updates ASR fields selectively:
- url
- model
- language
- format
- sampleRate
- enableTurnDetection
- turnDetectionType
- turnDetectionThreshold
- turnDetectionSilenceDurationMs
If API Key is updated, it synchronizes ASR and TTS:

asr.setApiKey(apiKey);
tts.setApiKey(apiKey);
updateEnvValue("AI_BAILIAN_API_KEY", apiKey);

It calls writeAsrConfigToYaml(asr) to write back to YAML.
It calls asrService.reload(voiceProperties) to reload ASR.
If API Key is updated, it also calls ttsService.reload(voiceProperties).

Notes:

This method does not have @Transactional.
ASR and TTS share the Bailian API Key.
Updating the ASR API Key also affects TTS.

TTS Configuration Management

`GET /api/llm-provider/voice/tts` Get TTS Configuration

Returns:

Result<TtsConfigDTO>

Flow:

Service acquires the read lock.
It reads voiceProperties.getQwen().getTts() from VoiceInterviewProperties.
It builds TtsConfigDTO:
- model
- maskedApiKey
- voice
- format
- sampleRate
- mode
- languageType
- speechRate
- volume
It returns the masked TTS configuration.

`PUT /api/llm-provider/voice/tts` Update TTS Configuration

Returns:

Result<Void>

Flow:

Service acquires the write lock.
It reads runtime ASR and TTS configuration references.
It updates TTS fields selectively:
- model
- voice
- format
- sampleRate
- mode
- languageType
- speechRate
- volume
If API Key is updated, it synchronizes TTS and ASR:

tts.setApiKey(apiKey);
asr.setApiKey(apiKey);
updateEnvValue("AI_BAILIAN_API_KEY", apiKey);

It calls writeTtsConfigToYaml(tts) to write back to YAML.
It calls ttsService.reload(voiceProperties) to reload TTS.
If API Key is updated, it also calls asrService.reload(voiceProperties).

Notes:

TTS update logic is symmetrical with ASR.
ASR/TTS API Keys are always updated together.

ASR Connectivity Test

`POST /api/llm-provider/voice/asr/test` Test ASR Connection

Returns:

Result<ProviderTestResult>

Flow:

Service acquires the read lock.
It reads ASR configuration from voiceProperties.getQwen().getAsr().
It parses the WebSocket URL:
- wss defaults to port 443.
- ws defaults to port 80.
It runs a TCP Socket connection test:

socket.connect(address, 5000);
socket.close();

On success, it returns:

ProviderTestResult(success=true, "ASR WebSocket connection succeeded: host")

On failure, it returns the failure reason.

Differences from Provider connectivity testing:

ASR testing only checks TCP Socket connectivity.
It does not send a WebSocket handshake.
It does not call the real ASR recognition API.
Provider testing sends a real HTTP request to the LLM service.

Cache and Runtime Behavior

After Provider configuration changes, registry.reload() is called. This method clears internal caches:

clientCache.clear();
embeddingModelCache.clear();

Therefore, configuration changes do not immediately create new clients. Instead, clients are rebuilt on demand the next time business code uses the Provider. This avoids forcing update APIs to bear the initialization cost of model clients and ensures old configuration does not remain in cache for too long.

It is important to note that reload only clears caches; it does not synchronize configuration sources. If DB mode is enabled, the system prioritizes database configuration instead of re-importing from YAML or .env.

Current Issues and Optimization Directions

The module already supports DB mode and Legacy mode, but the configuration synchronization boundary still needs to be clarified:

After DB mode is enabled, changes to YAML and .env are not automatically written back to the database.
Restarting the project only reloads runtime configuration and cannot resolve inconsistencies between DB configuration and file configuration.
Manual reload only clears runtime caches and does not re-import configuration sources.
ASR/TTS configuration still comes from VoiceInterviewProperties, which is not the same storage as Provider DB configuration.
ASR/TTS update methods have no transaction, so writing YAML, writing .env, and reloading services may partially succeed.

Future improvements:

Add a configuration import API for DB mode to sync Providers from YAML and .env into the database.
Add a one-time startup migration strategy and clearly define whether DB or configuration files have priority.
Add a version number or update time to Provider configuration to help diagnose whether caches have refreshed.
Move ASR/TTS configuration into unified configuration storage to reduce inconsistencies caused by multiple sources.
Add failure compensation or clearer error messages for YAML writes, .env writes, and service reloads.

Summary

The llm-provider module is the unified configuration entry for large model capabilities. It manages not only Chat Providers, but also Embedding Providers, default models, runtime caches, and voice ASR/TTS configuration. Its key value is decoupling model configuration from business calls, allowing upper-layer features such as knowledgebase, RAG chat, and voice interview to obtain model capabilities through the unified Provider registry. The next focus is to further clarify DB and file-configuration synchronization so configuration sources are clearer and runtime state is more controllable.

Transformer 20-Step Visual Study Notes

Fri, 05 Jun 2026 23:10:00 +0800

Transformer 20-Step Visual Study Notes

This note is based on the interactive explanation from Transformer Explainer. It follows the site’s 20 steps and organizes them into a study note for understanding how GPT-style Transformers perform next-token prediction. The site uses GPT-2 small as the example model and visualizes the full pipeline from input tokens to output probabilities.

First, remember one sentence: the core task of GPT-style Transformers is next-token prediction. Given the prompt:

Data visualization empowers users to

The model needs to answer:

What is the most likely next token?

To answer this, a Transformer goes through tokenization, embedding, positional encoding, repeated Transformer blocks, self-attention, MLP, logits, probability distribution, sampling strategy, and other steps.

All screenshots in this note are taken from Transformer Explainer, developed by the Georgia Tech Polo Club team. They are used here for personal study notes. It is best to read this together with the original interactive site.

What Is Transformer

Transformer is the most common foundation architecture for modern large language models. Text-generation models such as GPT, Llama, and Gemini can all be understood as expanded versions of the Transformer architecture.

Its most important ability is not “memorizing answers”, but learning language patterns from large amounts of text and then predicting the next token from context at inference time. This prediction is repeated: predict one token, append it to the text, then predict the next one.

Source: Transformer Explainer, https://poloclub.github.io/transformer-explainer/

How Transformers Work

When a Transformer generates text, it does not write an entire paragraph at once. It generates step by step. Each step performs the same task: predict a probability distribution for the next token based on the existing context.

For example, if the current input is:

Data visualization empowers users to

The model may assign the highest probability to visualize, while also assigning relatively high probabilities to tokens such as create, see, and make. The final selected token is also affected by sampling parameters such as temperature, top-k, and top-p.

Source: Transformer Explainer

Transformer Architecture

A text-generation Transformer can be divided into three major parts:

Embedding: converts human-readable text into vectors the model can process.
Transformer Blocks: repeatedly refine each token representation, mainly through Self-Attention and MLP.
Output Probabilities: converts the final vector into probabilities over all tokens in the vocabulary.

At a high level, the information flow is:

text input -> token -> embedding -> Transformer Blocks -> logits -> probabilities -> sample next token

Source: Transformer Explainer

Embedding

The model cannot directly understand strings. Embedding converts each token into a sequence of numbers, or a vector. This vector is not manually designed; it is learned during training.

If two tokens often appear in similar contexts, their embeddings tend to be closer in high-dimensional space. You can think of embedding as an internal “semantic coordinate” used by the model.

GPT-2 small has a hidden dimension of 768, so each token is represented as a 768-dimensional vector.

Source: Transformer Explainer

Token Embedding

Tokenization splits input text into tokens. A token can be a full word or a subword. For example, empowers in the example is split into em and powers.

Each token has a unique ID. GPT-2 has a vocabulary size of 50,257, so its token embedding matrix is roughly:

$$ 50257 \times 768 $$

After the model gets the token ID, it looks up the corresponding 768-dimensional vector from this large matrix.

Source: Transformer Explainer

Positional Encoding

Self-Attention does not naturally know sequence order. If the model only receives a set of token vectors, it does not know which token comes first and which comes later.

So positional encoding is needed. GPT-2 uses learnable positional embeddings and adds the semantic token vector and the position vector:

$$ x_i = \text{TokenEmbedding}_i + \text{PositionEmbedding}_i $$

This lets the model know both “what this token is” and “where it is.”

Source: Transformer Explainer

Repetitive Transformer Blocks

Embedding is only the input representation, not a fully contextualized semantic representation. The real context modeling happens inside Transformer Blocks.

GPT-2 small has 12 Transformer Blocks. Each block roughly contains:

Multi-Head Self-Attention: lets tokens exchange information.
MLP: nonlinearly processes each token representation.
Residual, LayerNorm, and Dropout: stabilize training and improve generalization.

The meaning of stacking multiple layers is that lower layers tend to capture local and lexical information, while higher layers more easily form complex semantic and task-related representations.

Source: Transformer Explainer

Multi-Head Self Attention

The goal of Self-Attention is to let each token update itself based on context. For example, the token to has different meanings in different sentences. It needs to “look at” the previous context Data visualization empowers users to form a more accurate representation.

Multi-Head means the model does not use just one attention perspective. It uses multiple heads in parallel. GPT-2 small has 12 attention heads. Different heads can learn different relationships, such as syntactic relations, short-distance collocations, and long-distance semantic dependencies.

Source: Transformer Explainer

Query, Key, Value

Self-Attention maps each token’s input vector into three vectors:

Query (Q): what information the current token wants to search for.
Key (K): features by which each token can be matched by others.
Value (V): the actual information content to be aggregated and passed along.

They come from linear transformations:

$$ Q = XW_Q,\quad K = XW_K,\quad V = XW_V $$

A simple analogy is a search engine:

Query is the search query.
Key is the page title or index.
Value is the page content.

The model first computes relevance between Query and Key, then reads Value with weighted aggregation.

Source: Transformer Explainer

Multi-Head

GPT-2 small has an embedding dimension of 768 and 12 attention heads, so each head usually processes:

$$ 768 / 12 = 64 $$

The benefit of multi-head attention is learning multiple relationships in parallel. One head may focus on adjacent words, another on subject-verb relationships, and another on more distant semantic hints.

Multiple heads are not duplicate work; they give the model multiple “viewing angles.”

Source: Transformer Explainer

Masked Self Attention

GPT-style models generate text from left to right. When predicting the current position, they must not peek at future tokens, so causal mask, also called masked self-attention, is used.

The core formula is:

$$ \text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right)V $$

Where:

$QK^T$: computes pairwise similarity between tokens.
$\sqrt{d_k}$: scaling factor that prevents dot products from becoming too large and making softmax too sharp.
$M$: mask matrix that sets future positions to $-\infty$.
softmax: turns scores into probabilities.
multiplying by $V$: aggregates information according to attention weights.

Source: Transformer Explainer

Attention Output and Concatenation

Each head outputs a context-enhanced token representation. Since GPT-2 small has 12 heads, it produces 12 sets of results.

The model then concatenates the outputs from these heads and applies a linear projection back to the original hidden dimension 768:

head_1, head_2, ..., head_12 -> concat -> linear projection

The purpose is to let different heads extract information separately, then fuse those perspectives into one unified representation.

Source: Transformer Explainer

MLP

Attention handles information flow between tokens. MLP processes each token’s own representation nonlinearly.

GPT-2’s MLP usually contains two linear transformations with GELU activation in between:

$$ \text{MLP}(x)=W_2\cdot \text{GELU}(W_1x+b_1)+b_2 $$

The first layer expands the dimension from 768 to 3072, and the second layer compresses it back to 768. Expanding the dimension lets the model represent more complex features in a higher-dimensional space.

Note that MLP does not communicate across tokens like Attention. It processes each token independently.

Source: Transformer Explainer

Output Logit

After all Transformer Blocks, the model takes the output vector at the last position and uses it to predict the next token.

This vector goes through the final linear layer and is mapped to the vocabulary size:

$$ \text{logits}=h_{\text{last}}W_{\text{vocab}}+b $$

GPT-2 has a vocabulary size of 50,257, so logits are a vector of length 50,257. Each number corresponds to the raw score of a candidate token.

A logit is not a probability. It can be any real number and must go through softmax to become a probability distribution.

Source: Transformer Explainer

Probabilities

Softmax converts logits into probabilities:

$$ p_i=\frac{e^{z_i}}{\sum_j e^{z_j}} $$

After conversion:

Every token probability is between 0 and 1.
The probabilities of all tokens sum to 1.

In the figure, after the example input, the model considers tokens such as visualize, create, see, and make to be likely next tokens.

Source: Transformer Explainer

Temperature

Temperature scales logits before softmax:

$$ p_i=\frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)} $$

Where $T$ is the temperature:

$T < 1$: the probability distribution becomes sharper. High-score tokens are more likely to be selected, and output is more stable.
$T = 1$: logits are not additionally adjusted.
$T > 1$: the probability distribution becomes flatter. Low-probability tokens have more chances to be selected, and output becomes more diverse.

In simple terms, lower temperature is more conservative, while higher temperature is more divergent.

Source: Transformer Explainer

Sampling Strategy

After obtaining a probability distribution, the model still needs to decide how to choose the next token. Common strategies include:

Greedy Search: always choose the highest-probability token. Stable, but can be rigid.
Top-k: keep only the k highest-probability tokens, then sample from them.
Top-p: keep the smallest token set whose cumulative probability reaches p. Also called nucleus sampling.

Top-k is more like a fixed candidate pool, while top-p is a dynamic candidate pool. In practice, temperature and top-k/top-p are often tuned together.

Source: Transformer Explainer

Residual Connection

A residual connection adds a layer’s input directly to its output:

$$ y = x + F(x) $$

Its purpose is to preserve original information and make gradients pass through deep networks more easily. Without residual connections, training becomes harder as the model gets deeper, and information from early layers is more likely to be lost.

In Transformers, residual connections usually surround both Attention and MLP.

Source: Transformer Explainer

Layer Normalization

Layer Normalization normalizes the values inside a token vector, making the mean and variance more stable:

$$ \text{LayerNorm}(x)=\gamma\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta $$

It reduces training instability and makes the input distribution of each layer more controllable. GPT-2 uses a pre-norm style: LayerNorm is applied before entering Attention and MLP.

Intuitively, LayerNorm is like adjusting the numerical scale to a more suitable range before each key computation.

Source: Transformer Explainer

Dropout

Dropout is a regularization method used during training. It randomly sets part of the connections or activations to zero, preventing the model from over-relying on certain local features.

The intuition is: during training, do not let the model follow the exact same path every time, forcing it to learn more robust representations.

Important notes:

Dropout is mainly used during training.
Dropout is disabled during inference.
Many newer large models use less Dropout than early models because their training data is extremely large.

Source: Transformer Explainer

One Flowchart Summary

The inference flow of a GPT-style Transformer can be compressed into this chain:

flowchart LR
 A["Input text"] --> B["Tokenization"]
 B --> C["Token Embedding"]
 C --> D["Positional Encoding"]
 D --> E["Transformer Block x N"]
 E --> F["Multi-Head Self-Attention"]
 F --> G["MLP"]
 G --> H["Final Linear"]
 H --> I["Logits"]
 I --> J["Softmax Probabilities"]
 J --> K["Temperature / Top-k / Top-p"]
 K --> L["Sample next token"]

Key Differences from RNN

Combining this with the previous RNN note, the difference can be understood like this:

Aspect	RNN	Transformer
Information transfer	Passed step by step through hidden state	Self-Attention lets tokens directly read one another
Parallelism	Strong time-step dependency, hard to parallelize	Tokens in the same layer can be computed in parallel
Long-distance dependency	Long path, easy to decay	Any positions can directly connect
Context representation	Compressed into hidden state	Keeps explicit token representations for the whole context
Large-model training	Less efficient to scale	Better suited to large-scale GPU/TPU matrix computation

This is why modern LLMs mainly use Transformers: they are not only strong in modeling, but also better engineered for large-scale pretraining.

LLM on XEDCZQ Blog

AI Interview Project: llm-provider Module

Llm-provider Module Design and Implementation

Module Capability Overview

Flowchart

Core Design

Provider List Query

GET /api/llm-provider/list Get All Providers

GET /api/llm-provider/{id} Get a Single Provider

Provider Creation and Update

POST /api/llm-provider Create a Provider

PUT /api/llm-provider/{id} Update a Provider

Provider Deletion and Reload

DELETE /api/llm-provider/{id} Delete a Provider

POST /api/llm-provider/reload Manually Reload Provider Cache

Provider Connectivity Test

POST /api/llm-provider/{id}/test Test Provider Connection

Default Provider Management

GET /api/llm-provider/default-provider Get Default Providers

PUT /api/llm-provider/default-provider Set Default Chat Provider

PUT /api/llm-provider/default-embedding-provider Set Default Embedding Provider

ASR Configuration Management

GET /api/llm-provider/voice/asr Get ASR Configuration

PUT /api/llm-provider/voice/asr Update ASR Configuration

TTS Configuration Management

GET /api/llm-provider/voice/tts Get TTS Configuration

PUT /api/llm-provider/voice/tts Update TTS Configuration

ASR Connectivity Test

POST /api/llm-provider/voice/asr/test Test ASR Connection

Cache and Runtime Behavior

Current Issues and Optimization Directions

Summary

Transformer 20-Step Visual Study Notes

Transformer 20-Step Visual Study Notes

What Is Transformer

How Transformers Work

Transformer Architecture

Embedding

Token Embedding

Positional Encoding

Repetitive Transformer Blocks

Multi-Head Self Attention

Query, Key, Value

Multi-Head

Masked Self Attention

Attention Output and Concatenation

MLP

Output Logit

Probabilities

Temperature

Sampling Strategy

Residual Connection

Layer Normalization

Dropout

One Flowchart Summary

Key Differences from RNN

Sources

`GET /api/llm-provider/list` Get All Providers

`GET /api/llm-provider/{id}` Get a Single Provider

`POST /api/llm-provider` Create a Provider

`PUT /api/llm-provider/{id}` Update a Provider

`DELETE /api/llm-provider/{id}` Delete a Provider

`POST /api/llm-provider/reload` Manually Reload Provider Cache

`POST /api/llm-provider/{id}/test` Test Provider Connection

`GET /api/llm-provider/default-provider` Get Default Providers

`PUT /api/llm-provider/default-provider` Set Default Chat Provider

`PUT /api/llm-provider/default-embedding-provider` Set Default Embedding Provider

`GET /api/llm-provider/voice/asr` Get ASR Configuration

`PUT /api/llm-provider/voice/asr` Update ASR Configuration

`GET /api/llm-provider/voice/tts` Get TTS Configuration

`PUT /api/llm-provider/voice/tts` Update TTS Configuration

`POST /api/llm-provider/voice/asr/test` Test ASR Connection