Exploring AI
Detail Deep Dive
# Detail Deep Dive
Complete Guide: Model Architecture Features, Variants & Reverse Engineering
Feature Group 1: Core Model Architecture
Variant A: Mixture-of-Experts (MoE) [This Spec]
- Concept: Instead of running one monolithic neural network for every token, the network routes inputs to a small subset of specialized sub-networks ("experts").
- Deep Dive: A dynamic router layer evaluates incoming token embeddings and computes softmax probabilities across $N$ experts. It selects the top-$K$ highest-scoring experts to process that specific token, bypassing all other experts. This decoupling of total capacity from per-token computation allows models to scale to trillions of parameters while keeping latency and FLOPS comparable to much smaller models.
- Analogy: A modern multi-specialty medical center. When a patient arrives, a triage nurse (the router) evaluates the symptoms and sends them strictly to 2 or 3 relevant specialists (e.g., a cardiologist and a radiologist) rather than calling every doctor in the hospital into the room.
Variant B: Dense Models (e.g., LLaMA 3, GPT-3, Mistral)
- Concept: Every single parameter in the neural network is activated and calculated for every single input token.
- Deep Dive: Information passes through identical, shared Feed-Forward Networks (FFNs) at every layer. Dense models maximize knowledge density per parameter, providing maximum capability per parameter, but becoming prohibitively expensive to compute as parameter counts scale past hundreds of billions.
- Analogy: A single general practitioner who personally handles every patient, performing all diagnostics, tests, and treatments themselves using 100% of their medical training for every visitor.
Variant C: Linear / Recurrent State-Space Models (SSM) (e.g., Mamba, RWKV)
- Concept: Replaces quadratic attention ($O(N^2)$) with linear state-space models ($O(N)$) or recurrent neural mechanisms that compress sequence history into a fixed-size hidden state.
- Deep Dive: Instead of comparing every token against every prior token in a growing memory buffer, these architectures maintain a dynamic memory state vector that updates recurrently as new tokens arrive. This enables near-instant token generation speeds and linear memory scaling over long context windows.
- Analogy: A real-time radio translator who keeps a running summary in their head, updating their mental state with each word heard, rather than re-reading a transcript from page 1 every time a new sentence is spoken.
Variant D: Hybrid MoE-SSM Architectures (e.g., Jamba)
- Concept: Combines State-Space Model (SSM) sequence layers with Mixture-of-Experts (MoE) routing layers.
- Deep Dive: Sequence processing is handled by SSM/Mamba layers (eliminating quadratic KV-cache memory growth), while parameter capacity is scaled using sparse MoE layers in place of standard feed-forward blocks. This achieves high inference throughput while maintaining massive parameter scale.
- Analogy: An automated high-speed conveyor belt system (SSM) that streams materials forward in constant time, passing through specialized artisan stations (MoE) that selectively turn on only when specific work is required.
Feature Group 2: Parameter Activation Dynamics
Variant A: Granular / High-Sparsity MoE [This Spec: 2.8T Total / 104B Active]
- Concept: Activates a tiny fraction (under 5–10%) of a massive parameter pool per token across hundreds of micro-experts.
- Deep Dive: By dividing the network into 800+ tiny experts and activating 16 per token, tokens receive hyper-targeted processing. The model achieves 2.8 Trillion parameters of stored knowledge, yet only incurs the computational cost of a ~100B parameter model per forward pass.
- Analogy: A mega-consultancy firm with 896 niche contractors (e.g., a tax lawyer specializing strictly in maritime import regulations). For a specific query, the system routes your case to 16 precise contractors.
Variant B: Coarse-Grained / Low-Sparsity MoE (e.g., Mixtral 8x7B)
- Concept: Activates a large fraction (25–30%) of total parameters across a small number of larger experts.
- Deep Dive: The model splits feed-forward blocks into 8 large expert networks and routes tokens to the top-2 experts per layer. This provides stable training convergence and high capacity, but lower specialization granularity per expert.
- Analogy: A boutique law firm with 8 senior partners. For any client issue, the 2 most relevant partners team up to manage the case together.
Variant C: Fully Dense (100% Activation Ratio)
- Concept: Total Parameters = Activated Parameters.
- Deep Dive: For a 70B parameter model, all 70B weights actively execute matrix multiplications for every generated token, providing high computational efficiency on GPUs at the cost of higher per-token compute demands.
- Analogy: A orchestra where all 100 musicians play their instrument on every single beat, regardless of whether the piece is a soft whisper or a loud climax.
Feature Group 3: Layer Structure & Composition
Variant A: Hybrid Sparse/Dense Interleaved Stacks [This Spec: 69 KDA + 24 MLA / 1 Dense + 92 MoE]
- Concept: Mixes different attention mechanisms and expert routing configurations across the vertical stack of layers.
- Deep Dive: The network uses 69 Kernelized Delta Attention (KDA) layers for ultra-fast sequence processing alongside 24 Multi-Head Latent Attention (MLA) layers for compressed KV-cache retrieval, combined with 1 universal dense layer and 92 MoE layers. Early/dense layers stabilize basic syntax representations while sparse layers handle higher-order abstract reasoning.
- Analogy: A hybrid manufacturing plant where Stage 1 is a universal assembly line (Dense), Stages 2–70 use automated rapid conveyers (KDA), and Stages 71–93 use specialized precision inspection stations (MLA & MoE).
Variant B: Pure Dense Layering
- Concept: Every layer in the transformer stack is identical in design and fully activated.
- Deep Dive: Stacks 32 to 96 identical Transformer layers where each layer contains standard Self-Attention followed by a standard dense Feed-Forward Network (FFN).
- Analogy: A 90-floor office building where every single floor has the exact same layout, floorplan, and staffing structure.
Variant C: Fully Sparse MoE Layering
- Concept: Every single layer in the network stack routes tokens dynamically to experts.
- Deep Dive: Maximizes capacity across all depth stages. Requires careful router load-balancing (such as Quantile Balancing or auxiliary loss terms) to prevent expert collapse across 90+ consecutive sparse layers.
- Analogy: An automated package sorting facility where every single intersection across 90 successive conveyer belts independently routes parcels based on micro-scanned tags.
Feature Group 4: Attention Mechanisms
Variant A: Kernelized / Linear Attention (KDA) & Gated MLA [This Spec]
- Concept: Combines linear-time kernelized attention (KDA) for ultra-fast sequence scanning with latent compressed multi-head attention (Gated MLA) for low-memory retrieval.
- Deep Dive: KDA reformulates matrix multiplication order in attention to bypass the $O(N^2)$ sequence length bottleneck. Gated MLA projects Key and Value matrices down into a low-rank latent vector space, cutting Key-Value (KV) cache VRAM requirements by over 80–90% during inference over long context windows.
- Analogy: Combining a high-speed scanner that skims 1,000 pages in seconds with an ultra-compressed shorthand indexing system in the margin.
Variant B: Standard Multi-Head Attention (MHA)
- Concept: Every attention head maintains separate Key, Value, and Query projection matrices.
- Deep Dive: Offers maximum representation fidelity, but KV-cache VRAM usage scales linearly with the number of heads ($H$) and sequence length ($N$). At long context windows (e.g., 100K+ tokens), KV-cache memory can easily overwhelm GPU VRAM.
- Analogy: 96 independent researchers taking their own individual, uncompressed, word-for-word handwritten notes for every single page of a giant legal contract.
Variant C: Grouped-Query Attention (GQA)
- Concept: Multiple Query heads share a single Key/Value head group (e.g., 8 Query heads share 1 KV head).
- Deep Dive: Reduces KV-cache memory footprint by 4x to 8x with virtually zero degradation in model performance. This is the standard attention mechanism used in LLaMA 2/3, Mistral, and Qwen models.
- Analogy: Dividing a 96-person committee into 8 working groups, where each group shares 1 assigned court reporter who maintains the master notes for everyone in that group.
Feature Group 5: Vocabulary & Activation Functions
Variant A: SiTU-GLU Activation & 160K Vocab [This Spec]
- Concept: Combines a large multi-byte tokenizer dictionary (160,000 entries) with a gated non-linear activation unit (SiTU-GLU) to prevent activation explosions in large sparse models.
- Deep Dive:
- 160K Vocab: Captures complex code constructs, mathematical symbols, and non-English scripts as single tokens, improving context compression efficiency.
- SiTU-GLU: A Gated Linear Unit variant using SiLU-style smooth gating designed to maintain numerical stability during training of extreme MoE models with hundreds of routed experts.
- Analogy: An unabridged international dictionary paired with a high-precision digital dimmer switch that smoothly regulates energy flow through complex circuits.
Variant B: Compact Vocab (32K) & Standard Activation (ReLU / GELU)
- Concept: Small vocabulary dictionaries with basic step-function or smooth non-linear activations.
- Deep Dive: Common in early LLMs (e.g., GPT-3, early LLaMA). 32K vocabularies require less embedding parameter memory, but split foreign words and code into multiple sub-word chunks, consuming context length faster.
- Analogy: A basic pocket dictionary containing only root words, requiring you to spell out complex technical words character-by-character.
Feature Group 6: Vision Encoder & Multimodality
Variant A: Scratch-Trained Native Vision Encoders (MoonViT-V2) [This Spec]
- Concept: A vision transformer (401M parameters) trained from scratch alongside the language model using next-token prediction, rather than using a frozen, pre-aligned vision model.
- Deep Dive: Instead of taking an off-the-shelf vision encoder (like SigLIP or CLIP) and aligning it to the LLM via cross-attention adapters or projection layers, MoonViT-V2 learns text and visual representations jointly from day one, offering higher optimization stability and native image-text reasoning.
- Analogy: Raising a child from birth in a fully bilingual environment (visuals and text learned together), rather than teaching them text first and hiring an interpreter later to translate images into words.
Variant B: Contrastive Pre-trained Adapters (CLIP / SigLIP)
- Concept: Connects a pre-trained image classifier/encoder to a text model using a projection MLP layer or cross-attention bridge.
- Deep Dive: The vision encoder is trained separately on image-text matching tasks (contrastive learning) and then frozen. A linear projection layer or perceiver resampler maps image embeddings into the language model's token space.
- Analogy: Hiring an external visual analyst to view photographic evidence and write a brief written report, which is then handed to the main analyst.
Feature Group 7: Quantization & Numerical Precision
Variant A: Microscaling Formats with QAT (MXFP4 Weights / MXFP8 Activations) [This Spec]
- Concept: Trains the model from scratch (or SFT) using 4-bit weights and 8-bit activations using Microscaling (MX) vector block formats.
- Deep Dive: Standard quantization rounds numbers post-hoc, causing precision loss. MX formats group blocks of 32 weights under a shared exponent scale factor. Combined with Quantization-Aware Training (QAT), the model learns optimal parameters under low-bit constraints, allowing a 2.8T parameter model to run in a fraction of standard FP16 VRAM.
- Analogy: Designing a building using modular, standardized mini-bricks from day one. Because the architect planned for low-scale precision during the original blueprint design, no structural details are ruined.
Variant B: Uncompressed High-Precision (FP16 / BF16)
- Concept: 16-bit floating-point format (2 bytes per weight parameter).
- Deep Dive: Maximum mathematical precision without quantization noise, but requires massive VRAM footprints (e.g., a 2.8T model in BF16 requires ~5.6 Terabytes of VRAM just to load weights).
- Analogy: Printing a massive architectural blueprint on ultra-heavy gloss paper at maximum photographic resolution—stunning clarity, but requiring giant shipping crates to carry around.
Variant C: Post-Training Quantization (PTQ - INT4 / GGUF / AWQ)
- Concept: The model is trained in 16-bit precision and compressed down to 4-bit or 8-bit integers after training completes.
- Deep Dive: Quick and easy to execute after weights are released, but aggressive post-hoc rounding can cause "outlier weight collapse" in ultra-large models, leading to degradation in complex reasoning tasks.
- Analogy: Taking a finished full-size oil painting and digitizing it down to a low-resolution compressed image file—some subtle color gradients and fine details get lost.
MoE Expert Architecture & Reverse-Engineering Guide
What is an "Expert" in Neural Network Code?
Inside deep learning frameworks like PyTorch, an Expert is not a separate AI model. It is simply a standard Feed-Forward Network (FFN) block (a two- or three-layer MLP containing linear projections like gate_proj, up_proj, and down_proj).
In an MoE layer:
1. Router Network: A linear projection layer (router = nn.Linear(hidden_dim, num_experts)) evaluates the input token vector.
2. Top-K Selection: The router outputs logits for all 896 experts and selects the top 16 highest-scoring expert indices.
3. Execution: The token vector is routed exclusively through those 16 selected MLP expert blocks, and their outputs are combined using a weighted sum based on the router's softmax scores.
┌─────────────────────────┐
│ Input Token │
└────────────┬────────────┘
│
┌───────────v───────────┐
│ Router Gate │
└───────────┬───────────┘
│ Selects Top 16 of 896
┌───────────────────┼───────────────────┐
│ │ │
┌────────v────────┐ ┌────────v────────┐ ┌────────v────────┐
│ Expert 12 │ │ Expert 104 │ │ Expert 891 │ ... (16 Active Experts)
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ │ │
└───────────────────┼───────────────────┘
│ Weighted Sum
┌────────────v────────────┐
│ Output Vector │
└─────────────────────────┘
How to Reverse Engineer Expert Counts & Architecture Specs
You can inspect the exact architectural parameters of any open-weights MoE model checkpoint using terminal CLI commands or Python without needing to load or run the full model weights.
Method 1: Instant CLI Config Inspection (Remote Repositories)
Every HuggingFace repository contains a config.json file detailing its parameters. You can inspect it directly using python one-liners:
# Extract Expert Count and Active Experts from any HuggingFace MoE Model:
python3 -c "import urllib.request, json; \
data = json.loads(urllib.request.urlopen('[https://huggingface.co/moonshotai/Kimi-K3/raw/main/config.json').read](https://huggingface.co/moonshotai/Kimi-K3/raw/main/config.json').read)()); \
print('Total Experts: ', data.get('num_experts') or data.get('n_routed_experts') or data.get('num_local_experts')); \
print('Active Experts: ', data.get('num_experts_per_tok') or data.get('num_selected_experts')); \
print('Shared Experts: ', data.get('num_shared_experts') or data.get('n_shared_experts', 0)); \
print('Total Layers: ', data.get('num_hidden_layers')); \
print('Hidden Dimension:', data.get('hidden_size'))"
---
### Universal Python Configuration Inspector
```python
from transformers import AutoConfig
# Load architecture metadata without downloading multi-terabyte model weights
model_id = "moonshotai/Kimi-K3"
config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
# Key lookup for common MoE naming conventions
total_experts = getattr(config, "num_experts",
getattr(config, "n_routed_experts",
getattr(config, "num_local_experts", None)))
active_experts = getattr(config, "num_experts_per_tok",
getattr(config, "num_selected_experts", None))
shared_experts = getattr(config, "num_shared_experts",
getattr(config, "n_shared_experts", 0))
print("=== REVERSE-ENGINEERED MODEL METADATA ===")
print(f"Architecture Type: {config.model_type}")
print(f"Total Layers: {config.num_hidden_layers}")
print(f"Total Routed Experts: {total_experts}")
print(f"Active Experts per Token: {active_experts}")
print(f"Shared Experts: {shared_experts}")
print(f"Attention Heads: {config.num_attention_heads}")
print(f"Hidden Dimension: {config.hidden_size}")
Reverse-Engineering Local .safetensors Checkpoints
If you have local model files on disk, you can inspect tensor shapes directly in Python without loading tensors into GPU VRAM:
from safetensors import safe_open
# Open a local safetensors shard
shard_path = "model-00001-of-00030.safetensors"
with safe_open(shard_path, framework="pt") as f:
tensor_keys = f.keys()
# Locate router gate layers and expert weight matrices
router_keys = [k for k in tensor_keys if "gate" in k or "router" in k]
expert_keys = [k for k in tensor_keys if "experts" in k]
print("Sample Router Weight Keys:", router_keys[:2])
print("Sample Expert Weight Keys:", expert_keys[:2])
# Inspect 3D tensor shapes: [num_experts, hidden_dim, intermediate_dim]
for key in expert_keys:
if "gate_proj.weight" in key or "w1.weight" in key or "mlp.experts" in key:
shape = f.get_slice(key).get_shape()
print(f"\nTensor Key: {key}")
print(f"Raw Tensor Shape: {shape}")
if len(shape) == 3:
print(f"--> Inferred Expert Count: {shape[0]}")
print(f"--> Expert Input/Output Dims: {shape[1]} x {shape[2]}")
break
Zero-RAM PyTorch Model Structure Printing (meta Device)
Using PyTorch's meta device, you can instantiate the complete neural network class in RAM with 0 bytes of allocated weight memory, allowing you to print the entire internal module tree:
from transformers import AutoModelForCausalLM, AutoConfig
# 1. Load config
config = AutoConfig.from_pretrained("moonshotai/Kimi-K3", trust_remote_code=True)
# 2. Instantiate empty model structure on meta device (0 MB VRAM used)
with torch.device("meta"):
model = AutoModelForCausalLM.from_config(config, trust_remote_code=True)
# 3. Print complete module hierarchy to inspect exact expert block layout
print(model)
Complete Guide: Model Architecture Features, Variants & Reverse Engineering
Feature Group 1: Core Model Architecture
Variant A: Mixture-of-Experts (MoE) [This Spec]
- Concept: Instead of running one monolithic neural network for every token, the network routes inputs to a small subset of specialized sub-networks ("experts").
- Deep Dive: A dynamic router layer evaluates incoming token embeddings and computes softmax probabilities across $N$ experts. It selects the top-$K$ highest-scoring experts to process that specific token, bypassing all other experts. This decoupling of total capacity from per-token computation allows models to scale to trillions of parameters while keeping latency and FLOPS comparable to much smaller models.
- Analogy: A modern multi-specialty medical center. When a patient arrives, a triage nurse (the router) evaluates the symptoms and sends them strictly to 2 or 3 relevant specialists (e.g., a cardiologist and a radiologist) rather than calling every doctor in the hospital into the room.
Variant B: Dense Models (e.g., LLaMA 3, GPT-3, Mistral)
- Concept: Every single parameter in the neural network is activated and calculated for every single input token.
- Deep Dive: Information passes through identical, shared Feed-Forward Networks (FFNs) at every layer. Dense models maximize knowledge density per parameter, providing maximum capability per parameter, but becoming prohibitively expensive to compute as parameter counts scale past hundreds of billions.
- Analogy: A single general practitioner who personally handles every patient, performing all diagnostics, tests, and treatments themselves using 100% of their medical training for every visitor.
Variant C: Linear / Recurrent State-Space Models (SSM) (e.g., Mamba, RWKV)
- Concept: Replaces quadratic attention ($O(N^2)$) with linear state-space models ($O(N)$) or recurrent neural mechanisms that compress sequence history into a fixed-size hidden state.
- Deep Dive: Instead of comparing every token against every prior token in a growing memory buffer, these architectures maintain a dynamic memory state vector that updates recurrently as new tokens arrive. This enables near-instant token generation speeds and linear memory scaling over long context windows.
- Analogy: A real-time radio translator who keeps a running summary in their head, updating their mental state with each word heard, rather than re-reading a transcript from page 1 every time a new sentence is spoken.
Variant D: Hybrid MoE-SSM Architectures (e.g., Jamba)
- Concept: Combines State-Space Model (SSM) sequence layers with Mixture-of-Experts (MoE) routing layers.
- Deep Dive: Sequence processing is handled by SSM/Mamba layers (eliminating quadratic KV-cache memory growth), while parameter capacity is scaled using sparse MoE layers in place of standard feed-forward blocks. This achieves high inference throughput while maintaining massive parameter scale.
- Analogy: An automated high-speed conveyor belt system (SSM) that streams materials forward in constant time, passing through specialized artisan stations (MoE) that selectively turn on only when specific work is required.
Feature Group 2: Parameter Activation Dynamics
Variant A: Granular / High-Sparsity MoE [This Spec: 2.8T Total / 104B Active]
- Concept: Activates a tiny fraction (under 5–10%) of a massive parameter pool per token across hundreds of micro-experts.
- Deep Dive: By dividing the network into 800+ tiny experts and activating 16 per token, tokens receive hyper-targeted processing. The model achieves 2.8 Trillion parameters of stored knowledge, yet only incurs the computational cost of a ~100B parameter model per forward pass.
- Analogy: A mega-consultancy firm with 896 niche contractors (e.g., a tax lawyer specializing strictly in maritime import regulations). For a specific query, the system routes your case to 16 precise contractors.
Variant B: Coarse-Grained / Low-Sparsity MoE (e.g., Mixtral 8x7B)
- Concept: Activates a large fraction (25–30%) of total parameters across a small number of larger experts.
- Deep Dive: The model splits feed-forward blocks into 8 large expert networks and routes tokens to the top-2 experts per layer. This provides stable training convergence and high capacity, but lower specialization granularity per expert.
- Analogy: A boutique law firm with 8 senior partners. For any client issue, the 2 most relevant partners team up to manage the case together.
Variant C: Fully Dense (100% Activation Ratio)
- Concept: Total Parameters = Activated Parameters.
- Deep Dive: For a 70B parameter model, all 70B weights actively execute matrix multiplications for every generated token, providing high computational efficiency on GPUs at the cost of higher per-token compute demands.
- Analogy: A orchestra where all 100 musicians play their instrument on every single beat, regardless of whether the piece is a soft whisper or a loud climax.
Feature Group 3: Layer Structure & Composition
Variant A: Hybrid Sparse/Dense Interleaved Stacks [This Spec: 69 KDA + 24 MLA / 1 Dense + 92 MoE]
- Concept: Mixes different attention mechanisms and expert routing configurations across the vertical stack of layers.
- Deep Dive: The network uses 69 Kernelized Delta Attention (KDA) layers for ultra-fast sequence processing alongside 24 Multi-Head Latent Attention (MLA) layers for compressed KV-cache retrieval, combined with 1 universal dense layer and 92 MoE layers. Early/dense layers stabilize basic syntax representations while sparse layers handle higher-order abstract reasoning.
- Analogy: A hybrid manufacturing plant where Stage 1 is a universal assembly line (Dense), Stages 2–70 use automated rapid conveyers (KDA), and Stages 71–93 use specialized precision inspection stations (MLA & MoE).
Variant B: Pure Dense Layering
- Concept: Every layer in the transformer stack is identical in design and fully activated.
- Deep Dive: Stacks 32 to 96 identical Transformer layers where each layer contains standard Self-Attention followed by a standard dense Feed-Forward Network (FFN).
- Analogy: A 90-floor office building where every single floor has the exact same layout, floorplan, and staffing structure.
Variant C: Fully Sparse MoE Layering
- Concept: Every single layer in the network stack routes tokens dynamically to experts.
- Deep Dive: Maximizes capacity across all depth stages. Requires careful router load-balancing (such as Quantile Balancing or auxiliary loss terms) to prevent expert collapse across 90+ consecutive sparse layers.
- Analogy: An automated package sorting facility where every single intersection across 90 successive conveyer belts independently routes parcels based on micro-scanned tags.
Feature Group 4: Attention Mechanisms
Variant A: Kernelized / Linear Attention (KDA) & Gated MLA [This Spec]
- Concept: Combines linear-time kernelized attention (KDA) for ultra-fast sequence scanning with latent compressed multi-head attention (Gated MLA) for low-memory retrieval.
- Deep Dive: KDA reformulates matrix multiplication order in attention to bypass the $O(N^2)$ sequence length bottleneck. Gated MLA projects Key and Value matrices down into a low-rank latent vector space, cutting Key-Value (KV) cache VRAM requirements by over 80–90% during inference over long context windows.
- Analogy: Combining a high-speed scanner that skims 1,000 pages in seconds with an ultra-compressed shorthand indexing system in the margin.
Variant B: Standard Multi-Head Attention (MHA)
- Concept: Every attention head maintains separate Key, Value, and Query projection matrices.
- Deep Dive: Offers maximum representation fidelity, but KV-cache VRAM usage scales linearly with the number of heads ($H$) and sequence length ($N$). At long context windows (e.g., 100K+ tokens), KV-cache memory can easily overwhelm GPU VRAM.
- Analogy: 96 independent researchers taking their own individual, uncompressed, word-for-word handwritten notes for every single page of a giant legal contract.
Variant C: Grouped-Query Attention (GQA)
- Concept: Multiple Query heads share a single Key/Value head group (e.g., 8 Query heads share 1 KV head).
- Deep Dive: Reduces KV-cache memory footprint by 4x to 8x with virtually zero degradation in model performance. This is the standard attention mechanism used in LLaMA 2/3, Mistral, and Qwen models.
- Analogy: Dividing a 96-person committee into 8 working groups, where each group shares 1 assigned court reporter who maintains the master notes for everyone in that group.
Feature Group 5: Vocabulary & Activation Functions
Variant A: SiTU-GLU Activation & 160K Vocab [This Spec]
- Concept: Combines a large multi-byte tokenizer dictionary (160,000 entries) with a gated non-linear activation unit (SiTU-GLU) to prevent activation explosions in large sparse models.
- Deep Dive:
- 160K Vocab: Captures complex code constructs, mathematical symbols, and non-English scripts as single tokens, improving context compression efficiency.
- SiTU-GLU: A Gated Linear Unit variant using SiLU-style smooth gating designed to maintain numerical stability during training of extreme MoE models with hundreds of routed experts.
- Analogy: An unabridged international dictionary paired with a high-precision digital dimmer switch that smoothly regulates energy flow through complex circuits.
Variant B: Compact Vocab (32K) & Standard Activation (ReLU / GELU)
- Concept: Small vocabulary dictionaries with basic step-function or smooth non-linear activations.
- Deep Dive: Common in early LLMs (e.g., GPT-3, early LLaMA). 32K vocabularies require less embedding parameter memory, but split foreign words and code into multiple sub-word chunks, consuming context length faster.
- Analogy: A basic pocket dictionary containing only root words, requiring you to spell out complex technical words character-by-character.
Feature Group 6: Vision Encoder & Multimodality
Variant A: Scratch-Trained Native Vision Encoders (MoonViT-V2) [This Spec]
- Concept: A vision transformer (401M parameters) trained from scratch alongside the language model using next-token prediction, rather than using a frozen, pre-aligned vision model.
- Deep Dive: Instead of taking an off-the-shelf vision encoder (like SigLIP or CLIP) and aligning it to the LLM via cross-attention adapters or projection layers, MoonViT-V2 learns text and visual representations jointly from day one, offering higher optimization stability and native image-text reasoning.
- Analogy: Raising a child from birth in a fully bilingual environment (visuals and text learned together), rather than teaching them text first and hiring an interpreter later to translate images into words.
Variant B: Contrastive Pre-trained Adapters (CLIP / SigLIP)
- Concept: Connects a pre-trained image classifier/encoder to a text model using a projection MLP layer or cross-attention bridge.
- Deep Dive: The vision encoder is trained separately on image-text matching tasks (contrastive learning) and then frozen. A linear projection layer or perceiver resampler maps image embeddings into the language model's token space.
- Analogy: Hiring an external visual analyst to view photographic evidence and write a brief written report, which is then handed to the main analyst.
Feature Group 7: Quantization & Numerical Precision
Variant A: Microscaling Formats with QAT (MXFP4 Weights / MXFP8 Activations) [This Spec]
- Concept: Trains the model from scratch (or SFT) using 4-bit weights and 8-bit activations using Microscaling (MX) vector block formats.
- Deep Dive: Standard quantization rounds numbers post-hoc, causing precision loss. MX formats group blocks of 32 weights under a shared exponent scale factor. Combined with Quantization-Aware Training (QAT), the model learns optimal parameters under low-bit constraints, allowing a 2.8T parameter model to run in a fraction of standard FP16 VRAM.
- Analogy: Designing a building using modular, standardized mini-bricks from day one. Because the architect planned for low-scale precision during the original blueprint design, no structural details are ruined.
Variant B: Uncompressed High-Precision (FP16 / BF16)
- Concept: 16-bit floating-point format (2 bytes per weight parameter).
- Deep Dive: Maximum mathematical precision without quantization noise, but requires massive VRAM footprints (e.g., a 2.8T model in BF16 requires ~5.6 Terabytes of VRAM just to load weights).
- Analogy: Printing a massive architectural blueprint on ultra-heavy gloss paper at maximum photographic resolution—stunning clarity, but requiring giant shipping crates to carry around.
Variant C: Post-Training Quantization (PTQ - INT4 / GGUF / AWQ)
- Concept: The model is trained in 16-bit precision and compressed down to 4-bit or 8-bit integers after training completes.
- Deep Dive: Quick and easy to execute after weights are released, but aggressive post-hoc rounding can cause "outlier weight collapse" in ultra-large models, leading to degradation in complex reasoning tasks.
- Analogy: Taking a finished full-size oil painting and digitizing it down to a low-resolution compressed image file—some subtle color gradients and fine details get lost.
MoE Expert Architecture & Reverse-Engineering Guide
What is an "Expert" in Neural Network Code?
Inside deep learning frameworks like PyTorch, an Expert is not a separate AI model. It is simply a standard Feed-Forward Network (FFN) block (a two- or three-layer MLP containing linear projections like gate_proj, up_proj, and down_proj).
In an MoE layer:
1. Router Network: A linear projection layer (router = nn.Linear(hidden_dim, num_experts)) evaluates the input token vector.
2. Top-K Selection: The router outputs logits for all 896 experts and selects the top 16 highest-scoring expert indices.
3. Execution: The token vector is routed exclusively through those 16 selected MLP expert blocks, and their outputs are combined using a weighted sum based on the router's softmax scores.
┌─────────────────────────┐
│ Input Token │
└────────────┬────────────┘
│
┌───────────v───────────┐
│ Router Gate │
└───────────┬───────────┘
│ Selects Top 16 of 896
┌───────────────────┼───────────────────┐
│ │ │
┌────────v────────┐ ┌────────v────────┐ ┌────────v────────┐
│ Expert 12 │ │ Expert 104 │ │ Expert 891 │ ... (16 Active Experts)
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ │ │
└───────────────────┼───────────────────┘
│ Weighted Sum
┌────────────v────────────┐
│ Output Vector │
└─────────────────────────┘
How to Reverse Engineer Expert Counts & Architecture Specs
You can inspect the exact architectural parameters of any open-weights MoE model checkpoint using terminal CLI commands or Python without needing to load or run the full model weights.
Method 1: Instant CLI Config Inspection (Remote Repositories)
Every HuggingFace repository contains a config.json file detailing its parameters. You can inspect it directly using python one-liners:
# Extract Expert Count and Active Experts from any HuggingFace MoE Model:
python3 -c "import urllib.request, json; \
data = json.loads(urllib.request.urlopen('[https://huggingface.co/moonshotai/Kimi-K3/raw/main/config.json').read](https://huggingface.co/moonshotai/Kimi-K3/raw/main/config.json').read)()); \
print('Total Experts: ', data.get('num_experts') or data.get('n_routed_experts') or data.get('num_local_experts')); \
print('Active Experts: ', data.get('num_experts_per_tok') or data.get('num_selected_experts')); \
print('Shared Experts: ', data.get('num_shared_experts') or data.get('n_shared_experts', 0)); \
print('Total Layers: ', data.get('num_hidden_layers')); \
print('Hidden Dimension:', data.get('hidden_size'))"
---
### Universal Python Configuration Inspector
```python
from transformers import AutoConfig
# Load architecture metadata without downloading multi-terabyte model weights
model_id = "moonshotai/Kimi-K3"
config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
# Key lookup for common MoE naming conventions
total_experts = getattr(config, "num_experts",
getattr(config, "n_routed_experts",
getattr(config, "num_local_experts", None)))
active_experts = getattr(config, "num_experts_per_tok",
getattr(config, "num_selected_experts", None))
shared_experts = getattr(config, "num_shared_experts",
getattr(config, "n_shared_experts", 0))
print("=== REVERSE-ENGINEERED MODEL METADATA ===")
print(f"Architecture Type: {config.model_type}")
print(f"Total Layers: {config.num_hidden_layers}")
print(f"Total Routed Experts: {total_experts}")
print(f"Active Experts per Token: {active_experts}")
print(f"Shared Experts: {shared_experts}")
print(f"Attention Heads: {config.num_attention_heads}")
print(f"Hidden Dimension: {config.hidden_size}")
Reverse-Engineering Local .safetensors Checkpoints
If you have local model files on disk, you can inspect tensor shapes directly in Python without loading tensors into GPU VRAM:
from safetensors import safe_open
# Open a local safetensors shard
shard_path = "model-00001-of-00030.safetensors"
with safe_open(shard_path, framework="pt") as f:
tensor_keys = f.keys()
# Locate router gate layers and expert weight matrices
router_keys = [k for k in tensor_keys if "gate" in k or "router" in k]
expert_keys = [k for k in tensor_keys if "experts" in k]
print("Sample Router Weight Keys:", router_keys[:2])
print("Sample Expert Weight Keys:", expert_keys[:2])
# Inspect 3D tensor shapes: [num_experts, hidden_dim, intermediate_dim]
for key in expert_keys:
if "gate_proj.weight" in key or "w1.weight" in key or "mlp.experts" in key:
shape = f.get_slice(key).get_shape()
print(f"\nTensor Key: {key}")
print(f"Raw Tensor Shape: {shape}")
if len(shape) == 3:
print(f"--> Inferred Expert Count: {shape[0]}")
print(f"--> Expert Input/Output Dims: {shape[1]} x {shape[2]}")
break
Zero-RAM PyTorch Model Structure Printing (meta Device)
Using PyTorch's meta device, you can instantiate the complete neural network class in RAM with 0 bytes of allocated weight memory, allowing you to print the entire internal module tree:
from transformers import AutoModelForCausalLM, AutoConfig
# 1. Load config
config = AutoConfig.from_pretrained("moonshotai/Kimi-K3", trust_remote_code=True)
# 2. Instantiate empty model structure on meta device (0 MB VRAM used)
with torch.device("meta"):
model = AutoModelForCausalLM.from_config(config, trust_remote_code=True)
# 3. Print complete module hierarchy to inspect exact expert block layout
print(model)