Exploring AI

LLM Model Summary, Document Format

This specification sheet describes a massive, multimodal Artificial Intelligence (AI) foundation model (with roughly 2.8 trillion total parameters). Below is an explanation of what each component means in plain English:

1. Core Model Scale & Architecture

  • Architecture: Mixture-of-Experts (MoE)

    • What it means: Instead of running the entire neural network for every single word or task, the model is split into specialized sub-networks called "experts." For any given input, a routing system sends the task only to the most relevant experts.

  • Total Parameters: 2.8T (Trillion)

    • What it means: The model has 2,800,000,000,000 learnable "weights" or connections in total. This total size represents the sum of all knowledge stored across all expert sub-networks.

  • Activated Parameters: 104B (Billion)

    • What it means: Despite having 2.8T total parameters, only 104 billion parameters are actually powered on (activated) to process any single token (word/character). This makes compute latency significantly faster and drastically cuts memory demands during execution.

2. Layer Structure & Attention Mechanisms

  • Number of Layers: 93

    • What it means: Information flows through 93 sequential processing blocks (layers) from input to output.

  • Number of Dense Layers: 1

    • What it means: Out of 93 layers, 92 layers use the Mixture-of-Experts (MoE) routing system, while 1 layer is "dense" (all parameters are activated for every token).

  • Attention-Layer Composition: 69 KDA + 24 Gated MLA

    • What it means: This describes the types of attention layers used across the 93 total layers:
      • KDA (Key-Difference Attention / Kernelized Attention variants): Highly efficient mechanisms designed to process long context windows quickly.
      • Gated MLA (Multi-Head Latent Attention): Compresses key/value representations to minimize memory usage (KV cache) while retaining deep contextual reasoning.

  • Attention Hidden Dimension: 7168

    • What it means: The width of the vector representation passed through the attention layers. Higher numbers mean the model can represent richer concepts per token.

  • Number of Attention Heads: 96

    • What it means: Each attention layer splits its work into 96 parallel streams, allowing the model to simultaneously look for 96 different types of relationships between words in a sentence.

3. Mixture-of-Experts (MoE) Routing Details

  • Latent MoE Dimension: 3584

    • What it means: The size of the hidden feature space fed into the expert router.

  • MoE Hidden Dimension (per Expert): 3072

    • What it means: The internal size/capacity of each individual expert network.

  • Number of Experts: 896

    • What it means: The model is partitioned into 896 distinct expert modules across its MoE layers.

  • Selected Experts per Token: 16

    • What it means: For every individual word/token processed, the router picks the top 16 best-suited experts out of 896 to handle it.

  • Number of Shared Experts: 2

    • What it means: Two experts are always activated for every single token, regardless of routing decisions, ensuring base-level knowledge remains consistent across all tasks.

4. Input Capabilities & Context Memory

  • Vocabulary Size: 160K (160,000)

    • What it means: The model understands 160,000 unique sub-words, characters, and symbols across multiple languages and code types.

  • Context Length: 1,048,576 (1M+ Tokens)

    • What it means: The model can retain and analyze roughly 1 million tokens in a single session—equivalent to roughly 750,000 words, several long books, or massive codebases at once.

  • Attention Mechanism: KDA & Gated MLA

    • What it means: Specialized attention designs that make processing 1 million tokens computationally feasible without crashing hardware memory.

  • Activation Function: SiTU-GLU

    • What it means: A mathematical gating formula used inside neuron layers to determine whether a signal should pass forward. Gated Linear Units (GLUs) generally improve stability and learning efficiency in large models.

5. Vision & Multimodal Capabilities

  • Vision Encoder: MoonViT-V2

    • What it means: A specialized visual pre-processor (a Vision Transformer) that converts images into visual vectors the text model can understand.

  • Parameters of Vision Encoder: 401M (Million)

    • What it means: The visual system has 401 million parameters dedicated solely to analyzing image features before passing them to the main language model.

  • Modality: Text, Image

    • What it means: The model natively handles both text inputs/outputs and visual inputs (images, diagrams, screenshots).

6. Optimization & Hardware Efficiency

  • Quantization: MXFP4 weights / MXFP8 activations (Quantization-Aware Training)
    • What it means:
      • MXFP4 Weights: Model weights are stored using ultra-compressed 4-bit floating point numbers (Microscaling Format), drastically cutting down VRAM footprint.
      • MXFP8 Activations: Signals during processing run in 8-bit precision.
      • Quantization-Aware Training: The model was trained from scratch knowing it would be compressed, preventing quality loss that usually happens when shrinking standard models post-training.

Detailed Guide to AI Model Specifications & Architectures

Welcome! Below is a comprehensive, component-by-component deep dive into the technical specifications of a modern frontier AI model. Each section explains what the component is, what alternative types exist, and how it works using real-world analogies.

1. Core Model Architecture & Scale

A. Architecture (Structure Type)

  • Value in Spec: Mixture-of-Experts (MoE)
  • What it means: The overall blueprint used to construct the neural network.
  • Alternative Types:
    • Dense Models: Every parameter is used for every single token (e.g., standard LLaMA 3, GPT-3). High quality per parameter, but computationally expensive.
    • Dense with Linear/Recurrent Attention: Models like Mamba or RWKV that replace traditional attention with state-space models for faster inference.
    • Hybrid Architectures: Combines MoE with linear state-space layers (e.g., Jamba).
  • Deep Dive: What is an "Expert" in MoE?
    • An expert is simply a sub-network (a feed-forward layer) dedicated to processing specific patterns. Rather than passing an input through one giant block, a router network evaluates the incoming word/token and decides which 16 sub-networks out of 896 are best suited to analyze it.
  • Analogy:
    • Dense Model: A single genius doctor who reads every medical chart, conducts every test, and performs every surgery themselves.
    • MoE Model: A massive, highly specialized hospital system. When a patient arrives, a triage nurse (the router) sends them directly to 16 specific specialists (the experts)—such as a cardiologist, a radiologist, and a neurologist—rather than calling every single doctor in the building.

B. Total Parameters vs. Activated Parameters

  • Values in Spec: Total = 2.8T (Trillion), Activated = 104B (Billion)
  • What it means: Total parameters represent the total capacity and stored knowledge of the entire network. Activated parameters are the subset actually turned on and calculated during a single step.
  • Alternative Configurations:
    • 100% Activation (Dense): Total parameters = Activated parameters (e.g., 70B total / 70B activated).
    • Low-Sparsity MoE: Small number of experts, high activation ratio (e.g., Mixtral 8x7B activates 2 out of 8 experts = ~13B out of 47B total).
    • High-Sparsity / Granular MoE: Thousands of tiny experts, low activation ratio (like this 2.8T model, where under 4% of parameters run per token).
  • Analogy:
    • Total Parameters: A library containing 2.8 million books.
    • Activated Parameters: The 104 books you pull off the shelves to research a specific query. You have access to the entire library's knowledge, but you don't read every book at the same time.

2. Layer Structure & Attention Mechanisms

A. Number of Layers & Dense vs. Sparse Layers

  • Values in Spec: 93 Total Layers (1 Dense + 92 MoE)
  • What it means: Neural networks refine information step-by-step through sequential layers.
  • Alternative Layer Structures:
    • Pure Dense: All 93 layers activate 100% of their parameters.
    • Interleaved / Patterned MoE: Every 2nd or 4th layer is MoE, while the rest are dense.
    • Fully Sparse MoE: Every single layer uses expert routing.
  • Analogy: Imagine a 93-stage assembly line for custom cars.
    • Dense Layer (Layer 1): The universal chassis inspector where every car must go through the exact same evaluation.
    • MoE Layers (Layers 2–93): Specialized customization stations where cars are routed only to the paint bays, engine tuners, or interior upholsterers needed for that specific vehicle type.

B. Attention Mechanism (KDA & Gated MLA)

  • Values in Spec: 69 KDA + 24 Gated MLA
  • What it means: Attention allows the model to connect related words across long distances (e.g., matching "she" to "Alice" 5 pages earlier).
  • Alternative Types of Attention:
    • Standard Multi-Head Attention (MHA): Traditional attention mechanism. Highly accurate, but uses massive amounts of GPU memory for long documents.
    • Grouped-Query Attention (GQA): Grouping key/value heads to save memory (used in LLaMA 2/3).
    • Multi-Head Latent Attention (MLA): Compresses key-value matrices into a latent vector, drastically reducing RAM/VRAM consumption during generation.
    • Kernelized / Linear Attention (KDA): Approximates traditional attention mathematically to allow extremely fast context processing over millions of words.
  • Analogy:
    • Standard Attention (MHA): Taking verbatim, word-for-word handwritten notes on every single line of a textbook while reading.
    • Gated MLA / KDA: Creating a compressed, shorthand index of key concepts in the margins so you can scan a 1,000-page book in seconds without running out of desk space.

C. Attention Hidden Dimension & Attention Heads

  • Values in Spec: Hidden Dimension = 7168, Heads = 96
  • What it means: The hidden dimension represents the depth of representation per token. Attention heads split this dimension into parallel "lens views."
  • Alternative Configurations:
    • Narrow / Shallow: Dimension 2048, 16 Heads (Common in smaller 3B–7B models).
    • Balanced: Dimension 4096, 32 Heads (Common in medium 13B–70B models).
    • Ultra-Wide: Dimension 7168+, 96+ Heads (Used in frontier-class flagship models).
  • Analogy: Imagine analyzing a legal contract.
    • Hidden Dimension (7168): The total depth of detail written down about every sentence.
    • 96 Attention Heads: 96 distinct inspectors reading the same contract simultaneously—one looks strictly for dates, another for dollar figures, another for grammar, another for legal liabilities, and so on.

3. MoE Routing & Expert Architecture

A. Number of Experts & Selection Dynamics

  • Values in Spec: 896 Experts, 16 Selected per Token, 2 Shared Experts
  • What it means: Defines how fine-grained the expert distribution is and how they are selected.
  • Alternative Routing Strategies:
    • Coarse-Grained (e.g., Top-2 out of 8): Few, large experts. Easy to train, but less specialized.
    • Fine-Grained (e.g., Top-16 out of 896): Many small, specialized experts. Allows hyper-precise task assignment.
    • Shared Experts: Keeping dedicated experts active for all tokens to capture universal features like basic syntax or logic.
  • Analogy:
    • Coarse MoE: Asking 2 general handymen to fix a house problem.
    • Fine-Grained MoE (This Model): Calling a specialty crew of 16 precision contractors (electrician, plumber, drywall installer, painter, locksmith, etc.) out of a labor force of 896.
    • Shared Experts: Two master supervisors who stay on-site non-stop to oversee safety and general quality, regardless of which contractors are working.

4. Context Window & Tokenization

A. Vocabulary Size & Activation Functions

  • Values in Spec: Vocab Size = 160K, Activation = SiTU-GLU
  • What it means:
    • Vocabulary Size: How many distinct sub-word pieces the tokenizer recognizes.
    • Activation Function: The mathematical gate determining neuron firing intensity.
  • Alternative Types:
    • Vocab Alternatives: Small (32K - early GPT models), Medium (128K - LLaMA 3), Ultra-large (256K+ - multilingual models).
    • Activation Alternatives: ReLU (older, basic step-function), GELU (smooth probabilistic step), SwiGLU / SiTU-GLU (advanced gated units that improve mathematical reasoning and overall stability).
  • Analogy:
    • Vocab Size:
      • 32K Vocab: Knowing basic root words and having to spell out rare words letter-by-letter.
      • 160K Vocab: Possessing an expansive dictionary that includes medical terms, legal jargon, Chinese logograms, and programming syntax as single instant-lookup entries.
    • Activation Function: The dimmer switch on a light bulb vs. a simple on/off toggle. SiTU-GLU acts as a smart dimmer switch that fine-tunes how much energy passes through based on complex context signals.

B. Context Length

  • Value in Spec: 1,048,576 Tokens (~1 Million)
  • What it means: The maximum amount of text/data the model can hold in its active memory at one time.
  • Alternative Context Capacities:
    • Standard: 4K–8K tokens (Short conversations, single articles).
    • Extended: 32K–128K tokens (Full documents, small code repos).
    • Ultra-Long / Million-Scale: 1M–2M+ tokens (Entire libraries, massive software codebases, hours of audio/video transcriptions).
  • Analogy:
    • 4K Context: A sticky note on your desk.
    • 128K Context: A notebook filled with project notes.
    • 1M+ Context: A massive whiteboard wall that spans an entire room, where you can pin whole textbooks, full software systems, and months of logs all at once without losing sight of any detail.

5. Vision Encoder & Multimodality

A. Vision Encoder Architecture

  • Value in Spec: MoonViT-V2 (401M Parameters)
  • What it means: A dedicated sub-model that takes raw pixels and converts them into semantic embedding vectors that the main language model can process alongside text.
  • Alternative Vision Architectures:
    • CLIP-style Encoders: Standard ViT models trained on contrastive image-text pairs (e.g., OpenAI CLIP).
    • Flamingo / Cross-Attention Adapters: Interleaving vision capabilities directly into frozen language models.
    • Native Native Multimodal (e.g., Gemini style): Unified tokenizers handling images, audio, and text in a single stream from the beginning.
  • Analogy: Imagine a court reporter working with a blind judge. The Vision Encoder (MoonViT) acts as an expert court investigator who views photos, diagrams, and maps, then translates those visual details into a detailed written brief for the main language model to analyze.

6. Quantization & Precision Formats

A. Quantization Strategy

  • Values in Spec: MXFP4 weights / MXFP8 activations (Quantization-Aware Training)
  • What it means: Shrinking the numerical precision of model numbers to reduce GPU VRAM requirements and accelerate processing speed.
  • Alternative Formats:
    • FP16 / BF16 (Uncompressed): 16-bit floating point. Max precision, huge memory footprint.
    • Post-Training Quantization (PTQ): Training in 16-bit, then compressing down to 4-bit/8-bit afterward. Faster, but can degrade model intelligence.
    • Microscaling Formats (MXFP4 / MXFP8): Uses sub-vector scaling blocks to preserve numerical precision at low bit counts.
    • Quantization-Aware Training (QAT): Training the model while restricting it to lower bit settings, teaching it to adapt during training rather than forcing compression later.
  • Analogy:
    • FP16: Writing down every measurement on a blueprint to 10 decimal places.
    • Post-Training Quantization (PTQ): Taking a high-precision blueprint and rounding all numbers to the nearest whole foot after construction has started—risking structural mistakes.
    • Quantization-Aware Training (QAT): Designing and constructing the building using modular standard bricks from day one. You use far less paper and storage, but everything fits together perfectly without losing stability.

Filters

No filters available for this view.

Reset All