Motif-3-Base
Motif Technologies · base checkpoint
Open in the comparison table Hugging Face ↗ Paper ↗
KV calculations use one sequence at 131,072 tokens and 16-bit cache elements. They include bounded sliding-window caches. This is retained cache payload, not peak prefill memory. The comparison table also offers sourced mixed-precision layouts, including quantization scales.
Release
- Released
2026-08-10
Source: paper · p.1 arXiv stamp
arXiv:2608.09119v1 [cs.AI] 10 Aug 2026
Card is undated; the Hugging Face repository was created 2026-08-09.
Date the weights (or, if earlier, the technical report) were first made public, YYYY-MM-DD, as dated by the publisher. A Hugging Face repository creation date is used only when no dated publisher page exists, with a note.
- License
mit
Source: model-card · README.md; YAML front matter
license: mit
License of the released weights as declared by the publisher (model card metadata or license file), e.g. apache-2.0, mit, or the name of a custom license.
- Input modalities
text
Inferred from the source; see the explanation below.
Source: model-card · README.md front matter
pipeline_tag: text-generation
No modality statement; text-generation pipeline tag and a causal LM config with no vision or audio config.
Input modalities of the released checkpoint as the publisher states them, from: text, image, video, audio, joined with ' + ' in that order ("text", "text + image + video").
Parameters
- Total params
314B parameters
Source: paper · p.1 Abstract
We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion parameters activated for each token.
Total parameter count as reported by the publisher. See params.note for whether embeddings, MTP heads or vision towers are included.
- Active params
13.2B parameters
Source: paper · p.1 Abstract
We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion parameters activated for each token.
Parameters used per token in the forward pass, as reported by the publisher.
- Active non-embedding (stated)
Not disclosed
Active parameters excluding the input embedding and the output head (unembedding). Stated by the publisher, or computed from params.active by subtracting vocab × d_model for each of the embedding and output head that params.active is stated to include (basis inferred, note with the arithmetic). Not disclosed when the scope of params.active is not stated.
- Counted total params
314.6B parameters
Calculated: embedding + output head + all experts + other weights, Motif-Technologies/Motif-3-Base@ac7c9a7 safetensors headers; reproduces stated total (backbone) and active (with embedding and output head)
Parameters of the text backbone counted from the released checkpoint's safetensors headers (embedding, output head, all experts and other weights; excluding MTP modules, vision/audio towers and n-gram lookup tables). Shown only when the counts reproduce the publisher's stated total and active parameters within 5% under some scope.
- Counted embedding + head
1.804B parameters
Calculated: input embedding + output head, Motif-Technologies/Motif-3-Base@ac7c9a7 safetensors headers
Input embedding plus output head (once if tied), counted from the checkpoint.
- Non-emb. params (counted)
312.8B parameters
Calculated: dense + routed backbone weights; excludes input embedding, output head, lookup tables, MTP and other modalities, Motif-Technologies/Motif-3-Base@ac7c9a7 safetensors headers; reproduces stated total (backbone) and active (with embedding and output head)
Language backbone weights excluding input embeddings, output head, lookup/Engram tables, MTP modules and vision/audio towers. Equals params.counted_total minus params.counted_embedding. Same validation as params.counted_total.
- Largest layer (counted)
6.125B parameters
Calculated: max(sum of dense and routed weights sharing a full layers.<index> tensor path), all experts included, Motif-Technologies/Motif-3-Base@ac7c9a7 safetensors headers
Largest group of language-backbone weights sharing a full layers.<index> tensor path, including every routed expert. Excludes lookup/Engram tables, embeddings, output head, MTP and other modalities. Hybrid checkpoints may store mixers and FFNs as separate layers. All-gather volume also depends on wrapping, sharding and dtype.
- Active non-emb. (counted)
11.17B parameters
Calculated: non-expert weights + routed experts × 8/384, excluding embedding and output head, Motif-Technologies/Motif-3-Base@ac7c9a7 safetensors headers; reproduces stated total (backbone) and active (with embedding and output head)
Active parameters per token excluding the input embedding and output head, counted from the checkpoint as all non-expert weights plus routed-expert weights × top_k / experts. Same validation as params.counted_total.
Shape
- Layers
53
Source: config · /num_hidden_layers
Number of transformer blocks in the main backbone (attention or token mixer plus FFN), excluding MTP modules.
- d_model
4,096
Source: config · /hidden_size
Residual stream width.
- Vocab
220,160
Source: config · /vocab_size
Embedding table rows (vocab_size in config, may include padding).
- Tied embeddings
No
Source: config · /tie_word_embeddings
Input embedding and output head share weights.
- Tokenizer
SuperBPE trained from scratch (stage 2 adds multi-word superword tokens), 220,160 vocab
Source: paper · p.15 §4.1 Tokenizer
We trained the Motif 3 tokenizer from scratch using the SuperBPE reference implementation
Tokenizer type and lineage as described by the publisher (e.g. byte-level BPE shared with an earlier model).
- d_model / layers
77.283
Calculated: d_model / layers
Width-to-depth ratio, d_model divided by the number of blocks.
Attention
- Token mixers
GDLA (MLA latent KV + grouped differential attention), SWA
Source: paper · p.1 Abstract
Motif 3 is built around Grouped Differential Latent Attention (GDLA), which integrates grouped differential attention with the compressed key-value representation of Multi-head Latent Attention.
Token-mixing layer types in the backbone, e.g. GQA, MLA, SWA, Gated DeltaNet, Mamba-2, DSA (DeepSeek sparse attention).
- Layer layout
3 SWA : 1 global (global first: layers 0, 4, …, 52)
Inferred from the source; see the explanation below.
Source: paper · p.19 §4.3 Model configuration
Under zero-based indexing, layers 0, 4, 8, . . . use full attention, while the three intervening layers use sliding-window attention.
Both layer types use GDLA; config use_sliding_window true, sliding_window_pattern interleave. The source gives zero-based order explicitly; modeling_motif.py uses layer_idx % sliding_window_period, with config period 4.
Token-mixer order, summarized as a pattern or layer positions. The fact's optional sequence lists each mixer in order; a ratio alone does not establish the order. Auxiliary prediction modules and standalone FFNs are excluded.
- Global-attention layers
14
Inferred from the source; see the explanation below.
Source: code · modeling_motif.py; MotifGDLAttention.__init__
elif pattern == "interleave" and layer_idx % period != 0:
Config has no per-layer list; with /sliding_window_period 4 and /num_hidden_layers 53, layers 0, 4, …, 52 are full attention.
Layers whose KV cache grows with full context (dense or sparse global attention, MLA). Excludes sliding-window and linear/recurrent layers.
- SWA : global
2.78571
Calculated: attn.local_layers / attn.global_layers
attn.local_layers divided by attn.global_layers, the sliding-window to global interleave ratio (5 for '5 SWA : 1 global').
- Sliding-window layers
39
Inferred from the source; see the explanation below.
Source: code · modeling_motif.py; MotifGDLAttention.__init__
elif pattern == "interleave" and layer_idx % period != 0:
53 layers (/num_hidden_layers) minus the 14 with layer_idx % 4 == 0 (/sliding_window_period 4).
Layers with a fixed attention window.
- Query heads
80
Source: paper · p.19 §4.3 Model configuration
GDLA uses 80 query heads and 16 KV heads
num_attention_heads 80, num_key_value_heads 16. All 80 heads (64 signal + 16 noise, g = 4) compute attention; each group of 4 signal + 1 noise heads shares one of 16 KV heads up-projected from the latent. Each noise output is subtracted from its group's signal outputs, so W_O sees 64 heads × 128.
Query heads in the global-attention layers.
- Heads × head dim
15,360
Calculated: heads × head_dim
attn.heads × attn.head_dim, the query projection width of global layers.
- QK head dim
192
Source: paper · p.19 §4.3 Model configuration
with a query-key head dimension of 192, a value-head dimension of 128, and a 64-dimensional rotary component.
config head_dim 192 = 128 NoPE + 64 RoPE.
Query/key dimension per head in global-attention layers (for MLA, nope + rope dims).
- V head dim
128
Source: paper · p.19 §4.3 Model configuration
with a query-key head dimension of 192, a value-head dimension of 128, and a 64-dimensional rotary component.
Value dimension per head in global layers (MLA v_head_dim; stated for MLA even when equal to the QK head dim).
- Window
128 tokens
Source: paper · p.19 §4.3 Model configuration
the remaining layers use sliding-window attention with a window of 128 tokens.
config sliding_window 128; HF code passes sliding_window + 1 to flash attention.
Sliding-window size of the local layers.
- SWA KV heads
16
Source: config · /num_key_value_heads
KV heads in sliding-window layers (stated even when equal to attn.kv_heads).
- SWA query heads
80
Source: config · /num_attention_heads
MotifGDLAttention reads the same head counts and dims for SWA and full layers; only the window and rotary embedding differ.
Query heads in sliding-window layers (stated even when equal to attn.heads).
- SWA QK head dim
192
Source: config · /head_dim
QK 192, V 128 (/v_head_dim), as in global layers.
Query/key head dimension in sliding-window layers. Value head dimensions may differ; see attn.local_v_head_dim.
- MLA KV latent
512
Source: paper · p.19 §4.3 Model configuration
The query and KV low-rank dimensions are 1,024 and 512, respectively.
config kv_lora_rank 512; v_head_dim 128.
Width of the compressed KV latent that MLA caches per token per layer.
- MLA Q latent
1,024
Source: paper · p.19 §4.3 Model configuration
The query and KV low-rank dimensions are 1,024 and 512, respectively.
Width of the low-rank query projection in MLA; 0 when queries use a full-rank projection.
- Decoupled RoPE dim
64
Source: paper · p.19 §4.3 Model configuration
with a query-key head dimension of 192, a value-head dimension of 128, and a 64-dimensional rotary component.
config qk_rope_head_dim 64.
Width of the shared positional key that MLA caches next to the latent.
- Positional encoding
partial RoPE (decoupled 64-dim RoPE key shared across heads + 128 NoPE dims); YaRN on global layers only
Source: paper · p.19 §4.3 Batch size and context-length schedule
The full-attention layers use a DeepSeek-YaRN extension from the original 4K context length to 256K, with an extension factor of 64, while the sliding-window layers retain local RoPE without long-range extension.
Positional scheme in global layers: RoPE, partial RoPE, NoPE, or mixed per layer type.
- RoPE base
10,000
Source: config · /rope_scaling/rope_theta
Top-level rope_theta is also 10000.
RoPE base frequency of the global layers at release.
- SWA RoPE base
10,000
Source: config · /swa_rope_theta
SWA layers build their own rotary embedding with rope_scaling None (no YaRN).
RoPE base frequency of the sliding-window layers (stated even when equal to attn.rope_theta).
- QK norm
No
Source: code · modeling_motif.py; MotifGDLAttention.forward
kv_proj = self.wkv_b(self.kv_norm(kv_latent))
RMSNorm is applied only to the Q and KV latents before up-projection, not per head.
Queries and keys are normalized per head before the dot product.
- Output gate
element-wise sigmoid gate from the normalized query latent, on the differential output before W_O
Source: paper · p.5 §2.2.3
Before the final output projection, the differential representation is modulated by an element-wise sigmoid gate computed from the normalized query latent
config elementwise_attn_output_gate true; diff_v2 true selects the input-dependent λ variant (the only one in HF code).
Gate applied to the attention output before the output projection, with its placement.
- Attention scaling
global layers: 1/sqrt(192) × (0.1·ln 64 + 1)² (YaRN mscale); SWA layers: 1/sqrt(192)
Source: code · modeling_motif.py; MotifGDLAttention.__init__
self.scaling = self.scaling * mscale_val * mscale_val
/rope_factor 64, /mscale 1. Differential output: attn1 - sigmoid(lambda_proj(x)) * attn2, λ per token and signal head (report Eq. 12–13).
Query-key scaling when it is not 1/sqrt(head_dim), e.g. query_pre_attn_scalar, length-dependent (SSMax-style) scaling, or value scaling.
- QKV bias
No
Source: code · modeling_motif.py; MotifGDLAttention.__init__
self.wq_a = nn.Linear(self.hidden_size, self.q_lora_rank, bias=False)
wq_b, wkv_a, wkv_b, lambda_proj and wo are also bias=False.
Query/key/value projections have bias terms.
Feed-forward
- Activation
Expert-Specific PolyNorm (gated: PolyNorm(gate) × up; per-expert sigmoid-bounded coefficients)
Source: paper · p.6 §2.4
Motif 3 instead replaces the SiLU gate activation with Expert-Specific PolyNorm, whose polynomial coefficients and bias are learned independently for each expert.
config hidden_act poly_norm. PolyNorm(z) = Σ_{n=1..3} a_n·z^n/RMS(z^n) + b; dense layers and the shared expert use a single PolyNorm. Code multiplies the activation output by /polynorm_output_scale 0.5.
FFN activation (SwiGLU, GeGLU, ...).
- Dense FFN dim
12,288
Source: paper · p.18 §4.3 Model configuration
The first two layers use dense feed-forward networks with an intermediate dimension of 12,288
Intermediate width of the dense FFN layers that are actually executed.
- Activation clamp
none (soft FFN-output magnitude penalty instead)
Source: paper · p.16 §4.2 FFN magnitude regularization
We avoid this forward-pass intervention because activation outliers are not necessarily meaningless numerical artifacts.
Routed-expert PolyNorm bias is clipped to [-0.5, 0.5] (Eq. 24, /polynorm_bias_clamp); the HF code's ±1e6 /hidden_clamp is a numerical guard.
Clamping of GLU gate / up activations (e.g. SwiGLU limit 7), with the layers it applies to.
- FFN bias
No
Source: code · modeling_motif.py; MotifMLP.__init__
self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
Up and down projections also bias=False; the PolyNorm activation itself has a learned scalar bias.
FFN projections have bias terms.
- Dense FFN / d_model
3
Calculated: ffn.dim / d_model
ffn.dim divided by shape.d_model.
Mixture of experts
- Routed experts
384
Source: paper · p.18 §4.3 Model configuration
Each MoE layer contains 384 routed experts and one shared expert.
config num_experts 384.
Routed experts per MoE layer.
- Experts per token
8
Source: paper · p.18 §4.3 Model configuration
The router uses normalized sigmoid scores and selects eight routed experts per token.
config experts_top_k 8.
Routed experts selected per token.
- Expert dim
1,280
Source: paper · p.18 §4.3 Model configuration
Each routed expert has an intermediate dimension of 1,280.
Intermediate width of one routed expert.
- Leading dense layers
2
Source: paper · p.18 §4.3 Model configuration
The first two layers use dense feed-forward networks with an intermediate dimension of 12,288, while the remaining 51 layers use sparse MoE feed-forward networks.
config n_dense_first_layers 2, interleave_moe_layer_step 1.
Number of initial blocks that use a dense FFN instead of MoE.
- Router score
normalized sigmoid
Source: paper · p.18 §4.3 Model configuration
The router uses normalized sigmoid scores and selects eight routed experts per token.
Routing computed in FP32; config score_func sigmoid, route_norm true.
Gating function over expert logits (softmax, sigmoid).
- Load balancing
aux-loss-free bias (coefficient 1e-3) + sequence-wise aux loss 1e-4 (2.4e-4 layer 2, 2.0e-4 layers 3–5) + cosine-decayed Gaussian router-logit noise
Source: paper · p.19 §4.3 Hyperparameters
The sequence-wise auxiliary load-balancing loss is summed over tokens and uses a default weight of 1 × 10−4 . We increase this weight to 2.4 × 10−4 for layer 2 and to 2.0 × 10−4 for layers 3–5.
Bias coefficient 1 × 10−3 in the same paragraph; router noise in §4.2. Collapsed routers were replaced with an adjacent layer's router and biases.
Load-balancing method, with coefficients where disclosed.
- Routed scaling factor
2
Source: config · /route_scale
Multiplier applied to the combined routed-expert output (routed_scaling_factor).
- Renormalize top-k
Yes
Source: config · /route_norm
Selected expert weights are renormalized to sum to 1.
- Expert dim / d_model
0.3125
Calculated: moe.expert_dim / d_model
moe.expert_dim divided by shape.d_model.
- Sparsity (experts / top-k)
48
Calculated: experts / top_k
Routed experts divided by experts selected per token.
- Active / total
0.0420382
Calculated: params.active / params.total
params.active divided by params.total.
Multi-token prediction
- MTP depth
1
Source: paper · p.7 Table 1
MTP head 1 layer for self-speculative decoding
One DeepSeek-V3-style MTP module (§2.5); the index holds only mtp_layers.0.
Additional future tokens predicted during pretraining (0 when the model was trained without MTP).
- MTP modules
1
Source: config · model.safetensors.index.json; weight_map keys
model.mtp_layers.0.input_proj.weight
Only mtp_layers.0 in the index; training code builds one module (multi_token_preds=1).
MTP modules with distinct weights. 1 with mtp.depth > 1 means one module is applied recursively.
- MTP module
concat(h, RMSNorm(next-token embedding)) → linear 2d→d → GDLA block (SWA, no mHC) with dense PolyNorm FFN → RMSNorm → shared output head
Source: code · llm_training/motif/model/model.py; compute_mtp_loss, Motif.__init__
h_out = model.mtp_layer[str(token_idx)](torch.cat([h_in, next_h], dim=-1))
Code comment: 'Force dense FFN, SWA, and no MHC for MTP blocks'.
Contents of one MTP module (mixer and FFN type, norms, how the previous hidden state and next-token embedding are combined).
- MTP FFN
dense (12288)
Source: code · llm_training/motif/model/model.py; Motif.__init__
interleave_moe_layer_step=0: always use dense FeedForward (no MoE)
Dense FFN width is intermediate_dim 12288; index has mlp.gate_proj/up_proj/down_proj, no experts.
FFN in the MTP module: 'dense' or 'MoE', with its width when it differs from the backbone.
- MTP loss weight
0.2
Source: paper · p.19 §4.3 Hyperparameters
The MTP auxiliary objective uses a loss weight of 0.2.
Training code multiplies by mtp_depth_decay 0.95 and divides the summed loss by 1 + weight.
Weight of the MTP loss, including its schedule.
- MTP weights released
Yes
Source: config · /weight_map | has mtp_layers.
The card says the MTP head is not shipped, but shard model-00104-of-00155 contains mtp_layers.0 (GDLA + dense PolyNorm FFN, 19 tensors); the HF modeling code does not load it.
The released checkpoint contains the MTP module weights.
Normalization and stability
- Norm
RMSNorm
Source: paper · p.2 §2.1
We apply RMSNorm [74] before each sublayer
Normalization layer type in the blocks.
- Norm placement
pre
Source: paper · p.2 §2.1
We apply RMSNorm [74] before each sublayer and use modified manifold-constrained hyper-connections (mHC) [68] in place of conventional residual additions.
Pre-norm, post-norm, or sandwich (pre and post).
- Norm epsilon
1e-05
Source: config · /rms_norm_eps
Epsilon of the block normalization layers.
- Residual stream
modified mHC: 4 streams, Sinkhorn-Knopp doubly stochastic residual mixing, post-mapping multiplier annealed 2 → 1 during pretraining
Source: paper · p.5–6 §2.3
We use an expansion rate of n = 4, corresponding to four parallel residual streams.
config mhc_enabled true, mhc_expansion_rate 4. Mappings computed in FP32 from an RMSNorm of the flattened streams; 20 Sinkhorn iterations (/mhc_sinkhorn_iters); embedding copied to all 4 streams and streams averaged before the final norm (MotifModel.forward).
Residual design when it is not a plain sum, e.g. hyper-connections with n streams, gated residuals, depth-scaled residuals, FP32 residual accumulation.
Context length
- Pretraining context
4.096K tokens
Source: paper · p.19 §4.3 Batch size and context-length schedule
Pretraining begins with sequences of up to 4K tokens.
Sequence length of the main pretraining stage.
- Trained context
262.1K tokens
Source: model-card · README.md; 2. Model Summary
Context Length 262,144 (256K)
Report §4.3: a dedicated long-context stage with sequences of up to 256K tokens.
Longest sequence length trained on after context extension.
- Advertised context
262.1K tokens
Source: model-card · README.md; Key Features
Native 256K context (262,144 tokens), trained with window-aware context parallelism.
config max_position_embeddings 262144.
Longest context the publisher says the model supports, possibly beyond trained length via RoPE scaling.
- Extension method
4K → 32K during the LR decay phase → 256K stage; DeepSeek-YaRN factor 64 on global layers only
Source: paper · p.19 §4.3 Batch size and context-length schedule
During the learning-rate decay phase, we transition to a maximum sequence length of 32K tokens, followed by a dedicated long-context stage with sequences of up to 256K tokens.
Config rope_scaling: rope_type yarn, factor 64, original_max_position_embeddings 4096 (original_seq_len), beta_fast 32, beta_slow 1, theta 10000.
How context was extended (stages, RoPE base change, YaRN factor).
Pretraining recipe
- Pretraining tokens
12.5T tokens
Source: paper · p.14 §4.1 Data
Motif 3 is pretrained on approximately 12.5 trillion tokens, measured with the Motif tokenizer
Tokens seen in pretraining including annealing and context extension, as reported.
- Optimizer
Muon + AdamW (AdamW for embeddings, output head and vector parameters)
Source: paper · p.10 §3.3
Embeddings, the output projection, and vector-shaped parameters are instead optimized with AdamW.
Each routed expert is orthogonalized as its own matrix; Newton–Schulz in BF16.
Optimizer used for the bulk of parameters.
- Peak LR
0.0003
Source: paper · p.19 §4.3 Hyperparameters
Training starts with a learning rate of 3 × 10−4 and follows a warmup-stable-decay (WSD) learning-rate schedule.
Peak learning rate of the main pretraining stage.
- Final LR
3e-05
Source: paper · p.19 §4.3 Hyperparameters
The subsequent 256K long-context stage continues from this checkpoint and further reduces the learning rate to 3 × 10−5 .
Learning rate at the end of the decay.
- LR schedule
WSD: stable at 3e-4, decay to 7.5e-5 during the 4K → 32K transition, then 256K stage decays to 3e-5
Source: paper · p.19 §4.3 Hyperparameters
The stable phase performs the majority of pretraining at the maximum learning rate, after which the decay phase reduces the learning rate to 7.5 × 10−5 while transitioning from 4K to 32K sequences.
Learning-rate schedule shape with phase boundaries.
- Warmup
Not disclosed
Warmup length in steps or tokens.
- Batch size
up to 75M tokens
Source: paper · p.19 §4.3 Batch size and context-length schedule
The global training batch contains up to 75 million tokens.
Global batch size, including any batch-size ramp: sequences × sequence length (token total), or the token total alone when only that is stated.
- Batch (tokens)
75M tokens
Source: paper · p.19 §4.3 Batch size and context-length schedule
The global training batch contains up to 75 million tokens.
Stated as an upper bound; no ramp described.
Global batch size in tokens during the main pretraining stage, after any ramp (sequences × sequence length when stated in sequences, marked inferred).
- Weight decay
Not disclosed
Decoupled weight decay coefficient.
- Adam betas
Not disclosed
(beta1, beta2) of the Adam-family optimizer or its AdamW fallback.
- Grad clip
Not disclosed
Global gradient-norm clipping threshold.
- Training precision
MXFP8 for MoE expert grouped GEMMs and EP dispatch; BF16 elsewhere; FP32 master weights, gradients, router logits and Muon states; BF16 gradient communication with FP32 reduction
Source: paper · p.9 §3.2
More specifically, we apply MXFP8 [59] to the MoE experts
Matmul precision of pretraining (BF16, FP8 recipe, NVFP4 recipe) with the scaling granularity where disclosed. Never inferred from the released checkpoint dtype.
- Stability measures
QK-Clip (τ=100, 200 in the 256K stage; every 10 steps; key-side share 1/(1+√G)); FFN-output magnitude penalty (weight 2e-4, threshold 128, 1,024 on the final layer); mHC post-multiplier annealing; FP32 routing; router replacement on collapse
Source: paper · p.11 §3.3 QK-Clip; p.19 §4.3
We perform monitoring and clipping every 10 steps, using τ = 100 during main pre-training and τ = 200 during the 256K-token long-context stage.
Penalty values: 'The FFN magnitude regularizer uses a loss weight of 2.0 × 10−4 and a threshold of 128' (p.19).
Interventions used during pretraining to keep it stable (QK-clip, z-loss, logit caps, spike handling), with coefficients.
- Document masking
document-masked packing
Source: paper · p.13 §3.5 Attention-Workload-Aware Re-Scheduling
Document-masked packing concatenates multiple documents into a fixed-length sequence while preventing cross-document attention.
Stated in the long-context training section; not stated separately for the 4K stage.
Whether attention crosses document boundaries within a packed sequence.
- Data mix
~70% NVIDIA Nemotron pretraining collection (some components filtered), most of the rest in-house web, STEM and synthetic; dedicated Korean, multilingual, legal and financial data; dynamic mixture 19 → 57 datasets; reasoning data <5%
Source: paper · p.14 §4.1 Data
The retained Nemotron data account for approximately 70% of the full Motif 3 pretraining corpus.
Long-context stage uses ~5% of the corpus reconstructed as long-context data (§4.3).
Pretraining data composition as disclosed (domain shares, synthetic fraction, repetition).
- Data cutoff
March 2026
Source: paper · p.14 §4.1 Data
The latest data included in the corpus were collected through March 2026, which we use as the model’s knowledge cutoff.
Pretraining data cutoff date.
- Hardware
B200 GPUs (count not stated); EP 8, DP-shard 8, no PP
Source: paper · p.7 §3.1
With expert and parameter sharding, optimizer-state CPU offloading, and the memory capacity of B200 GPUs, our 314B-parameter model fits within the available device memory
Accelerators used for pretraining, count and duration where stated.
- Accelerators
B200
Source: paper · p.7 §3.1
With expert and parameter sharding, optimizer-state CPU offloading, and the memory capacity of B200 GPUs
Count not stated.
Accelerator count and type for the main pretraining run, as the count with thousands separators and the chip's short name ("2,048 H800", "2,048 GH200"). A stated range or bound stays as stated ("2,048–4,096 GH200").
- Parallelism
EP8 · HSDP (DP-shard 8 · DP-replicate)
Source: paper · p.7 §3.1
Outside the expert-parallel dimension, we use a sharded data-parallel (DP-shard) degree of 8 and assign the remaining ranks to the replicated data-parallel (DP-replicate) dimension.
Same paragraph: EP = 8 within one eight-GPU node, FSDP sharding, no pipeline parallelism.
Parallelism layout of the main pretraining run as stated, as degree-annotated terms in the order TP, PP, VPP, EP, CP, DP ("PP16 · EP64 · ZeRO-1 DP"); a term without a stated degree has none ("FSDP").
- Stages
main
- Seq len
- 4096
- Lr
- 0.0003
decay
- Seq len
- 32768
Source: paper · p.19 §4.3
During the learning-rate decay phase, we transition to a maximum sequence length of 32K tokens
LR decays to 7.5e-5 while moving from 4K to 32K sequences.
long-context
- Seq len
- 262144
Source: paper · p.19 §4.3
followed by a dedicated long-context stage with sequences of up to 256K tokens.
LR decays to 3e-5; ~5% of the corpus rebuilt as long-context data; EP 8 + CP 8.
Source: paper · p.19 §4.3
Ordered pretraining stages. Each item: name, tokens, seq_len, lr, batch_tokens, batch_sequences, steps, note, quote, and source when it differs from the fact's source.
Scaling & HP transfer
- Tokens / param
39.8089
Calculated: train.tokens / params.total
train.tokens divided by params.total.
- Tokens / active param (non-embedding)
1,118.61
Calculated: train.tokens / params.counted_active_non_embedding
train.tokens divided by params.counted_active_non_embedding, or by the stated params.active_non_embedding when there is no validated checkpoint count.
KV cache
- KV bytes / token
16,172 bytes
Calculated: kv.bytes_at_context / 131072 tokens; uniform BF16, payload only; quantization overhead excluded. Global KV: 14 × (576 × 16/8) B/entry; SWA KV: 39 × (576 × 16/8) B/entry
Retained KV cache divided by context length, including bounded SWA caches and indexer keys. Exports use uniform BF16 at 131,072 tokens. The website can select uniform payload precision (no quantization overhead) or sourced component formats (including scales and documented padding). This is cache payload, not peak prefill memory or persistent prefix storage.
- Global KV bytes / token
16,128 bytes
Calculated: 16128 B / 1 tokens (Global KV); uniform BF16, payload only; quantization overhead excluded. Global KV: 14 × (576 × 16/8) B/entry; SWA KV: 39 × (576 × 16/8) B/entry
Long-context global KV bytes per token, including indexer keys; excludes bounded SWA caches. Compression rates and cross-layer cache reuse are included. Exports use uniform BF16; the website's storage selector also applies here. This is the field to compare with global-KV figures in reports, using the matching storage recipe.
- KV bytes / token / B active
1,225 bytes
Calculated: kv.bytes_per_token (131072 tokens) / (params.active / 1e9); uniform BF16, payload only; quantization overhead excluded. Global KV: 14 × (576 × 16/8) B/entry; SWA KV: 39 × (576 × 16/8) B/entry
kv.bytes_per_token divided by active parameters in billions (131,072 tokens in exports).
- KV at context / B active
160,581,818 bytes
Calculated: kv.bytes_at_context (131072 tokens) / (params.active / 1e9); uniform BF16, payload only; quantization overhead excluded. Global KV: 14 × (576 × 16/8) B/entry; SWA KV: 39 × (576 × 16/8) B/entry
kv.bytes_at_context divided by active parameters in billions (131,072 tokens in exports).
- KV at context
2,119,680,000 bytes
Calculated: 16128 B × 131072 (Global KV) + 44928 B × min(131072, 128) (SWA KV); uniform BF16, payload only; quantization overhead excluded. Global KV: 14 × (576 × 16/8) B/entry; SWA KV: 39 × (576 × 16/8) B/entry
Retained KV cache for one sequence at the selected context, including SWA windows and indexer keys. Exports use uniform BF16 at 131,072 tokens. Excludes recurrent state, compressor tail state, draft/MTP caches, temporary prefill tensors, allocator metadata and persistent prefix snapshots. Does not estimate peak prefill or total serving memory.
Evaluations
- AA Intelligence Index
34
Source: leaderboard · Artificial Analysis model page /models/motif-3, Wayback snapshot 2026-10-19
Motif 3 scores 34 on the Artificial Analysis Intelligence Index
Score of Motif-3, post-trained from this base. Variant: Motif 3. Index v4.3.2.
Artificial Analysis Intelligence Index v4.3.2 of the post-trained model, from a dated Wayback snapshot of its Artificial Analysis page. The note names the variant (reasoning setting).