LongCat-2.0
Meituan · reasoning checkpoint
Post-trained checkpoint (thinking and non-thinking modes); no base checkpoint released.
Open in the comparison table Hugging Face ↗ Paper ↗
KV calculations use one sequence at 131,072 tokens and 16-bit cache elements. They include bounded sliding-window caches. This is retained cache payload, not peak prefill memory. The comparison table also offers sourced mixed-precision layouts, including quantization scales.
Release
- Released
2026-06-30
Source: blog · blog bundle; hero date of 'Introducing LongCat-2.0'
date:`2026-06-30`
Publisher-dated announcement; the weight files were uploaded to Hugging Face on 2026-07-05 (repository commit history).
Date the weights (or, if earlier, the technical report) were first made public, YYYY-MM-DD, as dated by the publisher. A Hugging Face repository creation date is used only when no dated publisher page exists, with a note.
- License
mit
Source: model-card · README.md; front matter
license: mit
License of the released weights as declared by the publisher (model card metadata or license file), e.g. apache-2.0, mit, or the name of a custom license.
- Input modalities
text
Inferred from the source; see the explanation below.
Source: model-card · README.md front matter
pipeline_tag: text-generation
No modality statement; the card's pipeline tag is text-generation and config.json (LongcatCausalLM) has no vision or audio config.
Input modalities of the released checkpoint as the publisher states them, from: text, image, video, audio, joined with ' + ' in that order ("text", "text + image + video").
Parameters
- Total params
1.6T parameters
Source: model-card · README.md; Model Introduction
a large-scale MoE language model with 1.6 trillion total parameters and ~48 billion activated per token
Total parameter count as reported by the publisher. See params.note for whether embeddings, MTP heads or vision towers are included.
- Active params
48B parameters
Source: model-card · README.md; Model Introduction
a large-scale MoE language model with 1.6 trillion total parameters and ~48 billion activated per token
Parameters used per token in the forward pass, as reported by the publisher.
- Active non-embedding (stated)
Not disclosed
Active parameters excluding the input embedding and the output head (unembedding). Stated by the publisher, or computed from params.active by subtracting vocab × d_model for each of the embedding and output head that params.active is stated to include (basis inferred, note with the arithmetic). Not disclosed when the scope of params.active is not stated.
- Counted total params
Not calculated
Calculated: checkpoint count (1503.6B backbone) does not reproduce the stated total and active counts
Parameters of the text backbone counted from the released checkpoint's safetensors headers (embedding, output head, all experts and other weights; excluding MTP modules, vision/audio towers and n-gram lookup tables). Shown only when the counts reproduce the publisher's stated total and active parameters within 5% under some scope.
- Counted embedding + head
Not calculated
Calculated: checkpoint count (1503.6B backbone) does not reproduce the stated total and active counts
Input embedding plus output head (once if tied), counted from the checkpoint.
- Non-emb. params (counted)
Not calculated
Calculated: checkpoint count (1503.6B backbone) does not reproduce the stated total and active counts
Language backbone weights excluding input embeddings, output head, lookup/Engram tables, MTP modules and vision/audio towers. Equals params.counted_total minus params.counted_embedding. Same validation as params.counted_total.
- Active non-emb. (counted)
Not calculated
Calculated: checkpoint count (1503.6B backbone) does not reproduce the stated total and active counts
Active parameters per token excluding the input embedding and output head, counted from the checkpoint as all non-expert weights plus routed-expert weights × top_k / experts. Same validation as params.counted_total.
- Counting scope
The 1.6T total includes 135B N-gram Embedding parameters
Source: model-card · README.md; N-gram Embedding
135B N-gram Embedding parameters are included in the model
What the publisher's counts include or exclude (MTP modules, embeddings, vision encoder, n-gram tables).
Shape
- Layers
38
Source: config · /num_layers
Counts LongCat-Flash-style shortcut-connected blocks: each holds two attention + dense-FFN sublayers (self_attn.0/.1, mlps.0/.1 in the weight map) and one MoE, so the backbone has 76 MLA and 76 dense-FFN sublayers.
Number of transformer blocks in the main backbone (attention or token mixer plus FFN), excluding MTP modules.
- d_model
8,192
Source: config · /hidden_size
Residual stream width.
- Vocab
163,840
Source: config · /vocab_size
Embedding table rows (vocab_size in config, may include padding).
- Tied embeddings
No
Inferred from the source; see the explanation below.
Source: config · model.safetensors.index.json; weight_map
"lm_head.weight"
Weight map stores lm_head.weight separately from model.embed_tokens.weight.
Input embedding and output head share weights.
- Lookup memory
N-gram Embedding, n-gram size 5, 135B params; embedding space expanded ~100× over the token vocabulary
Source: blog · blog bundle; N-gram Embedding
the n-gram size is configured to 5, and 135B N-gram Embedding parameters are included in the model
Weight map has 16 N-gram tables (model.oe_embed_tokens0-15) with projections (oe_embed_proj0-15), and a separate set in the MTP module; config /oe_vocab_size_ratio 100.567, /oe_neighbor_num 5, /oe_split_num 4.
Hashed n-gram or conditional-memory tables outside the dense backbone (Engram, n-gram embeddings), with size and placement.
- d_model / layers
215.579
Calculated: d_model / layers
Width-to-depth ratio, d_model divided by the number of blocks.
Attention
- Token mixers
MLA, LSA
Source: model-card · README.md; LongCat Sparse Attention
we introduce LongCat Sparse Attention (LSA)
Config /attention_method is MLA; LSA is the DSA-derived sparse selection inside those layers.
Token-mixing layer types in the backbone, e.g. GQA, MLA, SWA, Gated DeltaNet, Mamba-2, DSA (DeepSeek sparse attention).
- Global-attention layers
76
Inferred from the source; see the explanation below.
Source: config · model.safetensors.index.json; weight_map
model.layers.37.self_attn.1.kv_a_proj_with_mqa.weight
38 blocks (/num_layers) × 2 MLA sublayers, each with its own kv_a_proj_with_mqa and therefore its own latent KV cache; counts attention sublayers, unlike shape.layers.
Layers whose KV cache grows with full context (dense or sparse global attention, MLA). Excludes sliding-window and linear/recurrent layers.
- Query heads
64
Source: config · /num_attention_heads
Query heads in the global-attention layers.
- Heads × head dim
12,288
Calculated: heads × head_dim
attn.heads × attn.head_dim, the query projection width of global layers.
- QK head dim
192
Source: config · /qk_nope_head_dim + /qk_rope_head_dim
Query/key dimension per head in global-attention layers (for MLA, nope + rope dims).
- V head dim
128
Source: config · /v_head_dim
Value dimension per head in global layers (MLA v_head_dim; stated for MLA even when equal to the QK head dim).
- MLA KV latent
512
Source: config · /kv_lora_rank
Config also sets mla_scale_q_lora and mla_scale_kv_lora true; no modeling code is published to define them.
Width of the compressed KV latent that MLA caches per token per layer.
- MLA Q latent
1,536
Source: config · /q_lora_rank
Width of the low-rank query projection in MLA; 0 when queries use a full-rank projection.
- Decoupled RoPE dim
64
Source: config · /qk_rope_head_dim
Width of the shared positional key that MLA caches next to the latent.
- Positional encoding
RoPE (MLA decoupled, 64 dims)
Source: config · config.json
"qk_rope_head_dim": 64
Positional scheme in global layers: RoPE, partial RoPE, NoPE, or mixed per layer type.
- RoPE base
1,000,000
Source: config · /rope_theta
RoPE base frequency of the global layers at release.
- QK norm
Not disclosed
Queries and keys are normalized per head before the dot product.
- Sparse attention
LSA: 2048-token budget per query = 16 sink tokens + 1024-token sliding window + indexer-selected tokens; one indexer pass shared by every 2 attention sublayers
Source: paper · p.8 §3.1
We set sink size Ksink = 16 and sliding window size Kswa = 1024
Released config matches: /index_topk 2048, /index_init_tokens 16, /index_local_tokens 1024, /cli_factor 2; index_n_heads 32 query heads of index_head_dim 128, index_k_norm_type rms; dsa_mtp_cli true shares one index across the 3 MTP steps. Card: 'the target model shares an index every 2 layers'; indexer weights exist only in self_attn.0 of each block. Hierarchical Indexing is training-free and enabled only for selected ultra-long-context tasks (blog).
Token-selection mechanism inside global layers, e.g. DSA lightning indexer with top-k.
- QKV bias
No
Source: config · /attention_bias
Query/key/value projections have bias terms.
- Indexer key dim
128
Source: config · /index_head_dim
Indexer wk weight is [128, 8192], one key head; present only in self_attn.0 of each block, so 38 indexer caches for 76 MLA sublayers.
Width of the sparse-attention indexer key cached per token in each global layer (DSA lightning indexer), counted in the KV cache.
Feed-forward
- Activation
Not disclosed
FFN activation (SwiGLU, GeGLU, ...).
- Dense FFN dim
12,288
Source: config · /ffn_hidden_size
Two dense FFN sublayers per block (mlps.0/.1 in the weight map) alongside one MoE.
Intermediate width of the dense FFN layers that are actually executed.
- Dense FFN / d_model
1.5
Calculated: ffn.dim / d_model
ffn.dim divided by shape.d_model.
Mixture of experts
- Routed experts
768
Source: config · /n_routed_experts
Config also declares 128 identity zero-experts (/zero_expert_num, /zero_expert_type); the router classifier is [896, 8192], scoring 768 + 128 experts, with an e_score_correction_bias. No modeling code is published.
Routed experts per MoE layer.
- Experts per token
12
Source: config · /moe_topk
Routed experts selected per token.
- Expert dim
2,048
Source: config · /expert_ffn_hidden_size
Intermediate width of one routed expert.
- Leading dense layers
0
Inferred from the source; see the explanation below.
Source: config · model.safetensors.index.json; weight_map
"model.layers.0.mlp.router.classifier.weight"
Every block from layer 0 carries an MoE branch next to its dense branches.
Number of initial blocks that use a dense FFN instead of MoE.
- Router score
Not disclosed
Gating function over expert logits (softmax, sigmoid).
- Load balancing
Not disclosed
Load-balancing method, with coefficients where disclosed.
- Routed scaling factor
9
Source: config · /routed_scaling_factor
Multiplier applied to the combined routed-expert output (routed_scaling_factor).
- Renormalize top-k
Not disclosed
Selected expert weights are renormalized to sum to 1.
- Expert dim / d_model
0.25
Calculated: moe.expert_dim / d_model
moe.expert_dim divided by shape.d_model.
- Sparsity (experts / top-k)
64
Calculated: experts / top_k
Routed experts divided by experts selected per token.
- Active / total
0.03
Calculated: params.active / params.total
params.active divided by params.total.
Multi-token prediction
- MTP depth
Not disclosed
Additional future tokens predicted during pretraining (0 when the model was trained without MTP).
- MTP weights released
Yes
Source: config · /weight_map | has mtp.
One MTP layer is stored (model.mtp.layers.0: MLA with indexer, dense FFN, enorm/hnorm + eh_proj, own embedding and N-gram tables); config /mtp_num_layers 3 with /mtp_replicate_modules true reuses it for 3 draft steps; mtp_disable_over_tokenizer true is undocumented. Pretraining MTP depth is not disclosed.
The released checkpoint contains the MTP module weights.
Normalization and stability
- Norm
Not disclosed
Normalization layer type in the blocks.
- Norm placement
Not disclosed
Pre-norm, post-norm, or sandwich (pre and post).
- Norm epsilon
1e-05
Source: config · /rms_norm_eps
No modeling code is published; the norm type is not stated.
Epsilon of the block normalization layers.
Context length
- Pretraining context
Not disclosed
Sequence length of the main pretraining stage.
- Trained context
1M tokens
Source: model-card · README.md; Model Introduction
train LongCat-2.0 on hundreds of billions of tokens of 1M-context data
Stated as 1M; the blog says CP scales past 512 'enabling native 1M-length training'. Released config /max_position_embeddings is 262,144.
Longest sequence length trained on after context extension.
- Advertised context
1M tokens
Source: blog · blog bundle; Inference
Serving a 1.6T-parameter model over a 1M-token context presents a significant challenge
Stated as 1M; released config /max_position_embeddings is 262,144.
Longest context the publisher says the model supports, possibly beyond trained length via RoPE scaling.
- Extension method
YaRN (factor 120, beta_fast 32, beta_slow 1, mscale 1) over an 8,192-token original context
Source: config · config.json; rope_scaling
"original_max_position_embeddings": 8192, "rope_type": "deepseek_yarn", "factor": 120
Config rope_scaling keys factor, beta_fast, beta_slow, mscale, mscale_all_dim; the stages that used it are not disclosed.
How context was extended (stages, RoPE base change, YaRN factor).
Pretraining recipe
- Pretraining tokens
>35T
Source: model-card · README.md; Model Introduction
Pretraining spans millions of accelerator-days across more than 35 trillion tokens
Tokens seen in pretraining including annealing and context extension, as reported.
- Optimizer
Muon
Source: blog · blog bundle; Training at Scale
We deploy the Muon optimizer at large scale on our accelerators
Optimizer used for the bulk of parameters.
- Peak LR
Not disclosed
Peak learning rate of the main pretraining stage.
- Final LR
Not disclosed
Learning rate at the end of the decay.
- LR schedule
Not disclosed
Learning-rate schedule shape with phase boundaries.
- Warmup
Not disclosed
Warmup length in steps or tokens.
- Batch size
Not disclosed
Global batch size, including any batch-size ramp: sequences × sequence length (token total), or the token total alone when only that is stated.
- Weight decay
Not disclosed
Decoupled weight decay coefficient.
- Adam betas
Not disclosed
(beta1, beta2) of the Adam-family optimizer or its AdamW fallback.
- Grad clip
Not disclosed
Global gradient-norm clipping threshold.
- Training precision
Not disclosed
Matmul precision of pretraining (BF16, FP8 recipe, NVFP4 recipe) with the scaling granularity where disclosed. Never inferred from the released checkpoint dtype.
- Hardware
over 50K AI ASICs in superpods of up to 48 machines; millions of accelerator-days
Source: blog · blog bundle; Training
LongCat-2.0 is pre-trained on over 50K AI ASICs
6D parallelism (TP/CP/EP/DP/PP plus EMBP for the N-gram Embeddings), ZeRO-1, selective recomputation and allocator-level OOM-aware offloading; CP scaled past 512 for 1M-length training.
Accelerators used for pretraining, count and duration where stated.
- Accelerators
>50,000 AI ASICs
Accelerator count and type for the main pretraining run, as the count with thousands separators and the chip's short name ("2,048 H800", "2,048 GH200"). A stated range or bound stays as stated ("2,048–4,096 GH200").
- Parallelism
TP · PP · EP · CP · DP · EMBP
Source: blog · blog bundle; Training, Training at Scale
Beyond standard TP/CP/EP/DP/PP, we introduce EMBP to parallelize and accelerate N-gram Embeddings.
No degrees stated; EMBP parallelizes the N-gram Embeddings; the same section states ZeRO-1.
Parallelism layout of the main pretraining run as stated, as degree-annotated terms in the order TP, PP, VPP, EP, CP, DP ("PP16 · EP64 · ZeRO-1 DP"); a term without a stated degree has none ("FSDP").
- Stages
pre-training
- Tokens
- >35T
Source: model-card · README.md; Model Introduction
Pretraining spans millions of accelerator-days across more than 35 trillion tokens
Sparse attention is reached through a dense warmup computed forward-only alongside the KL loss (blog, Long Context Training).
1M long-context
- Seq len
- 1000000
Source: model-card · README.md; Model Introduction
train LongCat-2.0 on hundreds of billions of tokens of 1M-context data
Hundreds of billions of tokens.
Source: model-card · README.md; Model Introduction
Sparse attention is reached through a dense warmup computed forward-only alongside the KL loss (blog, Long Context Training).
Ordered pretraining stages. Each item: name, tokens, seq_len, lr, batch_tokens, batch_sequences, steps, note, quote, and source when it differs from the fact's source.
KV cache
- KV bytes / token
107,008 bytes
Calculated: kv.bytes_at_context / 131072 tokens; uniform BF16, payload only; quantization overhead excluded. Global KV: 76 × (704 × 16/8) B/entry
Retained KV cache divided by context length, including bounded SWA caches and indexer keys. Exports use uniform BF16 at 131,072 tokens. The website can select uniform payload precision (no quantization overhead) or sourced component formats (including scales and documented padding). This is cache payload, not peak prefill memory or persistent prefix storage.
- Global KV bytes / token
107,008 bytes
Calculated: 107008 B / 1 tokens (Global KV); uniform BF16, payload only; quantization overhead excluded. Global KV: 76 × (704 × 16/8) B/entry
Long-context global KV bytes per token, including indexer keys; excludes bounded SWA caches. Compression rates and cross-layer cache reuse are included. Exports use uniform BF16; the website's storage selector also applies here. This is the field to compare with global-KV figures in reports, using the matching storage recipe.
- KV bytes / token / B active
2,229 bytes
Calculated: kv.bytes_per_token (131072 tokens) / (params.active / 1e9); uniform BF16, payload only; quantization overhead excluded. Global KV: 76 × (704 × 16/8) B/entry
kv.bytes_per_token divided by active parameters in billions (131,072 tokens in exports).
- KV at context / B active
292,203,179 bytes
Calculated: kv.bytes_at_context (131072 tokens) / (params.active / 1e9); uniform BF16, payload only; quantization overhead excluded. Global KV: 76 × (704 × 16/8) B/entry
kv.bytes_at_context divided by active parameters in billions (131,072 tokens in exports).
- KV at context
14,025,752,576 bytes
Calculated: 107008 B × 131072 (Global KV); uniform BF16, payload only; quantization overhead excluded. Global KV: 76 × (704 × 16/8) B/entry
Retained KV cache for one sequence at the selected context, including SWA windows and indexer keys. Exports use uniform BF16 at 131,072 tokens. Excludes recurrent state, compressor tail state, draft/MTP caches, temporary prefill tensors, allocator metadata and persistent prefix snapshots. Does not estimate peak prefill or total serving memory.
Evaluations
- AA Intelligence Index
19
Source: leaderboard · Artificial Analysis model page /models/longcat-2-0, Wayback snapshot 2026-10-19
LongCat 2.0 scores 19 on the Artificial Analysis Intelligence Index
Variant: LongCat 2.0. Index v4.3.2.
Artificial Analysis Intelligence Index v4.3.2 of the post-trained model, from a dated Wayback snapshot of its Artificial Analysis page. The note names the variant (reasoning setting).