archdb

LLM architectures and pretraining recipes, with sources.

Models

Open a model to see its architecture, pretraining details and sources.

Model architectures
ModelTotal parametersActive parametersAttentionLayersHidden size
A.X K2SK Telecom688B parameters33B parametersMLA, DSA (SGA)617,168
Apertus-v1.5-70B-baseSwiss AI70B parameters70B parametersGQA808,192
Beam-1.0-501B-A23BReflection AI501.3B parameters23.4B parametersGQA, SWA525,120
Command A+ (command-a-plus-05-2026)Cohere218B parameters25B parametersGQA, SWA324,096
DeepSeek-V3-BaseDeepSeek671B parameters37B parametersMLA617,168
DeepSeek-V3.2DeepSeek671B parameters37B parametersMLA, DSA617,168
DeepSeek-V4-Flash-BaseDeepSeek284B parameters13B parametersSWA, CSA, HCA (CSA and HCA each with a 128-token SWA branch)434,096
DeepSeek-V4-ProDeepSeek1.6T parameters49B parametersCSA, HCA (each with a 128-token SWA branch)617,168
DeepSeek-V4.1-FlashDeepSeek552B parameters16B parametersSWA, CSA2405,120
Gemma-4-26B-A4BGoogle DeepMind25.2B parameters3.8B parametersGQA, SWA302,816
Gemma-4-31BGoogle DeepMind30.7B parameters30.7B parametersGQA, SWA605,376
GLM-5.3Z.ai744B parameters40B parametersMLA, DSA786,144
GLM-5.3-FlashZ.ai320B parameters18B parametersKDA, MLA, DSA454,096
gpt-oss-120bOpenAI116.8B parameters5.13B parametersGQA, SWA362,880
Granite-4.1-30B-BaseIBM28.87B parameters28.87B parametersGQA644,096
Hy3-preview-BaseTencent295B parameters21B parametersGQA804,096
Hy4-previewTencent770B parameters49B parametersMLA, DSA786,144
InklingThinking Machines Lab975B parameters41B parametersGQA, SWA666,144
Inkling-SmallThinking Machines Lab276B parameters12B parametersGQA, SWA424,096
K-EXAONE 2.0LG AI Research750B parameters37B parametersGQA, SWA786,144
K2-Horizon-375B-A23B (mid_4_6000)MBZUAI IFM375B parameters23B parametersGQA616,144
K2-Horizon-MoVA-36B-A4B (mid_4_10000)MBZUAI IFM36B parameters4B parametersGQA (MoVA: routed value projections in 45 of 48 layers)482,560
Kimi-K2-BaseMoonshot AI1.04T parameters32.6B parametersMLA617,168
Kimi-K2.5Moonshot AI1T parameters32B parametersMLA617,168
Kimi-K3Moonshot AI2.78T parameters104.2B parametersKDA, MLA937,168
Laguna-M.1-basePoolside225.8B parameters23.4B parametersGQA704,096
Laguna-S-2.1Poolside118B parameters8B parametersGQA, SWA483,072
Laguna-XS-2.1Poolside33B parameters3B parametersGQA, SWA402,048
Laguna-XS.2Poolside33.4B parameters3B parametersGQA, SWA402,048
Ling-2.6-1T-baseinclusionAI1T parameters63B parametersLightning, MLA808,192
Ling-3.0-flash-baseinclusionAI124B parameters5.1B parametersKDA, MLA422,560
Ling-3.1-flashinclusionAI560B parameters25B parametersKDA, MLA724,096
LongCat-2.0Meituan1.6T parameters48B parametersMLA, LSA388,192
MAI-Thinking-1Microsoft AI962B parameters34.7B parametersGQA, SWA786,656
MiMo-V2-Flash-BaseXiaomi309B parameters15B parametersGQA, SWA484,096
MiMo-V2.5-Pro-BaseXiaomi1.02T parameters42B parametersGQA, SWA706,144
MiMo-V2.6-FlashXiaomi310B parameters15B parametersGQA, SWA484,096
MiMo-V2.6-ProXiaomi1.02T parameters42B parametersGQA, SWA706,144
MiniMax-M1MiniMax456B parameters45.9B parametersLightning, GQA806,144
MiniMax-M2.7MiniMax229.9B parameters9.8B parametersGQA623,072
MiniMax-M3MiniMax428B parameters23B parametersGQA, MSA606,144
Mistral-Large-3-675B-Base-2512Mistral AI675B parameters41B parametersMLA617,168
Mistral-Medium-3.5-128BMistral AI128B parameters128B parametersGQA8812,288
Mistral-Small-4-119B-2603Mistral AI119B parameters6.5B parametersMLA364,096
Motif-3-BaseMotif Technologies314B parameters13.2B parametersGDLA (MLA latent KV + grouped differential attention), SWA534,096
Muse-Spark-Open-816B-A42BMeta816B parameters42B parametersGQA, SWA628,192
Nemotron-3-Super-120B-A12B-BaseNVIDIA120.6B parameters12.7B parametersGQA, Mamba-2484,096
Nemotron-3-Ultra-550B-A55B-BaseNVIDIA550B parameters55B parametersGQA, Mamba-2608,192
Olmo-3-1125-32BAi232B parameters32B parametersGQA, SWA645,120
Qwen3-Next-80B-A3B-InstructQwen80B parameters3B parametersGated DeltaNet, GQA482,048
Qwen3.5-122B-A10BQwen122B parameters10B parametersGated DeltaNet, GQA483,072
Qwen3.5-27BQwen27B parameters27B parametersGated DeltaNet, GQA645,120
Qwen3.5-35B-A3BQwen35B parameters3B parametersGated DeltaNet, GQA402,048
Qwen3.5-397B-A17BQwen397B parameters17B parametersGated DeltaNet, GQA604,096
Qwen3.8-2.4T-A95BQwen2.4T parameters95B parametersGated DeltaNet, GQA928,192
Qwen3.8-27BQwen27B parameters27B parametersGated DeltaNet, GQA645,120
Qwen3.8-Flash-NextQwen125B parameters6B parametersGated DeltaNet, QSA482,560
Solar-Open2-250BUpstage250.3B parameters15B parametersGQA, KDA484,096
Step-3StepFun321B parameters38B parametersMFA617,168
Step-3.5-Flash-BaseStepFun196B parameters11B parametersGQA, SWA454,096
Step-5-PreviewStepFun600B parameters27B parametersSWA, DSA (Sparse GQA)924,096
Trinity-Large-BaseArcee AI398B parameters13B parametersGQA, SWA603,072