LLM Cheatsheet Mind Map: Frontier vs Open-Weight Models, VRAM, Licences, Trading and Restricted Content
Frontier APIs against open weights: dense and mixture-of-experts architectures, the current open models of every major lab, how much GPU memory each needs, what their licences allow, and where they fit in trading and restricted-content work.
How you get the model
API, weights, source
- What it is
- Four ways a model reaches you: a proprietary API (weights never leave the vendor), open weights you download and run, a fully open-source release (weights plus code and data under an open licence), and gated or research-only checkpoints. Open-weight does not mean open-source: most "open" models publish weights only, under licences that range from Apache 2.0 to custom revenue-gated terms.
- When to use it
- Decide the access model first: it fixes where the data goes, who pays for GPUs and what the licence lets you ship.
Frontier API
Strongest, vendor-hosted
- What it is
- GPT, Claude, Gemini, Grok and Muse Wonder are reachable only through their vendors' APIs or cloud marketplaces. You get the strongest models, no hardware and per-token billing; data leaves your network under a processing agreement, and regional processing or zero retention are contract settings.
- When to use it
- Default for hard reasoning, long agentic coding and anything without a rule that forbids sending data out.
Open weights
Download and self-host
- What it is
- The trained weights are published (usually on Hugging Face) so you can run, quantise and fine-tune them on your own GPUs. Training data and code normally stay closed, and each checkpoint carries its own licence.
- When to use it
- Data that must stay on your hardware, fine-tuning, predictable high volume, or offline and air-gapped sites.
Open source
Weights, code and data
- What it is
- A release you can fully reproduce: weights, training code and enough data documentation under an open licence. Rare at frontier scale; an Apache 2.0 or MIT licence on the weights alone makes a model permissive, not reproducible.
- When to use it
- Research, audits and regulated settings that must explain how the model was built.
Gated and custom terms
Accept before download
- What it is
- Some repositories require you to accept a licence or share contact details before downloading (Llama 4, Muse Spark Open), and some licences add revenue thresholds, attribution or reseller clauses on top of the weights.
- When to use it
- Read the licence file in the repository, not the family's blog post: terms change between checkpoints of the same family.
Private deployment
VPC, on-prem, air-gapped
- What it is
- Open weights served by vLLM, SGLang or llama.cpp inside your VPC, on rented GPUs or on your own servers; prompts, embeddings and logs never leave the network. Both vLLM and SGLang expose OpenAI-compatible endpoints, so switching from an API is a base-URL change.
- When to use it
- Client data under NDA, health or financial records, proprietary trading logic.
Hybrid routing
API and self-hosted
- What it is
- A router sends each request to the cheapest model that does it well: sensitive fields to the self-hosted model, hard reasoning to a frontier API, fixed-option decisions to a decision model, with a fallback when a provider is down or a model retires.
- When to use it
- Most production systems: neither "all API" nor "all local".
Architectures
Dense, MoE, hybrid
- What it is
- How a model spends its parameters per token decides its memory, speed and cost. The two axes to keep apart: total parameters (what must sit in memory) and active parameters (what is computed per token).
- When to use it
- Read both numbers on every model card before buying or renting a GPU.
Dense
Every weight, every token
- What it is
- Every parameter takes part in every token: a 27B dense model keeps 27B weights in memory and computes all of them per token. Simple to serve, predictable latency, the easiest to fine-tune. Examples: Qwen3.8-27B, Gemma 4 31B, Muse Glimmer, Granite 4.1, Ministral 3.
- When to use it
- Single-GPU deployments and fine-tuning on your own data.
Mixture of experts
A router picks experts
- What it is
- A router sends each token to a few expert networks out of many, so compute follows the active parameters while memory follows the total. Qwen3-30B-A3B routes to 8 of 128 experts (30.5B total, 3.3B active); Kimi K3 to 16 of 896 plus 2 shared (reported).
- When to use it
- Frontier quality at mid-size compute, when you can afford the memory for all experts.
- Key figures
- "MoE reduces computation, not memory": all experts must be loaded even though few are active per token.Source: Hugging Face blog, Mixture of Experts Explained
Hybrid Mamba and attention
Cheaper long context
- What it is
- State-space layers (Mamba-2) mixed with a few attention layers and MoE blocks: the KV cache grows only in the attention layers, so long contexts cost less memory. NVIDIA Nemotron 3 uses it; IBM Granite 4.0 did, while Granite 4.1 returned to dense.
- When to use it
- Long-running agents and long documents on limited GPU memory.
Reasoning models
Think, then answer
- What it is
- The model spends extra tokens working through the problem before the answer; gpt-oss exposes low, medium and high reasoning effort. More latency and more KV cache per request, better results on maths, code and planning.
- When to use it
- Open problems; skip it for grounded summarising, where it adds cost without accuracy.
- AI angle
- Budget the reasoning tokens in the KV cache, not just the answer.
Microsoft Decision-1
Scores fixed options
- What it is
- A decision model post-trained by Microsoft from Qwen3.5-9B: you pass a text and a fixed set of options and it returns a calibrated probability per option instead of generating text. Available in Microsoft Foundry; no downloadable weights were announced.
- When to use it
- Intent classification, tool and model routing, grading answers, continue-retry-escalate steps in agents.
- Key figures
- $0.042 per million input tokens, output free; 32,768-token input limit.Source: Microsoft Learn, Foundry Models sold by Azure
Classic classifiersTrees and encoders
- What it is
- Gradient-boosted trees (LightGBM, XGBoost, CatBoost) for tabular and numeric data, encoder transformers (ModernBERT, DeBERTa) for text labels, TabPFN for small tabular sets without training. They need a fraction of an LLM's memory and often run on CPU.
- When to use it
- Fraud, risk, price features and any numeric prediction: an LLM is not the right tool there.
Frontier APIs
No downloadable weights
- What it is
- The strongest proprietary models of each lab, as listed on 26 October 2026. Six of the eight language-model flagships publish a context window of one million tokens or more.
- When to use it
- Benchmark one of these against your best open candidate on your own evaluation set before deciding to self-host.
GPT-6 AstraOpenAI flagship
- What it is
- OpenAI's flagship, released 3 September 2026, with a 1,050,000-token window. Cheaper tiers: GPT-6.1 Terra and GPT-6 Sol; OpenAI also publishes open weights separately as gpt-oss.
- When to use it
- General agents, tool calling, financial analysis.
- Key figures
- $10 input / $50 output per million tokens.Source: AI Board report, 26 Oct 2026
Claude Fable 5.1Anthropic flagship
- What it is
- Anthropic's flagship, released 1 September 2026, with a 1,000,000-token window; Claude Opus 5.5 and Sonnet 5.5 are the tiers most workloads start on. No open weights.
- When to use it
- Coding agents on large codebases, long documents, complex workflows.
- Key figures
- $10 input / $50 output per million tokens.Source: AI Board report, 26 Oct 2026
Gemini 4 ArgonGoogle flagship
- What it is
- Google's flagship reasoning model, generally available on 13 October 2026, with a 1,048,576-token window. Google's open-weight line is Gemma.
- When to use it
- Video, audio and image understanding, million-token documents.
- Key figures
- $2 input / $10 output per million tokens (introductory).Source: AI Board report, 26 Oct 2026
Grok 4.7
xAI flagship
- What it is
- xAI's flagship for coding, agents and knowledge work, released 21 September 2026, with a 500,000-token window and real-time access to the X ecosystem.
- When to use it
- Social sentiment and market chatter in near real time.
- Key figures
- $2 input / $6 output per million tokens; the rate doubles above 200K tokens.Source: AI Board report, 26 Oct 2026
Muse Wonder
Meta flagship
- What it is
- Meta's most capable Muse model, released 12 October 2026 on the Meta Model API without open weights. Three days later Meta published Muse Spark Open with downloadable weights.
- When to use it
- Teams already on Meta's API; for self-hosting use Muse Spark Open or Llama 4.
- Key figures
- $4 input / $20 output per million tokens, 1,000,000-token window.Source: AI Board report, 26 Oct 2026
Open-weight modelsEvery major lab
- What it is
- The downloadable models of each lab in November 2026, with total and active parameters, context and licence. Sizes run from phone-class Gemma 4 E2B to Kimi K3 at 2.8 trillion parameters.
- When to use it
- Shortlist by licence first, then by what fits your memory, then evaluate quality.
Meta: Llama and Muse
Four open lines
- What it is
- Llama 4 (April 2025) remains downloadable under the Llama 4 Community License; in 2026 Meta added Muse Glimmer (dense, Apache 2.0) and Muse Spark Open (frontier MoE, Muse Model License).
- When to use it
- The ecosystem most self-hosted deployments already run.
Llama 4 Scout
109B / 17B active MoE
- What it is
- Mixture of experts with 16 experts, 109B total and 17B active parameters and a 10M-token context window. Multimodal (text and image input).
- When to use it
- Very long documents and codebases on a single data-centre GPU.
- Key figures
- Fits a single H100 GPU with Int4 quantisation.Source: Meta, Llama 4 announcement
Llama 4 Maverick
400B / 17B active MoE
- What it is
- Mixture of experts with 128 experts, 400B total and 17B active parameters, a 1M-token window and image input.
- When to use it
- General assistant and multimodal RAG when an 8-GPU host is available.
- Key figures
- Runs on a single H100 host (8 GPUs).Source: Meta, Llama 4 announcement
Muse Spark Open
816B / 42B active MoE
- What it is
- Released 15 October 2026 on Hugging Face, Google Cloud and Microsoft Foundry: a mixture of experts with 256 experts and 8 active per token, 816B total and 42B active parameters, up to a 1M-token window (reported), under the custom Muse Model License.
- When to use it
- Frontier-class coding and agents on your own cluster, if the licence fits your business model.
- Key figures
- Reported minimum of about 700 GB of GPU memory in NVFP4, about 2 TB in BF16.Source: The New Stack, Muse Spark Open
Muse Glimmer
30B dense, Apache 2.0
- What it is
- A dense model of about 30B parameters released in August 2026 for local agents, function calling and coding, under Apache 2.0: the most permissive licence Meta has used for a strong model.
- When to use it
- A commercial product on a 24–32 GB GPU without Llama's licence conditions.
OpenAI gpt-ossTwo MoE sizes, Apache 2.0
- What it is
- OpenAI's open-weight reasoning models, released 5 August 2025 under Apache 2.0 plus the gpt-oss usage policy, with configurable reasoning effort, tool use and a 128K window. Weights ship natively in MXFP4.
- When to use it
- Reasoning on your own hardware with the simplest licence of the frontier labs.
gpt-oss-120b
117B / 5.1B active
- What it is
- Mixture of experts with 117B total and 5.1B active parameters; the very low active count makes it fast once loaded.
- When to use it
- Production reasoning on one 80 GB or 96 GB GPU.
- In code
vllm serve openai/gpt-oss-120b- Key figures
- Fits a single 80 GB GPU thanks to MXFP4.Source: OpenAI, gpt-oss model card
gpt-oss-20b
21B / 3.6B active
- What it is
- Mixture of experts with 21B total and 3.6B active parameters, aimed at local and on-device inference.
- When to use it
- The strongest reasoning model for a 16 GB laptop GPU.
- In code
ollama run gpt-oss:20b- Key figures
- Runs within 16 GB of memory.Source: OpenAI, gpt-oss model card
Google Gemma 4E2B to 31B, Apache 2.0
- What it is
- Five sizes: E2B (2.3B effective) and E4B (4.5B effective) for phones and edge, 12B dense, 26B A4B (MoE, 128 experts plus 1 shared, 8 active, 25.2B total / 3.8B active) and 31B dense, with 256K context on the larger three. The Gemma 4 model card states Apache 2.0, unlike earlier Gemma terms.
- When to use it
- Multimodal apps from the phone to a workstation GPU under one permissive family.
- Key figures
- 12B, 26B A4B and 31B target consumer GPUs and workstations; E2B and E4B target phones and edge devices.Source: Gemma 4 model card
Mistral AI
European, mixed licences
- What it is
- Mistral publishes open weights under different licences per model: Apache 2.0 for Large 3, Small 4 and Ministral 3, a modified MIT for Medium 3.5, and the Mistral Ecosystem Open-Weight License for Large 4.
- When to use it
- EU sovereignty: the same model self-hosted or on Mistral's European endpoints.
Mistral Large 3
675B / 41B, Apache 2.0
- What it is
- Released 2 December 2025: a mixture of experts with 675B total and 41B active parameters, a 256K window and Apache 2.0, the largest permissively licensed MoE of its generation.
- When to use it
- A frontier-scale model you can resell or fine-tune without revenue clauses.
- Key figures
- Estimate: about 675 GB of weights in FP8, so an 8 × H200 node (1,128 GB) holds it with room for KV cache.Source: Computed from the Mistral 3 announcement
Mistral Large 4
About 1T MoE, preview
- What it is
- Public preview since 6 October 2026, weights on Hugging Face since 29 October (reported): a multimodal mixture of experts of about one trillion parameters under the Mistral Ecosystem Open-Weight License, which lets you use and fine-tune it but requires a commercial licence to resell it as a model service.
- When to use it
- Internal frontier-scale workloads; not for a model-as-a-service business without a deal.
Ministral 3
3B, 8B, 14B dense
- What it is
- Dense vision-capable models of 3B, 8B and 14B parameters with a 256K window, Apache 2.0, released 2 December 2025.
- When to use it
- Edge devices, laptops and cheap fine-tunes.
Alibaba Qwen
Widest open line
- What it is
- Sizes from 0.6B to 2.4T with coder, vision, omni, embedding, reranker and guard variants. Qwen3 and Qwen3.8-27B are Apache 2.0; the Qwen3.8 Max-class weights use a custom licence.
- When to use it
- Private RAG, multilingual assistants, local coding agents, fine-tuned domain models.
Qwen3.8-27B
Dense, 262K, Apache 2.0
- What it is
- Dense multimodal model of 27B language parameters (August 2026), 262,144 tokens native context extensible to about one million, Apache 2.0. The card recommends SGLang or vLLM for serving.
- When to use it
- The most practical open model for one 24–48 GB GPU.
- In code
vllm serve Qwen/Qwen3.8-27B --max-model-len 65536- Key figures
- Estimate: 27B × 4.8 bits ÷ 8 ≈ 16 GB of weights at Q4_K_M, 54 GB at BF16.Source: Computed from the Qwen3.8-27B model card
Qwen3.8-2.4T-A95B
Max-class open MoE
- What it is
- The open Max-class weights: 2.4T total and 95B active parameters, 512 experts with 10 routed plus 1 shared per token, 262,144 tokens native extensible to 1,010,000, under the custom "qwen3.8-max" licence.
- When to use it
- Data-centre clusters only; check the model-service clause before building a product on it.
- Key figures
- Model-as-a-service providers above $50M a year reportedly need a separate licence.Source: SCMP, Qwen3.8-Max licence
Qwen3-30B-A3B
30.5B / 3.3B MoE
- What it is
- Mixture of experts from April 2025: 30.5B total, 3.3B active, 128 experts with 8 active, Apache 2.0. Fast like a 3B model, memory like a 30B one.
- When to use it
- Snappy local agents on a 24 GB GPU at Q4.
Qwen3-Coder-480B-A35B
Agentic coding MoE
- What it is
- Coding mixture of experts from July 2025: 480B total, 35B active, 160 experts with 8 active, 256K native context, Apache 2.0.
- When to use it
- A coding agent when the source code may not leave your network.
DeepSeek
Frontier MoE, MIT
- What it is
- DeepSeek publishes its frontier models as MIT-licensed mixtures of experts with sparse attention and a 1M-token window, the most permissive licence at that scale.
- When to use it
- Cost-efficient reasoning, maths and code, self-hosted or through its low-priced API.
DeepSeek-V4-Pro
1.6T / 49B, MIT
- What it is
- Released April 2026: 1.6T total and 49B active parameters, 1M-token window, MIT; experts stored in FP4 and the rest in FP8 (reported).
- When to use it
- Data-centre scale only: plan for a terabyte of GPU memory.
- Key figures
- $1.32 input / $3.96 output per million tokens on DeepSeek's API at peak, half off-peak.Source: AI Board report, 26 Oct 2026
DeepSeek-V4-Flash
284B / 13B, MIT
- What it is
- The smaller V4: 284B total, 13B active, 1M window, MIT. V4.1-Flash (10 September 2026) followed with native image input.
- When to use it
- Private mid-tier workhorse on a 4–8 GPU server.
- Key figures
- Estimate: 284B × 4 bits ÷ 8 ≈ 142 GB of weights at 4-bit.Source: Computed from the DeepSeek-V4-Flash model card
DeepSeek-V3.2
685B MoE, MIT
- What it is
- The December 2025 generation: about 685B total and 37B active parameters, 128K window, MIT, the first with DeepSeek Sparse Attention.
- When to use it
- Still widely quantised and served; a known quantity for existing clusters.
Moonshot Kimi K3
2.8T / ~104B MoE
- What it is
- Weights published in July 2026 (reported): 2.8T total and about 104B active parameters, 896 experts with 16 routed plus 2 shared, a 1,048,576-token window and image input, under the custom Kimi K3 License. Its predecessor Kimi K2 (1T / 32B) used a modified MIT.
- When to use it
- Long-horizon agentic coding on a multi-node cluster.
Z.ai GLM-5.x
Coding and reasoning MoE
- What it is
- GLM-5 (February 2026, 744B / 40B, MIT) and GLM-5.3-Flash (August 2026, about 320B / 18B multimodal MoE, MIT, reported). The flagship GLM-5.3 moved to a custom "glm-5.3" licence, so the family no longer shares one licence.
- When to use it
- Chinese and English coding agents; pick the Flash for MIT terms.
NVIDIA Nemotron 3
Hybrid Mamba MoE
- What it is
- Hybrid Mamba-2, attention and MoE models under the OpenMDW License 1.1, which allows commercial use: Nemotron 3.5 Lightning (30B / 3B, NVFP4 variant) for sub-agents and tool calling, and Nemotron 3 Ultra (550B / 55B) for data-centre reasoning.
- When to use it
- Long-running agents where KV-cache memory is the bottleneck.
IBM Granite 4.1
3B, 8B, 30B, Apache 2.0
- What it is
- Dense base and instruct models of 3B, 8B and 30B parameters (April 2026), Apache 2.0, designed for enterprise RAG, extraction and classification; IBM also publishes the Granite Guardian safety models.
- When to use it
- Document pipelines where licence clarity matters more than leaderboard rank.
VRAM sizing
Will it fit?
- What it is
- GPU memory must hold the weights, the KV cache of every live request, activations and the runtime's own buffers. The weights are the part you can calculate exactly; the rest grows with context length and concurrency.
- When to use it
- Size the GPU before choosing the model, not after the out-of-memory error.
Weight memory formula
Params × bits ÷ 8
- What it is
- Weight memory in bytes = total parameters × bits per weight ÷ 8. A 7B model needs 14 GB at 16-bit, 7 GB at 8-bit, 3.5 GB at 4-bit; a 70B model 140, 70 and 35 GB. Use total parameters, never active ones, and add 10–30 % for KV cache and runtime as a starting margin.
- When to use it
- The first check on any model card.
- In code
weights_gb = params_billion * bits_per_weight / 8 # 27B at Q4_K_M: 27 * 4.8 / 8 = 16.2 GB
Quantisation formats
Fewer bits per weight
- What it is
- Storing weights in fewer bits trades a little quality for a lot of memory. GGUF (llama.cpp, Ollama) is the laptop format; AWQ, GPTQ and FP8 are common on vLLM; MXFP4 and NVFP4 are the 4-bit floats of the newest GPUs.
- When to use it
- Q4 to fit a bigger model on the same GPU; 8-bit or 16-bit when accuracy on numbers and code matters.
BF16, FP16 and FP8
16 and 8 bits
- What it is
- BF16 and FP16 store 2 bytes per weight, the quality reference and the format most checkpoints ship in. FP8 (formats E4M3 and E5M2) halves that with hardware support on Hopper and Blackwell GPUs.
- When to use it
- Fine-tuning (BF16) and data-centre serving (FP8).
- Key figures
- FP8 defines two encodings: E4M3 for weights and activations, E5M2 for gradients.Source: FP8 Formats for Deep Learning (arXiv 2209.05433)
GGUF Q8_0 and Q6_K
Near-lossless
- What it is
- llama.cpp's 8-bit and 6-bit formats: Q8_0 uses about 8.5 bits per weight and Q6_K about 6.56, both very close to 16-bit quality.
- When to use it
- Local models where you have the memory and want the quality.
- Key figures
- Q8_0 ≈ 8.5 bits per weight, Q6_K ≈ 6.56.Source: GitHub, llama.cpp quantize README
GGUF Q4_K_M
The local default
- What it is
- The 4-bit K-quant most local guides start from, at roughly 4.8 bits per weight: about a third of the 16-bit size with a small quality loss.
- When to use it
- Fitting a 14B model on 16 GB or a 27–32B model on 24 GB.
- In code
llama-server -hf <repo>:Q4_K_M -c 16384 -ngl 99
MXFP4 and NVFP4
Hardware 4-bit floats
- What it is
- Both store E2M1 4-bit floats with shared scales: MXFP4 (Open Compute Project) shares one 8-bit power-of-two scale per 32 values; NVFP4 (NVIDIA Blackwell) one FP8 scale per 16 values plus a per-tensor FP32 scale. gpt-oss ships in MXFP4.
- When to use it
- Large MoE models on Blackwell GPUs at 4-bit with native speed.
- Key figures
- NVFP4: blocks of 16 values with an FP8 E4M3 scale.Source: NVIDIA blog, Introducing NVFP4
KV cache
Grows with context
- What it is
- Every token in every live request keeps a key and a value per layer: KV bytes = 2 × layers × KV heads × head dimension × bytes per value × tokens × batch. Illustration: a model with 48 layers, 8 KV heads and head size 128 in FP16 needs about 6.4 GB for one 32K-token request.
- When to use it
- Why a model that "fits" fails at long context or ten concurrent users; cap max-model-len or quantise the cache to FP8.
- In code
kv_bytes = 2 * layers * kv_heads * head_dim * bytes * tokens * batch vllm serve <model> --kv-cache-dtype fp8 --max-model-len 32768- Key figures
- KV cache size = 2 × layers × KV heads × head dimension × precision bytes × sequence length × batch.Source: NVIDIA blog, Mastering LLM Techniques: Inference Optimization
The MoE memory trap
Total, not active
- What it is
- A 284B-A13B model computes like a 13B but must load 284B parameters. Memory planning uses the total; only speed benefits from the active count. Exceptions are tricks, not free: expert offloading to CPU RAM costs PCIe bandwidth on every token.
- When to use it
- Read "A3B", "A22B" in a model name as speed, never as memory.
Offloading and multi-GPU
Split the model
- What it is
- When weights exceed one GPU: tensor parallelism splits each layer across GPUs in a node (fast with NVLink), pipeline parallelism splits layers across nodes, and llama.cpp can keep some layers in system RAM. Each split adds communication; PCIe and network bandwidth become the bottleneck.
- When to use it
- Tensor parallel inside one 8-GPU node first; offload to CPU only for experiments.
- In code
vllm serve <model> --tensor-parallel-size 8 llama-server -m model.gguf -ngl 30 # 30 layers on GPU, rest in RAM
Fine-tuning memory
Full, LoRA, QLoRA
- What it is
- Full fine-tuning with Adam in mixed precision needs about 16–18 bytes per parameter (weights, gradients, two optimiser states, an FP32 master copy) before activations, so a 7B model needs over 110 GB. LoRA trains small adapters on frozen weights; QLoRA keeps the frozen base in 4-bit.
- When to use it
- QLoRA on one GPU for domain adaptation; full fine-tuning only with a multi-GPU budget and a reason.
- Key figures
- Mixed-precision Adam training: about 18 bytes per parameter plus activations.Source: Hugging Face Transformers, Model training anatomy
- QLoRA fine-tunes a 65B model on a single 48 GB GPU.Source: QLoRA paper (arXiv 2305.14314)
Serving engines
vLLM, SGLang, llama.cpp
- What it is
- vLLM and SGLang serve many users at once with paged KV cache and continuous batching on data-centre GPUs; llama.cpp and Ollama run GGUF models on laptops, Macs and CPUs. All four expose an OpenAI-compatible API.
- When to use it
- Ollama or llama.cpp to prototype, vLLM or SGLang for production traffic.
- In code
ollama run qwen3:30b-a3b python -m sglang.launch_server --model-path <model> --tp 4
GPU memory tiers
16 GB to 8-GPU nodes
- What it is
- What fits at each memory size, for inference with a modest context. Bandwidth decides tokens per second; capacity decides what loads at all.
- When to use it
- Pick the tier from the largest model you must serve, plus its KV cache.
16 GB laptop GPU
Up to 14B at Q4
- What it is
- 7–8B at 8-bit or 16-bit, 14B at Q4 with a modest context, gpt-oss-20b, Gemma 4 E4B and 12B at Q4, plus every classic classifier. 32B needs heavy quantisation or CPU offloading; 70B does not fit.
- When to use it
- Development, private RAG prototypes, local classification and routing.
24–32 GB desktop
RTX 4090, RTX 5090
- What it is
- RTX 4090 (24 GB, 1,008 GB/s) and RTX 5090 (32 GB, 1,792 GB/s): 14B at 16-bit, Qwen3.8-27B, Gemma 4 31B, Muse Glimmer and Qwen3-30B-A3B at Q4 with room for context.
- When to use it
- The serious local workstation: agents, RAG and QLoRA on small models.
- Key figures
- RTX 5090: 32 GB GDDR7, 1,792 GB/s.Source: NVIDIA RTX 5090 spec page
48–96 GB workstation
RTX PRO 6000 Blackwell
- What it is
- 48 GB cards (RTX 6000 Ada, L40S) take 27–32B at 8-bit or a 70B at Q4. The RTX PRO 6000 Blackwell (96 GB GDDR7, 1,792 GB/s) holds gpt-oss-120b or a 70B at 8-bit on one card.
- When to use it
- Small-team production inference and QLoRA of 30–70B models.
- Key figures
- RTX PRO 6000 Blackwell: 96 GB GDDR7 with ECC, 1,792 GB/s.Source: NVIDIA RTX PRO 6000 spec page
80–141 GB data-centre GPU
H100, H200
- What it is
- H100 (80 GB HBM3, 3.35 TB/s) and H200 (141 GB HBM3e, 4.8 TB/s): gpt-oss-120b or Llama 4 Scout at Int4 on one GPU, 70B at 16-bit across two, high concurrency thanks to HBM bandwidth.
- When to use it
- Rented by the hour for production serving and fine-tuning.
- Key figures
- H200: 141 GB HBM3e at 4.8 TB/s.Source: NVIDIA H200 spec page
180–288 GB accelerators
B200, B300, MI355X
- What it is
- NVIDIA B200 (180 GB HBM3e in HGX systems) and B300 (288 GB), AMD MI300X (192 GB, 5.3 TB/s) and MI355X (288 GB, 8 TB/s): a 284B MoE at 4-bit fits on one or two cards, and Blackwell adds native NVFP4.
- When to use it
- Large MoE models with fewer GPUs and less cross-GPU traffic.
- Key figures
- AMD Instinct MI355X: 288 GB HBM3E, 8 TB/s.Source: AMD Instinct MI350 series spec page
8-GPU nodes and clusters
1 TB and beyond
- What it is
- 8 × H200 gives 1,128 GB, 8 × B300 about 2.3 TB. Mistral Large 3 at FP8, Muse Spark Open at NVFP4 and DeepSeek-V4-Pro fit one node; Qwen3.8-2.4T and Kimi K3 need multiple nodes or aggressive quantisation.
- When to use it
- Frontier-scale open models for a company that must not use an API.
Unified memory
DGX Spark, Mac Studio
- What it is
- NVIDIA DGX Spark (128 GB LPDDR5x at 273 GB/s) and Mac Studio with M3 Ultra (up to 512 GB at 819 GB/s) share memory between CPU and GPU: huge capacity for the price, but several times less bandwidth than HBM, so large models load yet generate slowly.
- When to use it
- Running a 100–400B MoE on a desk for evaluation, one user at a time.
- Key figures
- Mac Studio M3 Ultra: up to 512 GB of unified memory, 819 GB/s.Source: Apple Mac Studio tech specs
Licences
What you may ship
- What it is
- The licence decides whether you can sell a product on the model, fine-tune and redistribute it, and under what name. Families do not share one licence: Mistral, Qwen and GLM each use several.
- When to use it
- Read the LICENSE file of the exact checkpoint, then the acceptable-use policy it links.
Apache 2.0
Any purpose, keep notices
- What it is
- Use, modify and distribute "for any purpose" with no field-of-use limits; keep the notices, state your changes, and accept the patent clause. Used by gpt-oss (plus its usage policy), Gemma 4, Qwen3, Qwen3.8-27B, Muse Glimmer, Mistral Large 3, Small 4, Ministral 3 and Granite 4.1.
- When to use it
- The first filter for SaaS, client deliverables and redistributed fine-tunes.
MIT
Without restriction
- What it is
- Use "without restriction" on condition that the copyright and permission notice travel with the weights. Used by DeepSeek V3.2 and V4, GLM-5 and GLM-5.3-Flash, and Xiaomi MiMo-V2-Flash.
- When to use it
- Frontier-scale MoE with no commercial strings.
Modified MIT
MIT plus a display clause
- What it is
- MIT with an extra condition for very large users. Kimi K2: products above 100M monthly active users or $20M monthly revenue must show "Kimi K2" in the interface. Mistral Medium 3.5 uses its own modified MIT with revenue exceptions.
- When to use it
- Fine for almost everyone; check the threshold if you might cross it.
Llama 4 Community
700M MAU, "Built with Llama"
- What it is
- Commercial use allowed, but: companies above 700 million monthly active users on the release date must ask Meta for a licence; products show "Built with Llama"; derived model names start with "Llama"; the Llama 4 Acceptable Use Policy applies; and the multimodal rights are not granted to individuals or companies domiciled in the EU (their end users are exempt).
- When to use it
- EU companies building on Llama 4 vision: check the multimodal clause first.
Muse Model License
Notice plus reseller clause
- What it is
- Meta's custom licence for Muse Spark Open: products carry a "Built with Muse Spark Open" notice, and model-as-a-service providers above a revenue threshold need a written agreement with Meta (reported; sources disagree on the amount).
- When to use it
- Internal use and most products are fine; an inference-API business needs legal review.
Revenue-gated custom licences
Free until you are big
- What it is
- The 2026 pattern: weights free to use, but large model-as-a-service resellers need a separate deal. Reported thresholds: Qwen3.8-Max weights above $50M a year, MiniMax M3 above $20M revenue (with "Built with MiniMax M3"), and Kimi K3 for large providers; Mistral Large 4 requires a commercial licence for any model-service reseller.
- When to use it
- Using the model inside your own product is usually fine; selling raw inference is what triggers the clause.
OpenMDW 1.1
NVIDIA Nemotron 3
- What it is
- The open model licence of Nemotron 3 Ultra and Nemotron 3.5; it allows commercial use. Earlier Nemotron 3 releases used the NVIDIA Open Model License, so check which one a checkpoint ships with.
- When to use it
- NVIDIA-stack deployments that want commercial terms without revenue gates.
Licence checklist
Six questions
- What it is
- 1. May I use it commercially? 2. Is there a user or revenue threshold? 3. Must I show attribution or name derivatives a certain way? 4. May I redistribute modified weights? 5. Which acceptable-use policy is attached? 6. Are some rights withheld by region (Llama 4 multimodal in the EU)?
- When to use it
- Run it per checkpoint and keep the answers with the model version in your registry.
Trading use casesText in, features out
- What it is
- In trading, LLMs turn unstructured text (news, filings, calls, social posts) into structured features, help write and review strategy code, and assist research. Prices, signals and execution stay with numeric models and deterministic code.
- When to use it
- Put the LLM before the model, never in the order path.
Why self-host for trading
The strategy is the IP
- What it is
- Prompts describe positions, rules and research ideas: the firm's intellectual property. An open model on your own GPUs keeps it inside, with fixed weights that do not change behind a version name, which also makes backtests reproducible.
- When to use it
- Strategy research and anything mentioning live positions; public-news summaries can still use an API.
News and sentiment labels
Classify at volume
- What it is
- Classify each headline or post by asset, event type and direction with a decision model or a small local model (Qwen 8B, gpt-oss-20b), and store the label with its timestamp as a feature.
- When to use it
- Thousands of items a day on a 16–24 GB GPU; Grok when you need X posts in real time.
- AI angle
- Calibrated probabilities from a decision model can gate which labels reach the feature store.
Filings and earnings calls
Structured extraction
- What it is
- Extract guidance, risk factors and management tone from filings and call transcripts into JSON with a schema, using a long-context model (Qwen3.8-27B at 262K, Llama 4 Scout) and validating every number against the source text.
- When to use it
- Fundamental features for medium-horizon strategies.
Research copilot
Read, compare, summarise
- What it is
- An agent that reads papers, reports and your own research notes, answers with citations and drafts hypotheses for a person to test. Reasoning models (DeepSeek-V4, gpt-oss-120b) self-hosted, or a frontier API for public material.
- When to use it
- Faster literature review; never an automatic source of trades.
Strategy code assistant
Local coding model
- What it is
- Qwen3-Coder or a Qwen3.8 model on your own GPU writes backtest code, indicators and data pipelines without sending the strategy to a third party; tests and a person review every change.
- When to use it
- Quant teams whose code must not leave the network.
Signals stay numericTrees, not tokens
- What it is
- Price and order-book features go to LightGBM, XGBoost or similar models with walk-forward validation; LLM-derived labels join them as extra columns. Latency-critical paths run in milliseconds without any LLM call.
- When to use it
- The signal model; the LLM only enriches its inputs.
Look-ahead bias
The model knows the future
- What it is
- A model trained on text up to its knowledge cutoff has read what happened after your backtest dates. Labels it produces for 2024 news may encode 2025 outcomes, inflating backtest results.
- When to use it
- Backtest only on periods after the model's cutoff, or keep the same pinned weights for labelling and live use.
Regulators on AI
Same rules apply
- What it is
- ESMA (30 May 2024): firms using AI remain fully responsible under MiFID II for organisation, conduct and client best interest. FINRA Regulatory Notice 24-09 (27 June 2024): existing supervision, communications and record-keeping rules apply to generative AI. The SEC withdrew its predictive data analytics proposal on 12 June 2025.
- When to use it
- Log prompts, outputs and model versions like any other record when the output reaches a client.
- Key figures
- Firms using AI stay fully responsible under MiFID II.Source: ESMA public statement, 30 May 2024
Human approval and limits
Risk stays deterministic
- What it is
- Position limits, kill switches and order validation are code, not prompts. An agent may propose a trade; a rule engine and, for anything material, a person approve it.
- When to use it
- Every system where model output can move money.
Restricted content
Policies, guards, law
- What it is
- Frontier APIs refuse or limit some categories by policy. Self-hosting removes the vendor's filter but not the licence's use policy or the law: you become the operator responsible for what the model produces.
- When to use it
- Sensitive but legitimate work that a general-purpose API is tuned to refuse.
API usage policies
The vendor decides
- What it is
- OpenAI's usage policies set universal rules (protect people, respect privacy, keep minors safe, human review for high-stakes automation). Anthropic's Usage Policy adds extra requirements for high-risk uses such as legal, health and financial advice. Both can change and both apply to every request.
- When to use it
- Check the policy before designing a product around an API.
Use policies in licences
Llama 4, Gemma, gpt-oss
- What it is
- Open weights can still carry a use policy. The Llama 4 Acceptable Use Policy bans illegal activity, harassment, unlicensed professional practice, weapons, self-harm content and deception; the Gemma Prohibited Use Policy (for Gemma releases under the Gemma terms) also bans sexually explicit content made for gratification; gpt-oss ships Apache 2.0 with a usage policy.
- When to use it
- A policy in the licence binds you even on your own GPUs.
Permissive licences
No field-of-use limits
- What it is
- Apache 2.0 and MIT place no restriction on what the model is used for. That moves every content decision to you and to the law of the country where you operate; the base model's own safety training still shapes what it will write.
- When to use it
- Products with their own documented content policy and moderation.
Legitimate sensitive uses
Where self-hosting helps
- What it is
- Security red-teaming and malware analysis in a sandbox, medical and legal drafting on private records, moderation training data that must contain the harmful examples it learns to catch, and mature fiction for verified adults where it is lawful.
- When to use it
- A documented purpose, an authorised user base and logs: the conditions that make it defensible.
Hard lines
Illegal under any licence
- What it is
- Child sexual abuse material, help with weapons capable of mass harm, non-consensual intimate imagery of real people, fraud and targeted harassment are illegal or banned regardless of the licence; every use policy above lists them, and hosting the model yourself only makes you the one accountable.
- When to use it
- Block these at input and output with a guard model, whatever the product.
Guard models
Classify input and output
- What it is
- Small classifiers that sit in front of and behind the main model: Llama Guard 4 (12B, multimodal, MLCommons hazard categories), gpt-oss-safeguard 20B and 120B (reason over a policy you write, Apache 2.0), Qwen3Guard (0.6B, 4B and 8B, streaming variant, 119 languages, Apache 2.0), IBM Granite Guardian (harm, jailbreak and RAG groundedness, Apache 2.0) and ShieldGemma 2 (4B image safety).
- When to use it
- Your own content policy, enforced locally: gpt-oss-safeguard reads it as a prompt, so changing the policy needs no retraining.
- AI angle
- A 4–8B guard adds a few GB of VRAM and milliseconds per request.
Operator duties
Age, logs, disclosure
- What it is
- Verify age where content requires it, keep logs of prompts and outputs, offer reporting and human review, and label synthetic content: the EU AI Act (Regulation 2024/1689, Article 50) requires machine-readable marking of AI-generated content and disclosure of deepfakes.
- When to use it
- Any public product on self-hosted weights, especially in the EU.
- Key figures
- Providers must mark AI-generated audio, image, video and text in a machine-readable format.Source: EU AI Act, Article 50
Sources, read 2026-11-11: the model cards and announcements of each lab on Hugging Face and their own sites; the AI Board report "AI Models for Executives 2026" (26 October 2026) for flagship names, dates and prices; llama.cpp, NVIDIA and Hugging Face documentation for quantisation, KV cache and MoE memory; NVIDIA, AMD and Apple spec pages; the Apache 2.0 and MIT licence texts, the Llama 4 and Gemma use policies, the OpenAI and Anthropic usage policies; ESMA (30 May 2024), FINRA Regulatory Notice 24-09, the SEC (12 June 2025) and Regulation (EU) 2024/1689. "Reported" marks figures confirmed only by press coverage; memory sizes computed as parameters × bits ÷ 8 are estimates for weights alone. Read the licence file of the exact checkpoint before you deploy it.