HARDWARE FIT CHECK • Qwen 3.6 35B-A3B MoE + NVIDIA GeForce RTX 4090 (24GB GDDR6X)24.1 GB / 24 GB VRAM

Can I Run Qwen 3.6 35B-A3B MoE on NVIDIA GeForce RTX 4090 (24GB GDDR6X)?

Full VRAM footprint analysis, tokens/sec generation benchmark, and copy-paste Ollama & vLLM runtime commands for running Qwen 3.6 35B-A3B MoE (35B) on NVIDIA GeForce RTX 4090 (24GB GDDR6X) (24 GB).

⚠️ Exceeds VRAM — Requires Partial CPU/RAM Offload (Slow)24.1 GB (100%)~68 t/s

Model & Target Hardware

interactive

Model35B MoE

Ultra-Fast 100+ tok/s Coding & Local RAG on 24GB GPUs

Target Hardware24GB • 1008 GB/s

▼ Fine-Tune Precision (Q4_K_M, 32K ctx, Q8_0 KV)quant / context

Qwen 3.6 35B-A3B MoE on NVIDIA GeForce RTX 4090 (24GB GDDR6X)

⚠️ Partial Offload — Slow CPU/RAM Spillover

⚡ PARTIAL OFFLOAD

Required VRAM

24.1 GB

100% of hardware capacity (24 GB)

Est. Decode Speed

~68 tok/s

3B active params @ 1008 GB/s

Available Headroom

-0.1 GB

Needs +0.1 GB more VRAM

Memory Allocation & Headroom

24.1 GB allocated of 24 GB capacity

100% utilized

Weights: 21 GB

KV: (32K): 1.61 GB

Overhead: 1.49 GB

Deficit: -0.1 GB

⚡ Architecture & Throughput Transparency: Single-user decode speed (~68 tok/s) is memory-bandwidth bound at batch size = 1 on NVIDIA GeForce RTX 4090 (24GB GDDR6X) (1008 GB/s). Prompt prefill / TTFT (~986 tok/s) is compute-bound and utilizes GPU Tensor cores. Actual latency varies with FlashAttention-2 and runtime engine (Ollama/vLLM/llama.cpp).

OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama run qwen3.6:35b-a3b

⚠️ Out of VRAM? Don't downgrade model quality.

Out of VRAM

Spin up an on-demand cloud GPU instance in 60 seconds with pre-installed Ollama and vLLM. (Qwen 3.6 35B-A3B MoE @ 24.1 GB VRAM)

RunPod Cloud GPUsBest 1-Click Ollama & vLLM

$0.34/hr (RTX 4070 Ti) – $2.39/hr (H100)

Launch 1-Click GPU Pod → →

Vast.ai GPU MarketplaceCheapest Dual RTX 3090 / 4090

$0.18/hr (RTX 3060) – $0.55/hr (RTX 4090)

Rent Spot GPU → →

Lambda GPU CloudEnterprise 8x H100 Clusters

$1.99/hr (GH200 / H100)

Check H100 Availability → →

Quantization Precision Matrix

32K ctx • KV: Q8_0

Quantization	Weights	KV Cache	Total VRAM	Est. Speed	Fit Status
FP1616 bpw	70 GB	1.61 GB	74.81 GB (312%)	0 t/s	OOM
Q8_08.5 bpw	37.1 GB	1.61 GB	40.76 GB (170%)	0 t/s	OOM
Q6_K6.56 bpw	28.7 GB	1.61 GB	32.06 GB (134%)	~53 t/s	PARTIAL
Q5_K_M5.68 bpw	24.85 GB	1.61 GB	28.08 GB (117%)	~60 t/s	PARTIAL
Q4_K_M4.8 bpw	21 GB	1.61 GB	24.1 GB (100%)	~68 t/s	PARTIAL
FP44.25 bpw	18.55 GB	1.61 GB	21.56 GB (90%)	~344 t/s	FITS
IQ3_XS3.3 bpw	14.35 GB	1.61 GB	17.21 GB (72%)	~420 t/s	FITS