This Tiny Qwen 27B Model Runs on a 12GB GPU

Highlights

• Saluki 27B compresses a 55-gigabyte original model down to 7.9 gigabytes while outperforming the full-sized model on specific tool-calling benchmarks.

• The model achieves weights at roughly 2.35 bits per weight through aggressive quantization.

• A community tester running Saluki on a 12-gigabyte RTX 3060 achieves 22 tokens per second with 170,000 context length.

• Saluki scores 79 on the AIME 2025 math benchmark compared to the original model score of 96.7.

• Independent tool-use tests show 200 consecutive tool calls returning clean, parsable outputs without text leakage.

Timeline

1. 00:00 - 02:45

   Model Overview and Compression Mechanics

   Standard local deployment of large models requires massive hardware, such as 64-gigabyte Macs or 80-gigabyte enterprise setups. Saluki compresses this requirement significantly by discarding general performance metrics like perplexity and math reasoning to focus entirely on tool calling and argument formatting.

   • Saluki 27B reduces the 55-gigabyte original model footprint to 7.9 gigabytes.
   • The model targets local AI agents by prioritizing tool-calling accuracy over general capabilities.
   • Weight quantization drops precision down to roughly 2.35 bits per weight.
2. 02:45 - 04:14

   Benchmark Results and Unknown Training Details

   The foundation relies on RCO quantization techniques that optimize size budgets per tensor. Although the resulting tool-calling score exceeds the original full model baseline, the lack of complete training documentation makes it difficult to separate the impact of the quantization technique from the subsequent fine-tuning.

   • Saluki builds upon an existing 2-bit quantization from ISTA Das Lab.
   • The model scores 88 on Underdog's tool-calling benchmark compared to 84 for the full-sized model.
   • Full training details and full-precision checkpoints remain unreleased by the creators.
3. 04:14 - 06:17

   Hardware Compatibility and Inference Speeds

   Alternative low-bit alternatives require specialized forks like Prism ML's custom Llama CPP, whereas Saluki integrates smoothly with standard local stacks. Community variants incorporating multi-token prediction heads further accelerate generation speeds on consumer hardware like the RTX 3080.

   • Saluki operates using stock Llama CPP and Ollama without requiring custom forks.
   • A 12-gigabyte RTX 3060 runs the model at 22 tokens per second with a 170,000 token context.
   • Independent tests on a DGX Spark yield an 86 score on single-turn BFCL benchmarks.
4. 06:17 - 08:45

   Performance Trade-offs and Practical Limitations

   While the file size is small, pre-fill bottlenecks can make real-world agent execution slower during context processing. Users requiring robust coding and math capabilities should utilize a standard 4-bit quantization if hardware permits, reserving Saluki specifically for lightweight, local tool-calling agents.

   • Math benchmark scores drop from 96.7 on the original model down to 79 for Saluki.
   • Agent execution loops experience slower pre-fill phases despite the smaller file size.
   • Optimal tool-calling performance requires disabling thinking mode and setting temperature to zero.