Is a Faster Model Actually Faster? Test the Whole Job — Morgan Suo, MiniMax

Highlights

• A research task analyzing 14 company reports requires the model to find correct documents, compile reporting periods, separate businesses, and explain numbers.

• Sparse attention used in the first three layers of M3.1 Flash uses a small selection step to score text blocks before main attention calculates the uses.

• Changing keys and values from 8 bits to 4 bits in the KV cache reduces data storage and movement while keeping the query at 8 bits.

• When concurrent requests rise from 32 to 64, total output per second grows 1.9 times, but the P95 wait time for the first token jumps from 1.5 seconds to 11.3 seconds.

• Reducing reasoning effort from max to high drops accuracy from 80% to 68% in coding tests.

Timeline

1. 00:00 - 04:21

   Defining the Research Job and Standards

   Switching to a faster model often introduces hidden costs, such as missed sources that require human intervention to fix. A practical research case study tracking AI server demand and printed circuit board suppliers illustrates the complex multi-step demands placed on modern language models. Analysts need reports that clearly distinguish between reported profit and cash flow, pointing directly to relevant financial disclosures and source documents.

   • AI applications must be evaluated on whether they successfully finish the entire user workflow rather than just returning fast model outputs.
   • A standard financial research task involves analyzing supply chain data across 14 company reports to examine whether printed circuit board suppliers turn demand into cash flow.
   • A finished research output requires verifiable sources, consistent reporting periods, and transparent calculations that analysts can check.
2. 04:21 - 09:02

   Optimizations in M3.1 Flash: Sparse Attention and KV Cache

   Model modifications affect performance across different layers and tasks. While sparse attention helps process long context windows by focusing on selected text blocks, critical details like a short cash flow footnote must not be overlooked. Similarly, dropping KV cache precision from 8 bits to 4 bits reduces memory footprint, but overall average scores can mask specific performance drops in certain metrics.

   • M3.1 Flash modifies attention mechanisms, data storage, and reasoning controls to alter computational efficiency.
   • Sparse attention applies a selection step to score document blocks before calculating main attention across long inputs.
   • Quantizing keys and values in the KV cache to 4 bits reduces data movement while queries remain at 8 bits.
3. 09:02 - 13:10

   Mixture of Experts and Speculative Decoding Trade-offs

   Lower precision calculations require precise ordering of scale and restore steps to maintain accuracy. Speculative decoding introduces drafting and checking overhead, meaning overall latency includes verification steps, tool calls, and retries. Serving tests demonstrate that high system concurrency can severely degrade tail latency, making median throughput an unreliable indicator of user experience.

   • Mixture of Experts architecture utilizes four-bit weights and activations for selected experts while keeping shared experts outside the change.
   • Speculative decoding uses a draft model to propose tokens that a target model subsequently checks and corrects.
   • Increasing concurrent requests from 32 to 64 increases throughput by 1.9x but causes the P95 wait time for the first token to jump from 1.5 seconds to 11.3 seconds.
4. 13:10 - 17:35

   Reasoning Control and Complete Workflow Evaluation

   Configuring thinking budgets directly impacts model accuracy and reliability. A complete workflow evaluation must account for all failed attempts, tool calls, and human corrections required to produce an acceptable report. Ultimately, the utility of a faster model depends entirely on whether it successfully delivers verified, usable results within acceptable timeframes.

   • Reasoning control settings dictate the number of tokens allocated for model thinking.
   • Lowering reasoning effort from max to high reduces output token count but drops task accuracy from 80% to 68%.
   • Evaluating model updates requires measuring trust, total cost including failed attempts, and reproducibility across prompt settings and sources.