The Model Swap Workshop: GPT-5.5 vs Kimi vs DeepSeek vs Mistral — Pamela Fox, Microsoft

Highlights

• GPT-5.5 and Kimi K2.6 count letter occurrences in zero-shot prompts correctly, whereas DeepSeek V4 Flash and Mistral Large 3 require explicit chain-of-thought instructions like word splitting to avoid miscounting.

• Smaller language models answering RAG questions frequently fall back on internal model weights rather than strictly refusing to answer when context documents omit target facts.

• Multi-turn agent loops highlight major latency disparities: GPT-5.5 completes compound operations in a single tool call or parallel turns, while DeepSeek V4 Flash takes four sequential calls to solve the exact same calculation.

• Mistral Large 3 drops early instructions in agent chains once context windows expand, omitting budget verification and activity steps unless explicitly steered.

• Automating prompt optimization with DSPy and the GEPA algorithm produces tuned system prompts that elevate smaller models to parity with larger models on complex formatting constraints.

Timeline

1. 00:00 - 05:01

   Workshop Environment and Multi-Model Setup

   Participants access models deployed through a Microsoft Foundry project endpoint via shared environment credentials. The architecture allows switching models by toggling variable strings across identical execution scripts. This multi-model agility protects engineering systems against rate limits, vendor outages, and policy cutoffs.

   • Repository dependencies install via UV sync or pre-configured GitHub Codespaces environments.
   • The comparison suite evaluates four deployed models: GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3.
   • Standard industry benchmarks fail to predict how an LLM performs inside specific, end-to-end production application scenarios.
2. 05:01 - 08:36

   Basic Character Counting and Chain-of-Thought Steering

   Counting tests reveal baseline reasoning fidelity across models using OpenAI-compatible response calls. Models without intrinsic numerical accuracy improve immediately when prompted to segment sentences into discrete words and sum intermediate counts. Prompt modifications serve as the first layer of mitigation when swapping to less capable models.

   • GPT-5.5 and Kimi identify 13 letter occurrences correctly on a raw counting prompt.
   • DeepSeek V4 Flash and Mistral Large 3 fail initial zero-shot counts, returning six and eight.
   • Enforcing a per-word breakdown and explicit reasoning trace resolves the miscount on Mistral.
3. 08:36 - 13:05

   RAG Groundedness and Model Weight Hallucinations

   Smaller architectures display a stronger tendency to supplement missing context with pre-training knowledge rather than acknowledging absence of data. Forcing per-sentence source citations nudges non-compliant models closer to acknowledging missing context. Groundedness evaluations remain mandatory when evaluating candidate models for retrieval architectures.

   • A grounded RAG prompt tests negative constraint compliance using a dataset containing only carpenter bee data while asking about honeybees.
   • GPT-5.5 and DeepSeek accurately state that the provided sources lack information on honeybee nest construction.
   • Mistral answers from its internal pre-trained weights despite explicit instructions to cite only provided sources.
4. 13:05 - 17:14

   Multimodal Parsing and Linguistic Vulnerabilities

   Multimodal capabilities are largely fixed within model weights, making prompt engineering ineffective for correcting visual misinterpretations. Shifting prompt languages introduces cross-lingual lexical confusion, breaking species identification in non-English contexts. Production vision pipelines require domain-specific validation in every target language.

   • GPT-5.5 and Kimi identify biological snout differences between crocodilians and locate items in small-font tabular images.
   • DeepSeek V4 Flash does not support image input modalities.
   • Mistral fails table value extraction and misidentifies species, confusing crocodilians when prompted in Spanish.
5. 17:14 - 21:00

   Function Calling Precision and Low-Level Schemas

   Testing single-turn function calling evaluates whether an engine selects appropriate endpoints and adheres to parameter types. Underspecified parameter schemas lead models to include pronouns or extraneous tokens in named entity fields. Explicit prompt constraints like specifying first and last names resolve argument drift across different model families.

   • JSON schemas serve as the prompt engineering surface for function calling.
   • DeepSeek injects unrequested parenthetical strings into tool argument fields.
   • Discrepancies in parameter formatting require tightening JSON schema descriptions to eliminate ambiguity.
6. 21:00 - 24:00

   Tool Calling Loops and Computational Efficiency

   Looping tool interfaces reveal execution efficiency differences that impact end-to-end latency and infrastructure costs. While both models arrive at identical mathematical results, sequential tool execution incurs multiple network roundtrips. Agent architectures must measure total tool invocations alongside final task success.

   • GPT-5.5 batches multi-step arithmetic into a single mathematical expression inside one tool call.
   • DeepSeek V4 Flash invokes four sequential tool calls across separate network roundtrips to compute identical arithmetic.
   • Kimi intermittently generates redundant self-verification tool calls after already achieving the correct solution.
7. 24:00 - 30:29

   Multi-Step Agent Frameworks and Tool Parallelization

   Evaluating travel planning inside LangChain highlights distinct execution paths across model engines. Parallel dispatch dramatically cuts turn duration when querying independent external systems. Smaller models exhibit context decay toward the end of multi-turn flows, necessitating specialized outer execution loops to force completion.

   • GPT-5.5 executes parallel tool requests for flights and hotels on turn one, followed by budget checks and activity queries.
   • Kimi and DeepSeek fail to parallelize tool calls, extending execution turns and overall latency.
   • Mistral halts execution after searching flights and hotels, omitting the final activity search step.
8. 30:29 - 39:51

   Automated Evaluation Pipelines and LLM-as-a-Judge

   Automated evaluation requires isolating test datasets from production drift by capturing edge cases like date math and multi-lingual inputs. Running LLM judges adds latency and compute overhead, meaning comprehensive test suites are ill-suited for per-commit CI runs. Decoupling judge logic from application execution allows engineers to refine evaluation rubrics independently.

   • String and list assertion checks provide lightweight verification for deterministic tool call parameters.
   • LLM-as-a-judge patterns require full traces, tool definitions, and system prompts to evaluate fuzzy outputs.
   • Production retrieval evaluations require 200 to 800 test cases calibrated against human domain experts.
9. 39:51 - 45:15

   Spec-Driven Evaluation with the Assert Framework

   Assert bootstraps evaluation suites prior to production deployments by generating edge-case user scenarios across stratified operational dimensions. Standard metrics like coherence and fluency are obsolete because modern models generate coherent text by default. Custom behavioral evaluators targeted at specific business rules provide actionable signal on agent reliability.

   • Microsoft Research Assert compiles YAML agent specifications into behavior rubrics and simulated user test cases.
   • Assert test suites run across target models to generate comparative behavioral violation scores.
   • GPT-5.5 records the lowest violation counts compared to Kimi and DeepSeek on travel planning benchmarks.
10. 45:15 - 51:18

    Prompt Optimization via DSPy, Ollama, and Anthropic Citations

    Treating prompts as compiled artifacts allows engineering teams to deploy cheaper models while maintaining strict formatting guarantees. The architecture allows pointing code directly at local Ollama instances or Anthropic Messages endpoints with minimal changes. Model swapping workflows transition software development from manual heuristic tuning to programmatic constraint optimization.

    • DSPy uses the GEPA genetic algorithm to hill-climb and compile optimal system prompts against programmatic constraints.
    • Compiling prompts for Mistral via DSPy bridges quality gaps on complex formatting tasks without manual prompt hacking.
    • Anthropic models integrate native citations at the API level by extracting structured reference spans directly from documents.