From Scratch to SOTA: Training a 3B State-Space Vision Model — Krishna Prasad Srinivasan, Sarvam

India remains largely unrepresented in machine-readable datasets because historical and modern documents have never been systematically digitized. Extracting text from Indian languages involves multi-unicode character fusion where visual word shapes differ significantly from underlying machine representations. Sarvam addresses this gap as a sovereign foundation model company building native voice, text, and vision capabilities.

Standard vision-language models process full document pages containing 5,000 to 10,000 visual tokens, resulting in massive computational overhead due to quadratic sequence interactions. By switching to a 3B parameter State-Space Model (SSM) architecture focusing on block-level extraction, inference costs drop significantly while retaining processing accuracy.

A strong linguistic prior allows the model to resolve smudged or ambiguous text visually, similar to human reading context. After continual visual pre-training and fine-tuning across math, tables, handwriting, and 22 languages, stage four applies RL. Because document extraction correctness can be validated deterministically using unit tests for character rates and table structures, RL provides a scalable performance boost.

Sarvam Vision powers Akshar, an agentic workbench featuring human-in-the-loop verification, block-level grounding, and confidence scoring. Sovereign data control is critical for government and enterprise adoption where privacy rules prevent off-shore data transfer. The team is also preparing to open-source the Sarvam Indic Benchmark covering historical and modern documents spanning from the 1800s to present day.