Bye, Bye OpenAI & Anthropic?

Highlights

• The sub-q (sub-quadratic) model targets a 12 million token context window, enough to ingest entire large codebases.

• Initial benchmarks indicate sub-q performs comparably to Opus 4.6 on retrieval tasks and software engineering benchmarks.

• Inference costs for sub-q are reportedly 5% of those for current Opus models.

• Sub-q utilizes Sub-quadratic Selective Attention (SSA), a content-dependent routing method that selectively attends to relevant tokens instead of evaluating every pair.

• Current frontier models rely on dense attention, which exhibits quadratic complexity, making massive context windows computationally prohibitive.

Timeline

1. 00:00 - 03:31

   Sub-q Announcement and Scope

   The announcement introduces a major potential shift in LLM architecture. By enabling a 12 million token window, the model aims to directly load massive codebases or multiple large legal documents, bypassing the constraints that currently force developers to use inefficient retrieval-augmented generation (RAG) systems. If functional, this architecture directly addresses the limitation where models only see small portions of data at a time.

   • Alexander Wedin announced sub-q, a new large language model architecture designed for long-context tasks.
   • The model promises a 12 million token context window, significantly exceeding current industry standards.
   • Inference costs are projected to be 5% of current frontier models like Opus 4.7.
   • Successful implementation could eliminate the need for current workarounds like RAG and sub-agents for handling large datasets or codebases.
2. 03:31 - 07:09

   Benchmark Performance

   The current technical evidence consists of three benchmarks. While the model demonstrates proficiency in retrieving information and generating code, the limited scope of testing and potential for benchmark gaming warrant caution. Despite these limitations, the results suggest the model's ability to act upon retrieved information in long-context scenarios is at least competitive with established frontier models.

   • The model was tested against RULER, MRCRv2, and software engineering benchmarks.
   • Performance on retrieval tasks matches Opus 4.6 in many scenarios, though it lags in specific instances.
   • Software engineering benchmarks show the model generating code at a level comparable to Opus models.
   • Benchmark data remains limited, and the absence of a detailed model card makes comprehensive evaluation difficult.
3. 07:09 - 21:07

   Dense vs. Sparse Attention Mechanisms

   Dense attention requires comparing all prior tokens to generate a new one, causing memory and compute requirements to scale quadratically. Sparse attention attempts to mitigate this by only attending to a subset of tokens. While previous attempts like sliding windows or global summary tokens limited model intelligence, the proposed Sub-quadratic Selective Attention (SSA) claims to achieve efficiency by using content-dependent routing to activate only necessary tokens, theoretically retaining model quality while significantly boosting performance.

   • Current frontier models use dense attention, which exhibits quadratic complexity by comparing every token pair.
   • Quadratic complexity makes massive context windows, such as 12 million tokens, computationally impossible to manage.
   • Sub-q uses Sub-quadratic Selective Attention (SSA) to dynamically route and attend only to tokens carrying actual signal.
   • Previous sparse attention approaches, such as sliding windows or global summary tokens, suffered from significant loss of information and context.