Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish

Highlights

• A natural experiment involving a sudden shift in Grok usage across Cursor users showed no statistical correlation with changes in pull request throughput.

• Tool and model choice account for less than 1% of the variance in developer PR throughput.

• Skill utilization accounts for 3% of productivity variance, and achieving over 30 skill runs per week yields a 30% output bump.

• Token consumption accounts for 9% of productivity variance, where the top 10% of consumers double their output compared to the median.

• Organizational environment and team culture account for 12% of output variance, creating a 2x productivity spread between top and bottom companies at fixed token spend.

• The individual engineer accounts for roughly 75% of productivity variance, with a 3.5x spread between the 10th and 90th percentile developers.

• Top-performing engineers remain largely unchanged after AI adoption, as only 44% cycle out of the top productivity quartile every six months.

Timeline

1. 00:00:00 - 01:07:18

   Introduction and the Golf Analogy

   The speaker introduces the foundational premise that unsatisfactory AI outcomes stem from human and organizational factors rather than the model itself. A comparison to golf equipment illustrates the intuition that better tools should help, but historical PGA Tour data confirms that technology upgrades only tell part of the story. Jellyfish telemetry data provides an empirical foundation to measure what actually moves the needle in software engineering output.

   • Golfers often blame equipment for poor performance, mirroring how engineers attribute productivity bottlenecks to AI models.
   • PGA Tour driving data shows a 20% increase in distance since 1980, proving that equipment technology genuinely impacts performance.
   • Data from Jellyfish tracks hundreds of thousands of developers across 1,300 companies to analyze real-world AI engineering workflows.
2. 01:07:18 - 04:54:00

   The Natural Experiment on Model Choice

   When Cursor promoted Grok in its user interface, it created a natural experiment involving thousands of developers shifting model preference rapidly. Plotting individual PR throughput changes against the shift to Grok demonstrated a flat statistical distribution centered around zero. Subsequent analysis across various model and harness switches confirmed that raw model choice does not significantly alter developer output.

   • Cursor's UI updates caused a massive surge in Grok model adoption among enterprise engineers.
   • A statistical analysis of merged PRs per engineer against Grok adoption percentages revealed no correlation with output changes.
   • Switching between model families, autocomplete, and agentic modes yields statistically insignificant changes in PR throughput.
3. 04:54:00 - 07:18:00

   Deconstructing Engineering Productivity Levers

   To find what truly drives productivity, a curated dataset of 50,000 engineers across 447 companies is analyzed using a statistical variance decomposition model. The resulting map assigns specific percentages to each factor influencing weekly merged PRs per engineer. While tool selection captures disproportionate attention on social media, it plays a negligible role in actual output compared to human and organizational variables.

   • A statistical variance model breaks down PR throughput determinants into tools, skills, consumption, environment, and the individual.
   • Tools account for less than 1% of output variance across engineering teams.
   • Skill utilization accounts for 3%, consumption accounts for 9%, company environment accounts for 12%, and the individual accounts for roughly 75%.
4. 07:18:00 - 11:16:00

   The Impact of Skills and Token Consumption

   Analyzing skill execution reveals that advanced workflow customizations are underutilized in real-world environments. However, when engineers actively incorporate skills into their workflow, productivity increases significantly. Similarly, token consumption acts like rocket fuel, yielding higher output with a saturating curve that rewards higher active usage days and token volume.

   • Nearly half of all developer weeks involve zero skill use, and only 12% utilize skills at a modern AI engineering standard.
   • Executing over 30 skill runs per engineer per week produces a 30% output bump while holding other factors constant.
   • Token consumption exhibits diminishing returns, where the top 10% of token spenders double output compared to the median at a 10x token cost.
5. 11:16:00 - 16:27:00

   Organizational Environment and Individual Variance

   Organizational factors such as codebase health, code review processes, and team culture heavily influence output even under fixed token consumption. The individual engineer remains the single largest determinant of productivity. Historical tracking confirms that AI does not level the playing field or reshuffle the rankings; craft and individual capability still dictate engineering success.

   • Company environment and team culture account for 12% of output variance, generating a 2x productivity spread between top and bottom companies.
   • Individual developer variance is even larger, showing a 3.5x productivity spread between the 10th and 90th percentiles.
   • Longitudinal data from Jellyfish shows that top-performing engineers before AI remain top performers after AI adoption.