What Does an AI Engineer Do Now? - Aparna Dhinakaran, Arize AI
Key Takeaway
AI engineering has shifted from writing individual features to building self-improving loops and agent factories that consume telemetry data, run automated evaluations, and deploy code fixes autonomously.
Highlights
At frontier labs, roughly 5,000 users run thousands of always-on agents concurrently.
An AI engineer builds the systemic loop for writing features, running code reviews, and executing evals rather than just building isolated AI features.
Signal reads traces, identifies telemetry patterns, and generates pull requests automatically as an always-on agent.
Every agent run operates inside a sandbox where all inputs, actions, and outputs are strictly proxied and traced.
AI engineers must build evaluation datasets and evaluate their evaluators to align automated judging with human judgment.
Timeline
The Evolution of the AI Engineer and the Frontier Gap
An analysis of current usage tiers reveals a massive gap between the six billion casual LLM users and the elite five thousand developers at frontier labs running complex agent swarms. Over recent decades, software abstraction layers have accelerated this shift, moving engineering focus away from manual coding toward managing agent factories.
- A small fraction of users at frontier labs operate thousands of always-on agents simultaneously.
- Abstraction layers have rapidly evolved from high-level programming languages to local agents, cloud agents, and internal agent factories.
- The primary role of the AI engineer has shifted toward orchestrating teams and swarms of automated agents.
Building the Autonomous Engineering Loop with Signal
The modern AI engineering workflow requires building automated loops that handle building, evaluating, shipping, and monitoring without human intervention. Arize AI developed Signal to ingest telemetry traces, maintain historical memory of past issues, and propose repository fixes. Each execution runs within an isolated sandbox governed by a single proxy to ensure durability and security.
- AI engineers design systems to handle non-deterministic code reviews and telemetry analysis via agents.
- Signal operates as an always-on agent that reviews trace telemetry to detect anomalies and submit pull requests.
- Agent runs execute inside a secure sandbox where outputs are saved to memory and infrastructure is torn down afterward.
Writing Evals and Aligning Automated Judges
Because non-deterministic tasks require automated verification, AI engineers must build robust evaluation frameworks. Trust in these evaluators is established by constructing golden datasets and verifying that automated judges agree with human experts before deployment.
- Evals test pull requests and prevent regressions across both development and production workflows.
- Teams must evaluate their evaluators by testing judge outputs against golden datasets of human judgment.
- Defining good behavior through rigorous alignment is a core responsibility for AI engineers.
Closing the Gap Between Labs and Enterprises
Successful production deployments rely on continuous tracing of both primary agents and underlying coding agents. By feeding online evaluation data back into the system, AI engineers create self-improving products that continuously optimize themselves based on real-world telemetry.
- Production environments require tracing both application agents and the coding agents that build them.
- Enterprise data is significantly messier than constrained lab environments, requiring systemic optimization.
- AI engineers bridge the enterprise gap by implementing comprehensive tracing, online evals, and self-improving feedback loops.
Description
Aparna Dhinakaran of Arize AI revisits what an AI engineer actually does as the field moves from writing features to running factories of agents. The job now is building the loop: evals that review non-deterministic work, telemetry that agents consume, and agents that act on it. She walks through how Arize built Signal, an always-on agent that reads traces, remembers past issues and opens PRs, covering triggers, sandboxes, durable state, and evaluating the evaluators. Chapters 00:00 Intro: what does an AI engineer do? 00:38 Karpathy's funnel: who's running thousands of agents 01:45 Rising abstraction, from code to agent factories 02:23 Then vs now: build the loop, not the feature 03:31 Signal: an always-on agent for your traces 04:45 Agents are now the main consumers of telemetry 05:19 What triggers an always-on agent 06:01 Inside a run: context, sandbox, proxy 07:11 What persists and what gets torn down 08:13 Designing durable agents 09:14 Model and harness agnostic 10:21 Writing evals, and evaluating the evaluators 12:48 Trace everything, including your coding agents 14:17 Closing the gap between labs and businesses Aparna Dhinakaran, Arize AI Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!