Architecture for Tackling Serverless Cold Starts and Costs in Vercel Eve Agents

When deploying a Vercel Eve-based agent to production, you immediately run into the stateless nature inherent to serverless environments. Once a request finishes, the instance shuts down and the execution state vanishes. However, hitting a database like PostgreSQL every time to restore the session introduces over 100ms of latency and causes DB costs to skyrocket.

To overcome these serverless limitations, here are three architectural patterns used in actual production setups.

1. Lowering Session Restoration Latency Below 50ms with Upstash Redis

Creating a new DB connection for every request in a serverless environment is a shortcut to ruining both infrastructure costs and response times. Placing Upstash Redis—which communicates via HTTP REST API—as a session caching layer solves this issue.

`
[User Request]
│
▼
┌──────────────┐ < 50ms (HTTP REST) ┌────────────────────────┐
│ Vercel Eve │ ────────────────────────> │ Upstash Redis │
│ Agent │ <──────────────────────── │ (Session State Storage)│
└──────────────┘ Session Context Restored└────────────────────────┘
│
│ Compress History (Sliding Window + Summary)
▼
┌──────────────┐
│ LLM Provider │
└──────────────┘

Session data is retrieved within 50ms over the REST API. Replacing direct RDB queries with a cache significantly cuts down Read Capacity consumption.

Evaluation Metric	Traditional RDB (PostgreSQL)	DynamoDB (On-Demand)	Upstash Redis (HTTP REST)
Connection Method	TCP Socket	AWS SDK	HTTP/REST API
Avg Read Latency	50ms - 200ms	10ms - 20ms	1ms - 5ms (Edge < 50ms)
Serverless Fit	Low (Connection Exhaustion)	Moderate (Connection delay exists)	Very High (Supports Scale-to-Zero)
Cost Structure	Hourly billing per provisioned instance	Request unit billing (RCU/WCU)	Command request billing ($0.20/100k)
Primary Use Case	ACID Transactions, Source Storage	Persistent Storage & Search	Session Caching, Rate Limiting, Agent Memory

Keep the session saving and restoration code straightforward.

`typescript
import { Redis } from "@upstash/redis";

const redis = Redis.fromEnv();

interface AgentSessionContext {
userId: string;
currentStep: string;
intermediateThoughts: Record<string, unknown>[];
lastActiveTimestamp: number;
}

export async function restoreSessionContext(sessionId: string): Promise<AgentSessionContext | null> {
const cacheKey = session:context:${sessionId};
const cachedContext = await redis.get(cacheKey);
return cachedContext ?? null;
}

export async function saveSessionContext(
sessionId: string,
context: AgentSessionContext,
ttlSeconds: number = 3600
): Promise {
const cacheKey = session:context:${sessionId};
await redis.set(cacheKey, JSON.stringify(context), { ex: ttlSeconds });
}

As conversations grow longer, token costs continue to rise. Use a pattern that retains only the recent 6 turns in their raw form while summarizing older messages with a lightweight model to place at the top of the prompt.

`typescript
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";

interface Message {
role: "user" | "assistant" | "system";
content: string;
}

export async function compressConversationHistory(
messages: Message[],
recentWindowSize: number = 6
): Promise<Message[]> {
if (messages.length <= recentWindowSize) return messages;

const systemMessage = messages.find((m) => m.role === "system");
const nonSystemMessages = messages.filter((m) => m.role !== "system");

const olderMessages = nonSystemMessages.slice(0, nonSystemMessages.length - recentWindowSize);
const recentMessages = nonSystemMessages.slice(nonSystemMessages.length - recentWindowSize);

const summaryResponse = await generateText({
model: openai("gpt-4o-mini"),
prompt: 다음 대화의 핵심 사실과 결정 사항만 200자 이내로 요약하세요:\n\n${JSON.stringify(olderMessages)},
});

const compressedHistory: Message[] = [];
if (systemMessage) compressedHistory.push(systemMessage);
compressedHistory.push({
role: "system",
content: [이전 대화 요약]: ${summaryResponse.text},
});
compressedHistory.push(...recentMessages);

return compressedHistory;
}

2. Defensive Patterns for External API Latency and Failure

Encountering a 429 (Rate Limit) or 5xx error while calling external tools will crash the entire agent inference process. You need to implement exponential backoff mixed with Full Jitter alongside a circuit breaker.

The exponential backoff formula avoids bottlenecks by mixing in a random variation instead of increasing delay times linearly.

T_{ ext{delay}} = minleft(T_{ ext{max}}, T_{ ext{base}} imes 2^{ ext{attempt}} ight) imes left(0.5 + ext{random}(0, 1.0) ight)

`typescript
export interface RetryConfig {
maxRetries: number;
baseDelayMs: number;
maxDelayMs: number;
}

export async function executeWithExponentialBackoff(
fn: () => Promise,
config: RetryConfig = { maxRetries: 3, baseDelayMs: 200, maxDelayMs: 8000 }
): Promise {
let attempt = 0;

while (true) {
try {
return await fn();
} catch (error: any) {
attempt++;
const statusCode = error?.status || error?.response?.status;
const isUnretryable = statusCode && statusCode >= 400 && statusCode < 500 && statusCode !== 429;

  if (attempt > config.maxRetries || isUnretryable) throw error;

  const calculatedDelay = Math.min(
    config.maxDelayMs,
    config.baseDelayMs * Math.pow(2, attempt)
  );
  const jitteredDelay = calculatedDelay * (0.5 + Math.random());

  await new Promise((resolve) => setTimeout(resolve, jitteredDelay));
}

}
}

When outages persist, use a circuit breaker to immediately block requests (Fail-Fast) and route through fallback logic.

External API Response Status	Circuit Breaker State	Behavioral Mechanism	Agent Handling Result
HTTP 200 OK	Closed	Normal pass-through and success counter increment	Provides external data to the agent normally
HTTP 429 / 503	Closed $
ightarrow$ Open	Executes exponential backoff; switches to Open upon reaching failure threshold	Retries, then opens the circuit
Circuit OPEN State	Open	Blocks external API network requests (Fail-Fast)	Uses alternative Tool or outputs Fallback message
After Cooldown Expiration	Half-Open	Verifies external service recovery via a single probing request	On success, normalizes circuit; on failure, re-blocks circuit

`typescript
export class CircuitBreaker {
private state: 'CLOSED' | 'OPEN' | 'HALF_OPEN' = 'CLOSED';
private failureCount = 0;
private lastStateChange = Date.now();

constructor(
private failureThreshold: number = 5,
private cooldownPeriodMs: number = 30000
) {}

async execute(requestFn: () => Promise, fallbackFn: () => Promise): Promise {
const now = Date.now();

if (this.state === 'OPEN') {
  if (now - this.lastStateChange > this.cooldownPeriodMs) {
    this.state = 'HALF_OPEN';
    this.lastStateChange = now;
  } else {
    return await fallbackFn();
  }
}

try {
  const result = await requestFn();
  if (this.state === 'HALF_OPEN') {
    this.state = 'CLOSED';
    this.failureCount = 0;
    this.lastStateChange = now;
  }
  return result;
} catch (error) {
  this.failureCount++;
  if (this.failureCount >= this.failureThreshold || this.state === 'HALF_OPEN') {
    this.state = 'OPEN';
    this.lastStateChange = now;
  }
  return await fallbackFn();
}

}
}

3. Timeout-Free Asynchronous Human-in-the-Loop Integration

Serverless functions have execution time limits. Keeping a request open while waiting for approval on payments or DB deletions will trigger a timeout error.

`
[Agent Action] ──> Eve Tool (needsApproval: true)
│
▼
[Checkpoint Saved & Instance Terminated]
│
├─> Slack Notification (Interactive Card)
│
[Human Approve] ───────>│ (Webhook POST Callback)
│
▼
[Resume Agent & Proceed Transaction]

Provide needsApproval: true to the Eve tool, halt execution, and save only a checkpoint.

`typescript
import { defineTool } from "@vercel/eve";
import { z } from "zod";

export const deleteDatabaseTool = defineTool({
name: "delete_database", description: "특정 테넌트의 영구 데이터베이스 레코드를 삭제합니다.",
needsApproval: true,
input: z.object({
tenantId: z.string(),
reason: z.string(),
}),
execute: async ({ tenantId }) => {
return await db.tenant.delete({ where: { id: tenantId } });
},
});

Human approval is received via a webhook callback to resume the process.

`typescript
import { createWebhook } from "@vercel/workflows";

export async function handleApprovalWorkflow(event: { approvalId: string; payload: any }) {
const webhook = createWebhook();

await sendSlackApprovalCard({
approvalId: event.approvalId,
callbackUrl: webhook.url,
payload: event.payload,
});

try {
const { approved, userReason } = await webhook.timeout("12h");

if (!approved) {
  await rollbackPreviousSteps(event.payload);
  return { status: "REJECTED", reason: userReason };
}

return await proceedAction(event.payload);

} catch (error) {
await rollbackPreviousSteps(event.payload);
return { status: "TIMEOUT_CANCELLED" };
}
}

4. CI/CD Prompt Verification and Canary Routing

Hallucination issues occurring after prompt modifications are hard to catch through manual testing. Construct your pipeline so that PRs can only be merged if they pass DeepEval metrics.

Evaluation Metric	Acceptance Threshold	Evaluation Criteria
Faithfulness	$ge 0.85$	Presence of factual distortion relative to provided Context
Answer Relevancy	$ge 0.75$	Degree of alignment with the user query's intent
Hallucination Rate	$le 0.10$	Proportion of hallucinations occurring in the test set
Tool Calling Accuracy	$ge 0.90$	Correct selection of OpenAPI spec tools and type compliance rate

Run Pytest in GitHub Actions to block builds if thresholds are not met.

`yaml
name: Eve Agent Prompt Evaluation Pipeline

on:
pull_request:
branches: [ main ]

jobs:
evaluate-agent:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

  - name: Set up Python
    uses: actions/setup-python@v5
    with:
      python-version: '3.11'

  - name: Install Evaluation Dependencies
    run: |
      pip install deepeval pytest

  - name: Run DeepEval Regression Suite
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    run: |
      pytest test_agent_evals.py --deepeval-metric-threshold=0.85

During deployment, integrate Edge Config with middleware to route and apply the new prompt to only 10% of traffic initially.

`typescript
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';
import { get } from '@vercel/edge-config';

export async function middleware(req: NextRequest) {
const res = NextResponse.next();
let variant = req.cookies.get('agent_canary_variant')?.value;

if (!variant) {
const canaryRate = (await get('canary_traffic_rate')) || 0.10;
variant = Math.random() < canaryRate ? 'canary' : 'control';
res.cookies.set('agent_canary_variant', variant, { path: '/', httpOnly: true });
}

res.headers.set('x-agent-prompt-version', variant === 'canary' ? 'v2-canary' : 'v1-stable');
return res;
}

export const config = {
matcher: '/api/agent/:path*',
};