Streaming AI Chat UIs with the Vercel AI SDK and Provider Switching

Across several clients I kept building the same thing: a chat interface that streams responses token-by-token and can switch between OpenAI and Claude depending on the task or the client's preference. The Vercel AI SDK does a lot of the heavy lifting here, but there's a gap between the demo and a production chat UI that's worth writing down.

Why Streaming Matters

A non-streaming chat response feels broken even when it's fast. The user sends a message, the spinner spins for 8 seconds, then a wall of text appears. Streaming changes the perceived experience entirely. Tokens start appearing in under a second, and the user reads along as the model generates. Same total latency, completely different feel.

The AI SDK's streamText on the server and useChat on the client handle the streaming protocol so you're not manually parsing SSE chunks:

// app/api/chat/route.ts
import { streamText } from "ai"
import { openai } from "@ai-sdk/openai"
 
export async function POST(req: Request) {
  const { messages } = await req.json()
  const result = streamText({
    model: openai("gpt-4o"),
    messages,
  })
  return result.toDataStreamResponse()
}

// client
const { messages, input, handleSubmit, handleInputChange } = useChat()

That's the demo. It works. The production version is where it gets interesting.

Provider Switching Without Rewriting the Frontend

The clean part of the AI SDK is that the model is just a value. openai("gpt-4o") and anthropic("claude-3-5-sonnet") both satisfy the same LanguageModel interface, so switching providers is swapping one line. The streaming, tool calls, and message format all get normalised by the SDK.

I made the model selection dynamic based on the request:

import { openai } from "@ai-sdk/openai"
import { anthropic } from "@ai-sdk/anthropic"
 
function selectModel(provider: string) {
  switch (provider) {
    case "claude":
      return anthropic("claude-3-5-sonnet-20241022")
    case "openai":
    default:
      return openai("gpt-4o")
  }
}
 
const result = streamText({
  model: selectModel(provider),
  messages,
})

The frontend doesn't know or care which provider is behind the stream. A dropdown sets provider, it goes in the request body, the server picks the model. The same useChat hook renders both.

Here's where the abstraction leaks. System prompts and token limits differ between providers. Claude and GPT-4o respond differently to the same prompt, and the prompt that works beautifully for one can underperform on the other. I kept per-provider system prompt variants rather than pretending one prompt fits both. The SDK normalises the transport, not the behaviour.

The Production Gaps

Stopping a stream. Users want to stop a runaway response. useChat exposes a stop function, but you also need to handle the server side. An aborted client request should cancel the upstream provider call so you're not billed for tokens nobody reads. The AI SDK propagates the abort signal if you pass it through, but it's easy to forget and quietly pay for cancelled generations.

Persisting conversations. The demo holds messages in React state, so refresh and they're gone. Production needs persistence. I used the onFinish callback on streamText to save the completed assistant message to the database, paired with loading prior messages into the initial useChat state. The streaming and the persistence are separate concerns. The SDK handles the first, you own the second.

Error states mid-stream. A provider can fail after streaming has started: rate limit, content filter, network drop. A half-rendered message that just stops is confusing. I added an onError handler that surfaces a clear inline error and offers a retry, rather than leaving the user staring at a truncated sentence.

Markdown rendering during streaming. Models emit Markdown, and rendering it incrementally is awkward. A half-streamed code fence or list renders as garbage until the closing token arrives. I rendered raw text during active streaming and switched to full Markdown rendering on completion, which avoided the flicker of half-parsed syntax.

Tool Calls Across Providers

Several of these chat UIs needed tool calling, where the model invokes a function to look up data or take an action. The AI SDK normalises tool definitions across providers, so a tool defined once works whether the model is GPT-4o or Claude:

const result = streamText({
  model: selectModel(provider),
  messages,
  tools: {
    getOrderStatus: {
      description: "Look up the status of an order by ID",
      parameters: z.object({ orderId: z.string() }),
      execute: async ({ orderId }) => fetchOrderStatus(orderId),
    },
  },
})

Both providers support tool calling, but they differ in how aggressively they call tools and how they format arguments. Claude tended to be more conservative about calling tools, GPT-4o more eager. For consistent behaviour I had to tune the tool descriptions and sometimes the system prompt per provider. Again, the SDK normalises the mechanism, not the judgment.

What I'd Tell Someone Starting

The Vercel AI SDK is the right default for streaming chat in a Next.js app. It removes the genuinely tedious parts (SSE parsing, the streaming protocol, provider-specific message formats, tool-call plumbing) and the provider abstraction is real and useful.

But treat "provider switching" as "transport switching." You still own the prompts, the persistence, the error UX, the stream cancellation, and the per-provider behavioural tuning. The SDK gets you to a streaming chat in an afternoon. Getting to a good one is the week after that.