Building an in-app AI assistant on Workers AI

Every app gets "how do I..." questions that the docs already answer, asked by people who are mid-task and not about to go read docs. So Emberstack has Ember: a sparkle button next to the theme toggle in /app that opens a chat drawer.

It's modelled on an assistant from another project that called Anthropic and OpenAI directly. Ember uses Workers AI instead, through the same AI binding the rest of the stack uses. No API key, no second bill.

Picking a model

Workers AI has a long catalog. For a chat drawer I wanted 2 things: answers that start streaming straight away, and a model Cloudflare isn't about to retire.

That ruled out the reasoning models (they think before they speak, and you watch a spinner) and landed on @cf/meta/llama-4-scout-17b-16e-instruct, which is pinned in the catalog. It's 1 line in src/lib/assistant/config.ts if you want something else.

Streaming, and the bug in the obvious version

The call itself is short:

const stream = await env.AI.run(ASSISTANT_MODEL, {
  messages,
  stream: true,
  max_tokens: 1024,
})

That returns a stream of server-sent events, one data: line per chunk of text.

The obvious way to read it is to take each network read, split it on newlines, and parse each line. That has a quiet bug: a single event can arrive split across 2 reads. Split on the newline and you get half a JSON object on each side, both fail to parse, and a word goes missing from the answer.

The fix is to keep the unfinished last line and prepend it to the next read:

buffer += text
const lines = buffer.split('\n')
buffer = lines.pop() ?? ''

I tested it by feeding a real model reply to the parser 7 bytes at a time. It rebuilt the sentence exactly.

There's a second wrinkle: older Workers AI models stream {"response": "..."} and newer ones stream OpenAI-style choices[0].delta.content. The parser reads both, so swapping models later doesn't break anything.

Keeping chats private

Conversations and messages live in D1, in 2 small tables. Each conversation belongs to one user in one team.

The easy version of a "load conversation" endpoint takes an ID from the URL and returns the messages. If it never checks whose conversation that is, any signed-in user can read anyone's chat by counting upward.

In Emberstack, every query takes the owner and filters on it:

const owned = (id: number, owner: Owner) =>
  and(
    eq(assistantConversations.id, id),
    eq(assistantConversations.userId, owner.userId),
    eq(assistantConversations.organizationId, owner.organizationId),
  )

Ask for someone else's conversation and you get a 404, same as one that doesn't exist.

Model output is untrusted input

Replies render as Markdown while they stream, using streaming-markdown. It builds DOM nodes (never raw HTML), which is good, but it writes link URLs straight from the model. A model can be talked into writing a javascript: link.

So the renderer is wrapped: links and images only keep http:, https:, mailto: and relative URLs. Everything else becomes #.

What it knows

The system prompt describes each page of the app, plus a few facts about the person asking: name, team, role, plan and the page they're on. It's also told plainly what it can't do: take actions, change settings or read their data. When someone asks it to, it says so and points them to the right page.

The bug you could see

The first version had dark gray text on a blue bubble. Barely readable.

The cause was Tailwind Typography: the prose class sets its own text colors, and those beat the bubble's light text. The fix was to point the prose color variables at currentColor, so replies take the bubble's color in both light and dark themes, and to move replies to a neutral bubble while your own messages took the blue.

Make it yours

The prompt lives in src/lib/assistant/prompt.ts. When you add a feature, add a line describing it there, and Ember can point people to it. The next step I'd take is tools, so it can answer "how many notes do I have?" by actually looking.