Sentry Session Replay and User Feedback for AI Chat Interfaces

Error logs tell you what broke. Session Replay shows you what the user experienced. For AI chat interfaces, Session Replay is particularly powerful: when a user submits a bug report about a 'wrong answer' or a broken UI interaction, you can watch their exact session — keystrokes, clicks, network requests, and console errors — instead of asking them to reproduce it.

Combined with Sentry's User Feedback widget, you can let users flag bad AI responses directly in your app, automatically attaching their session replay, console logs, and the full request/response to the feedback report.

Enabling Session Replay

// sentry.client.config.ts import * as Sentry from "@sentry/nextjs"; Sentry.init({ dsn: process.env.NEXT_PUBLIC_SENTRY_DSN, replaysSessionSampleRate: 0.1, // Record 10% of all sessions replaysOnErrorSampleRate: 1.0, // Record 100% of sessions with errors integrations: [ Sentry.replayIntegration({ // Mask sensitive content maskAllInputs: true, // masks all input fields maskAllText: false, // only mask specific elements blockAllMedia: false, // Block specific elements from being recorded block: [ ".sensitive-data", "[data-sentry-block]", ], // Unmask specific elements that are safe to record unmask: [ ".safe-to-record", ], }), ], });

Masking AI Conversation Content

AI conversations often contain sensitive user queries. Configure Replay to mask message content while still recording the UI interaction:

// components/ChatMessage.tsx export function ChatMessage({ message }: { message: Message }) { return ( <div className="chat-message"> {/* Add data-sentry-mask to hide content in replays */} <p data-sentry-mask className="message-content"> {message.content} </p> {/* Keep metadata visible for debugging */} <span className="message-meta"> {message.role} · {message.timestamp} </span> </div> ); }

User Feedback Widget for Bad AI Responses

Let users flag bad AI responses with one click. Each feedback report automatically attaches a link to the session replay:

// components/AIResponseFeedback.tsx import * as Sentry from "@sentry/nextjs"; interface Props { messageId: string; responseContent: string; } export function AIResponseFeedback({ messageId, responseContent }: Props) { const [submitted, setSubmitted] = React.useState(false); const handleBadResponse = async (reason: "wrong" | "incomplete" | "harmful") => { // Capture a Sentry event with full context const eventId = Sentry.captureMessage("User flagged bad AI response", { level: "warning", tags: { feedback_reason: reason, message_id: messageId }, extra: { // Only send first 500 chars — don't log full conversations response_preview: responseContent.slice(0, 500), }, }); // Open Sentry's feedback dialog, pre-linked to the event const feedback = Sentry.getFeedback(); if (feedback) { const dialog = await feedback.createDialog({ formTitle: "What was wrong with this response?", submitButtonLabel: "Send Feedback", messagePlaceholder: "Describe what the AI got wrong...", associatedEventId: eventId, }); dialog.open(); } setSubmitted(true); }; if (submitted) { return <span className="text-sm text-gray-500">Thanks for your feedback</span>; } return ( <div className="flex gap-2 mt-2"> <button onClick={() => handleBadResponse("wrong")} className="text-xs text-gray-400 hover:text-red-500" title="This answer is wrong" > 👎 Wrong </button> <button onClick={() => handleBadResponse("incomplete")} className="text-xs text-gray-400 hover:text-yellow-500" title="This answer is incomplete" > ⚠️ Incomplete </button> </div> ); }

Attaching Custom Context to Replays

Add user and session context so replays are immediately useful when you open them:

// After user authenticates: Sentry.setUser({ id: user.id, email: user.email, // optional — shows in Sentry UI plan: user.plan, // custom attribute }); // Add context for the current AI session: Sentry.setContext("ai_session", { sessionId: aiSession.id, model: aiSession.model, conversationLength: aiSession.messages.length, }); // Tag specific operations for filtering in Sentry: Sentry.setTag("feature", "rag-chat"); Sentry.setTag("model", "gpt-4o-mini");

Viewing Replays

In Sentry → Session Replay, you'll see a grid of recorded sessions. Filter by:

• `has:errors` — sessions where a JS error occurred
• Custom tags like `feature:rag-chat` — only AI chat sessions
• User ID — replay sessions for a specific user who submitted feedback
• Duration — longest sessions (often indicate confusion or bugs)

Click any replay to watch it. The player shows browser events in sync with network requests, console logs, and any captured Sentry events — including your manual `captureMessage()` calls with the 'User flagged bad AI response' message.

Linking Feedback to Traces

// When a user reports a bad response, you can also // capture the full trace context for the LLM call that generated it async function captureAIFeedback(messageId: string, reason: string) { // Retrieve the stored trace ID for this message const traceId = await redis.get(`message_trace:${messageId}`); Sentry.captureMessage("AI quality feedback", { level: "warning", tags: { reason, message_id: messageId }, extra: { traceId }, // Link to the distributed trace for that specific LLM call }); } // In your API route — store the trace ID when generating responses const traceId = Sentry.getActiveSpan()?.spanContext().traceId; if (traceId) { await redis.setex(`message_trace:${messageId}`, 3600, traceId); }

Sentry's free tier includes 50 replays/month and 5k feedback submissions. The Team plan at $26/month includes 500 replays. For most AI apps in early development, the free tier is sufficient to catch the most common UX failures.

Privacy Checklist