RAG-Powered AI Support Agent — 62% of Tier-1 Tickets Automated in 5 Weeks

The Situation

Our client runs a B2B project management SaaS with 8,000 enterprise customers. Their support team was handling ~1,200 tickets per week. A ticket breakdown analysis showed:

• 38% — How-to questions answered in the docs
• 24% — Integration and API questions (Zapier, Salesforce, Slack)
• 17% — Billing and account management
• 14% — Bug reports and escalations
• 7% — Feature requests

The first three categories — 79% of volume — were answerable by someone who had read the documentation carefully and knew the product well. The support team was spending most of their time on these, with an average first-response time of 4.2 hours.

Architecture

Knowledge Base Ingestion

We built a documentation ingestion pipeline that:

1. Crawls the Gitbook-hosted documentation (420 pages)
2. Scrapes the public Changelog and release notes
3. Indexes the Zendesk macro library (280 pre-written responses)
4. Pulls the Salesforce and Slack integration guides

Each document is chunked at 512 tokens with 64-token overlap, embedded using text-embedding-3-large, and stored in Pinecone with metadata (source URL, section, last-updated date).

The ingestion pipeline runs nightly — documentation changes are reflected in the AI's answers within 24 hours without any manual intervention.

The Agent

We used LangChain to orchestrate the agent loop. On each incoming Zendesk ticket:

1. Intent classification — GPT-4o mini classifies the ticket into one of 12 intent categories (how-to, integration, billing, bug, etc.)
2. RAG retrieval — top-8 chunks retrieved from Pinecone using the ticket body as the query
3. Context assembly — retrieved chunks + customer account data (plan tier, integrations enabled, prior tickets) assembled into the prompt
4. Response generation — GPT-4o generates a response grounded in the retrieved documentation
5. Confidence scoring — a separate call evaluates the generated response against the source documents; responses below 0.78 confidence are flagged for human review

Zendesk Integration

The agent runs as a Zendesk app. When a new ticket arrives:

• Tickets with intent confidence > 0.85 are responded to automatically (no human in loop)
• Tickets with confidence 0.70–0.85 are sent to a human with the AI draft pre-filled (one-click accept)
• Tickets below 0.70 confidence, or tagged as bug reports, go directly to the human queue

We spent significant time on the deflection vs. resolution distinction. A deflecting response that sends the customer to a documentation page they've already read is worse than no response. We added a "prior resolution" check that queries the customer's ticket history — if they've been sent the same doc link in the past 30 days, the agent escalates.

CSAT Measurement

We A/B tested AI vs. human responses for 6 weeks before full deployment, surfacing CSAT surveys to both cohorts. AI-resolved tickets scored 4.3/5 vs 4.1/5 for human-resolved tier-1 tickets. The AI responses were faster and more precise — linking directly to the relevant documentation section rather than paraphrasing it.

Results

62% of tier-1 tickets now resolved autonomously — no human touch.

4.2 hours → 8 minutes average first-response time for AI-handled tickets.

$298k annual cost reduction — the support team was redeployed to tier-2 and tier-3 work rather than made redundant.

4.3/5 CSAT on AI-resolved tickets — higher than the 4.1/5 baseline for human-resolved tier-1.

5 weeks from kick-off to full production — the clean Zendesk integration surface and Gitbook documentation structure made this faster than typical AI support deployments.

What Made This Work

Three things made this project successful that are often underestimated:

1. Documentation quality matters more than model quality. The client's Gitbook documentation was well-structured, comprehensive, and kept up to date. RAG systems are only as good as what they retrieve. Teams with poor documentation get poor AI support agents — no model compensates.

2. The "prior resolution" check. The most common failure mode in support AI is repeating advice the customer has already received. Adding customer ticket history context halved the escalation rate from frustrated customers.

3. Confidence transparency. Every AI response includes a small "This answer was generated by our support AI, based on our documentation" disclosure. CSAT didn't drop. Customers value speed and accuracy more than knowing a human wrote the response.