Our 30‑Day Experiment: Building an AI‑Automated FAQ Bot for a Niche SaaS Product

Why We Built an FAQ Bot

Our SaaS product serves a small but highly technical audience. Support tickets repeatedly asked the same questions about API rate limits, webhook payloads, and billing cycles. Answering each manually consumed ~15 hours per week. An automated FAQ bot promised to deflect repetitive queries, freeing time for feature work and reducing response latency for users.

We set a 30‑day deadline to ship a minimum viable bot that could:

1. Understand natural‑language variations of the top 20 FAQs.
2. Return accurate, pre‑written answers or link to documentation.
3. Escalate to a human agent when confidence fell below a threshold.
4. Log interactions for continuous improvement.

Phase 1: Scope and Data Collection (Days 1‑5)

Identify Core FAQs

We exported the last three months of support tickets from Zendesk, filtered for closed tickets tagged “question,” and counted occurrences. The top 20 items covered 78 % of volume. Each entry was rewritten into a concise answer (max two sentences) and stored in a Google Sheet with columns:

• intent_id (numeric)
• training_phrases (semicolon‑separated variations)
• answer (plain text)
• source_url (link to docs or KB article)

Choose a Bot Framework

We evaluated two options that fit the budget and skill set:

Feature	Dialogflow ES (free tier)	Rasa Open Source
Hosting	Fully managed (Google)	Self‑hosted (Docker)
NLU limits	1000 intents, 10k training phrases/month	Unlimited (self‑hosted)
Integration	One‑click webhook, easy Zapier/Make	Requires custom webhook code
Cost	Free until usage exceeds quota	Free (self‑hosted) + server cost

Because we wanted zero infrastructure overhead for the experiment, we started with Dialogflow ES. Rasa remained a fallback if we hit limits.

Phase 2: Model Building (Days 6‑15)

Import Intents

Dialogflow ES accepts a CSV upload for intents. We formatted the sheet as:

Intent,Training Phrases,Response
API rate limit,"What is the limit?;How many calls per minute?",You can make 120 requests per minute per API key.

Each intent received 5‑8 training phrases covering synonyms, typos, and phrasing variations.

Set Up Fallback and Confidence Threshold

Dialogflow returns a confidence score (0‑1). We configured the fallback intent to trigger when the top score < 0.35. The fallback response asked the user to rephrase or offered to create a ticket.

Webhook for Dynamic Answers

Static responses sufficed for most FAQs, but a few required real‑time data (e.g., current subscription tier). We deployed a simple Node.js webhook on Glitch (free tier) that:

1. Received the intent_id from Dialogflow.
2. Queried our internal API (protected by a bearer token stored as an environment variable).
3. Returned a JSON payload with fulfillmentText containing the live answer.

The webhook URL was added in the Dialogflow fulfillment settings and enabled for the relevant intents.

Testing Dialogflow Agent

We used the built‑in simulator to send each training phrase and verified:

• Correct intent mapping (> 0.8 confidence).
• Proper webhook call (checked logs in Glitch).
• Accurate answer returned.

Any mismatches prompted additional training phrases or intent merging.

Phase 3: Channel Integration (Days 16‑22)

Website Widget

Dialogflow provides a demo integration via an iframe snippet. We copied the script, replaced the agent ID, and embedded it in the product’s help center page (built with Hugo). The widget appeared as a floating button in the lower‑right corner.

Slack Bot via Make

Many users asked questions in our public Slack community. We built a Make scenario:

1. Watch new messages in the #support channel (filter for messages not from bots).
2. Send the text to Dialogflow via the “Detect Intent” API.
3. If confidence ≥ 0.35, post the answer as a threaded reply.
4. Else, send a direct message to the on‑call engineer (using Slack’s users.info endpoint).

The scenario ran every minute, used the free Make tier (1000 operations/month), and consumed ~200 ops/day during the test.

Email-to-Bot Gateway

For users who emailed support@, we set up a Zapier zap:

• Trigger: New email in Gmail (labelled “support”).
• Action: Call Dialogflow detect intent.
• Action: If confidence ≥ 0.35, send reply email with answer; else, create a Zendesk ticket and notify the team.

Zapier’s free plan allowed 100 tasks/month; we limited the zap to peak hours (9 AM‑6 PM UTC) to stay within limits.

Phase 4: Monitoring and Improvement (Days 23‑30)

Logging

Every Dialogflow request was logged to a Google Sheet via the webhook (append row). Columns included timestamp, user ID (hashed), intent, confidence, answer source, and whether fallback fired.

Metrics Tracked

Metric	Target	Actual (end of 30 days)
Deflection rate (bot answered without fallback)	≥ 60 %	68 %
Average response time (bot)	< 5 s	3.2 s
Escalation to human (fallback + ticket)	≤ 30 %	28 %
User satisfaction (post‑chat thumbs‑up/down)	≥ 4/5	4.2/5

Iterative Tuning

We reviewed the log weekly:

• Added 12 new training phrases for intents that hovered near the confidence threshold.
• Merged two overlapping intents (“API key expiration” and “Renewing API key”) after noticing frequent confusion.
• Updated the webhook to return a richer payload (including a “Learn more” link) for three intents where users asked for deeper docs.

At day 30 we exported the final agent zip from Dialogflow and archived it in a private GitHub repo for version control.

Results and Lessons

Quantitative Outcome

• Ticket volume dropped from ~150/week to ~48/week (68 % reduction).
• Support engineer time saved ≈ 10 hours/week, redirected to bug fixes and documentation updates.
• Cost: Dialogflow ES remained within the free quota (< 500 requests/day). Make and Zapier stayed on free tiers; the only expense was the $5/month Glitch plan for the webhook (could be replaced by a free Render.com instance).

Qualitative Insights

1. Data quality drives performance – The initial 20‑item list covered most tickets, but edge cases (multi‑part questions) still fell to fallback. A simple “ask for clarification” intent improved user experience.
2. Hybrid static+dynamic approach works – Static answers for evergreen facts; webhook calls for data that changes (subscription, usage).
3. Low‑code integration is viable – Make and Zapier allowed us to connect Slack and email without writing glue code, saving ~8 hours of engineering time.
4. Fallback handling is critical – A polite fallback that offers to create a ticket or rephrase reduced frustration; users rarely abandoned the chat after a fallback.
5. Version control matters – Exporting the Dialogflow agent after each iteration let us roll back a problematic intent merge quickly.

Free/Cheapest Alternatives

If Dialogflow’s free tier ever became a constraint, the drop‑in replacement is Rasa Open Source:

• Deploy via Docker (docker run -p 5005:5005 rasa/rasa:latest).
• Train with the same intent‑example CSV format (converted to Rasa’s YAML).
• Use the built‑in REST webhook for channel integration (Slack, web widget via Rasa Webchat).
• Hosting cost: a $5/month VPS (e.g., Hetzner CX11) easily handles < 10 RPS.

For the workflow automation, n8n (self‑hosted) offers a free alternative to Make/Zapier with similar Slack and email nodes.

Closing Thoughts

A 30‑day timeline is realistic for a narrowly scoped FAQ bot when you:

• Start with a concrete, data‑driven FAQ list.
• Choose a managed NLU service to avoid infrastructure overhead.
• Leverage low‑code platforms for channel hookups.
• Instrument logging from day 1 to guide iterative improvements.

The bot is not a “set‑and‑forget” solution; it thrives on weekly reviews of logs and user feedback. For niche SaaS products with repetitive support questions, this approach delivers measurable time savings with minimal ongoing cost.

On pricing: plans and rates change, and this post was written on a specific date. Check the vendor's current pricing page before you commit to anything.