Building a Public AI FAQ Bot on Discord: A Solo Founder's 4-Week Iteration Log

Building in public means sharing the structural compromises, API bottlenecks, and architectural shifts that happen when software meets actual users.

This log details the four-week journey of building and deploying an AI-powered FAQ bot for a public Discord community. The goal was simple: deflect repetitive support tickets by answering user questions directly from documentation, using Retrieval-Augmented Generation (RAG).

Here is the exact technical roadmap, the mistakes made, and how they were resolved.

---

Week 1: The Basic RAG Stack and Discord Gateway

The initial architecture focused on getting a functional prototype running on a local machine. The core requirement was to ingest a documentation folder (Markdown files) and query it via a Discord slash command.

Technical Stack Decisions

• Bot Framework: discord.py (Python). It is mature, stable, and handles asynchronous events natively.
• Vector Database: pgvector running on a local PostgreSQL instance. While Pinecone offers a managed service, pgvector allows you to keep your data structure relational and hosted within a single database.
• LLM and Embeddings: OpenAI text-embedding-3-small for vector generation and gpt-4o-mini for response generation.

Implementation Steps

1. Parsing and Chunking: The Markdown files were parsed using a simple recursive character text splitter from LangChain, targeting a chunk size of 800 characters with a 100-character overlap.
2. Database Schema: A simple table was created in PostgreSQL:

   CREATE EXTENSION IF NOT EXISTS vector;
   CREATE TABLE doc_chunks (
       id SERIAL PRIMARY KEY,
       content TEXT,
       metadata JSONB,
       embedding VECTOR(1536)
   );
   

3. The Discord Listener: A basic slash command /ask was registered. When triggered, it converted the user's query into an embedding, ran a cosine similarity search in PostgreSQL, and sent the top 3 chunks to the LLM as context.

Lessons from Week 1

• Slash Command Timeouts: Discord expects an acknowledgment of a slash command within 3 seconds. Embedding generation and LLM streaming frequently exceeded this threshold, causing "Interaction Failed" errors.
• Solution: Call await interaction.response.defer() immediately upon receiving the command to buy 15 minutes of execution time, then use interaction.followup.send() to deliver the final answer.

---

Week 2: Thread Context and Message History

A single-turn /ask command is insufficient. Users naturally ask follow-up questions. Week 2 was dedicated to moving the bot from a stateless command runner to a stateful conversational partner.

Implementing Thread-Based QA

Instead of relying solely on slash commands, the bot was refactored to listen to new threads created in a specific #help channel.

[User posts question in #help] 
       │
       ▼
[Bot creates a Public Thread]
       │
       ▼
[Bot reads last 5 messages in thread] ──► [Generates Search Query via LLM]
                                                   │
                                                   ▼
[Sends response in thread] ◄── [RAG Pipeline runs with Context]

Managing State and Token Usage

To prevent token costs from ballooning, the bot does not feed the entire thread history to the vector search. Instead:

1. We retrieve the last 5 messages in the thread.
2. We pass these messages to a lightweight model (like gpt-4o-mini or a fast alternative via Groq) with a prompt instructing it to condense the thread history into a single, optimized search query.
3. We run the vector search using this optimized query, not the raw user message.

Cost and Tooling Alternatives

OpenAI’s API is highly reliable, but costs can accumulate with active communities.

• Alternative: For a completely free, local setup during development, you can use Ollama running llama3 locally.
• Cheap Production Alternative: Groq offers low-latency hosting for open models like Llama 3 at highly competitive rates. Always check current pricing for both OpenAI and Groq before deploying, as token pricing fluctuates frequently.

---

Week 3: Guardrails, Noise Filtering, and Hallucinations

By Week 3, the bot was live in a beta channel. Users quickly exposed the limitations of naive RAG: asking off-topic questions, trying to prompt-inject the bot, and receiving highly confident, incorrect answers (hallucinations) when the documentation lacked the answer.

Implementing a Relevance Threshold

If a user asked, "What is your favorite movie?", the vector database still returned the most mathematically similar documentation chunks (e.g., something about video rendering files), and the LLM tried to synthesize an answer from it.

We introduced a two-tier defense:

1. Cosine Distance Filter: If the cosine similarity score of the top-retrieved chunk was below 0.65, the database query was discarded.
2. System Prompt Guardrails: The system prompt was hardened to restrict the model's creative freedom.

You are a technical support assistant for Product X. 
You must answer the user's question using ONLY the provided context blocks.
If the context blocks do not contain the answer, state clearly: "I cannot find that information in our current documentation. Let me tag a moderator for you."
Do not make up commands, URLs, or features.

Handling the "No Answer" Fallback

When the bot failed to find an answer, it automatically added a specific Discord tag (e.g., @Support-Needed) to the thread, alerting human moderators to step in. This turned a frustrating AI failure into a functional ticketing system.

---

Week 4: Caching, Telemetry, and Production Scaling

With daily active users increasing, redundant questions (e.g., "How do I reset my password?") started driving up API costs. Week 4 focused on optimization and monitoring.

Implementing a Redis Cache

Before running a vector search and calling the LLM, we implemented a semantic cache using Redis.

Step	Action	Outcome
1	User submits query	Vectorize query
2	Query Redis Index	Look for existing embeddings within a high similarity threshold (0.95)
3	Cache Hit	Return cached JSON response instantly (0ms LLM cost)
4	Cache Miss	Run full RAG pipeline, store result in Redis with a 24-hour TTL

Monitoring and Logging

To track performance, we logged every interaction to a PostgreSQL telemetry table, tracking:

• User query
• Retrieved chunk IDs
• LLM latency (milliseconds)
• Token count
• User feedback (using Discord's emoji reactions: 👍 / 👎)

This feedback loop allowed us to identify gaps in our documentation. If multiple users received answers that got a 👎 reaction, we could look up those specific threads, update the Markdown documentation, and re-index the database.

---

Key Metrics and Takeaways

After 4 weeks of iteration, the system achieved the following metrics:

• Deflection Rate: ~42% of support threads were resolved by the bot without human intervention (measured by threads closed without a moderator response).
• Average Latency: Reduced from 4.2 seconds (Week 1) to 1.1 seconds (Week 4, utilizing Redis caching).
• Cost: Average cost per resolved thread was approximately $0.012 using gpt-4o-mini.

Building a production-ready Discord bot is less about the sophistication of the LLM and more about handling the chaotic nature of user inputs, managing API rate limits, and building tight feedback loops to continuously improve your knowledge base.

---