Table of Contents
- Key takeaways
- What is machine learning in customer support?
- How is machine learning used in a customer support chatbot?
- Intent classification: working out what the customer wants
- Retrieval: finding the right help content
- Generation: writing the reply
- Evaluation: measuring whether the chatbot works
- How do you build a customer support chatbot using machine learning?
- How the three routes were ranked
- 1. Build it yourself with an ML library or a model API: best for a chatbot inside your own product
- 2. SiteGPT: best for a website chatbot that answers from your existing content
- 3. A helpdesk's built-in AI agent: best for teams that already work tickets in a helpdesk
- How to build the chatbot in 7 steps
- Worked example: 2,000 conversations a month
- What the model usage costs if you build it yourself
- What the same volume costs on the other two routes
- How to score the test run
- Should you build or buy a customer service chatbot?
- Common mistakes when building a machine learning chatbot
- Frequently asked questions
- Basics
- How is machine learning used in customer service?
- Does a customer service chatbot need machine learning?
- What is the difference between machine learning and NLP in a chatbot?
- Building
- What do you need to build a machine learning chatbot?
- Which type of AI system is best for improving customer service?
- Why does a chatbot forget what the customer said earlier in the conversation?
- Can you build a customer service chatbot without code?
- Models and data
- Do you need to train your own model?
- How much training data does an intent classifier need?
- Is fine-tuning worth it for a customer service chatbot?
- Cost and evaluation
- How much does it cost to run a machine learning chatbot?
- How do you evaluate a customer service chatbot?
- What is a good accuracy score for a chatbot?
- Sources
Created by
Do not index
Created time
Oct 26, 2023 09:01 PM
Machine learning in a customer support chatbot does four jobs: it classifies what the customer wants, retrieves the right help content, generates the reply, and scores the results so the team can improve them.
TL;DR
- A modern customer service chatbot uses machine learning in four places: intent classification, retrieval, generation, and evaluation.
- Most teams never train a model. They index their own content and let a pretrained language model answer from it.
- There are three ways to build one: write it yourself on a model API or ML library, use a no-code retrieval platform, or switch on the AI inside a helpdesk.
- In the worked example below, 2,000 conversations a month cost $17.70 in OpenAI usage on gpt-6-luna. Engineering time is the larger cost of that route.
- Build it yourself when the chatbot is part of your product. Buy when it answers questions about your product.
Key takeaways
Machine learning job | What it does | Typical tool | Number to watch |
Intent classification | Sorts a message into a known request type | Dialogflow intents, or a scikit-learn text classifier | Precision and recall per intent |
Retrieval | Finds the help passages closest in meaning to the question | Embeddings with Sentence Transformers or OpenAI file search | Context recall |
Generation | Writes the reply from the retrieved passages | A language model such as gpt-6-luna | Faithfulness score |
Evaluation | Scores the chatbot on a fixed test set | A labeled question set, scikit-learn metrics, Ragas | Pass rate before and after each change |
What is machine learning in customer support?
Machine learning in customer support is software that learns patterns from examples of customer messages and help content, so it can handle a new message nobody scripted. A rule-based system only follows the steps a person wrote down.
The difference shows up when a customer phrases something in a new way. A rule that looks for the word "refund" misses "can I get my money back". A model trained on examples of refund requests can match both.
Machine learning is used across customer service, and a chatbot is the most visible use. Teams also use it for three other jobs:
- Ticket routing. A classifier reads each new ticket and sends it to the right queue.
- Sentiment detection. A model flags messages that sound upset so a person can step in early.
- Topic analysis. A model groups past conversations so the team can see which questions come up most.
This guide covers the chatbot. The definition, the chatbot types, and what a chatbot can and cannot handle are in the guide to what a customer service chatbot is.
How is machine learning used in a customer support chatbot?
Machine learning is used in a customer support chatbot for four jobs: classifying the request, retrieving the right content, generating the reply, and evaluating the result. Older chatbots used only the first job. Chatbots built on large language models use all four.
Job | Input | Output | Who trains the model |
Intent classification | One customer message | A label such as "refund" or "order status" | You, with example phrases |
Retrieval | The question and your indexed content | The few passages most likely to hold the answer | Nobody. A pretrained embedding model is used as it is |
Generation | The question, the passages, and written instructions | A reply in plain language | The model provider |
Evaluation | A fixed set of test questions | Scores per question type | Nobody. This is measurement |
Intent classification: working out what the customer wants
Intent classification assigns each message to one of a fixed list of request types. The team writes example phrases for every intent, and a model learns to match new messages to the closest one.
Google's Dialogflow documentation recommends at least 10 to 20 training phrases per intent, depending on how complex the intent is. Its built-in machine learning then matches wordings that are similar to those phrases.
A team that writes its own classifier often starts with scikit-learn. Its TfidfVectorizer turns raw text into numeric TF-IDF features, and a classifier learns from those features and the labels.
Intent classification still matters in a language-model chatbot. It is the reliable way to send a refund request to a fixed flow or a person, where a generated answer should not make the decision.
Retrieval: finding the right help content
Retrieval finds the passages in your own content that are most likely to answer the question. It works on meaning, so a question and a passage can match without sharing a single keyword.
The method relies on embeddings. OpenAI's documentation describes an embedding as a list of numbers that represents a piece of text, where a small distance between two lists means the texts are closely related.
The Sentence Transformers documentation describes the same process for open-source models. Every entry in the content is embedded once, the question is embedded at search time, and the closest entries are returned. Its guide says this direct approach works for collections of up to about 1 million entries. A typical help center is far smaller than that.
Retrieval needs no training by you. The embedding model is already trained, and your content is only indexed. What is RAG explains the full mechanism.
Generation: writing the reply
Generation is the step where a large language model writes the reply from the retrieved passages and the team's written instructions. The combination of retrieval and generation is called retrieval-augmented generation, or RAG.
The method was named in a 2020 paper by Patrick Lewis and colleagues. Their models paired a pretrained language model with a searchable index of Wikipedia. The paper reported that these models produced more specific and more factual text than a model working from its own parameters alone.
The model's weights do not change in this step. That is why a chatbot can answer from a page that was published this morning. It is also why the chatbot repeats an out-of-date page with full confidence.
Each request to a language model is independent. OpenAI's documentation states that text generation requests are stateless, so the application must send the earlier messages again on every turn. A chatbot that forgets what the customer said two messages ago is usually missing this step.
Evaluation: measuring whether the chatbot works
Evaluation scores the chatbot against questions with known correct outcomes. It is the only way to know whether a change made the chatbot better or worse.
Each job has its own metric:
- Intent classification uses precision and recall. The scikit-learn documentation defines precision as the ability not to label a negative sample as positive. It defines recall as the ability to find all the positive samples.
- Retrieval uses context recall. The Ragas documentation defines it as how many of the relevant pieces of information were retrieved.
- Generation uses faithfulness. Ragas defines it as the number of claims in the reply that the retrieved content supports, divided by the total number of claims in the reply. The score runs from 0 to 1.
The worked example further down applies these metrics to a 200-question test set. The broader testing routine, including tone checks, is in the guide to chatbot training.
How do you build a customer support chatbot using machine learning?
You build a customer support chatbot using machine learning by choosing one of three routes, loading your content, writing the rules for what the bot must decline, and testing it on a fixed question set. The route decides how much of the machine learning you do yourself.
How the three routes were ranked
The routes are ordered by how much machine learning work the team owns, from most to least. No route is best overall. Each one is judged on four criteria:
- Control. Whether you choose the model, the retrieval method, and the logic.
- Skills needed. Whether the route needs an engineer.
- Time to a working chatbot. Whether the first version takes minutes or a project.
- Pricing unit. Whether you pay per token, per message, or per resolved conversation.
Route | Best for | You choose the model | Needs an engineer | Pricing unit |
1. Build it yourself | A chatbot inside your own product | Yes | Yes | Per token, plus engineering time |
2. SiteGPT | A website chatbot that answers from existing content | No | No | Per plan, with a monthly message quota |
3. A helpdesk's AI agent, such as Fin | Teams that already work tickets in a helpdesk | No | No | Per resolved conversation |
1. Build it yourself with an ML library or a model API: best for a chatbot inside your own product
Building it yourself means your team writes the code that classifies, retrieves, generates, and evaluates. This route comes first because it is the only one where the reader does the machine learning work, and it gives the most control.
What you assemble
- An intent classifier for the requests that need a fixed flow. scikit-learn or Dialogflow covers this.
- A retrieval layer. Sentence Transformers runs open-source embedding models on your own hardware. OpenAI's file search is a hosted tool in the Responses API that searches uploaded files by meaning and by keyword.
- A language model called through an API, with conversation history sent on every turn.
- A chat widget, a handoff to a person, logging, and a test harness.
Pros
- Full choice of model, retrieval method, and business logic.
- Usage is billed per token, and the bill is low at small volumes.
- The chatbot can call your own systems, such as an order database.
Cons
- An engineer has to build it and keep it running.
- The widget, the handoff, and the analytics are your work too.
- Fine-tuning is a shrinking option on OpenAI. Its pricing page says the fine-tuning platform is being wound down and is closed to new users.
Cost. OpenAI lists gpt-6-luna at $0.10 per 1M input tokens and $0.50 per 1M output tokens. gpt-6.1-sol is $2.00 and $10.00. File search costs $2.50 per 1,000 tool calls, and the first 1 GB of storage is free.
The build routes on OpenAI models, including custom GPTs, are compared in the guide to building a GPT chatbot.
2. SiteGPT: best for a website chatbot that answers from your existing content
SiteGPT is a no-code retrieval platform. You add your content, and it handles the indexing, the retrieval, the language model, and the chat widget.
What you get
- Content sources: website URLs, sitemaps, uploaded files, raw text, Google Drive, Dropbox, OneDrive, Box, Notion, GitHub, and YouTube.
- Scheduled content refresh: monthly on Growth, weekly on Scale, daily on Enterprise. Starter is manual refresh only.
- Chat integrations on Growth and above: Google Chat, Messenger, Slack, Crisp, Freshdesk Messaging, and Zoho SalesIQ. Zendesk is on Scale and above.
- Analytics that do some of the evaluation work. Growth lists knowledge gaps and the questions the chatbot could not answer. Scale adds AI-classified topics, sentiment, and resolution outcomes for every conversation.
- 95+ languages.
Pros
- No engineer and no model work. The setup steps are in how to create an AI chatbot.
- A fixed monthly price with a known message quota.
- A person can take over. The Growth plan's iOS app sends a push notification when a visitor asks for a human.
Cons
- You do not train a model or bring your own. The plans run on GPT-4.1-mini and GPT-4.1.
- There is no visual flow canvas for scripted, branching conversations.
- There is no free plan. The 7-day trial needs a credit card.
- There is no Intercom or HubSpot integration today.
- The Starter plan has no chat integrations and no scheduled refresh.
Cost. Starter is $59 a month billed monthly, or $39 a month billed yearly, with up to 4k GPT-4.1-mini messages a month. Growth is $129 monthly or $79 a month billed yearly, with up to 10k. Scale is $429 monthly or $259 a month billed yearly, with up to 40k.
3. A helpdesk's built-in AI agent: best for teams that already work tickets in a helpdesk
A helpdesk AI agent is the chatbot that a ticketing platform sells alongside its inbox. Fin is the clearest example, because it runs with the Intercom helpdesk and with others, including Zendesk, Freshdesk, and Salesforce.
What you get
- An AI agent that answers inside the same tool where agents already work tickets.
- Handoff to the agents who already work in that helpdesk.
- Pricing tied to results. Fin charges $0.99 per outcome, with at most one outcome per conversation.
Pros
- Nothing to connect if the team already uses the helpdesk.
- No charge when the customer asks for a person or Fin cannot answer.
- Spending limits and alerts can cap the monthly bill.
Cons
- The cost rises with every resolved conversation, so a busy month costs more.
- You do not choose the model or the retrieval method.
- Add-ons raise the bill. Fin lists a Pro add-on from $100 per workspace a month and Copilot at $29 per teammate a month.
Cost. $0.99 per outcome on top of the helpdesk's own seat prices. Fin offers a 14-day trial.
How to build the chatbot in 7 steps
These seven steps apply to all three routes. On a platform, steps 4 and 5 take minutes. In your own code, they are most of the project.
- Collect real questions. Export 200 recent customer questions from email, chat, or tickets. Remove personal data.
- Label each question. Mark it as answerable from content, needs a fixed flow or a person, or out of scope. This labeled list becomes the test set.
- Pick the route. Use the decision matrix below. Choose before writing any code.
- Load and index the content. Add the pages and files that answer the questions from step 1. Fix contradictions between pages first. The guide to training a chatbot on your own data covers the cleanup.
- Write the instructions and the routing. Tell the model to answer only from the content and to decline when the content has no answer. Send refunds, cancellations, and account changes to a fixed flow or a person.
- Run the test set. Score the chatbot on the 200 questions. Record the numbers.
- Launch, then re-test after every change. A new page, a new instruction, or a new model version can each lower the score. Run the same 200 questions again each time.
Worked example: 2,000 conversations a month
This example prices the three routes and scores one test run. The volume is 2,000 conversations a month, with 3 customer questions per conversation. That is 6,000 model calls a month.
What the model usage costs if you build it yourself
Each call sends about 3,000 input tokens: the instructions, the retrieved passages, and the conversation so far. Each reply is about 300 output tokens.
Line | Calculation | gpt-6-luna | gpt-6.1-sol |
Input tokens | 6,000 calls x 3,000 = 18M tokens | 18 x $0.10 = $1.80 | 18 x $2.00 = $36.00 |
Output tokens | 6,000 calls x 300 = 1.8M tokens | 1.8 x $0.50 = $0.90 | 1.8 x $10.00 = $18.00 |
File search calls | 6,000 calls at $2.50 per 1,000 | $15.00 | $15.00 |
Total per month | Sum of the three lines | $17.70 | $69.00 |
Cost per conversation | Total / 2,000 | $0.009 | $0.035 |
Three results stand out:
- The retrieval tool costs more than the model on gpt-6-luna. File search is $15.00 of the $17.70.
- The larger model costs 20 times more for the tokens. That is $54.00 against $2.70.
- Indexing the content is close to free. OpenAI's embeddings guide lists text-embedding-3-small at about 62,500 pages per dollar, so 500 pages cost under one cent.
These figures leave out the engineer who builds and maintains the chatbot. That is the real cost of this route. The full budget, including build time, is in the guide to how much it costs to build a chatbot.
What the same volume costs on the other two routes
Fin bills per outcome, so its cost depends on how many of the 2,000 conversations it resolves.
Share of conversations Fin resolves | Outcomes | Cost at $0.99 per outcome |
30% | 600 | $594 a month |
50% | 1,000 | $990 a month |
70% | 1,400 | $1,386 a month |
The helpdesk's seat prices come on top of these figures.
SiteGPT bills per plan, and each plan has a message quota.
SiteGPT plan | Billed monthly | Billed yearly, per month | GPT-4.1-mini messages a month | Price per message, monthly billing | Price per message, yearly billing |
Starter | $59 | $39 | 4,000 | $0.0148 | $0.0098 |
Growth | $129 | $79 | 10,000 | $0.0129 | $0.0079 |
Scale | $429 | $259 | 40,000 | $0.0107 | $0.0065 |
Each quota is ten times smaller if every message uses GPT-4.1: 400, 1,000, and 4,000 messages. Count the messages in your own conversations before you pick a plan.
The chart compares prices, and the routes do not include the same things. The own-build figures cover model usage only. The SiteGPT figure is the Growth plan billed monthly with its 10k message quota. The Fin figure assumes half the conversations are resolved and leaves out helpdesk seats.
How to score the test run
The test set has 200 questions. 160 can be answered from the content. 40 cannot, and the correct outcome for those is to decline and offer a person.
Question type | Count | Correct outcome | Result | Score |
Answerable | 160 | A correct answer | 136 correct, 14 wrong, 10 declined | 136 / 160 = 85% |
Not answerable | 40 | Decline and offer a person | 36 declined, 4 invented an answer | 36 / 40 = 90% |
All questions | 200 | Either outcome above | 172 passed | 172 / 200 = 86% |
The number to fix first is the wrong answers. 14 wrong plus 4 invented is 18 of 200, so 9% of the test questions got a wrong answer. The 10 declined questions are a smaller problem, because those customers reach a person.
The intent classifier gets its own score. Take the refund intent. 50 of the 200 messages are refund requests. The classifier labels 48 messages as refunds, and 42 of those are right.
- Precision: 42 / 48 = 87.5%. Of the messages sent to the refund flow, 6 did not belong there.
- Recall: 42 / 50 = 84%. The classifier missed 8 refund requests.
- F1 score: 2 x 0.875 x 0.84 / (0.875 + 0.84) = 85.7%.
Low recall is the costlier error here. A missed refund request gets a generated answer when it should reach the fixed flow. Adding more example phrases for that intent is the usual fix.
Should you build or buy a customer service chatbot?
Build the chatbot when it is part of your product or needs logic no vendor offers. Buy it when its job is to answer questions about your product from content you already have.
If you need... | Pick... | Why |
A chatbot inside your own app that calls your own systems | Build it yourself | Only your code can reach your data and logic |
A specific model, or a model hosted on your own hardware | Build it yourself | Platforms choose the model for you |
To learn how the machine learning works | Build it yourself | The other routes hide those parts |
A website chatbot live this week with no engineer | Indexing, the model, and the widget are included | |
Answers from a site, files, and cloud drives that change often | Scheduled refresh runs monthly on Growth and weekly on Scale | |
A fixed monthly price | Plans have a set price and a set message quota | |
An AI agent inside the helpdesk your agents already use | The helpdesk's AI agent | Handoff and ticket data are already connected |
To pay only when a conversation is resolved | Fin | It charges $0.99 per outcome and nothing when it cannot answer |
Scripted, branching flows drawn on a canvas | None of the three routes above is built for that |
If the chatbot must also take actions, such as issuing a refund on its own, compare the options in AI agent vs chatbot.
Common mistakes when building a machine learning chatbot
- Training a model when indexing would do. Most question answering needs retrieval over your content. It does not need a new model.
- Launching with no test set. A team with no baseline cannot tell whether a change helped.
- Testing only answerable questions. The test set also needs questions the chatbot should decline. The guide to chatbot hallucinations covers why.
- Letting a generated answer make a commitment. Refunds and account changes belong in a fixed flow or with a person.
- Dropping the conversation history. The model does not remember earlier turns unless the application sends them.
- Indexing content that contradicts itself. The chatbot will state one of the two versions as fact.
- Counting only the model bill. In the worked example, tokens cost $2.70 a month. The engineer costs far more.
The day-to-day rules for running the chatbot after launch are in chatbot best practices.
Frequently asked questions
Basics
How is machine learning used in customer service?
Machine learning is used in customer service to answer questions through chatbots, route tickets to the right queue, flag upset customers, and group conversations by topic. In a chatbot it does four jobs: intent classification, retrieval, generation, and evaluation.
Does a customer service chatbot need machine learning?
No. A rule-based chatbot follows buttons and scripts with no machine learning. It works for fixed tasks such as booking a slot. A chatbot that answers open questions in the customer's own words needs machine learning.
What is the difference between machine learning and NLP in a chatbot?
Machine learning is the general method of learning patterns from data. Natural language processing, or NLP, is the field that applies it to text and speech. Intent classification, embeddings, and language models are all NLP built with machine learning. To see how intents, entities, and retrieval work together inside one chatbot, read the guide to NLP chatbots.
Building
What do you need to build a machine learning chatbot?
You need real customer questions, the content that answers them, a model, and a test set. If you write the code yourself, you also need a retrieval layer, a chat widget, a handoff to a person, and an engineer to maintain them.
Which type of AI system is best for improving customer service?
A system that is strong at natural language processing fits most customer service work, because the work is reading and answering messages. Add prediction models later, for jobs such as forecasting ticket volume.
Why does a chatbot forget what the customer said earlier in the conversation?
The chatbot forgets because language model requests are stateless. The application has to store the conversation under a session ID and send the earlier messages with each new request. A larger context window does not help if the history is never sent.
Can you build a customer service chatbot without code?
Yes. A no-code retrieval platform such as SiteGPT indexes your content and supplies the model and the widget. A helpdesk's AI agent is the other no-code route.
Models and data
Do you need to train your own model?
No. Most chatbots use a pretrained language model and retrieve your content at answer time. Training your own model makes sense for narrow jobs such as intent classification, where a small classifier trained on labeled examples is enough.
How much training data does an intent classifier need?
Google's Dialogflow documentation recommends at least 10 to 20 training phrases per intent, depending on the intent's complexity. Start there, then add phrases for any intent with low recall on the test set.
Is fine-tuning worth it for a customer service chatbot?
Rarely. Fine-tuning changes how a model writes, and it does not keep facts current. Retrieval does that job. OpenAI's pricing page also says its fine-tuning platform is being wound down and is closed to new users.
Cost and evaluation
How much does it cost to run a machine learning chatbot?
In the worked example, 2,000 conversations a month cost $17.70 in OpenAI usage on gpt-6-luna and $69.00 on gpt-6.1-sol. A SiteGPT plan starts at $59 a month billed monthly, or $39 a month billed yearly. Fin costs $990 a month if it resolves half of those conversations.
How do you evaluate a customer service chatbot?
Run a fixed set of labeled questions and score each job. Use precision and recall for intent classification, context recall for retrieval, and faithfulness for generated answers. Track the overall pass rate, and run the set again after every change.
What is a good accuracy score for a chatbot?
No single number fits every team. Compare the score with the cost of an error. In the worked example, an 86% pass rate still meant 9% of the test questions got a wrong answer, and that share is the number to bring down first.
Sources
All sources were checked on October 12, 2026.
Written by
