Are you wondering how to recreate the magical ability that ChatGPT has to continue a conversation and refer to things previously discussed using the GPT API?
If you are, this is the article for you.
The Basics – How to create a chat bot
You probably already created your own chatbot if you ask this question.
The prompt required for GPT to act as a simple chatbot is relatively simple to construct.
First, you must provide an interface for a user to ask questions. This might be through a web page or an application of some sort.
Once the user has entered their question, you can build a prompt and include the query text.
You are a friendly support person. The customer will ask you questions, and you will provide polite responses Q: My phone won't start. What do I do? A:
The API may answer with the text:
Try plugging your phone into the charger for an hour, and then turn it on. The most common cause for a phone not starting is that the battery is dead.
Once GPT responds with an answer, you can show the response to the user and allow the user to ask another question.
Sometimes the user wants to continue the discussion:
You are a friendly support person. The customer will ask you questions, and you will provide polite responses Q: I've tried that. What else can I try? A:
The problem is that GPT has no idea what you are talking about. You answered with a suggestion, but the user was not happy.
Furthermore, the user thinks the chatbot knows them already. The question they ask requires the context of the previous conversation.
Without reference to the previous question, the AI has no idea what they have tried or what the issue is. It doesn’t even know they are talking about a phone.
The only way to solve this is to give the chatbot some context. We can do this by including the original question in our query.
You are a friendly support person. The customer will ask you questions, and you will provide polite responses Q: My phone won't start. What do I do? A: Try plugging your phone into the charger for an hour and then turn it on. The most common cause for a phone not starting is that the battery is dead. Q: I've tried that. What else can I try? A:
The API may answer with the text:
Hold the button in for 15 seconds. It may need a reset.
Hopefully, that will have fixed the problem (or maybe not)
The user may have another question.
You are a friendly support person. The customer will ask you questions, and you will provide polite responses Q: I did that. It worked. But the screen is blank A:
Once again, the AI has no idea what we are talking about or what has already been tried.
However, this time it knows we are talking about a blank screen. But it could be a phone, tablet, laptop, or even a car’s TV screen or navigation system.
We could include the last conversation again – but can you spot the major problem with that?
You are a friendly support person. The customer will ask you questions, and you will provide polite responses Q: I've tried that. What else can I try? A: Hold the button in for 15 seconds. It may need a reset. Q: I did that. It worked. But the screen is blank A:
In case you didn’t notice, there is no mention of the fact we are talking about a phone. The phone was initially mentioned in the dialog from two conversations ago. It certainly wasn’t in the previous question-and-answer pair.
So now we need to include two conversations in our prompt
You are a friendly support person. The customer will ask you questions, and you will provide polite responses Q: My phone won't start. What do I do? A: Try plugging your phone into the charger for an hour and then turn it on. The most common cause for a phone not starting is that the battery is dead. Q: I've tried that. What else can I try? A: Hold the button in for 15 seconds. It may need a reset. Q: I did that. It worked. But the screen is blank A:
Now the AI knows we are talking about a phone and that we have charged it for an hour, as well as holding the power button for 15 seconds.
By now, you should be noticing a pattern. We are including previous conversations in our prompt. This is to give GPT the context of the discussion.
How to include previous conversations
Including previous conversations is pretty simple. You have seen a few examples in the section above.
Essentially you need to include each pair of answers above the user’s new question. The pairs should be chronological, with the oldest interactions at the top.
Over time, the prompt will get longer and longer until you hit a barrier.
Each GPT model has a maximum number of tokens you can pass to it. In the case of Davinci-003, it is 4096 tokens (about 3000 words, plus or minus)
When you hit this limit, the API will start to throw errors.
So what should you do in this case?
The only solution is to drop the oldest pair of the conversation. This will be the pair at the top of the list.
However, it can be tough to predict when you will run out of tokens. The words require a different number of tokens depending on the language you are conversing in. In the case of English, each word takes between 1 and 3 tokens (roughly). But if you converse in Russian, sometimes the individual letters within a word require a token each.
A better rule of thumb is to limit the history to 3 or 4 pairs of conversations.
Whichever way you decide to go, you will face a new problem…
What if the user asks a question that relates to a conversation that occurred more than 4 or 5 cycles ago?
The answer is fairly obvious – but it is probably not what you want to hear.
GPT will no longer have the knowledge contained in those conversations, and this causes a phenomenon called “Drift.”
The AI will drift away from the original topic as the conversation continues. This is not usually a problem, as an unaware user will subconsciously pull the AI back on course by the wording of the subsequent questions or interactions.
People have commented that ChatGPT can keep track of old conversations. However, OpenAI has stated in their documentation that ChatGPT also uses 4096 tokens and has no persistent memory.
Unfortunately, this is one of the things you need to live with when creating chatbots with GPT (I will talk about a potential solution in a minute)
Summarizing where we are at
So, before I look at a possible solution to provide long-term memory, let’s review where we are at:
- GPT has no natural memory of conversations
- We can include previous interactions with our prompt to give context
- There is a token limit for the number of interactions we can include
- Tokens cost money, so this could become quite expensive
- Once we run out of capacity, GPT starts to lose its memory of old conversations
- There may be a way to add persistent memory – but it will require some work at your end
Adding persistent memory
If giving your GPT chatbot some form of memory is important to you, your only choice is to consider Embedding.
I have an entirely free course on embedding at this link –> https://blinkdata.com/openai-embedding-tutorial/
You need to be aware that the current GPT models for embedding only support the English language. You will also have to do a reasonable amount of work to use this functionality.
It’s up to you how you code it. But here is the general strategy you can use:
- Create a local database with a userid, a datestamp, a text field for a Q/A pair, and a vector field
- Each time a user asks a question, store the question and generated answer in the text field. Also include the current date and time and the user ID or session ID
- When a user asks a question, check to see how many records are in the database for the user. If there are 4 or less, simply include the last interactions in your prompt in chronological order above the final question.
- If you have more than 4 records, pick the 4 or 5 top matches from your semantic search. Include them as the Q and A pairs in your prompt. (Instead of using the most recent conversation entries)
- You can delete the old conversation when a user ends their chat or periodically
I hope you have found this helpful. If you have questions, contact me on the community forums at OpenAI. By doing this, other users can also help solve your issues, and the knowledge will be available for everyone to search for.