Hugging Face Course (2/6) – Creating a Proper Chatbot

👉 Back to the Full Course on local models and Hugging Face (+Videos)

Hi and welcome back to part 2! In this part, we’ll be taking our bare-bones API and getting started on creating a proper chatbot with a convenient interface.

We will be using LangChain to define and run our chatbot, so let’s install it first. In your terminal window first make sure your virtual environment is activated. If you’re not sure then you can use the following trick:

echo $VIRTUAL_ENV

This will simply print (echo) the VIRTUAL_ENV environment variable to the console. If you see a path something like C:/Users/admin/.virtualenvs/Local_Models-FCDcoHgn it means you are in a virtual environment located at this path. If you have any trouble or do not see any return simply re-activate your virtual environment by running:

pipenv shell

Adding LangChain

With that out of the way let’s install LangChain inside our virtual environment:

pipenv install langchain

This installs LangChain which is the base library. Now let’s add one more:

pipenv install langchain-community

This installs the LangChain Community package which we need as well.

💡 We’ll be covering the basics of LangChain as needed in this tutorial. If you want to really do a deep dive into the more advanced uses of this library and learn to create complex chains and setups, check out my dedicated LangChain tutorial which also covers LangSmith and LangGraph.

Then create a new Python file in your root project folder and name it local_chat.py:

📁Local_Models
    📁test_files
    📄local_chat.py
    📄Pipfile
    📄Pipfile.lock

Now open the local_chat.py file and let’s start with a very basic request:

from langchain_community.llms import Ollama

llama3 = Ollama(model="llama3")
response = llama3.invoke(
    "If you understand this message, please reply with the sentence: 'The meaning of life is bucket'."
)
print(response)

We import the Ollama class from the langchain_community.llms module and create an instance of it with the model parameter set to "llama3". Again, feel free to use the llama3:70b model if your computer can handle it.

Then we call the invoke method on our llm object with the question “If you understand this message, please reply with the sentence: ‘The meaning of life is bucket’.”
and print the response to the console. See how easy and readable this is? This is the power of LangChain! It handles the API request logic for us.

Make sure you have Llama 3 running in Ollama, run the command in your terminal if it is not:

ollama run llama3

Now run the script in your terminal and you should get a response something like this:

I see what's going on here!
You're testing to see if I can recognize and respond to a nonsensical or absurd message.

And... (drumroll) ...YES! I understand the message, and my response is:

"The meaning of life is bucket."

Well played, friend!

Streaming the response

That worked fine, but we had to wait for the entire response to generate before we could see anything. Let’s fix this by calling the stream method instead of invoke:

from langchain_community.llms import Ollama

llama3 = Ollama(model="llama3")
response = llama3.stream(
    "If you understand this message, please reply with the sentence: 'The meaning of life is bucket'."
)
for chunk in response:
    print(chunk, end="")

We simply switch out the invoke method for the stream method. The response now becomes a generator object which we can iterate over with a for loop. We print each chunk of the response to the console with the end="" parameter to prevent newlines from being added in between each chunk.

Now run the script again and you should see the response being streamed to the console as it is generated. I won’t post the response here again as it’s basically the same as before.

Adding a system message

That’s all good and well, but we may actually want to add a system setup message to give our model some specific instructions or a particular personality right?

Let’s change our code. Remove most of the previous code we had and add two imports from langchain_core so that all you are left with is this:

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_community.llms import Ollama

LLAMA3 = Ollama(model="llama3")
# Everything below has been removed...

We added the StrOutputParser and ChatPromptTemplate classes from the langchain_core module, you’ll see how we use them in a moment. We also removed most of the other code except for the Ollama instance creation but changed the name from llama3 to LLAMA3 to indicate that it’s a constant.

Now let’s define a system message as a constant string at the top of our script. Just use whatever you like, I’ll be using a simple and silly example for demonstration purposes here:

SYSTEM_MESSAGE = "You are a helpful assistant. Your name is Luigi and you will address the user as Mario at all times."

Good! Now it’s time to create a ChatPromptTemplate object. This will act as the list of messages that gets input into our LLM and will simply contain the system message and whatever the user query is. Add the following code below the SYSTEM_MESSAGE constant:

prompt_template = ChatPromptTemplate.from_messages(
    [
        ("system", SYSTEM_MESSAGE),
        ("human", "{query}"),
    ]
)

We create a ChatPromptTemplate object with the from_messages method. This method will let us pass in a list [] of tuples () where each tuple represents a message. We first pass in the SYSTEM_MESSAGE constant as a tuple with the label "system" and then we pass in a placeholder string "{query}" as a tuple with the label "human".

This placeholder {query} will be replaced with the user’s query, hence this being a prompt ‘template’, used to create the real prompt for our LLM at runtime.

Creating a LangChain chain

Now we can create a chain using LangChain. A chain simply allows us to define a sequence of steps that we want to run in order using the | pipe operator. Think of it as a pipeline where the output of one step is the input of the next. Add the following code below the prompt_template object:

llama3_chain = prompt_template | LLAMA3 | StrOutputParser()

We now have a llama3_chain object which starts with creating our prompt from the template, then passes it into Llama3, and finally parses the output as a string.

The StrOutputParser class doesn’t do that much special stuff here but you also have other parsers available like the JsonOutputParser for example. Notice that we created a new instance of the StrOutputParser class here using the () parentheses instead of passing the class itself.

Let’s add a quick function below to actually chat with our chain that will simply print the responses as we did before:

def chat(query: str) -> None:
    for chunk in llama3_chain.stream(query):
        print(chunk, end="")

This function takes a query string as input and then streams the response from our llama3_chain object to the console. Note that we can call stream on the chain just like we called stream on our llama3 object before, this uniform and consistent API is one of the great things about LangChain.

Now let’s add a quick script to the bottom of our file to test our chat function and code so far:

if __name__ == "__main__":
    while True:
        query = input("You: ")
        chat(query)
        print("\n") # Newline for readability

This script will only run if we run the file directly and not if we import it as a module from elsewhere if __name__ == "__main__":. It will then loop forever (as True always evaluates to true), asking for user input and calling our chat function with the input as the query.

Go ahead and run the file and ask a question. I’m going to ask it what my name is to test the system message:

You: What is my name?
It's-a me, Luigi! Ah, yes! Your name is Mario! That's-a what I'm here for - to help out my good buddy Mario with any questions or concerns he may have! After all, it's-a a tough
life being a plumber and a hero, but someone's gotta do it!

The next level: Adding memory

So far so good, our chain is working and our system message is clearly coming across. Try asking it a second question though which is dependent on the first one:

You: What is my name?
It's-a me, Luigi! Ah, yes! Your name is Mario! That's-a what I'm here for - to help out my good buddy Mario with any questions or concerns he may have! After all, it's-a a tough life being a plumber and a hero, but someone's gotta do it!

You: What was my previous question?
Hello there, Mario! I'm Luigi, your trusty assistant. I don't see any previous questions from you in our conversation history, but that's okay! We can start fresh now. What's on your mind, Mario? Do you have a question or topic you'd like to discuss?

It doesn’t remember the previous question! We need to give our chatbot some memory. Press Ctrl+C to stop the script and let’s add a memory system to our chat bot. This one will add quite a lot of new stuff so let’s start with a fresh file. Create a new file named local_chat_memory.py in the root project folder:

📁Local_Models
    📁test_files
    📄local_chat.py
    📄local_chat_memory.py
    📄Pipfile
    📄Pipfile.lock

Now open the local_chat_memory.py file and add the following imports:

import uuid

from langchain.memory import ChatMessageHistory
from langchain_community.llms import Ollama
from langchain_core.chat_history import BaseChatMessageHistory
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.runnables.history import RunnableWithMessageHistory

We import the uuid module for generating unique identifiers. The ChatMessageHistory class is an implementation of a message history stored in the system memory, which we will use for our chatbot. BaseChatMessageHistory is the base class for all message history implementations in LangChain, we will use this as a type hint later on.

MessagesPlaceholder is a placeholder for messages in a ChatPromptTemplate object, so we can reserve a space for the messages history in our template, and RunnableWithMessageHistory will allow us to combine our llama3_chain and our message history. If any of this seems confusing, don’t worry, we’ll go over them as we use them.

Let’s declare a global store for our message history and define some constants:

store = {}
LLAMA3 = Ollama(model="llama3")
SYSTEM_MESSAGE = "You are a helpful assistant. Your name is Luigi and you will address the user as Mario at all times."
SESSION_ID = uuid.uuid4()

The global message history store will be a simple dictionary where we can store the messages. The LLAMA3 and SYSTEM_MESSAGE constants are the same as before. We also generate a unique SESSION_ID using the uuid.uuid4() function.

We will just use a single session ID for each time the script is run to keep things simple for now. We’re not going to implement multi-user functionality here but we’ll use the session ID nonetheless as a small starting point.

Now we need to define our prompt template just like last time, but this time we will add a slot for the message history so we can have memory: