💡 Full Course with Videos and Course Certificate (PDF): https://academy.finxter.com/university/openai-api-function-calls-and-embeddings/
Welcome back to part 6 of this tutorial series. In this part, we’re going to be looking at embeddings. What is an embedding? Without getting into algorithmic arithmetic an embedding is basically a numerical (vector) representation of text that makes no sense to us, but makes sense to the computer and allows it to compare the similarity in meaning of certain words and pieces of text.
What makes this so exciting is that it does not literally compare the words, but rather the meaning of the words. For example, if we were to compare the words “happy” and “contented” it would not compare the words themselves, but rather the meaning of the words. This is because the words “happy” and “contented” are very similar in meaning, but not in spelling. This is what makes embeddings so powerful. It allows us to compare or search by similarity of meaning, even if the words used are completely different!
Imagine having a large document and you need to find a particular quote or passage but you only half remember the exact words used. You can totally just do a similarity search using embeddings! As long as you input something with a similar meaning, you’ll be able to easily find the passage you need even if you don’t remember the exact wording. The uses for this are really endless in many different and clever ways, so we’ll be exploring a couple of possible uses in the last two parts of this tutorial series.
Preparing our data
Before we get started we’ll need something to search in. For this first part, we’ll be using a list of quotes from famous people throughout time. There are about 250 quotes in the list. I’ve provided the list as a separate text file with this tutorial, but if for some reason it is not available you can find the list at the end of this tutorial. Just copy the quotes and paste them into a new file named ‘Fx_quotes.txt’ and save it in your tutorial’s root folder
> Fx_quotes.txt
Now if you read through the quotes file you’ll notice they all have their authors and sometimes extra information attached. As we only want to compare the meaning of the quotes themselves, and not any extra information that is tagged on, let’s have a bit of fun. This is not technically required, but it’s quite easy to do and for tutorial purposes, it will be very easy to show visually what is going on, so we’ll be writing the quotes to a separate Python file as a list of lists.
First, create a file in your base directory named ‘Fa_write_quotes_to_list.py’.
> Fa_write_quotes_to_list.py
Now open the file and add the following code:
quotes = []
with open("Fx_quotes.txt", "r") as file:
quotes = file.read().split("\n")
with open("Fx_quotes.py", "w") as file:
file.write("quotes = [\n")
for quote in quotes:
# split the quote only on the first occurrence of the '-' character
quote = quote.split(" - ", 1)
file.write(f"{quote},\n")
file.write("]")
First, we create an empty list named quotes. Then we open the Fx_quotes.txt file in read mode (make sure the files are both in the root folder). We then read the quotes from the file, split them on the newline characters, so the end of each line, and assign this to the quotes variable. We now have a list of quotes with each ‘quote-author’ pair as a single combined item in the list.
Next, we open the Fx_quotes.py file in write mode (it will be created automatically if it doesn’t exist). We then write the string “quotes = [\n” to the file. This is just a Python variable named quotes followed by the opening square bracket and a newline (enter key) character. We then loop through each quote in the quotes list, splitting the quote on the first occurrence of the ‘-‘ character. This will split the quote into two parts, the quote itself and the author. We then write the quote to the file, followed by a comma and a newline character. We then write the closing square bracket to the file. This will close the list and the file.
Note we use the ‘with open’ context manager in both cases which means we don’t have to worry about closing the files ourselves. The context manager will automatically close the files for us when the code block has finished executing.
If you run this file you’ll see a new file appear in your base directory. Opening Fx_quotes.py you should see the following:
quotes = [ ["A man is great not because he hasn't failed; a man is great because failure hasn't stopped him.", 'Confucius'], ['Happiness is a gift and the trick is not to expect it, but to delight in it when it comes.', 'Charles Dickens'], ['Everyday you can take a tiny step in the right direction.', 'Unknown'], ['Sometimes adversity is what you need to face in order to become successful.', 'Zig Ziglar'], ['Opportunity comes when you continually push forward.', 'Sonia Ricotti'], ['Always seek out the seed of triumph in every adversity.', 'Og Mandino'], ['Reality is nothing but a collective hunch.', 'Lily Tomlin'], ['Best be yourself, imperial, plain, and true.', 'Robert Browning'], ["Resilience is not what happens to you. It's how you react to, respond to, and recover from what happens to you.", 'Jeffrey Gitomer'], ['Time is a game played beautifully by children.', 'Heraclitus'], ['Most people fail in life because they major in minor things.', 'Tony Robbins'], ['Anyone who has ever made anything of importance was disciplined.', 'Andrew Hendrixson'], ['Do anything, but let it produce joy.', 'Walt Whitman'], ...many more quotes... ]
Now we have a separate list of lists with the quote in index 0 and the author info in index 1. We’ll need to generate a separate embedding for every single quote in the list and store them somewhere. Once we have done so we can compare them to any piece of text we enter to find similar entries. If this seems confusing, don’t worry, we’ll be showing exactly how the process works.
Generating an embedding
Before we start generating embeddings for all our quotes let’s actually look at how to generate an embedding in the first place and exactly what an embedding is and looks like. Create a new file in your root tutorial directory called ‘Fb_getting_an_embedding.py’. Again the silly alphabetic naming is just to keep the files in a nice order so you can easily find anything if you want to reference these tutorial code files later on.
> Fb_getting_an_embedding.py
Now open the file and let’s start with our imports up top:
import openai
from decouple import config
openai.api_key = config("CHATGPT_API_KEY")
EMBEDDING_MODEL = "text-embedding-ada-002"
We’ll go back to the basics for a moment, getting rid of our simplified GPT calling utility as we’ll be using a different model. We’ll be using the text-embedding-ada-002 model as it is specifically designed for generating embeddings. We import openai, as the embeddings endpoint is also an openai API. We import the config function and set our openai API key as we did for the previous parts, embeddings use the same API key as normal ChatGPT. We then set the EMBEDDING_MODEL variable to the text-embedding-ada-002 model.
Now we’ll create a simple call function:
def get_quote_embedding(quote):
response = openai.Embedding.create(
model=EMBEDDING_MODEL,
input=quote,
)
return response
print(get_quote_embedding("Please embed this sentence for me!"))
This is very self-explanatory and similar to normal ChatGPT calls except we use a openai.Embedding.create instead of ChatCompletion.create. It takes a model and an input text, we pass in the model we defined in our variable and the quote input our function takes. Finally, we add a call to this function below the function block which will embed the text “Please embed this sentence for me!” and print the result.
Now let’s generate our first embedding and see what it looks like. Go ahead and run the file:
{
"data": [
{
"embedding": [
-0.03016725555062294,
0.005489848554134369,
0.0014944026479497552,
-0.014989439398050308,
0.002586822025477886,
0.02618442475795746,
0.010024355724453926,
0.0032814627047628164,
-0.005856511183083057,
-0.015339282341301441,
-0.01650991104543209,
... many more numbers ...
],
"index": 0,
"object": "embedding"
}
],
"model": "text-embedding-ada-002-v2",
"object": "list",
"usage": {
"prompt_tokens": 7,
"total_tokens": 7
}
}
You should see a massive list of numbers representing vectors, about 1500 lines just for this simple short sentence. While this doesn’t really make visual sense to us this is the magic that will allow the computer to compare the similarity between different text’s vectors, and thereby the similarity between the texts themselves. The computer will not be comparing the text but the text’s vectors with each other.
Generating our quote embeddings
We will need to generate an embedding for every quote in our list and store them in some type of database. If we don’t save the embeddings we’d have to generate them anew every single time which takes time and wastes tokens. We’ll be using a CSV file for storage to get started, which is basically a simple text file with comma-separated values.
Create a new file in your base tutorial directory named ‘Fc_quote_embeddings.py’.
> Fc_quote_embeddings.py
Now open the file and add the following imports and setup to get started:
import openai
import pandas as pd
from decouple import config
from Fx_quotes import quotes
openai.api_key = config("CHATGPT_API_KEY")
EMBEDDING_MODEL = "text-embedding-ada-002"
total_tokens_used = 0
total_embeddings = 0
Most of these are familiar, we import and set up the openai module with our key and import the quotes we saved to a Python list earlier on. We also import the Pandas module. We won’t go too deep into Pandas, but will briefly cover the functions we use, so if you’re not familiar with Pandas no worries. Pandas is basically a spreadsheet/table in Python, it allows us to easily create, read, update, and delete data in a spreadsheet/table-like format with rows and columns of data. Note you may need to run ‘pip install pandas’ in your console if you don’t have Pandas installed yet.
Finally, we set two global variables to keep track of the total tokens used and the total embeddings generated, so we can give ourselves a sort of progress indicator as we run the file later on.
Now define a function that takes a quote as an argument and makes a single embedding API call, returning the response:
def get_quote_embedding(quote):
global total_tokens_used, total_embeddings
response = openai.Embedding.create(
model=EMBEDDING_MODEL,
input=quote,
)
tokens_used = response["usage"]["total_tokens"]
total_tokens_used += tokens_used
total_embeddings += 1
if (total_embeddings % 10) == 0:
print(
f"Generated {total_embeddings} embeddings so far with a total of {total_tokens_used} tokens used. ({int((total_embeddings / len(quotes)) * 100)}%)"
)
return response["data"][0]["embedding"]
The function takes a quote as an argument, and the second line gives us access to the global variables total_tokens_used and total_embeddings. We’ll be running this function over and over so we can increment these global variables each pass. We then make the API call just like we did before.
We get the tokens used from the response data and add the tokens used on this call to the global variable total_tokens_used. We also increment the total_embeddings global variable by 1. We then check if the total_embeddings is a multiple of 10, using the remainder operator. If the total embeddings number is cleanly divisible by 10 we print a progress message to the console to inform ourselves of the progress of the embedding generation so far. This just means every 10 times the function runs it will print the overall progress to the console once. We then return the response, but only the actual embedding itself which is located in [“data”][0][“embedding”].
Now we need some kind of data structure to hold our data. Below and outside the function block, continue:
embedding_df = pd.DataFrame(columns=["quote", "author", "embedding"])
A DataFrame is Pandas’ signature data structure, pd just stands for Pandas. All this is is an empty table with three columns; quote, author, and embedding, that we can use to store our data in.
Now we’ll need to loop over our list of quotes and call the above get_quote_embedding function once for each quote in the file and store our data in the DataFrame.
for index, quote in enumerate(quotes):
current_quote = quote[0]
try:
current_author = quote[1]
except IndexError:
current_author = "Unknown"
embedding = get_quote_embedding(current_quote)
embedding_df.loc[index] = [current_quote, current_author, embedding]
For each index and quote in the enumerate quotes (remember we already imported our quotes from the different file up top), we will run the following code. The current_quote = quote index 0, and we try to set the current_author to quote index 1. If there is no quote index 1, we set the current_author to “Unknown”. We then call our get_quote_embedding function on the current_quote and set the embedding variable to the response data’s embedding.
Finally, we select the embedding_df Dataframe and use the loc method to select the current index which will increase with each loop iteration. We set the values of the current index of this Dataframe to the current quote, author, and embedding, filling in the Dataframe with data loop by loop.
Note that certain code like current_quote = quote[0] is not technically required, but if you code in this style it’s much easier to read and if you come back to your code later you’ll be able to jump right back in. Writing your code as human-readable as possible is always preferable over clever one-liners that are hard to read.
Now that we’ve filled up our DataFrame with data, let’s save the DataFrame to a CSV file so we can load it later on. Then we’ll print a success message to finish up. The .head() method in the print statement just prints the first x rows of a DataFrame and is a built-in Pandas method.
embedding_df.to_csv("Fx_embedding_db.csv", index=False)
print(
f"""
Generated {total_embeddings} embeddings with a total of {total_tokens_used} tokens used. (Done!)
Succesfully saved embeddings to embedding_db.csv, printing dataframe head:
{embedding_df.head(5)}
"""
)
As you can see, Pandas provides an easy method for us to dump the DataFrame to a CSV (comma-separated values) text file using the .to_csv method. We pass in the filename we want to save to and set the index to False. This just means we don’t want to save the index column to the CSV file, as we don’t need it. The index column is a Pandas-generated ID column that just holds numbers like 0, 1, 2, 3, 4, etc.
Your complete ‘Fc_quote_embeddings.py’ file should now look like this:
import openai
import pandas as pd
from decouple import config
from Fx_quotes import quotes
openai.api_key = config("CHATGPT_API_KEY")
EMBEDDING_MODEL = "text-embedding-ada-002"
total_tokens_used = 0
total_embeddings = 0
def get_quote_embedding(quote):
global total_tokens_used, total_embeddings
response = openai.Embedding.create(
model=EMBEDDING_MODEL,
input=quote,
)
tokens_used = response["usage"]["total_tokens"]
total_tokens_used += tokens_used
total_embeddings += 1
if (total_embeddings % 10) == 0:
print(
f"Generated {total_embeddings} embeddings so far with a total of {total_tokens_used} tokens used. ({int((total_embeddings / len(quotes)) * 100)}%)"
)
return response["data"][0]["embedding"]
embedding_df = pd.DataFrame(columns=["quote", "author", "embedding"])
for index, quote in enumerate(quotes):
current_quote = quote[0]
try:
current_author = quote[1]
except IndexError:
current_author = "Unknown"
embedding = get_quote_embedding(current_quote)
embedding_df.loc[index] = [current_quote, current_author, embedding]
embedding_df.to_csv("Fx_embedding_db.csv", index=False)
print(
f"""
Generated {total_embeddings} embeddings with a total of {total_tokens_used} tokens used. (Done!)
Succesfully saved embeddings to embedding_db.csv, printing dataframe head:
{embedding_df.head(5)}
"""
)
Go ahead and run it, this will take a little bit, but you’ll see regular status updates in your console.