Welcome back to part 4 of this tutorial series. In this part, we’re going to be looking at embeddings. What is an embedding?
Without getting into too deep into algorithms an embedding is basically a numerical (vector) representation of text that makes no sense to us, but makes sense to the computer and allows it to compare the similarity in meaning of certain words and pieces of text. Think of an embedding as a way for the computer to understand, store, and compare the meaning of a piece of text.
What makes this so exciting is that it does not literally compare the words or characters in them, but rather the meaning of the words. For example, if we were to compare the words “happy” and “contented” it would not compare the words themselves, but rather the meaning of the words. This is because the words “happy” and “contented” are very similar in meaning, but very different in spelling. This is what makes embeddings so powerful. It allows us to compare or search by similarity of meaning, even if the words used are completely different!
Imagine having a large document and you need to find a particular quote or passage but you only half remember the exact words used. You can totally just do a similarity search using embeddings! As long as you input something with a similar meaning, you’ll be able to easily find the passage you need even if you don’t remember the exact wording. The uses for this are really endless in many different and clever ways, so we’ll be exploring a couple of possible uses in the next two parts of this tutorial series.
Preparing our data
Before we get started we’ll need something to search in. For this first part, we’ll be using a list of quotes from famous people throughout time. There are about 250 quotes in the list. I’ve provided the list as a separate text file with this tutorial, but if for some reason it is not available you can find the list at the end of this written tutorial. Just copy the quotes and paste them into a new file named 'quotes.txt' and save it in a new folder named '4_Embeddings_for_similarity' in your base tutorial directory:
📁FINX_FUNC_EMBED
📁1_Simple_function_call
📁2_Parallel_function_calling
📁3_Database_functions
📁4_Embeddings_for_similarity
📄quotes.txt
📄.env
Now if you read through the quotes file you’ll notice they all have their authors and sometimes extra information attached. As we only want to compare the meaning of the quotes themselves, and not any extra information that is tagged on, let’s have a bit of fun. This is not technically required, but it’s quite easy to do so let’s parse our quotes into a JSON object.
First, create a file in your '4_Embeddings_for_similarity' directory named '1_write_quotes_to_list.py':
📁FINX_FUNC_EMBED
📁1_Simple_function_call
📁2_Parallel_function_calling
📁3_Database_functions
📁4_Embeddings_for_similarity
📄1_write_quotes_to_list.py
📄quotes.txt
📄.env
Now open the file and start with the following code:
import json from pathlib import Path current_directory = Path(__file__).parent raw_quotes = []
We start by importing JSON to be able to output in JSON data format and pathlib.Path to get the current directory. We then set the current_directory variable to the current directory like we did before. We then create an empty list named raw_quotes which we’ll use to store our quotes in.
Next, let’s read our quotes:
with open(current_directory / "quotes.txt", "r", encoding="utf-8") as file:
raw_quotes = file.read().split("\n")
We open the quotes.txt file in read mode by using our current path variable and appending / "quotes.txt". Make sure to specify the 'utf-8' encoding just to be safe. We then read the quotes from the file, split them on the newline characters, so the end of each line, and assign this to the quotes variable. We now have a list of quotes with each quote as a single string in the list, including whatever extra information may be included, like this:
[
quote1 - info,
quote2 - info,
quote3 - info,
]
Now let’s parse our quotes and write them to a JSON file:
with open(current_directory / "quotes.json", "w", encoding="utf-8") as file:
quotes: list = []
for quote in raw_quotes:
# split the quote only on the first occurrence of the '-' character
quotes.append(quote.split(" - ", 1))
json.dump(quotes, file, indent=2)
We open 'quotes.json' in the current directory as a file in write mode (it will be created automatically if it doesn’t exist).
Again, make sure to specify the utf-8 encoding to be safe. We then create an empty list named quotes, and loop over our raw_quotes list, splitting each quote on the first occurrence of the '-' character. This will split the quote into two parts, the quote itself and the extra information. We then append this to our quotes list. Finally, we dump the quotes list to the file in JSON format, with an indent of 2 for readability.
Note we use the 'with open' context manager in both cases which means we don’t have to worry about closing the files ourselves. The context manager will automatically close the files for us when the code block has finished executing.
Go ahead and run this file and you’ll see a new file named 'quotes.json' appear in your '4_Embeddings_for_similarity' folder. Opening 'quotes.json' you should see the following:
[
[
"A man is great not because he hasn't failed; a man is great because failure hasn't stopped him.",
"Confucius"
],
[
"Happiness is a gift and the trick is not to expect it, but to delight in it when it comes.",
"Charles Dickens"
],
... many, many more quotes ...
]
See how nice and readable that is because we specified the indent value?
Now we have a separate list of lists with the quote in index 0 and the author info in index 1. So in order for the computer to understand the meaning of each quote, we’ll need to generate an embedding for every single quote in the list and store them somewhere.
Once we have done so we can compare them to any piece of text we enter to find similar entries. If this seems confusing, don’t worry, we’ll be showing exactly how the process works.
Generating an embedding
Before we start generating embeddings for all our quotes let’s actually look at how to generate an embedding in the first place and exactly what an embedding is and looks like. Create a new file in your '4_Embeddings_for_similarity' folder called '2_getting_an_embedding.py':
📁FINX_FUNC_EMBED
📁1_Simple_function_call
📁2_Parallel_function_calling
📁3_Database_functions
📁4_Embeddings_for_similarity
📄1_write_quotes_to_list.py
📄2_getting_an_embedding.py
📄quotes.txt
📄.env
Now open the file and let’s start with our imports up top:
from openai import OpenAI
from decouple import config
client = OpenAI(api_key=config("OPENAI_API_KEY"))
EMBEDDING_MODEL = "text-embedding-ada-002"
We import OpenAI and config from decouple, and then set up our client object as always. Note that we can use the same API key we use for ChatGPT to create embeddings. What is different is the model we’ll be using, which is the text-embedding-ada-002 model as it is specifically designed for generating embeddings.
Now we’ll create a simple call function:
def get_quote_embedding(quote):
response = client.embeddings.create(
model=EMBEDDING_MODEL,
input=quote,
)
return response
This is very self-explanatory and similar to normal ChatGPT calls except we use a client.embeddings.create instead. It takes a model and an input text, and we pass in the model we defined in our variable and the quote input our function takes.
Now let’s add a call to this function, pass in any string sentence you like, and wrap it inside a print statement:
print(get_quote_embedding("Please embed this sentence for me!"))
Now let’s generate our first embedding and see what it looks like. Go ahead and run the file and the response object in your terminal will look something like this (I made it a bit easier to read for you here):
CreateEmbeddingResponse(
data=[
Embedding(
embedding=[
-0.03006751835346222,
0.005477833561599255,
0.001486383960582316,
-0.014979923143982887,
0.0026430210564285517,
0.026231689378619194,
0.010040451772511005,
0.003284008475020528,
...over a thousand more numbers...
],
index=0,
object="embedding",
)
],
model="text-embedding-ada-002-v2",
object="list",
usage=Usage(prompt_tokens=7, total_tokens=7),
)
You should see a massive list of numbers representing vectors, about 1500 lines just for this simple short sentence. While this doesn’t really make visual sense to us this is the magic that will allow the computer to compare the similarity between different text’s vectors, and thereby the similarity between the meaning of the texts themselves. The computer will not be comparing the text but the text’s embedding numbers with each other. So an embedding is basically just a huge list of floating point numbers.
Generating our quote embeddings
Now that we know what an embedding is and how to get one, we will need to generate an embedding for every single quote in our list and store them all in some type of database or data format. If we don’t save the embeddings we would have to generate them anew every single time we want to compare something which takes a lot of time and wastes tokens. We’ll be using a CSV file for storage to get started, which is basically a simple text file with comma-separated values. There are also special vector databases like PineCone if you have a much larger project you need to host in the cloud.
Create a new file in your '4_Embeddings_for_similarity' folder named '3_generate_quote_embeddings.py':
📁FINX_FUNC_EMBED
📁1_Simple_function_call
📁2_Parallel_function_calling
📁3_Database_functions
📁4_Embeddings_for_similarity
📄1_write_quotes_to_list.py
📄2_getting_an_embedding.py
📄3_generate_quote_embeddings.py
📄quotes.txt
📄.env
Now open the file and add the following imports and setup to get started:
import json
from pathlib import Path
import pandas as pd
from decouple import config
from openai import OpenAI
client = OpenAI(api_key=config("OPENAI_API_KEY"))
current_directory = Path(__file__).parent
EMBEDDING_MODEL = "text-embedding-ada-002"
total_tokens_used = 0
total_embeddings = 0
Most of these are familiar, we import json and pathlib.Path, plus the OpenAI module and config. We then setup our client object and set the current_directory variable and also set the EMBEDDING_MODEL to the same one we used before.
We also import the Pandas module. We won’t go too deep into Pandas, but will briefly cover the functions we use, so if you’re not familiar with Pandas no worries. Pandas is basically a spreadsheet/table in Python, it allows us to easily create, read, update, and delete data in a spreadsheet/table-like format with rows and columns of data. Note you may need to run 'pip install pandas' in your console if you don’t have Pandas installed yet:
pip install pandas <-(run this comand in the terminal)
Finally, we set two global variables to keep track of the total tokens used and the total embeddings generated, so we can give ourselves a sort of progress indicator as we run the file later on.
Now let’s load our quotes from the JSON file we created earlier:
with open(current_directory / "quotes.json", "r", encoding="utf-8") as file:
quotes = json.load(file)
We open the quotes.json file in read mode, again specifying the utf-8 encoding to be safe, and then load the quotes into the quotes variable using the json.load method.
Let’s break down the bigger problem of getting embeddings for all quotes into smaller pieces by first creating a function that gets a single embedding for us. Define a function that takes a quote as an argument and makes a single embedding API call, returning the response:
def get_quote_embedding(quote: str) -> list[float]:
global total_tokens_used, total_embeddings
response = client.embeddings.create(
model=EMBEDDING_MODEL,
input=quote,
)
tokens_used = response.usage.total_tokens
total_tokens_used += tokens_used
total_embeddings += 1
if (total_embeddings % 10) == 0:
print(
f"Generated {total_embeddings} embeddings so far with a total of {total_tokens_used} tokens used. ({int((total_embeddings / len(quotes)) * 100)}%)"
)
return response.data[0].embedding
The function takes a quote as an argument and returns a list containing floating point numbers (an embedding). The second line gives us access to the global variables total_tokens_used and total_embeddings by referencing them inside this function, so we can keep count in the variables stored outside the function between runs. We’ll be running this function over and over so we can increment these global variables each pass. We then make the API call just like we did before.
We get the tokens used from the response data and add the tokens used on this call to the global variable total_tokens_used. We also increment the total_embeddings global variable by 1. We then check if the total_embeddings is a multiple of 10, using the remainder operator. If the total embeddings number is cleanly divisible by 10 we print a progress message to the console to inform ourselves of the progress of the embedding generation so far. This just means every 10 times the function runs it will print the overall progress to the console once. We then return the response, but only the actual embedding itself which is located in .data[0].embedding.
Now we need some kind of data structure to hold our data. Below and outside the function block, continue:
embedding_df = pd.DataFrame(columns=["quote", "author", "embedding"])
A DataFrame is Pandas’ signature data structure, pd just stands for Pandas. All this is is an empty table with three columns; quote, author, and embedding, that we can use to store our data in.
Now we’ll need to loop over our list of quotes and call the above get_quote_embedding function once for each quote in the file and store our data in the DataFrame.
for index, quote in enumerate(quotes):
current_quote = quote[0]
try:
current_author = quote[1]
except IndexError:
current_author = "Unknown"
embedding = get_quote_embedding(current_quote)
embedding_df.loc[index] = [current_quote, current_author, embedding]
For each index and quote in the enumerate quotes (remember we already imported our quotes from the different file up top), we will run the following code. The current_quote = quote index 0, and we try to set the current_author to quote index 1. If there is no quote index 1, we set the current_author to "Unknown". We then call our get_quote_embedding function on the current_quote and set the embedding variable to the response data’s embedding.
Finally, we select the embedding_df Dataframe and use the loc method to select the current index which will increase with each loop iteration, selecting the first row in the table on the first loop, the second row on the second loop, etc. We set the values of the current index of this Dataframe to the current quote, author, and embedding, filling in the Dataframe with data with each loop iteration.
Note that certain code like current_quote = quote[0] is not technically required, but if you code in this style it’s much easier to read and if you come back to your code later you’ll be able to jump right back in. Writing your code as human-readable as possible is always preferable over clever one-liners that are hard to read.
Now that we’ve filled up our DataFrame with data, let’s save the DataFrame to a CSV file so we can load it later on. Then we’ll print a success message to finish up:
embedding_df.to_csv(
current_directory / "embedding_db.csv", index=False, encoding="utf-8"
)
print(
f"""
Generated {total_embeddings} embeddings with a total of {total_tokens_used} tokens used. (Done!)
Succesfully saved embeddings to embedding_db.csv, printing dataframe head:
{embedding_df.head(5)}
"""
)
As you can see, Pandas provides an easy method for us to dump the DataFrame to a CSV (comma-separated values) text file using the .to_csv method. We pass in the current_directory + filename we want to save to and set the index to False. This just means we don’t want to save the index column to the CSV file, as we don’t need it. The index column is a Pandas-generated ID column that just holds numbers like 0, 1, 2, 3, 4, etc. In the final print statement, we just print a success message and the first 5 rows of the DataFrame to the console using the .head method, which just prints the first x rows based on the number you input as an argument.
Your complete '3_generate_quote_embeddings.py' file should now look like this:
import json
from pathlib import Path
import pandas as pd
from decouple import config
from openai import OpenAI
client = OpenAI(api_key=config("OPENAI_API_KEY"))
current_directory = Path(__file__).parent
EMBEDDING_MODEL = "text-embedding-ada-002"
total_tokens_used = 0
total_embeddings = 0
with open(current_directory / "quotes.json", "r", encoding="utf-8") as file:
quotes = json.load(file)
def get_quote_embedding(quote: str) -> list[float]:
global total_tokens_used, total_embeddings
response = client.embeddings.create(
model=EMBEDDING_MODEL,
input=quote,
)
tokens_used = response.usage.total_tokens
total_tokens_used += tokens_used
total_embeddings += 1
if (total_embeddings % 10) == 0:
print(
f"Generated {total_embeddings} embeddings so far with a total of {total_tokens_used} tokens used. ({int((total_embeddings / len(quotes)) * 100)}%)"
)
return response.data[0].embedding
embedding_df = pd.DataFrame(columns=["quote", "author", "embedding"])
for index, quote in enumerate(quotes):
current_quote = quote[0]
try:
current_author = quote[1]
except IndexError:
current_author = "Unknown"
embedding = get_quote_embedding(current_quote)
embedding_df.loc[index] = [current_quote, current_author, embedding]
embedding_df.to_csv(
current_directory / "embedding_db.csv", index=False, encoding="utf-8"
)
print(
f"""
Generated {total_embeddings} embeddings with a total of {total_tokens_used} tokens used. (Done!)
Succesfully saved embeddings to embedding_db.csv, printing dataframe head:
{embedding_df.head(5)}
"""
)
Go ahead and run it, this will take a little bit, but you’ll see regular status updates in your console.
Generated 10 embeddings so far with a total of 149 tokens used. (3%) Generated 20 embeddings so far with a total of 292 tokens used. (7%) Generated 30 embeddings so far with a total of 429 tokens used. (11%) Generated 40 embeddings so far with a total of 617 tokens used. (15%) Generated 50 embeddings so far with a total of 796 tokens used. (19%) ......
Don’t worry, this will cost you like $0.001 as the current price for Ada v2 tokens is $0.0001 per 1000 tokens. Make sure you let it run to 100%. You’ll see the basic structure of the DataFrame, which is just a table with three columns holding data:
quote author embedding 0 A man is great not because he hasn't failed; a... Confucius [-0.03543785214424133, -0.01987086981534958, 0... 1 Happiness is a gift and the trick is not to ex... Charles Dickens [-0.0009652393055148423, -0.009218506515026093... 2 Everyday you can take a tiny step in the right... Unknown [0.014701569452881813, 0.00505814841017127, 0.... 3 Sometimes adversity is what you need to face i... Zig Ziglar [0.0021813535131514072, -0.011264899745583534,... 4 Opportunity comes when you continually push fo... Sonia Ricotti [-0.008844518102705479, -0.042655542492866516,...
You’ll now also have a large CSV file named 'embedding_db.csv' in your base directory which contains the column names and then all the values for all quotes, separated by commas. This is like a simple embed