Hi and welcome to part 4 of this tutorial series! In this part, we’ll be continuing on with our exploration of the Google Gemini API and look at adding multiple modalities like images to our prompts. The first thing we’ll need is an image! Go ahead and download pink_vader.jpg here, or use any other image you like:
Now add a new folder to your project base directory and place the image inside it like so:
📂 GOOGLE_GEMINI
📂 images
🖼️ pink_vader.jpg
⚙️ .env
🐍 load_env.py
🐍 simple_chat.py
🐍 simple_request.py
🐍 utils.py
📄 Pipfile
📄 Pipfile.lock
Now in order for us to use these images with Google Gemini we need to send them over somehow. We cannot just put them into our text prompt, unfortunately. Not to worry, Google has a solution for this! We can use the File API. Basically, this API will allow us to upload an image to Google’s servers and we can then reference this uploaded image in our prompts as it is already on the Google servers.
Using the File API
So let’s set it up. First, create a new file named upload_image.py in your root project folder:
📂 GOOGLE_GEMINI
📂 images
🖼️ pink_vader.jpg
⚙️ .env
🐍 load_env.py
🐍 simple_chat.py
🐍 simple_request.py
🐍 upload_image.py ✨ New file
🐍 utils.py
📄 Pipfile
📄 Pipfile.lock
First we’ll just create a messy test run to see how it works. To start, we’ll need a genai object again which has our API key and stuff inside. Remember we made a separate file and function to abstract away this repetitive task? In upload_image.py:
from load_env import configure_genai genai = configure_genai()
Done! I love not doing the same thing over and over again! Now let’s define a path to our image and then upload it using the File API which we can access conveniently through the genai object:
image_path = "images/pink_vader.jpg"
image_upload = genai.upload_file(path=image_path, display_name="Pink Vader")
print(f"Uploaded file '{image_upload.display_name}' as: {image_upload.uri}")
This is all very readable right? We just define the relative path to our image in a string. Then we call the upload_file method on our genai object and pass in the path and a display_name for the image. The method returns a File object which we called image_upload and we print the display_name and the uri to see if it worked successfully.
Run this file and you should see something like the following:
Uploaded file 'Pink Vader' as: https://generativelanguage.googleapis.com/v1beta/files/holaithowqzp
This means that we have received a valid File object as a response and our call was successful. We can double-check by trying to retrieve the file from the API once again. Add the following code below:
file = genai.get_file(name=image_upload.name)
print(f"Retrieved file '{file.display_name}' from: {file.uri}")
Now go ahead and run the script one more time and you’ll see two lines this time:
Uploaded file 'Pink Vader' as: https://generativelanguage.googleapis.com/v1beta/files/lhq5updkqla4 Retrieved file 'Pink Vader' from: https://generativelanguage.googleapis.com/v1beta/files/lhq5updkqla4
We successfully uploaded and retrieved the image! Note that we have a different URL this time. This is because we made a second call so we uploaded the same image again to a separate file.
You can call genai.delete_file(sample_file.name) filling in the correct name from the File object to delete the file, but I would advise you not to worry about it or bother with all this. Google will delete old files for you after 48 hours, so there is an automatic cleanup!
Getting started on our chat with images
Now we could turn this into a function and import this from the other files, but there is really no need to do so, as genai.upload_file is already a one-liner. Create a new file named simple_chat_images.py in your project folder:
📂 GOOGLE_GEMINI
📂 images
🖼️ pink_vader.jpg
⚙️ .env
🐍 load_env.py
🐍 simple_chat.py
🐍 simple_chat_images.py ✨ New file
🐍 simple_request.py
🐍 upload_image.py
🐍 utils.py
📄 Pipfile
📄 Pipfile.lock
We’ll build on what we did in simple_chat.py but make it a bit more organized so that things stay readable as we up the complexity level. So open up your simple_chat_images.py file and first, we’ll basically copy the beginning of what we had:
from load_env import configure_genai
from utils import safety_settings
genai = configure_genai()
character = input("What is your favorite movie character? (e.g. Gollum): ")
movie = input("What movie are they from? (e.g. Lord of the Rings): ")
model = genai.GenerativeModel(
model_name="gemini-1.5-flash",
safety_settings=safety_settings.low,
system_instruction=f"You are helpful and provide good information but you are {character} from {movie}. You will stay in character as {character} no matter what. Make sure you find some way to relate your responses to {character}'s personality or the movie {movie} at least once every response.",
)
chat_session = model.start_chat(history=[])
This is pretty much what we had importing the configure and safety_settings utilities, then asking for a character and movie, setting up the model, and starting the chat session. We just pass in an empty list for the history as we don’t want to start with any previous history and as we know the history after we start will be managed for us.
From here on, we’ll do things a bit differently. Let’s get started on our main loop:
if __name__ == "__main__":
try:
while True:
text_query = input("\nPlease ask a question or type `/image` to upload an image first: ")
image_upload = None
if text_query.lower() == "/image":
image_path = input("Please provide the path to the image: ")
image_upload = genai.upload_file(path=image_path, display_name="User Image")
text_query = input("Please ask a question to go with your image upload: ")
# ...
We open up a try: block and start an infinite loop with while True: as we did before. This time we ask for either a query or for the user to type the command /image if they want to upload an image before they start their query. We set the image_upload variable to None so that the variable name exists and we can check if it has content or is still set to None later on.
If the query is /image we know that we need to get an image so we ask the user to provide the path to the image they want to upload. We then call the genai.upload_file method and store the result in the image_upload variable like we did in our test before. After this we ask for the text_query variable again, overriding the previous value of "/image" with the actual query the user wants to ask.
If the user did not type /image this while block is skipped and the text_query is simply the query the user typed the first time around. Now let’s finish this up:
if __name__ == "__main__":
try:
while True:
text_query = input("\nPlease ask a question or type `/image` to upload an image first: ")
image_upload = None
if text_query.lower() == "/image":
image_path = input("Please provide the path to the image: ")
image_upload = genai.upload_file(path=image_path, display_name="User Image")
text_query = input("Please ask a question to go with your image upload: ")
full_query = [image_upload, text_query] if image_upload else text_query
response = chat_session.send_message(full_query, stream=True)
for chunk in response:
if chunk.candidates[0].finish_reason == 3:
print(f"\n\033[1;31mPlease ask a more appropriate question!\033[0m", end="")
chat_session.rewind()
break
print(f"\033[1;34m{chunk.text}\033[0m", end="")
print("\n")
except KeyboardInterrupt:
print("Shutting down...")
We continue by setting a new variable named full_query which is a list containing the image_upload object and the text_query if image_upload is not None, otherwise it’s just the text_query. This is because to make a normal request we need to just give genai a simple string, but for an image/text request we give it a list with the image object and the string.
We then call chat_session.send_message with the full_query and stream=True to get a response in chunks. We iterate over the response chunks again and check for the finish_reason of 3 like before, making sure to rewind the chat session if an inappropriate response was blocked halfway thro