Hi and welcome to part 5 of the tutorial series. In this part we’ll make our chat much, much more powerful by looking into functions. But before we do, let’s continue on where we left off in the last part and implement our cost calculator function, and do some refactoring in our simple_chat_images.py file.
Let’s start by adding the import and also adding an extra constant at the top:
from load_env import configure_genai from utils import safety_settings from cost_calculator import print_cost_in_dollars # Add this line genai = configure_genai() MODEL_NAME = "gemini-1.5-flash" # Add this line
The reason we define the MODEL_NAME constant is because we will need it in multiple places now so we want to make sure there is only a single source of truth for it.
Skip down a couple of lines and make sure to update the model = genai.GenerativeModel() line to use the MODEL_NAME constant:
model = genai.GenerativeModel(
model_name=MODEL_NAME, # Update this line
safety_settings=safety_settings.low,
system_instruction=f"You are helpful and provide good information but you are {character} from {movie}. You will stay in character as {character} no matter what. Make sure you find some way to relate your responses to {character}'s personality or the movie {movie} at least once every response.",
)
Image upload error handling
Now, one of the problems you may have noticed while using the chat so far, is that if you accidentally enter the wrong path to the image, the program will crash. We can fix this by adding a try-except block around the image upload code, but I don’t want the if __name__ == "__main__": block to get too big, so let’s create a new function in the simple_chat_images.py file called upload_image.
I’m going to put this function right in between the chat_session variable declaration and the if __name__ == "__main__": block:
chat_session = model.start_chat(history=[])
## insert the new function here ##
def upload_image():
while True:
try:
image_path = input("Please provide the path to the image: ")
image_upload = genai.upload_file(path=image_path, display_name="User Image")
return image_upload
except FileNotFoundError:
print("File not found. Please try again with the correct path.")
if __name__ == "__main__":
#...
First we start an infinite loop using while True and then we try to get the image and upload it using genai.upload_file. If this is successful, we return the image_upload object to break out of the loop. If the path is not correct we simply catch the FileNotFoundError and print a message to the user, letting them try again without having to start all over again.
Putting it all together
Now we can refactor our if __name__ == "__main__": block to use this new function and also use our new print_cost_in_dollars function:
if __name__ == "__main__":
try:
while True:
text_query = input("\nPlease ask a question or type `/image` to upload an image first: ")
image_upload = None
## Remove several lines in this block ##
if text_query.lower() == "/image":
image_upload = upload_image() # Add this line
text_query = input("Please ask a question to go with your image upload: ")
full_query = [image_upload, text_query] if image_upload else text_query
response = chat_session.send_message(full_query, stream=True)
for chunk in response:
if chunk.candidates[0].finish_reason == 3:
print(f"\n\033[1;31mPlease ask a more appropriate question!\033[0m", end="")
chat_session.rewind()
break
print(f"\033[1;34m{chunk.text}\033[0m", end="")
print("\n")
# Add this line
print_cost_in_dollars(response.usage_metadata, MODEL_NAME)
except KeyboardInterrupt:
print("Shutting down...")
Giving it a good test
Ok, now let’s try it out! Run your file and have a chat. Make sure you also type a wrong image path to see if the error handling works as expected.
What is your favorite movie character? (e.g. Gollum): Bambi What movie are they from? (e.g. Lord of the Rings): Bambi Please ask a question or type `/image` to upload an image first: You must defeat Godzilla for me. He is coming and you are the only one who can save the world! Oh, my... Godzilla? That sounds very scary! I don't think I can defeat him. He's so big and strong! But I know my father would say to be brave, like when I learned to jump over the stream. I'm not sure I can be as brave as you need me to be. Maybe Thumper can help? He's really good at jumping too! Cost: $0.000114800
So far, so good. We can see that everything still works and the cost is displayed. Note that this will be 10x higher when using the pro model compared to the flash model, but it is a very small amount in any case.
Please ask a question or type `/image` to upload an image first: /image Please provide the path to the image: images/typo.jpg File not found. Please try again with the correct path. Please provide the path to the image: images/pink_vader.jpg Please ask a question to go with your image upload: Do you think this guy can help us to defeat Godzilla? Oh, wow! He looks so big and strong! But he also looks very scary... just like the Great Prince of the Forest! I think he would be a great protector of the forest. But I'm not sure he would want to fight Godzilla. I think he would prefer to just watch from a distance, like when I watched my father fight the other deer for the first time. Cost: $0.000236250
Great! The error handling works as expected and we got to retry our image upload. The cost is higher for the second call as we have some extra tokens from the image upload and the history from the first call in there.
Taking it to the next level
Now images and text are fun and all, but I want a more powerful AI, one that can execute certain tasks for us, like fetching some information that we need for instance. So let’s have a look at function calling, where we can give Gemini the ability to execute a certain code function for us. We’ll start with a simple but real and genuinely useful example.
I’m going to use the weatherapi.com API for this example, as it is free, easy to get an API key, and it will give us a great example use case for function calling and even parallel function calling afterward.
First, sign up for a free account on weatherapi.com and get your API key. They will automatically give you a pro trial for 14 days for free you don’t have to provide any credit card details and it will automatically switch back to a free account after the trial period, so there is zero hassle involved. The free account is more than powerful enough for our tutorial use.
Find your API key and copy it, pasting it as a new entry into your existing .env file:
GEMINI_API_KEY=your_api_key_here WEATHER_API_KEY=your_api_key_here
As a reminder, make sure not to use any spaces or quotes around the key, just the key itself. Save the file and close it.
Fetching the weather
Now let’s go ahead and create a simple function that will fetch the weather for a given location so that we can give this function to Gemini later on, but let’s first just focus on writing the function itself. Create a new file called weather.py in your main project directory:
📂 GOOGLE_GEMINI
📂 images
🖼️ pink_vader.jpg
⚙️ .env
🐍 cost_calculator.py
🐍 load_env.py
🐍 simple_chat.py
🐍 simple_chat_images.py
🐍 simple_request.py
🐍 upload_image.py
🐍 utils.py
🐍 weather.py ✨ New file
📄 Pipfile
📄 Pipfile.lock
Now let’s first code up a quick function to get the weather in a specific location by calling the API. We’ll use the requests library for this, so make sure to install it by running pipenv install requests in your terminal if you need to. Then start with the imports in your weather.py file:
import os from json import dumps import requests from dotenv import load_dotenv load_dotenv()
Remember the load_dotenv() function from the dotenv library? We now have two API keys in our .env file, so running this function will load up both of them. The os library is used to get the API keys from the environment variables like before. requests is used to make an internet request to the API, and dumps is used to convert the JSON response to a string as LLMs work with strings and not response objects.
Now let’s code up the function itself:
def get_current_weather(location):
if not location:
return (
"Please provide a location and call the get_current_weather_function again."
)
API_params = {
"key": os.environ["WEATHER_API_KEY"],
"q": location,
"aqi": "no",
"alerts": "no",
}
response = requests.get(
"http://api.weatherapi.com/v1/current.json", params=API_params
)
str_response: str = dumps(response.json())
return str_response
The function named get_current_weather takes a single argument, location, which is the location for which we want to get the weather. If no location is provided, the function will return a message asking for a location to be provided. The reason we return a string is that this function is written for LLMs, and giving a string response as to what has gone wrong may help the LLM to fix its mistake and try calling the function again.
The function then constructs the API parameters in a dictionary, loading our API key, and the location, and skipping over the air quality index and alerts that we don’t need. We then make a GET request to the weather API, passing in the parameter dictionary, which the request library will convert for us.
The response is a Response object from the requests library which comes with the .json() method to convert the received JSON response to a Python dictionary. As LLMs don’t work with Python dictionaries we immediately convert this to a string using the dumps function from the json library and return this string.
Making the function LLM-friendly
Now that’s all nice and well, and fine for humans to read and understand. But how can we give this function to a LLM and have it know what to do and input? We need to make some slight changes for Gemini to be able to understand our function. Luckily we can make these changes within the function itself without any real complexity. Change your function as follows:
def get_current_weather(location: str) -> str:
"""Get the current weather for a location using the WeatherAPI.
Args:
location (str): The location to get the current weather for, e.g. "London".
Returns:
str: A JSON string containing the current weather data in detail.
"""
if not location:
return (
"Please provide a location and call the get_current_weather_function again."
)
API_params = {
"key": os.environ["WEATHER_API_KEY"],
"q": location,
"aqi": "no",
"alerts": "no",
}
response: requests.models.Response = requests.get(
"http://api.weatherapi.com/v1/current.json", params=API_params
)
str_response: str = dumps(response.json())
return str_response
The first thing we added is the :str type-hints in the function definition. This is important as it tells Gemini that the location argument is a string. Since Gemini is supposed to call our function it should know exactly what the required input arguments and types are right?
After that, the biggest change is the very detailed docstring we added. First we state what the function’s purpose is so that Gemini can decide whether or not it should call this function in the context of the conversation:
Get the current weather for a location using the WeatherAPI.
Then we list the arguments that the function takes, and what type they should be. In this case, we have only our single input argument again. Make sure to also include a good example:
Args:
location (str): The location to get the current weather for, e.g. "London".
And last but not least, we describe the function’s return value, so that Gemini knows exactly what it can expect in return if it calls this function, helping it decide whether or not calling this function is the right thing to do at any point in the conversation:
Returns:
str: A JSON string containing the current weather data in detail.
The rest of the function is exactly identical to what we wrote before, we basically just added a load of documentation. Gemini is set up to read these type-hints and docstrings when you give it a function so following this rough structure will work for other functions as well.
Testing the function
Now scroll down to the bottom and give this a quick test by adding a if __name__ == "__main__": block and calling the function with a location:
if __name__ == "__main__":
print(get_current_weather("Seoul"))
print(get_current_weather("Amsterdam"))
Use any location you want for the test, I’m using Seoul and Amsterdam here. Run the weather.py file and you should see something like the following in your terminal output:
{"location": {"name": "Seoul", "region": "", "country": "South Korea", "lat": 37.57, "lon": 127.0, "tz_id": "Asia/Seoul", "localtime_epoch": 1719121763, "localtime": "2024-06-23 14:49"}, "current": {"last_updated_epoch": 1719121500, "last_updated": "2024-06-23 14:45", "temp_c": 29.2, "temp_f": 84.6, "is_day": 1, "condition": {"text": "Partly cloudy", "icon": "//cdn.weatherapi.com/weather/64x64/day/116.png", "code": 1003}, "wind_mph": 9.4, "wind_kph": 15.1, "wind_degree": 250, "wind_dir": "WSW", "pressure_mb": 998.0, "pressure_in": 29.47, "precip_mm": 0.1, "precip_in": 0.0, "humidity": 66, "cloud": 50, "feelslike_c": 30.0, "feelslike_f": 86.1, "windchill_c": 30.8, "windchill_f": 87.4, "heatindex_c": 32.4, "heatindex_f": 90.2, "dewpoint_c": 18.4, "dewpoint_f": 65.2, "vis_km": 10.0, "vis_miles": 6.0, "uv": 7.0, "gust_mph": 12.3, "gust_kph": 19.9}}
{"location": {"name": "Amsterdam", "region": "North Holland", "country": "Netherlands", "lat": 52.37, "lon": 4.89, "tz_id": "Europe/Amsterdam", "localtime_epoch": 1719121764, "localtime": "2024-06-23 7:49"}, "current": {"last_updated_epoch": 1719121500, "last_updated": "2024-06-23 07:45", "temp_c": 16.1, "temp_f": 61.0, "is_day": 1, "condition": {"text": "Sunny", "icon": "//cdn.weatherapi.com/weather/64x64/day/113.png", "code": 1000}, "wind_mph": 4.3, "wind_kph": 6.8, "wind_degree": 180, "wind_dir": "S", "pressure_mb": 1018.0, "pressure_in": 30.06, "precip_mm": 0.0, "precip_in": 0.0, "humidity": 94, "cloud": 0, "feelslike_c": 16.1, "feelslike_f": 61.0, "windchill_c": 16.4, "windchill_f": 61.4, "heatindex_c": 16.4, "heatindex_f": 61.4, "dewpoint_c": 13.6, "dewpoint_f": 56.6, "vis_km": 10.0, "vis_miles": 6.0, "uv": 5.0, "gust_mph": 7.1, "gust_kph": 11.4}}
We have one object for each call made, and they are packed with information including the local time, temperature, wind speed and direction, pressure, humidity, and more. It’s quite unpleasant to read through this format though, but that is where Gemini will help us out!
Creating our function chat
Now go ahead and make a copy of your simple_chat_images.py file and rename the copy to function_chat.py. Normally you should never make copies of the same code, violating the “don’t repeat yourself” principle, but for tutorial purposes, it might be nice to have a snapshot of the code at different points, with and without function calling:
📂 GOOGLE_GEMINI
📂 images
🖼️ pink_vader.jpg
⚙️ .env
🐍 cost_calculator.py
🐍 function_chat.py ✨ Copy of `simple_chat_images.py`
🐍 load_env.py
🐍 simple_chat.py
🐍 simple_chat_images.py
🐍 simple_request.py
🐍 upload_image.py
🐍 utils.py
🐍 weather.py
📄 Pipfile
📄 Pipfile.lock
So go ahead and open up function_chat.py and let’s make some changes. You will be surprised how few changes we’ll actually need to make here to add function calling. Let’s start with the imports: