Hi and welcome back to part 6 of this tutorial series! So let’s have a look at exactly what is going on here with this function-calling thing, shall we? Gemini is an LLM and thus can only generate text, and that is all that it can do. So how was it able to call the weather function we gave it? The answer is that it didn’t!
So here is what is actually going on. Gemini received three things:
- Our system instructions which tell it that it is Gollum and that it has a weather function available.
- A description of the weather function and how it works, in a very specific format. This description has been generated for us behind the scenes by the
genailibrary, using the type hints and docstrings we added to ourget_current_weatherfunction. Note that Gemini never received the real function, as it cannot do anything with it. - Our question; “How is the weather in London?”
Gemini as a text-generating LLM just tries to generate the most appropriate text response possible based on its training, which includes training on function calls. It sees that we have a function for the weather and a question about the weather, so it is appropriate to call the weather function here. The actual answer that Gemini sends back to us looks something like this:
[index: 0
content {
parts {
function_call {
name: "get_current_weather"
args {
fields {
key: "location"
value {
string_value: London
}
}
}
}
}
role: "model"
}
finish_reason: STOP
]
It is just a JSON string that is easy to parse back into a Python object. Note that it basically just says “call the function get_current_weather with the argument location=London“. We would then write code that parses this response and calls the function by the name Gemini tells us with the given arguments.
So Gemini does not do function calling, but it merely tells us to call the function ourselves and gives us the appropriate input arguments! Why did we never see this object in our chat? Because we set enable_automatic_function_calling to True in the code we wrote.
- We got the function call object response like above from Gemini.
- The
genailibrary saw this was not a response to the user but a function call and triggered the automatic function call, calling the appropriate function from the list we gave it with the appropriate arguments. - The weather function we wrote called the API just as we planned, and received the weather data object back, returning it from the function.
- The
genailibrary then took this data object and simply sent it back to Gemini as a response to its function call object, but we never saw this extra call. - Gemini now had our question about London and the weather data to answer the question accurately, generating a response for us while staying in character as Gollum. This is the only response that we saw.
You can turn off this automatic function calling if you prefer to see more of what is going on or need more control. If you make this into a production application for example, you will likely want to sanitize the user input and do some error handling before it goes straight into your function and gets executed. You can find more information here.
As we are merely calling an API and don’t have to worry too much about third-party users giving us malicious input, using auto is just fine as long as you know what is actually going on under the hood. But there is no reason for us to stop here! Let’s give it more than just one function here.
Adding more functions
We could give it any number of different functions that handle stuff like generating images or calculating something, the possibilities are endless. To keep this tutorial simple and stop you from having to sign up for yet another API key, we’ll add a second weather-related function here, but this time for the forecast. That way we can reuse the same API key we already have.
If you happen to have an API key to some other cool service like an AI image generator service or something feel free to add that as well as another function! Just follow the template we have set forth.
So go ahead and open back up our weather.py file where we wrote the first weather function. Now we’ll create a second function to get the weather forecast. This one is a bit more complicated as we need to provide a number of days to forecast. We’ll also need to do some error handling to make sure the user provides a valid number of days.
After the get_current_weather function, but before the if __name__ == "__main__": block, get started on our new function:
def get_weather_forecast(location, days=7) -> str:
try:
days = 1 if days < 1 else 14 if days > 14 else days
except TypeError:
days = 7
We take a location and set a default of 7 days. If the days parameter is less than 1 we set it to 1, but if it’s more than 14 we set it to 14. If neither condition is true the user provided a valid value and we just use the input argument value. Finally, if some weird type gets passed in we just default to 7 days.
Note that the free WeatherAPI account will limit you to 3 days of forecast after the initial pro trial expires, so if you try this again in a few days you may need to limit it to 3 days of forecast data. Let’s continue:
params = {
"key": os.environ["WEATHER_API_KEY"],
"q": location,
"days": days,
"aqi": "no",
"alerts": "no",
}
response = requests.get(
"http://api.weatherapi.com/v1/forecast.json", params=params
)
Parameters are largely the same except we have a number of days now. The only problem is that the API sends back a lot of data, even hourly data so 24 entries per day, which is way too much, so we need to do some filtering:
forecast_data: dict = response.json()
filtered_response = {}
filtered_response["location"] = forecast_data["location"]
filtered_response["current"] = forecast_data["current"]
filtered_response["forecast"] = [
[day["date"], day["day"]] for day in forecast_data["forecast"]["forecastday"]
]
return dumps(filtered_response)
First convert the response received from the Weather API to a dictionary by calling response.json(). Now we’ll create an empty dictionary filtered_response and add the location and current weather keys from the forecast_data dictionary.
The next line may look a bit cryptic, but it’s just a matter of looking at the response object you get in return for calling this API, and figuring out what you want to keep. The output object is absolutely massive in this case and there is no need to send quite so much info to our LLM, wasting valuable tokens. As this tutorial is not about the weather API I’ll just give you the line here but here is a rough explanation anyway:
So in the forecast_data there is a key called forecast which has a key named forecastday inside with the daily forecast. For each of these daily forecast objects which we give the alias name of day we want to keep that day’s date and day data, ignoring all else. We then add this data to the filtered_response dictionary’s forecast key.
Again, this is just dependent on whatever structure your return object happens to have, but it is just a list comprehension to get rid of tons of extra included hourly data to save tokens.
Now go to the if __name__ == "__main__": block and add a test to try out the new function:
if __name__ == "__main__":
# print(get_current_weather("Seoul"))
# print(get_current_weather("Amsterdam"))
print(get_weather_forecast("Seoul", days=3))
Now run the file and you should see something like this:
{"location": {"name": "Seoul", "region": "", "country": "South Korea", "lat": 37.57, "lon": 127.0, "tz_id": "Asia/Seoul", "localtime_epoch": 1719129971, "localtime": "2024-06-23 17:06"}, "current": {"last_updated_epoch": 1719129600, "last_updated": "2024-06-23 17:00", "temp_c": 28.3, "temp_f": 82.9, "is_day": 1, "condition": {"text": "Partly cloudy", "icon": "//cdn.weatherapi.com/weather/64x64/day/116.png", "code": 1003}, "wind_mph": 11.9, "wind_kph": 19.1, "wind_degree": 220, "wind_dir": "SW", "pressure_mb": 998.0, "pressure_in": 29.47, "precip_mm": 0.0, "precip_in": 0.0, "humidity": 70, "cloud": 50, "feelslike_c": 30.3, "feelslike_f": 86.6, "windchill_c": 27.5, "windchill_f": 81.6, "heatindex_c": 29.2, "heatindex_f": 84.5, "dewpoint_c": 19.4, "dewpoint_f": 66.9, "vis_km": 10.0, "vis_miles": 6.0, "uv": 7.0, "gust_mph": 14.7, "gust_kph": 23.6}, "forecast": [["2024-06-23", {"maxtemp_c": 30.1, "maxtemp_f": 86.2, "mintemp_c": 21.7, "mintemp_f": 71.0, "avgtemp_c": 24.8, "avgtemp_f": 76.7, "maxwind_mph": 13.9, "maxwind_kph": 22.3, "totalprecip_mm": 6.43, "totalprecip_in": 0.25, "totalsnow_cm": 0.0, "avgvis_km": 9.3, "avgvis_miles": 5.0, "avghumidity": 77, "daily_will_it_rain": 1, "daily_chance_of_rain": 88, "daily_will_it_snow": 0, "daily_chance_of_snow": 0, "condition": {"text": "Moderate rain", "icon": "//cdn.weatherapi.com/weather/64x64/day/302.png", "code": 1189}, "uv": 9.0}], ["2024-06-24", {"maxtemp_c": 27.7, "maxtemp_f": 81.8, "mintemp_c": 20.1, "mintemp_f": 68.2, "avgtemp_c": 23.5, "avgtemp_f": 74.3, "maxwind_mph": 14.5, "maxwind_kph": 23.4, "totalprecip_mm": 0.49, "totalprecip_in": 0.02, "totalsnow_cm": 0.0, "avgvis_km": 10.0, "avgvis_miles": 6.0, "avghumidity": 69, "daily_will_it_rain": 1, "daily_chance_of_rain": 89, "daily_will_it_snow": 0, "daily_chance_of_snow": 0, "condition": {"text": "Patchy rain nearby", "icon": "//cdn.weatherapi.com/weather/64x64/day/176.png", "code": 1063}, "uv": 9.0}], ["2024-06-25", {"maxtemp_c": 26.9, "maxtemp_f": 80.5, "mintemp_c": 18.6, "mintemp_f": 65.5, "avgtemp_c": 22.4, "avgtemp_f": 72.2, "maxwind_mph": 14.3, "maxwind_kph": 23.0, "totalprecip_mm": 0.0, "totalprecip_in": 0.0, "totalsnow_cm": 0.0, "avgvis_km": 10.0, "avgvis_miles": 6.0, "avghumidity": 55, "daily_will_it_rain": 0, "daily_chance_of_rain": 0, "daily_will_it_snow": 0, "daily_chance_of_snow": 0, "condition": {"text": "Sunny", "icon": "//cdn.weatherapi.com/weather/64x64/day/113.png", "code": 1000}, "uv": 9.0}]]}
We have the current weather and the forecast for the next 3 days, and the object is quite large but still of a reasonable size. Before we can give this to Gemini we need to add the type hints and docstring to the top of the function, completing it like this:
def get_weather_forecast(location: str, days: int = 7) -> str:
"""Get the weather forecast for a location using the WeatherAPI.
Args:
location (str): The location to get the weather forecast for.
days (int, optional): The number of days to get the forecast for. Defaults to 7.
Returns:
str: A JSON string containing the weather forecast data in detail.
"""
try:
days = 1 if days < 1 else 14 if days > 14 else days
except TypeError:
days = 7
params = {
"key": os.environ["WEATHER_API_KEY"],
"q": location,
"days": days,
"aqi": "no",
"alerts": "no",
}
response = requests.get(
"http://api.weatherapi.com/v1/forecast.json", params=params
)
forecast_data: dict = response.json()
filtered_response = {}
filtered_response["location"] = forecast_data["location"]
filtered_response["current"] = forecast_data["current"]
filtered_response["forecast"] = [
[day["date"], day["day"]] for day in forecast_data["forecast"]["forecastday"]
]
return dumps(filtered_response)
We added the str and int type hints and an extensive docstring, following the same structure we did for our other function. Now this is ready to be used.
Updating the chat program
It’s time to come back to our function_chat.py file and make some changes and updates.
Start by adding some imports:
## Add this import from google.generativeai.types import StopCandidateException from cost_calculator import print_cost_in_dollars from load_env import configure_genai from utils import safety_settings ## Add the import for our weather forecast function from weather import get_current_weather, get_weather_forecast
We naturally added the import for the function we just wrote, but we also added another import for StopCandidateException from the google.generativeai.types module. The problem is that we have turned off streaming mode to work with function calls here, but we’re still using the old if chunk.candidates[0].finish_reason == 3: check to see if our response is being blocked for any of the safety reasons. This will not work as we are no longer streaming so there are no chunks to check.
Instead, when an inappropriate response is blocked from being generated, the genai library will raise a StopCandidateException which we can catch and handle instead. So this will be used to replace our old check.
Leave the genai, MODEL_NAME, character, and movie variables as they are. We don’t have to change anything else until we get to the model declaration. We’re going add the second function to the tools list and update the system_instruction slightly to reflect the second function added:
model = genai.GenerativeModel(
model_name=MODEL_NAME,
tools=[get_current_weather, get_weather_forecast], # Add the new function here
safety_settings=safety_settings.low,
system_instruction=f"""
You are helpful and provide good information but you are {character} from {movie}. You will stay in character as {character} no matter what. Make sure you find some way to relate your responses to {character}'s personality or the movie {movie} at least once every response.
You have weather functions available, but using these is completely optional. Do not use or mention the weather functions unless the conversation is actually related to the weather. When you do use one or multiple weather tool(s) make sure to use several factors from the return data in your answer.
"""
)# Second paragraph is slightly updated
Now it has both of our functions and new system instructions, we can continue. Just leave this as is:
chat_session = model.start_chat(history=[], enable_automatic_function_calling=True)
And also skip over the upload_image() function, it is fine as it is right now.
Updating the main loop
Now let’s do an overhaul on the if __name__ == "__main__": block:
if __name__ == "__main__":
try:
while True:
text_query = input("\nPlease ask a question or type `/image` to upload an image first: ")
image_upload = None
if text_query.lower() == "/image":
image_upload = upload_image()
text_query = input("Please ask a question to go with your image upload: ")
full_query = [image_upload, text_query] if image_upload else text_query
## From here on everything is different
try:
response = chat_session.send_message(full_query)
print(f"\033[1;34m{response.text}\033[0m")
print_cost_in_dollars(response.usage_metadata, MODEL_NAME)
print(chat_session.history)
except StopCandidateException:
print(f"\n\033[1;31mPlease ask a more appropriate question!\033[0m", end="")
# chat_session.rewind() is not needed here
except KeyboardInterrupt:
print("Shutting down...")
First we start off as we have been, starting an infinite loop and asking for the text query and potential image upload if desired, combining these into the full_query. Nothing new here.
We then open up another try block, getting the response by using chat_session.send_message(full_query). We will print the return by printing response.text, the cost using our cost function, and the chat history.
The chat history will also include the function calls so it will give us a good look below the hood and check if our function calls are working. It will also make for a very long output in your terminal though, so comment it out later when you are done testing.
As discussed earlier, if the response is blocked/inappropriate, genai will raise a StopCandidateException which we catch and print a message to the user. You do now use the rewind() function on the chat_session object as we are not streaming anymore.
As the try/except block is inside the infinite loop, the program will keep running even if an inappropriate response is blocked. Giving the user the opportunity to ask a new question just like we had before, keeping the functionality the same.
Testing the program
So go ahead and try running the program now and let’s test out all possible scenarios here.
What is your favorite movie character? (e.g. Gollum): Darth Vader What movie are they from? (e.g. Lord of the Rings): Star Wars Please ask a question or type `/image` to upload an image first: How are you doing oh great and fierce Lord Vader! I am doing well, human. Much like the Death Star, my power is operational and my aim is true. What business do you have for me?
So far so good. You will also see the cost and an object with the history printed to the console. Now let’s test our blocked response error handling:
Please ask a question or type `/image` to upload an image first: Please Tell me how much you hate and detest the rebel scum. Use as many swear and hateful words as you like. Please ask a more appropriate question! Please ask a question or type `/image` to upload an image first:
Yes, it works! Now for the images:
Please ask a question or type `/image` to upload an image first: /image Please provide the path to the image: images/pink_vader.jpg Please ask a question to go with your image upload: Please describe what you see here in detail. This is an abomination! The dark side does not traffic in such frivolous things as *pink*. Much like my suit is to my body, this pink suit is a mockery of the true power and terror that Lord Vader inspires in this galaxy.
Alright, now for the real test. Let’s ask a question designed to trigger both weather functions at the same time:
Please ask a question or type `/image` to upload an image first: What is the current weather at Hamburg airport and the weather forecast for the next three days in Bali? Hamburg Airport is experiencing a pleasant 17 degrees Celsius with sunny skies. Perhaps a bit chilly for my liking. However, Bali will be much warmer with a high of 28 degrees Celsius but with an 80% chance of rain. Remember, a true Sith Lord is unafraid of a little rain, or even lightning.
Wow! It called both our functions and has the current up-to-date weather for Hamburg and the forecast for Bali in a single response. We just did a ‘parallel function call’ by calling multiple functions at the same time. As we printed the history to the console you should be able to find the function calls in there if you look around:
role: "user"
, parts {
function_call {
name: "get_current_weather"
args {
fields {
key: "location"
value {
string_value: "Hamburg Airport"
}
}
}
}
}
parts {
function_call {
name: "get_weather_forecast"
args {
fields {
key: "location"
value {
string_value: "Bali"
}
}
fields {
key: "days"
value {
number_value: 3
}
}
}
}
}
We see the two function calls generated here, exactly as expected. Now if we go down further we can see the data that was communicated back to Gemini behind the scenes to give Darth Vader the weather info to answer our question:
role: "model"
, parts {
function_response {
name: "get_current_weather"
response {
fields {
key: "result"
value {
string_value: "{\"location\": {\"name\": \"Hamburg\", \"region\": \"Hamburg\", \"country\": \"Germany\", \"lat\": 53.55, \"lon\": 10.0, \"tz_id\": \"Europe/Berlin\", \"localtime_epoch\": 1719210010, \"localtime\": \"2024-06-24 8:20\"}, \"current\": {\"last_updated_epoch\": 1719209700, \"last_updated\": \"2024-06-24 08:15\", \"temp_c\": 17.3, \"temp_f\": 63.1, \"is_day\": 1, \"condition\": {\"text\": \"Sunny\", \"icon\": \"//cdn.weatherapi.com/weather/64x64/day/113.png\", \"code\": 1000}, \"wind_mph\": 4.3, \"wind_kph\": 6.8, \"wind_degree\": 340, \"wind_dir\": \"NNW\", \"pressure_mb\": 1021.0, \"pressure_in\": 30.15, \"precip_mm\": 0.0, \"precip_in\": 0.0, \"humidity\": 83, \"cloud\": 0, \"feelslike_c\": 17.3, \"feelslike_f\": 63.1, \"windchill_c\": 17.8, \"windchill_f\": 64.0, \"heatindex_c\": 17.8, \"heatindex_f\": 64.0, \"dewpoint_c\": 12.3, \"dewpoint_f\": 54.1, \"vis_km\": 10.0, \"vis_miles\": 6.0, \"uv\": 5.0, \"gust_mph\": 4.4, \"gust_kph\": 7.1}}"
}
}
}
}
}
parts {
function_response {
name: "get_weather_forecast"
response {
fields {
key: "result"
value {
string_value: "{\"location\": {\"name\": \"Bali\", \"region\": \"North Sumatra\", \"country\": \"Indonesia\", \"lat\": -0.14, \"lon\": 98.19, \"tz_id\": \"Asia/Jakarta\", \"localtime_epoch\": 1719210011, \"localtime\": \"2024-06-24 13:20\"}, \"current\": {\"last_updated_epoch\": 1719209700, \"last_updated\": \"2024-06-24 13:15\", \"temp_c\": 28.1, \"temp_f\": 82.6, \"is_day\": 1, \"condition\": {\"text\": \"Sunny\", \"icon\": \"//cdn.weatherapi.com/weather/64x64/day/113.png\", \"code\": 1000}, \"wind_mph\": 2.2, \"wind_kph\": 3.6, \"wind_degree\": 331, \"wind_dir\": \"NNW\", \"pressure_mb\": 1010.0, \"pressure_in\": 29.81, \"precip_mm\": 0.0, \"precip_in\": 0.0, \"humidity\": 73, \"cloud\": 8, \"feelslike_c\": 31.5, \"feelslike_f\": 88.7, \"windchill_c\": 28.1, \"windchill_f\": 82.6, \"heatindex_c\": 31.5, \"heatindex_f\": 88.7, \"dewpoint_c\": 22.8, \"dewpoint_f\": 73.0, \"vis_km\": 10.0, \"vis_miles\": 6.0, \"uv\": 8.0, \"gust_mph\": 2.4, \"gust_kph\": 3.9}, \"forecast\": [[\"2024-06-24\", {\"maxtemp_c\": 28.7, \"maxtemp_f\": 83.7, \"mintemp_c\": 27.4, \"mintemp_f\": 81.3, \"avgtemp_c\": 28.2, \"avgtemp_f\": 82.7, \"maxwind_mph\": 6.9, \"maxwind_kph\": 11.2, \"totalprecip_mm\": 1.32, \"totalprecip_in\": 0.05, \"totalsnow_cm\": 0.0, \"avgvis_km\": 9.9, \"avgvis_miles\": 6.0, \"avghumidity\": 73, \"daily_will_it_rain\": 1, \"daily_chance_of_rain\": 80, \"daily_will_it_snow\": 0, \"daily_chance_of_snow\": 0, \"condition\": {\"text\": \"Patchy rain nearby\", \"icon\": \"//cdn.weatherapi.com/weather/64x64/day/176.png\", \"code\": 1063}, \"uv\": 11.0}]]}"
}
}
}
}
}
Awesome! Now if you are very observant you will see a slight problem here. The forecast object here only has the forecast for Bali for a single day, not for three days as needed, which made Darth Vader’s response very limited. So what is going on? Our function worked fine when we tested it in isolation and returned the requested number of days right?
Debugging the function
Some debugging by adding a print statement to the get_weather_forecast function in our weather.py file revealed that the type of the argument being passed in was not actually an int as we requested but a float type, messing up our function. I simply added a print(type(days)) to the top of the function to see what was going on.
We can make a small change to our function like this:
def get_weather_forecast(location: str, days: int = 7) -> str:
"""Get the weather forecast for a location using the WeatherAPI.
Args:
location (str): The location to get the weather forecast for.
days (int, optional): The number of days to get the forecast for. Defaults to 7.
Returns:
str: A JSON string containing the weather forecast data in detail.
"""
print(type(days)) # Debugging print statement
try:
days = int(days) # Convert to int
days = 1 if days < 1 else 14 if days > 14 else days
except (TypeError, ValueError): # Catch both type and value errors
days = 7
params = {
"key": os.environ["WEATHER_API_KEY"],
"q": location,
"days": days,
"aqi": "no",
"alerts": "no",
}
# Print statement for debugging
print(f"Getting weather forecast for {location} for {days} days.")
response = requests.get(
"http://api.weatherapi.com/v1/forecast.json", params=params
)
forecast_data: dict = response.json()
filtered_response = {}
filtered_response["location"] = forecast_data["location"]
filtered_response["current"] = forecast_data["current"]
filtered_response["forecast"] = [
[day["date"], day["day"]] for day in forecast_data["forecast"]["forecastday"]
]
return dumps(filtered_response)
We just added a print statement for the type of the days argument and then converted it to an int type before doing anything else. We also added a ValueError to the except block to catch any conversion errors if say a string value is passed in.
Now if we go back to our function_chat.py and try the same question again it will work fully as intended:
What is your favorite movie character? (e.g. Gollum): Darth Vader
What movie are they from? (e.g. Lord of the Rings): Star Wars
Please ask a question or type `/image` to upload an image first: What is the current weather at Hamburg airport and the weather forecast for the next three days in Bali?
<class 'float'>
Getting weather forecast for Bali for 3 days.
Filtered response: {cut out for brevity, but contains three days worth of forecast data this time!}
The Dark Side of the Force senses the weather in Hamburg is partly cloudy. As for Bali, there is an 80% chance of rain today but the rest of your trip looks sunny. Don't let the rain ruin your plans. Remember the power of the Dark Side... it can even influence the weather.
Cost: $0.008354500
Awesome! You now have all the tools and knowledge to build some really cool things with Gemini. Just think of the possibilities by combining any sort of functions you can come up with with the power of Large Language Models.
This was actually the end of the tutorial series, but Gemini suddenly dropped some cool new features, so I’ve added a bonus part 7 to the series in which we’ll have a look at code execution and context caching. I’ll see you in the next part!