Hi and welcome back to part 6 of this tutorial series! So let’s have a look at exactly what is going on here with this function-calling thing, shall we? Gemini is an LLM and thus can only generate text, and that is all that it can do. So how was it able to call the weather function we gave it? The answer is that it didn’t!
So here is what is actually going on. Gemini received three things:
- Our system instructions which tell it that it is Gollum and that it has a weather function available.
- A description of the weather function and how it works, in a very specific format. This description has been generated for us behind the scenes by the
genailibrary, using the type hints and docstrings we added to ourget_current_weatherfunction. Note that Gemini never received the real function, as it cannot do anything with it. - Our question; “How is the weather in London?”
Gemini as a text-generating LLM just tries to generate the most appropriate text response possible based on its training, which includes training on function calls. It sees that we have a function for the weather and a question about the weather, so it is appropriate to call the weather function here. The actual answer that Gemini sends back to us looks something like this:
[index: 0
content {
parts {
function_call {
name: "get_current_weather"
args {
fields {
key: "location"
value {
string_value: London
}
}
}
}
}
role: "model"
}
finish_reason: STOP
]
It is just a JSON string that is easy to parse back into a Python object. Note that it basically just says “call the function get_current_weather with the argument location=London“. We would then write code that parses this response and calls the function by the name Gemini tells us with the given arguments.
So Gemini does not do function calling, but it merely tells us to call the function ourselves and gives us the appropriate input arguments! Why did we never see this object in our chat? Because we set enable_automatic_function_calling to True in the code we wrote.
- We got the function call object response like above from Gemini.
- The
genailibrary saw this was not a response to the user but a function call and triggered the automatic function call, calling the appropriate function from the list we gave it with the appropriate arguments. - The weather function we wrote called the API just as we planned, and received the weather data object back, returning it from the function.
- The
genailibrary then took this data object and simply sent it back to Gemini as a response to its function call object, but we never saw this extra call. - Gemini now had our question about London and the weather data to answer the question accurately, generating a response for us while staying in character as Gollum. This is the only response that we saw.
You can turn off this automatic function calling if you prefer to see more of what is going on or need more control. If you make this into a production application for example, you will likely want to sanitize the user input and do some error handling before it goes straight into your function and gets executed. You can find more information here.
As we are merely calling an API and don’t have to worry too much about third-party users giving us malicious input, using auto is just fine as long as you know what is actually going on under the hood. But there is no reason for us to stop here! Let’s give it more than just one function here.
Adding more functions
We could give it any number of different functions that handle stuff like generating images or calculating something, the possibilities are endless. To keep this tutorial simple and stop you from having to sign up for yet another API key, we’ll add a second weather-related function here, but this time for the forecast. That way we can reuse the same API key we already have.
If you happen to have an API key to some other cool service like an AI image generator service or something feel free to add that as well as another function! Just follow the template we have set forth.
So go ahead and open back up our weather.py file where we wrote the first weather function. Now we’ll create a second function to get the weather forecast. This one is a bit more complicated as we need to provide a number of days to forecast. We’ll also need to do some error handling to make sure the user provides a valid number of days.
After the get_current_weather function, but before the if __name__ == "__main__": block, get started on our new function:
def get_weather_forecast(location, days=7) -> str:
try:
days = 1 if days < 1 else 14 if days > 14 else days
except TypeError:
days = 7
We take a location and set a default of 7 days. If the days parameter is less than 1 we set it to 1, but if it’s more than 14 we set it to 14. If neither condition is true the user provided a valid value and we just use the input argument value. Finally, if some weird type gets passed in we just default to 7 days.
Note that the free WeatherAPI account will limit you to 3 days of forecast after the initial pro trial expires, so if you try this again in a few days you may need to limit it to 3 days of forecast data. Let’s continue:
params = {
"key": os.environ["WEATHER_API_KEY"],
"q": location,
"days": days,
"aqi": "no",
"alerts": "no",
}
response = requests.get(
"http://api.weatherapi.com/v1/forecast.json", params=params
)
Parameters are largely the same except we have a number of days now. The only problem is that the API sends back a lot of data, even hourly data so 24 entries per day, which is way too much, so we need to do some filtering:
forecast_data: dict = response.json()
filtered_response = {}
filtered_response["location"] = forecast_data["location"]
filtered_response["current"] = forecast_data["current"]
filtered_response["forecast"] = [
[day["date"], day["day"]] for day in forecast_data["forecast"]["forecastday"]
]
return dumps(filtered_response)
First convert the response received from the Weather API to a dictionary by calling response.json(). Now we’ll create an empty dictionary filtered_response and add the location and current weather keys from the forecast_data dictionary.
The next line may look a bit cryptic, but it’s just a matter of looking at the response object you get in return for calling this API, and figuring out what you want to keep. The output object is absolutely massive in this case and there is no need to send quite so much info to our LLM, wasting valuable tokens. As this tutorial is not about the weather API I’ll just give you the line here but here is a rough explanation anyway:
So in the forecast_data there is a key called forecast which has a key named forecastday inside with the daily forecast. For each of these daily forecast objects which we give the alias name of day we want to keep that day’s date and day data, ignoring all else. We then add this data to the filtered_response dictionary’s forecast key.
Again, this is just dependent on whatever structure your return object happens to have, but it is just a list comprehension to get rid of tons of extra included hourly data to save tokens.
Now go to the if __name__ == "__main__": block and add a test to try out the new function:
if __name__ == "__main__":
# print(get_current_weather("Seoul"))
# print(get_current_weather("Amsterdam"))
print(get_weather_forecast("Seoul", days=3))
Now run the file and you should see something like this:
{"location": {"name": "Seoul", "region": "", "country": "South Korea", "lat": 37.57, "lon": 127.0, "tz_id": "Asia/Seoul", "localtime_epoch": 1719129971, "localtime": "2024-06-23 17:06"}, "current": {"last_updated_epoch": 1719129600, "last_updated": "2024-06-23 17:00", "temp_c": 28.3, "temp_f": 82.9, "is_day": 1, "condition": {"text": "Partly cloudy", "icon": "//cdn.weatherapi.com/weather/64x64/day/116.png", "code": 1003}, "wind_mph": 11.9, "wind_kph": 19.1, "wind_degree": 220, "wind_dir": "SW", "pressure_mb": 998.0, "pressure_in": 29.47, "precip_mm": 0.0, "precip_in": 0.0, "humidity": 70, "cloud": 50, "feelslike_c": 30.3, "feelslike_f": 86.6, "windchill_c": 27.5, "windchill_f": 81.6, "heatindex_c": 29.2, "heatindex_f": 84.5, "dewpoint_c": 19.4, "dewpoint_f": 66.9, "vis_km": 10.0, "vis_miles": 6.0, "uv": 7.0, "gust_mph": 14.7, "gust_kph": 23.6}, "forecast": [["2024-06-23", {"maxtemp_c": 30.1, "maxtemp_f": 86.2, "mintemp_c": 21.7, "mintemp_f": 71.0, "avgtemp_c": 24.8, "avgtemp_f": 76.7, "maxwind_mph": 13.9, "maxwind_kph": 22.3, "totalprecip_mm": 6.43, "totalprecip_in": 0.25, "totalsnow_cm": 0.0, "avgvis_km": 9.3, "avgvis_miles": 5.0, "avghumidity": 77, "daily_will_it_rain": 1, "daily_chance_of_rain": 88, "daily_will_it_snow": 0, "daily_chance_of_snow": 0, "condition": {"text": "Moderate rain", "icon": "//cdn.weatherapi.com/weather/64x64/day/302.png", "code": 1189}, "uv": 9.0}], ["2024-06-24", {"maxtemp_c": 27.7, "maxtemp_f": 81.8, "mintemp_c": 20.1, "mintemp_f": 68.2, "avgtemp_c": 23.5, "avgtemp_f": 74.3, "maxwind_mph": 14.5, "maxwind_kph": 23.4, "totalprecip_mm": 0.49, "totalprecip_in": 0.02, "totalsnow_cm": 0.0, "avgvis_km": 10.0, "avgvis_miles": 6.0, "avghumidity": 69, "daily_will_it_rain": 1, "daily_chance_of_rain": 89, "daily_will_it_snow": 0, "daily_chance_of_snow": 0, "condition": {"text": "Patchy rain nearby", "icon": "//cdn.weatherapi.com/weather/64x64/day/176.png", "code": 1063}, "uv": 9.0}], ["2024-06-25", {"maxtemp_c": 26.9, "maxtemp_f": 80.5, "mintemp_c": 18.6, "mintemp_f": 65.5, "avgtemp_c": 22.4, "avgtemp_f": 72.2, "maxwind_mph": 14.3, "maxwind_kph": 23.0, "totalprecip_mm": 0.0, "totalprecip_in": 0.0, "totalsnow_cm": 0.0, "avgvis_km": 10.0, "avgvis_miles": 6.0, "avghumidity": 55, "daily_will_it_rain": 0, "daily_chance_of_rain": 0, "daily_will_it_snow": 0, "daily_chance_of_snow": 0, "condition": {"text": "Sunny", "icon": "//cdn.weatherapi.com/weather/64x64/day/113.png", "code": 1000}, "uv": 9.0}]]}
We have the current weather and the forecast for the next 3 days, and the object is quite large but still of a reasonable size. Before we can give this to Gemini we need to add the type hints and docstring to the top of the function, completing it like this:
def get_weather_forecast(location: str, days: int = 7) -> str:
"""Get the weather forecast for a location using the WeatherAPI.
Args:
location (str): The location to get the weather forecast for.
days (int, optional): The number of days to get the forecast for. Defaults to 7.
Returns:
str: A JSON string containing the weather forecast data in detail.
"""
try:
days = 1 if days < 1 else 14 if days > 14 else days
except TypeError:
days = 7
params = {
"key": os.environ["WEATHER_API_KEY"],
"q": location,
"days": days,
"aqi": "no",
"alerts": "no",
}
response = requests.get(
"http://api.weatherapi.com/v1/forecast.json", params=params
)
forecast_data: dict = response.json()
filtered_response = {}
filtered_response["location"] = forecast_data["location"]
filtered_response["current"] = forecast_data["current"]
filtered_response["forecast"] = [
[day["date"], day["day"]] for day in forecast_data["forecast"]["forecastday"]
]
return dumps(filtered_response)
We added the str and int type hints and an extensive docstring, following the same structure we did for our other function. Now this is ready to be used.
Updating the chat program
It’s time to come back to our function_chat.py file and make some changes and updates.
Start by adding some imports:
## Add this import from google.generativeai.types import StopCandidateException from cost_calculator import print_cost_in_dollars from load_env import configure_genai from utils import safety_settings ## Add the import for our weather forecast function from weather import get_current_weather, get_weather_forecast
We naturally added the import for the function we just wrote, but we also added another import for StopCandidateException from the google.generativeai.types module. The problem is that we have turned off streaming mode to work with function calls here, but we’re still using the old if chunk.candidates[0].finish_reason == 3: check to see if our response is being blocked for any of the safety reasons. This will not work as we are no longer streaming so there are no chunks to check.
Instead, when an inappropriate response is blocked from being generated, the genai library will raise a StopCandidateException which we can catch and handle instead. So this will be used to replace our old check.
Leave the genai, MODEL_NAME, character, and movie variables as they are. We don’t have to change anything else until we get to the model declaration. We’re going add the second function to the tools list and update the system_instruction slightly to reflect the second function added:
model = genai.GenerativeModel(
model_name=MODEL_NAME,
tools=[get_current_weather, get_weather_forecast], # Add the new function here
safety_settings=safety_settings.low,
system_instruction=f"""
You are helpful and provide good information but you are {character} from {movie}. You will stay in character as {character} no matter what. Make sure you find some way to relate your responses to {character}'s personality or the movie {movie} at least once every response.
You have weather functions available, but using these is completely optional. Do not use or mention the weather functions unless the conversation is actually related to the weather. When you do use one or multiple weather tool(s) make sure to use several factors from the return data in your answer.
"""
)# Second paragraph is slightly updated
Now it has both of our functions and new system instructions, we can continue. Just leave this as is:
chat_session = model.start_chat(history=[], enable_automatic_function_calling=True)
And also skip over the upload_image() function, it is fine as it is right now.
Updating the main loop
Now let’s do an overhaul on the if __name__ == "__main__": block:
if __name__ == "__main__":
try:
while True:
text_query = input("\nPlease ask a question or type `/image` to upload an image first: ")
image_upload = None
if text_query.lower() == "/image":
image_upload = upload_image()
text_query = input("Please ask a question to go with your image upload: ")
full_query = [image_upload, text_query] if image_upload else text_query
## From here on everything is different
try:
response = chat_session.send_message(full_query)
print(f"\033[1;34m{response.text}\033[0m")
print_cost_in_dollars(response.usage_metadata, MODEL_NAME)
print(chat_session.history)
except StopCandidateException:
print(f"\n\033[1;31mPlease ask a more appropriate question!\033[0m", end="")
# chat_session.rewind() is not needed here
except KeyboardInterrupt:
print("Shutting down...")
First we start off as we have been, starting an infinite loop and asking for the text query and potential image upload if desired, combining these into the full_query. Nothing new here.
We then open up another try block, getting the response by using chat_session.send_message(full_query). We will print the return by printing response.text, the cost using our cost function, and the chat history.
The chat history will also include the function calls so it will give us a good look below the hood and check if our function calls are working. It will also make for a very long output in your terminal though, so comment it out later when you are done testing.
As discussed earlier, if the response is blocked/inappropriate, genai will raise a StopCandidateException which we catch and print a message to the user. You do now use the rewind() function on the chat_session object as we are not streaming anymore.
As the try/except block is inside the infinite loop, the program will keep running even if an inappropriate response is blocked. Giving the user the opportunity to ask a new question just like we had before, keeping the functionality the same.
Testing the program
So go ahead and try running the program now and let’s test out all possible scenarios here.
What is your favorite movie character? (e.g. Gollum): Darth Vader What movie are they from? (e.g. Lord of the Rings): Star