Welcome back to part 3 of this course, where we’ll be looking at the new ChatGPT 4, named GPT-4 Turbo!
GPT-4 Turbo is even more capable and now has a knowledge cutoff of April 2023, making it significantly less behind on its knowledge of current events.
GPT-4 Turbo is also 3X cheaper for input tokens and 2X cheaper for output tokens compared to the original GPT-4 model. The maximum number of output tokens for this model is 4096.
It also introduces a 128k maximum input context, which is absolutely insane, and there is also a ‘vision’ version that can analyze our images. More details on all of this as we go through this part, which is going to be a lot of fun!
First, create a new folder in your base directory named 3_GPT4_turbo. Inside this folder, create a new file named gpt4_turbo.py:
📁FINX_OPENAI_UPDATES (root project folder)
📁1_Parallel_function_calling
📁2_JSON_mode_and_seeds
📁3_GPT4_turbo
📄gpt4_turbo.py
📄.env
Inside this file let’s start with our imports:
from decouple import config
from openai import OpenAI
from pathlib import Path
client = OpenAI(api_key=config("OPENAI_API_KEY"))
And then let’s give GPT-4 Turbo a quick test run (You can skip this one and just look along with my results, we’ll be removing this basic code again in a second):
messages = [
{
"role": "system",
"content": "You make up fun children's short stories according to the user request.",
},
{"role": "user", "content": "Tell me a story about a unicorn in space."},
]
response = client.chat.completions.create(
model="gpt-4-1106-preview",
messages=messages,
stop=["\n"],
)
print(response.choices[0].message.content, "\n")
Now if I run this same function with GPT 3.5-Turbo and GPT 4-Turbo we can really feel the difference in quality:
3.5: Once upon a time, in a magical land far, far away, there lived a majestic unicorn named Luna. Luna was not an ordinary unicorn - she had the ability to travel through space and visit other worlds. One day, as she was wandering through the enchanted forest, she stumbled upon a mysterious portal that appeared out of nowhere. 4: Once upon a time, in a shimmering galaxy far, far away, there lived a unique and magical creature known as Stellar, the space-faring unicorn. Stellar had a coat that sparkled with the radiance of a thousand stars and a mane that flowed behind him like the trails of comets across the night sky. His horn, forged from cosmic crystal, glowed with an ethereal light that reflected the colors of distant nebulae.
We can see that 4 is much more descriptive and at a higher level of writing. It really paints a vivid picture like a writer would.
Pricing
So we said that GPT-4 Turbo was cheaper than GPT-4, so I guess that makes GPT-4 Turbo the new GPT 4, as it’s both more capable and cheaper.
Input tokens are 3x cheaper than GPT-4, and output tokens are 2x cheaper, and the new GPT-3.5 Turbo is also cheaper than the older 3.5 Turbo version, but what are the costs exactly?
The new GPT-4 Turbo model is priced at Input: $0.01 and Output: $0.03 per 1000 tokens, which while indeed cheaper than the old GPT-4, is still 10 times more expensive than the new version of GPT-3.5 Turbo, which is priced at Input: $0.001 and Output: $0.002 per 1000 tokens.
This is why it’s good to use the new 3.5 Turbo model whenever you can as it’s often simply good enough and much cheaper.
Both the new 3.5 Turbo and 4 Turbo models have the parallel function calling feature, JSON mode, seeds, and improvements so the 3.5 Turbo model certainly isn’t shabby either.
One of the points that stands out about GPT-4 Turbo is that we no longer have to worry about the context limit, or how big a prompt we can send to ChatGPT at once. the new GPT-4 Turbo has a whopping context limit of 128k tokens, which is about 300 pages of text in a single prompt. Although there are some caveats to this as we’ll soon find out.
The new GPT-3.5 Turbo we have been using so far also has an increased context limit of 16k tokens by the way, which in itself is very high already.
Testing Out the Context Limit
So let’s test out the context limit.
In order to do this we first need a very large text, so let’s go to our trusty copyright-free book website Gutenberg, and download a book.
I’ll be using “The Importance of Being Earnest: A Trivial Comedy for Serious People”, which is actually a historical play that is now in the public domain. You can download a full text version at the link below.
Inside the 3_GPT4_turbo folder, create a new file called book.txt and just copy and paste the entire text from the website inside this text file:
📁FINX_OPENAI_UPDATES (root project folder)
📁1_Parallel_function_calling
📁2_JSON_mode_and_seeds
📁3_GPT4_turbo
📄book.txt (has the whole book text inside)
📄gpt4_turbo.py
📄.env
Now go back to your 'gpt4_turbo.py' file and remove everything except the imports and the client variable, as we don’t want to keep generating stories about unicorns, I think we’ve had enough of those for now!
This is all that will be left in your 'gpt4_turbo.py' file:
from decouple import config
from openai import OpenAI
from pathlib import Path
client = OpenAI(api_key=config("OPENAI_API_KEY")) # type:
Now we’re going to use pathlib to define a variable that holds the path to our book:
path_to_book = Path(__file__).parent / "book.txt"
This line uses the Path class from the pathlib module, __file__ is a built-in variable in Python that contains the path of the Python file that is currently being executed.
The .parent attribute of a Path object gives the directory containing the file, in this case, the '3_GPT4_turbo' folder. As this is also the folder that contains our book.txt file, we can use the / operator to append the filename to the path and voila we have a path to our book.txt file!
Book Summarizer
So let’s write a book summarizer. As it’s fairly simple, we’ll just go over it in one go:
def book_summary_GPT(file_path):
with open(file_path, "r", encoding="utf8") as file:
book = file.read()
messages = [
{
"role": "system",
"content": "You are a book-summarizing AI. You will receive a book as the query and you will return a summary that is not too long and summarizes the important main points and happenings of the book. Make sure to use only the text provided for your summary, and not any other knowledge you may have.",
},
{
"role": "user",
"content": book,
},
]
response = client.chat.completions.create(
model="gpt-4-1106-preview",
messages=messages,
)
content = response.choices[0].message.content
# write the output to summary.txt
with open("summary.txt", "w") as file:
file.write(content)
print(content)
return content
We take a file_path as input and then open it in read mode with the encoding utf8. We then read the text into a variable named book. We set up the messages with a system instruction for summarizing a book.
We make a call to "gpt-4-1106-preview" which is GPT-4 Turbo, and we pass in our messages.
We then get the content and write it to a file named summary.txt where 'w' means write mode. This file will be created if it doesn’t exist. We then print and return the content.
The issue – rate limits
Ok sounds simple enough, so let’s run it.
(Hint, we’re going to run into some trouble!)
Alternatively, if you’re concerned about the costs of sending this many tokens you can look along on my screen for this part, but it won’t be super expensive unless you make hundreds of calls:
book_summary_GPT(path_to_book)
Now if we run this! BAM error!
openai.RateLimitError: Error code: 429 - {'error': {'message': 'Request too large for gpt-4-1106-preview in organization org-oMYMXpp7Cr9pG1rG5Z8a1T2w on tokens per min (TPM): Limit 10000, Requested 34407. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}
So what’s going on here? I thought the model had a 128k limit and we only tried to send 34k tokens.
Well, the model may have a context limit of 128,000 tokens, but we’re only allowed to use 10,000 tokens per minute, which essentially makes it completely useless!
Why is that?
It turns out there is a tier list based on how much you have used the API so far:
Tier 1 $5 used (+ certain number of days elapsed since) 10,000 tokens per minute Tier 2 $50 used (+ certain number of days elapsed since) 40,000 tokens per minute Tier 3 $100 used (+ certain number of days elapsed since) 80,000 tokens per minute Tier 4 $250 used (+ certain number of days elapsed since) 300,000 tokens per minute Tier 5 $1,000 used (+ certain number of days elapsed since) 300,000 tokens per minute
So yeah, that’s something to be aware of, as of now, start of November 2023 you will not be able to utilize the full potential of this model unless you’ve been using the API professionally on your website.
While using it extensively for local development and testing projects only, I never exceed 1-2 dollars per month, so I’ve yet to even become a tier 2 user!
These limits will likely be loosened in the future though, so by the time you watch this tutorial, this may no longer be an issue. Note that the whole book we wanted to pass in was still only 34,407 tokens in total, meaning without the tokens per minute rate limit we could have passed in this whole book three times over in a single prompt!
A Shorter Test
So let’s run it with just the first 30,000 characters passed in then shall we? It’s still a load of text and we should be able to see the potential of handling this much text at once successfully.
In your book_summary_GPT function, find the following:
{
"role": "user",
"content": book,
},
and simply change it to:
{
"role": "user",
"content": book[:30_000],
},
Now if you run the print statement again, you’ll get a second surprise and a wise lesson:
book_summary_GPT(path_to_book)
Note that interestingly, the entire play is summarized even though we only provided a portion. I suspect the play and other public domain works are part of the training data that ChatGPT was trained on, or at the very least some summaries of the play were.
I tried to find a public works book that is not already present in ChatGPT’s knowledge data-bank, but it’s really hard to do so, which kind of proves a point.
If we need a summary of an existing work we can just not pass in all this data and ask for a summary of “The Importance of Being Earnest” by Oscar Wilde to achieve the same thing!
ChatGPT has so much knowledge that often you really won’t need to pass in all that data to ask it to do something on a certain topic.
It’s still nice to know you’ll be able to do so after the limits are loosened, and this will mostly be useful for proprietary business data that is unique to your business, because if it’s part of the open internet, at this rate, chances are ChatGPT already knows it.
Also keep in mind that passing such massive prompts should always be avoided when possible considering the costs, which go up linearly with the insane amount of tokens you can theoretically pass to GPT-4 Turbo.
Where a normal prompt often is a couple of lines or paragraphs worth of characters a whole book equates to many, many of those normal calls at once.
GPT-4 Turbo Vision
Now let’s have a look at something very exciting. GPT-4 Turbo Vision! Go ahead and make a new file in the '3_GPT4_turbo' directory named 'gpt4_turbo_vision.py' and let’s get started.
📁FINX_OPENAI_UPDATES (root project folder)
📁1_Parallel_function_calling
📁2_JSON_mode_and_seeds
📁3_GPT4_turbo
📄book.txt (has the whole book text inside)
📄gpt4_turbo.py
📄gpt4_turbo_vision.py
📄.env
First get started with your imports:
from decouple import config
from openai import OpenAI
client = OpenAI(api_key=config("OPENAI_API_KEY"))
And let’s just do a test run with the most simple possible way to use GPT-4 Turbo vision, by linking to an image that is already on the internet. I’ll be using the following random image, as imgur will allow us to just hotlink to their files directly:
Now create a function:
response = client.chat.completions.create(
model="gpt-4-vision-preview",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "What’s in this image?"},
{
"type": "image_url",
"image_url": "https://i.imgur.com/M6bTG.jpeg",
},
],
}
],
max_tokens=300,
)
print(response.choices[0])
Make sure you pass in the specific "gpt-4-vision-preview" model and notice how easy this syntax all is. We pass in a user message but in this case, the content is a list. We have the first entry which is of type text and the second entry which is of type image_url, allowing us to send both of these as our query.
Just a quick note on the max_tokens parameter, usually you don’t really have to worry about setting it unless you want a limit, but for this model, there seems to be a pretty low default, so go ahead and set this to like 300 to make sure your response message does not get cut off halfway through.
So let’s run the file to test it out.
The image URL we chose deliberately does not give any hints as to what is in the image, so any analysis will be purely from GPT vision. My output was:
This image shows a view from inside a refrigerator looking outwards. There's a person who appears to be a man standing and looking into the fridge, holding a cat. The fridge contains various items such as bottles, what appears to be mustard, and other condiments. There's a dish rack with yellow utensils above on one of the fridge shelves, indicating that there might be more storage space above the fridge. The kitchen seems to have a window with blinds, allowing natural light into the room, and there's a plant visible through the window.
Now that is seriously impressive, and also really fun. Before we look at a more practical use case for this, let’s discuss some of the details and specs of this model.
Costs and Limitations
If you do this on literally hundreds of images especially of high quality, your usage costs will go up faster compared to text-only models, as the images have to be analyzed and processed by the API.
For example, a 1024 x 1024 square image in detail: high mode costs 765 tokens. (more on detail settings in a moment) which amounts to $0.00765 at current pricing.
That’s not a huge deal if you process a couple of images but if you do thousands it racks up quickly.
The model also still has some limitations, which I’m sure will largely disappear with future iterations. For now, the model is not as good with non-English languages, especially if they have an alphabet beyond Latin, rotated images and graphs, the exact location in space of objects, and things like counting.
Building DinnerGPT
So let’s look at a more realistic implementation where the image is hosted locally on our server.
Say the user uploads an image to our server through our website or mobile application. Our application is called: “What’s for dinner!” and the service we provide is for the user to upload a picture of their fridge and we will tell them what they can cook with the ingredients they have in their fridge.
We want to make a call to GPT-4 Turbo and upload our image to ask it for the answer.
First of all, let’s get two images of fridge contents to test with. If you want to really make this fun, take one picture each of your open fridge and your open freezer and use those!
I’ll be using two images from the internet as I’m about to head out on a short trip to Japan soon and the current contents of our fridge and freezer are therefore rather shabby at the moment.
First go to https://unsplash.com/photos/green-and-pink-plastic-container-AEU9UZstCfs and download this image using the ‘Download free’ button in the top right. I will be using the ‘medium’ size from the choice menu.
These images are free for us to use and you don’t even have to log in or sign up. Thanks “Ello” from Unsplash!
Save this image as 'fridge.jpg' in the '3_GPT4_turbo' folder.
Then for the second image, I will use https://unsplash.com/photos/a-refrigerator-filled-with-lots-of-food-and-drinks-ZUhM8LE_HGc with a thank you to Unsplash user “Staton” for this one. Save it as fridge2.jgp in the same folder.
Finally, make a new file called dinner.py for our code, your folders should now look like this:
📁FINX_OPENAI_UPDATES (root project folder)
📁1_Parallel_function_calling
📁2_JSON_mode_and_seed