Hi and welcome back! In this part, we’re going to look at some alternatives to speed stuff up or outsource the processing power to OpenAI’s servers altogether. First, we’ll look at faster-whisper at a basic level. If you’re not sure whether you want to use this you can also just watch this part and decide whether or not you want to install it for further use later as we’re just going to cover it quickly before moving on to the web API version for the rest of this part.
So what is faster-whisper? Faster-Whisper is a quicker version of OpenAI’s Whisper speech-to-text model. As OpenAI released the whisper model as open-source this has naturally allowed others to try to build on and optimize it further. It uses CTranslate2, a fast engine for Transformer models, and is up to 4 times faster and uses considerably less memory than the original openai/whisper while claiming to maintain the same accuracy. You can find the GitHub repository here.
You can use this for the same apps we have built so far, just as a faster version of the Whisper model, so we won’t be building a new app specifically for this, as it would get repetitive and I don’t want to waste your time! You just need some syntax changes to make your app work with faster-whisper instead of the original whisper model. So we’ll take a look at the basics of fast-whisper, let you decide if you want to use/implement it, and then move on to the web-API version.
Installing faster-whisper
Note: If you do not plan on using faster-whisper or are not quite sure, there is no point in going through the install procedures, and you can skip ahead a couple of minutes to the web-API version, or just watch/read along and decide later if you want to use it.
Basically, to install faster-whisper you just have to run the following command in your terminal:
pip install faster-whisper
And to support GPU execution you need to have the appropriate libraries for CUDA installed, which are cuBLAS and cuDNN. This can be the slightly trickier part of the install, and again I cannot really give you platform-specific instructions or help you with the specific troubleshooting if you run into challenges. As always in software development, if you’re lucky you won’t have any problems, and if you’re not, you spend some time on Google and Stackoverflow to find the solution. If you just want to run faster-whisper on your CPU, which will of course be slower but may not be a big deal for small-scale development on your own machine, you can skip the cuBLAS and cuDNN installs.
Using faster-whisper
So let’s give it a spin to see how it works! First create a new file in your project root directory called 4_faster_whisper.py:
📁FINX_WHISPER (project root folder)
📁output_temp_files
📁output_video
📁styles
📁test_audio_files
📁utils
📄1_basic_call_english_only.py
📄1_multiple_languages.py
📄2_whisper_pods.py
📄3_subtitle_master.py
📄4_faster_whisper.py (✨new file)
📄settings.py
📄.env
And inside let’s start with our imports:
from faster_whisper import WhisperModel from settings import TEST_AUDIO_DIR model_size = "small"
We import the WhisperModel class from the faster_whisper package, and the TEST_AUDIO_DIR variable from our settings.py file, and then set a string variable to the value small. Like whisper, faster-whisper also comes with different sizes of models. Using the same naming convention we have tiny.en, base.en, small.en, and medium.en as our English-only models. For the multi-language models, we can choose between tiny, base, small, medium, or one of several versions of the full-size model, namely: large-v1, large-v2, large-v3, or large.
Next, we’ll create a new instance of the WhisperModel class, picking only one of the two options below:
model = WhisperModel(model_size, device="cpu", compute_type="int8") # Choose only one of these, depending on if you're running on CPU or GPU (cuda). (I'll be using the second option) model = WhisperModel(model_size, device="cuda", compute_type="float16")
More options are available, like running on cuda using int8_float16 or even using float32, see here for more details.
The .transcribe method for faster-whisper is slightly different:
segments, info = model.transcribe(
str(TEST_AUDIO_DIR / "dutch_long_repeat_file.mp3"),
beam_size=5,
)
As you can see we get two returns when calling model.transcribe instead of the single dictionary output we had before. The first is a list of segments which contains the transcription. The second is a NamedTuple (a Tuple with named fields) which allows us to access information like the language (info.language), language probability (info.language_probability), etc. So let’s add some print statements to print the information and then the transcription itself to the console:
print(f"Detected language '{info.language}' with probability {info.language_probability}")
for segment in segments:
print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")
The first print statement just has us access some of the properties of the info object we discussed. The second print statement loops over the list of segments, and for each segment it will print the segment’s start time, end time, and the text of the segment itself. The :.2f is a formatting string that tells Python to print the number with two decimal places, for example: 1.23 instead of 1.23456789.
One interesting thing to note here though is that segments is not actually a list. Segments is a generator, which is a different type of iterable. What this means is that the segments will be generated when you request them and not beforehand. In other words, the transcription only begins when we iterate over the segments and not before. Calling .transcribe() on our model did not start the transcription as vanilla whisper did. You can either loop over the segments as we did above, or you can convert the generator to a list by converting it to a list list(segments).
One of the nice things about this generator is that we can very easily see the live transcription and print it to the console while it is still generating, which is exactly what this code will do. So let’s run it and see what happens:
Estimating duration from bitrate, this may be inaccurate Detected language 'nl' with probability 0.931703 [0.00s -> 3.04s] Hoi allemaal, dit is weer een testbestandje. [3.04s -> 6.88s] Deze keer om te testen of de Nederlandse taal goed herkent gaat worden. [6.88s -> 12.68s] Hierna kunnen we ook proberen deze tekst te laten vertalen naar het Engels om te zien hoe goed dat gaat. [12.68s -> 13.88s] Ik ben benieuwd. [13.88s -> 16.84s] Hoi allemaal, dit is weer een testbestandje. [16.84s -> 20.72s] Deze keer om te testen of de Nederlandse taal goed herkent gaat worden. [20.72s -> 26.48s] Hierna kunnen we ook proberen deze tekst te laten vertalen naar het Engels om te zien hoe goed dat gaat. [26.48s -> 27.68s] Ik ben benieuwd. [27.68s -> 30.72s] Hoi allemaal, dit is weer een testbestandje. [30.72s -> 34.60s] Deze keer om te testen of de Nederlandse taal goed herkent gaat worden. [34.60s -> 40.36s] Hierna kunnen we ook proberen deze tekst te laten vertalen naar het Engels om te zien hoe goed dat gaat. [40.36s -> 41.52s] Ik ben benieuwd.
You can see the output streaming to the console as the model transcribes. Unless you run over CPU you will also notice a pretty good speed. Now as you’re probably not Dutch I’ll just tell you the transcription above is perfect except for the one small (herkent/herkend) issue we had before, but as you know this can be fixed by loading a larger model size.
Play around with any audio file you want and see what model size you need. If you use English files pick a .en model for greater efficiency. Also be aware that you can pass in options into the .transcribe method much like the vanilla whisper model, for instance:
segments, info = model.transcribe(
str(TEST_AUDIO_DIR / "dutch_long_repeat_file.mp3"),
beam_size=5,
word_timestamps=True, # uncomment this line to get word timestamps
# without_timestamps=True, # uncomment this line to get rid of timestamps and just transcribe
)
In conclusion, faster-whisper is a nice optimization to look into if you’re considering deploying this model in a production application somewhere. There are also other optimized versions of the whisper model out there that you can check out, like distil-whisper. Play around and see which gives you the best trade-offs between speed and accuracy. I’ll leave the rest up to you as we move on from faster-whisper to check out the web-API version.
Web-API version
Another option we have is to simply not deploy the model anywhere but outsource this to OpenAI’s fast servers. This is kind of like making a ChatGPT call except we request a transcription instead of a chat completion. The OpenAI servers are also very optimized for machine-learning calculations (obviously) and as you’ll see they are therefore quite fast!
So let’s take a look at the pricing first. The cost for using the Whisper API is $0.006 per minute transcribed, rounded to the nearest second. This means a 20-minute video would cost you $0.12. This is a good solution if you don’t want to deploy the model yourself, perhaps your application will only be used occasionally and it’s simply not worth it to invest that much into having a model running somewhere. For a high-use application dealing with longer files and many users, this is not the way to go though.
So let’s take a quick look at how this would work practically, by building one last quick application, but this time using the web API. Our application will take any video in any language as input and will return a short quiz with questions about the video. First, create a new file in your utils folder named openai_api.py:
📁FINX_WHISPER (project root folder)
📁output_temp_files
📁output_video
📁styles
📁test_audio_files
📁utils
📄command.py
📄openai_api.py (✨new file)
📄podcast.py
📄subtitles.py
📄video.py
📄1_basic_call_english_only.py
📄1_multiple_languages.py
📄2_whisper_pods.py
📄3_subtitle_master.py
📄4_faster_whisper.py
📄settings.py
📄.env
Inside openai_api.py, let’s start with our imports and some basic setup:
import typing
from pathlib import Path
from decouple import config
from openai import OpenAI
CLIENT = OpenAI(api_key=str(config("OPENAI_API_KEY")))
MODEL = "whisper-1"
ResponseFormat = typing.Literal["text", "srt", "vtt"]
We’ll use typing to define our allowed response formats. The rest is all imports we have used before, config as we’ll need to load our API key and OpenAI to call the APIs for Whisper and ChatGPT. We create our CLIENT just like last time and we save the MODEL in a string variable, whisper-1 is the only option for the Whisper API for now.
Finally, we define a type alias named ResponseFormat which is a Literal type, which means it can only be one of the three strings we have defined, text, srt, or vtt. We can use this as a type hint later to indicate that if a particular variable is of type ResponseFormat then it should have one of these three values and nothing else. (json and verbose_json are also possible if you prefer JSON object output, but we will be skipping them as they are useless for our purposes.)
Now we’ll define our transcription utility function:
def transcribe(
file: Path,
language: str | None = None,
translate: bool = False,
response_format: ResponseFormat = "text",
) -> str:
print("Transcribing file...")
options = {
"file": file,
"model": MODEL,
"response_format": response_format,
}
if translate:
transcript = CLIENT.audio.translations.create(**options)
else:
if language:
options["language"] = language
transcript = CLIENT.audio.transcriptions.create(**options)
if type(transcript) != str:
raise TypeError(
f"Expected a string value to be returned, but got {type(transcript)} instead."
)
print(f"Transcription successful:\n{transcript[:100]}...")
return transcript
We define a function called transcribe which takes a file of type Path, a language of type str or None, which defaults to None, in which case the API will try to detect the language automatically. We also have a translate boolean which defaults to False, and a response_format which has to be of type ResponseFormat, so one of the three values we defined in the type alias, and defaults to text. The function returns a string.
We print a message to indicate the transcription is starting and then create a dictionary named options in which we pass in some options that are needed for both a translation and a transcript call, so the shared options if you will. These are the file, model, and response_format. If the user requests a translation we call the CLIENT.audio.translations.create method, passing in the **options dictionary as arguments as is. If translation = False it must be a transcription. For transcriptions, we can add the language key to the options dictionary to specify the language, but if the user didn’t provide it we can leave it out and it will just take a bit longer to do the auto-detection. This time we call the CLIENT.audio.transcriptions.create method, again passing in the **options dictionary which optionally now contains the language key.
Finally, we check if the transcript is a string, and if not we raise a TypeError to indicate something went wrong, just to make sure the user is not requesting JSON from this endpoint, which is possible and would crash the rest of our code. Otherwise, we print a message to indicate the transcription was successful and return the transcript.
Video to Quiz
As we’re going to be building a video-to-quiz app, we need one more utility function inside this openai_api.py file, which will take a transcript and generate some questions for us. Continue below the transcribe function: