Welcome back to part 5, where we’ll take a look at OpenAI’s new text-to-speech API.

Text-to-speech is nothing new, the robot voices have been famous for a long, long time. What is new is the quality is so good it’s almost scary now! So let’s jump right in.

Create a new folder called '5_Text_to_speech' and inside, create a new file named 'text_to_speech.py' like this:

📁FINX_OPENAI_UPDATES (root project folder)
    📁1_Parallel_function_calling
    📁2_JSON_mode_and_seeds
    📁3_GPT4_turbo
    📁4_DALLE
    📁5_Text_to_speech
        📄text_to_speech.py

Inside, we’ll get started with our imports:

from decouple import config
from typing import Literal, get_args
from pathlib import Path
from openai import OpenAI

All of these should be thoroughly familiar by now except for the typing imports which we’ll cover as we use them.

Note that the typing imports are not required to use the API specifically, I just happened to use them for this example code to clarify what range of options are available for certain arguments, you may leave them out if you dislike type hints in Python, or try them out if you’re not yet familiar with them.

We continue our code as follows:

client = OpenAI(api_key=config("OPENAI_API_KEY"))
current_directory = Path(__file__).parent
Model = Literal["tts-1", "tts-1-hd"]
Voice = Literal["alloy", "echo", "fable", "onyx", "nova", "shimmer"]

We set up our client and current_directory path as is well familiar by now.

We also set up two type aliases, Model and Voice, which are both Literal types.

Literal types are a special type hint that allows us to specify a range of options for a given variable.

In this case, we’re specifying that the Model type can only be one of two options, "tts-1" or "tts-1-hd". The Voice type can only be one of six options, "alloy", "echo", "fable", "onyx", "nova", or "shimmer".

If you’re not that familiar with types, note that 'Model' is a type alias, it’s not a variable on its own. All it says is that all variables that are of the type "Model" should have a value of either "tts-1" or "tts-1-hd".

model_name: Model = "tts-1"         #valid
model_name: Model = "robo-voice"    #invalid as it's not an option on the Model type alias

This is a great way to make sure we don’t accidentally pass in an invalid argument, and also to make it clear to anyone reading our code what the valid options are.

Note that you won’t actually get an error when passing in an invalid type unless you have a type checker enabled in your IDE, but even without using a type checker the type hints will still help clarify your code.

Calling the TTS API

Ok so now let’s have a quick reusable function that both uses and clarifies how to use the OpenAI text-to-speech API:

def text_to_speech(
    input_text: str,
    model: Model,
    voice: Voice,
    file_name: str,
):
    response = client.audio.speech.create(
        model=model,
        voice=voice,
        input=input_text,
    )
    response.stream_to_file(f"{current_directory}/{file_name}.mp3")

We take an input text as a string, a model which has to be of type Model, so one of the available options in the type alias, a voice which has to be of type Voice, and a file_name which is just a string.

We then call the OpenAI API client’s audio.speech.create endpoint with the given arguments.

Note the handly stream_to_file method provided on the response object which allows us to save the response to a file by providing a path as argument. We combine the current_directory defined on top of our file and the file_name argument to create the file path.

Testing the Different Voices

So let’s have a listen to the various voices:

for voice in get_args(Voice):
    text_to_speech(
        input_text="My name is Spongebob and I love pineapples!",
        model="tts-1",
        voice=voice,
        file_name=voice,
    )

The get_args function we imported from typing gets all the possible options from the Voice type alias, which is the six voices we defined earlier.

We then loop over them and call our text_to_speech function with the given arguments. If you run this an mp3 file will be created for each voice and saved in the current directory. (alloy.mp3, echo.mp3, etc…)

So go ahead and run this and then have a listen to the 6 different voice files.

The voices really are different and have different inflections and tones to them. Of course, nothing beats a real voice actor but they sound very natural and human-like for computer voices.

tts-1 vs tts-1-hd

Now let’s have a listen to the different models to see if the HD option makes a big difference.

We’ll generate one file for each model using the echo voice:

for model in get_args(Model):
    text_to_speech(
        input_text="My name is Spongebob and I love pineapples!",
        model=model,
        voice="echo",
        file_name=model,
    )

Run this and another two mp3 files will be created, "tts-1.mp3" and "tts-1-hd.mp3".

I don’t hear a big change in quality, though this is a very small sample. But interestingly enough there is a change in the voice inflections and emphasis on the word Love in the HD version, so that’s interesting. It doesn’t seem to be limited to a mere sound quality change.

Building a Talking ChatGPT

That’s enough introduction, let’s build something cool with it!

Create a new file called 'talking_gpt.py' and let’s create a ChatGPT version that will speak the answer to us:

📁FINX_OPENAI_UPDATES (root project folder)
    📁1_Parallel_function_calling
    📁2_JSON_mode_and_seeds
    📁3_GPT4_turbo
    📁4_DALLE
    📁5_Text_to_speech
        📄text_to_speech.py
        📄talking_gpt.py

The example we’ll be looking at here is to create a talkingGPT that will take our question and then speak the answer to us. We will explore downloading and then playing the response versus streaming it directly to our speakers before the entire response is generated by streaming.

A small disclaimer before we start this part, handling audio files can be a bit tricky when doing this for the first time and you may find you run into specific errors with missing dependencies or codecs.

I’ve tried to use as simple audio libraries and a solution as possible to keep your issues to a minimum, like avoiding the somewhat buggy pydub library.

But if you do run into issues, I suggest you google the error message and find a solution that way. Having to troubleshoot on your own and fix potential bugs is an almost daily part of software development and I unfortunately cannot help you solve system-specific issues.

That being said you hopefully shouldn’t have any with the libraries chosen and used here.

First, open a shell to install a couple of dependencies we need. Run the following command in your terminal:

pip install PyAudio soundfile playsound

Now open your 'talking_gpt.py' file and let’s get started with our imports, Python built-in imports first:

import io
from distutils.util import strtobool
from pathlib import Path
from typing import Literal
from time import sleep, time

The io module contains Python’s core tools for working with streams, the time module will be used to time our responses. The strtobool function is a simple function that converts a string like 'yes' or 'no' to a boolean value, and the remaining two are already familiar from previous parts.

Now our other library imports:

import pyaudio
import requests
import soundfile as sf
from decouple import config
from openai import OpenAI
from playsound import playsound
import threading

We will use pyaudio to work with audio in our streaming function, requests to make a manual HTTP request to the OpenAI API, and soundfile to play audio from our buffer.

Playsound will be used as the simplest library to play an mp3 file already fully saved to disk for our download-first non-streaming example.

We also import threading as we will need it for the streaming example, gathering the data on a separate thread while we play the audio on the main thread simultaneously.

Now start with some setup:

API_KEY = config("OPENAI_API_KEY")
client = OpenAI(api_key=API_KEY)
current_directory = Path(__file__).parent
buffer_file_location = current_directory / "buffer.opus"
Model = Literal["tts-1", "tts-1-hd"]
Voice = Literal["alloy", "echo", "fable", "onyx", "nova", "shimmer"]

We will use our API_KEY several times so load it in a variable.

We set up our client and current_directory path as is well familiar by now and the two type aliases from the previous part make a reappearance to define the possible options.

The buffer_file_location is a Path object that points to a file called "buffer.opus" in our current directory. We will use this to store our streamed audio while playing it. (More on this later)

Download and Then Play

Now let’s code up the first example, which simply downloads the whole text-to-speech audio response and then plays it after it has been fully downloaded:

def download_and_play(input: str, model: Model = "tts-1", voice: Voice = "onyx"):
    start_time = time()
    response = client.audio.speech.create(
        model=model,
        voice=voice,
        input=input,
    )
    # Note that at this point the whole audio file is downloaded and stored in memory in it's entirety. The 'stream_to_file' method is a bit of a misnomer as it doesn't actually stream the audio here but just saves it to a file.
    response.stream_to_file(f"{current_directory}/audio.mp3")
    time_until_playback = time() - start_time
    print(f"Time until playback: {time_until_playback} seconds")
    playsound(f"{current_directory}/audio.mp3")

We take an input string, a model, and a voice as arguments, setting defaults for the model and voice.

We then call time() to get a reference of the exact current time to measure the time it will take for our audio to start playing from the initial request.

We get a response by calling client.audio.speech.create and passing in the given arguments.

We then call the stream_to_file method on the response object to save the response to a file. Note that the entire response has already been received at this point, so this is not really streaming, as you will see when we compare times later on.

Finally, we calculate the time it took for the audio to start playing by getting the current time and subtracting the starting time, then printing it out.

We then call playsound to play the file we just saved to disk.

Try it out by adding a print statement and running the file:

download_and_play("This is a test.")

You should hear the onyx voice say "This is a test." and see the time it took to play the audio in seconds:

Time until playback: 1.0087647438049316 seconds

Now change the above download_and_play call like this:

download_and_play(
    "This is a test. When the test gets much longer the difference will be much more obvious as to the speed in generation times taken for the test."
)

Now run the file again (making sure you removed the earlier call to the method) and you will see the issue:

Time until playback: 2.1658453941345215 seconds

The time it takes to generate the audio goes up linearly with the length, and for a long answer, this will easily go up to 15 seconds or longer. This is a big problem if we want anyone to actually have the patience to use our amazing talking GPT.

Streaming the Audio During Generation

Make sure you comment out the "download_and_play("…")" line before continuing as we don’t want to keep running this test.

For the second example, we will be streaming the response and playing it as it is still being generated. This will greatly speed up the loading time and cut down on the waiting between the question asked and the audio playing.

Note that the OpenAI library doesn’t yet handle audio streaming properly, so we cannot use the OpenAI library or the client.audio.speech.create method that we would normally call.

We’ll send a manual request to the API endpoint for streaming purposes, manually handling the request-response cycle.

I’ve not yet seen a proper implementation of this on the internet, nor are there any examples in the documentation and some of the solutions I have seen have the user download the entire file before playing it, yet they are mistaken into thinking they are streaming it, still taking a long time.

I will show you how to actually really stream the audio before downloading the full file, though it will get a little bit hacky.

Let’s go over this function in parts as it is a bit longer, and then after the full explanation we’ll give the entire function in a single block again so that you can properly copy it as the explanation blocks will be a bit long in our first look:

def stream_audio(input: str, model: Model = "tts-1", voice: Voice = "onyx"):
    start_time = time()
    py_audio = pyaudio.PyAudio()

Again, we take an input string and a model and voice from the possibilities in our Model and Voice type aliases, providing a default value for both. We then call time() to get a reference of the current exact time so we can calculate the time taken later on.

We then create an instance of the PyAudio class from the pyaudio library and catch it under the variable name 'py_audio'. With pyaudio, you can easily use Python to play and record audio on a variety of platforms.

The py_audio variable now holds a PyAudio object, which can be used to control audio streams.

For example, you could open a new stream and start playing audio, or even record audio from a microphone. We’ll see how the playing works in a moment.

Continue the function:

    url = "https://api.openai.com/v1/audio/speech"
    headers = {
        "Authorization": f"Bearer {API_KEY}",
    }

    response = requests.post(
        url,
        headers=headers,
        json={
            "model": model,
            "input": input,
            "voice": voice,
            "response_format": "opus",
        },
        stream=True,
    )

We define the URL endpoint, as we’ll be calling it manually without using the client object this time.

We then define the authorization header which will hold our API key to authorize our request to the OpenAI API. Normally the OpenAI library handles sending this header behind the scenes.

Now we make a request using the requests library, doing a post request to the speech endpoint we defined in 'url', sending along our headers, and adding a json object.

The json object contains the arguments we would normally pass to the client.audio.speech.create method, but we also add a “response_format” argument which we set to “opus“.

OPUS is an audio format that was developed for streaming audio over the internet. This alone will cut down the time taken to receive the audio.

Note that finally, we pass stream=True to the request, which is something that is not yet possible with the OpenAI library, though it might be by now if you’re watching this in the future!

This is one of the reasons we’re making a manual API request.

    if response.status_code == 200:
        chunk_size = 4096
        buffer = io.BytesIO()

If the response status code is 200, which means the request was successful, we continue.

We define a chunk_size of 4096 bytes.

We then create a BytesIO object, which is a stream implementation using an in-memory bytes buffer, or basically Python will create a small buffer in memory where we can store our stream.

A quick side note before we move on. We will be writing our audio both to this stream and to a physical file on disk.

This is technically double but it is done to keep this tutorial as simple as possible. If you make a production-grade implementation you will want to get rid of this duplication but for our purposes, it’s not a big deal, and makes it a lot easier to visually show what is going on.

With that out of the way, let’s define an inner function (still inside the stream_audio function) that handles collecting our data:

def stream_audio(...):
    ...
    if response.status_code == 200:
        ...
        ...
        def collect_data():
            with open(buffer_file_location, "wb") as f:
                chunks_written = 0
                for chunk in response.iter_content(chunk_size=chunk_size):
                    buffer.write(chunk)
                    f.write(chunk)
                    chunks_written += 1
                    if chunks_written % 3 == 0:
                        print(f"Buffer size: {buffer.tell()} bytes")

We define a collect_data function that takes no arguments.

We then open a file in write binary mode, catching it under the variable name 'f'.

We define a chunks_written variable and set it to 0 and then loop over the response object’s iter_content method, which will return the response data in chunks of the given chunk_size.

This means we can start looping over the response data before the entire response has been received.

We then write the chunk to our buffer and to the file, incrementing the chunks_written variable by 1. We then check if chunks_written is divisible by 3, just to limit the number of print messages, and if so we print out the current size of our buffer using buffer.tell(). This is just to show you how the buffer is growing as we’re writing to it.

Now that we have a function to collect the data as it comes in and write it to our buffer and file, we need another function to play the audio of unknown length as it is still in the process of coming in.

We’ll call this function play_audio:

def stream_audio(...):
    ...
    if response.status_code == 200:
        ...
        def collect_data():
            ...
        def play_audio():
            with sf.SoundFile(buffer_file_location, "r") as audio_file:
                stream = py_audio.open(
                    format=pyaudio.paInt16,
                    channels=audio_file.channels,
                    rate=audio_file.samplerate,
                    output=True,
                )

                dtype = "int16"
                data = audio_file.read(chunk_size, dtype=dtype)
                while len(data) > 0:
                    stream.write(data.tobytes())
                    data = audio_file.read(chunk_size, dtype=dtype)

                stream.stop_stream()
                stream.close()</