Whisper + GPT-3.5 + Telegram Bot = J.A.R.V.I.S.

Since the release of OpenAI’s Whisper and GPT-3.5, I’ve been thinking about building an assistant just like Iron Man’s J.A.R.V.I.S. It makes total sense, right? Since they released a speech-to-text API and a text-generative API, why not integrate all this to build a personal assistant?

Part of that desire also comes from being frustrated with Amazon’s Alexa “robotic” answers. “Here is what I found on Google.” C’mon! I wanted something more humane! Even knowing that GPT’s answers might not always be true makes it more humane!

So that’s exactly what I did in this article. Just connected the dots! With ~100 lines of Python code, I could create a version of what I would call TalkGPT.

The whole project took less than three hours to build. I’ve also added a /leia command (read, in Portuguese) that reads any given text for me just for fun.

The bot is open source — the repository lives at marciojmo/ausmi. I named it A.U.S.M.I., which stands for “Apenas Um Sistema Muito Inteligente” (Just Another Rather Very Intelligent System, aka J.A.R.V.I.S.), but in Brazilian Portuguese.

Ingredients

• Telegram Bot Token: you need to create a Telegram bot using Telegram’s BotFather.
• OpenAI API Key: you also need an API key from OpenAI to use the Whisper and GPT APIs.
• Python 3: the best language in the world.

The idea is quite simple

1. Receive voice messages on Telegram.
2. Convert voice to text using OpenAI’s Whisper API.
3. Generate a response using OpenAI’s GPT API.
4. Convert the response text into speech using the Google Text-to-Speech (gTTS) library.
5. Send back the spoken answer on Telegram.

Let’s take a closer look at each step.

1. Receive voice messages on Telegram

When our bot receives a voice message, we first download it as ogg (the format used by Telegram). We do that by invoking the download_voice_as_ogg method, which takes a Telegram Voice object and saves it in the desired folder.

async def download_voice_as_ogg(voice):
    voice_file = await voice.get_file()
    ogg_filepath = os.path.join(AUDIOS_DIR, f"{generate_unique_name()}.ogg")
    await voice_file.download_to_drive(ogg_filepath)
    return ogg_filepath

2. Convert voice to text using OpenAI’s Whisper API

To invoke OpenAI’s Whisper, we first need to convert our ogg file into mp3 (ogg is not supported by Whisper yet). To do that, we’ll make use of the pydub library. The code is encapsulated in the convert_ogg_to_mp3 method.

def convert_ogg_to_mp3(ogg_filepath):
    mp3_filepath = os.path.join(AUDIOS_DIR, f"{generate_unique_name()}.mp3")
    audio = pydub.AudioSegment.from_file(ogg_filepath, format="ogg")
    audio.export(mp3_filepath, format="mp3")
    return mp3_filepath

To invoke the Whisper API, we just read the audio file as binary and invoke the API by passing the opened file as an argument. The API response will be a dict that contains the transcribed text in the key text.

def convert_speech_to_text(audio_filepath):
    with open(audio_filepath, "rb") as audio:
        transcript = openai.Audio.transcribe("whisper-1", audio)
        return transcript["text"]

3. Generate a response using OpenAI’s GPT API

After having the transcribed text in hand, we invoke OpenAI’s GPT-3.5-turbo API. The code is encapsulated into the generate_response method.

Note that we could specify the context for GPT on the messages array to impersonate AUSMI, for example, but I’m sending only the text of our message to keep it simple.

def generate_response(text):
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "user", "content": text}
        ]
    )
    answer = response["choices"][0]["message"]["content"]
    return answer

4. Convert the response text into speech using gTTS

With the GPT answer in our hands, we need to convert the answer back into voice and reply to the user. This is done by the method convert_text_to_speech, which uses the Google Text-to-Speech library (gTTS).

Note that the default language code is set to pt. Change that for your own language, or add a new bot command to set that on the fly! This method is also used when we run the command /leia, which reads a text prompt for the user.

def convert_text_to_speech(text, language_code='pt'):
    output_filepath = os.path.join(AUDIOS_DIR, f"{generate_unique_name()}.mp3")
    tts = gtts.gTTS(text=text, lang=language_code)
    tts.save(output_filepath)
    return output_filepath

5. Send back the spoken answer on Telegram

Once we have the audio file, we use the reply_audio method of the Telegram Bot API to send the answer back to the user and delete the audio files from the server. Here is the whole handle_voice method:

async def handle_voice(update: telegram.Update,
                       context: telegram.ext.ContextTypes.DEFAULT_TYPE) -> None:
    ogg_filepath = await download_voice_as_ogg(update.message.voice)
    mp3_filepath = convert_ogg_to_mp3(ogg_filepath)
    transcripted_text = convert_speech_to_text(mp3_filepath)
    answer = generate_response(transcripted_text)
    answer_audio_path = convert_text_to_speech(answer)
    await update.message.reply_audio(audio=answer_audio_path)
    os.remove(ogg_filepath)
    os.remove(mp3_filepath)
    os.remove(answer_audio_path)

That’s it. A simple yet powerful bot.

I hope you enjoyed this article, and may the code be with you!