Speech AI

Speech AI is AI that works with the human voice. It turns speech into text (speech recognition) and text into speech (speech synthesis), and some newer models hold a spoken conversation directly. A business uses it to transcribe meetings and customer calls, to add subtitles to videos, or to answer the phone with a voice assistant.

How does speech recognition work?

Speech recognition, also called ASR (automatic speech recognition), turns audio into text. OpenAI's Whisper, released with open code and weights in September 2022, shows how. Whisper learned from 680,000 hours of audio with transcripts collected from the web (Radford et al., 2022). It works on pieces of 30 seconds and turns each into a Mel spectrogram, a picture of which pitches sound when. A transformer then writes the text. The same model can recognise the language and translate speech into English. OpenAI wrote that Whisper approaches human robustness and accuracy on English; its model card reports strong results in about ten languages and lower accuracy in languages with less training data. Cloud services such as Google Cloud Speech-to-Text also transcribe live audio as it comes in.

How does speech synthesis work?

Text-to-speech (TTS) long worked in two steps. In Tacotron 2 (2017), one network turns text into a Mel spectrogram and a WaveNet vocoder turns that into sound. Listeners gave it a mean score of 4.53 out of 5, against 4.58 for professional recordings (Shen et al.). Newer systems copy a voice from a short sample: VALL-E (2023) imitates an unknown speaker from a 3-second recording (Wang et al.).

Can AI hold a spoken conversation?

ChatGPT's first Voice Mode chained three models: one turned speech into text, GPT-3.5 or GPT-4 wrote a reply, and a third read it aloud. It took 2.8 to 5.4 seconds on average to answer and lost tone, background noise and the difference between speakers (OpenAI, May 2024). GPT-4o, launched in May 2024, is one model trained on text, images and audio together. It answers spoken input in 320 milliseconds on average, close to human response time in conversation (GPT-4o System Card).

Where does it go wrong?

• Transcripts can contain text nobody said. In about 1% of Whisper's transcriptions, Koenecke et al. found whole invented phrases or sentences, and 38% of those contained harms such as violence or false associations. They occurred more often when speakers paused for long stretches (Careless Whisper, 2024).
• Accuracy drops for accents, dialects and languages with little training data, according to Whisper's model card.
• A cloned voice can be used to impersonate someone. OpenAI lets GPT-4o speak only in preset voices made with voice actors, and a separate classifier checks that the output keeps to them.

Example

Suppose an accounting firm wants minutes of its client meetings. In September 2026, OpenAI's speech-to-text guide recommends gpt-transcribe for recordings, at $0.0045 per minute (pricing): $0.27 for a one-hour meeting. To mark who said what, it needs gpt-4o-transcribe-diarize at $0.006 per minute, or $0.36. Files may be at most 25 MB, so a long recording may have to be split. A language model then turns the transcript into minutes, and someone checks them before they go out. A firm that wants recordings to stay in the office can run Whisper's open weights on its own computer.